diff --git a/README.md b/README.md index 8106c00..982f9dc 100644 --- a/README.md +++ b/README.md @@ -77,5 +77,6 @@ This writes `results/real-.md` and `.json` (committed) and per-song output u Caveats: -- The fit summary (matched MIDI onsets, median residual, flagged windows) is a consistency measure between the MIDI and the audio, not ground truth; a high match rate does not prove the alignment is right. +- The fit summary is a consistency measure between the MIDI and the audio, not ground truth: *precision* (share of the recording's detected attacks with a warped MIDI onset within ±70 ms; a 30 s window is flagged below 60 %), *lift* (precision minus its off-grid chance level; flagged below 0.25), a chroma check that flags suspected whole-beat/bar shifts, "MIDI onsets audible as attacks" (information only: a full mix exposes far fewer attacks than the score has notes), and the number of warned stretches (> 5 % and > 15 % tempo change). A high precision does not prove the alignment is right. +- `align auto` also reports a constant-offset baseline (best single offset, −2..+2 s); with no manual anchors it keeps the baseline (one auto anchor) unless the DTW alignment beats its precision by 2 points. - The page plays Ogg Opus; Safari needs 18.4 or later. diff --git a/docs/superpowers/specs/2026-10-30-reference-alignment-design.md b/docs/superpowers/specs/2026-10-30-reference-alignment-design.md index d42ac01..e2a3f89 100644 --- a/docs/superpowers/specs/2026-10-30-reference-alignment-design.md +++ b/docs/superpowers/specs/2026-10-30-reference-alignment-design.md @@ -93,7 +93,7 @@ aligned MIDI. starting from the raw MIDI; a "reset to raw MIDI" action exists. - **Fit quality:** median absolute onset residual (ms) between matched recording onsets and warped MIDI onsets, overall and per 30 s window; shown on the page - and in benchmark output. + and in benchmark output. (Amended 2026-10-30: see "Fit quality — Amended" below.) - **Playback of the warped MIDI:** the server re-renders the warped MIDI audio when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page requests it on Play. Waveforms are sent as precomputed peaks, not full audio. @@ -118,6 +118,49 @@ aligned MIDI. match rate = share of warped MIDI onsets with a recording onset within ±70 ms, overall and per 30 s window. A window is flagged "don't trust" when its match rate < 0.5 or median residual > 60 ms; the page highlights flagged windows. + *(Superseded by the amendment below.)* +- **Fit quality — Amended 2026-10-30 after real-song evidence (rulings A7, A8).** + On real full mixes the onset picker finds ~1.4 attacks/s while a scored MIDI + has ~13 unique onsets/s, so the match rate above caps near 40 % even when + perfectly aligned (every window was flagged on every song), and onset metrics + cannot see whole-beat shifts (they alias onto the grid). + - **Flag metric = precision:** share of recording onsets (onset flux, k = 4, + parabolic sub-frame peak times) with a warped scored-role MIDI onset within + ±70 ms; per 30 s window by recording time. A window with < 8 recording + onsets is null and never flagged; otherwise flagged when precision < 0.6. + - **Chance-corrected lift:** precision minus the mean precision with the + warped MIDI shifted by +0.11, +0.23 and +0.37 s (off-grid); reported per + window and overall; a window is also flagged when lift < 0.25. + - **Beat/bar-shift check:** per window, mean chroma cosine between the + recording (features chroma, 10 Hz) and a symbolic MIDI chroma (piano roll of + the warped pitched scored-role notes, no render) at the current alignment + vs the MIDI moved by ±1, ±2 beats and ±1 bar along the MIDI's own + tempo/time-signature grid. Flagged "beat shift suspected: ±k beat(s)/bar" + when a shift beats the current alignment by > 0.05 (the margin keeps + repetitive harmony, where shifts tie, from flagging). "+1 beat" means the + MIDI content belongs one beat later in the recording. + - **Information only:** the old match rate, renamed "MIDI onsets audible as + attacks" (JSON `audibleRate`), and the median residual (recording onsets → + nearest warped MIDI onset). + - **Stretch warnings:** the fit summary (CLI, page, real report) includes the + number of warned stretches (> 5 % tempo change) and of stretches with + |tempo change| > 15 %. + - **Constant-offset baseline:** grid search −2..+2 s in 10 ms steps for the + offset maximising overall precision; precision peaks within 0.1 of the best + are told apart by the symbolic chroma cosine (beat aliasing), and within a + peak the offset with the smallest mean residual is taken. Reported with its + precision. Without manual anchors, `autoAlign` returns the baseline as one + auto anchor (`chosen: "baseline"`) unless the DTW alignment's precision + beats it by ≥ 0.02 (`chosen: "dtw"`); with manual anchors there is never a + substitution. + - **Aligner guards:** both DTW passes use slope-limited steps (Sakoe–Chiba + P = 1: (1,1), (1,2), (2,1)), so the path cannot jump; the first segment may + start late in the recording (open begin) and the final segment uses an open + end (the path may stop before the recording ends, or before the MIDI ends, + each unconsumed MIDI frame costing 1), so a length mismatch is not absorbed + as a tempo ramp. Per-bar offsets more than ¼ beat from the median of their + ±3 neighbouring bars are rejected (iterated until stable), and no auto + anchor is placed in the MIDI's final partial bar. - **Editing model:** dragging an anchor only re-fits its two neighbouring stretches. Adding or moving a manual anchor deletes the auto anchors in the two affected stretches (those stretches become plain uniform scaling) until the user diff --git a/src/align/auto.ts b/src/align/auto.ts index 27e4f66..12f2e17 100644 --- a/src/align/auto.ts +++ b/src/align/auto.ts @@ -2,12 +2,12 @@ // Pure: no I/O, no Bun/process/fs. The SoundFont bytes are passed in. import type { Audio, Role } from "../core/types.ts"; import type { Smf } from "./smf.ts"; -import { makeWarp, type Anchor, type Warp } from "./warp.ts"; +import { makeWarp, stretches, type Anchor, type Warp } from "./warp.ts"; import { renderMidi } from "./render.ts"; import { features, pickOnsets, type Features } from "./features.ts"; -import { dtw } from "./dtw.ts"; +import { dtw, type DtwOptions } from "./dtw.ts"; import { trackKeys } from "./roles.ts"; -import { fitQuality, type FitReport } from "./fit.ts"; +import { MATCH_TOL_SEC, fitQuality, meanCosine, precisionOf, uniqueSorted, type FitReport, type ShiftInputs, type StretchCounts } from "./fit.ts"; const FINE_RATE = 50; const COARSE_FACTOR = 5; // coarse pass at 10 Hz @@ -16,10 +16,23 @@ const FINE_BANDS_SEC = [0.5, 1, 2]; // fine pass band half-widths tried in turn const ANCHOR_HALF_WIN_SEC = 1; // path points within ±1 s (warped time) of a bar estimate its offset const MIN_POINTS = 10; // diagonal path points required to place a bar anchor const MANUAL_CLEARANCE_SEC = 1; // no auto anchor within 1 s (MIDI time) of a manual anchor +const OUTLIER_BEATS = 0.25; // per-bar offsets further than ¼ beat from the robust median of ±3 neighbours are rejected +const OUTLIER_NEIGHBOURS = 3; +const BASELINE_RANGE_SEC = 2, BASELINE_STEP_SEC = 0.01; // constant-offset baseline grid +const BASELINE_MARGIN = 0.02; // DTW must beat the baseline's precision by this much to be kept (no manual anchors) +const BASELINE_PLATEAU = 0.05; // within a precision peak, offsets this close to its best are refined by residual +const BASELINE_TIE = 0.1; // precision peaks this close to the best are told apart by symbolic chroma (beat aliasing) +const OVER_TEMPO = 0.15; // stretches with |tempo change| above this are counted separately in the fit const ZERO = 1e-9; -/** Bar start times (MIDI seconds) from the tempo map and time signatures (4/4 when none), up to durationSec. */ -export function barTimes(smf: Smf): number[] { +export interface MeterGrid { beats: number[]; bars: number[]; /** end of the last bar */ end: number; /** the MIDI ends before its last bar does */ lastBarPartial: boolean } + +/** + * Beat and bar start times (MIDI seconds) from the tempo map and time signatures (4/4 when none; a beat is one + * denominator note), for bars starting before `untilSec` (default: the MIDI duration). A signature change mid-bar + * restarts the bar there. + */ +export function meterGrid(smf: Smf, untilSec = smf.durationSec): MeterGrid { const tempos = smf.tempoMap.length ? [...smf.tempoMap].sort((a, b) => a.sec - b.sec) : [{ sec: 0, bpm: 120 }]; const sigs = [...smf.timeSigs].sort((a, b) => a.sec - b.sec); const bpmAt = (t: number) => { let b = tempos[0]!.bpm; for (const p of tempos) if (p.sec <= t + ZERO) b = p.bpm; return b; }; @@ -34,19 +47,78 @@ export function barTimes(smf: Smf): number[] { } return t; }; - const out: number[] = []; + const beats: number[] = [], bars: number[] = []; let t = 0; - while (t < smf.durationSec - ZERO && out.length < 100000) { - out.push(t); + while (t < untilSec - ZERO && bars.length < 100000) { + bars.push(t); let num = 4, den = 4; for (const s of sigs) if (s.sec <= t + ZERO) { num = s.num; den = s.den; } - let next = advance(t, (num * 4) / den); - const change = sigs.find((s) => s.sec > t + ZERO && s.sec < next - ZERO); // signature change mid-bar: restart there - if (change) next = change.sec; - if (!(next > t)) break; - t = next; + const change = sigs.find((s) => s.sec > t + ZERO)?.sec ?? Infinity; + let b = t; + for (let k = 0; k < num; k++) { + beats.push(b); + const nb = advance(b, 4 / den); + if (nb >= change - ZERO) { b = change; break; } // signature change mid-bar: restart there + b = nb; + } + if (!(b > t)) break; + t = b; + } + return { beats, bars, end: t, lastBarPartial: bars.length > 0 && t > smf.durationSec + 1e-3 }; +} + +/** Bar start times (MIDI seconds), up to durationSec. */ +export const barTimes = (smf: Smf): number[] => meterGrid(smf).bars; +/** Beat start times (MIDI seconds) of the bars starting before durationSec. */ +export const beatTimes = (smf: Smf): number[] => meterGrid(smf).beats; + +/** Time `k` grid steps after `t` on a sorted grid (fractional positions; linear extrapolation outside it). */ +function gridShift(grid: number[], t: number, k: number): number { + const n = grid.length; + if (n < 2) return t; + const posOf = (x: number): number => { + if (x <= grid[0]!) return (x - grid[0]!) / (grid[1]! - grid[0]!); + if (x >= grid[n - 1]!) return n - 1 + (x - grid[n - 1]!) / (grid[n - 1]! - grid[n - 2]!); + let lo = 0, hi = n - 1; + while (hi - lo > 1) { const m = (lo + hi) >> 1; if (grid[m]! <= x) lo = m; else hi = m; } + return lo + (x - grid[lo]!) / (grid[hi]! - grid[lo]!); + }; + const timeOf = (p: number): number => { + if (p <= 0) return grid[0]! + p * (grid[1]! - grid[0]!); + if (p >= n - 1) return grid[n - 1]! + (p - n + 1) * (grid[n - 1]! - grid[n - 2]!); + const i = Math.floor(p); + return grid[i]! + (p - i) * (grid[i + 1]! - grid[i]!); + }; + return timeOf(posOf(t) + k); +} + +const med = (xs: number[]): number => { const s = [...xs].sort((a, b) => a - b), h = s.length >> 1; return s.length % 2 ? s[h]! : (s[h - 1]! + s[h]!) / 2; }; + +/** + * Per-bar offsets (recording − warped, s) at MIDI time `midiSec`, with each bar's beat length (s): iteratively drops + * the bar deviating most from the robust median of its ±3 kept neighbours' offsets, while that deviation exceeds ¼ + * beat. The neighbours' offsets are transported to the bar along their median pairwise (Theil–Sen) slope, so a smooth + * ramp (a genuine tempo difference) is kept whole instead of being eroded from its ends, where a plain one-sided median + * lags the ramp; beat/bar jumps are still rejected. + */ +export function rejectOffsetOutliers(bars: T[]): T[] { + const kept = [...bars]; + for (;;) { + let worst = -1, worstDev = 0; + for (let k = 0; k < kept.length; k++) { + const nb = [...kept.slice(Math.max(0, k - OUTLIER_NEIGHBOURS), k), ...kept.slice(k + 1, k + 1 + OUTLIER_NEIGHBOURS)]; + if (!nb.length) continue; + const slopes: number[] = []; + for (let a = 0; a < nb.length; a++) for (let b = a + 1; b < nb.length; b++) slopes.push((nb[b]!.off - nb[a]!.off) / (nb[b]!.midiSec - nb[a]!.midiSec)); + const slope = slopes.length ? med(slopes) : 0; + const x = kept[k]!.midiSec; + const pred = med(nb.map((b) => b.off + slope * (x - b.midiSec))); + const dev = Math.abs(kept[k]!.off - pred) / kept[k]!.beatSec; + if (dev > OUTLIER_BEATS && dev > worstDev) { worst = k; worstDev = dev; } + } + if (worst < 0) return kept; + kept.splice(worst, 1); } - return out; } /** Aggregate 50 Hz features by `k` frames (centred): chroma energy summed and re-normalised, onset max, re-z-scored. */ @@ -91,19 +163,38 @@ function costFn(A: Features, B: Features): (i: number, j: number) => number { }; } -type Path = [number, number][]; -/** Align rendered frames [i0, i1] with recording frames [j0, j1] (inclusive, fine rate), coarse then banded fine. */ -function alignSegment(fine: [Features, Features], coarse: [Features, Features], i0: number, i1: number, j0: number, j1: number, bandsSec: number[]): Path { +type Path = [number, number][]; +type Open = { openBegin: boolean; openEnd: boolean }; + +/** Slope-limited DTW; falls back to the unconstrained symmetric steps when the slope limit makes the end unreachable. */ +function dtwSlope(costAt: (i: number, j: number) => number, n: number, m: number, open: Open, band?: { center: (i: number) => number; width: number }): Path { + const opts: DtwOptions = { steps: "slope", ...open }; + try { + return dtw(costAt, n, m, band, opts); + } catch (e) { + if (band || !(e instanceof Error) || !/unreachable/.test(e.message)) throw e; + return dtw(costAt, n, m, undefined, { ...open, steps: "symmetric" }); + } +} + +/** + * Align rendered frames [i0, i1] with recording frames [j0, j1] (inclusive, fine rate), coarse then banded fine, both + * slope-limited. `open.openBegin`/`openEnd` let the path start after j0 / stop before j1 (or before i1). + */ +function alignSegment(fine: [Features, Features], coarse: [Features, Features], i0: number, i1: number, j0: number, j1: number, bandsSec: number[], open: Open): Path { const K = COARSE_FACTOR; const ci0 = Math.round(i0 / K), ci1 = Math.max(ci0, Math.min(coarse[0].chroma.length - 1, Math.round(i1 / K))); const cj0 = Math.round(j0 / K), cj1 = Math.max(cj0, Math.min(coarse[1].chroma.length - 1, Math.round(j1 / K))); const cc = costFn(coarse[0], coarse[1]); - const cp = dtw((i, j) => cc(ci0 + i, cj0 + j), ci1 - ci0 + 1, cj1 - cj0 + 1); - // coarse map: mean recording frame per rendered frame, in segment-relative coarse units - const sum = new Float64Array(ci1 - ci0 + 1), cnt = new Float64Array(ci1 - ci0 + 1); + const cp = dtwSlope((i, j) => cc(ci0 + i, cj0 + j), ci1 - ci0 + 1, cj1 - cj0 + 1, open); + // coarse map: mean recording frame per rendered frame, in segment-relative coarse units; rows the path does not + // reach (open end on the last column) continue at slope 1 + const nRows = ci1 - ci0 + 1; + const sum = new Float64Array(nRows), cnt = new Float64Array(nRows); for (const [i, j] of cp) { sum[i]! += j; cnt[i]! += 1; } - const cmap = Array.from(sum, (s, i) => s / cnt[i]!); + const cmap = Array.from(sum, (v, i) => (cnt[i] ? v / cnt[i]! : NaN)); + for (let i = 1; i < nRows; i++) if (Number.isNaN(cmap[i]!)) cmap[i] = cmap[i - 1]! + 1; const centre = (i: number): number => { // fine row (segment-relative) -> fine column (segment-relative) const x = Math.min(cmap.length - 1, Math.max(0, (i0 + i) / K - ci0)); const a = Math.floor(x), b = Math.min(cmap.length - 1, a + 1); @@ -111,7 +202,7 @@ function alignSegment(fine: [Features, Features], coarse: [Features, Features], return (y + cj0) * K - j0; }; const fc = costFn(fine[0], fine[1]); - const p = bandedWithRetry((i, j) => fc(i0 + i, j0 + j), i1 - i0 + 1, j1 - j0 + 1, centre, bandsSec); + const p = bandedWithRetry((i, j) => fc(i0 + i, j0 + j), i1 - i0 + 1, j1 - j0 + 1, centre, bandsSec, open); if (p) return p.map(([i, j]) => [i0 + i, j0 + j]); return densify(cp, ci0, cj0, i0, i1, j0, j1); } @@ -119,19 +210,20 @@ function alignSegment(fine: [Features, Features], coarse: [Features, Features], /** * Fallback: the coarse path expanded to a contiguous fine path from (i0, j0) to (i1, j1). Each coarse step becomes K * fine steps in the same direction (so coarse diagonals stay diagonal and still yield anchors), clamped to the - * segment; any remainder at the end is closed with horizontal/vertical steps. + * segment; the lead-in to the coarse path's first cell and any remainder at the end use horizontal/vertical steps + * (which carry no timing). */ function densify(cp: Path, ci0: number, cj0: number, i0: number, i1: number, j0: number, j1: number): Path { const K = COARSE_FACTOR; const out: Path = [[i0, j0]]; let i = i0, j = j0; - // the coarse grid origin may sit up to K/2 fine frames off (i0, j0); align to it first const step = (di: number, dj: number) => { const ni = Math.min(i1, i + di), nj = Math.min(j1, j + dj); if (ni !== i || nj !== j) { i = ni; j = nj; out.push([i, j]); } }; - while (i < Math.min(i1, ci0 * K)) step(1, 0); - while (j < Math.min(j1, cj0 * K)) step(0, 1); + const [si, sj] = cp[0] ?? [0, 0]; // the coarse path may start off the corner (open begin) + while (i < Math.min(i1, (ci0 + si) * K)) step(1, 0); + while (j < Math.min(j1, (cj0 + sj) * K)) step(0, 1); for (let k = 1; k < cp.length; k++) { const di = cp[k]![0] - cp[k - 1]![0], dj = cp[k]![1] - cp[k - 1]![1]; for (let q = 0; q < K; q++) step(di, dj); @@ -140,13 +232,14 @@ function densify(cp: Path, ci0: number, cj0: number, i0: number, i1: number, j0: return out; } -/** Banded fine DTW, widening the band (±0.5 s, ±1 s, ±2 s) when the end cell is unreachable; null if all fail. */ +/** Banded fine DTW (slope-limited), widening the band (±0.5 s, ±1 s, ±2 s) when the end is unreachable; null if all fail. */ export function bandedWithRetry( costAt: (i: number, j: number) => number, n: number, m: number, centre: (i: number) => number, bandsSec = FINE_BANDS_SEC, + open: Open = { openBegin: false, openEnd: false }, ): Path | null { for (const sec of bandsSec) { try { - return dtw(costAt, n, m, { center: centre, width: Math.round(sec * FINE_RATE) }); + return dtw(costAt, n, m, { center: centre, width: Math.round(sec * FINE_RATE) }, { steps: "slope", ...open }); } catch (e) { if (!(e instanceof Error) || !/unreachable/.test(e.message)) throw e; } @@ -189,38 +282,153 @@ function compose(manual: Anchor[], auto: Anchor[]): Anchor[] { const durationOf = (a: Audio) => (a.channels[0]?.length ?? 0) / a.sampleRate; -/** Fit of `warp` against recording onsets: warped onsets of scored-role notes (roles keyed by `trackKeys`). */ -function fitFor(recOnsets: number[], smf: Smf, roles: Record, warp: Warp, durSec: number): FitReport { +/** What the fit needs from the recording: onset-flux peaks (s) and 10 Hz chroma. Compute once per song. */ +export interface RecordingCues { onsets: number[]; chroma: Float32Array[]; chromaRate: number; durSec: number } + +function cuesFrom(fA: Features, cA: Features, durSec: number): RecordingCues { + return { onsets: pickOnsets(fA.onset, FINE_RATE), chroma: cA.chroma, chromaRate: cA.rate, durSec }; +} + +/** Onset-flux peaks and chroma of the raw recording. Compute once and reuse with `fitFromCues` to re-score edits cheaply. */ +export function recordingCues(audio: Audio): RecordingCues { + const fA = features(audio, FINE_RATE); + return cuesFrom(fA, downsample(fA, COARSE_FACTOR), durationOf(audio)); +} + +/** Scored-role tracks (by `trackKeys`). */ +function scoredTracks(smf: Smf, roles: Record) { const keys = trackKeys(smf.tracks); - const midi = smf.tracks.flatMap((t, i) => (roles[keys[i]!] ? t.notes.map((n) => warp.toAudio(n.onset)) : [])); - return fitQuality(recOnsets, midi, durSec); + return smf.tracks.filter((_, i) => roles[keys[i]!]); } -/** Onset-flux peaks of the raw recording (seconds). Compute once and reuse with `fitFromOnsets` to re-score edits cheaply. */ -export function recordingOnsets(audio: Audio): number[] { - return pickOnsets(features(audio, FINE_RATE).onset, FINE_RATE); +/** Piano-roll chroma (pitch classes of sounding pitched notes, 1 per note) with MIDI time placed by `place`. */ +function symbolicChroma(tracks: Smf["tracks"], place: (m: number) => number, nFrames: number, rate: number): Float32Array[] { + const out = Array.from({ length: nFrames }, () => new Float32Array(12)); + for (const t of tracks) { + if (t.isDrums) continue; + for (const n of t.notes) { + const a = Math.max(0, Math.round(place(n.onset) * rate)); + const b = Math.min(nFrames, Math.max(a + 1, Math.round(place(n.offset) * rate))); + for (let f = a; f < b; f++) out[f]![n.pitch % 12]! += 1; + } + } + return out; } -/** `alignmentFit` with precomputed recording onsets; `durSec` = recording duration. */ -export function fitFromOnsets(onsets: number[], smf: Smf, roles: Record, anchors: Anchor[], durSec: number): FitReport { - return fitFor(onsets, smf, roles, makeWarp(anchors), durSec); +/** Warped MIDI onsets of scored-role notes. */ +const scoredOnsets = (tracks: Smf["tracks"], place: (m: number) => number) => tracks.flatMap((t) => t.notes.map((n) => place(n.onset))); + +/** Fit of `anchors`: precision/lift/audible rate, the beat/bar-shift chroma check and stretch-warning counts. */ +export function fitFromCues(cues: RecordingCues, smf: Smf, roles: Record, anchors: Anchor[]): FitReport { + const warp = makeWarp(anchors); + const tracks = scoredTracks(smf, roles); + const nFrames = cues.chroma.length, rate = cues.chromaRate; + const g = meterGrid(smf, smf.durationSec + 30); + const barSteps = (m: number, k: number) => gridShift(g.bars, m, k); + const beatSteps = (m: number, k: number) => gridShift(g.beats, m, k); + // "+k beat(s)": the MIDI content belongs k beats later in the recording than the current alignment puts it + const cands: [string, (m: number) => number][] = [ + ["+1 beat", (m) => beatSteps(m, 1)], ["−1 beat", (m) => beatSteps(m, -1)], + ["+2 beats", (m) => beatSteps(m, 2)], ["−2 beats", (m) => beatSteps(m, -2)], + ["+1 bar", (m) => barSteps(m, 1)], ["−1 bar", (m) => barSteps(m, -1)], + ]; + const shift: ShiftInputs = { + rate, recording: cues.chroma, + current: symbolicChroma(tracks, warp.toAudio, nFrames, rate), + shifts: cands.map(([label, f]) => ({ label, chroma: symbolicChroma(tracks, (m) => warp.toAudio(f(m)), nFrames, rate) })), + }; + const st = stretches(anchors, smf.durationSec); + const counts: StretchCounts = { total: st.length, warned: st.filter((s) => s.warn).length, over15: st.filter((s) => Math.abs(1 / s.ratio - 1) > OVER_TEMPO).length }; + return fitQuality(cues.onsets, scoredOnsets(tracks, warp.toAudio), cues.durSec, { shift, stretches: counts }); } -/** Fit quality of any alignment (e.g. a hand-made one): onset-flux peaks of the raw recording vs the warped MIDI. */ +/** Fit quality of any alignment (e.g. a hand-made one) against the raw recording. */ export function alignmentFit(audio: Audio, smf: Smf, roles: Record, anchors: Anchor[]): FitReport { - return fitFromOnsets(recordingOnsets(audio), smf, roles, anchors, durationOf(audio)); + return fitFromCues(recordingCues(audio), smf, roles, anchors); } +/** Best constant offset (recording − MIDI, s) and its precision; `cosine` = mean symbolic-chroma cosine there. */ +export interface Baseline { offsetSec: number; precision: number | null; cosine: number | null } + +/** + * Constant-offset baseline: grid search of −2..+2 s in 10 ms steps maximising precision. Whole-beat shifts alias onto + * the onset grid (equal precision), so the precision peaks within 0.1 of the best are told apart by the mean chroma + * cosine of the symbolic MIDI vs the recording. Within a peak precision is flat across the ±70 ms tolerance, so the + * offset with the smallest mean residual (among those within 0.05 of the peak's precision) is reported. + */ +export function constantOffsetBaseline(cues: RecordingCues, smf: Smf, roles: Record): Baseline { + const tracks = scoredTracks(smf, roles); + const midi = uniqueSorted(scoredOnsets(tracks, (m) => m)); + const steps = Math.round(BASELINE_RANGE_SEC / BASELINE_STEP_SEC); + const offs = Array.from({ length: 2 * steps + 1 }, (_, k) => Math.round((k - steps) * BASELINE_STEP_SEC * 100) / 100); + const prec = offs.map((o) => precisionOf(cues.onsets, midi, o)); + if (prec.every((p) => p === null)) return { offsetSec: 0, precision: null, cosine: null }; + const best = Math.max(...prec.map((p) => p ?? -1)); + // peaks: runs of consecutive offsets within the tie margin + const runs: number[][] = []; + for (let k = 0; k < offs.length; k++) { + if ((prec[k] ?? -1) < best - BASELINE_TIE) continue; + if (k > 0 && (prec[k - 1] ?? -1) >= best - BASELINE_TIE) runs[runs.length - 1]!.push(k); else runs.push([k]); + } + // within a run, precision is flat across the ±70 ms tolerance (with spikes at its edges, where neighbouring notes + // start to match): take the offset with the smallest mean residual among those within 0.05 of the run's best + const refine = (run: number[]): number => { + const top = Math.max(...run.map((k) => prec[k]!)); + let pick = run[0]!, bestRes = Infinity; + for (const k of run) { + if (prec[k]! < top - BASELINE_PLATEAU) continue; + const r = meanResidual(cues.onsets, midi, offs[k]!); + if (r < bestRes) { bestRes = r; pick = k; } + } + return pick; + }; + const cosAt = (o: number) => meanCosine(cues.chroma, symbolicChroma(tracks, (m) => m + o, cues.chroma.length, cues.chromaRate), 0, cues.chroma.length, 0); + let pick = refine(runs[0]!), pickCos = runs.length > 1 ? cosAt(offs[pick]!) : null; + for (const run of runs.slice(1)) { + const k = refine(run), c = cosAt(offs[k]!); + if (c !== null && (pickCos === null || c > pickCos)) { pick = k; pickCos = c; } + } + return { offsetSec: offs[pick]!, precision: prec[pick]!, cosine: pickCos ?? cosAt(offs[pick]!) }; +} + +/** Mean |recording onset − nearest (MIDI + shift)| over recording onsets matched within ±70 ms. */ +function meanResidual(rec: number[], midi: number[], shift: number): number { + let s = 0, n = 0; + for (const t of rec) { + const x = t - shift; + let lo = 0, hi = midi.length; + while (lo < hi) { const m = (lo + hi) >> 1; if (midi[m]! < x) lo = m + 1; else hi = m; } + const d = Math.min(lo < midi.length ? Math.abs(midi[lo]! - x) : Infinity, lo > 0 ? Math.abs(midi[lo - 1]! - x) : Infinity); + if (d <= MATCH_TOL_SEC + 1e-9) { s += d; n++; } + } + return n ? s / n : Infinity; +} + +/** A single auto anchor realising a constant offset: at the bar nearest the middle of the MIDI (positive audio time). */ +function baselineAnchors(smf: Smf, offsetSec: number): Anchor[] { + const bars = barTimes(smf).filter((b) => b + offsetSec > 0.01); + const mid = smf.durationSec / 2; + const m = bars.length ? bars.reduce((a, b) => (Math.abs(b - mid) < Math.abs(a - mid) ? b : a)) : Math.max(0, -offsetSec) + 1; + return [{ midiSec: m, audioSec: m + offsetSec, kind: "auto" }]; +} + +/** `dtwAnchors`/`dtwPrecision`: the DTW alignment and its precision, also when the baseline replaced it. */ +export interface AutoResult { anchors: Anchor[]; fit: FitReport; baseline: Baseline; chosen: "baseline" | "dtw"; dtwAnchors: Anchor[]; dtwPrecision: number | null } + /** * Align `smf` to `audio`, honouring `manual` anchors exactly. The MIDI is rendered under the manual warp, then each - * stretch between consecutive manual anchors (plus start/end) is aligned by a 10 Hz DTW refined at 50 Hz in a band - * around it. One auto anchor is placed per bar (skipping bars within 1 s of a manual anchor). `roles` is keyed by - * `trackKeys`; tracks without a role are rendered but excluded from the fit. + * stretch between consecutive manual anchors (plus start/end) is aligned by a slope-limited 10 Hz DTW refined at 50 Hz + * in a band around it; the first stretch may start late in the recording (open begin) and the last one may end early + * on either axis (open end), so extra intro/outro material does not force the path. One auto anchor is placed per bar + * (skipping bars within 1 s of a manual anchor and the final partial bar); per-bar offsets more than ¼ beat from the + * median of their ±3 neighbours are rejected. Without manual anchors the result is replaced by the constant-offset + * baseline (one auto anchor) unless it beats the baseline's precision by ≥ 0.02. `roles` is keyed by `trackKeys`; + * tracks without a role are rendered but excluded from the fit. */ export function autoAlign( audio: Audio, smf: Smf, roles: Record, manual: Anchor[], sf2: Uint8Array, opts: { fineBandsSec?: number[] } = {}, // band half-widths for the fine pass; [] forces the coarse-path fallback -): { anchors: Anchor[]; fit: FitReport } { +): AutoResult { const man = makeWarp(manual).anchors.map((a) => ({ ...a, kind: "manual" as const })); const W0 = makeWarp(man); const audioDur = durationOf(audio); @@ -229,17 +437,18 @@ export function autoAlign( const fR = features(rendered, FINE_RATE), fA = features(audio, FINE_RATE); const cR = downsample(fR, COARSE_FACTOR), cA = downsample(fA, COARSE_FACTOR); const lastR = fR.chroma.length - 1, lastA = fA.chroma.length - 1; + const midiEnd = Math.max(1, Math.min(lastR, Math.round(W0.toAudio(smf.durationSec) * FINE_RATE))); // last rendered frame with MIDI content // segment boundaries: manual anchors map warped time == recording time, so both axes split at the same instants - const cuts = man.map((a) => Math.round(a.audioSec * FINE_RATE)).filter((f) => f > 0 && f < Math.min(lastR, lastA)); + const cuts = man.map((a) => Math.round(a.audioSec * FINE_RATE)).filter((f) => f > 0 && f < Math.min(midiEnd, lastA)); const bounds = [...new Set(cuts)].sort((a, b) => a - b); - // per segment: warped-time span and diagonal path steps (they carry timing; horizontal/vertical runs are stalls, - // silence or unmatched material and do not) + // per segment: warped-time span and diagonal path steps (they carry timing; horizontal/vertical steps do not) const segs: { lo: number; hi: number; diag: { w: number; off: number }[] }[] = []; let prev = 0; for (const b of [...bounds, -1]) { - const i1 = b < 0 ? lastR : b, j1 = b < 0 ? lastA : b; - const path = alignSegment([fR, fA], [cR, cA], prev, i1, prev, j1, opts.fineBandsSec ?? FINE_BANDS_SEC); + const i1 = b < 0 ? midiEnd : b, j1 = b < 0 ? lastA : b; + const open = { openBegin: prev === 0, openEnd: b < 0 }; + const path = alignSegment([fR, fA], [cR, cA], prev, i1, prev, j1, opts.fineBandsSec ?? FINE_BANDS_SEC, open); const diag: { w: number; off: number }[] = []; for (let k = 1; k < path.length; k++) { const [i, j] = path[k]!, [pi, pj] = path[k - 1]!; @@ -249,8 +458,12 @@ export function autoAlign( prev = b; } - const auto: Anchor[] = []; - for (const m of barTimes(smf)) { + const grid = meterGrid(smf); + const lastBar = grid.lastBarPartial ? grid.bars[grid.bars.length - 1] : undefined; + const beatAt = (m: number) => { const k = grid.beats.findIndex((b) => b > m + ZERO); return k > 0 ? grid.beats[k]! - grid.beats[k - 1]! : 0.5; }; + const perBar: { midiSec: number; w: number; off: number; beatSec: number }[] = []; + for (const m of grid.bars) { + if (m === lastBar) continue; // never anchor the final partial bar if (man.some((a) => Math.abs(a.midiSec - m) < MANUAL_CLEARANCE_SEC)) continue; const w = W0.toAudio(m); const seg = segs.find((s) => w < s.hi) ?? segs[segs.length - 1]!; @@ -258,9 +471,25 @@ export function autoAlign( if (off === null) continue; const a = w + off; if ((m > 0 ? a <= 0 : a < 0) || a > audioDur) continue; // no degenerate first stretch - auto.push({ midiSec: m, audioSec: a, kind: "auto" }); + perBar.push({ midiSec: m, w, off, beatSec: beatAt(m) * (W0.toAudio(m + 1) - w) }); } + const auto = rejectOffsetOutliers(perBar).map((p): Anchor => ({ midiSec: p.midiSec, audioSec: p.w + p.off, kind: "auto" })); const anchors = compose(man, auto); - const fit = fitFor(pickOnsets(fA.onset, FINE_RATE), smf, roles, makeWarp(anchors), audioDur); - return { anchors, fit }; + const cues = cuesFrom(fA, cA, audioDur); + const fit = fitFromCues(cues, smf, roles, anchors); + const baseline = constantOffsetBaseline(cues, smf, roles); + const beats = fit.precision !== null && baseline.precision !== null && fit.precision >= baseline.precision + BASELINE_MARGIN; + if (man.length || baseline.precision === null || beats) return { anchors, fit, baseline, chosen: "dtw", dtwAnchors: anchors, dtwPrecision: fit.precision }; + const b = baselineAnchors(smf, baseline.offsetSec); + return { anchors: b, fit: fitFromCues(cues, smf, roles, b), baseline, chosen: "baseline", dtwAnchors: anchors, dtwPrecision: fit.precision }; +} + +/** e.g. "DTW (precision 93.5%) · constant-offset baseline +0.06 s: precision 88.0%"; with "chosen: baseline" the + * single-anchor constant offset replaced a DTW alignment that was not better by 2 points. */ +export function autoSummary(r: { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null }): string { + const pc = (x: number | null) => (x === null ? "n/a" : `${(x * 100).toFixed(1)}%`); + const off = `${r.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(r.baseline.offsetSec).toFixed(2)} s`; + const base = `constant-offset baseline ${off}: precision ${pc(r.baseline.precision)}`; + return r.chosen === "dtw" ? `chosen: DTW (precision ${pc(r.dtwPrecision)}) · ${base}` + : `chosen: baseline (DTW precision ${pc(r.dtwPrecision)} not better by ≥ 2 points) · ${base}`; } diff --git a/src/align/cli.ts b/src/align/cli.ts index f2b84cd..3dd2c92 100644 --- a/src/align/cli.ts +++ b/src/align/cli.ts @@ -1,7 +1,7 @@ import { parseArgs } from "node:util"; import { addSong, loadSong, readAlignment, trackKeys, writeAlignment } from "../ref/manifest.ts"; import { parseSmf } from "./smf.ts"; -import { autoAlign } from "./auto.ts"; +import { autoAlign, autoSummary } from "./auto.ts"; import { fitSummary } from "./fit.ts"; import { SF2_PATH, startServer } from "./server.ts"; @@ -51,8 +51,9 @@ async function auto(argv: string[]) { const sf2 = new Uint8Array(await Bun.file(SF2_PATH).arrayBuffer()); const t0 = performance.now(); const r = autoAlign(s.audio, s.smf, roles, manual, sf2); - await writeAlignment(id, { anchors: r.anchors, savedAt: new Date().toISOString(), fit: r.fit }, root); + await writeAlignment(id, { anchors: r.anchors, savedAt: new Date().toISOString(), fit: r.fit, auto: { chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision } }, root); console.log(`${id}: ${r.anchors.length} anchors (${manual.length} manual kept) in ${((performance.now() - t0) / 1000).toFixed(1)} s -> ${root}/${id}/alignment.json`); + console.log(`method: ${autoSummary(r)}`); console.log(`fit: ${fitSummary(r.fit)}`); } diff --git a/src/align/dtw.ts b/src/align/dtw.ts index bd36d5a..b5782d7 100644 --- a/src/align/dtw.ts +++ b/src/align/dtw.ts @@ -1,18 +1,33 @@ -// Pure (banded) dynamic time warping. Only the two latest accumulated-cost rows are kept; backpointers are one byte -// per cell (O(n·band) when banded), so a 3000×3000 unbanded run uses ≈ 9 MB. +// Pure (banded) dynamic time warping. Only the latest accumulated-cost rows are kept; backpointers are one byte per +// cell (O(n·band) when banded), so a 3000×3000 unbanded run uses ≈ 9 MB. export interface DtwBand { center: (i: number) => number; width: number } +/** + * steps: "symmetric" (default) — steps (1,0), (0,1), (1,1), each paying the cost of the cell entered. + * "slope" — slope-limited (Sakoe–Chiba P = 1, asymmetric weights): (1,1), (1,2) via (i, j−1), (2,1) via + * (i−1, j). The local slope stays within [1/2, 2], so the path cannot make abrupt jumps; each row (query frame) + * carries weight 1, so paths that consume all rows are comparable whatever columns they cover. + * openBegin: the path may start anywhere on row 0 (the target may have extra material before the query). + * openEnd: the path may end anywhere on the last row or the last column (target or query has extra material at the + * end); candidates are compared by mean cost per query row, each query row left unconsumed costing 1 (the + * cost of sound vs silence), so a trivial path into a cheap corner cannot win. + */ +export interface DtwOptions { steps?: "symmetric" | "slope"; openBegin?: boolean; openEnd?: boolean } -const DIAG = 0, UP = 1, LEFT = 2; // from (i-1,j-1), (i-1,j), (i,j-1) +const DIAG = 0, UP = 1, LEFT = 2, S12 = 3, S21 = 4; // from (i-1,j-1), (i-1,j), (i,j-1), (i-1,j-2) via (i,j-1), (i-2,j-1) via (i-1,j) +const START = 5; +const UNCONSUMED_ROW_COST = 1; -/** Minimum-cost monotone path from (0,0) to (n-1,m-1) with steps (1,0), (0,1), (1,1). */ +/** Minimum-cost monotone path from (0,0) to (n-1,m-1) (or open ends, see `DtwOptions`). */ export function dtw( costAt: (i: number, j: number) => number, n: number, m: number, band?: DtwBand, + opts: DtwOptions = {}, ): [number, number][] { if (n <= 0 || m <= 0) return []; + const slope = opts.steps === "slope", openBegin = !!opts.openBegin, openEnd = !!opts.openEnd; const lo = new Int32Array(n); const hi = new Int32Array(n); const off = new Int32Array(n + 1); @@ -22,8 +37,8 @@ export function dtw( const c = Math.round(band.center(i)); a = Math.max(0, c - band.width); b = Math.min(m - 1, c + band.width); - if (i === 0) a = 0; - if (i === n - 1) b = m - 1; + if (i === 0 && !openBegin) a = 0; + if (i === n - 1 && !openEnd) b = m - 1; if (a > b) { // centre outside the matrix: clamp to the nearest edge a = b = Math.min(m - 1, Math.max(0, c)); } @@ -32,36 +47,71 @@ export function dtw( off[i + 1] = off[i]! + (b - a + 1); } const bp = new Uint8Array(off[n]!); - let prev = new Float64Array(m).fill(Infinity); - let cur = new Float64Array(m).fill(Infinity); + // accumulated cost rows (row i in D[i % 3]) and cell-cost rows (row i in C[i % 2]); full width, cleared by range + const D = [0, 1, 2].map(() => new Float64Array(m).fill(Infinity)); + const C = [0, 1].map(() => new Float64Array(m)); + const inRow = (r: number, j: number) => r >= 0 && j >= lo[r]! && j <= hi[r]!; + const acc = (r: number, j: number) => (inRow(r, j) ? D[r % 3]![j]! : Infinity); + // open-end candidates: mean cost per query row, each row left unconsumed (ending on the last column) costing 1 + let endScore = Infinity, endI = n - 1, endJ = m - 1; + const consider = (i: number, v: number, j: number) => { + const score = (v + (n - 1 - i) * UNCONSUMED_ROW_COST) / n; + if (v < Infinity && score < endScore - 1e-12) { endScore = score; endI = i; endJ = j; } + }; for (let i = 0; i < n; i++) { const a = lo[i]!, b = hi[i]!, base = off[i]! - a; - const pa = i > 0 ? lo[i - 1]! : 0, pb = i > 0 ? hi[i - 1]! : -1; - cur.fill(Infinity, Math.max(0, a - 1), Math.min(m, b + 2)); + const cur = D[i % 3]!, cc = C[i % 2]!, pc = C[(i + 1) % 2]!; + if (i >= 3) cur.fill(Infinity, lo[i - 3]!, hi[i - 3]! + 1); + for (let j = a; j <= b; j++) cc[j] = costAt(i, j); for (let j = a; j <= b; j++) { let best = Infinity, dir = DIAG; - if (i === 0 && j === 0) best = 0; - else { - if (i > 0 && j > 0 && j - 1 >= pa && j - 1 <= pb) { best = prev[j - 1]!; dir = DIAG; } - if (i > 0 && j >= pa && j <= pb && prev[j]! < best) { best = prev[j]!; dir = UP; } + const d = cc[j]!; + if (i === 0 && (j === 0 || openBegin)) { best = 0; dir = START; } + else if (slope) { + const dg = acc(i - 1, j - 1); + if (dg + d < best) { best = dg + d; dir = DIAG; } + const h = acc(i - 1, j - 2); + if (h < Infinity) { + const v = h + ((j - 1 >= a ? cc[j - 1]! : costAt(i, j - 1)) + d) / 2; + if (v < best) { best = v; dir = S12; } + } + const u = acc(i - 2, j - 1); + if (u < Infinity) { + const v = u + (inRow(i - 1, j) ? pc[j]! : costAt(i - 1, j)) + d; + if (v < best) { best = v; dir = S21; } + } + } else { + const dg = acc(i - 1, j - 1); + if (dg < best) { best = dg; dir = DIAG; } + const up = acc(i - 1, j); + if (up < best) { best = up; dir = UP; } if (j > a && cur[j - 1]! < best) { best = cur[j - 1]!; dir = LEFT; } + if (best < Infinity) best += d; } if (best === Infinity) { cur[j] = Infinity; continue; } - cur[j] = best + costAt(i, j); + cur[j] = dir === START ? d : best; bp[base + j] = dir; } - // clear stale values outside this row's range so the next row never reads them - if (i > 0) prev.fill(Infinity, pa, pb + 1); - [prev, cur] = [cur, prev]; + if (openEnd && i < n - 1 && b === m - 1) consider(i, cur[m - 1]!, m - 1); } - if (prev[m - 1]! === Infinity) throw new Error("dtw: end cell unreachable within band"); + // choose the end cell (last-column candidates were collected during the sweep) + let ei = n - 1, ej = m - 1; + if (openEnd) { + for (let j = lo[n - 1]!; j <= hi[n - 1]!; j++) consider(n - 1, acc(n - 1, j), j); + if (!(endScore < Infinity)) throw new Error("dtw: end cell unreachable within band"); + ei = endI; ej = endJ; + } else if (acc(n - 1, m - 1) === Infinity) throw new Error("dtw: end cell unreachable within band"); const path: [number, number][] = []; - let i = n - 1, j = m - 1; + let i = ei, j = ej; for (;;) { path.push([i, j]); - if (i === 0 && j === 0) break; const d = bp[off[i]! - lo[i]! + j]!; - if (d === DIAG) { i--; j--; } else if (d === UP) i--; else j--; + if (d === START) break; + if (d === DIAG) { i--; j--; } + else if (d === UP) i--; + else if (d === LEFT) j--; + else if (d === S12) { path.push([i, j - 1]); i--; j -= 2; } + else { path.push([i - 1, j]); i -= 2; j--; } } return path.reverse(); } diff --git a/src/align/features.ts b/src/align/features.ts index 2b86020..2996703 100644 --- a/src/align/features.ts +++ b/src/align/features.ts @@ -108,7 +108,11 @@ export function features(audio: Audio, rate: number): Features { return { rate, onset, chroma }; } -/** Peak-pick an onset-strength curve (adaptive threshold median + k·MAD, 50 ms minimum gap). Times in seconds. */ +/** + * Peak-pick an onset-strength curve (adaptive threshold median + k·MAD, 50 ms minimum gap, the stronger peak wins). + * Peak times are refined to sub-frame precision by parabolic interpolation over the peak and its two neighbours. + * Times in seconds. + */ export function pickOnsets(onset: Float32Array, rate: number, k = 4): number[] { const n = onset.length; if (n === 0) return []; @@ -118,15 +122,20 @@ export function pickOnsets(onset: Float32Array, rate: number, k = 4): number[] { const mad = dev[n >> 1]! * 1.4826; const thr = med + k * Math.max(mad, 1e-3); const minGap = 0.05; - const picked: number[] = []; + const frames: number[] = []; for (let i = 0; i < n; i++) { const v = onset[i]!; if (v <= thr || v < (onset[i - 1] ?? -Infinity) || v <= (onset[i + 1] ?? -Infinity)) continue; - const t = i / rate; - const last = picked.length - 1; - if (last >= 0 && t - picked[last]! < minGap - 1e-9) { - if (v > onset[Math.round(picked[last]! * rate)]!) picked[last] = t; - } else picked.push(t); + const last = frames.length - 1; + if (last >= 0 && (i - frames[last]!) / rate < minGap - 1e-9) { + if (v > onset[frames[last]!]!) frames[last] = i; + } else frames.push(i); } - return picked; + return frames.map((i) => { + if (i === 0 || i === n - 1) return i / rate; + const a = onset[i - 1]!, b = onset[i]!, c = onset[i + 1]!; + const den = a - 2 * b + c; + const d = den < 0 ? Math.max(-0.5, Math.min(0.5, (0.5 * (a - c)) / den)) : 0; + return (i + d) / rate; + }); } diff --git a/src/align/fit.ts b/src/align/fit.ts index 7877d3b..12bb5e1 100644 --- a/src/align/fit.ts +++ b/src/align/fit.ts @@ -1,13 +1,58 @@ -// Fit quality of an alignment: how well warped MIDI onsets land on onsets picked from the raw recording. +// Fit quality of an alignment: how well onsets picked from the raw recording land on the warped MIDI onsets. // Pure: no I/O. Recording onsets must come from the audio itself (onset flux), never from transcription. +// +// Calibrated on real mixes (ruling A7, spec "Fit quality", amended 2026-10-30): the onset picker finds ~1.4 attacks/s +// in a full mix while a scored MIDI has ~13 unique onsets/s, so the share of MIDI onsets that are heard ("audible +// rate", the old match rate) is capped near 40% even when perfectly aligned and carries no information. The flag +// metric is therefore *precision*: the share of recording onsets with a warped MIDI onset within ±70 ms. Because a +// dense score matches almost anything, precision is chance-corrected by the precision at off-grid shifts (lift), and a +// symbolic chroma check catches whole-beat / whole-bar shifts that onset metrics cannot see (they alias onto the grid). -export interface FitWindow { startSec: number; matchRate: number | null; medianResidualMs: number | null; flagged: boolean } -export interface FitReport { matchRate: number | null; medianResidualMs: number | null; windows: FitWindow[] } +/** A beat/bar shift that explains the recording's chroma better than the current alignment. */ +export interface BeatShift { label: string; gain: number } +export interface FitWindow { + startSec: number; + /** Recording onsets in this window (by recording time). */ + recOnsets: number; + /** Share of this window's recording onsets with a warped MIDI onset within ±70 ms; null with < 8 recording onsets. */ + precision: number | null; + /** precision − mean precision at the off-grid shifts (chance-corrected); null when precision is null. */ + lift: number | null; + /** Information only (JSON `audibleRate`): share of unique MIDI onsets (in this window by warped time) heard as attacks. */ + audibleRate: number | null; + /** Median |recording onset − nearest warped MIDI onset| over matched recording onsets (ms). */ + medianResidualMs: number | null; + beatShift: BeatShift | null; + flagged: boolean; + /** Why the window is flagged, e.g. "precision 41%", "lift 0.12", "beat shift suspected: +1 beat". */ + reasons: string[]; +} +export interface StretchCounts { total: number; warned: number; over15: number } +export interface FitReport { + precision: number | null; + lift: number | null; + /** Information only: share of unique MIDI onsets heard as attacks ("MIDI onsets audible as attacks"). */ + audibleRate: number | null; + medianResidualMs: number | null; + windows: FitWindow[]; + /** Stretches of the alignment: total, warned (> 5% tempo change) and with |tempo change| > 15%. */ + stretches?: StretchCounts; +} + +/** Symbolic-chroma inputs for the beat/bar-shift check: frames at `rate` Hz in recording time, 12-d (any scale). */ +export interface ShiftInputs { rate: number; recording: Float32Array[]; current: Float32Array[]; shifts: { label: string; chroma: Float32Array[] }[] } export const MATCH_TOL_SEC = 0.07; export const WINDOW_SEC = 30; -const FLAG_MIN_RATE = 0.5; -const FLAG_MAX_RESIDUAL_MS = 60; +export const FLAG_MIN_PRECISION = 0.6; +export const FLAG_MIN_LIFT = 0.25; +export const MIN_REC_ONSETS = 8; +/** Off-grid shifts (s) of the warped MIDI used as the chance level of precision. */ +export const CHANCE_SHIFTS_SEC = [0.11, 0.23, 0.37]; +/** A shifted alignment must beat the current one by this much mean chroma cosine to flag the window. */ +export const SHIFT_MARGIN = 0.05; +/** The current symbolic chroma must cover at least this share of the window's sounding frames for the check to run. */ +const SHIFT_MIN_COVER = 0.25; const UNIQUE_EPS = 1e-6; function median(xs: number[]): number | null { @@ -16,52 +61,126 @@ function median(xs: number[]): number | null { return s.length % 2 ? s[m]! : (s[m - 1]! + s[m]!) / 2; } -/** |residual| in seconds to the nearest recording onset, or null when none lies within ±70 ms. */ -function nearest(sorted: number[], t: number): number | null { +/** |distance| from `t` to the nearest value in `sorted` (Infinity when empty). */ +function nearestDist(sorted: number[], t: number): number { let lo = 0, hi = sorted.length; while (lo < hi) { const m = (lo + hi) >> 1; if (sorted[m]! < t) lo = m + 1; else hi = m; } let best = Infinity; if (lo < sorted.length) best = Math.min(best, Math.abs(sorted[lo]! - t)); if (lo > 0) best = Math.min(best, Math.abs(sorted[lo - 1]! - t)); - return best <= MATCH_TOL_SEC + 1e-9 ? best : null; + return best; +} +const hit = (d: number) => d <= MATCH_TOL_SEC + 1e-9; + +/** Sorted, de-duplicated copy. */ +export function uniqueSorted(xs: number[]): number[] { + const out: number[] = []; + for (const t of [...xs].sort((a, b) => a - b)) if (!out.length || t - out[out.length - 1]! > UNIQUE_EPS) out.push(t); + return out; } -function summarise(residuals: (number | null)[]): { matchRate: number | null; medianResidualMs: number | null } { - if (!residuals.length) return { matchRate: null, medianResidualMs: null }; - const hits = residuals.filter((r): r is number => r !== null); - const med = median(hits); - return { matchRate: hits.length / residuals.length, medianResidualMs: med === null ? null : med * 1000 }; +/** Share of `rec` (any order) with a value of sorted `midi` + `shift` within ±70 ms; null when `rec` or `midi` is empty. */ +export function precisionOf(rec: number[], midi: number[], shift = 0): number | null { + if (!rec.length || !midi.length) return null; + let n = 0; + for (const t of rec) if (hit(nearestDist(midi, t - shift))) n++; + return n / rec.length; +} + +function cos(a: Float32Array, b: Float32Array): number { + let d = 0, na = 0, nb = 0; + for (let q = 0; q < 12; q++) { d += a[q]! * b[q]!; na += a[q]! * a[q]!; nb += b[q]! * b[q]!; } + return na > 1e-12 && nb > 1e-12 ? d / Math.sqrt(na * nb) : 0; +} + +/** Mean chroma cosine over the sounding recording frames of [f0, f1); null when the candidate covers too little. */ +export function meanCosine(rec: Float32Array[], cand: Float32Array[], f0: number, f1: number, minCover = SHIFT_MIN_COVER): number | null { + let sum = 0, sounding = 0, covered = 0; + for (let t = f0; t < Math.min(f1, rec.length); t++) { + const r = rec[t]!; + if (!r.some((v) => v !== 0)) continue; + sounding++; + const c = cand[t]; + if (!c || !c.some((v) => v !== 0)) continue; + covered++; + sum += cos(r, c); + } + if (!sounding || covered / sounding < minCover) return null; + return sum / sounding; +} + +function shiftCheck(s: ShiftInputs, f0: number, f1: number): BeatShift | null { + const cur = meanCosine(s.recording, s.current, f0, f1); + if (cur === null) return null; + let best: BeatShift | null = null; + for (const c of s.shifts) { + const v = meanCosine(s.recording, c.chroma, f0, f1, 0); + if (v !== null && v - cur > SHIFT_MARGIN && (!best || v - cur > best.gain)) best = { label: c.label, gain: v - cur }; + } + return best; +} + +const pct = (x: number) => `${Math.round(x * 100)}%`; + +/** + * Recording onsets are matched to the nearest warped MIDI onset within ±70 ms (non-exclusive). Overall and per 30 s + * window (by recording onset time): precision, lift (precision − mean precision with the MIDI shifted by +0.11, +0.23, + * +0.37 s), median residual; plus the audible rate (info) and, with `extra.shift`, the beat/bar-shift chroma check. + * A window with < 8 recording onsets is null and unflagged; otherwise it is flagged when precision < 0.6, lift < 0.25, + * or a shifted alignment beats the current one by > 0.05 mean chroma cosine. + */ +export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number, extra: { shift?: ShiftInputs; stretches?: StretchCounts } = {}): FitReport { + const rec = [...audioOnsets].sort((a, b) => a - b); + const midi = uniqueSorted(midiOnsets); + const nWin = Math.max(1, Math.ceil(Math.max(durSec, (rec[rec.length - 1] ?? 0) + 1e-9) / WINDOW_SEC)); + const winOf = (t: number) => Math.min(nWin - 1, Math.max(0, Math.floor(t / WINDOW_SEC))); + const recWin: number[][] = Array.from({ length: nWin }, () => []); + for (const t of rec) recWin[winOf(t)]!.push(t); + const midiHit = midi.map((t) => hit(nearestDist(rec, t))); + + const stats = (r: number[]) => { + const precision = midi.length ? precisionOf(r, midi) : null; + if (precision === null) return { precision: null, lift: null, medianResidualMs: null }; + const chance = CHANCE_SHIFTS_SEC.map((s) => precisionOf(r, midi, s)!); + const dists = r.map((t) => nearestDist(midi, t)).filter(hit); + const med = median(dists); + return { precision, lift: precision - chance.reduce((a, b) => a + b, 0) / chance.length, medianResidualMs: med === null ? null : med * 1000 }; + }; + const audible = (lo: number, hi: number) => { + let n = 0, h = 0; + midi.forEach((t, k) => { if (t >= lo && t < hi) { n++; if (midiHit[k]) h++; } }); + return n ? h / n : null; + }; + + const windows = recWin.map((r, w): FitWindow => { + const startSec = w * WINDOW_SEC; + const lo = w === 0 ? -Infinity : startSec, hi = w === nWin - 1 ? Infinity : startSec + WINDOW_SEC; + const s = r.length >= MIN_REC_ONSETS ? stats(r) : { precision: null, lift: null, medianResidualMs: null }; + const beatShift = extra.shift && s.precision !== null + ? shiftCheck(extra.shift, Math.round(startSec * extra.shift.rate), w === nWin - 1 ? Infinity : Math.round((startSec + WINDOW_SEC) * extra.shift.rate)) + : null; + const reasons: string[] = []; + if (s.precision !== null && s.precision < FLAG_MIN_PRECISION) reasons.push(`precision ${pct(s.precision)}`); + if (s.lift !== null && s.lift < FLAG_MIN_LIFT) reasons.push(`lift ${s.lift.toFixed(2)}`); + if (beatShift) reasons.push(`beat shift suspected: ${beatShift.label}`); + return { startSec, recOnsets: r.length, ...s, audibleRate: audible(lo, hi), beatShift, flagged: reasons.length > 0, reasons }; + }); + const all = rec.length && midi.length ? stats(rec) : { precision: null, lift: null, medianResidualMs: null }; + return { ...all, audibleRate: audible(-Infinity, Infinity), windows, ...(extra.stretches ? { stretches: extra.stretches } : {}) }; } /** - * Match each unique MIDI onset (already warped to audio time) to the nearest recording onset within ±70 ms - * (non-exclusive). Reports match rate and median |residual| overall and per 30 s window (by MIDI onset time). - * A window is flagged when its match rate < 0.5 or its median residual > 60 ms; empty windows are null, unflagged. + * One line, e.g. "precision 91.2% (lift +0.70) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms · + * 1/8 windows flagged (0:30 beat shift suspected: +1 beat) · stretches: 3/40 warned, 1 > 15%". */ -export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number): FitReport { - const audio = [...audioOnsets].sort((a, b) => a - b); - const midi: number[] = []; - for (const t of [...midiOnsets].sort((a, b) => a - b)) if (!midi.length || t - midi[midi.length - 1]! > UNIQUE_EPS) midi.push(t); - const res = midi.map((t) => nearest(audio, t)); - - const lastOnset = midi.length ? midi[midi.length - 1]! : 0; - const nWin = Math.max(1, Math.ceil(Math.max(durSec, lastOnset + 1e-9) / WINDOW_SEC)); - const buckets: (number | null)[][] = Array.from({ length: nWin }, () => []); - midi.forEach((t, k) => buckets[Math.min(nWin - 1, Math.max(0, Math.floor(t / WINDOW_SEC)))]!.push(res[k]!)); - const windows = buckets.map((b, w): FitWindow => { - const s = summarise(b); - const flagged = s.matchRate !== null && (s.matchRate < FLAG_MIN_RATE || (s.medianResidualMs ?? 0) > FLAG_MAX_RESIDUAL_MS); - return { startSec: w * WINDOW_SEC, ...s, flagged }; - }); - return { ...summarise(res), windows }; -} - -/** One-line summary, e.g. "82.3% of MIDI onsets matched · median residual 14.2 ms · 2/9 windows flagged (0:30, 2:00)". */ export function fitSummary(f: FitReport): string { - if (f.matchRate === null) return "no scored MIDI onsets"; + if (f.precision === null) return f.audibleRate === null ? "no scored MIDI onsets" : "no recording onsets"; const mmss = (s: number) => `${Math.floor(s / 60)}:${String(Math.floor(s % 60)).padStart(2, "0")}`; const flagged = f.windows.filter((w) => w.flagged); const med = f.medianResidualMs === null ? "n/a" : `${f.medianResidualMs.toFixed(1)} ms`; - const list = flagged.length ? ` (${flagged.map((w) => mmss(w.startSec)).join(", ")})` : ""; - return `${(f.matchRate * 100).toFixed(1)}% of MIDI onsets matched · median residual ${med} · ${flagged.length}/${f.windows.length} windows flagged${list}`; + const lift = f.lift === null ? "" : ` (lift ${f.lift >= 0 ? "+" : "−"}${Math.abs(f.lift).toFixed(2)})`; + const aud = f.audibleRate === null ? "n/a" : `${(f.audibleRate * 100).toFixed(1)}%`; + const list = flagged.length ? ` (${flagged.map((w) => `${mmss(w.startSec)} ${w.reasons.join(", ")}`).join("; ")})` : ""; + const st = f.stretches ? ` · stretches: ${f.stretches.warned}/${f.stretches.total} warned, ${f.stretches.over15} > 15%` : ""; + return `precision ${(f.precision * 100).toFixed(1)}%${lift} · MIDI onsets audible as attacks ${aud} · median residual ${med} · ${flagged.length}/${f.windows.length} windows flagged${list}${st}`; } diff --git a/src/align/page/app.ts b/src/align/page/app.ts index 6de729c..6b73166 100644 --- a/src/align/page/app.ts +++ b/src/align/page/app.ts @@ -255,14 +255,23 @@ async function runBusy(label: string, f: () => Promise) { try { await f(); } catch (e) { setStatus((e as Error).message, true); } finally { busy = false; showOverlay(null); updateUi(); } } +interface AutoInfo { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null } +const pctOf = (x: number | null) => (x === null ? "n/a" : `${(x * 100).toFixed(1)}%`); +/** "DTW, precision 91.0% vs constant offset +0.06 s 88.0%" or "constant offset +0.06 s (precision 91.0%; DTW 89.5% was not better)". */ +function autoNote(a: AutoInfo): string { + const off = `${a.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(a.baseline.offsetSec).toFixed(2)} s`; + return a.chosen === "dtw" ? `DTW, precision ${pctOf(a.dtwPrecision)} vs constant offset ${off} ${pctOf(a.baseline.precision)}` + : `constant offset ${off} (precision ${pctOf(a.baseline.precision)}; DTW ${pctOf(a.dtwPrecision)} was not better by 2 points)`; +} + const autoAlign = () => runBusy("Auto-aligning around your anchors… (a few seconds to half a minute)", async () => { const manual = anchors.filter((a) => a.kind === "manual"); - const j = await http<{ anchors: Anchor[]; fit: FitReport }>(songUrl("/auto"), send("POST", { anchors: manual })); + const j = await http<{ anchors: Anchor[]; fit: FitReport; auto: AutoInfo }>(songUrl("/auto"), send("POST", { anchors: manual })); selected = -1; commit(j.anchors, anchors, true); fit = j.fit; fitKey = keyOf(j.anchors); updateUi(); - setStatus(`Auto-aligned: ${j.anchors.length - manual.length} automatic anchors around your ${manual.length}. Not saved yet.`); + setStatus(`Auto-aligned: ${j.anchors.length - manual.length} automatic anchors around your ${manual.length} (${autoNote(j.auto)}). Not saved yet.`); }); const save = () => runBusy("Saving…", async () => { @@ -357,7 +366,7 @@ function updateUi() { fitEl.classList.toggle("stale", stale); if (fit) { fitEl.textContent = `Fit${stale ? " (updating…)" : ""}: ${fitSummary(fit)}`; - fitEl.classList.toggle("bad", (fit.matchRate ?? 1) < 0.5 || fit.windows.some((w) => w.flagged)); + fitEl.classList.toggle("bad", fit.windows.some((w) => w.flagged)); } const man = anchors.filter((a) => a.kind === "manual").length; countsEl.textContent = `· anchors: ${man} yours, ${anchors.length - man} automatic`; diff --git a/src/align/server.ts b/src/align/server.ts index 6a4e19e..458a3f0 100644 --- a/src/align/server.ts +++ b/src/align/server.ts @@ -8,7 +8,7 @@ import type { Role } from "../core/types.ts"; import { audioOgg, loadSong, readAlignment, readManifest, writeAlignment, type AlignmentFile, type LoadedSong } from "../ref/manifest.ts"; import { trackKeys } from "./roles.ts"; import { makeWarp, stretches, type Anchor } from "./warp.ts"; -import { autoAlign, fitFromOnsets, recordingOnsets } from "./auto.ts"; +import { autoAlign, fitFromCues, recordingCues, type AutoResult, type RecordingCues } from "./auto.ts"; import { renderMidi } from "./render.ts"; import { peaks } from "./peaks.ts"; import { stretchInfo } from "./tempo.ts"; @@ -32,9 +32,9 @@ export interface AlignServer { url: string; origin: string; token: string; port: interface SongState extends LoadedSong { roles: Record; - onsets: number[]; + cues: RecordingCues; peaks: { bucketSec: number; min: number[]; max: number[] }; - auto?: { anchors: Anchor[]; fit: FitReport }; // first automatic pass, reused while no alignment.json exists + auto?: AutoResult; // first automatic pass, reused while no alignment.json exists } class HttpError extends Error { constructor(readonly status: number, msg: string) { super(msg); } } @@ -98,7 +98,7 @@ export async function startServer(opts: ServeOptions = {}): Promise const s = await loadSong(id, root); const keys = trackKeys(s.smf.tracks); const roles = Object.fromEntries(keys.map((k) => [k, s.entry.roles[k] ?? null])); - const onsets = recordingOnsets(s.audio); + const cues = recordingCues(s.audio); const peaksFile = Bun.file(join(root, id, "peaks.json")); const recWav = Bun.file(join(root, id, "recording.wav")); let pk: SongState["peaks"] | null = null; @@ -114,7 +114,7 @@ export async function startServer(opts: ServeOptions = {}): Promise await Bun.write(peaksFile, JSON.stringify({ version: PEAKS_VERSION, peaks: pk })); } log(`loaded ${id} (${s.audioDur.toFixed(1)} s) in ${((performance.now() - t0) / 1000).toFixed(1)} s`); - return { ...s, roles, onsets, peaks: pk }; + return { ...s, roles, cues, peaks: pk }; } function song(id: string): Promise { @@ -127,17 +127,19 @@ export async function startServer(opts: ServeOptions = {}): Promise return p; } - const fitOf = (s: SongState, anchors: Anchor[]) => fitFromOnsets(s.onsets, s.smf, s.roles, anchors, s.audioDur); + const fitOf = (s: SongState, anchors: Anchor[]) => fitFromCues(s.cues, s.smf, s.roles, anchors); const stretchesOf = (s: SongState, anchors: Anchor[]) => stretchInfo(stretches(anchors, s.smf.durationSec), s.smf.tempoMap); const state = (s: SongState, anchors: Anchor[], fit = fitOf(s, anchors)) => ({ anchors, stretches: stretchesOf(s, anchors), fit }); function runAuto(s: SongState, manual: Anchor[]) { const t0 = performance.now(); const r = autoAlign(s.audio, s.smf, s.roles, manual, sf2); - log(`auto-aligned ${s.entry.id}: ${r.anchors.length} anchors (${manual.length} manual) in ${((performance.now() - t0) / 1000).toFixed(1)} s`); + log(`auto-aligned ${s.entry.id}: ${r.anchors.length} anchors (${manual.length} manual, ${r.chosen}) in ${((performance.now() - t0) / 1000).toFixed(1)} s`); return r; } + const autoInfo = (r: AutoResult) => ({ chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision }); + /** Saved alignment with validated anchors; an unreadable/invalid file yields `error` instead of failing the request. */ async function savedAlignment(id: string): Promise<{ file: AlignmentFile | null; error?: string }> { try { @@ -175,7 +177,7 @@ export async function startServer(opts: ServeOptions = {}): Promise id, title: s.entry.title, audioDur: s.audioDur, midiDur: s.smf.durationSec, tracks: trackKeys(s.smf.tracks).map((name) => ({ name, role: s.roles[name] ?? null })), recordingPeaks: s.peaks, ...state(s, anchors, fit), midiTempo: s.smf.tempoMap, - source: saved ? "saved" : "auto", savedAt: saved?.savedAt ?? null, + source: saved ? "saved" : "auto", savedAt: saved?.savedAt ?? null, ...(saved ? {} : { auto: autoInfo(s.auto!) }), ...(error ? { warning: `${error}. Showing the automatic alignment instead; saving will overwrite the file.` } : {}), }); } @@ -208,7 +210,7 @@ export async function startServer(opts: ServeOptions = {}): Promise const manual = parseAnchors(await readJson(req)).filter((a) => a.kind === "manual"); const s = await song(id); const r = runAuto(s, manual); - return Response.json(state(s, r.anchors, r.fit)); + return Response.json({ ...state(s, r.anchors, r.fit), auto: autoInfo(r) }); } if (route === "POST reset") return Response.json(state(await song(id), [])); if (route === "PUT alignment") { diff --git a/src/align/warp.ts b/src/align/warp.ts index c005864..ffe5ac2 100644 --- a/src/align/warp.ts +++ b/src/align/warp.ts @@ -23,7 +23,9 @@ function sortedCopy(anchors: Anchor[]): Anchor[] { const ratioOf = (a: Anchor, b: Anchor) => (b.audioSec - a.audioSec) / (b.midiSec - a.midiSec); -// Slopes used beyond the outermost anchors: the nearest interior stretch, or 1 with fewer than two anchors. +// Slopes used beyond the outermost anchors: the nearest interior stretch, or 1 with fewer than two anchors. So +// before the first anchor the map extrapolates with the *first* stretch's slope (there is no implicit 0 -> 0 anchor: +// a warp whose first anchor is (m0, a0) maps MIDI 0 to a0 − m0·slope₀), and after the last anchor with the last one's. function edgeRatios(A: Anchor[]): { first: number; last: number } { if (A.length < 2) return { first: 1, last: 1 }; return { first: ratioOf(A[0]!, A[1]!), last: ratioOf(A[A.length - 2]!, A[A.length - 1]!) }; @@ -43,6 +45,11 @@ function piecewise(X: number[], Y: number[], x: number, first: number, last: num return Y[lo]! + ((x - X[lo]!) * (Y[hi]! - Y[lo]!)) / (X[hi]! - X[lo]!); } +/** + * Monotonic piecewise-linear MIDI <-> recording map through the anchors. Outside the anchors it extrapolates linearly: + * before the first anchor with the first stretch's slope, after the last anchor with the last stretch's slope (slope + * 1 with a single anchor, identity with none). + */ export function makeWarp(anchors: Anchor[]): Warp { const A = sortedCopy(anchors); const midi = A.map((a) => a.midiSec), audio = A.map((a) => a.audioSec); diff --git a/src/bench/real.ts b/src/bench/real.ts index 9af3b29..d96981e 100644 --- a/src/bench/real.ts +++ b/src/bench/real.ts @@ -7,12 +7,20 @@ import { matchNotes, type MatchResult } from "../eval/notes.ts"; import { DRUM_CLASSES, ROLE_ORDER, drumClass, toClassPitch, type DrumClass } from "../eval/score.ts"; import { readManifest, readAlignment, loadSong, trackKeys } from "../ref/manifest.ts"; import { makeWarp, type Anchor } from "../align/warp.ts"; -import { autoAlign, alignmentFit } from "../align/auto.ts"; +import { autoAlign, autoSummary, constantOffsetBaseline, fitFromCues, recordingCues, type Baseline } from "../align/auto.ts"; import { fitSummary, type FitReport } from "../align/fit.ts"; export interface RealSongResult { id: string; title: string; - alignment: { source: "manual" | "auto"; manual: number; auto: number; savedAt: string | null }; + /** + * source: "saved" = reference//alignment.json, "computed" = auto-aligned this run (--auto). chosen: how the auto + * alignment was made ("dtw" or the constant-offset "baseline"), null when unknown (saved from the page). + * baseline: the constant-offset baseline computed this run, for comparison. + */ + alignment: { + source: "saved" | "computed"; manual: number; auto: number; savedAt: string | null; + chosen: "baseline" | "dtw" | null; baseline: Baseline | { offsetSec: number; precision: number | null }; dtwPrecision: number | null; + }; fit: FitReport; perRole: Partial>; /** F1 at the lenient +-100 ms tolerance, per role (null when the role has no notes on either side). */ @@ -37,19 +45,37 @@ export async function runReal(opts: { runId: string; auto: boolean; root?: strin }); const sf2 = opts.auto ? new Uint8Array(await Bun.file(SF2).arrayBuffer()) : null; const results: RealSongResult[] = []; - const skipped: string[] = []; + const skipped: SkippedSong[] = []; + const skip = (id: string, reason: string) => { skipped.push({ id, reason }); console.log(`${id}: skipped (${reason})`); }; for (const entry of songs) { - const saved = await readAlignment(entry.id, root); - if (!saved && !opts.auto) { skipped.push(entry.id); console.log(`${entry.id}: skipped (no alignment.json; pass --auto to compute one)`); continue; } + let saved: Awaited>; + try { + saved = await readAlignment(entry.id, root); + if (saved) makeWarp(saved.anchors); // validates the anchors + } catch (e) { skip(entry.id, (e as Error).message); continue; } + if (!saved && !opts.auto) { skip(entry.id, "no alignment.json; pass --auto to compute one"); continue; } const t0 = performance.now(); const { smf, audio } = await loadSong(entry.id, root); const keys = trackKeys(smf.tracks); const roleOf = (i: number): Role | null => entry.roles[keys[i]!] ?? null; + const roles = Object.fromEntries(keys.map((k) => [k, entry.roles[k] ?? null])); - let anchors: Anchor[], savedAt: string | null, source: "manual" | "auto"; - if (saved) { anchors = saved.anchors; savedAt = saved.savedAt ?? null; source = "manual"; } - else { anchors = autoAlign(audio, smf, entry.roles, [], sf2!).anchors; savedAt = null; source = "auto"; } - const fit = alignmentFit(audio, smf, entry.roles, anchors); + const cues = recordingCues(audio); + let anchors: Anchor[], alignment: RealSongResult["alignment"]; + if (saved) { + anchors = saved.anchors; + alignment = { + source: "saved", manual: 0, auto: 0, savedAt: saved.savedAt ?? null, + chosen: saved.auto?.chosen ?? null, baseline: constantOffsetBaseline(cues, smf, roles), dtwPrecision: saved.auto?.dtwPrecision ?? null, + }; + } else { + const r = autoAlign(audio, smf, roles, [], sf2!); + anchors = r.anchors; + alignment = { source: "computed", manual: 0, auto: 0, savedAt: null, chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision }; + } + alignment.manual = anchors.filter((a) => a.kind === "manual").length; + alignment.auto = anchors.filter((a) => a.kind === "auto").length; + const fit = fitFromCues(cues, smf, roles, anchors); const warp = makeWarp(anchors); const res = await pipeline({ mix: audio }, { oracleStems: false, oracleTempo: false, oracleNotes: false }); @@ -77,7 +103,7 @@ export async function runReal(opts: { runId: string; auto: boolean; root?: strin const seconds = (performance.now() - t0) / 1000; const result: RealSongResult = { id: entry.id, title: entry.title, - alignment: { source, manual: anchors.filter((a) => a.kind === "manual").length, auto: anchors.filter((a) => a.kind === "auto").length, savedAt }, + alignment, fit, perRole, lenient, drumsPerClass, excludedTracks: smf.tracks.map((t, i) => (roleOf(i) ? null : keys[i]!)).filter((k): k is string => k !== null), capo: entry.capo ?? {}, seconds, @@ -100,16 +126,25 @@ const NA = "n/a"; const fmt = (x: number | null | undefined) => (x == null ? NA : x.toFixed(2)); const row = (cells: string[]) => `| ${cells.join(" | ")} |`; -export function renderRealReport(runId: string, results: RealSongResult[], skipped: string[] = []): string { +export interface SkippedSong { id: string; reason: string } + +export function renderRealReport(runId: string, results: RealSongResult[], skipped: SkippedSong[] = []): string { const o: string[] = [`# STSheets real-song benchmark: ${runId}`, ""]; o.push("> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). " + - "The fit line reports the *consistency of the alignment* (warped MIDI onsets vs onset-flux peaks of the recording), not an independent validation of it: " + - "a poor alignment lowers every score below. Capo is informational and not applied.", ""); + "The fit line reports the *consistency of the alignment*, not an independent validation of it: precision = share of the recording's onset-flux peaks with a warped MIDI onset within ±70 ms " + + "(flag < 60% per 30 s window), lift = precision minus its value with the MIDI shifted off-grid (flag < 0.25), a symbolic-chroma check for whole-beat/bar shifts, " + + "and \"MIDI onsets audible as attacks\" (information only; a full mix exposes far fewer attacks than the score has notes). " + + "A poor alignment lowers every score below. Capo is informational and not applied.", ""); for (const r of results) { const a = r.alignment; - const src = a.source === "auto" ? `auto (${a.manual} manual / ${a.auto} auto anchors, computed this run)` : `saved (${a.manual} manual / ${a.auto} auto anchors, saved ${a.savedAt ?? "unknown"})`; + const counts = `${a.manual} manual / ${a.auto} auto anchors`; + const src = a.source === "computed" ? `computed this run (${counts})` : `saved ${a.savedAt ?? "(time unknown)"} (${counts})`; + const pc = (x: number | null) => (x === null ? NA : `${(x * 100).toFixed(1)}%`); + const off = `${a.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(a.baseline.offsetSec).toFixed(2)} s`; + const method = a.chosen ? autoSummary({ chosen: a.chosen, baseline: a.baseline, dtwPrecision: a.dtwPrecision }) + : `method unknown (saved from the page) · constant-offset baseline ${off}: precision ${pc(a.baseline.precision)}`; o.push(`## ${r.title} (${r.id})`, ""); - o.push(`- Alignment: ${src}`, `- Consistency of the alignment: ${fitSummary(r.fit)}`); + o.push(`- Alignment: ${src}`, `- Method: ${method}`, `- Consistency of the alignment: ${fitSummary(r.fit)}`); const capos = Object.entries(r.capo); o.push(`- Capo: ${capos.length ? capos.map(([k, v]) => `${k}: ${v}`).join(", ") : "none listed"}`); o.push(`- Excluded tracks (no role): ${r.excludedTracks.length ? r.excludedTracks.join(", ") : "none"}`, `- Pipeline + scoring: ${r.seconds.toFixed(1)} s`, ""); @@ -128,6 +163,6 @@ export function renderRealReport(runId: string, results: RealSongResult[], skipp o.push(""); } } - if (skipped.length) o.push(`Skipped (no alignment.json): ${skipped.join(", ")}`, ""); + if (skipped.length) o.push(`Skipped: ${skipped.map((s) => `${s.id} (${s.reason})`).join("; ")}`, ""); return o.join("\n"); } diff --git a/src/ref/manifest.ts b/src/ref/manifest.ts index c0e8aef..ae9f9f1 100644 --- a/src/ref/manifest.ts +++ b/src/ref/manifest.ts @@ -45,7 +45,9 @@ export async function addSong(e: Omit & { roles?: SongEntry[ // ---- per-song caches and alignment file (reference//) ---- -export interface AlignmentFile { anchors: Anchor[]; savedAt: string; fit: FitReport | null } +/** `auto`: how `bun run align auto` produced the anchors (absent for alignments saved from the page). */ +export interface AlignmentFile { anchors: Anchor[]; savedAt: string; fit: FitReport | null; auto?: AutoInfo } +export interface AutoInfo { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null } const inflight = new Map>(); diff --git a/tests/align/auto.test.ts b/tests/align/auto.test.ts index ae5aeed..d33d954 100644 --- a/tests/align/auto.test.ts +++ b/tests/align/auto.test.ts @@ -1,5 +1,5 @@ import { test, expect, beforeAll } from "bun:test"; -import { alignmentFit, autoAlign, bandedWithRetry, barTimes, fitFromOnsets, recordingOnsets } from "../../src/align/auto.ts"; +import { alignmentFit, autoAlign, bandedWithRetry, barTimes, beatTimes, constantOffsetBaseline, fitFromCues, meterGrid, recordingCues, rejectOffsetOutliers } from "../../src/align/auto.ts"; import { makeWarp, type Anchor, type Warp } from "../../src/align/warp.ts"; import type { RefTrack, Smf } from "../../src/align/smf.ts"; import type { Audio, Role, TruthTrack } from "../../src/core/types.ts"; @@ -49,18 +49,22 @@ function recoveryErrors(smf: Smf, got: Anchor[], trueMidiToAudio: (m: number) => .map((m) => Math.abs(w.toAudio(m) - trueMidiToAudio(m))); } -test("recovers a known piecewise warp: median error < 40 ms, fit match rate > 0.7", () => { +test("recovers a known piecewise warp: median error < 40 ms, fit precision ≥ 0.8, DTW beats the constant baseline", () => { const D = distortion(song.song.bpm); const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio); const t0 = performance.now(); - const { anchors, fit } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [], SF2); + const { anchors, fit, baseline, chosen } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [], SF2); const ms = performance.now() - t0; const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix)); - debug(`synthetic warp: ${anchors.length} anchors, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.matchRate?.toFixed(3)} / ${fit.medianResidualMs?.toFixed(1)} ms, ${ms.toFixed(0)} ms`); + debug(`synthetic warp: ${anchors.length} anchors, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.precision?.toFixed(3)} lift ${fit.lift?.toFixed(3)} / ${fit.medianResidualMs?.toFixed(1)} ms, baseline ${baseline.offsetSec} s ${baseline.precision?.toFixed(3)}, ${ms.toFixed(0)} ms`); + expect(chosen).toBe("dtw"); expect(anchors.every((a) => a.kind === "auto")).toBe(true); expect(median(err)).toBeLessThan(0.04); expect(p90(err)).toBeLessThan(0.08); - expect(fit.matchRate!).toBeGreaterThan(0.7); + expect(fit.precision!).toBeGreaterThanOrEqual(0.8); + expect(fit.precision!).toBeGreaterThanOrEqual(baseline.precision! + 0.02); + expect(fit.windows.some((w) => w.flagged)).toBe(false); + expect(fit.stretches!.total).toBe(anchors.length + 1); expect(fit.windows.length).toBe(2); expect(anchors.every((a) => a.audioSec > 0 || a.midiSec === 0)).toBe(true); }, BIG); @@ -86,7 +90,7 @@ test("a silent intro in the recording does not drag the path", () => { const { anchors, fit } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2); const err = recoveryErrors(smf, anchors, (m) => m + 4, durOf(mix)); expect(median(err)).toBeLessThan(0.04); - expect(fit.matchRate!).toBeGreaterThan(0.7); + expect(fit.precision!).toBeGreaterThanOrEqual(0.8); }, BIG); test("5-minute input aligns in < 15 s", () => { @@ -94,12 +98,18 @@ test("5-minute input aligns in < 15 s", () => { const mix: Audio = { sampleRate: song.mix.sampleRate, channels: song.mix.channels.map((c) => { const o = new Float32Array(c.length * reps); for (let r = 0; r < reps; r++) o.set(c, r * c.length); return o; }) }; const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t + 0.3, reps, period); const t0 = performance.now(); - const { anchors, fit } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2); + const { anchors, fit, baseline, chosen } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2); const sec = (performance.now() - t0) / 1000; - debug(`5-min (${durOf(mix).toFixed(0)} s) input: ${sec.toFixed(2)} s, ${anchors.length} anchors, fit ${fit.matchRate?.toFixed(3)}`); + debug(`5-min (${durOf(mix).toFixed(0)} s) input: ${sec.toFixed(2)} s, ${anchors.length} anchors, fit ${fit.precision?.toFixed(3)}, ${chosen}, baseline ${baseline.offsetSec} ${baseline.precision?.toFixed(3)}`); expect(durOf(mix)).toBeGreaterThan(290); expect(sec).toBeLessThan(15); expect(fit.windows.length).toBe(10); + // a constant offset: DTW cannot beat the baseline by 0.02, so the single-anchor baseline is returned + expect(baseline.offsetSec).toBeCloseTo(-0.3, 1); + expect(chosen).toBe("baseline"); + expect(anchors).toHaveLength(1); + expect(anchors[0]!.kind).toBe("auto"); + expect(anchors[0]!.audioSec - anchors[0]!.midiSec).toBeCloseTo(baseline.offsetSec, 6); }, BIG); test("barTimes follows tempo and time-signature changes", () => { @@ -130,13 +140,13 @@ test("fit exposes a bad alignment instead of hiding it", () => { const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio); const roles = rolesOf(song.truth.tracks); const bad = alignmentFit(song.mix, smf, roles, []); // identity: off by 0.4–0.8 s throughout - expect(bad.matchRate!).toBeLessThan(0.5); + expect(bad.precision!).toBeLessThan(0.6); // the −0.8 s section is exposed; the +0.4 s section is not, as 0.4 s is ≈ 40 ms from two 16ths at 68 bpm (grid alias) expect(bad.windows[1]!.flagged).toBe(true); const good = alignmentFit(song.mix, smf, roles, autoAlign(song.mix, smf, roles, [], SF2).anchors); expect(good.windows.some((w) => w.flagged)).toBe(false); // unmapped tracks are excluded: with no scored roles there is nothing to measure - expect(alignmentFit(song.mix, smf, Object.fromEntries(Object.keys(roles).map((k) => [k, null])), []).matchRate).toBeNull(); + expect(alignmentFit(song.mix, smf, Object.fromEntries(Object.keys(roles).map((k) => [k, null])), []).precision).toBeNull(); }, BIG); test("a deliberately wrong manual anchor (+0.3 s) is kept exactly and the result stays monotonic", () => { @@ -158,23 +168,122 @@ test("when every fine band fails, the coarse-path fallback still yields auto anc const D = distortion(song.song.bpm); const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio); const manual: Anchor = { midiSec: D.toAudio(30), audioSec: 30, kind: "manual" }; - const { anchors, fit } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [manual], SF2, { fineBandsSec: [] }); + const { anchors, fit, chosen } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [manual], SF2, { fineBandsSec: [] }); + expect(chosen).toBe("dtw"); // manual anchors present: never substituted by the baseline const autos = anchors.filter((a) => a.kind === "auto"); // both segments (before and after the manual anchor) get anchors expect(autos.filter((a) => a.audioSec < 30).length).toBeGreaterThan(3); expect(autos.filter((a) => a.audioSec > 30).length).toBeGreaterThan(3); expect(strictlyMonotone(anchors)).toBe(true); const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix)); - debug(`coarse fallback: ${autos.length} autos, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.matchRate?.toFixed(3)}`); + debug(`coarse fallback: ${autos.length} autos, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.precision?.toFixed(3)}`); expect(median(err)).toBeLessThan(0.06); // 10 Hz resolution: coarser than the fine pass, still hand-correctable expect(p90(err)).toBeLessThan(0.12); }, BIG); -test("fitFromOnsets with cached recordingOnsets equals alignmentFit", () => { +test("fitFromCues with cached recordingCues equals alignmentFit", () => { const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t); const roles = rolesOf(song.truth.tracks); const anchors: Anchor[] = [{ midiSec: 1, audioSec: 1.05, kind: "manual" }, { midiSec: 20, audioSec: 20.1, kind: "manual" }]; - const onsets = recordingOnsets(song.mix); - expect(onsets.length).toBeGreaterThan(10); - expect(fitFromOnsets(onsets, smf, roles, anchors, durOf(song.mix))).toEqual(alignmentFit(song.mix, smf, roles, anchors)); + const cues = recordingCues(song.mix); + expect(cues.onsets.length).toBeGreaterThan(10); + expect(cues.chroma.length).toBeGreaterThan(cues.durSec * cues.chromaRate - 2); + expect(fitFromCues(cues, smf, roles, anchors)).toEqual(alignmentFit(song.mix, smf, roles, anchors)); }); + +// ---- ruling A7: calibrated fit, beat-shift check, constant-offset baseline ---- + +test("fit precision: correct alignment ≥ 0.8, a 0.12 s off-grid shift < 0.5 (and flagged)", () => { + const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t); + const roles = rolesOf(song.truth.tracks); + const good = alignmentFit(song.mix, smf, roles, []); + const off = alignmentFit(song.mix, smf, roles, [{ midiSec: 10, audioSec: 10.12, kind: "manual" }]); + debug(`precision: correct ${good.precision?.toFixed(3)} (lift ${good.lift?.toFixed(3)}, audible ${good.audibleRate?.toFixed(3)}), +0.12 s ${off.precision?.toFixed(3)} (lift ${off.lift?.toFixed(3)})`); + expect(good.precision!).toBeGreaterThanOrEqual(0.8); + expect(good.lift!).toBeGreaterThanOrEqual(0.25); + expect(good.windows.some((w) => w.flagged)).toBe(false); + expect(off.precision!).toBeLessThan(0.5); + expect(off.windows.filter((w) => w.precision !== null).every((w) => w.flagged)).toBe(true); +}); + +test("beat-shift check flags an alignment that is one beat late, though onset precision cannot see it", () => { + const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t); + const roles = rolesOf(song.truth.tracks); + const beat = 60 / song.song.bpm; + const late = alignmentFit(song.mix, smf, roles, [{ midiSec: 10, audioSec: 10 + beat, kind: "manual" }]); + const good = alignmentFit(song.mix, smf, roles, []); + debug(`beat shift: late precision ${late.precision?.toFixed(3)}, windows ${JSON.stringify(late.windows.map((w) => [w.precision?.toFixed(2), w.lift?.toFixed(2), w.beatShift]))}`); + expect(good.windows.some((w) => w.beatShift)).toBe(false); + const shifted = late.windows.filter((w) => w.beatShift); + expect(shifted.length).toBeGreaterThan(0); + expect(shifted.every((w) => w.flagged && w.reasons.some((r) => r.startsWith("beat shift suspected: −")))).toBe(true); +}); + +test("constantOffsetBaseline grid-searches −2..+2 s for the offset with the best precision", () => { + const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t + 0.27); + const b = constantOffsetBaseline(recordingCues(song.mix), smf, rolesOf(song.truth.tracks)); + expect(b.offsetSec).toBeCloseTo(-0.27, 1); + expect(Math.round(b.offsetSec * 100)).toBe(b.offsetSec * 100); // 10 ms grid + expect(b.precision!).toBeGreaterThanOrEqual(0.8); +}); + +// ---- ruling A8: aligner guards ---- + +test("rejectOffsetOutliers drops bars > ¼ beat from the robust median of ±3 neighbours (slope-transported), iterating until stable", () => { + const bars = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: 0.1 + k * 0.004, beatSec: 0.5 })); + bars[5]!.off = 0.6; // a beat out + bars[6]!.off = 0.55; // its neighbour, also out + bars[15]!.off = -0.3; + const kept = rejectOffsetOutliers(bars).map((b) => b.midiSec / 2); + expect(kept).toEqual(Array.from({ length: 20 }, (_, k) => k).filter((k) => k !== 5 && k !== 6 && k !== 15)); + // a smooth ramp (genuine tempo change) is kept whole + const ramp = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: k * 0.05, beatSec: 0.5 })); + expect(rejectOffsetOutliers(ramp)).toHaveLength(20); + // ... also a steep one (10% tempo difference: 0.2 s per 2 s bar), whose ends a plain one-sided median would erode + // bar by bar; neighbours are transported along their median (Theil–Sen) slope + const steep = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: k * 0.2, beatSec: 0.5 })); + expect(rejectOffsetOutliers(steep)).toHaveLength(20); + // a beat jump inside the steep ramp is still rejected + steep[10]!.off += 0.5; + expect(rejectOffsetOutliers(steep).map((b) => b.midiSec / 2)).toEqual(Array.from({ length: 20 }, (_, k) => k).filter((k) => k !== 10)); +}); + +test("meterGrid: beats and bars from tempo/time signatures; the final bar is partial when the MIDI ends mid-bar", () => { + const smf: Smf = { durationSec: 9, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }, { sec: 4, bpm: 60 }], timeSigs: [{ sec: 0, num: 4, den: 4 }, { sec: 4, num: 3, den: 4 }] }; + const g = meterGrid(smf); + expect(g.bars).toEqual([0, 2, 4, 7]); + expect(g.beats).toEqual([0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 5, 6, 7, 8, 9]); + expect(g.lastBarPartial).toBe(true); // bar 7..10 ends after 9 + expect(meterGrid({ ...smf, durationSec: 10 }).lastBarPartial).toBe(false); + expect(beatTimes({ durationSec: 3, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }], timeSigs: [{ sec: 0, num: 6, den: 8 }] })).toEqual([0, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75]); +}); + +test("no auto anchor in the final partial bar", () => { + const D = distortion(song.song.bpm); + const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio); + const g = meterGrid(smf); + const last = g.bars[g.bars.length - 1]!; + const cut: Smf = { ...smf, durationSec: last + 0.5 * (last - g.bars[g.bars.length - 2]!) }; // ends half-way into the last bar + expect(meterGrid(cut).lastBarPartial).toBe(true); + const { anchors } = autoAlign(song.mix, cut, rolesOf(song.truth.tracks), [], SF2); + expect(anchors.some((a) => a.midiSec >= last - 1e-9)).toBe(false); +}, BIG); + +test("12 s of extra outro in the recording: open-end DTW, no ramp at the end", () => { + const D = distortion(song.song.bpm); + const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio); + const sr = song.mix.sampleRate, extra = 12 * sr; + // the outro repeats the first 12 s of the song (material that DTW could be tempted to match) + const mix: Audio = { sampleRate: sr, channels: song.mix.channels.map((c) => { const o = new Float32Array(c.length + extra); o.set(c); o.set(c.subarray(0, extra), c.length); return o; }) }; + const { anchors, chosen } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2); + expect(chosen).toBe("dtw"); + const autos = anchors.filter((a) => a.kind === "auto"); + const tail = autos.slice(-4).map((a) => a.audioSec - D.toMidi(a.midiSec)); + debug(`outro: last anchor offsets vs truth ${tail.map((e) => (e * 1000).toFixed(0)).join(", ")} ms`); + expect(tail.every((e) => Math.abs(e) < 0.06)).toBe(true); + const w = makeWarp(anchors); + const endErr = Math.abs(w.toAudio(smf.durationSec) - D.toMidi(smf.durationSec)); + expect(endErr).toBeLessThan(0.1); + const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix)); + expect(median(err)).toBeLessThan(0.04); +}, BIG); diff --git a/tests/align/dtw.test.ts b/tests/align/dtw.test.ts index 906744a..a92cb8f 100644 --- a/tests/align/dtw.test.ts +++ b/tests/align/dtw.test.ts @@ -43,3 +43,69 @@ test("3000x3000 unbanded dtw finishes in < 5 s", () => { expect(p[p.length - 1]).toEqual([n - 1, n - 1]); expect(ms).toBeLessThan(5000); }); + +// ---- slope-limited steps and open begin/end (ruling A8) ---- +const stepsOf = (p: [number, number][]) => p.slice(1).map(([i, j], k) => [i - p[k]![0], j - p[k]![1]] as const); +/** No two consecutive non-diagonal steps in the same direction: local slope stays within [1/2, 2]. */ +function slopeLimited(p: [number, number][]): boolean { + const s = stepsOf(p); + return s.every(([di, dj], k) => { + if (!((di === 1 && dj === 1) || (di === 1 && dj === 0) || (di === 0 && dj === 1))) return false; + const prev = s[k - 1]; + return !(k > 0 && di + dj === 1 && prev![0] === di && prev![1] === dj); + }); +} + +test("slope-limited steps: path never makes an abrupt jump even when the cost invites one", () => { + // the cost is 0 on a path that jumps 20 columns at i = 50; symmetric steps follow the jump, slope-limited ones cannot + const jumpCost = (i: number, j: number) => Math.min(1, Math.abs(j - (i < 50 ? i : i + 20)) * 0.2); + const sym = dtw(jumpCost, 100, 120); + expect(stepsOf(sym).some(([di, dj], k, s) => di === 0 && dj === 1 && s[k + 1]?.[0] === 0)).toBe(true); // a horizontal run + const p = dtw(jumpCost, 100, 120, undefined, { steps: "slope" }); + expect(p[0]).toEqual([0, 0]); + expect(p[p.length - 1]).toEqual([99, 119]); + expect(slopeLimited(p)).toBe(true); +}); + +test("slope-limited steps follow a smooth diagonal-with-kink path exactly like the default (banded too)", () => { + const p = dtw(cost, N, M, undefined, { steps: "slope" }); + expect(slopeLimited(p)).toBe(true); + for (const [i, j] of p) if (i < 45 || i > 60) expect(Math.abs(j - truthJ(i))).toBeLessThanOrEqual(1); + expect(dtw(cost, N, M, { center: truthJ, width: 15 }, { steps: "slope" })).toEqual(p); +}); + +test("slope-limited steps throw 'unreachable' when the aspect ratio exceeds 2", () => { + expect(() => dtw(cost, 10, 30, undefined, { steps: "slope" })).toThrow(/unreachable/); +}); + +test("open end: the path may stop before the recording ends (subsequence end)", () => { + // the query (rows) matches columns 0..99 diagonally; the last 40 columns are extra material + const c = (i: number, j: number) => (j < 100 ? Math.min(1, Math.abs(i - j) * 0.2) : 0.5); + const closed = dtw(c, 100, 140, undefined, { steps: "slope" }); + expect(closed[closed.length - 1]).toEqual([99, 139]); + const open = dtw(c, 100, 140, undefined, { steps: "slope", openEnd: true }); + expect(open[open.length - 1]).toEqual([99, 99]); + for (const [i, j] of open) expect(Math.abs(i - j)).toBeLessThanOrEqual(1); + // the query being longer than the target (its extra rows mismatch everything, costing more than an unconsumed row): + // the path may end on the last column + const longer = dtw((i, j) => (i < 60 ? Math.min(1, Math.abs(i - j) * 0.2) : 1.5), 100, 60, undefined, { steps: "slope", openEnd: true }); + expect(longer[longer.length - 1]).toEqual([59, 59]); +}); + +test("open begin: the path may start after the beginning of the target", () => { + const c = (i: number, j: number) => (j >= 30 ? Math.min(1, Math.abs(i + 30 - j) * 0.2) : 0.5); + const p = dtw(c, 100, 130, undefined, { steps: "slope", openBegin: true }); + expect(p[0]).toEqual([0, 30]); + expect(p[p.length - 1]).toEqual([99, 129]); + for (const [i, j] of p) expect(Math.abs(i + 30 - j)).toBeLessThanOrEqual(1); + // banded with a band that does not include column 0 at row 0 + expect(dtw(c, 100, 130, { center: (i) => i + 30, width: 10 }, { steps: "slope", openBegin: true })).toEqual(p); +}); + +test("open begin + end: a trivial path in a zero-cost corner does not win (unconsumed rows cost 1 each)", () => { + // the last column is free everywhere (e.g. silence vs silence), the real match is the diagonal of the first 100 columns + const c = (i: number, j: number) => (j === 119 ? 0 : j < 100 ? Math.min(1, Math.abs(i - j) * 0.2) : 0.6); + const p = dtw(c, 100, 120, undefined, { steps: "slope", openBegin: true, openEnd: true }); + expect(p.length).toBeGreaterThan(90); + expect(p[p.length - 1]).toEqual([99, 99]); +}); diff --git a/tests/align/features.test.ts b/tests/align/features.test.ts index b13bc6f..4876dd5 100644 --- a/tests/align/features.test.ts +++ b/tests/align/features.test.ts @@ -55,3 +55,16 @@ test("pickOnsets enforces 50 ms min gap and ignores flat input", () => { expect(pickOnsets(o, 100)).toEqual([0.22, 0.6]); expect(pickOnsets(new Float32Array(100), 100)).toEqual([]); }); + +test("pickOnsets refines peaks to sub-frame times by parabolic interpolation", () => { + const o = new Float32Array(100); + o[30] = 4; o[31] = 8; o[32] = 6; // peak leans right: δ = ½(4 − 6)/(4 − 16 + 6) = 1/6 frame + const got = pickOnsets(o, 50); + expect(got).toHaveLength(1); + expect(got[0]!).toBeCloseTo((31 + 1 / 6) / 50, 9); + // min-gap competition still compares the peak heights, not the interpolated times + const p = new Float32Array(100); + p[40] = 5; p[41] = 3; p[42] = 7; p[43] = 1; + expect(pickOnsets(p, 100)).toHaveLength(1); + expect(pickOnsets(p, 100)[0]!).toBeCloseTo((42 + 0.5 * (3 - 1) / (3 - 14 + 1)) / 100, 9); +}); diff --git a/tests/align/fit.test.ts b/tests/align/fit.test.ts index 69bb0b2..a0df875 100644 --- a/tests/align/fit.test.ts +++ b/tests/align/fit.test.ts @@ -1,72 +1,106 @@ import { test, expect } from "bun:test"; -import { fitQuality, fitSummary } from "../../src/align/fit.ts"; +import { fitQuality, fitSummary, FLAG_MIN_PRECISION, FLAG_MIN_LIFT, MIN_REC_ONSETS, SHIFT_MARGIN, type FitReport, type FitWindow } from "../../src/align/fit.ts"; -test("perfect fit: match rate 1, residual 0, one window per 30 s", () => { - const on = [1, 2, 3, 31, 32, 61]; - const r = fitQuality(on, on, 75); - expect(r.matchRate).toBe(1); - expect(r.medianResidualMs).toBe(0); - expect(r.windows.map((w) => w.startSec)).toEqual([0, 30, 60]); - expect(r.windows.every((w) => !w.flagged && w.matchRate === 1)).toBe(true); +const grid = (from: number, to: number, step: number) => { const o: number[] = []; for (let t = from; t < to - 1e-9; t += step) o.push(Math.round(t * 1000) / 1000); return o; }; + +test("constants match ruling A7", () => { + expect(FLAG_MIN_PRECISION).toBe(0.6); + expect(FLAG_MIN_LIFT).toBe(0.25); + expect(MIN_REC_ONSETS).toBe(8); + expect(SHIFT_MARGIN).toBe(0.05); }); -test("match within ±70 ms only; residual is median |audio − midi| of matched onsets", () => { - const midi = [1, 2, 3, 4]; - const audio = [1.01, 2.03, 3.069, 4.2]; // last is 200 ms away: unmatched - const r = fitQuality(audio, midi, 10); - expect(r.matchRate).toBe(0.75); +test("precision = share of recording onsets with a MIDI onset within ±70 ms; audibleRate = the reverse (info)", () => { + // recording hears every 2nd MIDI onset (dense score, sparse attacks) plus 2 extra attacks out of the score + const midi = grid(0.5, 20.5, 0.25); // 80 onsets + const rec = [...midi.filter((_, k) => k % 2 === 0), 3.13, 7.63]; // 40 + 2 + const r = fitQuality(rec, midi, 20.5); + expect(r.precision).toBeCloseTo(40 / 42, 6); + expect(r.audibleRate).toBeCloseTo(0.5, 6); + expect(r.medianResidualMs).toBe(0); + expect(r.windows).toHaveLength(1); + expect(r.windows[0]!.recOnsets).toBe(42); + // low audibleRate alone does not flag (that was the old, miscalibrated rule) + expect(r.windows[0]!.flagged).toBe(false); +}); + +test("lift: precision minus mean precision at off-grid shifts (+0.11, +0.23, +0.37 s)", () => { + const midi = grid(1, 29, 1); // one onset per second: shifts of 0.11–0.37 s never land + const r = fitQuality(midi, midi, 30); + expect(r.precision).toBe(1); + expect(r.lift).toBeCloseTo(1, 6); + expect(r.windows[0]!.lift).toBeCloseTo(1, 6); + // a dense MIDI grid (every 0.12 s) matches anything: precision 1 but no lift -> flagged + const dense = grid(0, 30, 0.12); + const rec = grid(0.5, 29.5, 0.5).map((t) => t + 0.031); + const d = fitQuality(rec, dense, 30); + expect(d.precision).toBe(1); + expect(d.lift!).toBeLessThan(FLAG_MIN_LIFT); + expect(d.windows[0]!.flagged).toBe(true); + expect(d.windows[0]!.reasons.join(" ")).toContain("lift"); +}); + +test("window flagged when precision < 0.6; null and unflagged with < 8 recording onsets", () => { + const midi = grid(1, 89, 1); + const rec = [ + ...grid(1, 11, 1), // window 0: 10 onsets, all on the score + ...grid(31.5, 41.5, 1), // window 1: 10 onsets, all half-way between score onsets + 61, 62, 63, // window 2: only 3 onsets + ]; + const r = fitQuality(rec, midi, 90); + expect(r.windows.map((w) => w.startSec)).toEqual([0, 30, 60]); + expect(r.windows[0]!.precision).toBe(1); + expect(r.windows[0]!.flagged).toBe(false); + expect(r.windows[1]!.precision).toBe(0); + expect(r.windows[1]!.flagged).toBe(true); + expect(r.windows[1]!.reasons[0]).toContain("precision"); + const w2 = r.windows[2]!; + expect(w2.recOnsets).toBe(3); + expect(w2.precision).toBeNull(); + expect(w2.lift).toBeNull(); + expect(w2.flagged).toBe(false); +}); + +test("no MIDI onsets: overall nulls, nothing flagged", () => { + const r = fitQuality([1, 2], [], 10); + expect(r.precision).toBeNull(); + expect(r.audibleRate).toBeNull(); + expect(r.lift).toBeNull(); + expect(r.windows.every((w) => !w.flagged)).toBe(true); +}); + +test("residual: median |recording − nearest MIDI| over matched recording onsets", () => { + const midi = grid(1, 11, 1); + const r = fitQuality(midi.map((t, k) => t + (k % 2 ? 0.02 : 0.04)), midi, 12); expect(r.medianResidualMs!).toBeCloseTo(30, 6); }); -test("nearest recording onset is used and may be shared by several MIDI onsets", () => { - const r = fitQuality([1.0], [0.98, 1.0, 1.02], 5); - expect(r.matchRate).toBe(1); - expect(r.medianResidualMs!).toBeCloseTo(20, 6); +test("beat-shift check: flags a window where a shifted symbolic chroma beats the current one by the margin", () => { + const rate = 10, n = 600; // 60 s, two windows + const pc = (k: number) => { const c = new Float32Array(12); c[k % 12] = 1; return c; }; + // chord changes every second (one "beat"); recording = sequence s(t); current MIDI = s shifted by one beat in window 1 + const seq = (t: number) => pc(Math.floor(t / rate) * 5); + const recording = Array.from({ length: n }, (_, t) => seq(t)); + const current = Array.from({ length: n }, (_, t) => (t < 300 ? seq(t) : seq(t - rate))); + const plus1 = Array.from({ length: n }, (_, t) => seq(t - rate)), minus1 = Array.from({ length: n }, (_, t) => (t < 300 ? seq(t - rate) : seq(t))); + const onsets = grid(0.5, 59.5, 0.5); + const r = fitQuality(onsets, onsets, 60, { shift: { rate, recording, current, shifts: [{ label: "+1 beat", chroma: plus1 }, { label: "−1 beat", chroma: minus1 }] } }); + expect(r.windows[0]!.beatShift).toBeNull(); + expect(r.windows[0]!.flagged).toBe(false); + expect(r.windows[1]!.beatShift?.label).toBe("−1 beat"); + expect(r.windows[1]!.beatShift!.gain).toBeGreaterThan(SHIFT_MARGIN); + expect(r.windows[1]!.flagged).toBe(true); + expect(r.windows[1]!.reasons).toContain("beat shift suspected: −1 beat"); }); -test("empty window gives nulls and is not flagged", () => { - const r = fitQuality([1, 2, 65], [1, 2, 65], 90); - expect(r.windows).toHaveLength(3); - const w = r.windows[1]!; - expect(w).toEqual({ startSec: 30, matchRate: null, medianResidualMs: null, flagged: false }); -}); - -test("no MIDI onsets at all gives overall nulls", () => { - const r = fitQuality([1, 2], [], 10); - expect(r.matchRate).toBeNull(); - expect(r.medianResidualMs).toBeNull(); - expect(r.windows).toEqual([{ startSec: 0, matchRate: null, medianResidualMs: null, flagged: false }]); -}); - -test("window flagged when match rate < 0.5", () => { - const midi = [1, 2, 3, 4, 31, 32]; - const audio = [1, 31, 32]; // window 0: 1/4 matched - const r = fitQuality(audio, midi, 60); - expect(r.windows[0]!.matchRate).toBe(0.25); - expect(r.windows[0]!.flagged).toBe(true); - expect(r.windows[1]!.flagged).toBe(false); - expect(r.matchRate).toBe(0.5); -}); - -test("window flagged when median residual > 60 ms even with full match", () => { - const midi = [1, 2, 3]; - const r = fitQuality(midi.map((t) => t + 0.065), midi, 10); - expect(r.matchRate).toBe(1); - expect(r.windows[0]!.medianResidualMs!).toBeCloseTo(65, 6); - expect(r.windows[0]!.flagged).toBe(true); -}); - -test("unsorted and duplicate MIDI onsets: duplicates count once", () => { - const r = fitQuality([2, 1], [2, 1, 1, 1], 5); - expect(r.matchRate).toBe(1); - const s = fitQuality([1], [1, 1, 1, 3], 5); - expect(s.matchRate).toBe(0.5); // unique onsets {1, 3} -}); - -test("fitSummary: one line with match rate, median residual and flagged windows", () => { - const w = (startSec: number, flagged: boolean) => ({ startSec, matchRate: 0.9, medianResidualMs: 10, flagged }); - expect(fitSummary({ matchRate: 0.8234, medianResidualMs: 14.24, windows: [w(0, false), w(30, true), w(60, false), w(90, true)] })) - .toBe("82.3% of MIDI onsets matched · median residual 14.2 ms · 2/4 windows flagged (0:30, 1:30)"); - expect(fitSummary({ matchRate: 1, medianResidualMs: 3, windows: [w(0, false)] })).toBe("100.0% of MIDI onsets matched · median residual 3.0 ms · 0/1 windows flagged"); - expect(fitSummary({ matchRate: null, medianResidualMs: null, windows: [] })).toBe("no scored MIDI onsets"); +test("fitSummary: precision, lift, audible rate, residual, flagged windows with reasons, stretch warnings", () => { + const w = (startSec: number, reasons: string[]): FitWindow => ({ startSec, recOnsets: 20, precision: 0.9, lift: 0.7, audibleRate: 0.4, medianResidualMs: 10, beatShift: null, flagged: reasons.length > 0, reasons }); + const f: FitReport = { + precision: 0.8234, lift: 0.6512, audibleRate: 0.3811, medianResidualMs: 14.24, + windows: [w(0, []), w(30, ["beat shift suspected: +1 beat"]), w(60, []), w(90, ["precision 41%"])], + stretches: { total: 40, warned: 3, over15: 1 }, + }; + expect(fitSummary(f)).toBe("precision 82.3% (lift +0.65) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms · 2/4 windows flagged (0:30 beat shift suspected: +1 beat; 1:30 precision 41%) · stretches: 3/40 warned, 1 > 15%"); + expect(fitSummary({ ...f, windows: [w(0, [])], stretches: undefined })).toBe("precision 82.3% (lift +0.65) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms · 0/1 windows flagged"); + expect(fitSummary({ precision: null, lift: null, audibleRate: null, medianResidualMs: null, windows: [] })).toBe("no scored MIDI onsets"); }); diff --git a/tests/align/server.test.ts b/tests/align/server.test.ts index 19bf40a..3f7f93a 100644 --- a/tests/align/server.test.ts +++ b/tests/align/server.test.ts @@ -127,12 +127,18 @@ describe("API", () => { expect(Math.max(...first.recordingPeaks.max)).toBeGreaterThan(0.05); expect(first.source).toBe("auto"); expect(first.savedAt).toBeNull(); - expect(first.anchors.length).toBeGreaterThan(2); + // the fixture's MIDI is the recording's own score (identity alignment): the constant-offset baseline wins + expect(first.auto.chosen).toBe("baseline"); + expect(first.anchors).toHaveLength(1); + expect(Math.abs(first.auto.baseline.offsetSec)).toBeLessThan(0.05); expect(first.anchors.every((a: Anchor) => a.kind === "auto")).toBe(true); expect(first.stretches.length).toBeGreaterThanOrEqual(first.anchors.length); // +1, minus an empty edge stretch expect(first.stretches[1].midiBpm).toBeGreaterThan(30); expect(first.stretches[1].bpm).toBeCloseTo(first.stretches[1].midiBpm / first.stretches[1].ratio, 6); - expect(first.fit.matchRate).toBeGreaterThan(0.5); + expect(first.fit.precision).toBeGreaterThan(0.6); + expect(first.fit.stretches.total).toBe(first.stretches.length); + expect(["dtw", "baseline"]).toContain(first.auto.chosen); + expect(typeof first.auto.baseline.offsetSec).toBe("number"); expect(first.midiTempo[0].bpm).toBeGreaterThan(30); expect(await Bun.file(join(root, "fix", "recording.wav")).exists()).toBe(true); expect(await Bun.file(join(root, "fix", "peaks.json")).exists()).toBe(true); @@ -164,7 +170,7 @@ describe("API", () => { const r = await api("/api/song/fix/fit", json({ anchors: [{ midiSec: 1, audioSec: 1.4, kind: "manual" }] })); const j = await r.json(); expect(j.fit.windows.length).toBeGreaterThan(0); - expect(j.fit.matchRate).toBeLessThan(first.fit.matchRate); + expect(j.fit.precision).toBeLessThan(first.fit.precision); expect(j.stretches.length).toBe(2); }); test("bad anchors are rejected with 400", async () => { @@ -173,12 +179,13 @@ describe("API", () => { expect((await api("/api/song/fix/fit", { method: "POST", body: "{not json" })).status).toBe(400); }); test("auto: keeps manual anchors and re-solves autos around them", async () => { - const manual: Anchor = { ...first.anchors[3], kind: "manual" }; - const r = await api("/api/song/fix/auto", json({ anchors: [manual, ...first.anchors.slice(5)] })); + const manual: Anchor = { midiSec: 10, audioSec: 10, kind: "manual" }; + const r = await api("/api/song/fix/auto", json({ anchors: [manual, { midiSec: 15, audioSec: 15.5, kind: "auto" }] })); const j = await r.json(); expect(j.anchors.filter((a: Anchor) => a.kind === "manual")).toEqual([manual]); expect(j.anchors.filter((a: Anchor) => a.kind === "auto").length).toBeGreaterThan(2); - expect(j.fit.matchRate).toBeGreaterThan(0.5); + expect(j.fit.precision).toBeGreaterThan(0.6); + expect(j.auto.chosen).toBe("dtw"); // manual anchors present: never the baseline expect(j.stretches.length).toBeGreaterThanOrEqual(j.anchors.length); }, BIG); test("reset: raw MIDI (no anchors)", async () => { @@ -217,6 +224,6 @@ describe("API", () => { const j = await g.json(); expect(j.source).toBe("auto"); expect(j.warning).toContain("alignment.json"); - expect(j.anchors.length).toBeGreaterThan(2); + expect(j.anchors.length).toBeGreaterThanOrEqual(1); }); }); diff --git a/tests/align/warp.test.ts b/tests/align/warp.test.ts index 2762ff8..c9d4df1 100644 --- a/tests/align/warp.test.ts +++ b/tests/align/warp.test.ts @@ -230,3 +230,10 @@ describe("deleteAnchor", () => { expect(() => deleteAnchor([M(1, 1)], 1)).toThrow(); }); }); + +test("extrapolation before the first anchor uses the first stretch's slope (no implicit 0 -> 0 anchor)", () => { + const w = makeWarp([{ midiSec: 10, audioSec: 12, kind: "auto" }, { midiSec: 20, audioSec: 23, kind: "auto" }]); // slope 1.1 + expect(w.toAudio(0)).toBeCloseTo(12 - 10 * 1.1, 9); + expect(w.toMidi(12 - 10 * 1.1)).toBeCloseTo(0, 9); + expect(makeWarp([{ midiSec: 10, audioSec: 12, kind: "auto" }]).toAudio(0)).toBe(2); // single anchor: slope 1 +}); diff --git a/tests/bench/real.test.ts b/tests/bench/real.test.ts index e7435ff..58be770 100644 --- a/tests/bench/real.test.ts +++ b/tests/bench/real.test.ts @@ -9,31 +9,46 @@ import { writeAlignment, type SongEntry } from "../../src/ref/manifest.ts"; import { createPipeline } from "../../src/pipeline/run.ts"; import { scoreSong } from "../../src/eval/score.ts"; import type { Role } from "../../src/core/types.ts"; +import type { FitReport, FitWindow } from "../../src/align/fit.ts"; const hasReference = await Bun.file("reference/songs.json").exists(); const mr = (f1: number | null, tp = 8, fp = 2, fn = 2) => ({ tp, fp, fn, precision: f1, recall: f1, f1, octaveErrors: 0, matches: [] as [number, number][] }); -const fit = (rate: number) => ({ matchRate: rate, medianResidualMs: 12.3, windows: [{ startSec: 0, matchRate: rate, medianResidualMs: 12.3, flagged: false }, { startSec: 30, matchRate: 0.2, medianResidualMs: 80, flagged: true }] }); +const win = (startSec: number, precision: number, reasons: string[]): FitWindow => ({ startSec, recOnsets: 40, precision, lift: 0.6, audibleRate: 0.4, medianResidualMs: 12.3, beatShift: null, flagged: reasons.length > 0, reasons }); +const fit = (p: number): FitReport => ({ + precision: p, lift: 0.62, audibleRate: 0.38, medianResidualMs: 12.3, + windows: [win(0, p, []), win(30, 0.5, ["precision 50%"])], stretches: { total: 41, warned: 4, over15: 2 }, +}); +const base = { offsetSec: 0.06, precision: 0.88 }; const fixture = (): RealSongResult[] => [ { - id: "a", title: "Song A", alignment: { source: "auto", manual: 0, auto: 40, savedAt: null }, fit: fit(0.8), + id: "a", title: "Song A", alignment: { source: "computed", manual: 0, auto: 40, savedAt: null, chosen: "dtw", baseline: base, dtwPrecision: 0.8 }, fit: fit(0.8), perRole: { bass: mr(0.5), guitar: mr(0.9), keys: mr(null, 0, 0, 0) }, lenient: { bass: 0.6, guitar: 0.95, keys: null }, drumsPerClass: null, excludedTracks: ["Click", "Pad"], capo: { Lead: 2 }, seconds: 3, }, { - id: "b", title: "Song B", alignment: { source: "manual", manual: 5, auto: 30, savedAt: "2026-10-29T10:00:00Z" }, fit: fit(0.6), + id: "b", title: "Song B", alignment: { source: "saved", manual: 0, auto: 1, savedAt: "2026-10-29T10:00:00Z", chosen: "baseline", baseline: base, dtwPrecision: 0.85 }, fit: fit(0.6), perRole: { guitar: mr(0.3) }, lenient: { guitar: 0.4 }, drumsPerClass: null, excludedTracks: [], capo: {}, seconds: 4, }, ]; describe("renderRealReport", () => { - const md = renderRealReport("fx", fixture()); - test("guitar row precedes the other roles; auto source and fit shown", () => { + const md = renderRealReport("fx", fixture(), [{ id: "c", reason: "reference/c/alignment.json: not valid JSON" }, { id: "d", reason: "no alignment.json" }]); + test("guitar row precedes the other roles; source, method, baseline, fit and stretch warnings shown", () => { expect(md).toContain("Song A"); expect(md).toContain("computed this run"); - expect(md).toContain("2026-10-29T10:00:00Z"); + expect(md).toContain("saved 2026-10-29T10:00:00Z"); expect(md).toContain("0 manual / 40 auto"); - expect(md).toContain("80.0% of MIDI onsets matched"); + expect(md).not.toMatch(/manual \(/); + expect(md).toContain("chosen: DTW (precision 80.0%)"); + expect(md).toContain("chosen: baseline (DTW precision 85.0% not better"); + expect(md).toContain("constant-offset baseline +0.06 s: precision 88.0%"); + expect(md).toContain("precision 80.0% (lift +0.62)"); + expect(md).toContain("MIDI onsets audible as attacks 38.0%"); + expect(md).toContain("1/2 windows flagged (0:30 precision 50%)"); + expect(md).toContain("stretches: 4/41 warned, 2 > 15%"); expect(md).toContain("consistency of the alignment"); + expect(md).toContain("c (reference/c/alignment.json: not valid JSON)"); + expect(md).toContain("d (no alignment.json)"); const a = md.slice(md.indexOf("Song A"), md.indexOf("Song B")); expect(a.indexOf("| guitar")).toBeGreaterThan(-1); expect(a.indexOf("| guitar")).toBeLessThan(a.indexOf("| bass")); @@ -59,15 +74,18 @@ describe("runReal", () => { id: "t1a-1", title: "Synthetic t1a/1", audio: join(dir, "mix.wav"), midi: join(ref, "t1a-1", "song.mid"), roles: Object.fromEntries(truth.tracks.map((t) => [t.name, t.role])), capo: {}, }; - await Bun.write(join(ref, "songs.json"), JSON.stringify([entry, { ...entry, id: "noalign", title: "No alignment" }])); + await Bun.write(join(ref, "songs.json"), JSON.stringify([entry, { ...entry, id: "noalign", title: "No alignment" }, { ...entry, id: "corrupt", title: "Corrupt alignment" }])); + await Bun.write(join(ref, "corrupt", "alignment.json"), "{ not json"); const dur = 600; await writeAlignment("t1a-1", { anchors: [{ midiSec: 0, audioSec: 0, kind: "manual" }, { midiSec: dur, audioSec: dur, kind: "manual" }], savedAt: "2026-10-30T00:00:00Z", fit: null }, ref); const results = await runReal({ runId: "t", auto: false, root: ref, out: join(root, "runs"), resultsDir: join(root, "results") }); expect(results.map((r) => r.id)).toEqual(["t1a-1"]); const r = results[0]!; - expect(r.alignment).toEqual({ source: "manual", manual: 2, auto: 0, savedAt: "2026-10-30T00:00:00Z" }); - expect(r.fit.matchRate).not.toBeNull(); + expect(r.alignment).toEqual({ source: "saved", manual: 2, auto: 0, savedAt: "2026-10-30T00:00:00Z", chosen: null, baseline: expect.any(Object), dtwPrecision: null }); + expect(Math.abs(r.alignment.baseline.offsetSec)).toBeLessThan(0.05); // identity alignment + expect(r.fit.precision).not.toBeNull(); + expect(r.fit.stretches).toBeDefined(); const pipe = await createPipeline({ demucs: new Uint8Array(await Bun.file("models/demucs/htdemucs_6s.onnx").arrayBuffer()), @@ -81,7 +99,10 @@ describe("runReal", () => { for (const f of ["notes.json", "out.gp", "score.json"]) expect(await Bun.file(join(root, "runs", "real-t", "t1a-1", f)).exists()).toBe(true); const json = await Bun.file(join(root, "results", "real-t.json")).text(); expect(json).not.toContain('"matches"'); - expect(await Bun.file(join(root, "results", "real-t.md")).text()).toContain("Synthetic t1a/1"); + const md = await Bun.file(join(root, "results", "real-t.md")).text(); + expect(md).toContain("Synthetic t1a/1"); + expect(md).toMatch(/corrupt \(.*alignment\.json.*not valid JSON/); + expect(md).toContain("noalign (no alignment.json; pass --auto to compute one)"); } finally { await rm(root, { recursive: true, force: true }); } }, 600_000);