STSheets/docs/superpowers/specs/2026-10-30-reference-alignment-design.md

Reference Song Alignment — Design

Date: 2026-10-30 Status: approved by user 2026-10-30 Builds on: docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md (branch feat/alignment, based on feat/synthetic-benchmark).

Intent

The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI. For two of the first three songs (Swimming, Holes) the MIDI does not follow the recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song, and it drifts differently per section, so a single stretch can't fix it. Until the MIDI follows the recording, real-song scores mostly measure misalignment, not transcription. (Last Night on Earth fits with one ~60 ms offset.)

We want (1) an automatic alignment step, (2) a local web page where a person can correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song to be enough), and (3) a real-song benchmark that scores the pipeline against the aligned MIDI.

Goals

1. Alignment model. A song's alignment is a monotonic piecewise-linear map from MIDI time to recording time, defined by anchors ({ midiSec, audioSec }). Between two anchors, the MIDI's own tempo map is scaled uniformly so the stretch fits. Start and end are implicit anchors.
2. Automatic first pass. Computes an alignment from the audio and the MIDI alone, honouring any manual anchors as fixed points (the automatic part only fills in between them). Reports a fit-quality number so we know whether to trust it.
3. Alignment web page (layout A, chosen by the user):
   • Two lanes on one shared, zoomable timeline: recording waveform on top, the MIDI rendered to audio (with the current alignment applied) underneath.
   • Anchors drawn as lines across both lanes; the user can add one at the playback cursor, drag either end (recording side / MIDI side), and delete it.
   • Each stretch between anchors shows its resulting tempo vs. the MIDI's own (e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large (likely an anchor on the wrong bar).
   • Playback from the cursor with a mix choice: both, recording only, MIDI only, recording left / MIDI right. Playback reflects the current anchors.
   • Save writes the alignment to disk; reopening the page loads it.
   • Song picker for the songs listed in the reference manifest.
4. Real-song benchmark. A command runs the pipeline (full mode) on each reference recording, warps the reference MIDI's note times through the saved alignment, and scores per role (guitar tracks merged; same pitch metric as the synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit quality. Writes a markdown + JSON summary.
5. Per-song metadata in the manifest: title, audio path, MIDI path, MIDI track → role mapping (auto-guessed from track names/programs, overridable), and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning.

Non-goals

• Editing notes, pitches, or instruments in the MIDI — only timing changes.
• Multi-user or hosted use; this is a local team tool.
• Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in the tab assigner — later work). Capo is recorded now, not yet used.
• Changing the synthetic benchmark.

Anti-goals

• Committing any reference audio, MIDI, rendered audio, or the manifest to git.
• An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar) without the fit-quality number or the page showing it.
• Scores reported for a real song without saying which alignment (auto/manual, fit quality) they were computed against.

Constraints

• Bun + TypeScript; follow CLAUDE.md (Bun.serve with HTML imports for the page, no vite/express; bun test).
• Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering, src/dsp (STFT, mono, resample), src/pipeline/run.ts, src/eval/notes.ts. Pure alignment logic lives outside src/bench/server code so it stays browser-portable (same rule as the pipeline).
• All reference material lives in the git-ignored reference/ folder (and the FLAC/MIDI files in the repo root, also ignored). The manifest is reference/songs.json (git-ignored — coordinator's call, to keep song lists out of VCS). Alignments are saved as reference/<songId>/alignment.json.
• Served on the user's network: the server binds a configurable host (the user reaches this machine at 10.20.20.2); bun run align --host 0.0.0.0.
• CPU only.

How it works (decided parts)

• Features for automatic alignment (coordinator's call): onset strength and chroma from the recording and from the MIDI rendered to audio, compared with dynamic time warping restricted to a band around the current path, solved separately between consecutive manual anchors (so anchors are hard constraints). The resulting path is reduced to anchors (about one per bar) marked auto; manual anchors are marked manual and always win.
• Default flow (coordinator's call): opening a song with no saved alignment runs the automatic pass first, so the user corrects a guess rather than starting from the raw MIDI; a "reset to raw MIDI" action exists.
• Fit quality: median absolute onset residual (ms) between matched recording onsets and warped MIDI onsets, overall and per 30 s window; shown on the page and in benchmark output. (Amended 2026-10-30: see "Fit quality — Amended" below.)
• Playback of the warped MIDI: the server re-renders the warped MIDI audio when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page requests it on Play. Waveforms are sent as precomputed peaks, not full audio.

Clarifications (coordinator's calls after builder review)

• Roles scored on real songs: guitar (all guitar tracks merged), vocals (lead + backing), bass, drums (existing heuristic transcriber, compared by drum class as in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with no role (e.g. brass, crowd) are excluded and listed in the report. The track→role mapping is auto-guessed from track names/programs when a song is added and can be overridden by editing reference/songs.json (no UI for it).
• Automatic aligner, coarse-to-fine: a first unconstrained DTW pass at a low frame rate (≈10 Hz) over the whole song (between manual anchors when present), then refinement in a band around that path at full rate (≈50 Hz). The bar for "auto" is good enough to correct by hand, not perfect.
• Fit quality (not circular): recording onsets come from the raw audio (onset flux peak-picking on the mix), never from the pipeline's transcription. MIDI onsets = unique onset times of all scored-role notes after warping. Matching within ±70 ms. Report (a) median absolute residual of matched onsets and (b) match rate = share of warped MIDI onsets with a recording onset within ±70 ms, overall and per 30 s window. A window is flagged "don't trust" when its match rate < 0.5 or median residual > 60 ms; the page highlights flagged windows. (Superseded by the amendment below.)
• Fit quality — Amended 2026-10-30 after real-song evidence (rulings A7, A8). On real full mixes the onset picker finds ~1.4 attacks/s while a scored MIDI has ~13 unique onsets/s, so the match rate above caps near 40 % even when perfectly aligned (every window was flagged on every song), and onset metrics cannot see whole-beat shifts (they alias onto the grid).
  • Flag metric = precision: share of recording onsets (onset flux, k = 4, parabolic sub-frame peak times) with a warped scored-role MIDI onset within ±70 ms; per 30 s window by recording time. A window with < 8 recording onsets is null and never flagged; otherwise flagged when precision < 0.6.
  • Chance-corrected lift: precision minus the mean precision with the warped MIDI shifted by +0.11, +0.23 and +0.37 s (off-grid); reported per window and overall; a window is also flagged when lift < 0.25.
  • Beat/bar-shift check: per window, mean chroma cosine between the recording (features chroma, 10 Hz) and a symbolic MIDI chroma (piano roll of the warped pitched scored-role notes, no render) at the current alignment vs the MIDI moved by ±1, ±2 beats and ±1 bar along the MIDI's own tempo/time-signature grid. Flagged "beat shift suspected: ±k beat(s)/bar" when a shift beats the current alignment by > 0.05 (the margin keeps repetitive harmony, where shifts tie, from flagging). "+1 beat" means the MIDI content belongs one beat later in the recording.
  • Information only: the old match rate, renamed "MIDI onsets audible as attacks" (JSON audibleRate), and the median residual (recording onsets → nearest warped MIDI onset).
  • Stretch warnings: the fit summary (CLI, page, real report) includes the number of warned stretches (> 5 % tempo change) and of stretches with |tempo change| > 15 %.
  • Constant-offset baseline: grid search −2..+2 s in 10 ms steps for the offset maximising overall precision; precision peaks within 0.1 of the best are told apart by the symbolic chroma cosine (beat aliasing), and within a peak the offset with the smallest mean residual is taken. Reported with its precision. Without manual anchors, autoAlign returns the baseline as one auto anchor (chosen: "baseline") unless the DTW alignment's precision beats it by ≥ 0.02 (chosen: "dtw"); with manual anchors there is never a substitution.
  • Aligner guards: both DTW passes use slope-limited steps (Sakoe–Chiba P = 1: (1,1), (1,2), (2,1)), so the path cannot jump; the first segment may start late in the recording (open begin) and the final segment uses an open end (the path may stop before the recording ends, or before the MIDI ends, each unconsumed MIDI frame costing 1), so a length mismatch is not absorbed as a tempo ramp. Per-bar offsets more than ¼ beat from the median of their ±3 neighbouring bars are rejected (iterated until stable), and no auto anchor is placed in the MIDI's final partial bar.
• Editing model: dragging an anchor only re-fits its two neighbouring stretches. Adding or moving a manual anchor deletes the auto anchors in the two affected stretches (those stretches become plain uniform scaling) until the user presses "Auto-align around my anchors", which re-solves auto anchors in every stretch between consecutive manual anchors. Auto never runs implicitly on drag.
• MIDI rendering: build an alphaTab MidiFile from the parsed SMF events (programs, channel 10 drums) and render with the existing SONiVOX SoundFont. If that proves impossible, a small built-in additive synth is an acceptable fallback, labelled in the UI. MIDI pitches already include capo, so capo does not affect rendering.
• Manifest seeding: bun run align add <audio> <midi> [--id] [--capo track=N] adds a song and auto-guesses roles; a one-off seeding of the three current songs (Holes: acoustic guitar capo 9) is part of the work.
• Playback/transport: the recording is transcoded once (ffmpeg) to a browser-friendly cached file under reference/<songId>/; waveforms are sent as precomputed peaks; the warped MIDI render is re-requested after edits (debounced), the previous render keeps playing until the new one arrives, and both buffers start at the same AudioContext time.
• Access: the server prints a URL with a random access token (like the mockup server) and rejects requests without it; file access is limited to manifest entries.
• bench real: report-only (no pass bar). Songs without alignment.json are skipped unless --auto (then the automatic alignment is computed and used, and labelled). The report states, per song, alignment source (manual/auto anchor counts, file timestamp) and fit quality. Per-run artefacts go to runs/real-<runId>/ (git-ignored); the summary results/real-<runId>.{md,json} (song titles, scores, alignment quality only — no audio/MIDI) is committed (user decision).
• Tests: the aligner is tested on generated songs with a known synthetic warp (piecewise tempo changes and offsets applied to the MIDI); tests touching real reference files skip when reference/ is absent.

Left to the builder

• Page structure, styling, zoom/scroll details, keyboard shortcuts.
• Exact DTW band width, feature frame rate, and bar-anchor reduction rule.
• How the MIDI is fed to alphaTab's synth (constructing a MidiFile from the parsed SMF events is expected to work; report if not).
• Audio transport format to the browser (e.g. FLAC/WAV/Opus) as long as playback works in current Chrome/Firefox/Safari.
• CLI shape (bun run align, bun run bench real are suggestions).