docs: reference alignment design

Co-Authored-By: Claude Opus 5.5 <[email protected]>

docs/superpowers/specs/2026-10-30-reference-alignment-design.md

@@ -0,0 +1,159 @@
+# Reference Song Alignment — Design
+
+Date: 2026-10-30
+Status: approved by user 2026-10-30
+Builds on: `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md` (branch `feat/alignment`, based on `feat/synthetic-benchmark`).
+
+## Intent
+
+The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI.
+For two of the first three songs (*Swimming*, *Holes*) the MIDI does not follow the
+recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song,
+and it drifts differently per section, so a single stretch can't fix it. Until the
+MIDI follows the recording, real-song scores mostly measure misalignment, not
+transcription. (*Last Night on Earth* fits with one ~60 ms offset.)
+
+We want (1) an automatic alignment step, (2) a local web page where a person can
+correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song
+to be enough), and (3) a real-song benchmark that scores the pipeline against the
+aligned MIDI.
+
+## Goals
+
+1. **Alignment model.** A song's alignment is a monotonic piecewise-linear map from
+   MIDI time to recording time, defined by anchors (`{ midiSec, audioSec }`).
+   Between two anchors, the MIDI's own tempo map is scaled uniformly so the
+   stretch fits. Start and end are implicit anchors.
+2. **Automatic first pass.** Computes an alignment from the audio and the MIDI
+   alone, honouring any manual anchors as fixed points (the automatic part only
+   fills in between them). Reports a fit-quality number so we know whether to trust it.
+3. **Alignment web page (layout A, chosen by the user):**
+   - Two lanes on one shared, zoomable timeline: recording waveform on top, the
+     MIDI rendered to audio (with the current alignment applied) underneath.
+   - Anchors drawn as lines across both lanes; the user can add one at the
+     playback cursor, drag either end (recording side / MIDI side), and delete it.
+   - Each stretch between anchors shows its resulting tempo vs. the MIDI's own
+     (e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large
+     (likely an anchor on the wrong bar).
+   - Playback from the cursor with a mix choice: both, recording only, MIDI only,
+     recording left / MIDI right. Playback reflects the current anchors.
+   - Save writes the alignment to disk; reopening the page loads it.
+   - Song picker for the songs listed in the reference manifest.
+4. **Real-song benchmark.** A command runs the pipeline (full mode) on each
+   reference recording, warps the reference MIDI's note times through the saved
+   alignment, and scores per role (guitar tracks merged; same pitch metric as the
+   synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit
+   quality. Writes a markdown + JSON summary.
+5. **Per-song metadata** in the manifest: title, audio path, MIDI path, MIDI
+   track → role mapping (auto-guessed from track names/programs, overridable),
+   and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning.
+
+## Non-goals
+
+- Editing notes, pitches, or instruments in the MIDI — only timing changes.
+- Multi-user or hosted use; this is a local team tool.
+- Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in
+  the tab assigner — later work). Capo is *recorded* now, not yet used.
+- Changing the synthetic benchmark.
+
+## Anti-goals
+
+- Committing any reference audio, MIDI, rendered audio, or the manifest to git.
+- An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar)
+  without the fit-quality number or the page showing it.
+- Scores reported for a real song without saying which alignment (auto/manual,
+  fit quality) they were computed against.
+
+## Constraints
+
+- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports for the
+  page, no vite/express; `bun test`).
+- Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering,
+  `src/dsp` (STFT, mono, resample), `src/pipeline/run.ts`, `src/eval/notes.ts`.
+  Pure alignment logic lives outside `src/bench`/server code so it stays
+  browser-portable (same rule as the pipeline).
+- All reference material lives in the git-ignored `reference/` folder (and the
+  FLAC/MIDI files in the repo root, also ignored). The manifest is
+  `reference/songs.json` (git-ignored — coordinator's call, to keep song lists
+  out of VCS). Alignments are saved as `reference/<songId>/alignment.json`.
+- Served on the user's network: the server binds a configurable host (the user
+  reaches this machine at `10.20.20.2`); `bun run align --host 0.0.0.0`.
+- CPU only.
+
+## How it works (decided parts)
+
+- **Features for automatic alignment (coordinator's call):** onset strength and
+  chroma from the recording and from the MIDI rendered to audio, compared with
+  dynamic time warping restricted to a band around the current path, solved
+  separately between consecutive manual anchors (so anchors are hard
+  constraints). The resulting path is reduced to anchors (about one per bar)
+  marked `auto`; manual anchors are marked `manual` and always win.
+- **Default flow (coordinator's call):** opening a song with no saved alignment
+  runs the automatic pass first, so the user corrects a guess rather than
+  starting from the raw MIDI; a "reset to raw MIDI" action exists.
+- **Fit quality:** median absolute onset residual (ms) between matched recording
+  onsets and warped MIDI onsets, overall and per 30 s window; shown on the page
+  and in benchmark output.
+- **Playback of the warped MIDI:** the server re-renders the warped MIDI audio
+  when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page
+  requests it on Play. Waveforms are sent as precomputed peaks, not full audio.
+
+
+## Clarifications (coordinator's calls after builder review)
+
+- **Roles scored on real songs:** guitar (all guitar tracks merged), vocals (lead +
+  backing), bass, drums (existing heuristic transcriber, compared by drum class as
+  in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with
+  no role (e.g. brass, crowd) are excluded and listed in the report. The
+  track→role mapping is auto-guessed from track names/programs when a song is
+  added and can be overridden by editing `reference/songs.json` (no UI for it).
+- **Automatic aligner, coarse-to-fine:** a first unconstrained DTW pass at a low
+  frame rate (≈10 Hz) over the whole song (between manual anchors when present),
+  then refinement in a band around that path at full rate (≈50 Hz). The bar for
+  "auto" is *good enough to correct by hand*, not perfect.
+- **Fit quality (not circular):** recording onsets come from the raw audio (onset
+  flux peak-picking on the mix), never from the pipeline's transcription. MIDI
+  onsets = unique onset times of all scored-role notes after warping. Matching
+  within ±70 ms. Report (a) median absolute residual of matched onsets and (b)
+  match rate = share of warped MIDI onsets with a recording onset within ±70 ms,
+  overall and per 30 s window. A window is flagged "don't trust" when its match
+  rate < 0.5 or median residual > 60 ms; the page highlights flagged windows.
+- **Editing model:** dragging an anchor only re-fits its two neighbouring
+  stretches. Adding or moving a manual anchor deletes the auto anchors in the two
+  affected stretches (those stretches become plain uniform scaling) until the user
+  presses "Auto-align around my anchors", which re-solves auto anchors in every
+  stretch between consecutive manual anchors. Auto never runs implicitly on drag.
+- **MIDI rendering:** build an alphaTab `MidiFile` from the parsed SMF events
+  (programs, channel 10 drums) and render with the existing SONiVOX SoundFont.
+  If that proves impossible, a small built-in additive synth is an acceptable
+  fallback, labelled in the UI. MIDI pitches already include capo, so capo does
+  not affect rendering.
+- **Manifest seeding:** `bun run align add <audio> <midi> [--id] [--capo track=N]`
+  adds a song and auto-guesses roles; a one-off seeding of the three current songs
+  (Holes: acoustic guitar capo 9) is part of the work.
+- **Playback/transport:** the recording is transcoded once (ffmpeg) to a
+  browser-friendly cached file under `reference/<songId>/`; waveforms are sent as
+  precomputed peaks; the warped MIDI render is re-requested after edits
+  (debounced), the previous render keeps playing until the new one arrives, and
+  both buffers start at the same AudioContext time.
+- **Access:** the server prints a URL with a random access token (like the
+  mockup server) and rejects requests without it; file access is limited to
+  manifest entries.
+- **`bench real`:** report-only (no pass bar). Songs without `alignment.json` are
+  skipped unless `--auto` (then the automatic alignment is computed and used, and
+  labelled). The report states, per song, alignment source (manual/auto anchor
+  counts, file timestamp) and fit quality. Per-run artefacts go to
+  `runs/real-<runId>/` (git-ignored); the summary `results/real-<runId>.{md,json}` (song titles, scores, alignment quality only — no audio/MIDI) is committed (user decision).
+- **Tests:** the aligner is tested on generated songs with a known synthetic warp
+  (piecewise tempo changes and offsets applied to the MIDI); tests touching real
+  reference files skip when `reference/` is absent.
+
+## Left to the builder
+
+- Page structure, styling, zoom/scroll details, keyboard shortcuts.
+- Exact DTW band width, feature frame rate, and bar-anchor reduction rule.
+- How the MIDI is fed to alphaTab's synth (constructing a `MidiFile` from the
+  parsed SMF events is expected to work; report if not).
+- Audio transport format to the browser (e.g. FLAC/WAV/Opus) as long as
+  playback works in current Chrome/Firefox/Safari.
+- CLI shape (`bun run align`, `bun run bench real` are suggestions).