STSheets/docs/superpowers/specs/2026-10-30-reference-alignment-design.md

# Reference Song Alignment — Design

Date: 2026-10-30
Status: approved by user 2026-10-30
Builds on: `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md` (branch `feat/alignment`, based on `feat/synthetic-benchmark`).

## Intent

The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI.
For two of the first three songs (*Swimming*, *Holes*) the MIDI does not follow the
recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song,
and it drifts differently per section, so a single stretch can't fix it. Until the
MIDI follows the recording, real-song scores mostly measure misalignment, not
transcription. (*Last Night on Earth* fits with one ~60 ms offset.)

We want (1) an automatic alignment step, (2) a local web page where a person can
correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song
to be enough), and (3) a real-song benchmark that scores the pipeline against the
aligned MIDI.

## Goals

1. **Alignment model.** A song's alignment is a monotonic piecewise-linear map from
   MIDI time to recording time, defined by anchors (`{ midiSec, audioSec }`).
   Between two anchors, the MIDI's own tempo map is scaled uniformly so the
   stretch fits. Start and end are implicit anchors.
2. **Automatic first pass.** Computes an alignment from the audio and the MIDI
   alone, honouring any manual anchors as fixed points (the automatic part only
   fills in between them). Reports a fit-quality number so we know whether to trust it.
3. **Alignment web page (layout A, chosen by the user):**
   - Two lanes on one shared, zoomable timeline: recording waveform on top, the
     MIDI rendered to audio (with the current alignment applied) underneath.
   - Anchors drawn as lines across both lanes; the user can add one at the
     playback cursor, drag either end (recording side / MIDI side), and delete it.
   - Each stretch between anchors shows its resulting tempo vs. the MIDI's own
     (e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large
     (likely an anchor on the wrong bar).
   - Playback from the cursor with a mix choice: both, recording only, MIDI only,
     recording left / MIDI right. Playback reflects the current anchors.
   - Save writes the alignment to disk; reopening the page loads it.
   - Song picker for the songs listed in the reference manifest.
4. **Real-song benchmark.** A command runs the pipeline (full mode) on each
   reference recording, warps the reference MIDI's note times through the saved
   alignment, and scores per role (guitar tracks merged; same pitch metric as the
   synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit
   quality. Writes a markdown + JSON summary.
5. **Per-song metadata** in the manifest: title, audio path, MIDI path, MIDI
   track → role mapping (auto-guessed from track names/programs, overridable),
   and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning.

## Non-goals

- Editing notes, pitches, or instruments in the MIDI — only timing changes.
- Multi-user or hosted use; this is a local team tool.
- Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in
  the tab assigner — later work). Capo is *recorded* now, not yet used.
- Changing the synthetic benchmark.

## Anti-goals

- Committing any reference audio, MIDI, rendered audio, or the manifest to git.
- An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar)
  without the fit-quality number or the page showing it.
- Scores reported for a real song without saying which alignment (auto/manual,
  fit quality) they were computed against.

## Constraints

- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports for the
  page, no vite/express; `bun test`).
- Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering,
  `src/dsp` (STFT, mono, resample), `src/pipeline/run.ts`, `src/eval/notes.ts`.
  Pure alignment logic lives outside `src/bench`/server code so it stays
  browser-portable (same rule as the pipeline).
- All reference material lives in the git-ignored `reference/` folder (and the
  FLAC/MIDI files in the repo root, also ignored). The manifest is
  `reference/songs.json` (git-ignored — coordinator's call, to keep song lists
  out of VCS). Alignments are saved as `reference/<songId>/alignment.json`.
- Served on the user's network: the server binds a configurable host (the user
  reaches this machine at `10.20.20.2`); `bun run align --host 0.0.0.0`.
- CPU only.

## How it works (decided parts)

- **Features for automatic alignment (coordinator's call):** onset strength and
  chroma from the recording and from the MIDI rendered to audio, compared with
  dynamic time warping restricted to a band around the current path, solved
  separately between consecutive manual anchors (so anchors are hard
  constraints). The resulting path is reduced to anchors (about one per bar)
  marked `auto`; manual anchors are marked `manual` and always win.
- **Default flow (coordinator's call):** opening a song with no saved alignment
  runs the automatic pass first, so the user corrects a guess rather than
  starting from the raw MIDI; a "reset to raw MIDI" action exists.
- **Fit quality:** median absolute onset residual (ms) between matched recording
  onsets and warped MIDI onsets, overall and per 30 s window; shown on the page
  and in benchmark output.
- **Playback of the warped MIDI:** the server re-renders the warped MIDI audio
  when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page
  requests it on Play. Waveforms are sent as precomputed peaks, not full audio.


## Clarifications (coordinator's calls after builder review)

- **Roles scored on real songs:** guitar (all guitar tracks merged), vocals (lead +
  backing), bass, drums (existing heuristic transcriber, compared by drum class as
  in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with
  no role (e.g. brass, crowd) are excluded and listed in the report. The
  track→role mapping is auto-guessed from track names/programs when a song is
  added and can be overridden by editing `reference/songs.json` (no UI for it).
- **Automatic aligner, coarse-to-fine:** a first unconstrained DTW pass at a low
  frame rate (≈10 Hz) over the whole song (between manual anchors when present),
  then refinement in a band around that path at full rate (≈50 Hz). The bar for
  "auto" is *good enough to correct by hand*, not perfect.
- **Fit quality (not circular):** recording onsets come from the raw audio (onset
  flux peak-picking on the mix), never from the pipeline's transcription. MIDI
  onsets = unique onset times of all scored-role notes after warping. Matching
  within ±70 ms. Report (a) median absolute residual of matched onsets and (b)
  match rate = share of warped MIDI onsets with a recording onset within ±70 ms,
  overall and per 30 s window. A window is flagged "don't trust" when its match
  rate < 0.5 or median residual > 60 ms; the page highlights flagged windows.
- **Editing model:** dragging an anchor only re-fits its two neighbouring
  stretches. Adding or moving a manual anchor deletes the auto anchors in the two
  affected stretches (those stretches become plain uniform scaling) until the user
  presses "Auto-align around my anchors", which re-solves auto anchors in every
  stretch between consecutive manual anchors. Auto never runs implicitly on drag.
- **MIDI rendering:** build an alphaTab `MidiFile` from the parsed SMF events
  (programs, channel 10 drums) and render with the existing SONiVOX SoundFont.
  If that proves impossible, a small built-in additive synth is an acceptable
  fallback, labelled in the UI. MIDI pitches already include capo, so capo does
  not affect rendering.
- **Manifest seeding:** `bun run align add <audio> <midi> [--id] [--capo track=N]`
  adds a song and auto-guesses roles; a one-off seeding of the three current songs
  (Holes: acoustic guitar capo 9) is part of the work.
- **Playback/transport:** the recording is transcoded once (ffmpeg) to a
  browser-friendly cached file under `reference/<songId>/`; waveforms are sent as
  precomputed peaks; the warped MIDI render is re-requested after edits
  (debounced), the previous render keeps playing until the new one arrives, and
  both buffers start at the same AudioContext time.
- **Access:** the server prints a URL with a random access token (like the
  mockup server) and rejects requests without it; file access is limited to
  manifest entries.
- **`bench real`:** report-only (no pass bar). Songs without `alignment.json` are
  skipped unless `--auto` (then the automatic alignment is computed and used, and
  labelled). The report states, per song, alignment source (manual/auto anchor
  counts, file timestamp) and fit quality. Per-run artefacts go to
  `runs/real-<runId>/` (git-ignored); the summary `results/real-<runId>.{md,json}` (song titles, scores, alignment quality only — no audio/MIDI) is committed (user decision).
- **Tests:** the aligner is tested on generated songs with a known synthetic warp
  (piecewise tempo changes and offsets applied to the MIDI); tests touching real
  reference files skip when `reference/` is absent.

## Left to the builder

- Page structure, styling, zoom/scroll details, keyboard shortcuts.
- Exact DTW band width, feature frame rate, and bar-anchor reduction rule.
- How the MIDI is fed to alphaTab's synth (constructing a `MidiFile` from the
  parsed SMF events is expected to work; report if not).
- Audio transport format to the browser (e.g. FLAC/WAV/Opus) as long as
  playback works in current Chrome/Firefox/Safari.
- CLI shape (`bun run align`, `bun run bench real` are suggestions).