STSheets/docs/superpowers/specs/2026-10-30-reference-alignment-design.md

# Reference Song Alignment — Design

Date: 2026-10-30
Status: approved by user 2026-10-30
Builds on: `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md` (branch `feat/alignment`, based on `feat/synthetic-benchmark`).

## Intent

The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI.
For two of the first three songs (*Swimming*, *Holes*) the MIDI does not follow the
recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song,
and it drifts differently per section, so a single stretch can't fix it. Until the
MIDI follows the recording, real-song scores mostly measure misalignment, not
transcription. (*Last Night on Earth* fits with one ~60 ms offset.)

We want (1) an automatic alignment step, (2) a local web page where a person can
correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song
to be enough), and (3) a real-song benchmark that scores the pipeline against the
aligned MIDI.

## Goals

1. **Alignment model.** A song's alignment is a monotonic piecewise-linear map from
   MIDI time to recording time, defined by anchors (`{ midiSec, audioSec }`).
   Between two anchors, the MIDI's own tempo map is scaled uniformly so the
   stretch fits. Start and end are implicit anchors.
2. **Automatic first pass.** Computes an alignment from the audio and the MIDI
   alone, honouring any manual anchors as fixed points (the automatic part only
   fills in between them). Reports a fit-quality number so we know whether to trust it.
3. **Alignment web page (layout A, chosen by the user):**
   - Two lanes on one shared, zoomable timeline: recording waveform on top, the
     MIDI rendered to audio (with the current alignment applied) underneath.
   - Anchors drawn as lines across both lanes; the user can add one at the
     playback cursor, drag either end (recording side / MIDI side), and delete it.
   - Each stretch between anchors shows its resulting tempo vs. the MIDI's own
     (e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large
     (likely an anchor on the wrong bar).
   - Playback from the cursor with a mix choice: both, recording only, MIDI only,
     recording left / MIDI right. Playback reflects the current anchors.
   - Save writes the alignment to disk; reopening the page loads it.
   - Song picker for the songs listed in the reference manifest.
4. **Real-song benchmark.** A command runs the pipeline (full mode) on each
   reference recording, warps the reference MIDI's note times through the saved
   alignment, and scores per role (guitar tracks merged; same pitch metric as the
   synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit
   quality. Writes a markdown + JSON summary.
5. **Per-song metadata** in the manifest: title, audio path, MIDI path, MIDI
   track → role mapping (auto-guessed from track names/programs, overridable),
   and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning.

## Non-goals

- Editing notes, pitches, or instruments in the MIDI — only timing changes.
- Multi-user or hosted use; this is a local team tool.
- Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in
  the tab assigner — later work). Capo is *recorded* now, not yet used.
- Changing the synthetic benchmark.

## Anti-goals

- Committing any reference audio, MIDI, rendered audio, or the manifest to git.
- An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar)
  without the fit-quality number or the page showing it.
- Scores reported for a real song without saying which alignment (auto/manual,
  fit quality) they were computed against.

## Constraints

- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports for the
  page, no vite/express; `bun test`).
- Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering,
  `src/dsp` (STFT, mono, resample), `src/pipeline/run.ts`, `src/eval/notes.ts`.
  Pure alignment logic lives outside `src/bench`/server code so it stays
  browser-portable (same rule as the pipeline).
- All reference material lives in the git-ignored `reference/` folder (and the
  FLAC/MIDI files in the repo root, also ignored). The manifest is
  `reference/songs.json` (git-ignored — coordinator's call, to keep song lists
  out of VCS). Alignments are saved as `reference/<songId>/alignment.json`.
- Served on the user's network: the server binds a configurable host (the user
  reaches this machine at `10.20.20.2`); `bun run align --host 0.0.0.0`.
- CPU only.

## How it works (decided parts)

- **Features for automatic alignment (coordinator's call):** onset strength and
  chroma from the recording and from the MIDI rendered to audio, compared with
  dynamic time warping restricted to a band around the current path, solved
  separately between consecutive manual anchors (so anchors are hard
  constraints). The resulting path is reduced to anchors (about one per bar)
  marked `auto`; manual anchors are marked `manual` and always win.
- **Default flow (coordinator's call):** opening a song with no saved alignment
  runs the automatic pass first, so the user corrects a guess rather than
  starting from the raw MIDI; a "reset to raw MIDI" action exists.
- **Fit quality:** median absolute onset residual (ms) between matched recording
  onsets and warped MIDI onsets, overall and per 30 s window; shown on the page
  and in benchmark output. (Amended 2026-10-30: see "Fit quality — Amended" below.)
- **Playback of the warped MIDI:** the server re-renders the warped MIDI audio
  when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page
  requests it on Play. Waveforms are sent as precomputed peaks, not full audio.


## Clarifications (coordinator's calls after builder review)

- **Roles scored on real songs:** guitar (all guitar tracks merged), vocals (lead +
  backing), bass, drums (existing heuristic transcriber, compared by drum class as
  in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with
  no role (e.g. brass, crowd) are excluded and listed in the report. The
  track→role mapping is auto-guessed from track names/programs when a song is
  added and can be overridden by editing `reference/songs.json` (no UI for it).
- **Automatic aligner, coarse-to-fine:** a first unconstrained DTW pass at a low
  frame rate (≈10 Hz) over the whole song (between manual anchors when present),
  then refinement in a band around that path at full rate (≈50 Hz). The bar for
  "auto" is *good enough to correct by hand*, not perfect.
- **Fit quality (not circular):** recording onsets come from the raw audio (onset
  flux peak-picking on the mix), never from the pipeline's transcription. MIDI
  onsets = unique onset times of all scored-role notes after warping. Matching
  within ±70 ms. Report (a) median absolute residual of matched onsets and (b)
  match rate = share of warped MIDI onsets with a recording onset within ±70 ms,
  overall and per 30 s window. A window is flagged "don't trust" when its match
  rate < 0.5 or median residual > 60 ms; the page highlights flagged windows.
  *(Superseded by the amendment below.)*
- **Fit quality — Amended 2026-10-30 after real-song evidence (rulings A7, A8).**
  On real full mixes the onset picker finds ~1.4 attacks/s while a scored MIDI
  has ~13 unique onsets/s, so the match rate above caps near 40 % even when
  perfectly aligned (every window was flagged on every song), and onset metrics
  cannot see whole-beat shifts (they alias onto the grid).
  - **Flag metric = precision:** share of recording onsets (onset flux, k = 4,
    parabolic sub-frame peak times) with a warped scored-role MIDI onset within
    ±70 ms; per 30 s window by recording time. A window with < 8 recording
    onsets is null and never flagged; otherwise flagged when precision < 0.6.
  - **Chance-corrected lift:** precision minus the mean precision with the
    warped MIDI shifted by +0.11, +0.23 and +0.37 s (off-grid); reported per
    window and overall; a window is also flagged when lift < 0.25.
  - **Beat/bar-shift check:** per window, mean chroma cosine between the
    recording (features chroma, 10 Hz) and a symbolic MIDI chroma (piano roll of
    the warped pitched scored-role notes, no render) at the current alignment
    vs the MIDI moved by ±1, ±2 beats and ±1 bar along the MIDI's own
    tempo/time-signature grid. Flagged "beat shift suspected: ±k beat(s)/bar"
    when a shift beats the current alignment by > 0.05 (the margin keeps
    repetitive harmony, where shifts tie, from flagging). "+1 beat" means the
    MIDI content belongs one beat later in the recording.
  - **Information only:** the old match rate, renamed "MIDI onsets audible as
    attacks" (JSON `audibleRate`), and the median residual (recording onsets →
    nearest warped MIDI onset).
  - **Stretch warnings:** the fit summary (CLI, page, real report) includes the
    number of warned stretches (> 5 % tempo change) and of stretches with
    |tempo change| > 15 %.
  - **Constant-offset baseline:** grid search −2..+2 s in 10 ms steps for the
    offset maximising overall precision; precision peaks within 0.1 of the best
    are told apart by the symbolic chroma cosine (beat aliasing), and within a
    peak the offset with the smallest mean residual is taken. Reported with its
    precision. Without manual anchors, `autoAlign` returns the baseline as one
    auto anchor (`chosen: "baseline"`) unless the DTW alignment's precision
    beats it by ≥ 0.02 (`chosen: "dtw"`); with manual anchors there is never a
    substitution.
  - **Aligner guards:** both DTW passes use slope-limited steps (Sakoe–Chiba
    P = 1: (1,1), (1,2), (2,1)), so the path cannot jump; the first segment may
    start late in the recording (open begin) and the final segment uses an open
    end (the path may stop before the recording ends, or before the MIDI ends,
    each unconsumed MIDI frame costing 1), so a length mismatch is not absorbed
    as a tempo ramp. Per-bar offsets more than ¼ beat from the median of their
    ±3 neighbouring bars are rejected (iterated until stable), and no auto
    anchor is placed in the MIDI's final partial bar.
- **Editing model:** dragging an anchor only re-fits its two neighbouring
  stretches. Adding or moving a manual anchor deletes the auto anchors in the two
  affected stretches (those stretches become plain uniform scaling) until the user
  presses "Auto-align around my anchors", which re-solves auto anchors in every
  stretch between consecutive manual anchors. Auto never runs implicitly on drag.
- **MIDI rendering:** build an alphaTab `MidiFile` from the parsed SMF events
  (programs, channel 10 drums) and render with the existing SONiVOX SoundFont.
  If that proves impossible, a small built-in additive synth is an acceptable
  fallback, labelled in the UI. MIDI pitches already include capo, so capo does
  not affect rendering.
- **Manifest seeding:** `bun run align add <audio> <midi> [--id] [--capo track=N]`
  adds a song and auto-guesses roles; a one-off seeding of the three current songs
  (Holes: acoustic guitar capo 9) is part of the work.
- **Playback/transport:** the recording is transcoded once (ffmpeg) to a
  browser-friendly cached file under `reference/<songId>/`; waveforms are sent as
  precomputed peaks; the warped MIDI render is re-requested after edits
  (debounced), the previous render keeps playing until the new one arrives, and
  both buffers start at the same AudioContext time.
- **Access:** the server prints a URL with a random access token (like the
  mockup server) and rejects requests without it; file access is limited to
  manifest entries.
- **`bench real`:** report-only (no pass bar). Songs without `alignment.json` are
  skipped unless `--auto` (then the automatic alignment is computed and used, and
  labelled). The report states, per song, alignment source (manual/auto anchor
  counts, file timestamp) and fit quality. Per-run artefacts go to
  `runs/real-<runId>/` (git-ignored); the summary `results/real-<runId>.{md,json}` (song titles, scores, alignment quality only — no audio/MIDI) is committed (user decision).
- **Tests:** the aligner is tested on generated songs with a known synthetic warp
  (piecewise tempo changes and offsets applied to the MIDI); tests touching real
  reference files skip when `reference/` is absent.

## Left to the builder

- Page structure, styling, zoom/scroll details, keyboard shortcuts.
- Exact DTW band width, feature frame rate, and bar-anchor reduction rule.
- How the MIDI is fed to alphaTab's synth (constructing a `MidiFile` from the
  parsed SMF events is expected to work; report if not).
- Audio transport format to the browser (e.g. FLAC/WAV/Opus) as long as
  playback works in current Chrome/Firefox/Safari.
- CLI shape (`bun run align`, `bun run bench real` are suggestions).