From 9756fc6faf54edb16e5a11fa12689df030cdf077 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 06:21:53 +0100 Subject: [PATCH] docs: reference alignment design Co-Authored-By: Claude Opus 5.5 --- .../2026-10-30-reference-alignment-design.md | 159 ++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-30-reference-alignment-design.md diff --git a/docs/superpowers/specs/2026-10-30-reference-alignment-design.md b/docs/superpowers/specs/2026-10-30-reference-alignment-design.md new file mode 100644 index 0000000..d42ac01 --- /dev/null +++ b/docs/superpowers/specs/2026-10-30-reference-alignment-design.md @@ -0,0 +1,159 @@ +# Reference Song Alignment — Design + +Date: 2026-10-30 +Status: approved by user 2026-10-30 +Builds on: `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md` (branch `feat/alignment`, based on `feat/synthetic-benchmark`). + +## Intent + +The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI. +For two of the first three songs (*Swimming*, *Holes*) the MIDI does not follow the +recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song, +and it drifts differently per section, so a single stretch can't fix it. Until the +MIDI follows the recording, real-song scores mostly measure misalignment, not +transcription. (*Last Night on Earth* fits with one ~60 ms offset.) + +We want (1) an automatic alignment step, (2) a local web page where a person can +correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song +to be enough), and (3) a real-song benchmark that scores the pipeline against the +aligned MIDI. + +## Goals + +1. **Alignment model.** A song's alignment is a monotonic piecewise-linear map from + MIDI time to recording time, defined by anchors (`{ midiSec, audioSec }`). + Between two anchors, the MIDI's own tempo map is scaled uniformly so the + stretch fits. Start and end are implicit anchors. +2. **Automatic first pass.** Computes an alignment from the audio and the MIDI + alone, honouring any manual anchors as fixed points (the automatic part only + fills in between them). Reports a fit-quality number so we know whether to trust it. +3. **Alignment web page (layout A, chosen by the user):** + - Two lanes on one shared, zoomable timeline: recording waveform on top, the + MIDI rendered to audio (with the current alignment applied) underneath. + - Anchors drawn as lines across both lanes; the user can add one at the + playback cursor, drag either end (recording side / MIDI side), and delete it. + - Each stretch between anchors shows its resulting tempo vs. the MIDI's own + (e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large + (likely an anchor on the wrong bar). + - Playback from the cursor with a mix choice: both, recording only, MIDI only, + recording left / MIDI right. Playback reflects the current anchors. + - Save writes the alignment to disk; reopening the page loads it. + - Song picker for the songs listed in the reference manifest. +4. **Real-song benchmark.** A command runs the pipeline (full mode) on each + reference recording, warps the reference MIDI's note times through the saved + alignment, and scores per role (guitar tracks merged; same pitch metric as the + synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit + quality. Writes a markdown + JSON summary. +5. **Per-song metadata** in the manifest: title, audio path, MIDI path, MIDI + track → role mapping (auto-guessed from track names/programs, overridable), + and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning. + +## Non-goals + +- Editing notes, pitches, or instruments in the MIDI — only timing changes. +- Multi-user or hosted use; this is a local team tool. +- Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in + the tab assigner — later work). Capo is *recorded* now, not yet used. +- Changing the synthetic benchmark. + +## Anti-goals + +- Committing any reference audio, MIDI, rendered audio, or the manifest to git. +- An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar) + without the fit-quality number or the page showing it. +- Scores reported for a real song without saying which alignment (auto/manual, + fit quality) they were computed against. + +## Constraints + +- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports for the + page, no vite/express; `bun test`). +- Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering, + `src/dsp` (STFT, mono, resample), `src/pipeline/run.ts`, `src/eval/notes.ts`. + Pure alignment logic lives outside `src/bench`/server code so it stays + browser-portable (same rule as the pipeline). +- All reference material lives in the git-ignored `reference/` folder (and the + FLAC/MIDI files in the repo root, also ignored). The manifest is + `reference/songs.json` (git-ignored — coordinator's call, to keep song lists + out of VCS). Alignments are saved as `reference//alignment.json`. +- Served on the user's network: the server binds a configurable host (the user + reaches this machine at `10.20.20.2`); `bun run align --host 0.0.0.0`. +- CPU only. + +## How it works (decided parts) + +- **Features for automatic alignment (coordinator's call):** onset strength and + chroma from the recording and from the MIDI rendered to audio, compared with + dynamic time warping restricted to a band around the current path, solved + separately between consecutive manual anchors (so anchors are hard + constraints). The resulting path is reduced to anchors (about one per bar) + marked `auto`; manual anchors are marked `manual` and always win. +- **Default flow (coordinator's call):** opening a song with no saved alignment + runs the automatic pass first, so the user corrects a guess rather than + starting from the raw MIDI; a "reset to raw MIDI" action exists. +- **Fit quality:** median absolute onset residual (ms) between matched recording + onsets and warped MIDI onsets, overall and per 30 s window; shown on the page + and in benchmark output. +- **Playback of the warped MIDI:** the server re-renders the warped MIDI audio + when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page + requests it on Play. Waveforms are sent as precomputed peaks, not full audio. + + +## Clarifications (coordinator's calls after builder review) + +- **Roles scored on real songs:** guitar (all guitar tracks merged), vocals (lead + + backing), bass, drums (existing heuristic transcriber, compared by drum class as + in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with + no role (e.g. brass, crowd) are excluded and listed in the report. The + track→role mapping is auto-guessed from track names/programs when a song is + added and can be overridden by editing `reference/songs.json` (no UI for it). +- **Automatic aligner, coarse-to-fine:** a first unconstrained DTW pass at a low + frame rate (≈10 Hz) over the whole song (between manual anchors when present), + then refinement in a band around that path at full rate (≈50 Hz). The bar for + "auto" is *good enough to correct by hand*, not perfect. +- **Fit quality (not circular):** recording onsets come from the raw audio (onset + flux peak-picking on the mix), never from the pipeline's transcription. MIDI + onsets = unique onset times of all scored-role notes after warping. Matching + within ±70 ms. Report (a) median absolute residual of matched onsets and (b) + match rate = share of warped MIDI onsets with a recording onset within ±70 ms, + overall and per 30 s window. A window is flagged "don't trust" when its match + rate < 0.5 or median residual > 60 ms; the page highlights flagged windows. +- **Editing model:** dragging an anchor only re-fits its two neighbouring + stretches. Adding or moving a manual anchor deletes the auto anchors in the two + affected stretches (those stretches become plain uniform scaling) until the user + presses "Auto-align around my anchors", which re-solves auto anchors in every + stretch between consecutive manual anchors. Auto never runs implicitly on drag. +- **MIDI rendering:** build an alphaTab `MidiFile` from the parsed SMF events + (programs, channel 10 drums) and render with the existing SONiVOX SoundFont. + If that proves impossible, a small built-in additive synth is an acceptable + fallback, labelled in the UI. MIDI pitches already include capo, so capo does + not affect rendering. +- **Manifest seeding:** `bun run align add