STSheets/docs/superpowers/plans/2026-10-30-reference-alignment.md

Reference Alignment Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Align team-supplied GP-exported MIDI to real recordings (automatic pass + manual anchor editing in a local web page) and score the pipeline on real songs against the aligned MIDI.

Architecture: Pure alignment logic in src/align/ (SMF parsing, warp model, MIDI rendering via alphaTab, features, DTW auto-aligner, fit quality). File/manifest I/O in src/ref/. A token-protected Bun.serve app (src/align/server.ts + HTML import page) exposes songs, waveform peaks, audio, warped MIDI renders, auto-align and save. src/bench/real.ts scores real songs.

Tech Stack: Bun 1.3 + TypeScript, alphaTab 1.8.4 (synth), existing src/dsp, src/pipeline/run.ts, src/eval/*, ffmpeg CLI (transcode only), Web Audio API in the page.

Spec: docs/superpowers/specs/2026-10-30-reference-alignment-design.md

Global Constraints

• Bun + TypeScript; follow CLAUDE.md (Bun.serve with HTML imports, no vite/express, bun test, Bun.file/Bun.write, Bun.$).
• src/align/** except server.ts, cli.ts and page/** must not use node:fs/Bun.*/process (browser-portable); I/O lives in src/ref/**, src/align/server.ts, src/bench/**.
• Never commit reference audio, MIDI, renders, reference/**, or *.flac/*.mid. Stage files by explicit path only; check git status --short before committing.
• Manifest: reference/songs.json; per song dir reference/<songId>/ holds alignment.json, cached transcoded audio, peaks.
• Alignment = anchors { midiSec, audioSec, kind: "manual" | "auto" }, monotonic piecewise-linear MIDI→audio map; implicit anchors at start (0→0 unless an anchor exists) and end (extrapolate with the last stretch's slope).
• Roles scored: guitar (merged), vocals, bass, drums (by class, existing heuristic transcriber), keys. Unmapped tracks excluded and listed.
• Fit quality: recording onsets from raw-audio onset flux (never from the pipeline's transcription); MIDI onsets = unique warped onset times of scored-role notes; match ±70 ms; report median |residual| (ms) and match rate, overall and per 30 s window; window flagged when match rate < 0.5 or median residual > 60 ms.
• Editing model: adding/moving a manual anchor deletes auto anchors in the two affected stretches; "Auto-align around my anchors" re-solves autos between consecutive manual anchors; auto never runs implicitly on drag.
• Server binds configurable --host (default 127.0.0.1) / --port; prints URL with random token; rejects requests without token (query ?key= once, then cookie); file access only for manifest entries.
• bench real: report-only; skips songs without alignment.json unless --auto; per-run artefacts runs/real-<runId>/ (ignored); summary results/real-<runId>.{md,json} (titles, scores, alignment source + fit only) committed.
• Commits end with the Co-Authored-By line from your own system reminder.

Review Focus

1. Anchors at or beyond the ends / out of order — warp must stay monotonic: reject (throw) non-increasing anchors; the page prevents dragging past neighbours. Test in Task 2.
2. Rendered MIDI clipping — SMF renders peak > 1 (Holes peaked 2.13); render must peak-normalise to −1 dBFS. Test in Task 3.
3. Silence / no onsets in a 30 s window — fit quality reports null for that window (not NaN) and doesn't flag it. Test in Task 5.
4. Path traversal / unknown song ids on the server — 404, never read outside manifest paths. Test in Task 6.
5. Huge DTW matrices — coarse pass at ≈10 Hz over a 5-min song is ~3000×3000; must stay under ~200 MB and ~5 s CPU. Test timing in Task 5.

---

File Structure

src/align/smf.ts          parse SMF → { durationSec, tracks: RefTrack[], tempoMap }
src/align/roles.ts        guessRole(track) → Role | null
src/align/warp.ts         anchors model: makeWarp, invert, stretches, edit ops
src/align/render.ts       renderMidi(tracks, warp, sf2) → Audio (alphaTab synth)
src/align/features.ts     onsetEnvelope, pickOnsets, chroma (frame-rate param)
src/align/dtw.ts          banded/unconstrained DTW
src/align/auto.ts         autoAlign(audio, tracks, manualAnchors) → Anchor[]
src/align/fit.ts          fitQuality(audioOnsets, warpedMidiOnsets) → FitReport
src/align/peaks.ts        waveform peaks (min/max per bucket)
src/align/server.ts       Bun.serve app (token, API, HTML import)
src/align/page/index.html, app.ts, style.css   layout-A page
src/align/cli.ts          `bun run align [serve|add|auto]`
src/ref/manifest.ts       read/write reference/songs.json, song dirs, transcode cache
src/bench/real.ts         real-song scoring + report
tests/align/**, tests/ref/**, tests/bench/real.test.ts

---

Task 1: SMF parser, role guessing, manifest + align add

Files: Create src/align/smf.ts, src/align/roles.ts, src/ref/manifest.ts, src/align/cli.ts; modify package.json ("align": "bun src/align/cli.ts"). Tests: tests/align/smf.test.ts, tests/ref/manifest.test.ts.

Interfaces (produces):

// smf.ts
export interface RefNote { onset: number; offset: number; pitch: number; velocity: number; channel: number }
export interface RefTrack { index: number; name: string; program: number; isDrums: boolean; notes: RefNote[] }
export interface TempoPoint { sec: number; bpm: number }
export interface Smf { durationSec: number; tracks: RefTrack[]; tempoMap: TempoPoint[]; timeSigs: { sec: number; num: number; den: number }[] }
export function parseSmf(bytes: Uint8Array): Smf
// roles.ts
export function guessRole(t: RefTrack): Role | null   // drums (ch 10) / vocal / bass / guitar|delay / piano|key|synth; else null
// manifest.ts
export interface SongEntry { id: string; title: string; audio: string; midi: string; roles: Record<string, Role | null>; capo: Record<string, number>; tuning?: Record<string, number[]> }
export async function readManifest(root = "reference"): Promise<SongEntry[]>
export async function addSong(e: Omit<SongEntry, "roles"> & { roles?: SongEntry["roles"] }, root = "reference"): Promise<SongEntry>  // fills roles via guessRole, keyed by track name
export function songDir(id: string, root = "reference"): string

CLI: bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"].

• Step 1: Write failing tests — build a tiny SMF in the test (format 1, ppq 480, tempo 120 then 60 at tick 1920, a guitar track program 25 with two notes, a ch-10 drum note): assert note onsets in seconds (0.0, 0.5 → second-tempo section at 2.0 s + 1.0 s), tempoMap has 2 points, drums isDrums, guessRole returns "guitar"/"drums"; running-status and note-on velocity 0 = note-off handled. Manifest: addSong in a temp root writes songs.json, guesses roles, rejects duplicate ids.
• Step 2: FAIL. Step 3: implement (port the parser in .scratch/realsong.ts but clean). Step 4: PASS + typecheck.
• Step 5: Commit feat(align): SMF parser, role guess, reference manifest.

Task 2: Warp model

Files: Create src/align/warp.ts; test tests/align/warp.test.ts.

Interfaces:

export type AnchorKind = "manual" | "auto";
export interface Anchor { midiSec: number; audioSec: number; kind: AnchorKind }
export interface Warp { toAudio(midiSec: number): number; toMidi(audioSec: number): number; anchors: Anchor[] }
export function makeWarp(anchors: Anchor[]): Warp            // sorts by midiSec; throws if audioSec not strictly increasing; 0 anchors = identity; 1 anchor = pure offset
export interface Stretch { fromMidi: number; toMidi: number; ratio: number; warn: boolean }  // ratio = audio dur / midi dur; warn when |1/ratio − 1| > 0.05
export function stretches(anchors: Anchor[], midiDur: number): Stretch[]
export function addManual(anchors: Anchor[], a: Omit<Anchor, "kind">): Anchor[]       // removes autos in the two stretches adjacent to the new anchor
export function moveAnchor(anchors: Anchor[], i: number, to: Omit<Anchor, "kind">): Anchor[] // becomes manual; same auto-removal; throws if it would cross a neighbour
export function deleteAnchor(anchors: Anchor[], i: number): Anchor[]

• Step 1: Failing tests: identity; single anchor = offset; two anchors interpolate and extrapolate with end slopes; toMidi(toAudio(x)) ≈ x; non-monotonic throws; stretches ratio + warn at 7%; addManual between autos removes only autos in adjacent stretches; moveAnchor crossing throws.
• Step 2–4: FAIL → implement → PASS. Step 5: Commit feat(align): warp model.

Task 3: Warped MIDI rendering + waveform peaks

Files: Create src/align/render.ts, src/align/peaks.ts; tests tests/align/render.test.ts.

Interfaces:

export function renderMidi(tracks: RefTrack[], warp: Warp, sf2: Uint8Array, opts?: { sampleRate?: number; mono?: boolean; durationSec?: number }): Audio
export function peaks(x: Float32Array, buckets: number): { min: Float32Array; max: Float32Array }

Approach (verified by controller spike): build at.midi.MidiFile (division 960) at a fixed 120 bpm (TempoChangeEvent(0, 500000) → 1920 ticks/s); place each note at round(warp.toAudio(onset) * 1920); drums on channel 9, others on channels 0–8, 10–15 assigned per track with a ProgramChangeEvent; render with AlphaSynth.exportAudio (dummy output, useSyncPoints = false) as in src/audio/render.ts (reuse its dummy output helper if exported; otherwise export it from there); peak-normalise to −1 dBFS; pad/trim to durationSec when given.

• Step 1: Failing tests: one guitar note at MIDI 1.0 s with warp offset +0.5 s → envelope onset in [1.48, 1.52] s; peak ≤ 10^(−1/20)+1e-3 on a dense 20-note chord; drums track audible; peaks of a known sine.
• Step 2–4. Step 5: Commit feat(align): warped MIDI rendering and peaks.

Task 4: Features + DTW

Files: Create src/align/features.ts, src/align/dtw.ts; tests tests/align/features.test.ts, tests/align/dtw.test.ts.

Interfaces:

export interface Features { rate: number; onset: Float32Array; chroma: Float32Array[] }  // per frame; chroma 12-dim L2-normalised
export function features(audio: Audio, rate: number): Features
export function pickOnsets(onset: Float32Array, rate: number): number[]   // seconds; adaptive threshold (median + k·MAD), 50 ms min gap
export function dtw(costAt: (i: number, j: number) => number, n: number, m: number, band?: { center: (i: number) => number; width: number }): [number, number][]  // path start (0,0) → end (n-1,m-1), steps (1,0),(0,1),(1,1)

Reuse src/dsp/fft.ts stft, src/dsp/mono.ts, src/dsp/resample.ts. Cost for alignment = (1 − chroma cosine) + λ·|onset_a − onset_b| (λ = 1, onsets z-normalised).

• Step 1: Failing tests: DTW recovers a known diagonal-with-kink path on synthetic cost; banded DTW equals unbanded when band covers the path; features rate/length correct; pickOnsets on click train ±10 ms.
• Step 2–4. Step 5: Commit feat(align): features and DTW.

Task 5: Auto aligner + fit quality

Files: Create src/align/auto.ts, src/align/fit.ts; tests tests/align/auto.test.ts, tests/align/fit.test.ts.

Interfaces:

export interface FitWindow { startSec: number; matchRate: number | null; medianResidualMs: number | null; flagged: boolean }
export interface FitReport { matchRate: number | null; medianResidualMs: number | null; windows: FitWindow[] }
export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number): FitReport  // ±70 ms, 30 s windows, flag rule per Global Constraints
export function autoAlign(audio: Audio, smf: Smf, roles: Record<string, Role | null>, manual: Anchor[], sf2: Uint8Array): { anchors: Anchor[]; fit: FitReport }

Approach: render the (manual-anchored) warped MIDI with renderMidi; coarse DTW at 10 Hz per segment between consecutive manual anchors (+ start/end); refine at 50 Hz in a band (±0.5 s) around the coarse path; convert path to MIDI→audio pairs; reduce to one auto anchor per bar using smf.tempoMap/timeSigs bar times (skip bars within 1 s of a manual anchor); compose with manual anchors; compute fit with pickOnsets on the recording and warped scored-role MIDI onsets.

• Step 1: Failing tests: synthetic known warp — take generated song t1c/1 (buildSong), write its truth as an SMF-like Smf object (no file needed), distort MIDI times with a known piecewise map (offset +0.4 s, section 2 ×1.06 tempo, section 3 −0.8 s) and align against the original mix: median |recovered − true| < 40 ms, overall fit matchRate > 0.7. With one correct manual anchor mid-song the result honours it exactly. fitQuality: empty window → nulls, not flagged; timing: 5-min synthetic input completes < 15 s.
• Step 2–4. Step 5: Commit feat(align): automatic alignment and fit quality.

Task 6: Alignment server + page (layout A)

Files: Create src/align/server.ts, src/align/page/index.html, src/align/page/app.ts, src/align/page/style.css; extend src/align/cli.ts (serve default: bun run align [--host] [--port]; bun run align auto <id> computes autoAlign and saves alignment.json, printing the fit summary), src/ref/manifest.ts (transcode cache: reference/<id>/audio.ogg via ffmpeg Opus 160k + recording.wav 44.1 kHz for analysis; alignment.json read/write). Test: tests/align/server.test.ts.

API (all require token):

• GET / → page (HTML import). GET /api/songs → [{ id, title, hasAlignment }].
• GET /api/song/:id → { id, title, audioDur, midiDur, tracks: [{name, role}], recordingPeaks, anchors, stretches, fit, midiTempo: tempoMap } (auto-aligns and returns unsaved anchors if no alignment.json).
• GET /api/song/:id/audio → cached Opus.
• POST /api/song/:id/render body { anchors } → WAV (mono 22.05 kHz) of warped MIDI + header-free JSON peaks via GET /api/song/:id/render-peaks?hash= (or return both; builder's call).
• POST /api/song/:id/auto body { anchors } (manual ones used) → { anchors, stretches, fit }.
• PUT /api/song/:id/alignment body { anchors } → saves { anchors, savedAt, fit }. POST /api/song/:id/reset → raw MIDI (no anchors).

Page behaviour (layout A, from the approved mockup): song picker; two lanes on a shared zoom/scroll timeline (wheel = zoom, drag background = scroll); recording peaks on top, warped-MIDI peaks below; anchors as green lines spanning both lanes, draggable at the recording end or the MIDI end (prevented from crossing neighbours), click-select + Delete key; "+ Anchor at cursor" (maps cursor recording time → warp.toMidi for the MIDI side); stretch labels "MIDI bpm → effective bpm (±%)" with ⚠ when warned; fit-flagged 30 s windows shaded red; click timeline to set cursor; Play/Pause from cursor; mix select (both / recording / MIDI / rec L + MIDI R) via Web Audio gain + stereo panner; MIDI render re-requested 400 ms after the last edit, old buffer keeps playing until replaced, both sources started at the same AudioContext.currentTime; buttons: Auto-align around my anchors, Save, Reset to raw MIDI; unsaved-changes indicator.

• Step 1: Failing tests (API): start server on port 0 with a temp reference root containing a fixture song built from data/t1a/1 (mix.wav as audio, an SMF written from its truth — add writeSmf(tracks) helper in src/align/smf.ts for tests); no token → 401; /api/songs lists it; unknown id and .. id → 404; /api/song/:id returns peaks + anchors; PUT then GET round-trips anchors; /render returns RIFF WAV.
• Step 2–4: FAIL → implement → PASS. Smoke: bun run align --host 0.0.0.0 prints a tokenised URL; load it with curl and confirm HTML + bundle are served.
• Step 5: Commit feat(align): alignment web page.

Task 7: Real-song benchmark

Files: Create src/bench/real.ts; modify src/bench/cli.ts (real subcommand); test tests/bench/real.test.ts.

Interfaces:

export interface RealSongResult { id: string; title: string; alignment: { source: "manual" | "auto"; manual: number; auto: number; savedAt: string | null }; fit: FitReport; perRole: Partial<Record<Role, MatchResult>>; drumsPerClass: Record<DrumClass, MatchResult> | null; excludedTracks: string[]; seconds: number }
export async function runReal(opts: { runId: string; auto: boolean; root?: string; out?: string }): Promise<RealSongResult[]>
export function renderRealReport(runId: string, results: RealSongResult[]): string

Behaviour: for each manifest song: load alignment.json (or autoAlign when --auto, else skip with a note); run createPipeline full mode on recording.wav; warp each scored-role MIDI note's onset/offset with makeWarp(anchors).toAudio; per role matchNotes (guitar merged; drums by class like scoreSong); write runs/real-<runId>/<id>/{notes.json,out.gp,score.json} and results/real-<runId>.{md,json} (strip matches). Report: per song alignment source/fit, role table (guitar first), excluded tracks.

• Step 1: Failing tests: renderRealReport fixture (guitar column first, fit shown, "auto" labelled); runReal on a temp root with the t1a fixture song + an identity alignment → guitar F1 equals the synthetic full-mode score for t1a/1 within 0.02; real reference/ test skipIf absent.
• Step 2–4. Step 5: Commit feat(bench): real-song benchmark.

Task 8: Seed songs, auto-align, first real benchmark

• Step 1: bun run align add the three songs (ids lnoe, swimming, holes; titles from file names; Holes --capo "Acoustic Guitar =9" — match the exact track name from the MIDI). Verify roles in reference/songs.json (Holes brass/crowd → null).
• Step 2: For each song run bun run align auto <id> (CLI: compute auto alignment and save it as alignment.json with all anchors auto), print fit summaries.
• Step 3: bun run bench real --run-id real-auto-2026-10-30 (uses saved auto alignments) → results files.
• Step 4: README section "Aligning reference songs" (commands, where files live, the page workflow). Commit README + results/real-auto-2026-10-30.{md,json} by explicit path; push.