STSheets/docs/superpowers/plans/2026-10-30-reference-alignment.md

# Reference Alignment Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Align team-supplied GP-exported MIDI to real recordings (automatic pass + manual anchor editing in a local web page) and score the pipeline on real songs against the aligned MIDI.

**Architecture:** Pure alignment logic in `src/align/` (SMF parsing, warp model, MIDI rendering via alphaTab, features, DTW auto-aligner, fit quality). File/manifest I/O in `src/ref/`. A token-protected `Bun.serve` app (`src/align/server.ts` + HTML import page) exposes songs, waveform peaks, audio, warped MIDI renders, auto-align and save. `src/bench/real.ts` scores real songs.

**Tech Stack:** Bun 1.3 + TypeScript, alphaTab 1.8.4 (synth), existing `src/dsp`, `src/pipeline/run.ts`, `src/eval/*`, ffmpeg CLI (transcode only), Web Audio API in the page.

**Spec:** `docs/superpowers/specs/2026-10-30-reference-alignment-design.md`

## Global Constraints

- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports, no vite/express, `bun test`, `Bun.file`/`Bun.write`, `Bun.$`).
- `src/align/**` except `server.ts`, `cli.ts` and `page/**` must not use `node:fs`/`Bun.*`/`process` (browser-portable); I/O lives in `src/ref/**`, `src/align/server.ts`, `src/bench/**`.
- Never commit reference audio, MIDI, renders, `reference/**`, or `*.flac/*.mid`. Stage files by explicit path only; check `git status --short` before committing.
- Manifest: `reference/songs.json`; per song dir `reference/<songId>/` holds `alignment.json`, cached transcoded audio, peaks.
- Alignment = anchors `{ midiSec, audioSec, kind: "manual" | "auto" }`, monotonic piecewise-linear MIDI→audio map; implicit anchors at start (0→0 unless an anchor exists) and end (extrapolate with the last stretch's slope).
- Roles scored: guitar (merged), vocals, bass, drums (by class, existing heuristic transcriber), keys. Unmapped tracks excluded and listed.
- Fit quality: recording onsets from raw-audio onset flux (never from the pipeline's transcription); MIDI onsets = unique warped onset times of scored-role notes; match ±70 ms; report median |residual| (ms) and match rate, overall and per 30 s window; window flagged when match rate < 0.5 or median residual > 60 ms.
- Editing model: adding/moving a manual anchor deletes auto anchors in the two affected stretches; "Auto-align around my anchors" re-solves autos between consecutive manual anchors; auto never runs implicitly on drag.
- Server binds configurable `--host` (default 127.0.0.1) / `--port`; prints URL with random token; rejects requests without token (query `?key=` once, then cookie); file access only for manifest entries.
- `bench real`: report-only; skips songs without `alignment.json` unless `--auto`; per-run artefacts `runs/real-<runId>/` (ignored); summary `results/real-<runId>.{md,json}` (titles, scores, alignment source + fit only) committed.
- Commits end with the Co-Authored-By line from your own system reminder.

## Review Focus

1. **Anchors at or beyond the ends / out of order** — warp must stay monotonic: reject (throw) non-increasing anchors; the page prevents dragging past neighbours. Test in Task 2.
2. **Rendered MIDI clipping** — SMF renders peak > 1 (Holes peaked 2.13); render must peak-normalise to −1 dBFS. Test in Task 3.
3. **Silence / no onsets in a 30 s window** — fit quality reports `null` for that window (not NaN) and doesn't flag it. Test in Task 5.
4. **Path traversal / unknown song ids** on the server — 404, never read outside manifest paths. Test in Task 6.
5. **Huge DTW matrices** — coarse pass at ≈10 Hz over a 5-min song is ~3000×3000; must stay under ~200 MB and ~5 s CPU. Test timing in Task 5.

---

## File Structure

```
src/align/smf.ts          parse SMF → { durationSec, tracks: RefTrack[], tempoMap }
src/align/roles.ts        guessRole(track) → Role | null
src/align/warp.ts         anchors model: makeWarp, invert, stretches, edit ops
src/align/render.ts       renderMidi(tracks, warp, sf2) → Audio (alphaTab synth)
src/align/features.ts     onsetEnvelope, pickOnsets, chroma (frame-rate param)
src/align/dtw.ts          banded/unconstrained DTW
src/align/auto.ts         autoAlign(audio, tracks, manualAnchors) → Anchor[]
src/align/fit.ts          fitQuality(audioOnsets, warpedMidiOnsets) → FitReport
src/align/peaks.ts        waveform peaks (min/max per bucket)
src/align/server.ts       Bun.serve app (token, API, HTML import)
src/align/page/index.html, app.ts, style.css   layout-A page
src/align/cli.ts          `bun run align [serve|add|auto]`
src/ref/manifest.ts       read/write reference/songs.json, song dirs, transcode cache
src/bench/real.ts         real-song scoring + report
tests/align/**, tests/ref/**, tests/bench/real.test.ts
```

---

### Task 1: SMF parser, role guessing, manifest + `align add`

**Files:** Create `src/align/smf.ts`, `src/align/roles.ts`, `src/ref/manifest.ts`, `src/align/cli.ts`; modify `package.json` (`"align": "bun src/align/cli.ts"`). Tests: `tests/align/smf.test.ts`, `tests/ref/manifest.test.ts`.

**Interfaces (produces):**
```ts
// smf.ts
export interface RefNote { onset: number; offset: number; pitch: number; velocity: number; channel: number }
export interface RefTrack { index: number; name: string; program: number; isDrums: boolean; notes: RefNote[] }
export interface TempoPoint { sec: number; bpm: number }
export interface Smf { durationSec: number; tracks: RefTrack[]; tempoMap: TempoPoint[]; timeSigs: { sec: number; num: number; den: number }[] }
export function parseSmf(bytes: Uint8Array): Smf
// roles.ts
export function guessRole(t: RefTrack): Role | null   // drums (ch 10) / vocal / bass / guitar|delay / piano|key|synth; else null
// manifest.ts
export interface SongEntry { id: string; title: string; audio: string; midi: string; roles: Record<string, Role | null>; capo: Record<string, number>; tuning?: Record<string, number[]> }
export async function readManifest(root = "reference"): Promise<SongEntry[]>
export async function addSong(e: Omit<SongEntry, "roles"> & { roles?: SongEntry["roles"] }, root = "reference"): Promise<SongEntry>  // fills roles via guessRole, keyed by track name
export function songDir(id: string, root = "reference"): string
```
CLI: `bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]`.

- [ ] **Step 1: Write failing tests** — build a tiny SMF in the test (format 1, ppq 480, tempo 120 then 60 at tick 1920, a guitar track program 25 with two notes, a ch-10 drum note): assert note onsets in seconds (0.0, 0.5 → second-tempo section at 2.0 s + 1.0 s), `tempoMap` has 2 points, drums `isDrums`, `guessRole` returns "guitar"/"drums"; running-status and note-on velocity 0 = note-off handled. Manifest: `addSong` in a temp root writes `songs.json`, guesses roles, rejects duplicate ids.
- [ ] **Step 2:** FAIL. **Step 3:** implement (port the parser in `.scratch/realsong.ts` but clean). **Step 4:** PASS + typecheck.
- [ ] **Step 5:** Commit `feat(align): SMF parser, role guess, reference manifest`.

### Task 2: Warp model

**Files:** Create `src/align/warp.ts`; test `tests/align/warp.test.ts`.

**Interfaces:**
```ts
export type AnchorKind = "manual" | "auto";
export interface Anchor { midiSec: number; audioSec: number; kind: AnchorKind }
export interface Warp { toAudio(midiSec: number): number; toMidi(audioSec: number): number; anchors: Anchor[] }
export function makeWarp(anchors: Anchor[]): Warp            // sorts by midiSec; throws if audioSec not strictly increasing; 0 anchors = identity; 1 anchor = pure offset
export interface Stretch { fromMidi: number; toMidi: number; ratio: number; warn: boolean }  // ratio = audio dur / midi dur; warn when |1/ratio − 1| > 0.05
export function stretches(anchors: Anchor[], midiDur: number): Stretch[]
export function addManual(anchors: Anchor[], a: Omit<Anchor, "kind">): Anchor[]       // removes autos in the two stretches adjacent to the new anchor
export function moveAnchor(anchors: Anchor[], i: number, to: Omit<Anchor, "kind">): Anchor[] // becomes manual; same auto-removal; throws if it would cross a neighbour
export function deleteAnchor(anchors: Anchor[], i: number): Anchor[]
```
- [ ] **Step 1: Failing tests:** identity; single anchor = offset; two anchors interpolate and extrapolate with end slopes; `toMidi(toAudio(x)) ≈ x`; non-monotonic throws; `stretches` ratio + warn at 7%; `addManual` between autos removes only autos in adjacent stretches; `moveAnchor` crossing throws.
- [ ] **Step 2–4:** FAIL → implement → PASS. **Step 5:** Commit `feat(align): warp model`.

### Task 3: Warped MIDI rendering + waveform peaks

**Files:** Create `src/align/render.ts`, `src/align/peaks.ts`; tests `tests/align/render.test.ts`.

**Interfaces:**
```ts
export function renderMidi(tracks: RefTrack[], warp: Warp, sf2: Uint8Array, opts?: { sampleRate?: number; mono?: boolean; durationSec?: number }): Audio
export function peaks(x: Float32Array, buckets: number): { min: Float32Array; max: Float32Array }
```
Approach (verified by controller spike): build `at.midi.MidiFile` (division 960) at a fixed 120 bpm (`TempoChangeEvent(0, 500000)` → 1920 ticks/s); place each note at `round(warp.toAudio(onset) * 1920)`; drums on channel 9, others on channels 0–8, 10–15 assigned per track with a `ProgramChangeEvent`; render with `AlphaSynth.exportAudio` (dummy output, `useSyncPoints = false`) as in `src/audio/render.ts` (reuse its dummy output helper if exported; otherwise export it from there); peak-normalise to −1 dBFS; pad/trim to `durationSec` when given.
- [ ] **Step 1: Failing tests:** one guitar note at MIDI 1.0 s with warp offset +0.5 s → envelope onset in [1.48, 1.52] s; peak ≤ 10^(−1/20)+1e-3 on a dense 20-note chord; drums track audible; `peaks` of a known sine.
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): warped MIDI rendering and peaks`.

### Task 4: Features + DTW

**Files:** Create `src/align/features.ts`, `src/align/dtw.ts`; tests `tests/align/features.test.ts`, `tests/align/dtw.test.ts`.

**Interfaces:**
```ts
export interface Features { rate: number; onset: Float32Array; chroma: Float32Array[] }  // per frame; chroma 12-dim L2-normalised
export function features(audio: Audio, rate: number): Features
export function pickOnsets(onset: Float32Array, rate: number): number[]   // seconds; adaptive threshold (median + k·MAD), 50 ms min gap
export function dtw(costAt: (i: number, j: number) => number, n: number, m: number, band?: { center: (i: number) => number; width: number }): [number, number][]  // path start (0,0) → end (n-1,m-1), steps (1,0),(0,1),(1,1)
```
Reuse `src/dsp/fft.ts` `stft`, `src/dsp/mono.ts`, `src/dsp/resample.ts`. Cost for alignment = (1 − chroma cosine) + λ·|onset_a − onset_b| (λ = 1, onsets z-normalised).
- [ ] **Step 1: Failing tests:** DTW recovers a known diagonal-with-kink path on synthetic cost; banded DTW equals unbanded when band covers the path; features rate/length correct; `pickOnsets` on click train ±10 ms.
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): features and DTW`.

### Task 5: Auto aligner + fit quality

**Files:** Create `src/align/auto.ts`, `src/align/fit.ts`; tests `tests/align/auto.test.ts`, `tests/align/fit.test.ts`.

**Interfaces:**
```ts
export interface FitWindow { startSec: number; matchRate: number | null; medianResidualMs: number | null; flagged: boolean }
export interface FitReport { matchRate: number | null; medianResidualMs: number | null; windows: FitWindow[] }
export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number): FitReport  // ±70 ms, 30 s windows, flag rule per Global Constraints
export function autoAlign(audio: Audio, smf: Smf, roles: Record<string, Role | null>, manual: Anchor[], sf2: Uint8Array): { anchors: Anchor[]; fit: FitReport }
```
Approach: render the (manual-anchored) warped MIDI with `renderMidi`; coarse DTW at 10 Hz per segment between consecutive manual anchors (+ start/end); refine at 50 Hz in a band (±0.5 s) around the coarse path; convert path to MIDI→audio pairs; reduce to one `auto` anchor per bar using `smf.tempoMap`/`timeSigs` bar times (skip bars within 1 s of a manual anchor); compose with manual anchors; compute fit with `pickOnsets` on the recording and warped scored-role MIDI onsets.
- [ ] **Step 1: Failing tests:** synthetic known warp — take generated song `t1c/1` (`buildSong`), write its truth as an SMF-like `Smf` object (no file needed), distort MIDI times with a known piecewise map (offset +0.4 s, section 2 ×1.06 tempo, section 3 −0.8 s) and align against the original mix: median |recovered − true| < 40 ms, overall fit matchRate > 0.7. With one correct manual anchor mid-song the result honours it exactly. `fitQuality`: empty window → nulls, not flagged; timing: 5-min synthetic input completes < 15 s.
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): automatic alignment and fit quality`.

### Task 6: Alignment server + page (layout A)

**Files:** Create `src/align/server.ts`, `src/align/page/index.html`, `src/align/page/app.ts`, `src/align/page/style.css`; extend `src/align/cli.ts` (`serve` default: `bun run align [--host] [--port]`; `bun run align auto <id>` computes autoAlign and saves `alignment.json`, printing the fit summary), `src/ref/manifest.ts` (transcode cache: `reference/<id>/audio.ogg` via ffmpeg Opus 160k + `recording.wav` 44.1 kHz for analysis; `alignment.json` read/write). Test: `tests/align/server.test.ts`.

**API (all require token):**
- `GET /` → page (HTML import). `GET /api/songs` → `[{ id, title, hasAlignment }]`.
- `GET /api/song/:id` → `{ id, title, audioDur, midiDur, tracks: [{name, role}], recordingPeaks, anchors, stretches, fit, midiTempo: tempoMap }` (auto-aligns and returns unsaved anchors if no `alignment.json`).
- `GET /api/song/:id/audio` → cached Opus.
- `POST /api/song/:id/render` body `{ anchors }` → WAV (mono 22.05 kHz) of warped MIDI + header-free JSON peaks via `GET /api/song/:id/render-peaks?hash=` (or return both; builder's call).
- `POST /api/song/:id/auto` body `{ anchors }` (manual ones used) → `{ anchors, stretches, fit }`.
- `PUT /api/song/:id/alignment` body `{ anchors }` → saves `{ anchors, savedAt, fit }`. `POST /api/song/:id/reset` → raw MIDI (no anchors).

**Page behaviour (layout A, from the approved mockup):** song picker; two lanes on a shared zoom/scroll timeline (wheel = zoom, drag background = scroll); recording peaks on top, warped-MIDI peaks below; anchors as green lines spanning both lanes, draggable at the recording end or the MIDI end (prevented from crossing neighbours), click-select + Delete key; "+ Anchor at cursor" (maps cursor recording time → `warp.toMidi` for the MIDI side); stretch labels "MIDI bpm → effective bpm (±%)" with ⚠ when warned; fit-flagged 30 s windows shaded red; click timeline to set cursor; Play/Pause from cursor; mix select (both / recording / MIDI / rec L + MIDI R) via Web Audio gain + stereo panner; MIDI render re-requested 400 ms after the last edit, old buffer keeps playing until replaced, both sources started at the same `AudioContext.currentTime`; buttons: Auto-align around my anchors, Save, Reset to raw MIDI; unsaved-changes indicator.
- [ ] **Step 1: Failing tests (API):** start server on port 0 with a temp reference root containing a fixture song built from `data/t1a/1` (mix.wav as audio, an SMF written from its truth — add `writeSmf(tracks)` helper in `src/align/smf.ts` for tests); no token → 401; `/api/songs` lists it; unknown id and `..` id → 404; `/api/song/:id` returns peaks + anchors; `PUT` then `GET` round-trips anchors; `/render` returns RIFF WAV.
- [ ] **Step 2–4:** FAIL → implement → PASS. Smoke: `bun run align --host 0.0.0.0` prints a tokenised URL; load it with `curl` and confirm HTML + bundle are served.
- [ ] **Step 5:** Commit `feat(align): alignment web page`.

### Task 7: Real-song benchmark

**Files:** Create `src/bench/real.ts`; modify `src/bench/cli.ts` (`real` subcommand); test `tests/bench/real.test.ts`.

**Interfaces:**
```ts
export interface RealSongResult { id: string; title: string; alignment: { source: "manual" | "auto"; manual: number; auto: number; savedAt: string | null }; fit: FitReport; perRole: Partial<Record<Role, MatchResult>>; drumsPerClass: Record<DrumClass, MatchResult> | null; excludedTracks: string[]; seconds: number }
export async function runReal(opts: { runId: string; auto: boolean; root?: string; out?: string }): Promise<RealSongResult[]>
export function renderRealReport(runId: string, results: RealSongResult[]): string
```
Behaviour: for each manifest song: load `alignment.json` (or `autoAlign` when `--auto`, else skip with a note); run `createPipeline` full mode on `recording.wav`; warp each scored-role MIDI note's onset/offset with `makeWarp(anchors).toAudio`; per role `matchNotes` (guitar merged; drums by class like `scoreSong`); write `runs/real-<runId>/<id>/{notes.json,out.gp,score.json}` and `results/real-<runId>.{md,json}` (strip `matches`). Report: per song alignment source/fit, role table (guitar first), excluded tracks.
- [ ] **Step 1: Failing tests:** `renderRealReport` fixture (guitar column first, fit shown, "auto" labelled); `runReal` on a temp root with the t1a fixture song + an identity alignment → guitar F1 equals the synthetic full-mode score for t1a/1 within 0.02; real `reference/` test `skipIf` absent.
- [ ] **Step 2–4.** **Step 5:** Commit `feat(bench): real-song benchmark`.

### Task 8: Seed songs, auto-align, first real benchmark

- [ ] **Step 1:** `bun run align add` the three songs (ids `lnoe`, `swimming`, `holes`; titles from file names; Holes `--capo "Acoustic Guitar =9"` — match the exact track name from the MIDI). Verify roles in `reference/songs.json` (Holes brass/crowd → null).
- [ ] **Step 2:** For each song run `bun run align auto <id>` (CLI: compute auto alignment and save it as `alignment.json` with all anchors `auto`), print fit summaries.
- [ ] **Step 3:** `bun run bench real --run-id real-auto-2026-10-30` (uses saved auto alignments) → results files.
- [ ] **Step 4:** README section "Aligning reference songs" (commands, where files live, the page workflow). Commit README + `results/real-auto-2026-10-30.{md,json}` by explicit path; push.