diff --git a/docs/superpowers/plans/2026-10-30-reference-alignment.md b/docs/superpowers/plans/2026-10-30-reference-alignment.md new file mode 100644 index 0000000..97cde7c --- /dev/null +++ b/docs/superpowers/plans/2026-10-30-reference-alignment.md @@ -0,0 +1,183 @@ +# Reference Alignment Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Align team-supplied GP-exported MIDI to real recordings (automatic pass + manual anchor editing in a local web page) and score the pipeline on real songs against the aligned MIDI. + +**Architecture:** Pure alignment logic in `src/align/` (SMF parsing, warp model, MIDI rendering via alphaTab, features, DTW auto-aligner, fit quality). File/manifest I/O in `src/ref/`. A token-protected `Bun.serve` app (`src/align/server.ts` + HTML import page) exposes songs, waveform peaks, audio, warped MIDI renders, auto-align and save. `src/bench/real.ts` scores real songs. + +**Tech Stack:** Bun 1.3 + TypeScript, alphaTab 1.8.4 (synth), existing `src/dsp`, `src/pipeline/run.ts`, `src/eval/*`, ffmpeg CLI (transcode only), Web Audio API in the page. + +**Spec:** `docs/superpowers/specs/2026-10-30-reference-alignment-design.md` + +## Global Constraints + +- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports, no vite/express, `bun test`, `Bun.file`/`Bun.write`, `Bun.$`). +- `src/align/**` except `server.ts` and `page/**` must not use `node:fs`/`Bun.*`/`process` (browser-portable); I/O lives in `src/ref/**`, `src/align/server.ts`, `src/bench/**`. +- Never commit reference audio, MIDI, renders, `reference/**`, or `*.flac/*.mid`. Stage files by explicit path only; check `git status --short` before committing. +- Manifest: `reference/songs.json`; per song dir `reference//` holds `alignment.json`, cached transcoded audio, peaks. +- Alignment = anchors `{ midiSec, audioSec, kind: "manual" | "auto" }`, monotonic piecewise-linear MIDI→audio map; implicit anchors at start (0→0 unless an anchor exists) and end (extrapolate with the last stretch's slope). +- Roles scored: guitar (merged), vocals, bass, drums (by class, existing heuristic transcriber), keys. Unmapped tracks excluded and listed. +- Fit quality: recording onsets from raw-audio onset flux (never from the pipeline's transcription); MIDI onsets = unique warped onset times of scored-role notes; match ±70 ms; report median |residual| (ms) and match rate, overall and per 30 s window; window flagged when match rate < 0.5 or median residual > 60 ms. +- Editing model: adding/moving a manual anchor deletes auto anchors in the two affected stretches; "Auto-align around my anchors" re-solves autos between consecutive manual anchors; auto never runs implicitly on drag. +- Server binds configurable `--host` (default 127.0.0.1) / `--port`; prints URL with random token; rejects requests without token (query `?key=` once, then cookie); file access only for manifest entries. +- `bench real`: report-only; skips songs without `alignment.json` unless `--auto`; per-run artefacts `runs/real-/` (ignored); summary `results/real-.{md,json}` (titles, scores, alignment source + fit only) committed. +- Commits end with the Co-Authored-By line from your own system reminder. + +## Review Focus + +1. **Anchors at or beyond the ends / out of order** — warp must stay monotonic: reject (throw) non-increasing anchors; the page prevents dragging past neighbours. Test in Task 2. +2. **Rendered MIDI clipping** — SMF renders peak > 1 (Holes peaked 2.13); render must peak-normalise to −1 dBFS. Test in Task 3. +3. **Silence / no onsets in a 30 s window** — fit quality reports `null` for that window (not NaN) and doesn't flag it. Test in Task 5. +4. **Path traversal / unknown song ids** on the server — 404, never read outside manifest paths. Test in Task 6. +5. **Huge DTW matrices** — coarse pass at ≈10 Hz over a 5-min song is ~3000×3000; must stay under ~200 MB and ~5 s CPU. Test timing in Task 5. + +--- + +## File Structure + +``` +src/align/smf.ts parse SMF → { durationSec, tracks: RefTrack[], tempoMap } +src/align/roles.ts guessRole(track) → Role | null +src/align/warp.ts anchors model: makeWarp, invert, stretches, edit ops +src/align/render.ts renderMidi(tracks, warp, sf2) → Audio (alphaTab synth) +src/align/features.ts onsetEnvelope, pickOnsets, chroma (frame-rate param) +src/align/dtw.ts banded/unconstrained DTW +src/align/auto.ts autoAlign(audio, tracks, manualAnchors) → Anchor[] +src/align/fit.ts fitQuality(audioOnsets, warpedMidiOnsets) → FitReport +src/align/peaks.ts waveform peaks (min/max per bucket) +src/align/server.ts Bun.serve app (token, API, HTML import) +src/align/page/index.html, app.ts, style.css layout-A page +src/align/cli.ts `bun run align [serve|add|auto]` +src/ref/manifest.ts read/write reference/songs.json, song dirs, transcode cache +src/bench/real.ts real-song scoring + report +tests/align/**, tests/ref/**, tests/bench/real.test.ts +``` + +--- + +### Task 1: SMF parser, role guessing, manifest + `align add` + +**Files:** Create `src/align/smf.ts`, `src/align/roles.ts`, `src/ref/manifest.ts`, `src/align/cli.ts`; modify `package.json` (`"align": "bun src/align/cli.ts"`). Tests: `tests/align/smf.test.ts`, `tests/ref/manifest.test.ts`. + +**Interfaces (produces):** +```ts +// smf.ts +export interface RefNote { onset: number; offset: number; pitch: number; velocity: number; channel: number } +export interface RefTrack { index: number; name: string; program: number; isDrums: boolean; notes: RefNote[] } +export interface TempoPoint { sec: number; bpm: number } +export interface Smf { durationSec: number; tracks: RefTrack[]; tempoMap: TempoPoint[]; timeSigs: { sec: number; num: number; den: number }[] } +export function parseSmf(bytes: Uint8Array): Smf +// roles.ts +export function guessRole(t: RefTrack): Role | null // drums (ch 10) / vocal / bass / guitar|delay / piano|key|synth; else null +// manifest.ts +export interface SongEntry { id: string; title: string; audio: string; midi: string; roles: Record; capo: Record; tuning?: Record } +export async function readManifest(root = "reference"): Promise +export async function addSong(e: Omit & { roles?: SongEntry["roles"] }, root = "reference"): Promise // fills roles via guessRole, keyed by track name +export function songDir(id: string, root = "reference"): string +``` +CLI: `bun run align add