chore: bun init, implementation plan, model manifest
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
f2ca686b87
commit
62cab141bd
@@ -0,0 +1,417 @@
|
||||
# Synthetic Benchmark Harness Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Generate original tier-1 songs as Guitar Pro files, render them to audio, run a baseline browser-portable transcription pipeline on the audio, and score the output against the source file.
|
||||
|
||||
**Architecture:** One shared symbolic `Song` model is written to alphaTex and turned into Guitar Pro / MIDI / audio by alphaTab. The pipeline (Demucs-6s ONNX → Basic Pitch ONNX / heuristic drums → tempo grid → quantise → tab assignment) produces timed notes plus a `Song` that goes back out through the same alphaTex writer. A scorer compares timed notes to the ground truth extracted from the GP file.
|
||||
|
||||
**Tech Stack:** Bun 1.3 + TypeScript (strict), `@coderline/[email protected]` (MPL-2.0), `[email protected]` (MIT), `@spotify/[email protected]` post-processing only (Apache-2.0), ffmpeg CLI (mp3 encode only), models: `Kani95/htdemucs-6s-ort` (Apache-2.0, sha256 `35cb61d4d35173ff1d16df7393f41f05651ffdfe13d3c10e06abc26708bd6933`), Spotify `nmp.onnx` (Apache-2.0).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Bun + TypeScript for everything; follow `CLAUDE.md` (bun test, `Bun.file`/`Bun.write`, `Bun.$`, `bunx`). No Python on any path in this plan.
|
||||
- Models run via `onnxruntime-node`; code touching ORT must only use APIs that also exist in `onnxruntime-web` (`InferenceSession.create`, `Tensor`, `session.run`).
|
||||
- CPU only. Use `executionProviders: ["cpu"]`.
|
||||
- Pipeline code (`src/pipeline/**`, `src/dsp/**`, `src/score/**`) must not import `node:fs`/`Bun.*`; file I/O lives in `src/bench/**`, `src/audio/wav.ts`, `src/audio/mp3.ts`. (Keeps the pipeline browser-portable.)
|
||||
- Generated material is original; no reference songs are recreated.
|
||||
- Tier-1 benchmark songs: at most two guitars, standard tuning `[64,59,55,50,45,40]` (MIDI, highest string first), 4/4.
|
||||
- Pitch score: onset tolerance ±50 ms, offsets ignored, string ignored; computed on timed notes **before** quantisation.
|
||||
- Song pass: merged-guitar pitch F1 ≥ 0.85. Tier pass: median song passes.
|
||||
- Rhythm alignment allows tempo factor ½×, 1×, 2× and whole-beat bar offsets; factor/offset reported separately.
|
||||
- Every result is labelled with its oracle flags (`oracleStems`, `oracleTempo`, `oracleNotes`).
|
||||
- Reports order instruments: guitar, vocals, then bass, drums, keys.
|
||||
- Generated audio, `data/`, `runs/`, `models/*.onnx` are git-ignored; `results/*.md` + `results/*.json` summaries are committed.
|
||||
- SoundFont: alphaTab's bundled `sonivox.sf2` (Apache-2.0).
|
||||
- Commits end with `Co-Authored-By: Claude Opus 5.5 <[email protected]>`.
|
||||
|
||||
## Out of Scope for This Plan
|
||||
|
||||
- The optional Python comparison track (spec: "optional") — deferred to a follow-up plan once baseline numbers exist.
|
||||
- Website UI, YouTube ingestion, later tiers.
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. **Durations that aren't a single note value or that cross a barline** (e.g. 5 sixteenths, a whole note starting on beat 3) — writer must emit tied notes so the GP round trip preserves every onset and pitch. Test in Task 2.
|
||||
2. **Silent or absent stems** (no vocals in t1a; a stem Demucs leaves near-silent) — transcribers return `[]`, scorer reports `null` metrics (not NaN) and the report shows "n/a". Tests in Tasks 6, 8.
|
||||
3. **Audio length not a multiple of the Demucs segment, or shorter than one segment** — separation output length must equal input length exactly. Test in Task 7.
|
||||
4. **Pitches outside the instrument's fret range** (octave errors from transcription) — tab assigner drops them, counts them, never throws. Test in Task 11.
|
||||
5. **Synth onset latency** — rendered audio must put note energy within 20 ms of ground-truth onset, otherwise every score is silently shifted. Test in Task 4.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
scripts/fetch-models.ts download + sha256-verify models into models/
|
||||
src/core/types.ts Song, Track, ScoreNote, TimedNote, TruthTrack, Role, Grid, Audio
|
||||
src/core/time.ts tick/second conversion (TPQ = 960)
|
||||
src/core/rng.ts seeded PRNG
|
||||
src/score/alphatex.ts songToAlphaTex(song)
|
||||
src/score/gp.ts songToScore / songToGp / truthFromGp (alphaTab)
|
||||
src/gen/harmony.ts keys, progressions, guitar chord shapes
|
||||
src/gen/parts.ts per-role part generators
|
||||
src/gen/tiers.ts tier specs t1a..t1d
|
||||
src/gen/generate.ts generateSong(tier, seed)
|
||||
src/audio/wav.ts read/write WAV, toMono, resample
|
||||
src/audio/mp3.ts ffmpeg encode
|
||||
src/audio/render.ts per-track stem rendering via alphaSynth
|
||||
src/audio/fx.ts drive, chorus, reverb, pan, gain
|
||||
src/audio/mix.ts mix stems
|
||||
src/dsp/fft.ts radix-2 FFT, torch-compatible stft/istft
|
||||
src/pipeline/separate.ts Demucs 6s ONNX
|
||||
src/pipeline/transcribe.ts Basic Pitch ONNX → TimedNote[]
|
||||
src/pipeline/drums.ts heuristic kick/snare/hat transcriber
|
||||
src/pipeline/tempo.ts tempo + downbeat grid
|
||||
src/pipeline/quantize.ts TimedNote[] → ScoreNote[]
|
||||
src/pipeline/tab.ts string/fret assignment (Viterbi)
|
||||
src/pipeline/run.ts orchestrates pipeline incl. oracle modes
|
||||
src/eval/notes.ts note matching / F1 / octave errors
|
||||
src/eval/rhythm.ts grid alignment + rhythm score
|
||||
src/eval/score.ts scoreSong
|
||||
src/bench/dataset.ts build song dirs on disk
|
||||
src/bench/report.ts markdown + JSON tier report
|
||||
src/bench/cli.ts `bun run bench …` entry
|
||||
tests/** bun tests mirroring src
|
||||
```
|
||||
|
||||
On-disk layout: `data/<tier>/<seed>/{song.json, truth.gp, stems/<trackName>.wav, mix.wav, mix.mp3}`; `runs/<runId>/<tier>/<seed>/{notes.json, out.gp, score.json}`; `results/<runId>.{md,json}`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Scaffolding, core types, time, RNG, model fetch
|
||||
|
||||
**Files:**
|
||||
- Create: `src/core/types.ts`, `src/core/time.ts`, `src/core/rng.ts`, `scripts/fetch-models.ts`
|
||||
- Modify: `package.json` (scripts), `.gitignore`, delete `index.ts`
|
||||
- Test: `tests/core/time.test.ts`, `tests/core/rng.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces (`src/core/types.ts`):
|
||||
```ts
|
||||
export type Role = "guitar" | "bass" | "keys" | "vocals" | "drums";
|
||||
export type FxPreset = "none" | "clean" | "clean-chorus" | "light-drive";
|
||||
export interface ScoreNote { tick: number; dur: number; pitch: number; string?: number; fret?: number }
|
||||
// string numbering: 1 = highest-pitched string (alphaTex convention). Drums: pitch = GM drum number.
|
||||
export interface Track { name: string; role: Role; program: number; tuning?: number[]; pan: number; fx: FxPreset; gainDb: number; notes: ScoreNote[] }
|
||||
export interface Song { title: string; tier: string; seed: number; bpm: number; timeSig: [number, number]; bars: number; key: string; tracks: Track[] }
|
||||
export interface TimedNote { onset: number; offset: number; pitch: number; velocity?: number; string?: number; fret?: number }
|
||||
export interface TruthTrack { name: string; role: Role; tuning?: number[]; notes: TimedNote[] }
|
||||
export interface Grid { bpm: number; firstDownbeat: number; beatsPerBar: number }
|
||||
export interface Audio { sampleRate: number; channels: Float32Array[] }
|
||||
export const TPQ = 960;
|
||||
export const GUITAR_STANDARD = [64, 59, 55, 50, 45, 40];
|
||||
export const BASS_STANDARD = [43, 38, 33, 28];
|
||||
```
|
||||
- Produces: `ticksToSeconds(tick: number, bpm: number): number`, `secondsToTicks(s: number, bpm: number): number` in `src/core/time.ts`; `createRng(seed: number): Rng` with `next(): number` (0..1), `int(lo: number, hi: number): number` (inclusive), `pick<T>(a: readonly T[]): T`, `chance(p: number): boolean` in `src/core/rng.ts` (mulberry32).
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
```ts
|
||||
test("ticks/seconds", () => {
|
||||
expect(ticksToSeconds(960, 60)).toBeCloseTo(1);
|
||||
expect(ticksToSeconds(3840, 120)).toBeCloseTo(2);
|
||||
expect(secondsToTicks(0.5, 120)).toBeCloseTo(960);
|
||||
});
|
||||
test("rng deterministic", () => {
|
||||
const a = createRng(42), b = createRng(42), c = createRng(43);
|
||||
const sa = [a.next(), a.next(), a.next()];
|
||||
expect([b.next(), b.next(), b.next()]).toEqual(sa);
|
||||
expect(c.next()).not.toBe(sa[0]);
|
||||
const r = createRng(1); for (let i = 0; i < 1000; i++) { const v = r.int(3, 5); expect(v >= 3 && v <= 5).toBe(true); }
|
||||
});
|
||||
```
|
||||
- [ ] **Step 2:** `bun test tests/core` → FAIL (modules missing).
|
||||
- [ ] **Step 3:** Implement types, time, rng. `package.json` scripts: `"test": "bun test"`, `"typecheck": "bunx tsc --noEmit"`, `"fetch-models": "bun scripts/fetch-models.ts"`, `"bench": "bun src/bench/cli.ts"`. `.gitignore` add `data/`, `runs/`, `models/**/*.onnx`. `scripts/fetch-models.ts` downloads `https://huggingface.co/Kani95/htdemucs-6s-ort/resolve/main/htdemucs_6s.onnx` → `models/demucs/htdemucs_6s.onnx` and `https://github.com/spotify/basic-pitch/raw/main/basic_pitch/saved_models/icassp_2022/nmp.onnx` → `models/basic-pitch/nmp.onnx`, skips existing files whose sha256 matches, fails loudly on mismatch (Demucs hash above; record the nmp hash from the current file on first run as a constant).
|
||||
- [ ] **Step 4:** `bun test tests/core` → PASS; `bun run typecheck` → no errors.
|
||||
- [ ] **Step 5:** Commit `feat: core types, time, rng, model fetch`.
|
||||
|
||||
### Task 2: alphaTex writer and Guitar Pro round trip
|
||||
|
||||
**Files:**
|
||||
- Create: `src/score/alphatex.ts`, `src/score/gp.ts`
|
||||
- Test: `tests/score/roundtrip.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 1 types.
|
||||
- Produces: `songToAlphaTex(song: Song): string`; `songToScore(song: Song): alphaTab.model.Score`; `songToGp(song: Song): Uint8Array` (Gp7Exporter); `truthFromGp(bytes: Uint8Array): { bpm: number; timeSig: [number, number]; tracks: TruthTrack[] }`; `songToMidi(song: Song): alphaTab.midi.MidiFile`.
|
||||
|
||||
Writer rules (the implementer cannot infer these):
|
||||
- Header `\title`, `\tempo <bpm>`, `\ts <n> <d>`; per track `\track "<name>"`, `\instrument <program>` (drums: `\instrument percussion \articulation defaults`, notes written as GM numbers), guitar/bass: `\staff {tabs}` + `\tuning (...)` written as note names highest string first, notes as `fret.string`; keys/vocals: `\staff {score}`, notes as note names (e.g. `C#4`).
|
||||
- Each track is one voice. Notes with equal `tick` form a chord beat; their beat duration is the shortest `dur` in the group, clipped to the next onset in the track. Gaps become rests.
|
||||
- Durations are decomposed greedily into {1, 2, 4, 8, 16, 32} with optional single dots, split at barlines, joined with ties (`-` tie syntax on the continuation note; verify alphaTex tie syntax against alphaTab 1.8 docs/diagnostics).
|
||||
- `truthFromGp` uses absolute tick = `masterBar.start + beat.playbackStart`, seconds via `ticksToSeconds` with the score tempo (tier 1 is constant tempo). Tied continuation notes are merged into the original note (one TimedNote). String converted to our numbering: `ours = stringCount - model.string + 1`. Drum pitch = `track.percussionArticulations[n.percussionArticulation].outputMidiNumber`. Role is stored as the alphaTex track short name (`\track "<name>" "<role>"`) and recovered from `track.shortName`.
|
||||
|
||||
- [ ] **Step 1: Write failing tests** — build a hand-made `Song` (bpm 90) with: a guitar track containing a C-major chord at tick 0 (5 notes with string/fret), a single note of `dur = 5 * 240` (5 sixteenths), and a whole note starting at beat 3 of bar 1 (crosses barline); a vocals track with a rest gap; a drums track with kick 36 + hat 42 together and a snare 38. Assertions:
|
||||
```ts
|
||||
const truth = truthFromGp(songToGp(song));
|
||||
// every source note appears exactly once with same pitch and onset (±1 ms), ties merged:
|
||||
expect(sortNotes(truth.tracks[0].notes).map(n => [n.pitch, round3(n.onset)])).toEqual(expectedGuitar);
|
||||
expect(truth.tracks[0].notes.find(n => n.pitch === tiedPitch)!.offset - onset).toBeCloseTo(4 * 60/90, 3);
|
||||
expect(truth.tracks[0].notes.every(n => n.string! >= 1 && n.string! <= 6)).toBe(true);
|
||||
expect(truth.tracks.find(t => t.role === "drums")!.notes.map(n => n.pitch).sort()).toEqual([36, 38, 42]);
|
||||
expect(truth.tracks.map(t => t.role)).toEqual(["guitar", "vocals", "drums"]);
|
||||
```
|
||||
plus: alphaTex parsing emits no error diagnostics (capture `Logger` or use the importer's diagnostics API) for this song.
|
||||
- [ ] **Step 2:** `bun test tests/score` → FAIL.
|
||||
- [ ] **Step 3:** Implement writer + gp.ts.
|
||||
- [ ] **Step 4:** `bun test tests/score` → PASS.
|
||||
- [ ] **Step 5:** Commit `feat: alphaTex writer and GP round trip`.
|
||||
|
||||
### Task 3: Tier-1 song generator
|
||||
|
||||
**Files:**
|
||||
- Create: `src/gen/harmony.ts`, `src/gen/parts.ts`, `src/gen/tiers.ts`, `src/gen/generate.ts`
|
||||
- Test: `tests/gen/generate.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: types, rng, `songToGp`/`truthFromGp` (test only).
|
||||
- Produces: `type TierId = "t1a" | "t1b" | "t1c" | "t1d"`; `TIERS: Record<TierId, TierSpec>`; `generateSong(tier: TierId, seed: number): Song`.
|
||||
|
||||
Tier content (decided):
|
||||
| Tier | Tracks | Notes |
|
||||
|---|---|---|
|
||||
| t1a | 1 clean guitar | solo; 66–96 bpm; mix of strummed open chords and arpeggios; one short single-note melodic phrase allowed |
|
||||
| t1b | guitar + bass + drums | bass on chord roots (quarters/halves); drums: kick/snare/closed-hat groove, crash on section starts |
|
||||
| t1c | t1b + keys (pad/piano chords) + vocals (program 53 "Voice Oohs", monophonic melody in C4–G5) | keys voiced in C3–C5 |
|
||||
| t1d | t1c + second guitar | guitars get distinct roles (strum vs arpeggio/lead line), pans −0.6 / +0.6, one `clean`, one `clean-chorus` or `light-drive` |
|
||||
|
||||
Common: 4/4, 16 bars split into sections (e.g. A A B A, 4 bars each, repeats identical), keys from {C, G, D, A, E, F} major + relative minors, progressions from a fixed list of pop progressions (I–V–vi–IV, vi–IV–I–V, I–vi–IV–V, I–IV–vi–V). Guitar chords come from an open-chord shape dictionary in `harmony.ts` (each shape = per-string fret or mute, highest string first) so every guitar note carries a valid `string`/`fret`; arpeggios draw from the same shape. Guitar programs: 25 (steel) or 27 (clean electric). Bass program 33, keys 0 or 89, drums percussion.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
```ts
|
||||
for (const tier of ["t1a","t1b","t1c","t1d"] as const) for (const seed of [1,2,3]) {
|
||||
const s = generateSong(tier, seed);
|
||||
expect(generateSong(tier, seed)).toEqual(s); // deterministic
|
||||
const guitars = s.tracks.filter(t => t.role === "guitar");
|
||||
expect(guitars.length).toBe(tier === "t1d" ? 2 : 1);
|
||||
for (const g of guitars) for (const n of g.notes) expect(GUITAR_STANDARD[n.string! - 1]! + n.fret!).toBe(n.pitch);
|
||||
expect(s.tracks.map(t => t.role).sort()).toEqual(EXPECTED_ROLES[tier]);
|
||||
const end = s.bars * 4 * TPQ;
|
||||
for (const t of s.tracks) for (const n of t.notes) expect(n.tick + n.dur).toBeLessThanOrEqual(end);
|
||||
expect(truthFromGp(songToGp(s)).tracks.length).toBe(s.tracks.length); // valid GP
|
||||
}
|
||||
const d = generateSong("t1d", 7); const [g1, g2] = d.tracks.filter(t => t.role === "guitar");
|
||||
expect(Math.sign(g1!.pan)).not.toBe(Math.sign(g2!.pan));
|
||||
expect(generateSong("t1a", 1)).not.toEqual(generateSong("t1a", 2));
|
||||
```
|
||||
- [ ] **Step 2:** `bun test tests/gen` → FAIL.
|
||||
- [ ] **Step 3:** Implement.
|
||||
- [ ] **Step 4:** `bun test tests/gen` → PASS.
|
||||
- [ ] **Step 5:** Commit `feat: tier-1 song generator`.
|
||||
|
||||
### Task 4: Audio I/O, stem rendering, effects, mixing
|
||||
|
||||
**Files:**
|
||||
- Create: `src/audio/wav.ts`, `src/audio/mp3.ts`, `src/audio/render.ts`, `src/audio/fx.ts`, `src/audio/mix.ts`
|
||||
- Test: `tests/audio/audio.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `songToMidi`, `songToScore`, types.
|
||||
- Produces: `readWav(path: string): Promise<Audio>`; `writeWav(path: string, a: Audio): Promise<void>` (PCM16); `toMono(a: Audio): Float32Array`; `resample(x: Float32Array, from: number, to: number): Float32Array` (windowed-sinc, used for 44100→22050 for Basic Pitch); `encodeMp3(wavPath: string, mp3Path: string): Promise<void>` (`ffmpeg -y -loglevel error -i … -b:a 192k`); `renderStems(song: Song, sf2: Uint8Array, sampleRate = 44100): Map<string, Audio>` (stereo, centred, one per track name, all the same length = song length + 2 s tail); `applyFx(a: Audio, preset: FxPreset, sampleRate: number): Audio`; `pan(a: Audio, p: number): Audio` (constant-power); `mixStems(stems: { audio: Audio; gainDb: number }[]): Audio` (sum, then peak-normalise to −1 dBFS).
|
||||
|
||||
Rendering: one `AlphaSynth.exportAudio` per track with `AudioExportOptions.trackVolume` muting all other tracks (verify key semantics; fallback: build a single-track Score per track), `useSyncPoints = false`, `render(500)` until `undefined`, de-interleave. Fx are simple and deterministic: `light-drive` = tanh waveshaper with pre-gain 4 + one-pole low-pass 5 kHz; `clean-chorus` = 2 modulated delay lines (~15 ms, 0.8 Hz); every pitched track gets a light shared Freeverb-style reverb (wet 0.15). Oracle stems written to disk are post-fx, post-pan — exactly what went into the mix.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
- WAV round trip: random stereo 1 s → write → read → max abs diff < 1e-4.
|
||||
- `pan(a, -1)`: right-channel RMS < 1e-3 × left RMS.
|
||||
- Rendering `generateSong("t1b", 1)`: stems map has 3 entries, all equal length ≥ song seconds; each stem RMS > 1e-3.
|
||||
- **Onset latency:** render a song with one guitar note at tick 960 (bpm 60 → 1.0 s): first sample where the 5 ms-RMS envelope exceeds 10% of its max lies in [0.98, 1.02] s.
|
||||
- `mixStems` peak ≈ 10^(−1/20) ± 1e-3.
|
||||
- [ ] **Step 2:** `bun test tests/audio` → FAIL.
|
||||
- [ ] **Step 3:** Implement.
|
||||
- [ ] **Step 4:** `bun test tests/audio` → PASS.
|
||||
- [ ] **Step 5:** Commit `feat: stem rendering, fx and mixing`.
|
||||
|
||||
### Task 5: Dataset builder and CLI skeleton
|
||||
|
||||
**Files:**
|
||||
- Create: `src/bench/dataset.ts`, `src/bench/cli.ts`
|
||||
- Test: `tests/bench/dataset.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `buildSong(tier: TierId, seed: number, root = "data"): Promise<string>` (returns song dir; writes `song.json`, `truth.gp`, `stems/<trackName>.wav`, `mix.wav`, `mix.mp3`; idempotent — skips if `mix.mp3` exists and `song.json` matches); `loadSongDir(dir: string): Promise<{ song: Song; truth: ReturnType<typeof truthFromGp>; mix: Audio; stems: Map<string, Audio> }>`.
|
||||
- CLI: `bun run bench gen --tiers t1a,t1b,t1c,t1d --seeds 1-5`.
|
||||
|
||||
- [ ] **Step 1: Write failing test** — `buildSong("t1a", 1, tmpDir)` creates all five artefacts; `loadSongDir` returns `truth.tracks[0].notes.length === song.tracks[0].notes.length` (ties are not generated by the tier-1 generator, so counts match) and `mix.channels.length === 2`.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS; run `bun run bench gen --tiers t1a --seeds 1-2` and confirm files exist.
|
||||
- [ ] **Step 5:** Commit `feat: dataset builder`.
|
||||
|
||||
### Task 6: Evaluation (notes, rhythm, song score)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/eval/notes.ts`, `src/eval/rhythm.ts`, `src/eval/score.ts`
|
||||
- Test: `tests/eval/notes.test.ts`, `tests/eval/rhythm.test.ts`, `tests/eval/score.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
```ts
|
||||
interface MatchResult { tp: number; fp: number; fn: number; precision: number | null; recall: number | null; f1: number | null; octaveErrors: number; matches: [number, number][] }
|
||||
matchNotes(ref: TimedNote[], est: TimedNote[], opts?: { onsetTol?: number /*0.05*/; key?: "pitch" | "pitchString" }): MatchResult
|
||||
interface GridAlignment { factor: 0.5 | 1 | 2; beatOffset: number }
|
||||
alignGrid(refNotes: ScoreNote[], estNotes: ScoreNote[]): GridAlignment
|
||||
rhythmScore(ref: ScoreNote[], est: ScoreNote[], align: GridAlignment): { onBeat: number | null; durationOk: number | null }
|
||||
scoreSong(truth: { bpm: number; timeSig: [number, number]; tracks: TruthTrack[] }, out: PipelineOutput, flags: OracleFlags): SongScore
|
||||
```
|
||||
`PipelineOutput` and `OracleFlags` are defined here in `src/eval/score.ts` (re-exported to Task 12):
|
||||
```ts
|
||||
interface OracleFlags { oracleStems: boolean; oracleTempo: boolean; oracleNotes: boolean }
|
||||
interface OutTrack { name: string; role: Role; notes: TimedNote[]; scoreNotes: ScoreNote[] }
|
||||
interface PipelineOutput { grid: Grid; tracks: OutTrack[] }
|
||||
interface SongScore { flags: OracleFlags; guitarMerged: MatchResult & { pass: boolean }; partSplit: { f1: number | null; outTracks: number; refTracks: number } | null;
|
||||
perRole: Partial<Record<Role, MatchResult>>; drumsPerClass: Record<"kick"|"snare"|"hihat"|"cymbal"|"tom", MatchResult> | null;
|
||||
rhythm: { factor: number; beatOffset: number; onBeat: number | null; durationOk: number | null } | null;
|
||||
stringAgreement: number | null; spuriousRate: number | null }
|
||||
```
|
||||
Rules: matching is maximum bipartite matching (onset within tol AND equal pitch); octave errors = unmatched est notes whose onset is within tol of an unmatched ref note with pitch ±12. F1 etc. are `null` when both ref and est are empty; precision `null` when est empty; recall `null` when ref empty. `guitarMerged` pools all guitar ref notes and all guitar out notes; `pass = f1 >= 0.85`. `partSplit`: when ≥2 guitar out tracks, Hungarian assignment maximising tp between out and ref guitar tracks, F1 over the assignment. `stringAgreement` = matched guitar notes with equal `string` / matched notes having a string. `spuriousRate` = fp / est count (guitar merged). Drum classes: kick {35,36}, snare {37,38,40}, hihat {42,44,46}, cymbal {49,51,52,53,55,57,59}, tom {41,43,45,47,48,50}; pitch compared by class. `alignGrid` tries factor ∈ {0.5,1,2} and beatOffset ∈ {−3..3} beats, picking the pair maximising exact tick-onset agreement of pitch-matched notes. Convention: `estTick = refTick * factor + beatOffset * TPQ`. `rhythmScore.onBeat` = share of pitch-matched notes landing on the identical aligned tick; `durationOk` = share with identical aligned duration.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
```ts
|
||||
const ref = [n(1.0,60), n(1.5,62), n(2.0,64)];
|
||||
expect(matchNotes(ref, [n(1.03,60), n(1.5,74), n(2.2,64)]).tp).toBe(1); // 74 = 62+12 → octave error; 2.2 s is outside tol
|
||||
expect(matchNotes(ref, [n(1.03,60), n(1.5,74), n(2.2,64)]).octaveErrors).toBe(1);
|
||||
expect(matchNotes(ref, ref).f1).toBe(1);
|
||||
expect(matchNotes([], []).f1).toBeNull();
|
||||
expect(matchNotes(ref, []).recall).toBe(0);
|
||||
expect(matchNotes(ref, []).precision).toBeNull();
|
||||
// grid: est written at double tempo and shifted one beat
|
||||
const r = sn([[0,240,60],[960,960,62]]); const e = sn([[960,480,60],[2880,1920,62]]);
|
||||
expect(alignGrid(r, e)).toEqual({ factor: 2, beatOffset: 1 });
|
||||
expect(rhythmScore(r, e, { factor: 2, beatOffset: 1 })).toEqual({ onBeat: 1, durationOk: 1 });
|
||||
```
|
||||
plus a `scoreSong` test with two ref guitar tracks and one merged out track (partSplit `null`-free path: `outTracks: 1`), and one with an absent vocals track on both sides → `perRole.vocals.f1 === null`.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: evaluation metrics`.
|
||||
|
||||
### Task 7: FFT/STFT and Demucs separation
|
||||
|
||||
**Files:**
|
||||
- Create: `src/dsp/fft.ts`, `src/pipeline/separate.ts`
|
||||
- Test: `tests/dsp/fft.test.ts`, `tests/pipeline/separate.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `fft(re: Float32Array, im: Float32Array): void` (in-place radix-2); `stft(x: Float32Array, nFft: number, hop: number): { re: Float32Array[]; im: Float32Array[] }` (frames × bins, torch semantics: hann window, `center=true`, reflect pad, `normalized=true`); `istft(spec, nFft, hop, length): Float32Array`; `type Stem = "drums" | "bass" | "other" | "vocals" | "guitar" | "piano"`; `createSeparator(modelBytes: Uint8Array): Promise<(mix: Audio) => Promise<Record<Stem, Audio>>>`.
|
||||
|
||||
Demucs details (from `htdemucs.py`, MIT): segment = 343980 samples, overlap 0.25, linear-fade (triangular) weights for overlap-add, whole-mix normalisation `(mix − mean(mono)) / std(mono)` before and inverse after. Per segment: spectral input `x` = `_spec`: nfft 4096, hop 1024, pad `hop/2*3 = 1536` each side reflect (right pad extended to `le*hop - T`, `le = ceil(T/hop)`), stft, drop last bin (→ 2048), keep frames `[2, 2+le)` (→ 336), complex-as-channels `[re_L, im_L, re_R, im_R]` → `[1,4,2048,336]`. Outputs: `output` `[1,6,4,2048,336]` → `_ispec` (re-add zero bin, pad 2 frames both sides, istft, crop `pad … pad+T`) and add `add_67` `[1,6,2,T]`. Stem order from manifest: drums, bass, other, vocals, guitar, piano. Verify graph-internal normalisation by checking the result quality test below; if separation is garbage, the per-segment normalisation must be applied outside the graph.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
- `fft` of an impulse → all-ones magnitude; `istft(stft(x)) ≈ x` (max err < 1e-4) for random 50 000-sample x with nFft 4096, hop 1024.
|
||||
- Separation on `data/t1b/1` (built via `buildSong`): output length === input length for every stem; SDR(guitar stem vs oracle guitar stem) > 3 dB and > SDR(mix vs oracle guitar stem); bass likewise.
|
||||
- A 3-second slice of the mix (shorter than one segment) returns 3 s stems without throwing.
|
||||
(Separation tests are slow; mark with a 300 s timeout.)
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS (`bun test tests/dsp tests/pipeline/separate.test.ts --timeout 300000`).
|
||||
- [ ] **Step 5:** Commit `feat: demucs 6-stem separation`.
|
||||
|
||||
### Task 8: Basic Pitch transcription
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pipeline/transcribe.ts`
|
||||
- Test: `tests/pipeline/transcribe.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `createTranscriber(modelBytes: Uint8Array): Promise<(audio: Audio, opts?: TranscribeOpts) => Promise<TimedNote[]>>` with `TranscribeOpts = { onsetThresh?: number /*0.5*/; frameThresh?: number /*0.3*/; minNoteMs?: number /*58*/; minPitch?: number; maxPitch?: number; monophonic?: boolean }`.
|
||||
|
||||
Approach: downmix + `resample` to 22050; window exactly like `@spotify/basic-pitch` `inference.js` (AUDIO_N_SAMPLES = 43844, overlap 30 frames, half-overlap zero pad, trim 15 frames each side, truncate to original frame count); ORT input `serving_default_input_2:0` `[n,43844,1]`; outputs map `StatefulPartitionedCall:{0,1,2}` → identify contour/note/onset by last-dim size (264 = contour, 88 = note & onset; disambiguate note vs onset by name order matching inference.js `OUTPUT_TO_TENSOR_NAME`). Post-process with `outputToNotesPoly`, `noteFramesToTime` imported from `@spotify/basic-pitch/esm/toMidi.js` (must not pull in tfjs — verify; if it does, vendor those functions with the Apache-2.0 header). `monophonic`: at each time keep only the note with highest amplitude, trimming overlaps. Silent input (RMS < 1e-4) returns `[]` without running the model.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
- On the oracle guitar stem of `data/t1a/1`: `matchNotes(truthGuitar, est).f1 > 0.5` (sanity floor; the real number goes in the benchmark).
|
||||
- On a vocals-like monophonic synthetic line (from `data/t1c/1` oracle vocals stem) with `monophonic: true`: no two output notes overlap in time.
|
||||
- 2 s of zeros → `[]`.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: basic pitch transcription`.
|
||||
|
||||
### Task 9: Tempo and downbeat grid
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pipeline/tempo.ts`
|
||||
- Test: `tests/pipeline/tempo.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `estimateGrid(audio: Audio, opts?: { minBpm?: number /*60*/; maxBpm?: number /*180*/; beatsPerBar?: number /*4*/ }): Grid`; `gridFromTruth(bpm: number, beatsPerBar: number): Grid` (oracle; `firstDownbeat = 0`).
|
||||
|
||||
Approach: spectral-flux onset envelope (hop 512 @ 44.1 kHz, log-magnitude, half-wave rectified), tempo by autocorrelation with a log-Gaussian prior centred at 100 bpm, beat phase by maximising envelope sum on the beat comb, downbeat by the beat-phase (0..3) whose comb has the most low-frequency (< 150 Hz) flux energy. Constant tempo only (tier 1).
|
||||
|
||||
- [ ] **Step 1: Write failing test** — for `t1b` and `t1c` seeds 1–5 mixes: `|est.bpm/true − f|/f < 0.02` for some f ∈ {0.5,1,2} in ≥ 8 of 10 songs; `firstDownbeat` mod beat period within 50 ms of 0 in ≥ 8 of 10.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: tempo grid estimation`.
|
||||
|
||||
### Task 10: Heuristic drum transcriber
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pipeline/drums.ts`
|
||||
- Test: `tests/pipeline/drums.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `transcribeDrums(audio: Audio): TimedNote[]` (pitches 36 kick, 38 snare, 42 hat, 49 cymbal).
|
||||
|
||||
Approach: band-split flux (kick < 150 Hz, snare 150 Hz–2 kHz with broadband noise, hats > 7 kHz, cymbal = hat-band onset with long decay > 300 ms), peak-pick each band with adaptive threshold (median + k·MAD), 30 ms min inter-onset per class. Labelled in reports as "heuristic baseline".
|
||||
|
||||
- [ ] **Step 1: Write failing test** — on oracle drum stems of `t1b` seeds 1–3: kick F1 > 0.7, snare F1 > 0.6 (class-wise via `scoreSong` drum classes or direct `matchNotes` on mapped pitches); silence → `[]`.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: heuristic drum transcription`.
|
||||
|
||||
### Task 11: Quantisation and tab assignment
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pipeline/quantize.ts`, `src/pipeline/tab.ts`
|
||||
- Test: `tests/pipeline/quantize.test.ts`, `tests/pipeline/tab.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `quantize(notes: TimedNote[], grid: Grid, stepsPerBeat = 4): ScoreNote[]` (ticks relative to `grid.firstDownbeat`, notes before it dropped; onsets snapped to nearest step, duration snapped to ≥ 1 step; chord notes within 35 ms collapse to the same tick); `assignTab(notes: ScoreNote[], tuning: number[], maxFret = 20): { notes: ScoreNote[]; dropped: number }`.
|
||||
|
||||
Tab assignment: group notes by tick into chords; candidates = all injective note→string mappings with `0 ≤ fret ≤ maxFret`; cost = fret span within chord (penalise > 4) + movement from previous hand position (mean non-zero fret) + 0.1·mean fret (prefer low positions) − small bonus for open strings; Viterbi over chords. Notes with no valid position on any string are dropped and counted; if a chord has more notes than strings, keep the highest-pitched `tuning.length` notes and count the rest as dropped.
|
||||
|
||||
- [ ] **Step 1: Write failing tests**
|
||||
```ts
|
||||
const g = { bpm: 120, firstDownbeat: 0.1, beatsPerBar: 4 };
|
||||
const q = quantize([t(0.1+0.5+0.02, 0.1+1.0, 60), t(0.1+0.75-0.015, 0.1+1.0, 64)], g);
|
||||
expect(q.map(n => [n.tick, n.dur, n.pitch])).toEqual([[960, 960, 60], [1440, 480, 64]]);
|
||||
const chordC = [48,52,55,60,64].map(p => sn(0, 960, p));
|
||||
const a = assignTab(chordC, GUITAR_STANDARD);
|
||||
expect(a.dropped).toBe(0);
|
||||
for (const n of a.notes) expect(GUITAR_STANDARD[n.string!-1]! + n.fret!).toBe(n.pitch);
|
||||
expect(new Set(a.notes.map(n => n.string)).size).toBe(5);
|
||||
expect(assignTab([sn(0, 960, 30)], GUITAR_STANDARD)).toEqual({ notes: [], dropped: 1 }); // below low E
|
||||
```
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: quantisation and tab assignment`.
|
||||
|
||||
### Task 12: Pipeline runner with oracle modes
|
||||
|
||||
**Files:**
|
||||
- Create: `src/pipeline/run.ts`
|
||||
- Test: `tests/pipeline/run.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: separator (T7), transcriber (T8), `estimateGrid`/`gridFromTruth` (T9), `transcribeDrums` (T10), `quantize`/`assignTab` (T11), `songToGp` (T2), `PipelineOutput`/`OracleFlags` (T6).
|
||||
- Produces: `createPipeline(models: { demucs: Uint8Array; basicPitch: Uint8Array }): Promise<(input: PipelineInput, flags: OracleFlags) => Promise<{ output: PipelineOutput; gp: Uint8Array }>>` where `PipelineInput = { mix: Audio; oracle?: { stems: Map<string, Audio>; truth: { bpm: number; timeSig: [number, number]; tracks: TruthTrack[] } } }`.
|
||||
|
||||
Behaviour: stems — non-oracle: Demucs; map guitar→one "Guitar" track (role guitar), bass→bass, piano→keys, vocals→vocals (monophonic, pitch 48–84), drums→drums; `other` is ignored (reported in notes.json as unused). Oracle stems: one output track per truth track, from its own oracle stem. Grid — `oracleTempo` ? `gridFromTruth` : `estimateGrid(mix)`. `oracleNotes`: pitched tracks take their truth notes instead of transcription (tab assignment still runs, measuring fingering only). Pitch ranges for Basic Pitch: guitar 40–88, bass 28–67, keys 21–108. Every pitched track is quantised; guitar/bass get `assignTab` with standard tunings; output `Song` (bpm from grid) → `songToGp`. Pipeline code does no file I/O.
|
||||
|
||||
- [ ] **Step 1: Write failing test** — on `data/t1a/1` with `{ oracleStems: true, oracleTempo: true, oracleNotes: false }`: output has 1 guitar track with > 0 notes; `truthFromGp(gp)` re-imports with a guitar track; with `oracleNotes: true` merged guitar F1 from `scoreSong` === 1 and `stringAgreement` is a number. On `data/t1c/1` with all flags false: output roles ⊇ {guitar, bass, keys, vocals, drums}.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: pipeline runner with oracle modes`.
|
||||
|
||||
### Task 13: Benchmark orchestration and reports
|
||||
|
||||
**Files:**
|
||||
- Create: `src/bench/report.ts`; Modify: `src/bench/cli.ts`
|
||||
- Test: `tests/bench/report.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `renderReport(run: BenchRun): string` (markdown), `BenchRun = { runId: string; date: string; modes: { name: string; flags: OracleFlags }[]; results: { tier: TierId; seed: number; mode: string; score: SongScore; seconds: number }[] }`.
|
||||
- CLI: `bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 --modes full,oracle-stems,oracle-tempo,oracle-notes --run-id <id>` builds missing songs, runs each mode, writes `runs/<id>/<tier>/<seed>/{notes.json,out.gp,score.json}` and `results/<id>.{md,json}`. Modes: `full` = all false; `oracle-stems` = stems only; `oracle-tempo` = stems + tempo; `oracle-notes` = all true.
|
||||
|
||||
Report layout: header (run id, date, machine, model hashes, SoundFont, vocals caveat); per tier a table of median/min/max per mode in this column order: guitar merged F1 (pass ✓/✗), guitar octave errors, spurious rate, part-split F1 (t1d), vocals F1, bass F1, drums kick/snare/hihat F1, keys F1, rhythm onBeat/durationOk with tempo factor/offset, string agreement; tier pass line (median song passes, `full` mode); per-song appendix. `null` renders as `n/a`.
|
||||
|
||||
- [ ] **Step 1: Write failing test** — `renderReport` on a fixture `BenchRun` (2 songs, 2 modes, one with vocals `null`): contains "Guitar" column before "Vocals" before "Bass"; contains "n/a"; tier pass line reads "PASS" when median guitar F1 ≥ 0.85 and "FAIL" otherwise.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS.
|
||||
- [ ] **Step 5:** Commit `feat: benchmark runner and report`.
|
||||
|
||||
### Task 14: Baseline benchmark run
|
||||
|
||||
- [ ] **Step 1:** `bun run fetch-models && bun run typecheck && bun test --timeout 300000` → all pass.
|
||||
- [ ] **Step 2:** `bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 --modes full,oracle-stems,oracle-tempo,oracle-notes --run-id baseline-2026-10-30` → `results/baseline-2026-10-30.md` exists with 4 tier sections.
|
||||
- [ ] **Step 3:** Listen-check is not possible; instead spot-check one `out.gp` per tier re-imports via `truthFromGp` and one `mix.mp3` is non-silent (ffprobe duration > 30 s).
|
||||
- [ ] **Step 4:** Add a short `README.md` (what the harness is, commands, where results live). Commit `chore: baseline benchmark results` and push to `origin main`.
|
||||
@@ -1,7 +1,8 @@
|
||||
# Synthetic Benchmark Harness — Design
|
||||
|
||||
Date: 2026-10-30
|
||||
Status: awaiting user approval of this document
|
||||
Status: approved by user 2026-10-30 for the initial stage. Later stages will also
|
||||
run tests on real-world songs and pick up items currently listed as non-goals.
|
||||
|
||||
## Intent
|
||||
|
||||
|
||||
Reference in New Issue
Block a user