Lucas WintherandClaude Opus 5.5
3db87c0317
Fit flags on precision (<0.6) and chance-corrected lift (<0.25), null with <8 recording onsets; old match rate kept as audibleRate (info). Symbolic-chroma beat/bar-shift check per window. Constant-offset baseline (-2..+2 s) replaces DTW without manual anchors unless DTW precision is better by >=0.02. Stretch warning counts in the fit summary. DTW: slope-limited steps, open begin/end options; per-bar outlier rejection (1/4 beat, Theil-Sen-transported +-3 bar median); no anchor in the final partial bar. bench real: saved/computed source, baseline + method, corrupt alignment.json skipped. Parabolic onset interpolation; warp extrapolation documented; spec amended. Co-Authored-By: Claude Opus 5.5 <[email protected]>
83 lines
4.6 KiB
Markdown
83 lines
4.6 KiB
Markdown
# STSheets synthetic benchmark
|
||
|
||
A harness that measures guitar-tab transcription quality without any real recordings. It generates original songs in four tiers (t1a to t1d, from solo guitar up to a full band with a second guitar part), writes them as Guitar Pro files, renders audio with a bundled SoundFont, then runs the pipeline (Demucs source separation, Basic Pitch transcription, tempo and beat detection, tab fingering) and scores the result against the known truth. A song passes when merged-guitar pitch F1 is at least 0.85. Oracle modes feed ground truth to later stages so errors can be attributed to a single stage.
|
||
|
||
## Prerequisites
|
||
|
||
- [Bun](https://bun.sh)
|
||
- ffmpeg (and ffprobe) on `PATH`
|
||
|
||
## Commands
|
||
|
||
```sh
|
||
bun install
|
||
bun run fetch-models # downloads the ONNX models into models/
|
||
bun test # one test is skipped unless the Demucs drum measurement is enabled by env
|
||
bun run bench gen --tiers t1a,t1b,t1c,t1d --seeds 1-5
|
||
bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 \
|
||
--modes full,oracle-stems,oracle-tempo,oracle-notes --run-id baseline-2026-10-30
|
||
```
|
||
|
||
## On-disk layout
|
||
|
||
- `data/<tier>/<seed>/` holds `song.json`, `truth.gp`, `mix.wav`, `mix.mp3` and `stems/`. Git-ignored.
|
||
- `runs/<run-id>/<tier>/<seed>/<mode>/` holds `notes.json`, `out.gp` and `score.json`. Git-ignored.
|
||
- `results/<run-id>.md` and `.json` are the committed reports.
|
||
|
||
## Results
|
||
|
||
The baseline is in `results/baseline-2026-10-30.md` (tier summaries plus a per-song appendix) with the raw numbers in the matching `.json`.
|
||
|
||
## Known caveats
|
||
|
||
- Vocals are rendered with a GM "Voice Oohs" patch, not real singing, so vocal numbers are indicative only.
|
||
- Drums use a heuristic baseline transcriber.
|
||
- `oracle-stems` mode feeds the true per-track stems, which reveals the track count and roles, so part-split F1 under oracle-stems (and the oracle modes built on it) is not meaningful.
|
||
- The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
|
||
- Fingering (string agreement) is informational, not pass/fail.
|
||
|
||
## Aligning reference songs
|
||
|
||
Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored `reference/` directory (never commit them; `*.flac`, `*.mid` and `runs/` are ignored too). Each song gets `reference/<id>/` with its audio, MIDI and `alignment.json`, and `reference/songs.json` lists the registered songs.
|
||
|
||
Register a song (guesses a scoring role per MIDI track; unmapped tracks are `null`):
|
||
|
||
```sh
|
||
bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]
|
||
```
|
||
|
||
The `--capo` track name must match the MIDI track name exactly, including any trailing space (e.g. `"Acoustic Guitar =9"`).
|
||
|
||
Compute and save an automatic alignment (all anchors `auto`; manual anchors from an earlier save are kept) and print its fit summary:
|
||
|
||
```sh
|
||
bun run align auto <id>
|
||
```
|
||
|
||
Launch the alignment page. A bare `bun run align` starts a long-running server; the key in the printed URL is required and changes on every start. For LAN use:
|
||
|
||
```sh
|
||
bun run align --host 0.0.0.0 --url-host 10.20.20.2
|
||
```
|
||
|
||
Page workflow:
|
||
|
||
1. Pick a song and add anchors where the MIDI and the recording visibly disagree (click the waveform/roll to pin a MIDI time to an audio time).
|
||
2. Play and mix: toggle the recording and the rendered MIDI, and per-role levels, to listen for sync.
|
||
3. Run auto-align around anchors; manual anchors are kept fixed and the rest is recomputed.
|
||
4. Save to write `alignment.json`.
|
||
|
||
Benchmark against the real songs (uses each saved `alignment.json`; `--auto` computes one on the fly for songs without it):
|
||
|
||
```sh
|
||
bun run bench real [--auto] --run-id <id>
|
||
```
|
||
|
||
This writes `results/real-<id>.md` and `.json` (committed) and per-song output under `runs/real-<id>/` (ignored).
|
||
|
||
Caveats:
|
||
|
||
- The fit summary is a consistency measure between the MIDI and the audio, not ground truth: *precision* (share of the recording's detected attacks with a warped MIDI onset within ±70 ms; a 30 s window is flagged below 60 %), *lift* (precision minus its off-grid chance level; flagged below 0.25), a chroma check that flags suspected whole-beat/bar shifts, "MIDI onsets audible as attacks" (information only: a full mix exposes far fewer attacks than the score has notes), and the number of warned stretches (> 5 % and > 15 % tempo change). A high precision does not prove the alignment is right.
|
||
- `align auto` also reports a constant-offset baseline (best single offset, −2..+2 s); with no manual anchors it keeps the baseline (one auto anchor) unless the DTW alignment beats its precision by 2 points.
|
||
- The page plays Ogg Opus; Safari needs 18.4 or later.
|