STSheets synthetic benchmark

A harness that measures guitar-tab transcription quality without any real recordings. It generates original songs in four tiers (t1a to t1d, from solo guitar up to a full band with a second guitar part), writes them as Guitar Pro files, renders audio with a bundled SoundFont, then runs the pipeline (Demucs source separation, Basic Pitch transcription, tempo and beat detection, tab fingering) and scores the result against the known truth. A song passes when merged-guitar pitch F1 is at least 0.85. Oracle modes feed ground truth to later stages so errors can be attributed to a single stage.

Prerequisites

• Bun
• ffmpeg (and ffprobe) on PATH

Commands

bun install
bun run fetch-models                  # downloads the ONNX models into models/
bun test                              # one test is skipped unless the Demucs drum measurement is enabled by env
bun run bench gen --tiers t1a,t1b,t1c,t1d --seeds 1-5
bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 \
  --modes full,oracle-stems,oracle-tempo,oracle-notes --run-id baseline-2026-10-30

On-disk layout

• data/<tier>/<seed>/ holds song.json, truth.gp, mix.wav, mix.mp3 and stems/. Git-ignored.
• runs/<run-id>/<tier>/<seed>/<mode>/ holds notes.json, out.gp and score.json. Git-ignored.
• results/<run-id>.md and .json are the committed reports.

Results

The baseline is in results/baseline-2026-10-30.md (tier summaries plus a per-song appendix) with the raw numbers in the matching .json.

Known caveats

• Vocals are rendered with a GM "Voice Oohs" patch, not real singing, so vocal numbers are indicative only.
• Drums use a heuristic baseline transcriber.
• oracle-stems mode feeds the true per-track stems, which reveals the track count and roles, so part-split F1 under oracle-stems (and the oracle modes built on it) is not meaningful.
• The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
• Fingering (string agreement) is informational, not pass/fail.

Aligning reference songs

Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored reference/ directory (never commit them; *.flac, *.mid and runs/ are ignored too). Each song gets reference/<id>/ with its audio, MIDI and alignment.json, and reference/songs.json lists the registered songs.

Register a song (guesses a scoring role per MIDI track; unmapped tracks are null):

bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]

The --capo track name must match the MIDI track name exactly, including any trailing space (e.g. "Acoustic Guitar =9").

Compute and save an automatic alignment (all anchors auto; manual anchors from an earlier save are kept) and print its fit summary:

bun run align auto <id>

Launch the alignment page. A bare bun run align starts a long-running server; the key in the printed URL is required and changes on every start. For LAN use:

bun run align --host 0.0.0.0 --url-host 10.20.20.2

Page workflow:

1. Pick a song and add anchors where the MIDI and the recording visibly disagree (click the waveform/roll to pin a MIDI time to an audio time).
2. Play and mix: toggle the recording and the rendered MIDI, and per-role levels, to listen for sync.
3. Run auto-align around anchors; manual anchors are kept fixed and the rest is recomputed.
4. Save to write alignment.json.

Benchmark against the real songs (uses each saved alignment.json; --auto computes one on the fly for songs without it):

bun run bench real [--auto] --run-id <id>

This writes results/real-<id>.md and .json (committed) and per-song output under runs/real-<id>/ (ignored).

Caveats:

• The fit summary is a consistency measure between the MIDI and the audio, not ground truth: precision (share of the recording's detected attacks with a warped MIDI onset within ±70 ms; a 30 s window is flagged below 60 %), lift (precision minus its off-grid chance level; flagged below 0.25), a chroma check that flags suspected whole-beat/bar shifts, "MIDI onsets audible as attacks" (information only: a full mix exposes far fewer attacks than the score has notes), and the number of warned stretches (> 5 % and > 15 % tempo change). A high precision does not prove the alignment is right.
• align auto also reports a constant-offset baseline (best single offset, −2..+2 s); with no manual anchors it keeps the baseline (one auto anchor) unless the DTW alignment beats its precision by 2 points.
• The page plays Ogg Opus; Safari needs 18.4 or later.