# STSheets synthetic benchmark

A harness that measures guitar-tab transcription quality without any real recordings. It generates original songs in four tiers (t1a to t1d, from solo guitar up to a full band with a second guitar part), writes them as Guitar Pro files, renders audio with a bundled SoundFont, then runs the pipeline (Demucs source separation, Basic Pitch transcription, tempo and beat detection, tab fingering) and scores the result against the known truth. A song passes when merged-guitar pitch F1 is at least 0.85. Oracle modes feed ground truth to later stages so errors can be attributed to a single stage.

## Prerequisites

- [Bun](https://bun.sh)
- ffmpeg (and ffprobe) on `PATH`

## Commands

```sh
bun install
bun run fetch-models                  # downloads the ONNX models into models/
bun test                              # one test is skipped unless the Demucs drum measurement is enabled by env
bun run bench gen --tiers t1a,t1b,t1c,t1d --seeds 1-5
bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 \
  --modes full,oracle-stems,oracle-tempo,oracle-notes --run-id baseline-2026-10-30
```

## On-disk layout

- `data/<tier>/<seed>/` holds `song.json`, `truth.gp`, `mix.wav`, `mix.mp3` and `stems/`. Git-ignored.
- `runs/<run-id>/<tier>/<seed>/<mode>/` holds `notes.json`, `out.gp` and `score.json`. Git-ignored.
- `results/<run-id>.md` and `.json` are the committed reports.

## Results

The baseline is in `results/baseline-2026-10-30.md` (tier summaries plus a per-song appendix) with the raw numbers in the matching `.json`.

## Known caveats

- Vocals are rendered with a GM "Voice Oohs" patch, not real singing, so vocal numbers are indicative only.
- Drums use a heuristic baseline transcriber.
- `oracle-stems` mode feeds the true per-track stems, which reveals the track count and roles, so part-split F1 under oracle-stems (and the oracle modes built on it) is not meaningful.
- The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
- Fingering (string agreement) is informational, not pass/fail.