Files

T

Lucas WintherandClaude Opus 5.5 3db87c0317 fix(align): calibrated fit (precision, lift, beat-shift check, baseline) and aligner guards (A7, A8)

Fit flags on precision (<0.6) and chance-corrected lift (<0.25), null with <8
recording onsets; old match rate kept as audibleRate (info). Symbolic-chroma
beat/bar-shift check per window. Constant-offset baseline (-2..+2 s) replaces
DTW without manual anchors unless DTW precision is better by >=0.02. Stretch
warning counts in the fit summary. DTW: slope-limited steps, open begin/end
options; per-bar outlier rejection (1/4 beat, Theil-Sen-transported +-3 bar
median); no anchor in the final partial bar. bench real: saved/computed
source, baseline + method, corrupt alignment.json skipped. Parabolic onset
interpolation; warp extrapolation documented; spec amended.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-10-30 19:50:48 +01:00

4.6 KiB

Raw Permalink Blame History

STSheets synthetic benchmark

A harness that measures guitar-tab transcription quality without any real recordings. It generates original songs in four tiers (t1a to t1d, from solo guitar up to a full band with a second guitar part), writes them as Guitar Pro files, renders audio with a bundled SoundFont, then runs the pipeline (Demucs source separation, Basic Pitch transcription, tempo and beat detection, tab fingering) and scores the result against the known truth. A song passes when merged-guitar pitch F1 is at least 0.85. Oracle modes feed ground truth to later stages so errors can be attributed to a single stage.

Prerequisites

Bun
ffmpeg (and ffprobe) on PATH

Commands

bun install
bun run fetch-models                  # downloads the ONNX models into models/
bun test                              # one test is skipped unless the Demucs drum measurement is enabled by env
bun run bench gen --tiers t1a,t1b,t1c,t1d --seeds 1-5
bun run bench run --tiers t1a,t1b,t1c,t1d --seeds 1-5 \
  --modes full,oracle-stems,oracle-tempo,oracle-notes --run-id baseline-2026-10-30

On-disk layout

data/<tier>/<seed>/ holds song.json, truth.gp, mix.wav, mix.mp3 and stems/. Git-ignored.
runs/<run-id>/<tier>/<seed>/<mode>/ holds notes.json, out.gp and score.json. Git-ignored.
results/<run-id>.md and .json are the committed reports.

Results

The baseline is in results/baseline-2026-10-30.md (tier summaries plus a per-song appendix) with the raw numbers in the matching .json.

Known caveats

Vocals are rendered with a GM "Voice Oohs" patch, not real singing, so vocal numbers are indicative only.
Drums use a heuristic baseline transcriber.
oracle-stems mode feeds the true per-track stems, which reveals the track count and roles, so part-split F1 under oracle-stems (and the oracle modes built on it) is not meaningful.
The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
Fingering (string agreement) is informational, not pass/fail.

Aligning reference songs

Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored reference/ directory (never commit them; *.flac, *.mid and runs/ are ignored too). Each song gets reference/<id>/ with its audio, MIDI and alignment.json, and reference/songs.json lists the registered songs.

bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]

The --capo track name must match the MIDI track name exactly, including any trailing space (e.g. "Acoustic Guitar =9").

Compute and save an automatic alignment (all anchors auto; manual anchors from an earlier save are kept) and print its fit summary:

bun run align auto <id>

Launch the alignment page. A bare bun run align starts a long-running server; the key in the printed URL is required and changes on every start. For LAN use:

bun run align --host 0.0.0.0 --url-host 10.20.20.2

Page workflow:

Pick a song and add anchors where the MIDI and the recording visibly disagree (click the waveform/roll to pin a MIDI time to an audio time).
Play and mix: toggle the recording and the rendered MIDI, and per-role levels, to listen for sync.
Run auto-align around anchors; manual anchors are kept fixed and the rest is recomputed.
Save to write alignment.json.

Benchmark against the real songs (uses each saved alignment.json; --auto computes one on the fly for songs without it):

bun run bench real [--auto] --run-id <id>

This writes results/real-<id>.md and .json (committed) and per-song output under runs/real-<id>/ (ignored).

Caveats:

The fit summary is a consistency measure between the MIDI and the audio, not ground truth: precision (share of the recording's detected attacks with a warped MIDI onset within ±70 ms; a 30 s window is flagged below 60 %), lift (precision minus its off-grid chance level; flagged below 0.25), a chroma check that flags suspected whole-beat/bar shifts, "MIDI onsets audible as attacks" (information only: a full mix exposes far fewer attacks than the score has notes), and the number of warned stretches (> 5 % and > 15 % tempo change). A high precision does not prove the alignment is right.
align auto also reports a constant-offset baseline (best single offset, −2..+2 s); with no manual anchors it keeps the baseline (one auto anchor) unless the DTW alignment beats its precision by 2 points.
The page plays Ogg Opus; Safari needs 18.4 or later.

4.6 KiB Raw Permalink Blame History Unescape Escape