STSheets/docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md

Synthetic Benchmark Harness — Design

Date: 2026-10-30 Status: approved by user 2026-10-30 for the initial stage. Later stages will also run tests on real-world songs and pick up items currently listed as non-goals.

Intent

STSheets will turn songs into sheet music and Guitar Pro tabs. Before we can improve transcription accuracy we need a way to measure it with a perfectly known answer. The harness generates original songs as Guitar Pro files, renders them to audio, runs our transcription pipeline on that audio, and scores the output against the file it came from. Because we write the source file, ground truth is exact: no alignment problems, no disputes about whether a human tab matches its recording.

It is also how we work without user data: the product keeps no user data, so the improvement loop runs on data we generate ourselves.

Background research: reports/Music transcription tooling survey.md and reports/Transcription accuracy improvement ideas.md.

Goals

1. Difficulty ladder. Generated songs are organised into tiers. Each step adds one kind of difficulty so a drop in score can be attributed.
   • Tier 1 (build now): simple, slow, floaty pop. Style references: Swimming (Lunar Vacation), Last Night on Earth (Green Day), Afire Love (Ed Sheeran). Standard tuning, 4/4, slow-to-moderate tempo, clean or lightly driven guitars with reverb/chorus, open chords, arpeggios, simple melodic lines; bass on roots; simple drums; keys/pads; a vocal melody.
   • Later tiers (design for, don't build): simpler rock/metal in the vein of Skillet and Anarbor (distortion, power chords, palm muting, faster tempos, possibly lower tunings), then up to Moon Walker, Patent Pending, Burn the Ballroom (faster, denser, more technique).
   • Benchmark tiers use at most two guitars. More guitars are allowed in the product later but are not part of this ladder.
2. Original material only. Songs are generated in the style of the references, never recreations of them.
3. Instrument priority (drives what we tune first and how reports are ordered):
   1. Guitar
   2. Vocals
   3. Bass, drums, keys (equal) All five are present and scored from the start; only the focus differs.
4. Scoring priority (the user's words: wrong notes are "the hardest no"):
   1. Pitch — correct notes at roughly the right time. Primary pass/fail.
   2. Rhythm — after quantisation, notes on the right beat with the right duration so the tab reads correctly. Secondary.
   3. Fingering (string/fret) and techniques (slides, bends, hammer-ons…) — measured and reported, but not part of pass/fail; users can fix these.
5. Repeatable. Same seed → same song, same audio, same scores.
6. Browser-portable where we own the code. The pipeline should be structured so it can later run in the browser (WASM/WebGPU).

Non-goals

• The website UI, upload flow, YouTube ingestion.
• Building the metal tiers now (but nothing should prevent adding them).
• Training or fine-tuning models (later phase; this harness is the prerequisite).
• Matching the reference songs themselves.

Anti-goals (would be failure even if it "works")

• Scores that look good because the pipeline was handed information it won't have in real use (tempo, stems, track count) without that being explicitly labelled as an oracle run.
• Fingering/technique differences dragging down the headline score.
• Anything that stores or depends on user data.
• Shipping components whose licence blocks commercial use without flagging it.

Constraints

• Language/runtime: Bun + TypeScript for everything we own: generator, Guitar Pro writing, rendering/mixing, pipeline orchestration, scoring, reports.
• Models run via ONNX Runtime (onnxruntime-node in the harness; the same model files can later run under onnxruntime-web).
• Python only as an optional comparison track for research models that exist only in Python (e.g. Noise2Fret). Never on the product path.
• Hardware: this machine, CPU only (16 cores, ~17 GB RAM, no GPU).
• Licences: anything on the product path must allow commercial use (e.g. Demucs MIT, Basic Pitch Apache-2.0, alphaTab MPL-2.0). Research-only things may be used in the comparison track if clearly labelled. SoundFonts used for rendering must have a licence that permits this use.
• Repo: /home/user/STSheets, git, remote https://gitea.lucaswinther.info/lucasw89/STSheets. Generated audio is not committed (regenerable from seeds); source, specs, and result summaries are.

How it works (decided parts)

seed + tier spec
   → song generator (TS)            → ground-truth score model
   → Guitar Pro file (alphaTab GP7 exporter)
   → render each track (alphaTab synth + SoundFont) → per-track stems
   → effects + pan + mix (our TS DSP) → mix.wav / mix.mp3
   → pipeline under test (decode → separate → transcribe per stem → tempo/beat
     → quantise → tab assignment → GP file)
   → scorer (output vs ground truth) → per-song JSON → tier report (markdown)

Generator

• Produces an original arrangement from a seed and a tier spec: tempo, key, chord progression, song sections (with repeats), and per-instrument parts.
• Guitar parts in tier 1: strummed open chords, fingerpicked/arpeggiated chords, simple single-note melodic lines. Two-guitar songs give the guitars distinct roles (e.g. strum + arpeggio) and pan them apart.
• Writes the song as a valid Guitar Pro file (alphaTab model → GP7 export) with tracks, tunings, string/fret, and technique marks. This file is the ground truth.
• Tier 1 should itself ramp: start with a single clean guitar, then add the band, then the second guitar. Exact sub-steps are the builder's call.

Renderer

• Renders each track separately (so isolated stems exist for oracle runs), then applies light effects (drive, chorus, reverb), panning, and mixes.
• Vocals have no real singing synthesis available; the vocal melody is rendered with a voice-like SoundFont patch. Known limitation: separation models are trained on real voices and may not treat this as "vocals". Report vocal results with that caveat.

Pipeline under test (baseline)

• Separation: Demucs (ONNX). Transcription: Basic Pitch (ONNX/TS) for pitched stems. Drums: a licence-clean option if one is runnable, otherwise report drums as "not yet supported" rather than faking it. Tempo/beat: must be estimated by the pipeline itself. Tab assignment: our own playability-based assigner. Output: Guitar Pro file via alphaTab.
• Pipeline stages are swappable so later experiments (from the ideas report) can replace one stage at a time.

Oracle modes (for error attribution)

The harness can feed the pipeline ground-truth information to find which stage causes errors. Every result is labelled with which oracles were on:

• Oracle stems: isolated rendered stems instead of separated ones.
• Oracle tempo/beats: true tempo map instead of estimated.
• Oracle notes: ground-truth notes into the tab assigner (tests fingering only).

Scorer

• Track matching: output tracks are matched to ground-truth tracks by instrument. The pipeline is not told how many guitars there are. (User's decision) For pass/fail, all guitar ground-truth notes are pooled and compared with all guitar output notes (merged guitar score). Whether the pipeline split the guitars into the right separate parts is a separate "part-split" metric: when it outputs several guitar tracks they are matched one-to-one by best note agreement (Hungarian), and unmatched tracks count against this metric only. Part-split becomes a pass criterion in a later tier, not in tier 1.
• (Coordinator's call) Separation uses a model with a dedicated guitar stem (e.g. 6-stem Demucs) so keys are not pooled into "guitar"; keys bleeding into the guitar stem is a known risk that the spurious-note rate will expose.
• Pitch (primary): note-level precision/recall/F1 on onset + pitch (onset tolerance ±50 ms, offsets ignored), per track. String choice is ignored here: same pitch on a different string counts as correct. Also report octave-error rate and spurious-note rate separately. (Coordinator's call) The primary pitch score is computed on the pipeline's timed notes in seconds, before quantisation, so a tempo or grid error cannot fail a song on pitch. Every pipeline run therefore emits a timed-notes file alongside the Guitar Pro output. Pitch F1 of the final GP file (mapped back to seconds via its own tempo map) is reported as a secondary number.
• Rhythm (secondary): on the quantised output, the share of correctly-pitched notes that sit on the correct beat position and have the correct notated duration; bar count / time signature agreement. (Coordinator's call) Before scoring rhythm, the scorer aligns the output grid to the true grid, allowing a tempo factor of ½×, 1× or 2× and a whole-beat bar offset (incl. pickup bars). The rhythm score is computed after that alignment; the detected tempo factor and bar offset are reported separately as their own "grid errors".
• Informational: string/fret agreement on matched notes, technique precision/recall, drum per-class F1 (GM drum map), vocal note F1.
• Reports: per-song JSON plus a tier-level markdown summary, guitar first, then vocals, then bass/drums/keys.

Pass criteria (builder's/coordinator's call, adjustable)

Initial targets: a tier-1 song "passes" when the guitar pitch F1 ≥ 0.85; a tier passes when its median song passes. These numbers are a starting point to be revised once baseline results exist.

Left to the builder

• Repo layout, module boundaries, CLI shape.
• Which SoundFont(s) (subject to licence) and exact effect implementations.
• Generator internals (progression/rhythm/melody rules), as long as output fits the tier descriptions.
• Exact tier-1 sub-steps and how many songs per tier.
• How the Demucs ONNX model is obtained or converted, and fallback choices if alphaTab rendering/export doesn't work under Bun (report any substitution).
• Quantisation method and tab-assignment cost model.
• Test strategy for the harness itself.