Add research reports and synthetic benchmark design

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:28:03 +01:00
commit f2ca686b87
13 changed files with 2929 additions and 0 deletions
@@ -0,0 +1,184 @@
+# Synthetic Benchmark Harness — Design
+
+Date: 2026-10-30
+Status: awaiting user approval of this document
+
+## Intent
+
+STSheets will turn songs into sheet music and Guitar Pro tabs. Before we can improve
+transcription accuracy we need a way to measure it with a perfectly known answer.
+The harness generates original songs as Guitar Pro files, renders them to audio,
+runs our transcription pipeline on that audio, and scores the output against the
+file it came from. Because we write the source file, ground truth is exact: no
+alignment problems, no disputes about whether a human tab matches its recording.
+
+It is also how we work without user data: the product keeps no user data, so
+the improvement loop runs on data we generate ourselves.
+
+Background research: `reports/Music transcription tooling survey.md` and
+`reports/Transcription accuracy improvement ideas.md`.
+
+## Goals
+
+1. **Difficulty ladder.** Generated songs are organised into tiers. Each step adds
+   one kind of difficulty so a drop in score can be attributed.
+   - **Tier 1 (build now):** simple, slow, floaty pop. Style references:
+     *Swimming* (Lunar Vacation), *Last Night on Earth* (Green Day),
+     *Afire Love* (Ed Sheeran). Standard tuning, 4/4, slow-to-moderate tempo,
+     clean or lightly driven guitars with reverb/chorus, open chords, arpeggios,
+     simple melodic lines; bass on roots; simple drums; keys/pads; a vocal melody.
+   - **Later tiers (design for, don't build):** simpler rock/metal in the vein of
+     Skillet and Anarbor (distortion, power chords, palm muting, faster tempos,
+     possibly lower tunings), then up to *Moon Walker*, *Patent Pending*,
+     *Burn the Ballroom* (faster, denser, more technique).
+   - Benchmark tiers use **at most two guitars**. More guitars are allowed in the
+     product later but are not part of this ladder.
+2. **Original material only.** Songs are generated in the *style* of the
+   references, never recreations of them.
+3. **Instrument priority** (drives what we tune first and how reports are ordered):
+   1. Guitar
+   2. Vocals
+   3. Bass, drums, keys (equal)
+   All five are present and scored from the start; only the focus differs.
+4. **Scoring priority** (the user's words: wrong notes are "the hardest no"):
+   1. **Pitch** — correct notes at roughly the right time. Primary pass/fail.
+   2. **Rhythm** — after quantisation, notes on the right beat with the right
+      duration so the tab reads correctly. Secondary.
+   3. **Fingering (string/fret) and techniques (slides, bends, hammer-ons…)** —
+      measured and reported, but *not* part of pass/fail; users can fix these.
+5. **Repeatable.** Same seed → same song, same audio, same scores.
+6. **Browser-portable where we own the code.** The pipeline should be structured
+   so it can later run in the browser (WASM/WebGPU).
+
+## Non-goals
+
+- The website UI, upload flow, YouTube ingestion.
+- Building the metal tiers now (but nothing should prevent adding them).
+- Training or fine-tuning models (later phase; this harness is the prerequisite).
+- Matching the reference songs themselves.
+
+## Anti-goals (would be failure even if it "works")
+
+- Scores that look good because the pipeline was handed information it won't have
+  in real use (tempo, stems, track count) without that being explicitly labelled
+  as an oracle run.
+- Fingering/technique differences dragging down the headline score.
+- Anything that stores or depends on user data.
+- Shipping components whose licence blocks commercial use without flagging it.
+
+## Constraints
+
+- **Language/runtime:** Bun + TypeScript for everything we own: generator, Guitar
+  Pro writing, rendering/mixing, pipeline orchestration, scoring, reports.
+- **Models** run via ONNX Runtime (`onnxruntime-node` in the harness; the same
+  model files can later run under `onnxruntime-web`).
+- **Python** only as an optional *comparison* track for research models that
+  exist only in Python (e.g. Noise2Fret). Never on the product path.
+- **Hardware:** this machine, CPU only (16 cores, ~17 GB RAM, no GPU).
+- **Licences:** anything on the product path must allow commercial use (e.g.
+  Demucs MIT, Basic Pitch Apache-2.0, alphaTab MPL-2.0). Research-only things
+  may be used in the comparison track if clearly labelled. SoundFonts used for
+  rendering must have a licence that permits this use.
+- **Repo:** `/home/user/STSheets`, git, remote
+  `https://gitea.lucaswinther.info/lucasw89/STSheets`. Generated audio is
+  not committed (regenerable from seeds); source, specs, and result summaries are.
+
+## How it works (decided parts)
+
+```
+seed + tier spec
+   → song generator (TS)            → ground-truth score model
+   → Guitar Pro file (alphaTab GP7 exporter)
+   → render each track (alphaTab synth + SoundFont) → per-track stems
+   → effects + pan + mix (our TS DSP) → mix.wav / mix.mp3
+   → pipeline under test (decode → separate → transcribe per stem → tempo/beat
+     → quantise → tab assignment → GP file)
+   → scorer (output vs ground truth) → per-song JSON → tier report (markdown)
+```
+
+### Generator
+- Produces an original arrangement from a seed and a tier spec: tempo, key,
+  chord progression, song sections (with repeats), and per-instrument parts.
+- Guitar parts in tier 1: strummed open chords, fingerpicked/arpeggiated chords,
+  simple single-note melodic lines. Two-guitar songs give the guitars distinct
+  roles (e.g. strum + arpeggio) and pan them apart.
+- Writes the song as a valid Guitar Pro file (alphaTab model → GP7 export) with
+  tracks, tunings, string/fret, and technique marks. This file is the ground truth.
+- Tier 1 should itself ramp: start with a single clean guitar, then add the band,
+  then the second guitar. Exact sub-steps are the builder's call.
+
+### Renderer
+- Renders each track separately (so isolated stems exist for oracle runs), then
+  applies light effects (drive, chorus, reverb), panning, and mixes.
+- Vocals have no real singing synthesis available; the vocal melody is rendered
+  with a voice-like SoundFont patch. **Known limitation:** separation models are
+  trained on real voices and may not treat this as "vocals". Report vocal results
+  with that caveat.
+
+### Pipeline under test (baseline)
+- Separation: Demucs (ONNX). Transcription: Basic Pitch (ONNX/TS) for pitched
+  stems. Drums: a licence-clean option if one is runnable, otherwise report drums
+  as "not yet supported" rather than faking it. Tempo/beat: must be estimated by
+  the pipeline itself. Tab assignment: our own playability-based assigner.
+  Output: Guitar Pro file via alphaTab.
+- Pipeline stages are swappable so later experiments (from the ideas report) can
+  replace one stage at a time.
+
+### Oracle modes (for error attribution)
+The harness can feed the pipeline ground-truth information to find which stage
+causes errors. Every result is labelled with which oracles were on:
+- **Oracle stems:** isolated rendered stems instead of separated ones.
+- **Oracle tempo/beats:** true tempo map instead of estimated.
+- **Oracle notes:** ground-truth notes into the tab assigner (tests fingering only).
+
+### Scorer
+- **Track matching:** output tracks are matched to ground-truth tracks by
+  instrument. The pipeline is not told how many guitars there are.
+  *(User's decision)* For pass/fail, all guitar ground-truth notes are pooled and
+  compared with all guitar output notes (merged guitar score). Whether the
+  pipeline split the guitars into the right separate parts is a separate
+  "part-split" metric: when it outputs several guitar tracks they are matched
+  one-to-one by best note agreement (Hungarian), and unmatched tracks count
+  against this metric only. Part-split becomes a pass criterion in a later tier,
+  not in tier 1.
+- *(Coordinator's call)* Separation uses a model with a dedicated guitar stem
+  (e.g. 6-stem Demucs) so keys are not pooled into "guitar"; keys bleeding into
+  the guitar stem is a known risk that the spurious-note rate will expose.
+- **Pitch (primary):** note-level precision/recall/F1 on onset + pitch (onset
+  tolerance ±50 ms, offsets ignored), per track. String choice is ignored here:
+  same pitch on a different string counts as correct. Also report octave-error
+  rate and spurious-note rate separately.
+  *(Coordinator's call)* The primary pitch score is computed on the pipeline's
+  timed notes in seconds, **before** quantisation, so a tempo or grid error
+  cannot fail a song on pitch. Every pipeline run therefore emits a timed-notes
+  file alongside the Guitar Pro output. Pitch F1 of the final GP file (mapped
+  back to seconds via its own tempo map) is reported as a secondary number.
+- **Rhythm (secondary):** on the quantised output, the share of correctly-pitched
+  notes that sit on the correct beat position and have the correct notated
+  duration; bar count / time signature agreement.
+  *(Coordinator's call)* Before scoring rhythm, the scorer aligns the output grid
+  to the true grid, allowing a tempo factor of ½×, 1× or 2× and a whole-beat bar
+  offset (incl. pickup bars). The rhythm score is computed after that alignment;
+  the detected tempo factor and bar offset are reported separately as their own
+  "grid errors".
+- **Informational:** string/fret agreement on matched notes, technique
+  precision/recall, drum per-class F1 (GM drum map), vocal note F1.
+- **Reports:** per-song JSON plus a tier-level markdown summary, guitar first,
+  then vocals, then bass/drums/keys.
+
+### Pass criteria (builder's/coordinator's call, adjustable)
+Initial targets: a tier-1 song "passes" when the guitar pitch F1 ≥ 0.85; a tier
+passes when its median song passes. These numbers are a starting point to be
+revised once baseline results exist.
+
+## Left to the builder
+
+- Repo layout, module boundaries, CLI shape.
+- Which SoundFont(s) (subject to licence) and exact effect implementations.
+- Generator internals (progression/rhythm/melody rules), as long as output fits
+  the tier descriptions.
+- Exact tier-1 sub-steps and how many songs per tier.
+- How the Demucs ONNX model is obtained or converted, and fallback choices if
+  alphaTab rendering/export doesn't work under Bun (report any substitution).
+- Quantisation method and tab-assignment cost model.
+- Test strategy for the harness itself.