From 19ac52c93a1e6754c936c704df5b433fb121c8a4 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 21:00:45 +0100 Subject: [PATCH] docs: accuracy experiment log E1-E4 Co-Authored-By: Claude Opus 5.5 --- docs/experiments/2026-10-31-accuracy.md | 30 +++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/experiments/2026-10-31-accuracy.md diff --git a/docs/experiments/2026-10-31-accuracy.md b/docs/experiments/2026-10-31-accuracy.md new file mode 100644 index 0000000..9029023 --- /dev/null +++ b/docs/experiments/2026-10-31-accuracy.md @@ -0,0 +1,30 @@ +# Accuracy experiments — 2026-10-31 + +Branch `exp/accuracy`. Every experiment is judged on **both** the synthetic tiers +(median merged-guitar pitch F1, ±50 ms) and the real reference song +*Last Night on Earth* (`lnoe`, the only reference with a trustworthy alignment: +constant +0.03 s, precision 93%). Swimming/Holes alignments are still flagged and +are not used for decisions. + +Baseline before these experiments: synthetic full-mode guitar F1 +t1a 0.43 · t1b 0.51 · t1c 0.39 · t1d 0.60; lnoe guitar F1 0.390. + +| # | Experiment | Synthetic (median guitar F1) | lnoe guitar F1 | Verdict | +|---|---|---|---|---| +| E1 | Transcribe guitar from Demucs guitar + piano + other (instead of guitar stem only) | t1a 0.87 · t1b 0.81 · t1c 0.73 · t1d 0.72 | 0.403 | **Adopted** (commit 0f14fbc). Big synthetic gain is mostly a SONiVOX-timbre artefact (Demucs routes the synth guitar to piano/other); real gain is small but positive. t1a now passes. | +| E2 | Basic Pitch thresholds grid (onset 0.3–0.6, frame 0.2–0.4, min note 58/100 ms) | best ≈ default (0.5/0.3/58) | best 0.411 vs 0.403 | Not adopted — defaults are at the optimum on both. | +| E3 | Error taxonomy | — | fp: 46% no reference note near, 16% octave, 21% 50–150 ms timing; fn: 47% different pitch at the onset (partial chords), 25% octave, 24% missed | Octave fixing upper bound only 0.40 → 0.46 (pitch-class F1) with no systematic direction. | +| E4 | Complete partial strummed chords to standard guitar voicings (open + E/A-shape barres; maj/min/5/7/m7/sus/maj7) | −0.03…+0.03 | +0.016 (min 3 notes) | Not adopted yet — small, single-song evidence. | + +## Learnings + +- Synthetic gains do not transfer 1:1 to real recordings while the synthetic guitar + is the SONiVOX GM patch; a more realistic guitar sound in the renderer would make + the synthetic benchmark more predictive. +- On real audio the limits are Basic Pitch's partial-chord recall on strummed + rhythm guitar and octave confusions, not thresholds. +- An undocumented MT3 ONNX export exists (HF `ModernMube/HTDemucs_onnx`, ~300 MB); + not pursued — input/token format would need reverse-engineering and MT3 is weak + on real recordings. + +Experiment scripts live in the git-ignored `.scratch/exp/`.