Add research reports and synthetic benchmark design

Co-Authored-By: Claude Opus 5.5 <[email protected]>

reports/Music transcription tooling survey.md

@@ -0,0 +1,207 @@
+# Separate stems, then transcribe each instrument
+
+As of late 2026, nobody has released an open-source system that turns a full song into good multi-instrument sheets and tabs. The components for building one do exist, and they work best as a chain: **separate the mix into stems, then send each stem to a specialist transcriber, then quantize the notes to a beat grid and render the result with alphaTab**. Quality depends heavily on the instrument. **Kick, snare, bass and lead vocals come out reliably. Piano and lead guitar are workable. String/fret tablature is still only about half right end to end**: the best 2026 research system, TART, scores **54% Tab F1** ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). Hi-hats, cymbals, backing vocals and lead-vs-rhythm guitar separation remain weak. The browser is now a realistic place to run the front of the pipeline. WebGPU ships in every major browser ([web.dev](https://web.dev/blog/webgpu-supported-major-browsers)), MIT-licensed HTDemucs ONNX ports separate stems at **about 8–22x real time on a mid-range GPU** ([HF monteslu](https://huggingface.co/monteslu/htdemucs-web-onnx)), and Basic Pitch (Apache-2.0) has an official browser build. The best models, however, are too large for the browser, have unclear or non-commercial licenses, or both. That includes the RoFormer separators, Kyutai's MuScriptor and the strongest drum models. The architecture that follows from this is browser-first, with a server-side quality tier. The key finding for the next phase: no commercial vendor publishes accuracy numbers ([audio2guitar](https://audio2guitar.com/blog/best-ai-guitar-tab-generators); [Songscription](https://www.songscription.ai/blog/best-ai-guitar-tab-generators)), so a reproducible benchmark against Guitar Pro references would be both an engineering tool and a differentiator.
+
+## RoFormer models lead separation, but only Demucs runs in browsers
+
+Source separation decides how well everything after it can do, and the field has moved past Demucs. **BS-RoFormer SW** is the best open-weight multi-stem model. It is a 6-stem BS-RoFormer variant (vocals, bass, drums, guitar, piano, other), modified by lucidrains with weights from viperx. MVSEP made it the default for new users and reports Multisong SDR of **vocals 11.30, bass 14.62, drums 14.11, guitar 9.05, piano 7.83, other 8.71** ([MVSEP news 64](https://mvsep.com/news/64?lang=en)). The original Demucs repo was **archived on 2025-01-01**. Its author keeps a slow fork at adefossez/demucs that takes bug fixes only ([facebookresearch/demucs](https://github.com/facebookresearch/demucs)). Demucs's 6-stem model is especially weak on the instruments this project cares about most. The README admits "the piano source is not working great" ([Demucs README](https://cdn.jsdelivr.net/gh/facebookresearch/demucs@main/README.md)), and MVSEP measures htdemucs_6s at **2.23 SDR for piano** against 7.83 for BS-RoFormer SW, with "other" dropping to 2.63 ([MVSEP algorithms](https://mvsep.com/en/algorithms)).
+
+| Model | Stems | Bass / drums / vocals / guitar / piano SDR (MVSEP) | Code / weights license | Browser-ready? |
+|---|---|---|---|---|
+| htdemucs_ft | 4 | 12.05 / 11.24 / 8.33 / – / – | MIT / MIT | Yes (WebGPU ONNX, 333 MB) |
+| htdemucs_6s | 6 | 11.42 / 10.59 / 8.17 / n/a / 2.23 | MIT / MIT | Yes (WebGPU ONNX, ~84 MB) |
+| SCNet XL | 4 | 13.81 / 13.42 / – / – / – | code open | No port found |
+| BS-RoFormer SW | 6 | 14.62 / 14.11 / 11.30 / 9.05 / 7.83 | community weights, **no license stated** | Unofficial ONNX export only, ~700 MB checkpoint |
+| MVSEP BS-RoFormer 124-band | 2 (vocals) | vocals 12.33 | service-only | No |
+
+Sources: [MVSEP algorithms](https://mvsep.com/en/algorithms); [demucs-rs](https://github.com/nikhilunni/demucs-rs); [HF monteslu](https://huggingface.co/monteslu/htdemucs-web-onnx). MVSEP numbers come from its own Multisong set and cannot be compared directly with MUSDB18 paper figures.
+
+Two gaps matter. First, **the cleanest stems are not the most legally clean**. Demucs is MIT end to end ([HF monteslu](https://huggingface.co/monteslu/htdemucs-web-onnx)). The RoFormer weights (viperx, KimberleyJensen, becruily, gabox, anvuew, jarredou) are community-trained, usually on undisclosed data, and were found with no license attached. The BS-RoFormer SW checkpoint has already moved hosts once, after the jarredou account disappeared ([HF jarredou commit](https://huggingface.co/jarredou/BS-ROFO-SW-Fixed/commit/f99fc5c2e4bc201413cfc1ca5767cf62ddb1a9a4); [openmirlab/bs-roformer-infer](https://github.com/openmirlab/bs-roformer-infer)). Second, **separation stops at "guitar"**. No open-weight model splits lead from rhythm guitar. Moises markets one as "first-ever" without publishing metrics ([Moises blog](https://moises.ai/blog/moises-news/new-guitar-separation-models/)), MVSEP offers only a demo ([MVSEP news](https://de2.mvsep.com/news)), and LALAL.AI puts all guitars in one stem ([LALAL.AI](https://www.lalal.ai/blog/andromeda-supports-guitars-updates-piano/)). Keyboards other than piano have no dedicated stem and fall into "other".
+
+For the server, two MIT-licensed tools wrap these models. **python-audio-separator** is a pip/CLI wrapper. It loads MDX, VR, Demucs and RoFormer checkpoints, ships 11 ensemble algorithms and karaoke presets, and runs on CUDA, Apple MPS/CoreML and CPU. Version 0.47.0 raised RoFormer overlap defaults, which means 4–6x more forward passes ([python-audio-separator](https://github.com/nomadkaraoke/python-audio-separator)). **ZFTurbo MSST** (`pip install msst`) is the training and inference framework behind most community models. It supports 17 architectures and includes DrumSep checkpoints ([MSST](https://github.com/ZFTurbo/Music-Source-Separation-Training)). The UVR5 GUI is still on v5.6 plus beta Roformer patches ([kocpc](https://en.kocpc.com.tw/?p=12798)). Community sentiment has followed the models. UVR was the top answer in r/edmproduction's stem-splitter thread, and by mid-2026 opinion had "hardened toward MVSEP" ([Futureproof Music School](https://futureproofmusicschool.com/blog/best-ai-stem-separation-tools)).
+
+Two kinds of sub-separation are mature enough to use. **Vocal karaoke models** split lead from backing vocals. Open BS-RoFormer karaoke weights reach about **10.1–10.2 lead SDR**, but backing-vocal stems top out around **5.6–7 SDR**, which is usable for detecting harmony but not for clean transcription ([MVSEP algorithms](https://mvsep.com/en/algorithms)). **DrumSep models** split a drum stem into kick (18–22 SDR), snare (13–17) and toms (13–16) well. Hi-hat, ride and crash come out poorly, at 3–9 SDR ([MVSEP algorithms](https://mvsep.com/en/algorithms)). LarsNet is an academic alternative, but MVSEP could not reproduce its metrics, and the LARS plug-in weights are CC BY-NC ([MVSEP LarsNet](https://mvsep.com/algorithms/31?lang=en); [LARS](https://github.com/EdoardoMor/LARS)).
+
+In the browser, **only Demucs is proven**:
+
+- **monteslu's htdemucs ONNX build** is 126 MB in mixed precision. It is split into 21 chained pieces so the UI never freezes, and keeps convolutions in fp32 because full fp16 "badly degrades drums and bass" ([HF monteslu](https://huggingface.co/monteslu/htdemucs-web-onnx)).
+- **6-stem WebGPU exports** of htdemucs_6s exist ([HF kramp](https://huggingface.co/kramp/htdemucs-6s-webgpu-onnx)).
+- **demucs-rs** (Apache-2.0) runs in the browser via WASM+WebGPU, but "significantly slower than the native CLI" ([demucs-rs](https://github.com/nikhilunni/demucs-rs)).
+- **CPU-only WASM is impractical.** freemusicdemixer took about 41 minutes for a 7-minute track single-threaded. Its weights are proprietary, and the repo was archived in April 2025 ([freemusicdemixer repo](https://github.com/sevagh/freemusicdemixer.com)).
+
+A community ONNX export of BS-RoFormer SW exists, with no model card and no published browser speed ([HF elicwhite](https://huggingface.co/elicwhite)). On a server GPU, htdemucs separates a 4-minute song in roughly 15–30 s, against 2–3 minutes on CPU ([MISST](https://github.com/Frikallo/MISST); [LinuxLinks](https://www.linuxlinks.com/machine-learning-linux-demucs-music-source-separation/2/)). No primary-source RoFormer timings were found.
+
+## Per-stem specialists beat all-in-one models for keys, vocals and bass
+
+Multi-instrument transcription made a real jump in 2026, but with a licensing catch. **Kyutai's MuScriptor** (ISMIR 2026) is a decoder-only Transformer trained on 1.45M rendered MIDI files plus about 11,000 hours of real recordings with aligned notes. It outputs notes over 36 instrument subgroups, including Voice, E-Piano, E-Bass, Clean E-Guitar and Distorted E-Guitar ([arXiv 2607.08168](https://arxiv.org/html/2607.08168v2)). On 372 held-out real tracks it scores **48.2 Multi F1 against 21.9 for YourMT3+**, with onset F1 of 60.4 against 32.5 ([arXiv 2607.08168](https://arxiv.org/html/2607.08168v2)). The code is MIT, but **the weights are CC BY-NC 4.0 and gated**. Released sizes are 103M (CPU-practical), 307M (default) and 1.4B. The paper lists slightly different parameter counts. There is no ONNX or browser path, and sheet output needs MuseScore 4 ([GitHub muscriptor](https://github.com/muscriptor/muscriptor)). **YourMT3+** is GPL-3.0, lightly maintained (30 commits, last news November 2024) and labelled "pre-release" ([GitHub YourMT3](https://github.com/mimbres/YourMT3)).
+
+Even the leader stays below 50% on real music, and adding instruments hurts sharply. In the 2025 AMT Challenge, F1 roughly halved going from one to three instruments. YourMT3-MoE fell from 0.759 to 0.392, and the main failures were hallucinated instruments and dense polyphony ([arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)). These models are best treated as a non-commercial quality reference, not as the product engine. Their weakness argues for separating first and then transcribing each stem, although **no controlled study of stems versus mix was found**. Songscription warns that stem bleed creates false notes ([Songscription](https://www.songscription.ai/blog/separate-stems-before-transcribing)).
+
+**Spotify's Basic Pitch** is the default lightweight transcriber:
+
+- **What it is:** Apache-2.0, polyphonic and instrument-agnostic. It outputs pitch bends, ships as CoreML, TFLite and ONNX, and has an official TypeScript/browser port, `@spotify/basic-pitch` ([basic-pitch](https://github.com/spotify/basic-pitch); [basic-pitch-ts](https://github.com/spotify/basic-pitch-ts)).
+- **Its own guidance:** it "works best on one instrument at a time", which suits a stem-based design.
+- **Where it ranks:** it came last in the AMT Challenge (0.063), mainly because it outputs no instrument labels. It was also the fastest entry ([arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)).
+- **A possible successor:** a September 2026 paper reports its 26K-parameter Harmonica-nano beating Basic Pitch's frame F1 by 14.6 points at 1,600x real time ([arXiv 2609.04640](https://arxiv.org/abs/2609.04640)). Code and license are unknown.
+- **Maintenance:** the TS port's mirror was "last synced over a year ago" ([ecosyste.ms](https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fbasic-pitch-ts)). An ONNX re-export for ORT-Web exists ([HF daserge](https://huggingface.co/daserge/basic-pitch-onnx)).
+- **User sentiment:** no primary-source user reviews were found.
+
+The best tool for each pitched instrument:
+
+- **Keyboard.** Solo piano is essentially solved on studio recordings. **Transkun** (12.9M params) reaches **93.48% note-with-offset F1** on MAESTRO v3 and is pip-installable ([arXiv 2605.17405](https://arxiv.org/pdf/2605.17405); [Transkun](https://github.com/yujia-yan/transkun)). The ByteDance high-resolution model (Apache-2.0 code, models pedal) was **archived on Dec 8, 2025** ([bytedance/piano_transcription](https://github.com/bytedance/piano_transcription)). Omnizart (MIT) is TensorFlow-era and incompatible with ARM Macs ([Omnizart](https://github.com/Music-and-Culture-Technology-Lab/omnizart)). The real constraint is the piano stem at about 7.8 SDR, not the transcriber. Organ and synth parts land in "other" and need Basic Pitch.
+- **Vocals.** Pitch tracking is nearly solved, but segmenting it into notes is not. **FCPE** matches RMVPE's 96.8% raw pitch accuracy with 10.6M instead of 90.4M parameters, at 5x the speed ([arXiv 2509.15140](https://arxiv.org/html/2509.15140v1)). **PESTO** handles a 3-minute file in about 13 s on a laptop CPU, against about 12 minutes for CREPE ([PESTO](https://github.com/SonyCSLParis/pesto)). For notes, **RMVPE → ROSVOT** is the de facto research chain ([ROSVOT](https://github.com/RickyL-2000/ROSVOT)), with SOME as a lighter option. ROSVOT's checkpoint is trained mainly on Mandarin ([TelkNet](https://telknet.cc/en/tools/music-vocal-midi)). Vibrato, slides and melisma are where notation breaks down. Whisper runs on WebGPU for lyrics ([Remotion](https://remotion.dev/docs/whisper-webgpu)).
+- **Bass.** No maintained dedicated bass transcriber exists. A 2024 project notes "a lack of automatic transcription solutions for the electric bass guitar" ([HKU](https://wp2023.cs.hku.hk/fyp23026/wp-content/uploads/sites/27/final_paper.pdf)). Bass is mostly monophonic and its stem is the second-cleanest at about 14.6 SDR, so running a pitch tracker or Basic Pitch on the stem is the pragmatic route. The low E (about 41 Hz) needs explicit testing.
+
+Licenses for Transkun, ROSVOT, SOME, RMVPE, FCPE and PESTO were not verified and must be checked before shipping.
+
+## Guitar tabs remain roughly half right end to end
+
+Producing tablature means solving two problems: finding the notes, then choosing a string and fret for each. The 2026 state of the art is **TART** (Berkeley, revised September 2026). It is a four-stage pipeline:
+
+1. A high-resolution audio-to-MIDI model.
+2. A 160K-parameter technique classifier.
+3. A 15M-parameter T5 string/fret assigner, "AudioFret", which is conditioned on audio.
+4. A BeatNet-quantized MusicXML/JAMS renderer.
+
+TART reports **81.35% note F1 (onset ±50 ms)**, **71.8% Tab F1 when given perfect MIDI**, and **54.08% Tab F1 end to end**. It attributes a 17.75-point "propagation cost" to note-detection errors ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). The test sets are GuitarSet and EGDB, mostly solo guitar, so dense distorted rock mixes will score lower. **No TART code or weights were found**, nor any for the September 2026 diffusion model Noise2Fret ([arXiv 2608.30854](https://arxiv.org/abs/2608.30854)). The training data is public and has improved:
+
+- **GOAT:** 5.9 h of DI electric guitar, amp-augmented to 29.5 h, CC BY 4.0, with tabs **in Guitar Pro format** ([arXiv 2509.22655](https://arxiv.org/abs/2509.22655)).
+- **GAPS:** 14 h of classical guitar ([arXiv 2408.08653](https://arxiv.org/abs/2408.08653)).
+- **SynthTab:** DadaGP's 26k songs rendered to audio ([arXiv 2309.09085](https://arxiv.org/pdf/2309.09085)).
+
+Distorted guitar is still under-evaluated. No paper benchmarks high-gain full-mix tabs.
+
+For string/fret assignment, learned models beat heuristics only on hard material. The Fretting-Transformer results ([arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)):
+
+| Material | Fretting-Transformer | Guitar Pro | TuxGuitar | A* |
+|---|---|---|---|---|
+| Easy (GuitarToday) | 98.4% | 97.6% | 98.3% | – |
+| Leduc (jazz) | 72.2% | 56.0% | 57.9% | 62.6% |
+| DadaGP | 81.6% | 76.6% | 78.8% | 78.9% |
+
+On the easy GuitarToday material, a trivial "lowest fret" rule nearly ties the model. Its code was not found. **midi2tab_Deconvolutions** has public code but assumes standard 6-string tuning ([arXiv 2510.10619](https://arxiv.org/html/2510.10619v1)). Given these numbers, a playability-cost Viterbi assigner is a sound license-clean first version. Its costs would be fret span, position shifts and open-string preference, and it would be aware of tuning and capo. A T5 assigner fine-tuned on DadaGP/GOAT is the upgrade path.
+
+Techniques and tuning are partly solved:
+
+- **Technique classification** works on curated data: TART reports 95.9% macro F1 across 9 classes, with vibrato the weakest at 89.4% because it is confused with bends ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). No open technique weights or end-to-end technique scores exist.
+- **Tuning and capo are user inputs** in every research system. No detector was found. Notes below E2 strongly hint at drop or down-tuning. Capo is effectively unobservable from audio.
+- **Bends** need a pitch contour rather than a class label. Basic Pitch's pitch-bend output, or an F0 track on the stem, can provide it.
+
+**Multiple guitars are the least solved requirement**, and users notice. Heise found Songsterr AI "fairly accurate" but unable to always tell multiple guitar parts apart ([heise](https://heise.de/-10356152)). No open model separates guitars from each other, so the practical approaches are untested DSP and symbolic tricks:
+
+- Split the mix into left, right, mid and side channels before and after separation. Hard-panned double-tracked rhythm parts dominate the sides, and centered leads dominate the mid.
+- Split voices symbolically after transcription: monophonic high line as lead, chordal material as rhythm.
+- Let the user declare how many guitars there are and where they are panned.
+
+These approaches need benchmarking in the next phase.
+
+For Guitar Pro I/O, **alphaTab** (MPL-2.0) is the clear fit for a website. Its `Gp7Exporter` writes .gp (GP7+) files to a Uint8Array in browser or Node, available since 1.2.0 ([alphaTab exporter](https://alphatab.net/docs/guides/exporter)). An alphaTex text exporter arrived in 1.7, and a GP8 drum-export bug was fixed in December 2025, which shows active maintenance ([alphaTab 1.7](https://alphatab.net/docs/releases/release1_7); [PR #2456](https://github.com/CoderLine/alphaTab/pull/2456)). On the Python side, **PyGuitarPro** reads and writes only GP3–5 under LGPL-3.0 ([PyGuitarPro](https://github.com/Perlence/PyGuitarPro)). ApolloTab claims GP3–GP8 support, which is unverified ([PyPI apollotab](https://pypi.org/project/apollotab/)). No dedicated bass-tab model exists either. Commercial tools such as Tabtify and Songscription offer 4/5/6-string bass tabs with no published accuracy ([Tabtify](https://tabtify.com/convert/audio-to-bass-tab); [Songscription](https://www.songscription.ai/bass-tab-generator)).
+
+## Drums: kick and snare are reliable, cymbals are not
+
+Drum transcription has a usable open stack, but the licenses split it in two. The 2025–26 options:
+
+| Model | Classes | Onset F1 | License | Notes |
+|---|---|---|---|---|
+| **ADT_STR** (Jan 2026) | 26 GM-derived | claims SOTA on ENST/MDB (no numbers retrieved) | **CC BY-SA 4.0**, weights on HF | Trained on synthetic data only |
+| **ADTOF** | 5 | MDB 86.9, IDMT 94.7, E-GMD 45.0 | **CC BY-NC-SA 4.0** | Trained on real music; PyTorch port; low maintenance (8 commits) |
+| **Noise-to-Notes** (diffusion) | 7 | E-GMD 89.7, MDB 87.9; velocity F1 82.8 | code/weights not found | ~50M params plus a 330M MERT encoder |
+| **OaF-Drums** (Magenta) | GM | E-GMD 83.4, MDB 75.2 | E-GMD data CC BY 4.0; code license not checked | Velocity-aware |
+| MuScriptor / YourMT3+ | drums, onset-only | 50.2 / 41.4 | NC / GPL | Too weak for drum notation on their own |
+
+Sources: [ADT_STR](https://github.com/pier-maker92/ADT_STR); [arXiv 2601.09520](https://arxiv.org/abs/2601.09520); [ADTOF](https://github.com/MZehren/ADTOF); [arXiv 2509.21739](https://arxiv.org/html/2509.21739v2); [Magenta](https://magenta.tensorflow.org/oaf-drums); [arXiv 2607.08168](https://arxiv.org/html/2607.08168).
+
+ADTOF collapses to 45 F1 on electronic-kit audio. That domain gap is a reason to evaluate on the project's own songs rather than trust published scores. An ISMIR 2025 diagnostic study confirms the expected weaknesses ([ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)):
+
+- **Toms and cymbals** are the worst classes.
+- **Simultaneous hits** are the main error source on drum-only audio.
+- **Melodic instruments and vocals** are the main limit in full mixes.
+
+This last point argues for transcribing a separated drum stem. There is no evidence that further splitting the stem into kick, snare and so on with DrumSep or LarsNet improves transcription. Velocity, and therefore ghost-note notation, is modeled only by OaF-Drums, hFT-Transformer and N2N.
+
+No drum transcription demo running in the browser was found ([browserTranscribe](https://github.com/jsphweid/browserTranscribe)). User sentiment is thin. An r/DRUM user found Klangio's Drum2Notes poor even on a simple isolated beat, while another called Songsterr AI "very good" ([Reddit r/DRUM](https://nyc1.lr.ggtyler.dev/r/DRUM/comments/1e5qmzg/ai_that_translates_audio_into_sheet_music)).
+
+## Rhythm, notation and ingestion decide whether output looks like music
+
+Raw note events only become sheet music once they sit on a beat grid. **Beat This!** (CPJKU, ISMIR 2024) is the default beat tracker for a commercial site. Its code and weights are **MIT**, it tracks beats and downbeats without a DBN, falls back to CPU, and has an **8 MB small model**, which makes it a candidate for a browser port ([beat_this](https://github.com/CPJKU/beat_this)). The older options each have a blocker:
+
+- **madmom** has had no PyPI release since 2018, breaks on Python ≥3.10 with NumPy ≥1.24, and has **non-commercial CC BY-NC-SA model files** ([PyPI madmom](https://pypi.org/project/madmom/); [madmom](https://github.com/CPJKU/madmom)).
+- **allin1** (MIT) adds verse/chorus labels but depends on madmom and NATTEN ([all-in-one](https://github.com/mir-aidj/all-in-one)).
+- **essentia.js** is the only mature in-browser analysis library, but it is AGPLv3 unless you buy a license ([Essentia licensing](https://essentia.upf.edu/licensing_information.html)).
+
+For chords, BTC's 170-class checkpoint is the practical baseline ([HF btc-chord](https://huggingface.co/puar-playground/btc-chord)). ChordFormer (2025) claims SOTA but its code is unconfirmed ([arXiv 2502.11840](https://arxiv.org/abs/2502.11840v1)). Research models for turning performance MIDI into a score are almost all piano-only. PM2S beats commercial software on MV2H ([ISMIR 2022](https://archives.ismir.net/ismir2022/paper/000047.pdf)). For guitar, bass, vocals and drums, the workable method is to interpolate onsets between detected beats and snap them to allowed subdivisions. This handles tempo drift better than a fixed BPM grid.
+
+On rendering, **alphaTab** is the only library with multi-track standard notation, guitar and bass tab, drum notation, SoundFont playback, audio export, and an External Cursor API that syncs the score to external audio or a YouTube video ([alphaTab 1.6](https://alphatab.net/docs/releases/release1_6); [features](https://demo.alphatab.net/features.html)). It is the natural core of the product. MusicXML can go to OSMD (BSD-3, playback still sponsor-only) or Verovio (LGPL-3.0 WASM, with MEI tablature since 2025) for print-quality standard notation ([OSMD](https://github.com/opensheetmusicdisplay/opensheetmusicdisplay); [Verovio changelog](https://data.safetycli.com/changelogs/verovio)). Editing is not built into these renderers, so a correction UI has to be built on the score model. Users ask for such an editor in every review.
+
+Ingestion splits cleanly into an easy part and a hard one. In the browser, Web Audio's `decodeAudioData` handles mp3, wav, m4a and ogg/opus in modern browsers, with ffmpeg.wasm (about 31 MB) lazy-loaded as a fallback ([ffmpeg.wasm #623](https://github.com/ffmpegwasm/ffmpeg.wasm/issues/623)). One timing caveat: MP3 decoders add **20–40 ms offsets** ([all-in-one](https://github.com/mir-aidj/all-in-one)), which is close to the 50 ms tolerance used in evaluation. **YouTube links are the risky input**:
+
+- **yt-dlp only works server-side.** Since February 2026 it needs an external JS runtime (Deno) ([openSUSE](https://src.opensuse.org/pool/yt-dlp/pulls/4/files)), and it increasingly needs PO tokens that it cannot generate itself ([yt-dlp PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide)).
+- **The token workaround is failing.** The bgutil token provider now warns that tokens "no longer bypass the bot check for majority of cases" ([PyPI bgutil](https://pypi.org/project/bgutil-ytdlp-pot-provider)).
+- **There is legal exposure.** The RIAA's 2020 takedown of youtube-dl cited anti-circumvention under DMCA §1201 ([XDA](https://www.xda-developers.com/youtube-dl-riaa-dmca/)).
+
+The lower-risk design: users upload their own file, and a YouTube link only embeds the official player, synced to the score through alphaTab's cursor API.
+
+## Users treat every AI transcriber as a first draft
+
+Commercial tools set the bar this project has to clear, and the feedback is consistent across them:
+
+- **Klangio** (Guitar2Tabs and siblings) takes YouTube links and exports Guitar Pro and MusicXML. It is rated 3.6/5 on the App Store, with reports of crashes on MusicXML export ([App Store](https://apps.apple.com/app/id1589660528)). Its Trustpilot reviews range from "legendary" to "didn't even come close" ([Trustpilot](https://uk.trustpilot.com/review/klang.io)). An independent August 2026 test saw accuracy drop on phone or noisy recordings ([agent-finder](https://agent-finder.co/reviews/klangio)). A competitor says it "tends to get the notes right" but lays them out unplayably and detects bends inconsistently ([Songscription](https://www.songscription.ai/blog/best-ai-guitar-tab-generators)).
+- **Songsterr AI** does guitar, bass and drums in one pass. Bass players are split between "dead on accurate" and "if you are after accuracy you'll not get that" ([BassBuzz](https://forum.bassbuzz.com/t/new-songsterr-ai-is-impressive/70519)).
+- **Complaints that recur** across products: unplayable fingerings, confused multiple guitars, inconsistent technique detection, paywalls on export, and per-song ticket pricing.
+
+These map directly onto features this design can deliver: a playability-aware assigner, explicit multi-guitar handling, an editor, open GP and MusicXML export, and free processing in the browser.
+
+On the open-source side, the usual recipe is yt-dlp → Demucs → Basic Pitch → music21 → engraver. The most complete example is **oh-sheet**, which outputs two-hand piano PDFs with a 25-fixture evaluation harness. Its license is ambiguous: the sidebar says MIT but the README badge says "Proprietary" ([oh-sheet](https://github.com/Oh-Sheet-Team/oh-sheet)). **No open end-to-end guitar-tab or drum pipeline exists**, so this project would be first in that space.
+
+## Recommended architecture: browser draft, server quality pass
+
+The design runs a complete **"draft" pipeline in the browser**, built only from permissive components, and offers an optional **server "quality" pipeline** that swaps in larger models. Both feed the same symbolic score model, rendered and exported by alphaTab. Each browser stage runs in its own Web Worker, terminated after use. That is the reliable way to free memory, because ORT-Web's 4 GB WASM heap does not shrink when a session is disposed ([ORT large models](https://onnxruntime.ai/docs/tutorials/web/large-models.html); [Chromium group](https://groups.google.com/a/chromium.org/g/chromium-extensions/c/-4CNcu5bUGA/m/SxP0RfNaCAAJ)). The browser draft uses WebGPU through ONNX Runtime Web or transformers.js v4 ([transformers.js releases](https://github.com/huggingface/transformers.js/releases)). The app routes to the server when WebGPU is missing: Firefox on Linux or Android, and older Android devices ([web.dev](https://web.dev/blog/webgpu-supported-major-browsers)).
+
+| Stage | Browser draft (permissive) | Server quality tier | License caveat for commercial use |
+|---|---|---|---|
+| Ingest | decodeAudioData; ffmpeg.wasm fallback; YouTube embed for sync only | yt-dlp extraction only after legal review | yt-dlp and YouTube ToS / DMCA §1201 risk; ffmpeg.wasm core is LGPL/GPL depending on build |
+| Separation | htdemucs_6s or htdemucs ONNX on WebGPU (MIT) | BS-RoFormer SW 6-stem via python-audio-separator / MSST; optional SCNet XL ensemble for bass and drums; htdemucs_ft as clean fallback | RoFormer community weights have **no stated license**; get legal sign-off or ship Demucs only |
+| Multi-guitar | L/R/mid/side split before and after separation; user declares number of guitars and panning | same, plus symbolic lead/rhythm voice split | none (in-house DSP) |
+| Vocals | lead stem → FCPE or PESTO F0 → simple note segmentation; Whisper WebGPU for lyrics | karaoke BS-RoFormer (lead vs backing) → RMVPE/FCPE → ROSVOT or SOME | karaoke weights unlicensed; ROSVOT/SOME/FCPE/PESTO licenses unverified |
+| Keys | piano stem → Basic Pitch ONNX (Apache-2.0) | piano stem → Transkun or ByteDance (archived, Apache code); "other" stem → Basic Pitch for synths and organ | verify Transkun license; ByteDance checkpoint license unstated |
+| Bass | bass stem → Basic Pitch or F0 tracker → 4/5-string Viterbi assigner | same, larger models | none beyond the separator's |
+| Guitar | guitar stem → Basic Pitch (with pitch bends) → in-house Viterbi string/fret assigner with tuning and capo inputs | high-res guitar CRNN (Riley/GAPS-style) or a fine-tuned model → T5 assigner trained on DadaGP/GOAT → technique tagger | GOAT CC BY 4.0; DadaGP and GuitarSet licenses unverified; train in-house since no TART code |
+| Drums | not yet viable (no browser ADT port; best small models are NC) | drum stem → **ADT_STR** (CC BY-SA); ADTOF only for internal comparison | ADTOF, LARS, madmom models are **non-commercial**; CC BY-SA requires share-alike on derived weights |
+| Beat / tempo / chords | Beat This! small (8 MB) via ONNX export, untested | Beat This! full; BTC chords | avoid madmom models and essentia.js (AGPL) |
+| Quantize / notation | beat-interpolated grid snapping → alphaTex → alphaTab render, playback, GP7 export | same, plus MusicXML via music21 for OSMD/Verovio views | alphaTab MPL-2.0 is file-level copyleft (share changes to its files) |
+| Reference / benchmark only | – | MuScriptor 307M for a full-band reference transcription | **CC BY-NC**: not for production without a Kyutai license |
+
+The table hides one real trade-off: **the legally cleanest stack is measurably worse at separation**. htdemucs_6s's piano stem is at 2.23 SDR against 7.83, and the Demucs family has no strong guitar stem ([MVSEP algorithms](https://mvsep.com/en/algorithms)). If the RoFormer weights cannot be cleared, there are two remaining paths. One is to train a 6-stem RoFormer in-house with MSST on licensed data. The other is to treat keys and guitar as "best effort" in the draft tier. On a consumer server GPU, the full quality pipeline is estimated at about 1–3 minutes per 4-minute song. That figure is an inference; no RoFormer timings were published.
+
+## Benchmark against Guitar Pro references by stage, not just end to end
+
+The next phase should copy TART's habit of attributing errors to each stage, because a single end-to-end score hides where the 17-point losses happen ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). No public tool compares two GP files ([arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)), so building that comparison is the first deliverable.
+
+**Corpus and ground truth.** Each user-supplied reference .gp file is paired with the *exact* recording it transcribes. Each pair is tagged by genre, gain level (clean, crunch, high-gain), number of guitars, panning, tuning, capo and presence of keys. Three public sets add controlled anchors: GuitarSet/EGDB for guitar notes, GOAT for electric tabs in GP format, and MDB/ENST for drums. One parsing risk needs an early check. alphaTab's documented importers cover GP3–5, GP6 and alphaTex ([alphaTab features](https://demo.alphatab.net/features.html)). Importing GP7/8, the format most user references will be in, is assumed but needs testing in week one. PyGuitarPro is a fallback for GP3–5 only. The parsed reference is expanded into note events (repeats unrolled; tuning and capo applied), each carrying score position, pitch, string, fret, duration and technique flags. Drum tracks are mapped to the classes the model can output: kick, snare, toms, hi-hat, cymbals.
+
+**Alignment.** Reference tabs have bar and beat positions but no audio timestamps. Bar-level anchors come from aligning the beats of the reference's synthesized audio to Beat This! output on the real recording, refined with chroma DTW. Alignment success is reported separately so that alignment failures are not counted as transcription errors. Audio is decoded to WAV once, server-side, which removes the 20–40 ms MP3 decoder offset ([all-in-one](https://github.com/mir-aidj/all-in-one)).
+
+**Metrics.** Generated tracks are matched to reference tracks with the Hungarian algorithm on note F1. This matching is the multi-guitar test. Swapped lead and rhythm parts, or merged parts, show up directly as matching failures. The per-track metrics (via mir_eval where applicable):
+
+- **Notes:** note F1 on onset ±50 ms plus pitch, then with offsets added.
+- **Tabs:** Tab F1 (pitch + onset + string + fret), and Tab Disambiguation Rate (correct string given correct pitch).
+- **Lenient variant:** counts a different but playable position as acceptable, since human tabs legitimately disagree. Learned assigners reach only 72–82% exact agreement on hard references ([arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)).
+- **Techniques:** precision and recall for each technique.
+- **Rhythm:** beat-level onset accuracy in score time, correct bar count, and correct time signature.
+- **Playability:** a fret-span and shift difficulty score.
+- **Drums:** per-class onset F1.
+
+**Ablations and baselines.** Each benchmark song runs through four controlled comparisons:
+
+- **Mix vs stem:** answers the open question of whether separation helps transcription.
+- **Browser draft vs server quality:** same song, same metrics, plus runtime and peak memory per device class.
+- **Oracle notes:** the reference's own notes fed into the assigner, which isolates fingering quality from note detection.
+- **Separation stems:** Demucs vs RoFormer stems, which prices the license trade-off in F1 points.
+
+Basic Pitch + Viterbi is the floor. MuScriptor is a non-commercial reference ceiling. Klangio and Songsterr GP exports are commercial comparisons where their terms allow. A small human study runs alongside: guitarists rate playability and time how long it takes to correct output in the editor. This captures the "useful draft" quality that users actually judge products on.
+
+## Conclusion
+
+The bottleneck has moved. Separation and note detection improved sharply in 2025–26. What now limits a product like this is everything after the notes: deciding which guitar played what, picking playable fingerings, placing notes on a credible rhythmic grid, and keeping the whole stack licensable. That favors a team that builds strong symbolic post-processing and a correction editor on top of commodity models, rather than one waiting for a better end-to-end model. The strongest end-to-end models (MuScriptor, TART) are either non-commercial or unreleased.
+
+The browser-first design is viable for drafts today. It lowers cost and protects privacy, and that matches the main user complaints about paywalls and per-song tickets. Its quality ceiling, however, is set by the separator the project is legally allowed to ship. Two decisions should therefore come early, before heavy engineering: legal clearance of the RoFormer weights (or a plan to train in-house), and a verified way to import GP7/8 for the benchmark. If both resolve well, a published, reproducible accuracy table against real Guitar Pro tabs would be a claim no competitor currently makes.

reports/Transcription accuracy improvement ideas.md

@@ -0,0 +1,191 @@
+# Search, vote and verify beyond stem transcription
+
+The biggest accuracy gains available to us in late 2026 come from a newly released model plus our own symbolic engineering, not from a bigger end-to-end model. The model is **Noise2Fret**, a diffusion audio-to-tab model with public code that lifts electric-guitar Tab F1 on GOAT from about **0.66 to 0.795** ([arXiv 2608.30854](https://arxiv.org/html/2608.30854)). The engineering is three layers wrapped around the transcriber: search over hypotheses (tuning and capo, fingering, diffusion samples), voting across redundancy the song already contains (repeated riffs, double-tracked guitars, the cleaner bass stem), and verification of the result (render-and-compare, model agreement) that marks the bars a user needs to check. Six of the coordinator's sixteen ideas are cheap to test against our Guitar Pro references now: tuning/capo search, repetition voting, pitch-contour technique detection, bass-informed rescoring, multi-guitar pan clustering and user hints. Four are worth staging behind them: render-and-compare confidence, the note-level ensemble ("note-ROVER"), diffusion n-best rescoring and licence-clean procedural data. Three should wait: per-song tone adaptation, iterative "peeling" separation and video fingering. A licence audit changes the plan as much as any model does. Several datasets the earlier survey treated as open or unverified turn out to be **non-commercial or research-only**: GuitarSet, GOAT, DadaGP and SynthTab. The only commercially clean real-audio guitar data we found is Guitar-TECHS plus our own recordings, so a shippable guitar model has to be trained mostly on audio we render ourselves. The literature offers almost no published numbers for ensembling, repetition voting, LM rescoring or test-time adaptation applied to music transcription. Our benchmark against our Guitar Pro references will therefore produce the first evidence on most of these ideas.
+
+This report builds on the baseline survey in `reports/Music transcription tooling survey.md`. That survey set out the pipeline: separate stems (Demucs in the browser, BS-RoFormer on the server), transcribe each stem (Basic Pitch, Transkun, ADT_STR, F0 trackers), assign string and fret with a Viterbi pass, quantize to Beat This! beats, and render with alphaTab. It also found TART's **54% end-to-end Tab F1** as the 2026 research ceiling ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). Nothing below repeats that material except where a correction is needed.
+
+## Four 2025–26 results move the guitar frontier, and one paradigm does not
+
+### Noise2Fret shows that generative tab models model fingering better
+
+The earlier survey recorded that no code was available for Noise2Fret. **That is now wrong**: code and trained models are on GitHub and Zenodo ([GitHub](https://github.com/RiccardoVib/Noise2Fret); [Zenodo](https://zenodo.org/records/23037260)). Noise2Fret diffuses in a continuous embedding of string-fret targets with a 1D conv U-Net, conditioned on the STFT, spectral flux, brightness and raw audio. Five auxiliary losses (pitch-class, circle-of-fifths, positional, string-similarity and hand-span feasibility) build musical and ergonomic priors into training. The results are large:
+
+| Dataset | Noise2Fret pitch F / Tab F | TabCNN Tab F | FretNet Tab F |
+|---|---|---|---|
+| GuitarSet | 0.856 / **0.845** | 0.717 | 0.727 |
+| GOAT (mostly distorted electric) | 0.800 / **0.795** | 0.656 | 0.662 |
+
+The model is also cheap: about 15M parameters and **341M FLOPs per pass, against 3.4B for TabCNN and 17.4B for FretNet** ([arXiv 2608.30854](https://arxiv.org/html/2608.30854)). The likely reason it wins is that one pitch can be fretted in several places, so the fingering distribution has several peaks, and a generative model can represent that where a classifier averages across them. Two caveats limit what this means for us. First, both test sets are close to solo DI guitar. No numbers exist for separated stems from dense mixes. Second, **the code licence is not stated** (the paper text is CC BY-SA 4.0), so Noise2Fret is cleared for benchmarking only until we get an answer.
+
+### Weak supervision is the most mature route to real-mix training data
+
+Three methods turn imperfectly aligned scores into training signal. Riley et al. align Guitar Pro scores to a pretrained model's activations with DTW and then snap each onset to the local activation peak. That reaches **98% onset F1 at 50 ms, against 62% for DTW alone**. About 4 hours of aligned commercial jazz guitar then set a zero-shot GuitarSet record of **87.3 onset F1 against 82.9** ([arXiv 2402.15258](https://arxiv.org/html/2402.15258v1)). CountEM (ISMIR 2025) goes further and drops local alignment entirely, training EM on note-occurrence histograms. A search snippet reports that fine-tuning on GAPS raised GuitarSet accuracy from **66.2% to 84.6%** ([arXiv 2511.14250](https://arxiv.org/pdf/2511.14250)). Histogram supervision tolerates tabs whose repeats or sections do not match the recording, and user-supplied GP files often have exactly that problem. A 2026 cycle-consistency method reaches 64.81 GuitarSet frame F1 from almost no paired data. Without any pairs, however, it can converge on pitch-shifted degenerate solutions ([arXiv 2605.24193](https://arxiv.org/html/2605.24193v1)). Our reference tabs plug directly into the Riley and CountEM recipes, which is the basis for hypothesis H16c below.
+
+### Foundation encoders help multi-instrument models, and small models keep improving
+
+The 2025 AMT Challenge winner, MIROS, put a **MusicFM** self-supervised encoder into YourMT3+ and added parallel per-instrument-group decoders. It scored **0.600 note-onset F1 against 0.393 for MT3** ([arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)). The same paper shows why the separation front-end still matters: MIROS precision fell from **0.907 on solo pieces to 0.464 on three-instrument pieces**. At the other end of the size range, Harmonica-nano (26.3K parameters) claims frame F1 **14.6 points above Basic Pitch** at 1,622× real time ([arXiv 2609.04640](https://arxiv.org/abs/2609.04640)). No weights are released, but it suggests the browser bottleneck is the separator, not the note model. Neither paper isolates what an SSL encoder contributes on guitar or bass. Testing a MusicFM or MERT head on our stems would therefore produce new evidence, and that test should wait until the encoder licences are verified.
+
+### General audio-language models remain unusable for note-level transcription
+
+Every source agrees on this. Gemini models score near ceiling when given MIDI, but "audio reduced accuracy across tasks" ([arXiv 2510.22455](https://arxiv.org/pdf/2510.22455)). GPT-4o (55.9%) and Gemini 2.5 Pro (53.1%) sit barely above chance on performance-audio reasoning ([arXiv 2601.11968](https://arxiv.org/pdf/2601.11968)). On HMU-Bench, model rankings on notation and on audio do not correlate (Spearman −0.04) ([arXiv 2610.15773](https://arxiv.org/html/2610.15773)). The structural reason is frame rate: a Whisper-style encoder emits one frame per ~40 ms ([arXiv 2407.10759](https://arxiv.org/pdf/2407.10759)), which is too coarse for ±50 ms onsets in dense passages. LLMs still have a narrower job. They can propose *edit operations* on the symbolic score, such as changing a time signature or respelling a passage, and a deterministic validator in the style of [arXiv 2510.02746](https://arxiv.org/pdf/2510.02746) then applies and checks each edit. That safeguard is needed because LLM graders miss octave errors ([arXiv 2610.15773](https://arxiv.org/html/2610.15773v1)).
+
+### The notation side has better quantizers and better metrics than the baseline assumed
+
+A beat-conditioned T5 quantizer takes performance MIDI plus beat annotations, which is exactly what Beat This! produces. It reaches **92.1% onset F1 and 90.2% note-value accuracy on Leduc jazz guitar GP files** ([arXiv 2604.22290](https://arxiv.org/html/2604.22290v1)). It assumes a one-to-one note correspondence, so it must run after note cleanup, and no code has been released. For evaluation, a 2026 study with 106 listeners and 3,180 ABX judgements found that **playback metrics track human preference far better than notation edit distance**. CLEWS scored ρ=0.971 and DTW-chroma ρ=0.891. On the notation side, MUSTER's onset error rate reached ρ=0.84, against 0.38–0.50 for OMR-NED. The code is MIT ([arXiv 2608.04511](https://arxiv.org/html/2608.04511); [AMT-Dual-Eval](https://github.com/pingw220/AMT-Dual-Eval)). The study covers piano only, but it matters twice for us. It argues for adding a playback metric to the benchmark, and it supports render-and-compare (H1 below), at least at system level.
+
+## Licence corrections leave almost no commercially clean real guitar data
+
+The new notes overturn several assumptions in the baseline survey. **These corrections change what we can train on:**
+
+| Item | Baseline survey said | Corrected status | Source |
+|---|---|---|---|
+| **GuitarSet** | licence unverified | **CC BY-NC 4.0**. mirdata's "MIT" applies to the loader code only. | [Zenodo 1422265](https://zenodo.org/records/1422265) |
+| **GOAT** | CC BY 4.0 | **Research-only, data by request**. CC BY covers the paper, not the data, and some content is covers of songs the authors do not own. | [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1) |
+| **DadaGP** | licence unverified | **Research-only, by email request**. The repo code is MIT; no data licence is stated. | [GitHub dadaGP](https://github.com/dada-bots/dadaGP); [arXiv 2107.14653](https://arxiv.org/pdf/2107.14653) |
+| **SynthTab** | public training data | **No licence stated.** It is derived from DadaGP and rendered with commercial Ample Sound plugins, so treat it as research-only. | [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4); [synthtab.dev](https://synthtab.dev) |
+| **Noise2Fret** | no code found | **Code and models public**; code licence not stated. | [GitHub](https://github.com/RiccardoVib/Noise2Fret) |
+
+The practical consequence is that nearly every published guitar model, including any fine-tune we would start from, was trained on non-commercial data. NC licensors do sue. **Jamendo sued NVIDIA in June 2026, seeking about $20.3M**, over commercial training on CC BY-NC MTG-Jamendo; these are allegations so far ([Music Business Worldwide](https://www.musicbusinessworldwide.com/nvidia-hit-with-us-copyright-suit-claiming-it-trained-ai-models-on-winamp-subsidiary-jamendos-music-without-permission/)). The table below scores everything we might train on or ship. "Eval only" means internal benchmarking. Whether even that is safe under each licence needs a legal view, which this report does not give.
+
+| Asset | Licence / status | Train production weights? | Ship in product? | Risk | Source |
+|---|---|---|---|---|---|
+| Guitar-TECHS (electric guitar techniques, chords) | CC BY 4.0 | Yes | n/a | Low. Has no power chords. | [arXiv 2501.03720](https://arxiv.org/html/2501.03720v1) |
+| Slakh2100 (145 h rendered multitrack) | CC BY 4.0 | Yes | n/a | Low | [Zenodo 4599666](https://zenodo.org/records/4599666) |
+| E-GMD, Groove (drums) | CC BY 4.0 (mirdata lists Groove as BY-SA) | Yes | n/a | Low; Magenta is the authoritative source | [Magenta E-GMD](https://magenta.tensorflow.org/datasets/e-gmd); [Magenta Groove](https://magenta.tensorflow.org/datasets/groove) |
+| StemGMD | CC BY 4.0 (per HF card) | Yes | n/a | Low–medium | [HF stemgmd](https://huggingface.co/datasets/schismaudio/stemgmd/blob/main/README.md) |
+| GuitarSet | CC BY-NC 4.0 | **No** | n/a | High | [Zenodo](https://zenodo.org/records/1422265) |
+| IDMT-SMT-Guitar v2 | CC BY-NC-ND 4.0 | **No** (not even derivatives) | n/a | High | [Zenodo 7544110](https://zenodo.org/records/7544110) |
+| GOAT | research-only, on request | **No** | n/a | High | [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1) |
+| DadaGP (and any tab LM trained on it: GTR-CTRL, ShredGP, ProgGP, MIDI-to-Tab) | research-only | **No** | **No** | High; weight licences not checked | [GitHub dadaGP](https://github.com/dada-bots/dadaGP) |
+| SynthTab | unlicensed, DadaGP-derived | **No** | n/a | High | [synthtab.dev](https://synthtab.dev) |
+| MAESTRO, MoisesDB, MedleyDB, M4Singer | CC BY-NC(-SA) | **No** | n/a | High | [MAESTRO](https://magenta.tensorflow.org/datasets/maestro); [moises-db](https://github.com/moises-ai/moises-db) |
+| MUSDB18-HQ | educational only | **No** | n/a | High | [Zenodo](https://zenodo.org/record/3338373) |
+| MTG-Jamendo (pseudo-label pool) | CC BY-NC dataset terms; per-track licences vary | Only per-track CC BY/CC0 subsets | n/a | High; active litigation | [UPF](https://repositori.upf.edu/items/e48c8c42-dba1-404d-85b7-5a0d6ef99e6d/full) |
+| GAPS, EGDB, Riley jazz set, Opencpop, MIR-ST500 | no licence found | Not until cleared | n/a | Unknown | [ISMIR 2024](https://ismir2024program.ismir.net/poster_171.html); [arXiv 2402.15258](https://arxiv.org/html/2402.15258v1) |
+| Noise2Fret code and weights | licence not stated; trained on GuitarSet/GOAT (NC/research) | Retrain on clean data | Not as published | High as published | [GitHub](https://github.com/RiccardoVib/Noise2Fret) |
+| MuScriptor weights | CC BY-NC 4.0, gated | **No** | **No** (Kyutai licence needed) | High; use as benchmark ceiling | [GitHub muscriptor](https://github.com/muscriptor/muscriptor) |
+| ADTOF | CC BY-NC-SA 4.0 | **No** | **No** | High | [ADTOF](https://github.com/MZehren/ADTOF) |
+| ADT_STR | CC BY-SA 4.0 | Share-alike probably attaches to derived weights | With SA obligations | Medium | [ADT_STR](https://github.com/pier-maker92/ADT_STR) |
+| MERT / MusicFM / MuQ encoders | not verified | Not until verified | Not until verified | Unknown | [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1) |
+| Demucs, Basic Pitch | MIT / Apache-2.0 | Yes (as teachers or bases) | Yes | Low | [HF monteslu](https://huggingface.co/monteslu/htdemucs-web-onnx); [basic-pitch](https://github.com/spotify/basic-pitch) |
+| pedalboard, DawDreamer (rendering) | GPL-3.0 | Offline data generation is fine in the usual reading | **No** (keep out of the bundle) | Low offline | [PyPI pedalboard](https://pypi.org/project/pedalboard/0.9.2); [DawDreamer](https://github.com/DBraun/DawDreamer) |
+| NAM trainer / community NAM profiles | MIT / per-uploader terms | Trainer yes; profiles only if terms allow or we capture our own | n/a | Medium | [neural-amp-modeler](https://github.com/sdatkinson/neural-amp-modeler) |
+| Commercial sample libraries (Ample Sound etc.) | EULA silent on ML training | Not until the EULA is checked | n/a | Medium | [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4) |
+| AMT-Dual-Eval metrics code | MIT | n/a | Yes | Low | [GitHub](https://github.com/pingw220/AMT-Dual-Eval) |
+| Our reference GP tabs and recordings | ours, but transcriptions of copyrighted songs | Internal use; ownership of the derivative needs legal review | Not as content | Medium | inference |
+| User corrections and uploads | needs explicit ToS consent | Only with consent | n/a | Medium | inference |
+
+The clean pool is therefore Slakh2100, E-GMD, Groove, StemGMD, Guitar-TECHS, our own recordings, and anything we render ourselves from material we hold rights to. Distilling from NC teachers such as MuScriptor is legally untested, so we treat those teachers as benchmark-only. For guitar specifically, this makes the coordinator's procedural-data idea (H3) close to mandatory.
+
+## Sixteen team hypotheses, judged against the evidence
+
+The ideas below are **untested hypotheses from our team, not literature results**. Each is linked to the nearest evidence in the notes. "Verdict" is our judgement of expected value against effort.
+
+| # | Team hypothesis | Closest evidence | Verdict |
+|---|---|---|---|
+| H1 | Render-and-compare per-bar error heatmap; re-decode the worst bars | Drexel resynthesis training; Inverse Drum Machine; playback metrics ρ≈0.89–0.97 at system level | **Phase 2.** Use it to *rank* candidates rather than as an absolute error score. |
+| H2 | Per-song tone-matched test-time fine-tune | No AMT test-time adaptation found; GOAT shows tone diversity saturates | **Defer.** Measure tone sensitivity first. |
+| H3 | Licence-clean procedural tabs → render → train (also for a guitar splitter) | SynthTab sim-to-real gap; GOAT re-amping gains | **Necessary, high effort.** Pair it with separator-in-the-loop. |
+| H4 | Pan clustering, double-track merge, role classifier, symbolic part splitter | ADRess/DUET; mixing conventions; note-level timbre clustering | **Phase 1–2**, high value |
+| H5 | Tuning/capo inference by decoder search, cross-checked with bass | DLfM 2024 tuning classifier; drop-D ambiguity | **Phase 1**, cheap |
+| H6 | Bass and chords constrain guitar/keys | Bass is the cleanest pitched stem (14.62 dB) | **Phase 1** as a soft prior only |
+| H7 | Two-mode guitar: chord+strum for rhythm, notes for lead | Power-chord templates; MERT strum detection | **Phase 2.** Needs its own metric. |
+| H8 | Repetition consolidation as a free ensemble | Chorus pooling; non-local statistics; drum repetition LMs | **Phase 1**, top value |
+| H9 | Note-ROVER across separators × transcribers | No published note-level ROVER; separator ensembles gain tenths of a dB | **Phase 2**, mainly as confidence |
+| H10 | Iterative peeling / score-informed separation | Jointist; Lin et al.; no measured guitar gain | **Defer** |
+| H11 | Diffusion n-best + LM/playability/render rescoring | Noise2Fret; ASR-style rescoring proposed, unpublished | **Phase 2**, after H1 and the Noise2Fret baseline |
+| H12 | Monophonic F0 contour for bends/vibrato/slides | Vibrato confused with bends in TART; contour inference | **Phase 1**, cheap |
+| H13 | Up-front user hints | Every research system takes tuning as input | **Phase 1** |
+| H14 | "Wrong bar → alternatives" UI, correction logging, time-to-correct | Holzapfel & Benetos; ENSEMBLE; ASR confidence markup | **Build alongside**; it is the product metric |
+| H15 | Audio-visual fingering from playthrough videos | Audio-visual *piano* AMT only | **Park.** Speculative and legally exposed. |
+| H16 | Reference GP tabs for (a) eval, (b) synthetic twins, (c) Riley-style fine-tuning | Riley 98% alignment; SynthTab gap | **Phase 0** for (a)+(b); Phase 3 for (c) |
+
+### Search over hypotheses is cheap and attacks errors the transcriber cannot fix
+
+**H5 (tuning/capo search)** is the best ratio of value to effort in the list. A wrong tuning makes every fret wrong even when every pitch is right. The evidence base is thin. A DLfM 2024 network classifies five tunings, including standard (90 songs) and drop D (86), but its accuracy figures were not retrieved ([ACM DL](https://dl.acm.org/doi/fullHtml/10.1145/3660570.3660574)). A 2023 student project found drop D hard to tell apart from E standard ([Rochester report](https://hajim.rochester.edu/ece/sites/zduan/teaching/ece477/projects/2023/DeclanParker_McCormackChew_FinalReport.pdf)). The proposed search needs no training. First estimate the global cents offset to catch Eb tuning. Then use the lowest sustained guitar pitch to narrow the candidate tunings. Finally run the fingering decoder under each candidate and keep the one with the lowest total cost. One-finger power chords on the low three strings are a strong drop-tuning signature. The baseline survey called capo "effectively unobservable", and that remains true acoustically: a capo-2 G shape and an open A shape sound almost identical. Capo inference is therefore a *notation-convention* prediction, and it is only as good as the fingering prior. Our references will tell us how often human tabbers write a capo at all. The bass cross-check is sound (a low D1 on bass supports drop D), as long as it stays a vote and not a veto.
+
+**H13 (user hints)** pairs naturally with H5. The benchmark should measure the *value of a hint* as the gap between Tab F1 with oracle tuning, number of guitars and lead side and Tab F1 with auto-detection. If that gap is small, we skip the questions and keep the upload flow frictionless. If it is large, we ask, pre-filled with the detected value.
+
+**H11 (diffusion n-best plus rescoring)** follows from Noise2Fret: repeated sampling gives multiple candidate tabs per segment at no extra modelling cost. The rescoring terms are the open question. The tab-LM shallow-fusion idea (log P_audio + λ·log P_LM) has **no published evidence for audio tab transcription** ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1) is the nearest, a single audio-conditioned T5). Historically, music LMs gave "only small gains" in AMT ([arXiv 2010.00059](https://arxiv.org/pdf/2010.00059)). The available LMs are DadaGP-trained, so they are research-only. A shipped rescorer would need an LM trained on clean data, which couples H11 to H3. The safest, licence-free term is pitch-validity constrained decoding, which masks string/fret outputs that would produce the wrong pitch ([arXiv 2607.26440](https://arxiv.org/pdf/2607.26440)). We should adopt it regardless of the H11 outcome.
+
+### Redundancy inside the song is a free ensemble
+
+**H8 (repetition consolidation)** is the hypothesis the literature most supports in spirit and least in numbers. Pooling repeated choruses improved lyric transcription accuracy and consistency ([McVicar, Ellis, Goto](https://staff.aist.go.jp/m.goto/PAPER/ICASSP2014mcvicar.pdf)). Non-local self-similarity fixes meter and bar-line errors in piano transcription ([arXiv 2008.12710](https://arxiv.org/pdf/2008.12710)). Drum transcribers regularised by bar-periodic LMs improved on ENST and RWC ([arXiv 2010.03749](https://arxiv.org/pdf/2010.03749)). Riff-based rock and metal are the best case for voting. The main failure mode is erasing real variation, such as the turnaround in the fourth repeat or a fill. Voting should therefore *propose* rather than overwrite. Where an instance disagrees with the cluster majority on a confident note, keep it and flag it. Consistent fingering across repeats is a second, readability win that Tab F1 will not capture, so we measure it directly.
+
+**H6 (cross-instrument consistency)** rests on a solid fact: at **14.62 dB SDR the bass stem is the cleanest pitched stem** from BS-RoFormer SW ([MVSEP news 64](https://mvsep.com/news/64?lang=en)), and in metal the bass usually doubles the rhythm-guitar root. The risk is genre. Funk, pop and jazz bass lines walk, play inversions and pedal, so a hard constraint would corrupt correct guitar notes. Bass roots and BTC chord labels should enter as a soft emission bonus inside the decoder. Clashes should be flagged rather than deleted, the same rule the notes recommend for out-of-key filtering. The benchmark should report the effect separately per genre tag.
+
+**H9 (note-ROVER)** has no precedent: no 2023–26 note-level multi-model voting for music was found ([EURASIP 2022, instrument-level voting only](https://link.springer.com/article/10.1186/s13636-022-00245-8)). The separator half of the ensemble promises little. MVSEP shows overlap 2→8 moving SDR from 11.75 to 11.77 ([MVSEP 128](https://hk.mvsep.com/algorithms/128)), and the Demucs shift trick adds roughly 0.2 dB at N× compute ([arXiv 1911.13254](https://arxiv.org/pdf/1911.13254)). The transcriber half is more promising, because Basic Pitch, Noise2Fret and a mix-level model fail in different ways. Even so, models running on the same stem share that stem's artefacts, so their errors correlate. We expect H9's main payoff to be **calibrated confidence** (agreement count) rather than raw F1. It is server-tier only, because every extra model multiplies browser compute.
+
+### Verification beats blind acceptance, but absolute spectral distance is a weak judge
+
+**H1 (render-and-compare)** has real support. Resynthesis has trained guitar tab models on 27+ hours of unlabeled YouTube audio ([Drexel thesis](https://researchdiscovery.drexel.edu/view/pdfCoverPage?instCode=01DRXU_INST&filePid=13549920770004721&download=true)). The Inverse Drum Machine learns separation from transcription labels alone ([arXiv 2505.03337](https://arxiv.org/abs/2505.03337)). Playback-similarity metrics rank whole systems in line with human preference ([arXiv 2608.04511](https://arxiv.org/html/2608.04511)). Bar-level reliability, however, is untested. A soundfont through a rough amp model will never match a real distorted stem, and the stem itself contains separation bleed. An absolute per-bar distance therefore mixes timbre mismatch with note errors. Our refinement is to use H1 *comparatively*: render several candidate transcriptions of the same bar and keep the one closest to the stem. The timbre error is then roughly constant across candidates and largely cancels. The distance features should be timbre-robust (log-CQT after spectral whitening, chroma, onset envelopes). The first measurement is whether per-bar distance predicts per-bar error on the references (AUROC), before anything is re-decoded. If it does not, H1 becomes a candidate ranker inside H11 and not a heatmap.
+
+**Confidence calibration** ties H1, H9 and H11 together. Dedicated work on note-level confidence in music transcription barely exists. TCP_α offers post-hoc confidence heads for frozen MIR models, framed as failure prediction under domain shift ([arXiv 2608.20326](https://arxiv.org/html/2608.20326v1)). In audio captioning, confidence-aware reranking improved both quality and calibration ([arXiv 2512.10170](https://arxiv.org/pdf/2512.10170)). We would combine the signals (posterior, model and sample disagreement, render residual, leakage into other stems, rule flags) into one score per note and per bar. That score would be calibrated on the dev split and judged by *how many reference errors land in the flagged bars*, which is what the user experiences.
+
+**H10 (peeling)** is the least practical idea here. Subtracting a resynthesised guitar from a mix needs a close match in timbre and phase, and a soundfont-plus-amp render cannot provide one for distorted guitar. Score-informed masking is the realistic variant. Older joint models report that transcription helps separation ([arXiv 2302.00286](https://arxiv.org/html/2302.00286); [arXiv 2108.03456](https://arxiv.org/abs/2108.03456)), but no 2024–26 guitar result reports both SDR and F1. Community "residual" multi-pass separation has no measured transcription benefit either. We defer H10 until H1's distance proves informative and H4's pan clustering has been tried, because pan clustering is a much cheaper way to reveal a hidden second guitar.
+
+### Multi-guitar handling remains unsolved
+
+**H4** targets the requirement that users notice most. Heise found Songsterr AI unable to always separate guitar parts ([heise](https://heise.de/-10356152)), and no research system assigns notes to multiple guitar parts in full mixes. The building blocks are old but sound. Avendano's panning index and ADRess find instruments by azimuth ([arXiv 1804.08300](https://arxiv.org/pdf/1804.08300); [Barry et al.](https://dafx.de/paper-archive/2004/P_240.PDF)). Mixing convention hard-pans double-tracked rhythm guitars and places the lead near centre ([Producer Society](https://producersociety.com/how-to-pan-guitars/)). Nobody has measured how often commercial mixes follow that convention, and the benchmark should report it from our corpus. Double-track merging has a further advantage: a hard-L/hard-R pair with near-identical notes is two noisy observations of one part, so merging is itself a two-voter ensemble *and* matches how humans tab. Where guitars share a pan position, the 2025 note-level timbre-clustering model ([arXiv 2509.12712](https://arxiv.org/html/2509.12712v3)) and the GuitarDuets joint framework ([arXiv 2507.01172](https://www.arxiv.org/pdf/2507.01172)) are the nearest research. The symbolic part splitter, which merges multi-track tabs and learns to split them, is clever free supervision. On DadaGP, though, it is research-only, so a shippable version must train on H3's procedural multi-guitar arrangements, where pan and role are known by construction. Mono, live and centre-stacked mixes will defeat the pan stage, so the role classifier (monophonic, high register, bends → lead; chordal, palm-muted, repetitive → rhythm) has to work on its own as well.
+
+**H7 (two-mode transcription)** turns a weakness into a readability win. High-gain rhythm guitar uses a handful of shapes, and power-chord templates for distorted guitar already exist ([GitHub ACR](https://github.com/Driemtax/ACR)). Strum and rhythm detection in polyphonic music works with MERT features ([arXiv 2510.05756](https://arxiv.org/html/2510.05756v1)). The catch is measurement. Our GP references are note-level and record the voicing actually played, octave doublings included, so a canonical two-note power chord will *lose* exact Tab F1 even when a guitarist would call it correct. H7 needs a chord-level metric (root + type + onset per beat against the reference reduced to chords) and an edits-to-correct measure, with Tab F1 reported but not used to decide. Guitar-TECHS has no power chords ([Zenodo](https://zenodo.org/records/14963133)), so the chord model's training data has to be rendered (H3).
+
+**H12 (pitch-contour techniques)** is cheap, browser-feasible and well motivated. TART's technique classifier is weakest on vibrato (89.4%) because it confuses vibrato with bends ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)), and older solo-guitar bend detection ranged from about 58% to 88% depending on technique ([ISMIR 2015](https://www.ismir2015.uma.es/articles/119_Paper.pdf)). An F0 contour separates the two by shape: a bend is a sustained rise, vibrato a periodic oscillation. Pitch survives distortion better than spectral shape does, although that is our inference, not a measurement. FCPE matches RMVPE's accuracy at 5× the speed ([arXiv 2509.15140](https://arxiv.org/html/2509.15140v1)), and PESTO runs a 3-minute file in about 13 s on a laptop CPU ([PESTO](https://github.com/SonyCSLParis/pesto)). Both licences still need checking. H12 only works on monophonic lead material, so it depends on H4 to isolate the lead.
+
+### Data and adaptation ideas carry the biggest uncertainty
+
+**H3 (licence-clean procedural data)** is forced by the licence table, but the evidence warns against expecting it to be sufficient. TabCNN trained only on SynthTab scored **43.1 Tab F1 on GuitarSet against 80.1 in-domain**, and 13.8 on IDMT ([arXiv 2309.09085](https://arxiv.org/html/2309.09085v4)). GOAT's NAM re-amping, drawing on about 7,000 profiles from more than 1,000 amps, raised note F1 on distorted test audio from **0.714 to 0.800**, but doubling the augmentation added nothing (0.806) ([arXiv 2509.22655](https://arxiv.org/html/2509.22655v1)). Two refinements raise H3's odds. First, mix the rendered guitars with rendered bass, drums and keys, and **run the mix through our actual separator before training**, so the model learns separator artefacts. This is plausibly the biggest gap in our pipeline, though nothing in the notes measures it. Second, keep acoustic realism and musical realism apart. Procedural riffs teach the acoustic model what notes sound like, but an LM trained on random riffs encodes random statistics, so H3 data should not be used to train H11's rescoring prior unless the generator is genuinely idiomatic. GuitarFlow-style flow-matching renderers ([arXiv 2510.21872](https://arxiv.org/pdf/2510.21872)) and string-wise DDSP guitars ([arXiv 2309.07658](https://arxiv.org/pdf/2309.07658)) could narrow the timbre gap later. Capturing our own NAM profiles avoids per-uploader licence terms.
+
+**H2 (tone-matched test-time adaptation)** is the weakest data idea. No AMT-specific test-time adaptation method was found ([arXiv 2609.01679](https://arxiv.org/pdf/2609.01679) warns of confirmation bias in self-training). Unsupervised domain adaptation did not help a piano model under microphone shift ([Fraunhofer](https://publica.fraunhofer.de/handle/publica/486541)). The GOAT AMP-XL saturation suggests that broad tone diversity already captures most of the benefit. Per-song fine-tuning also costs GPU minutes per upload and cannot run in the browser. Before building anything, run the diagnostic H16b makes possible: render each reference tab in a matched tone and in a clean tone, and compare scores. If the gap is small, tone is not the bottleneck and H2 is dead. If it is large, the cheaper versions come first: picking among a few tone-bucket fine-tunes, or adapting only normalisation statistics.
+
+**H16** gives the reference corpus three jobs, and they conflict. Every song used for fine-tuning (c) or for tuning thresholds and λ weights is spent as test data. We therefore propose a **frozen test split that nothing touches**, a dev split for tuning, and k-fold, leave-one-artist-out cross-validation for any fine-tuning, so that songs by the same band do not leak between folds. **Synthetic twins (b)** are the most underrated element. Rendering each reference GP through our renderer, then mixing and separating it, produces audio with exact ground truth for every stage. Comparing a song's twin with its real recording then splits each error into "rendering gap" and "model error", the per-stage attribution that TART argues for ([arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)). For (c), the Riley recipe (LR 1e-5, 10 s excerpts, about 100K steps) worked at about 4 hours of audio ([arXiv 2402.15258](https://arxiv.org/html/2402.15258v1)). Below about 1 hour it is unproven, and SynthTab saw small real-data fine-tunes sometimes *lower* scores ([arXiv 2309.09085](https://arxiv.org/html/2309.09085v4)). Expect H16c to pay off only once user-consented (recording, GP) pairs accumulate through H14.
+
+### The editor is both a product feature and the final metric
+
+**H14** is where all the confidence work reaches the user. The evidence that editing beats transcribing from scratch is modest and dated. Editing saved time for some participants, and mainly improved quality for less-skilled transcribers ([ISMIR 2019](https://archives.ismir.net/ismir2019/paper/000082.pdf); [DSH 2022](https://academic.oup.com/dsh/article/37/3/747/6408955)). Note-level correction is laborious: ENSEMBLE logged **53,411 actions, about 60 per 3-second clip** ([ISMIR 2016](https://archives.ismir.net/ismir2016/paper/000072.pdf)). The design implications are bar-level alternatives (drawn from H11 samples and H9 voters) rather than note-by-note editing, bulk "apply to every repeat" actions built on H8 clusters, and confidence highlighting modelled on ASR correction UIs ([arXiv 2503.15124](https://arxiv.org/pdf/2503.15124)). Logged corrections need explicit consent before they become training data. They are also biased toward flagged regions, so they are a poor source for estimating overall error rates.
+
+**H15 (audio-visual fingering)** is the most speculative idea. The notes contain only an audio-visual *piano* transcription paper ([TASLP 2024](https://dl.acm.org/doi/pdf/10.1109/TASLP.2024.3426303)) and nothing on guitar fretboard tracking. Video makes the YouTube acquisition risk worse: yt-dlp needs PO tokens that are failing, and youtube-dl drew an RIAA takedown under DMCA §1201 ([yt-dlp PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide); [XDA](https://www.xda-developers.com/youtube-dl-riaa-dmca/)). Camera angles, occlusion and edits vary from video to video. One legally clean variant is worth noting: the user's *own* webcam, in a later "play along and we fix your tab" mode. We park H15.
+
+## Ranked experiments: cheap symbolic search first, training last
+
+Each experiment below runs on the same benchmark. That benchmark parses the GP references with tuning, capo and repeats unrolled, aligns them to audio with beats plus DTW plus onset snapping, matches tracks with the Hungarian algorithm, and reports alignment failures separately. The baseline is the survey's **Basic Pitch + Viterbi assigner on BS-RoFormer stems**, with the Demucs stem as a second baseline. Success criteria are our proposed decision thresholds, not literature values, and are evaluated on the frozen test split after tuning on dev. Rank reflects expected Tab F1 or time-to-correct gain divided by engineering effort.
+
+| Rank | Experiment (hypothesis) | Effort | What to measure | Primary metric(s) | Baseline | Success criterion (proposed) |
+|---|---|---|---|---|---|---|
+| 1 | **E1 Noise2Fret drop-in** on separated guitar stems and on synthetic twins | Low (code public; licence check gates shipping) | Tab and note accuracy by gain level (clean/crunch/high-gain) | Tab F1, note F1 (±50 ms), TDR, lenient Tab F1 | BP + Viterbi | ≥ +5 Tab F1 points overall and no loss on clean material. Also confirms whether the GOAT gain survives separation. |
+| 2 | **E2 Tuning/capo search + hint value** (H5, H13) | Low | Tuning/capo classification; Tab F1 under detected vs oracle tuning | Tuning accuracy, capo accuracy, ΔTab F1 | Assume E standard, no capo | ≥ 90% tuning accuracy; Tab F1 within 1 point of oracle tuning. If not, ship the hint UI. |
+| 3 | **E3 Repetition consolidation** (H8) for guitar, bass, drums | Low–medium | Accuracy on repeated vs non-repeated bars; fingering consistency across repeats; variations wrongly erased | Note/Tab F1 on repeated bars, per-class drum F1, repeat-consistency %, false-overwrite rate | No consolidation | ≥ +3 F1 on repeated bars, ≤ 0.5 loss on non-repeated, false overwrites < 5% of changed notes |
+| 4 | **E4 F0-contour techniques** (H12) on the lead stem or cluster | Low | Bend, vibrato, slide detection vs GP technique flags | Per-technique P/R/F1; bend/vibrato confusion | Basic Pitch pitch-bend output | Bend F1 ≥ 0.7 and vibrato↔bend confusion halved |
+| 5 | **E5 Pan clustering + double-track merge + role classifier** (H4) | Medium | Whether we produce the right guitar tracks with the right notes | Hungarian track-match rate, per-track note F1, guitar-count accuracy, lead/rhythm role accuracy; % of corpus following the hard-pan convention | One "guitar" track | Correct guitar count in ≥ 75% of multi-guitar songs; matched-track note F1 ≥ single-stem F1 |
+| 6 | **E6 Bass/chord soft prior** (H6) | Low | Power-chord root errors; spurious clashing notes; per-genre effect | Note F1 on rhythm guitar and keys, root-error rate, clash-flag precision | No prior | ≥ 30% fewer root errors on rock/metal, no F1 loss on other genres |
+| 7 | **E7 Confidence stack**: render-and-compare residual (H1) + model agreement (H9) + calibration | Medium | Whether flags find the errors | Bar-level AUROC for "bar contains an error"; % of errors in top-20% flagged bars; ECE after calibration | Transcriber posterior only | AUROC ≥ 0.75; top-20% of bars capture ≥ 50% of errors |
+| 8 | **E8 Note-ROVER accuracy** (H9) | Medium | Whether voting beats the best single system | Note F1, precision at fixed recall, compute ×RT | Best single transcriber | ≥ +2 F1 at ≤ 3× compute, else keep for confidence only |
+| 9 | **E9 Diffusion n-best + rescoring** (H11, pitch-constrained decoding) | Medium (after E1, E7) | Gain from choosing among K samples | Tab F1 vs K; oracle-best-of-K (upper bound) | Single Noise2Fret sample | Rescored ≥ +2 Tab F1 and closing ≥ 30% of the gap to oracle-best-of-K |
+| 10 | **E10 Two-mode rhythm transcription** (H7) | Medium | Readability and correctness of rhythm sections | Chord-level F1 (root+type+onset), strum-direction accuracy, edits-to-correct; Tab F1 reported | Note-level for all sections | Chord F1 ≥ 0.8 and fewer edits per rhythm bar than note mode |
+| 11 | **E11 Sim-to-real attribution with synthetic twins** (H16b) and tone sensitivity (gates H2) | Medium | Gap between twin and real recording per stage; matched vs clean tone | ΔTab F1 (twin − real) per stage; ΔF1 (matched − clean tone) | n/a (diagnostic) | Produces a per-stage error budget. If tone ΔF1 < 3 points, drop H2. |
+| 12 | **E12 Separator-in-the-loop procedural training** (H3) | High | Commercial-clean model vs NC-trained model | Tab F1 on real references; the same on twins | Noise2Fret / BP as published | Clean model within 5 Tab F1 points of the NC-trained one |
+| 13 | **E13 Fine-tune on aligned references / CountEM** (H16c) | Medium–high | Gain from in-domain real audio | Tab F1 on held-out folds; variance across folds | Pre-fine-tune model | ≥ +3 Tab F1 consistently across folds (k-fold, leave-one-artist-out) |
+| 14 | **E14 SSL-encoder head** (MusicFM/MERT) for guitar and bass | High; licence-gated | Encoder vs Basic Pitch features | Note F1, Tab F1 | BP / Noise2Fret | ≥ +5 note F1 on high-gain material |
+| 15 | E15 Score-informed peeling (H10); E16 tone TTA (H2); E17 audio-visual (H15) | High | n/a | n/a | n/a | Start only if E7/E11 justify it |
+
+Two cross-cutting items run with every experiment. Notation quality is scored with MUSTER-style onset and note-value error rates alongside Tab F1. A playback metric (alphaTab render vs stem via DTW-chroma or CLEWS) is also computed; it needs no reference alignment and therefore covers songs where GP alignment fails ([arXiv 2608.04511](https://arxiv.org/html/2608.04511)). Drums get E3's bar-pattern voting, and keys get E6's harmony prior and E8's voting; both have their own per-instrument metrics.
+
+## A four-phase roadmap folds these experiments into the Guitar Pro benchmark
+
+The ordering follows three principles. Each phase's measurement infrastructure is reused by the next. Post-processing experiments run on a frozen transcriber, so their effects can be attributed. Training work starts only after the cheap wins and the error budget show where training would help.
+
+| Phase | Weeks (indicative) | Build / run | Exit gate |
+|---|---|---|---|
+| **0: Harness** | 1–3 | GP7/8 import check (a week-one risk from the survey); reference expansion; beat + DTW + onset-snap alignment (Riley-style); Hungarian track matching; metric suite (Tab/note F1, TDR, lenient Tab F1, MUSTER, playback DTW-chroma); corpus tags (genre, gain, guitar count, pan, tuning, capo); **frozen test / dev / k-fold splits**; synthetic-twin rendering of every reference; baseline runs on Demucs and RoFormer stems, mix vs stem, oracle-notes assigner | Baseline numbers per stage; alignment success rate reported; twins render cleanly |
+| **1: Symbolic wins on a frozen transcriber** | 3–6 | E2 tuning/capo + hint value; E3 repetition; E4 F0 techniques; E6 bass/chord prior; pitch-validity constraint in the assigner; start editor event logging (H14) | Keep every change that clears its criterion; publish the first internal accuracy table |
+| **2: Model swap, multi-guitar, confidence** | 5–10 | E1 Noise2Fret (licence request sent in Phase 0); E5 multi-guitar; E7 confidence stack and calibration; E8 note-ROVER; E9 n-best rescoring; E10 two-mode; E11 sim-to-real and tone-sensitivity diagnostics | Low-confidence-bar UI backed by measured AUROC; decision on whether the server tier runs ensembles; error budget per stage |
+| **3: Clean training** | 9–16 | E12 procedural + separator-in-the-loop data (own NAM captures, Slakh-style backing); retrain a clean Noise2Fret-class model and a lead/rhythm splitter; E13 CountEM/Riley fine-tune once consented pairs exist; E14 encoder probe if licences clear; distil to a browser student (fp16 WebGPU, test int8) | Clean model within 5 points of the NC-trained one; browser student within the planned budget |
+| **4: Speculative** | gated | E15 peeling, E16 tone TTA, E17 audio-visual. Start only if E7/E11 show the specific bottleneck each addresses. | Evidence-led go/no-go |
+
+Two decisions should be made in Phase 0, because they gate later phases. The first is to request licence terms from the Noise2Fret authors and to check the MERT/MusicFM licences. The second is to get a legal view on our own GP references (internal use, and whether they may later train weights) and on collecting user corrections under the ToS. A related pitfall applies to every experiment from Phase 1 on: λ weights, thresholds and calibration are fitted on dev, and the test split is touched once per phase.
+
+## Conclusion
+
+The research picture has inverted since the baseline survey. Then, the question was which transcriber to adopt. Now one strong generative model exists and is available for testing, and the larger unclaimed gains lie in exploiting structure the transcriber ignores. The constraint has moved from model quality to licensing and measurement. The best public guitar data is unusable for production weights, and almost none of the post-processing ideas have published effect sizes. Two consequences follow. First, our Guitar Pro benchmark, with synthetic twins for attribution and a frozen test split, is research infrastructure that the field lacks, not only QA. Second, the confidence stack carries more weight than any single accuracy trick. A system that is 70% right and says *which* 30% to check gives users more than one that is 75% right but silent about where it is wrong.
+
+The coordinator's ideas sort along one line: those that use evidence already in the song are cheap and testable now (tuning and capo search, repeats, double-tracks, bass roots, F0 contours), while those that generate new evidence (adaptation, peeling, video) are expensive and unproven. Render-and-compare sits between the two groups, and its value depends on a reframing: as a ranker among candidates it should work even though a soundfont never sounds like a real amp, whereas as an absolute error detector it probably will not. If Phase 1 shows the symbolic wins we expect, the product differentiator becomes the published accuracy table, together with an editor that learns from its users' corrections and gives the user a short list of alternatives instead of a blank bar.