Add research reports and synthetic benchmark design
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
@@ -0,0 +1,174 @@
|
||||
# In-browser music transcription feasibility and the commercial / open-source landscape (as of Oct 2026)
|
||||
|
||||
Note on sourcing: about 19 search/fetch calls. Many results were aggregator or competitor-written pages. These are flagged inline. Anything not backed by a fetched source is in Gaps or marked as an inference.
|
||||
|
||||
## A1. Browser ML runtimes (ONNX Runtime Web, transformers.js, TF.js, LiteRT.js, WebNN) and WebGPU support
|
||||
|
||||
### Takeaway
|
||||
WebGPU now ships in all major browsers: Chrome/Edge 113+, Chrome Android 121+, Safari 26 (Sept 2025, including iOS), and Firefox 141 on Windows and 145 on Apple Silicon macOS. Firefox on Linux/Android and Intel Macs is still partial. ONNX Runtime Web with the WebGPU EP, directly or through transformers.js v4, is the practical runtime for audio models. Google's LiteRT.js (July 2026) is a new option. WebNN is not usable in production: it is behind flags and its origin trial was paused.
|
||||
|
||||
### Cited Findings
|
||||
- WebGPU is supported in all major browsers as of November 2025 — [Wikipedia](https://en.wikipedia.org/wiki/WebGPU); [web.dev](https://web.dev/blog/webgpu-supported-major-browsers)
|
||||
- Chrome/Edge have supported WebGPU since 113. Chrome Android has it since 121, on Android 12+ devices with Qualcomm/ARM GPUs — [Wikipedia](https://en.wikipedia.org/wiki/WebGPU)
|
||||
- Firefox: WebGPU on Windows from Firefox 141 and on macOS Tahoe 26 ARM64 from Firefox 145. Linux, Android and Intel Macs are "in progress" — [web.dev](https://web.dev/blog/webgpu-supported-major-browsers)
|
||||
- Safari 26 (Sept 2025) shipped WebGPU on macOS, iPadOS, iOS and visionOS. Because all iOS browsers use WebKit, this covers every iOS browser. Some blogs cite earlier Safari versions, which conflicts with this — [Wikipedia](https://en.wikipedia.org/wiki/WebGPU)
|
||||
- Chromium Compatibility Mode for older Android (GLES 3.1) is described as a 2026 change — [Khronos SIGGRAPH 2026 deck link](https://www.khronos.org/developers/linkto/webgl-and-webgpu-sigg26). This is a secondary summary and was not fetched.
|
||||
- transformers.js v4: preview on npm `next` on Feb 9, 2026. The WebGPU backend was rewritten in C++ with the ONNX Runtime team. GA was reportedly March 2026, and 4.2.0 and 4.3.0 have since shipped (4.3.0 includes a "WebGPU upgrade") — [GitHub releases](https://github.com/huggingface/transformers.js/releases); [newreleases 4.3.0](https://newreleases.io/project/github/huggingface/transformers.js/release/4.3.0); [awesomeagents summary](https://awesomeagents.ai/news/transformers-js-v4-webgpu-browser-ml/). The GA date comes from secondary sources.
|
||||
- LiteRT.js was announced by Google on July 9, 2026. It uses XNNPACK for CPU and ML Drift for GPU (WebGPU), with WebNN for NPUs listed as upcoming — [Google Developers Blog](https://developers.googleblog.com/en/litertjs-googles-high-performance-web-ai-inference/); [Gigazine](https://gigazine.net/gsc_news/en/20260710-google-litert-js/)
|
||||
- WebNN origin trial: planned for M145–147, delayed, then moved to M147–149. A later post says it was disabled again because of release-blocking issues, with a new milestone to be announced — [blink-dev Intent to Experiment](https://groups.google.com/a/chromium.org/g/blink-dev/c/5CWKSChYo98). A Feb 2026 Phoronix report of an origin trial in Chrome 146 is outdated — [Phoronix](https://www.phoronix.com/news/Chrome-146-Beta)
|
||||
- The WebNN spec was at W3C Candidate Recommendation Draft stage as of its June 26, 2026 revision — [kingy.ai](https://kingy.ai/news/litert-js-local-browser-ai-backends-benchmarks-limits/). Secondary source.
|
||||
|
||||
### Inferences
|
||||
- Target ORT-Web with WebGPU as the main path and the WASM EP as the fallback. Plan for a WASM-only path on Firefox/Linux, Firefox Android and older Android devices.
|
||||
- TF.js was not researched separately. Basic Pitch's official TS port uses TF.js, so it remains relevant for that model, but there are ONNX re-exports (see A2).
|
||||
|
||||
### Gaps
|
||||
- No caniuse share figure for WebGPU was fetched.
|
||||
- TF.js maintenance status in 2026 was not checked.
|
||||
- MediaPipe Audio tasks were not checked.
|
||||
|
||||
## A2. Existing in-browser audio ML projects
|
||||
|
||||
### Takeaway
|
||||
Browser-side source separation is now proven. There are several 2025–26 HTDemucs ports on WebGPU, and the best of them run many times faster than realtime on a mid-range desktop GPU. Basic Pitch runs in the browser through basic-pitch-ts (TF.js) or an ONNX re-export. Whisper-style ASR on WebGPU is mature. No in-browser drum or guitar-tab transcription demo was found.
|
||||
|
||||
### Cited Findings
|
||||
- **freemusicdemixer.com (sevagh):** free, unlimited, client-side Demucs using WebAssembly. No WebGPU is mentioned in its description — [sevagh GitHub](https://www.Github.com/sevagh). A third-party listing says its free tier is limited to a lower-quality 4-stem model — [similarlabs](https://similarlabs.com/p/freemusicdemixer). This is an aggregator and unverified.
|
||||
- **demucs-wasm (npm):** a wrapper reverse-engineered from freemusicdemixer's MIT code — [README](https://cdn.jsdelivr.net/npm/[email protected]/README.md)
|
||||
- **demucs-rs:** Apache-2.0 Rust HTDemucs v4 built on Burn. The browser build is WASM plus WebGPU, and there is also a CLI, a macOS VST3/CLAP plugin and an MCP server. Models: htdemucs 84 MB, htdemucs_6s 84 MB (adds guitar and piano stems), htdemucs_ft 333 MB. The web build is "significantly slower" than the native CLI, and no benchmarks are published. ~158 stars — [GitHub](https://github.com/nikhilunni/demucs-rs)
|
||||
- **demucs-js (bakkot):** about 3x realtime on WebGPU. Without WebGPU it is "much, much slower" — [GitHub](https://github.com/bakkot/demucs-js)
|
||||
- **monteslu/htdemucs-web-onnx (Hugging Face):**
|
||||
- Size and precision: 126 MB mixed precision vs 173 MB fp32. Conv and ConvTranspose layers stay fp32 because full fp16 degraded the drums and bass stems. Per-stem correlation vs fp32 is at least 0.9998. About 2.4x faster than fp32.
|
||||
- Speed: on a Radeon RX 7600, about 22x realtime back-to-back, or about 8x realtime with UI-friendly pacing.
|
||||
- Why the model is split: it is cut into 21 chained ONNX pieces of 10–30 ms each, because a monolithic session blocks the GPU for 200–330 ms and drops frames. Chaining costs about 7% overhead.
|
||||
- Memory: keep boundary tensors on the GPU with `preferredOutputLocation: 'gpu-buffer'` and dispose them after each segment.
|
||||
- [HF model card](https://huggingface.co/monteslu/htdemucs-web-onnx)
|
||||
- Other WebGPU Demucs ONNX exports: [htdemucs-ft-webgpu](https://huggingface.co/monteslu/htdemucs-ft-webgpu), [htdemucs-6s-webgpu-onnx](https://huggingface.co/kramp/htdemucs-6s-webgpu-onnx), [timcsy/demucs-web](https://github.com/timcsy/demucs-web). The openDAW project also has an in-browser stem-separation "AI Demuxer" — [openDAW newsletter](https://buttondown.com/opendaw/archive/ai-demuxer-stem-separation/). None of these were fetched.
|
||||
- **Basic Pitch:**
|
||||
- The official web demo at basicpitch.spotify.com converts recordings to MIDI on the site — [About page](https://basicpitch.spotify.com/about)
|
||||
- basic-pitch-ts is a TypeScript library that aims for feature parity with the Python version — [ecosyste.ms](https://awesome.ecosyste.ms/projects/github.com%2Fspotify%2Fbasic-pitch-ts). Its mirror was "last synced over a year ago", so activity is uncertain.
|
||||
- An ONNX re-export for ORT-Web exists — [daserge/basic-pitch-onnx](https://huggingface.co/daserge/basic-pitch-onnx)
|
||||
- **Whisper and other ASR in the browser:**
|
||||
- @remotion/whisper-webgpu uses transformers.js and WebGPU with timestamps — [Remotion docs](https://remotion.dev/docs/whisper-webgpu)
|
||||
- The Voxtral-WebGPU HF Space — [HF](https://huggingface.co/spaces/webml-community/Voxtral-WebGPU)
|
||||
- Liquid AI's LFM2.5-Audio-1.5B WebGPU demo — [Liquid docs](https://docs.liquid.ai/examples/web/audio-webgpu-demo)
|
||||
- These are useful for lyrics, not notes.
|
||||
- **essentia.js (UPF MTG):** WASM build of Essentia for audio features, onsets and rhythm. AGPLv3 license, which matters for closed products. The README warns that the API is under rapid development — [essentia.js](https://essentia.upf.edu/essentiajs); [TISMIR paper](https://transactions.ismir.net/articles/111)
|
||||
- **Drum transcription in the browser:** none found. browserTranscribe lists drum transcription only as future work — [GitHub](https://github.com/jsphweid/browserTranscribe)
|
||||
|
||||
### Inferences
|
||||
- Separation (the heaviest stage) is about 3–22x realtime on desktop WebGPU, so a 4-minute song would separate in roughly 10 s to 80 s on a desktop GPU. This is extrapolated, not measured. Basic Pitch is a small model and should add little on top.
|
||||
- Guitar-tab and drum transcription would be new ground in the browser. No reference implementation was found.
|
||||
|
||||
### Gaps
|
||||
- No phone or integrated-GPU benchmarks for Demucs in the browser were found.
|
||||
- No published WASM-only (no WebGPU) timing for a full song was found.
|
||||
- Basic Pitch browser runtime numbers were not found.
|
||||
|
||||
## A3. Practical limits: model size, memory, runtime, workers, caching
|
||||
|
||||
### Takeaway
|
||||
The hard ceiling is the 4 GB wasm32 address space. ONNX Runtime Web cannot load models larger than 4 GB, and disposing sessions does not reclaim wasm linear memory. Demucs (84–333 MB) and Basic Pitch fit easily. The real risks are total memory when several models are loaded together, GPU long-task jank, and the cost of the first download.
|
||||
|
||||
### Cited Findings
|
||||
- ORT-Web cannot run models larger than 4 GB because of 32-bit wasm addressing. wasm64 or direct GPU weight loading are named as possible future routes — [ONNX Runtime large-models doc](https://onnxruntime.ai/docs/tutorials/web/large-models.html); [V8 4GB blog](https://v8.dev/blog/4gb-wasm-memory)
|
||||
- WebGPU can upload weights from an ArrayBuffer outside the wasm heap, so they do not count toward the 4 GB cap — [ORT commit via semanticdiff](https://app.semanticdiff.com/gh/microsoft/onnxruntime/commit/d2a1b7a35398d6612a4d71272a6f784a913c855a)
|
||||
- ORT splits inputs larger than maxStorageBufferBindingSize across multiple bindings — [ORT commit](https://app.semanticdiff.com/gh/microsoft/onnxruntime/commit/535f1f37c410575ecbbb47de8f902a137e498869)
|
||||
- Firefox bug: WebGPU `queue.writeBuffer()` fails when the wasm heap is larger than 2 GB, so Wasm4GB and Wasm64 modes are not supported there — [Mozilla bug 2022805](https://bugzilla.mozilla.org/show_bug.cgi?id=2022805)
|
||||
- Loading several sessions fills the same 4 GB space, and wasm linear memory is not reclaimed after dispose — [Chromium extensions group](https://groups.google.com/a/chromium.org/g/chromium-extensions/c/-4CNcu5bUGA/m/SxP0RfNaCAAJ)
|
||||
- Device maxBufferSize distribution data is available — [web3dsurvey](https://web3dsurvey.com/webgpu/limits/maxBufferSize/platform/Linux). The numbers were not fetched.
|
||||
- Long GPU bursts block rendering, which is why the model was chained into 21 pieces (see A2) — [monteslu HF](https://huggingface.co/monteslu/htdemucs-web-onnx)
|
||||
|
||||
### Inferences
|
||||
- Run each pipeline stage in a dedicated Web Worker. Terminating the worker, rather than calling session.dispose(), is the dependable way to free wasm memory between stages, given the reclaim issue above.
|
||||
- Process audio in segments, as Demucs already does. Use fp16 or mixed precision to keep downloads around 100 MB for separation.
|
||||
|
||||
### Gaps
|
||||
- No primary source was fetched on caching models with the Cache API or OPFS, or on storage quotas. transformers.js uses the Cache API by default (from memory, not verified here).
|
||||
- No source was found on iOS Safari per-tab memory limits.
|
||||
|
||||
## A4. Hybrid browser-first / server-fallback patterns
|
||||
|
||||
### Takeaway
|
||||
No formal write-up was found. Existing projects show the pattern in pieces: demucs-rs uses the same Rust core for the browser (WASM+WebGPU) and native GPU, and recommends the CLI for long tracks. oh-sheet runs fully server-side with a local and a remote engraving backend.
|
||||
|
||||
### Cited Findings
|
||||
- demucs-rs: one codebase targets the browser, CLI, plugin and MCP server. It recommends the CLI for batch work or long tracks because the web build is slower — [GitHub](https://github.com/nikhilunni/demucs-rs)
|
||||
- oh-sheet: a `local` engraving backend with a `remote_http` fallback when the local one fails — [GitHub](https://github.com/Oh-Sheet-Team/oh-sheet)
|
||||
|
||||
### Inferences
|
||||
- Suggested pattern: detect `navigator.gpu` and the adapter limits at runtime. Use WebGPU in the browser when it is available, and offer server GPU processing when WebGPU is missing (Firefox Linux/Android, old Android), on low-memory phones, or for the larger htdemucs_ft model. Use the same ONNX files on both sides so outputs match.
|
||||
|
||||
### Gaps
|
||||
- No published reference architecture or cost analysis for browser-first transcription with server fallback was found.
|
||||
|
||||
## B1. Commercial products: features, pricing, user sentiment
|
||||
|
||||
### Takeaway
|
||||
Users consistently describe AI tabs and sheets as useful drafts that need cleanup. Accuracy is good on clean, solo, single-line parts and drops on dense, distorted or multi-guitar mixes. Recurring complaints are fingering and position choices that a guitarist would not use, inconsistent bend and slide detection, confusion between multiple guitars, and paywalls or ticket friction. There is no independent accuracy benchmark across products.
|
||||
|
||||
### Cited Findings
|
||||
- **Klangio (Guitar2Tabs, Piano2Notes, Drum2Notes, Sing2Notes):**
|
||||
- Pricing: about 20 s free, then ticket packs ($1.99 for 1, $5.99 for 3, $9.99 for 10, $24.99 for 50, per App Store). Pro subscriptions are about $8.99–9.99/month or $89.99–99.99/year per app. The "Klangio Universe" bundle is $24.99/month or $249/year. Prices differ between listings — [Klangio App Store listings via search](https://apps.apple.com/app/id1589420929); [Drum2Notes](https://apps.apple.com/app/id6717586517). The klang.io/pricing page returned 404.
|
||||
- Features: inputs are live recording, file upload and YouTube. There is a DAW plugin and a public API. Exports are Guitar Pro, MusicXML and MIDI. Bends and slides come through in "Rock Mode" — [Songscription comparison](https://www.songscription.ai/blog/best-ai-guitar-tab-generators). This is a competitor source.
|
||||
- Accuracy: "tends to get the notes right" but lays them out unplayably. Bend and slide detection is inconsistent. The ticket model is described as friction — [same Songscription article](https://www.songscription.ai/blog/best-ai-guitar-tab-generators)
|
||||
- App Store: 3.6★ from 25 reviews. One review reports a crash on MusicXML export — [App Store](https://apps.apple.com/app/id1589660528)
|
||||
- Trustpilot: split between praise for accuracy and time saved and "flawed/unusable". One user reports a support problem after buying a yearly plan — [Trustpilot](https://uk.trustpilot.com/review/klang.io)
|
||||
- Musehub user: "even turns accidental background noises into tabs" — [Musehub](https://musehub.com/app/guitar2tabs)
|
||||
- **Songsterr AI:**
|
||||
- Generates guitar, bass and drum tabs in one pass. It is backed by a library of over 1M community tabs. The free tier has limits, and Plus is needed for full access and MIDI or Guitar Pro export. There is no MusicXML export — [Songscription](https://www.songscription.ai/blog/best-ai-guitar-tab-generators)
|
||||
- Heise: "fairly accurate" tabs from your own recordings, but it doesn't always separate multiple guitar parts correctly — [heise](https://heise.de/-10356152)
|
||||
- BassBuzz forum: one user called it "impressive", another said "if you are after accuracy you'll not get that" — [BassBuzz](https://forum.bassbuzz.com/t/new-songsterr-ai-is-impressive/70519)
|
||||
- **Songscription:** an AI transcription service from audio or YouTube. Exports Guitar Pro, MusicXML, MIDI and PDF, and includes a piano-roll editor. Free 30 s previews, and full exports need a paid plan. The source is its own marketing, which acknowledges self-interest and says the comparison is "not a controlled accuracy benchmark" — [Songscription](https://www.songscription.ai/blog/best-ai-guitar-tab-generators). It claims cleaner piano output than AnthemScore's "older offline model" — [Songscription vs AnthemScore](https://www.songscription.ai/blog/songscription-vs-anthemscore). This is a self-interested claim.
|
||||
- **AnthemScore (Lunaverus):**
|
||||
- Offline desktop app with a one-time purchase. Editions are about $31–$107 per the vendor. Directories list $19.97, which conflicts — [Lunaverus compare](https://www.lunaverus.com/compare); [SourceForge](https://sourceforge.net/software/product/AnthemScore/)
|
||||
- Accuracy: a MuseScore forum user claims about 96% on slow solo piano, with no stated method. Users say it works best on simple, solo-piano audio. The vendor says accuracy "varies from song to song" — [MuseScore forum](https://musescore.org/en/node/272456); [Lunaverus FAQ](https://lunaverus.com/faq)
|
||||
- Tip from a TalkBass user: warping or quantizing an isolated bass stem in Ableton first improves results — [TalkBass](https://www.talkbass.com/threads/absolutely-amazing-scoring-software.1262885/)
|
||||
- **Moises:**
|
||||
- Stems, chords and practice tools. Free tier is about 5 tracks/month. Premium is about $3.99/month ($2.99/month annual) and Pro is $9.99/month — [StemSplit review, updated Mar 2026](https://stemsplit.io/blog/moises-ai-review). This is a competitor source.
|
||||
- A roundup ranks Moises for quick editable parts from solo or small-ensemble recordings, and Chordify for chord charts synced to the song — [Worldmetrics](https://worldmetrics.org/best/transcribe-music-software/). This is an SEO roundup.
|
||||
- **LALAL.AI:** about $15/month (about $0.17/min). Scored 92% "clean" vocals vs 87% for Moises in a competitor's test with unclear methodology — [StemSplit](https://stemsplit.io/blog/moises-ai-review)
|
||||
- **ScoreCloud, AnthemScore:** labelled "freemium" by an aggregator, without tier prices — [ToolMage](https://www.toolmage.com/de/compare/anthemscore-vs-scorecloud/). Unverified.
|
||||
|
||||
### Inferences
|
||||
- Gaps worth targeting, based on the complaints:
|
||||
- Playable fingering and position assignment for tabs.
|
||||
- Handling more than one guitar part.
|
||||
- An editor for correcting output before export.
|
||||
- Open export formats (MusicXML is missing in Songsterr).
|
||||
- Free or local processing with no per-song tickets. Browser-side inference makes this cheap to offer.
|
||||
|
||||
### Gaps
|
||||
- Not researched within the call budget: Chordify pricing and sentiment, ScoreCloud, Guitar Pro 8 and MuseScore 4 audio import, Ultimate Guitar, RipX DAW, Melodyne, Samplab, Fadr, Riffusion, and Spleeter-based sites.
|
||||
- Searches for Fadr and RipX returned only generic comparison pages.
|
||||
- No Reddit threads with substantive accuracy discussion were retrieved. The only one found was an r/Guitar thread from July 2024 about UG tab quality, which is not about AI.
|
||||
- No independent cross-product accuracy benchmark exists in the sources found.
|
||||
|
||||
## B2. Open-source end-to-end pipelines (audio/YouTube → stems → MIDI → sheet/tab)
|
||||
|
||||
### Takeaway
|
||||
The common open-source recipe is yt-dlp → Demucs → Basic Pitch → music21 or MusicXML → LilyPond, MuseScore or Verovio. The most complete repo found is oh-sheet (YouTube or audio to a two-hand piano PDF). Several HF Spaces chain the same components. None does guitar tabs or drums end to end with good quality.
|
||||
|
||||
### Cited Findings
|
||||
- **oh-sheet:**
|
||||
- Pipeline: YouTube (yt-dlp), uploaded audio or MIDI input → optional Demucs stems → Basic Pitch ONNX (about 10 s/song on CPU) with beat, tempo and key detection → rule-based two-hand piano arrangement → "humanize" step → music21 MusicXML and LilyPond PDF.
|
||||
- Stack: FastAPI backend; Flutter web/mobile frontend with OpenSheetMusicDisplay and Tone.js.
|
||||
- Evaluation: an offline P/R/F1 harness on 25 synthesized fixtures, with no scores published.
|
||||
- Licensing: the sidebar says MIT but the README badge says "Proprietary". The premium engraver is closed.
|
||||
- Activity: about 381 stars.
|
||||
- [GitHub](https://github.com/Oh-Sheet-Team/oh-sheet)
|
||||
- **StemStudio HF Space:** Basic Pitch, music21 and Verovio, with a playback cursor on the score — [HF Space](https://yair-bab3-stemstudio-chords.hf.space/)
|
||||
- **Other HF apps:** one chains Demucs and Basic Pitch into multi-track MIDI with a librosa fallback, but was showing a runtime error. Another uses Demucs, Basic Pitch and LilyPond with .ly or MusicXML export — [search summary; e.g. Voice-To-MIDI Space](https://huggingface.co/spaces/mrblackdev/Voice-To-MIDI-VTM/blob/main/app.py); [roo-score static Space](https://kaidjuric-roo-score.static.hf.space/index.html). Not individually verified.
|
||||
- **n8n community proposal:** yt-dlp → Demucs → Basic Pitch per stem → MIDI, MusicXML and PDF using the MuseScore CLI. It is a plan, not a shipped project — [n8n forum](https://community.n8n.io/t/looking-to-build-an-ai-powered-music-transcription-workflow-in-n8n-youtube-multi-instrument-sheet-music/299515)
|
||||
- **GitHub topic page:** music-transcription projects, including a YouTube link → PDF score using Basic Pitch — [GitHub topic](https://github.com/topics/music-transcription)
|
||||
- Basic Pitch is Apache-2.0 — [PyPI](https://pypi.org/project/basic-pitch)
|
||||
|
||||
### Inferences
|
||||
- The components that can be reused in the browser line up with what server-side projects use: Demucs ONNX (WebGPU), Basic Pitch ONNX/TS, and Verovio (which has a WASM/JS build) or OpenSheetMusicDisplay for rendering MusicXML. In principle, the whole oh-sheet-style piano chain could run client-side.
|
||||
- oh-sheet's arrangement and humanize steps and its evaluation harness are the most useful pieces to study. Check the license before reusing any code.
|
||||
|
||||
### Gaps
|
||||
- No open-source end-to-end guitar-tab pipeline (audio → playable tab with string and fret assignment) was found.
|
||||
- No open-source end-to-end drum pipeline was found.
|
||||
- Quality of the HF Spaces was not tested.
|
||||
- The latest commit date for oh-sheet was not visible; one listing said its last push was April 2026.
|
||||
@@ -0,0 +1,224 @@
|
||||
# Open-source Automatic Drum Transcription (ADT): state as of Oct 2026
|
||||
|
||||
Research date: 2026-10-30. About 18 tool calls were made, so coverage is uneven. Some important primary pages could not be retrieved: the ADTOF MDPI paper returned 403, the Omnizart docs returned 404, and the YourMT3 README has no drum details. Claims that come from background knowledge and were **not verified** in this session are marked [UNVERIFIED].
|
||||
|
||||
## 1. Best open models/tools: classes, accuracy, license, maintenance, compute
|
||||
|
||||
### Takeaway
|
||||
No single open model wins outright.
|
||||
- **For drum-only notation that you can use commercially**, the best options are the 2026 synthetic-data model ADT_STR (CC BY-SA 4.0, 26 GM-derived classes, weights on HF) and Magenta OaF-Drums (velocity-aware, trained on E-GMD).
|
||||
- **ADTOF** is the best-known model trained on real music, but it has only 5 classes and is non-commercial.
|
||||
- **The newest research SOTA** is Noise-to-Notes (diffusion, ~50M params). Its code/weight availability is unconfirmed.
|
||||
- **Multi-instrument models** (MuScriptor 2026, YourMT3+) are much weaker on drums: about 41–50 onset F1.
|
||||
|
||||
### Cited Findings
|
||||
|
||||
**ADTOF (Zehren et al.)**
|
||||
- The repository content is CC BY-NC-SA 4.0, which means non-commercial use only. No separate code or weights license is stated. — [GitHub MZehren/ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- Its targets are "5 classes". It ships pre-trained models, including "ADTOF2", which is used in the demo video. — [GitHub MZehren/ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- The datasets total 359 h of crowdsourced (rhythm-game) data. They are distributed only as mel spectrograms, on request via Zenodo. — [GitHub MZehren/ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- Maintenance signals are low:
|
||||
- the repo has only 8 commits;
|
||||
- the latest noted change added a General MIDI instrument mapping (`instrumentsMapping.py`);
|
||||
- it was tested on macOS Ventura with Python 3.10 (TF/Keras + madmom).
|
||||
— [GitHub MZehren/ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- An **ADTOF-pytorch** port exists. It needs only PyTorch (no TF/Keras/madmom) and loses about 0.2% F-measure compared with the original. — [GitHub MZehren/ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- In the N2N paper's comparison, ADTOF scores:
|
||||
- 86.87 F1 on MDB-Drums and 94.67 F1 on IDMT, both real or real-ish data;
|
||||
- only 44.95 on E-GMD, which shows a domain mismatch with electronic-kit synthesized audio;
|
||||
- no velocity prediction.
|
||||
— [Noise-to-Notes arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
|
||||
**Noise-to-Notes (N2N), Sept 2025, v2 Mar 2026**
|
||||
- N2N treats ADT as conditional diffusion. It is a transformer decoder (EDGE-style, FiLM-conditioned) that takes log-mel plus MERT-330M layer-10 features. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- It uses 7 classes on E-GMD: kick, snare, tom, hi-hat, crash, ride, bell. It uses 5 classes on MDB and 3 on IDMT. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- Onset F1 results:
|
||||
|
||||
| Model | E-GMD | E-GMD velocity-F1 | MDB | IDMT |
|
||||
|---|---|---|---|---|
|
||||
| N2N, 10-step | 89.68 | 82.80 | 87.86 | 94.90 |
|
||||
| hFT-Transformer | 86.31 | 80.16 | 70.81 | 70.61 |
|
||||
| OaF Drums | 83.40 | 61.70 | 75.19 | 85.72 |
|
||||
| ADTOF | 44.95 | – | 86.87 | 94.67 |
|
||||
|
||||
— [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- Size and speed:
|
||||
- about 50M parameters, plus the 330M MERT feature extractor;
|
||||
- one sampling step takes 0.163 s per 5 s clip on an A100;
|
||||
- performance saturates at about 10 steps.
|
||||
— [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- The paper text I retrieved states no code, weights or license. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
|
||||
**ADT_STR, "Towards Realistic Synthetic Data for ADT" (Melucci, Merialdo, Akama; arXiv Jan 2026)**
|
||||
- The model is trained only on synthetic audio. That audio is rendered from MIDI using one-shot samples mined semi-supervised from unlabeled audio. It is a seq2seq model. — [arXiv 2601.09520](https://arxiv.org/abs/2601.09520)
|
||||
- It claims new SOTA on the ENST and MDB test sets, beating fully supervised methods. — [arXiv 2601.09520](https://arxiv.org/abs/2601.09520)
|
||||
- The code is CC BY-SA 4.0, so commercial use is allowed with share-alike.
|
||||
- It maps the 47 GM percussion notes onto **26 classes**, merging, for example, the crash cymbals together.
|
||||
- Three checkpoints are released (tau 0.4/0.6/0.8, safetensors), and tau-0.8 is the best. Weights are on HF as `Pierfrancesco/adt-str`. The repo is at v1.0.0.
|
||||
|
||||
— [GitHub pier-maker92/ADT_STR](https://github.com/pier-maker92/ADT_STR)
|
||||
|
||||
**MuScriptor (Rouard, Krause, Roebel, Simon-Gabriel, Défossez; ISMIR 2026 camera-ready; arXiv Jul 2026)**
|
||||
- It is an open-weight multi-instrument model in 4 sizes: 60M, 100M, 300M and 1.3B (the paper version). — [arXiv 2607.08168](https://arxiv.org/html/2607.08168)
|
||||
- Drums are "onset-only". Drums onset F1 is 50.2 for MuScriptor vs 41.4 for YourMT3+. Per dataset, results are mixed: RWC-P 42.1 vs 36.5, RWC-C 23.7 vs 8.7, RWC-R 36.5 vs 51.3. — [arXiv 2607.08168](https://arxiv.org/html/2607.08168)
|
||||
- Licensing and release:
|
||||
- the code is MIT, but the weights are **CC BY-NC 4.0 (non-commercial)**;
|
||||
- the released sizes are small 103M, medium 307M (the default) and large 1.4B;
|
||||
- small is described as practical on CPU, while large "really wants a GPU";
|
||||
- it exports a separate drum-kit PDF through MuseScore 4.
|
||||
— [GitHub muscriptor/muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
|
||||
**YourMT3 / YourMT3+**
|
||||
- The repo license is GPL-3.0, and the weights license is not stated separately. An HF Spaces demo appeared in Aug 2024, and the last dated news item is Nov 2024. — [GitHub mimbres/YourMT3](https://github.com/mimbres/YourMT3)
|
||||
|
||||
**MT3 family**
|
||||
- The MT3 design does not model velocity. This is a secondary source (a student review). — [baulab review](https://expo.baulab.info/2022-Fall/PraveenKumarSridhar/second_review.html)
|
||||
|
||||
**Magenta OaF-Drums / E-GMD**
|
||||
- The E-GMD dataset is CC BY 4.0. — [Magenta OaF Drums](https://magenta.tensorflow.org/oaf-drums)
|
||||
- In 2025 comparisons, OaF Drums still reaches 83.4 F1 on E-GMD and 75.2 on MDB. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
|
||||
**Other**
|
||||
- A momentum-NMF ADT method (Jul 2025) offers a classical, interpretable alternative. — [arXiv 2507.12596](https://arxiv.org/abs/2507.12596v1)
|
||||
- The STAR Drums dataset (Fraunhofer, 2025) was built from full mixes by separating the drum stem and auto-annotating it with an ADT system. — [Fraunhofer publica](https://publica.fraunhofer.de/handle/publica/499927)
|
||||
- A LinkedIn post says "Separate-and-Detect", a latent-diffusion drum transcription paper, was accepted at ISMIR 2026. This is unverified and secondary. — [LinkedIn](https://tw.linkedin.com/in/mia-chen-1788511a9)
|
||||
|
||||
### Inferences
|
||||
- **Practical choices for a drum-tab product:**
|
||||
- (a) ADT_STR, the only one found with a commercially usable license plus GM-level class granularity;
|
||||
- (b) ADTOF/ADTOF-pytorch, which is robust on real recordings but non-commercial and has only 5 classes;
|
||||
- (c) OaF-Drums, if velocity matters (license not checked here, but Magenta code is typically Apache-2.0 [UNVERIFIED]).
|
||||
- **ADTOF's 5 classes** are, to my knowledge, kick / snare / toms / hi-hat / cymbals+ride, and do not separate open from closed hi-hat [UNVERIFIED; the paper page was blocked].
|
||||
- **Omnizart drum** (MIT code) and the **madmom** drum models (BSD code; madmom model files are, as I recall, CC BY-NC-SA) are older baselines from 2018–2021, mostly kick/snare/hi-hat. Neither appears as a competitive baseline in 2025–26 papers [UNVERIFIED; docs 404].
|
||||
- **Multi-instrument models** are not adequate on their own for drum notation, because drum F1 is about 40–50. Use them only if the product already needs full-band transcription.
|
||||
|
||||
### Gaps
|
||||
- Exact ADTOF class list and MIDI mapping, per-class F1, and the license of model weights vs dataset (the MDPI paper was 403).
|
||||
- ADT_STR numeric F1 values (the README gives no numbers in the retrieved text) and the HF weight license.
|
||||
- Whether N2N code/weights are released.
|
||||
- Omnizart drum class count and maintenance status (docs 404). The Omnizart project appears dormant since about 2022 [UNVERIFIED].
|
||||
- No "Drumscribe" or "DrumTranscriber" open-source projects were found in searches. I did not search for them specifically.
|
||||
- No MIREX drum transcription results from 2024–2026 were found.
|
||||
|
||||
## 2. Per-class performance (hi-hats/cymbals weakest?)
|
||||
|
||||
### Takeaway
|
||||
Yes. The 2025 ISMIR diagnostic study confirms that toms and cymbals are the weakest classes. In drum-only audio, the main error source is simultaneous hits. In full mixes, it is interference from melodic instruments.
|
||||
|
||||
### Cited Findings
|
||||
- Weyers, Uhle, Müller, Lang (ISMIR 2025): toms and cymbals are the classes where SOTA models perform worst. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
- Transcription of drums within full mixes is limited mainly by melodic instruments and vocals. Performance improves by about 5% with a single kit, strong onsets only, or no simultaneous notes. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
- Drum-only transcription becomes nearly error-free once simultaneous onsets are removed. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
- A reviewer comment notes that hi-hat and ride patterns alternate weak and strong strokes, which makes the weak strokes hard to detect. The findings are mostly on re-synthesized audio, and MDB, the real test set, is only about 22 min. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
- ADT_STR publishes confusion matrices for its 26 classes on ENST and MDB. — [GitHub ADT_STR](https://github.com/pier-maker92/ADT_STR)
|
||||
|
||||
### Inferences
|
||||
- Expect kick and snare to be reliable. Expect hi-hat open/closed, crash vs ride vs china, and toms (which tom) to need manual correction.
|
||||
- Expect ghost notes on the snare and weak hi-hat strokes to be the most commonly missed onsets.
|
||||
|
||||
### Gaps
|
||||
- No per-class F1 table was retrieved for any current model.
|
||||
|
||||
## 3. Does pre-separation (Demucs drum stem, LarsNet, DrumSep) help?
|
||||
|
||||
### Takeaway
|
||||
There is indirect evidence that removing melodic interference helps, since it is the main limiting factor in full mixes. I found no controlled study showing that LarsNet or DrumSep sub-stem separation improves ADT F1.
|
||||
|
||||
### Cited Findings
|
||||
- In full mixes, the main limiting factor is melodic and vocal interference. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
- The STAR Drums pipeline separates the drum stem from full mixes and then annotates it with ADT. This shows the "separate then transcribe" pattern in use. — [Fraunhofer publica](https://publica.fraunhofer.de/handle/publica/499927)
|
||||
- LarsNet uses 5 parallel U-Nets, one per stem: kick, snare, toms, hi-hat, cymbals. It is trained on StemGMD and beats NMF/NMFD baselines. — [arXiv 2312.09663](https://arxiv.org/html/2312.09663v1)
|
||||
- The LarsNet paper treats ADT only as a related task and reports no ADT gains. — [arXiv 2312.09663](https://arxiv.org/html/2312.09663v1)
|
||||
- The LARS VST plug-in weights are CC BY-NC 4.0. — [GitHub EdoardoMor/LARS](https://github.com/EdoardoMor/LARS)
|
||||
- MVSEP says LarsNet separation quality is inferior to its DrumSep model, and that it could not reproduce the authors' metrics. — [MVSEP](https://mvsep.com/algorithms/31?lang=en)
|
||||
- In practice, some users isolate the drum track with Moises before analysis. — [Penn State Brandywine](https://www.brandywine.psu.edu/node/28566)
|
||||
|
||||
### Inferences
|
||||
- Use a full-drum-stem pre-pass (Demucs/HTDemucs) when the ADT model was trained on isolated drums, such as E-GMD-trained OaF or N2N. It is likely to help, given the interference finding.
|
||||
- ADTOF was trained on full mixes, so it may gain less.
|
||||
- Sub-stem separation adds compute, and its artifacts (bleed) may create false onsets. Treat it as an experiment, not a default.
|
||||
|
||||
### Gaps
|
||||
- No quantitative ADT-with/without-separation results were found.
|
||||
- DrumSep's architecture and license were not confirmed.
|
||||
|
||||
## 4. Velocity, ghost notes, fills
|
||||
|
||||
### Takeaway
|
||||
Velocity is only modeled by OaF-Drums, hFT-Transformer and N2N. ADTOF, DT-Ensemble, MT3 and MuScriptor are onset-only, so ghost notes and accents must be inferred or are lost.
|
||||
|
||||
### Cited Findings
|
||||
- ADTOF and DT-Ensemble do not predict velocity. Velocity F1 on E-GMD is N2N 82.8, hFT 80.2 and OaF 61.7. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- N2N's Annealed Pseudo-Huber loss raised velocity F1 on E-GMD from 66.1 to 76.2. — [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2)
|
||||
- MuScriptor's drum hits are "onset-only". — [arXiv 2607.08168](https://arxiv.org/html/2607.08168)
|
||||
- Weak (ghost-like) strokes and simultaneous hits are a key error source. — [ISMIR 2025 P5-10](https://ismir2025program.ismir.net/poster_130.html)
|
||||
|
||||
### Inferences
|
||||
- For ghost-note notation, a velocity threshold on OaF or N2N output (for example, velocity below about 40 drawn as a parenthesized notehead) is the only data-driven path.
|
||||
- With onset-only models, you could estimate velocity from onset peak amplitude as a heuristic.
|
||||
- Fills (dense tom runs) combine the two weakest factors: tom classification and simultaneous hits. Expect degraded accuracy there.
|
||||
|
||||
### Gaps
|
||||
- No study specifically evaluates ghost-note or fill accuracy.
|
||||
|
||||
## 5. Converting ADT output to notation (quantization, GM map, GP/MusicXML)
|
||||
|
||||
### Takeaway
|
||||
Most open models output GM-pitched MIDI or class onsets. Building notation requires beat tracking, quantization, meter inference and a percussion map. MIDI-to-score drum work exists, but tooling here is thin.
|
||||
|
||||
### Cited Findings
|
||||
- ADTOF added a General MIDI instrument mapping file. — [GitHub ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- ADT_STR classes follow General MIDI percussion, consolidated from 47 to 26. — [GitHub ADT_STR](https://github.com/pier-maker92/ADT_STR)
|
||||
- MuScriptor renders a drum-kit PDF through MuseScore 4. — [GitHub muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
- An academic work on "MIDI To Score Automated Drum Transcription" (HAL, 2022) addresses the MIDI-to-score step. — [HAL hal-03847232](https://hal.archives-ouvertes.fr/hal-03847232)
|
||||
- Commercial tools ask users for time signature and BPM to improve results. — [Songscription blog](https://www.songscription.ai/blog/transcribe-drums-with-ai)
|
||||
- Commercial drum-convert services document manual correction of errors. — [La Touche Musicale help](https://latouchemusicale.crunch.help/en/drum-convert/errors-in-the-transcribed-drum-sheet-music-what-to-do)
|
||||
|
||||
### Inferences
|
||||
- **Suggested pipeline:** ADT onsets plus classes → beat/downbeat tracking (e.g. madmom or Beat This [UNVERIFIED]) → grid quantization (16th or 32nd notes, triplet detection) → GM drum pitches.
|
||||
- **MusicXML percussion output:** a `<unpitched>` display-step/octave plus `<instrument>` per GM note. Typical choices are x noteheads for cymbals and hi-hat, and stems up for hands and down for feet.
|
||||
- **Guitar Pro drum track output:** GP uses GM percussion numbers directly, for example via alphaTab/alphaTex or PyGuitarPro [UNVERIFIED].
|
||||
- **Open vs closed hi-hat:** ADTOF's merged classes would need to map to a single hi-hat, typically 42.
|
||||
|
||||
### Gaps
|
||||
- No verified sources were retrieved on Guitar Pro or MusicXML percussion-map specifics, or on alphaTab drum import/export.
|
||||
|
||||
## 6. Commercial references and user sentiment
|
||||
|
||||
### Takeaway
|
||||
User evidence is sparse and anecdotal. Songsterr's AI drum transcription gets the most positive mentions. Drum2Notes (Klangio) got negative early feedback. All tools are treated as drafts needing correction.
|
||||
|
||||
### Cited Findings
|
||||
- An r/DRUM user (Jul 2024) tried the Drum2Notes trial and found it poor even on isolated drums with a simple beat. — [Reddit r/DRUM](https://nyc1.lr.ggtyler.dev/r/DRUM/comments/1e5qmzg/ai_that_translates_audio_into_sheet_music)
|
||||
- In the same thread, a user recommended Songsterr's AI as "very good". — [Reddit r/DRUM](https://nyc1.lr.ggtyler.dev/r/DRUM/comments/1e5qmzg/ai_that_translates_audio_into_sheet_music)
|
||||
- The Drum2Notes App Store listing warns that it cannot separate multiple instruments. — [App Store](https://apps.apple.com/app/id6717586517)
|
||||
- On the BassBuzz forum, users called Songsterr AI "dead on accurate", while others said it was "mostly hit with some miss". These comments concern bass, not necessarily drums. — [BassBuzz forum](https://forum.bassbuzz.com/t/new-songsterr-ai-is-impressive/70519)
|
||||
- Moises is primarily a stem separator. No user reviews of its drum-transcription accuracy were found. — [Penn State Brandywine](https://www.brandywine.psu.edu/node/28566)
|
||||
|
||||
### Gaps
|
||||
- No 2025–2026 user reviews were found. The Drum2Notes complaint is from 2024, so the product may have improved.
|
||||
- Nothing was found on "Drumscrib".
|
||||
|
||||
## 7. Browser deployment (ONNX/TF.js), model sizes
|
||||
|
||||
### Takeaway
|
||||
No ADT project found ships ONNX or TF.js builds. The small CRNN models (ADTOF, OaF) are the realistic browser candidates. N2N (50M plus 330M MERT) and MuScriptor (100M–1.4B) are server-side.
|
||||
|
||||
### Cited Findings
|
||||
- The ADTOF README makes no mention of ONNX or TF.js. — [GitHub ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- The PyTorch port could be exported with `torch.onnx`. — [GitHub ADTOF](https://github.com/MZehren/ADTOF)
|
||||
- Model sizes:
|
||||
- hFT-Transformer: 5.5M parameters;
|
||||
- N2N: about 50M, plus the MERT-330M feature extractor;
|
||||
- MuScriptor: 103M, 307M and 1.4B (small is CPU-practical).
|
||||
— [arXiv 2509.21739v2](https://arxiv.org/html/2509.21739v2); [GitHub muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
- ADT_STR ships safetensors weights. — [GitHub ADT_STR](https://github.com/pier-maker92/ADT_STR)
|
||||
|
||||
### Inferences
|
||||
- **Browser path:** ADTOF-pytorch → ONNX → onnxruntime-web (WASM/WebGPU), with the mel front-end reimplemented in JS. ADTOF CRNNs are roughly a few MB [UNVERIFIED size].
|
||||
- **Browser path for ADT_STR:** feasible only if the seq2seq model is small, which was not verified.
|
||||
- **License caution:** ADTOF and MuScriptor weights are non-commercial. Check this before shipping them in a product.
|
||||
|
||||
### Gaps
|
||||
- Parameter counts for ADTOF, OaF-Drums and ADT_STR were not verified.
|
||||
- No existing web demos running ADT client-side were found.
|
||||
@@ -0,0 +1,265 @@
|
||||
# Guitar and bass audio-to-tablature transcription (state as of Oct 2026)
|
||||
|
||||
Scope note: this covers about 17 search and fetch calls. Items marked "[unverified, prior knowledge]" were not checked in this session. They are listed only under Inferences or Gaps, never as cited findings. Several key arXiv items are dated Aug–Sep 2026 (IDs 2608.x and 2609.x), which fits today's date of 2026-10-30.
|
||||
|
||||
## Q1. Audio-to-tab models and SOTA (TabCNN, FretNet, Riley/high-res, GAPS, SynthTab, GOAT, TART, diffusion); electric vs. clean
|
||||
|
||||
### Takeaway
|
||||
The strongest documented 2026 system is TART (Berkeley; arXiv 2510.02597 in Oct 2025, then a revised "modular" version, arXiv 2609.11904, in Sep 2026, submitted to ISMIR 2026). It is a 4-stage pipeline: audio→MIDI, technique classifier, audio-conditioned T5 string/fret assignment, and tab rendering. It reports 81.35% audio-to-MIDI F50 and 71.8% Tab F1 with oracle MIDI. End to end it gets only 54.08% Tab F1. That means fully automatic tabs are still roughly half right at the string+fret+onset level. Electric and distorted guitar is still the weak spot. Newer data (GOAT, Guitar-TECHS, EGDB, real-tone augmentation) targets it, but no paper reports a dedicated high-gain or distorted benchmark.
|
||||
|
||||
### Cited Findings
|
||||
- **TART (2609.11904v1, submitted 10 Sep 2026)**
|
||||
- Stage 1 is a high-res CRNN based on Kong et al., fine-tuned on pooled guitar data with noise augmentation.
|
||||
- Stage 2 is a CNN-BiLSTM technique classifier with about 160K parameters.
|
||||
- Stage 3, "AudioFret", is a T5-style encoder-decoder with about 15M parameters. It adds per-note audio embeddings to the MIDI tokens and uses beam search 4 with a chord fret-span limit of 5.
|
||||
- Stage 4 uses BeatNet tempo and a 1/16 grid, and outputs **MusicXML + JAMS**. Guitar Pro export is not mentioned.
|
||||
- — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **TART results**
|
||||
- Audio→MIDI F50: 81.35% average, vs. 74.68% for Riley et al. and 69.88% for NoteEM. Per dataset: GuitarSet 87.4, EGDB 79.0, GuitarSet-noisy 82.2, EGDB-noisy 76.8.
|
||||
- String/fret Tab F1 with oracle MIDI: 71.8%, vs. 63.3% for the Fretting-Transformer. TabCNN averages 27.9% on EGDB.
|
||||
- End-to-end Tab F1: 54.08%. The paper attributes a 17.75-point "propagation cost" to Stage-1 errors.
|
||||
- — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **TART training data**
|
||||
- Stage 1: GAPS, Guitar-TECHS, François Leduc, GOAT-DI.
|
||||
- Stage 3: pretrained on SynthTab + DadaGP, then fine-tuned on GAPS, GOAT-DI, Guitar-TECHS.
|
||||
- Evaluation: GuitarSet, EGDB, and noisy variants of both. There is no separate electric-only evaluation.
|
||||
- — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **TART code**: no code repo was found. The arXiv page has a "Code, Data, Media" tab, but no link surfaced. The paper text is CC BY 4.0. — [arXiv 2609.11904](https://arxiv.org/pdf/2609.11904); [arXiv 2510.02597](https://arxiv.org/abs/2510.02597)
|
||||
- **Noise2Fret** (Simionato & Bigo, arXiv 2608.30854, 31 Aug 2026, accepted to ISMIR 2026)
|
||||
- A diffusion model that generates tab through a continuous latent of string/fret targets, conditioned on spectral features.
|
||||
- Adds five playability losses: pitch-class distance, positional distance, circle-of-fifths distance, string similarity, hand-span feasibility.
|
||||
- Evaluated on GuitarSet and GOAT. The abstract claims it beats baselines and is more efficient. No numbers or code are on the abstract page.
|
||||
- — [arXiv 2608.30854](https://arxiv.org/abs/2608.30854)
|
||||
- **GOAT** (Loth, Sarmento, Sarkar, Guo, Barthet, Sandler; ISMIR 2025)
|
||||
- 5.9 h of unique DI electric guitar recordings, amp-augmented to 29.5 h.
|
||||
- Tabs with string/fret and techniques, supplied in **Guitar Pro format** plus a token encoding.
|
||||
- License CC BY 4.0 (per the arXiv license link).
|
||||
- Reports "competitive" MIDI transcription and only preliminary tab results.
|
||||
- — [arXiv 2509.22655](https://arxiv.org/abs/2509.22655)
|
||||
- **High Resolution Guitar Transcription via Domain Adaptation** (Riley et al., arXiv 2402.15258, 2024; older)
|
||||
- Onset-only F50 on GuitarSet: 89.7 when supervised on Leduc + GuitarSet, and 87.3 zero-shot when trained only on Leduc-derived data. The zero-shot figure is more than 4 points above Maman & Bermano.
|
||||
- This is notes only, not string assignment.
|
||||
- The Leduc data is commercial and not free.
|
||||
- — [arXiv 2402.15258](https://arxiv.org/pdf/2402.15258); [GAPS](https://arxiv.org/html/2408.08653v2)
|
||||
- **GAPS** (ISMIR 2024): 14 h of classical (nylon) guitar audio aligned to scores. It is called the largest real-guitar dataset, several times the size of GuitarSet. Its benchmark model is claimed SOTA on GuitarSet in both supervised and zero-shot settings. — [ISMIR 2024 poster](https://ismir2024program.ismir.net/poster_171.html); [arXiv 2408.08653](https://arxiv.org/abs/2408.08653)
|
||||
- **SynthTab** (Zang, Zhong, Cwitkowitz, Duan; arXiv 2309.09085; older, 2023–24)
|
||||
- Synthesized audio rendered from DadaGP tablature, which covers 26,181 songs.
|
||||
- Pre-training TabCNNx4 on it improves same-dataset accuracy and especially cross-dataset generalization.
|
||||
- — [arXiv 2309.09085](https://arxiv.org/pdf/2309.09085)
|
||||
- **Electric robustness**: Leveraging Real Electric Guitar Tones and Effects (arXiv 2405.14679, 2024) augments TabCNN training with real electric tones and effects to improve robustness. Guitar-TECHS (arXiv 2501.03720) is an electric dataset spanning techniques and hardware. — [arXiv 2405.14679](https://arxiv.org/html/2405.14679v3); [arXiv 2501.03720](https://arxiv.org/pdf/2501.03720)
|
||||
- **Survey**: a 2025 IEEE Access journal paper, "Automatic Guitar Transcription With Deep Neural Networks" (Chieppa et al., Univ. Coimbra), exists and could serve as background. Not read in detail. — [PDF](https://mir.dei.uc.pt/pdf/Journals/MERGE/Access_2025_Chieppa.pdf)
|
||||
|
||||
### Inferences
|
||||
- A practical 2026 architecture is (a) source separation → (b) a strong guitar note transcriber (Riley/GAPS-style high-res CRNN) → (c) a learned string/fret assigner conditioned on tuning and capo → (d) technique tagging → (e) beat/tempo quantization → GP export. That is the TART decomposition.
|
||||
- End-to-end Tab F1 of about 54% on research benchmarks with clean or lightly noisy, mostly solo guitar means full-mix, distorted rock and metal will be markedly worse. Expect manual correction.
|
||||
- [unverified, prior knowledge] Relevant code: TabCNN (Wiggins & Kim, ISMIR 2019); FretNet and the Cwitkowitz repos (github.com/cwitkowitz/guitar-transcription-continuous, guitar-transcription-with-inhibition, amt-tools, all MIT-licensed as I recall); the Riley/GAPS model (github.com/xavriley). These are the realistic open-source starting points, because no TART code was found.
|
||||
|
||||
### Gaps
|
||||
- No TART or Noise2Fret code or weights were found. Licenses for any released models are unknown.
|
||||
- No paper found that benchmarks high-gain or distorted full-mix rock or metal tabs end to end.
|
||||
- Noise2Fret numbers are not retrieved. Only the abstract was read.
|
||||
- Licenses for GuitarSet, EGDB, and Guitar-TECHS were not verified this session.
|
||||
|
||||
## Q2. MIDI/notes → tab (string/fret assignment) algorithms and tools
|
||||
|
||||
### Takeaway
|
||||
Learned seq2seq assigners now beat both A* and the built-in Guitar Pro and TuxGuitar heuristics on hard material (jazz, DadaGP). On easy fingerstyle, simple heuristics are nearly as good. Open code exists for some newer work (midi2tab_Deconvolutions). The Fretting-Transformer and TART code links were not found.
|
||||
|
||||
### Cited Findings
|
||||
- **Fretting-Transformer** (Hamberger, Murgul, Schmidt, Heizmann; arXiv 2506.14223, June 2025)
|
||||
- A reduced T5 (d_model 128, 3+3 layers, 4 heads), with post-processing that forces 100% pitch accuracy.
|
||||
- Tab accuracy on GuitarToday / Leduc / DadaGP:
|
||||
|
||||
| Method | GuitarToday | Leduc | DadaGP |
|
||||
|---|---|---|---|
|
||||
| Fretting-Transformer | 98.41% | 72.19% | 81.58% |
|
||||
| A* | 89.39% | 62.60% | 78.92% |
|
||||
| Guitar Pro | 97.58% | 56.03% | 76.58% |
|
||||
| TuxGuitar | 98.30% | 57.94% | 78.83% |
|
||||
|
||||
- — [arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)
|
||||
- **Fretting-Transformer conditioning and caveats**
|
||||
- A conditioned variant takes CAPO and TUNING tokens, trained with capo 0–7 and standard, ½-step down, whole-step down, and drop D.
|
||||
- The model has higher (worse) difficulty scores than TuxGuitar on Leduc and DadaGP.
|
||||
- In the conditioned GuitarToday setting, a trivial lowest-fret baseline (98.30%) slightly beats the model (98.19%).
|
||||
- No code link was found. The paper is CC BY 4.0.
|
||||
- — [arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)
|
||||
- **MIDI-to-Tab** (Edwards et al., ISMIR 2024): a masked-language-model Transformer for string assignment. — [ISMIR 2024 poster](https://ismir2024program.ismir.net/poster_175.html); [arXiv 2408.05024](https://arxiv.org/html/2408.05024v1)
|
||||
- **midi2tab_Deconvolutions** (Kaliakatsos-Papakostas et al., arXiv 2510.10619, Oct 2025): public code at github.com/maximoskp/midi2tab_Deconvolutions. Assumes standard 6-string tuning and 24 frets. — [arXiv 2510.10619](https://arxiv.org/html/2510.10619v1)
|
||||
- **AudioFret**: TART's audio-conditioned assigner gets 71.8% vs. 63.3% Tab F1 for the Fretting-Transformer on GuitarSet/EGDB. Using audio timbre per note helps disambiguate strings. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **Classical approach**: a weighted DAG plus A* over string/fret candidates to find playable sequences. It is the standard baseline. — [arXiv 2510.02597](https://arxiv.org/pdf/2510.02597)
|
||||
|
||||
### Inferences
|
||||
- For a website, a cost-based DP/Viterbi assigner (fret span, position shifts, open-string preference, tuning/capo aware) is a cheap, license-clean baseline. It can be swapped later for a T5-style assigner trained on DadaGP/GOAT, which the Fretting-Transformer paper shows is about 98% on easy tabs and 72–82% on hard ones.
|
||||
- Benchmarking against hand-made GP tabs will penalize "valid but different" fingerings. See the Q8 metrics.
|
||||
|
||||
### Gaps
|
||||
- I did not check whether a usable open-source "tabify" library exists (MuseScore's tab conversion, TuxGuitar's algorithm as a library). The Fretting-Transformer paper treats TuxGuitar and Guitar Pro only as black-box baselines.
|
||||
|
||||
## Q3. Playing-technique detection
|
||||
|
||||
### Takeaway
|
||||
Technique classification is reasonably accurate at the note level on curated datasets. TART reports 95.9% macro F1 over 9 classes, with vibrato hardest. Datasets with technique labels exist: Guitar-TECHS, GOAT, EG-IPT, AGPT, IDMT-SMT.
|
||||
|
||||
### Cited Findings
|
||||
- **TART per-class F1** on a unified test set:
|
||||
- Palm mute 97.9, hammer-on/pull-off 97.4, picking 98.0
|
||||
- Harmonics 94.9, slide 94.7, bend 93.8, vibrato 89.4 (often confused with bend)
|
||||
- Macro F1 95.9
|
||||
- Training data: IDMT-SMT-Chords, Guitar-TECHS, AGPT, EG-IPT, Magcil.
|
||||
- These are classifier scores, not end-to-end tab-with-technique scores.
|
||||
- — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **GOAT** tabs annotate playing techniques. — [arXiv 2509.22655](https://arxiv.org/abs/2509.22655)
|
||||
|
||||
### Inferences
|
||||
- Bends need pitch-contour output (bend amount and shape), not just a class label, to map onto GP bend points. Continuous-pitch models such as FretNet [unverified, prior knowledge] are relevant here.
|
||||
|
||||
### Gaps
|
||||
- No end-to-end technique F1 on full songs was found. No open technique-detection model weights were found.
|
||||
|
||||
## Q4. Tuning detection and capo inference
|
||||
|
||||
### Takeaway
|
||||
Current research systems take tuning and capo as user input (TART, Fretting-Transformer) instead of inferring them. No automatic tuning or capo detector was found.
|
||||
|
||||
### Cited Findings
|
||||
- TART assumes standard tuning by default, and the user can specify tuning, capo, and tempo. It was trained on standard, ½-step down, whole-step down, and drop D, with capo 0–7. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- The Fretting-Transformer uses CAPO/TUNING conditioning tokens with the same four tunings. — [arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)
|
||||
- midi2tab_Deconvolutions assumes standard 6-string, 24-fret. — [arXiv 2510.10619](https://arxiv.org/html/2510.10619v1)
|
||||
|
||||
### Inferences
|
||||
- Down-tuning can be partly inferred from the lowest detected pitch: notes below E2 imply drop or down-tuning, or 7-string. Global detuning (for example A=432 or ½-step) can be inferred from a pitch-deviation histogram.
|
||||
- Capo is mostly unobservable from audio. A UI toggle plus "minimize fret positions" search is the pragmatic answer.
|
||||
- 7-string and 8-string support is absent from the papers found.
|
||||
|
||||
### Gaps
|
||||
- No published model was found for tuning, capo, or extended-range detection.
|
||||
|
||||
## Q5. Distorted electric guitar; training on DadaGP-rendered or synth audio
|
||||
|
||||
### Takeaway
|
||||
Synthetic rendering of DadaGP (SynthTab) and real-amp augmentation (2405.14679, GOAT amp renders) are the main strategies. Distortion remains under-evaluated.
|
||||
|
||||
### Cited Findings
|
||||
- SynthTab renders DadaGP's 26k songs to audio. Pre-training improves cross-dataset results. — [arXiv 2309.09085](https://arxiv.org/pdf/2309.09085)
|
||||
- GOAT amp-augments DI recordings from 5.9 h to 29.5 h. — [arXiv 2509.22655](https://arxiv.org/abs/2509.22655)
|
||||
- Real electric tones and effects augmentation improves TabCNN robustness. — [arXiv 2405.14679](https://arxiv.org/html/2405.14679v3)
|
||||
- TART says prior systems struggle with electric pickups and distortion, but it has no electric-only evaluation. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- Vendors and reviewers all say accuracy is best on clear, solo guitar and degrades on dense mixes and phone recordings. — [Songscription](https://www.songscription.ai/guitar-tab-generator); [agent-finder Klangio review, Aug 2026](https://agent-finder.co/reviews/klangio)
|
||||
|
||||
### Gaps
|
||||
- No numbers were found for high-gain distortion on full mixes.
|
||||
|
||||
## Q6. Bass tab transcription
|
||||
|
||||
### Takeaway
|
||||
No bass-specific research model for string/fret assignment was found in this session. Research is guitar-centric. Commercial tools (Klangio, Songscription, Tabtify, BassConvert) offer bass tabs, but there are no published accuracy numbers.
|
||||
|
||||
### Cited Findings
|
||||
- The TART, Noise2Fret, and Fretting-Transformer papers are guitar-only. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1); [arXiv 2608.30854](https://arxiv.org/abs/2608.30854)
|
||||
- **Songscription**: separates bass from the mix, outputs tab plus notation, and offers Guitar Pro export. — [Songscription bass](https://www.songscription.ai/bass-tab-generator)
|
||||
- **Tabtify**: supports 4-, 5-, and 6-string bass. It warns that kick drum, low guitar, and synth bass in dense mixes need cleanup. — [Tabtify](https://tabtify.com/convert/audio-to-bass-tab)
|
||||
- **BassConvert**: offers a faithful "transcription" mode vs. an easier "arrangement" mode. — [BassConvert](https://latouchemusicale.com/en/apps/bassconvert/)
|
||||
- **Klangio**: claims support for acoustic, electric, and bass guitar. — [Klangio Guitar2Tabs](https://klang.io/guitar2tabs/)
|
||||
|
||||
### Inferences
|
||||
- Bass is largely monophonic, so a bass stem plus a monophonic pitch tracker plus a cost-based 4/5-string assigner may be competitive. String ambiguity on bass is mostly between positions.
|
||||
|
||||
### Gaps
|
||||
- No bass tab dataset or benchmark was found. DadaGP contains bass tracks [unverified, prior knowledge].
|
||||
|
||||
## Q7. Guitar Pro file I/O and formats
|
||||
|
||||
### Takeaway
|
||||
Options for each job:
|
||||
- **Writing modern .gp (GP7/8)**: alphaTab ships a Gp7Exporter (since 1.2.0), returning a Uint8Array, so it works in browser or Node. It is the best fit for a website.
|
||||
- **Python**: PyGuitarPro reads and writes only GP3/4/5 and is LGPL-3.0. A newer PyPI package, ApolloTab, claims GP3–GP8/GPX support under LGPL-2.1 (unverified).
|
||||
- **Datasets**: GOAT ships Guitar Pro files.
|
||||
|
||||
### Cited Findings
|
||||
- **alphaTab export**
|
||||
- `alphaTab.exporter.Gp7Exporter` writes Guitar Pro 7+ (.gp) files via `exporter.export(api.score, api.settings)`, which returns a Uint8Array. Available since 1.2.0.
|
||||
- An alphaTex exporter was added in 1.7.0.
|
||||
- — [alphaTab exporter guide](https://alphatab.net/docs/guides/exporter); [Gp7Exporter ref](https://alphatab.net/docs/reference/types/exporter/gp7exporter/)
|
||||
- **alphaTab maintenance**: active. A bug where exported drum tracks were silent in GP8 RSE (wrong `<Pitched>` vs. `<Name>` XML element) was fixed in PR #2456, merged Dec 2025. — [alphaTab issue #2467](https://github.com/CoderLine/alphaTab/issues/2467); [PR #2456](https://github.com/CoderLine/alphaTab/pull/2456)
|
||||
- **Community use**: Songsterr Downloader uses alphaTab to convert Songsterr tabs into GP7. — [songsterr-downloader](https://github.com/Metaphysics0/songsterr-downloader)
|
||||
- **PyGuitarPro**: reads and writes GP3, GP4, GP5. LGPLv3. — [PyPI](https://pypi.python.org/pypi/PyGuitarPro); [GitHub](https://github.com/Perlence/PyGuitarPro)
|
||||
- **ApolloTab**: the PyPI listing (via search snippet) claims "full support for GP3/GP4/GP5/GPX/GP (GP7/GP8)" under LGPL 2.1. The page failed to load on fetch, so this is unverified. — [PyPI apollotab](https://pypi.org/project/apollotab/)
|
||||
- **dart_gp_tab_reader**: a Dart port mirroring PyGuitarPro, LGPL-3.0. It notes "Guitar Pro" is a registered trademark of Arobas Music. — [pub.dev](https://pub.dev/packages/dart_gp_tab_reader)
|
||||
- **Output formats**: TART outputs MusicXML + JAMS, not GP. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
|
||||
### Inferences
|
||||
- **Suggested website stack**: a Python ML backend emits an internal score (or alphaTex or MusicXML). The frontend renders it with alphaTab and exports .gp via Gp7Exporter. alphaTab can also import the reference GP files (gp3–gp8) for benchmarking. [unverified, prior knowledge] alphaTab is MPL-2.0, and its importers cover gp3–5, gpx, gp7/8, MusicXML, and alphaTex.
|
||||
- **DadaGP** [unverified, prior knowledge]: tokenizes GP files into a text token format via PyGuitarPro, so it inherits the GP3–5 limit. Converting GP7/8 references to GP5 loses some detail.
|
||||
- **TuxGuitar** [unverified, prior knowledge]: LGPL, Java desktop app, reads and writes GP3–5, and is not ideal as a server library.
|
||||
|
||||
### Gaps
|
||||
- alphaTab's license and its exporter's coverage of techniques (bends, slides, palm mute) were not verified this session.
|
||||
- ApolloTab's maintenance, maturity, and actual GP8 write fidelity are unknown.
|
||||
- The GP7/8 .gp format (zip + "Content/score.gpif" XML) has no official public spec. [unverified, prior knowledge]
|
||||
|
||||
## Q8. Evaluation metrics, and comparing against reference Guitar Pro tabs
|
||||
|
||||
### Takeaway
|
||||
The standard metrics are:
|
||||
- **Multi-pitch and note F1** (onset ±50 ms, "F50"), via mir_eval.
|
||||
- **Tab F1**: a note counts only if pitch, string, fret, and onset all match.
|
||||
- **Tab Disambiguation Rate (TDR)**: string correctness given correct pitch.
|
||||
- **Tab accuracy and difficulty scores** for MIDI→tab.
|
||||
|
||||
### Cited Findings
|
||||
- **TART** reports audio-to-MIDI F50, string/fret Tab F1 with oracle MIDI, and end-to-end Tab F1. It separates out the error propagated from Stage 1, which is 17.75 points. TDR and multipitch F1 are not reported in the version fetched. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1)
|
||||
- **Fretting-Transformer** uses pitch accuracy, tab accuracy against professional tabs, and a modified difficulty (playability) score. — [arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)
|
||||
- **Riley et al.** use onset-only F-measure at 50 ms. — [arXiv 2402.15258](https://arxiv.org/pdf/2402.15258)
|
||||
- **SynthTab** reports both multi-pitch and tablature estimation metrics. — [arXiv 2309.09085](https://arxiv.org/pdf/2309.09085)
|
||||
|
||||
### Inferences
|
||||
- **Proposed benchmark protocol against hand-made GP references**:
|
||||
1. Parse both GP files (alphaTab or PyGuitarPro) to note lists of (onset in beats/seconds, pitch, string, fret, duration, techniques), applying tuning and capo.
|
||||
2. Align tempo or beat grids. Use DTW if the reference has no audio sync.
|
||||
3. Report the following, per track (multiple guitars plus bass need track matching via the Hungarian algorithm on note F1):
|
||||
- (a) note F1 on pitch+onset
|
||||
- (b) Tab F1 on pitch+onset+string+fret
|
||||
- (c) TDR = string-correct / pitch-correct
|
||||
- (d) technique precision and recall per class
|
||||
- (e) rhythm and quantization accuracy at beat level
|
||||
- (f) a playability or difficulty score
|
||||
4. Allow octave- or position-equivalent "acceptable alternative fingering" as a secondary lenient metric. Human tabs often differ in position, as seen in the Fretting-Transformer's Leduc and DadaGP scores in the 72–82% range.
|
||||
- [unverified, prior knowledge] TDR originates in TabCNN-era work (Wiggins & Kim 2019). amt-tools implements tab precision, recall, F1, and TDR.
|
||||
|
||||
### Gaps
|
||||
- No standard public tool was found for "GP-vs-GP" tab diffing.
|
||||
|
||||
## Q9. Commercial tools and user sentiment
|
||||
|
||||
### Takeaway
|
||||
Commercial tools give no numeric accuracy claims. User sentiment is mixed: fine on clean solo guitar, often wrong on self-recorded or full-band audio. Reddit threads did not surface in searches, so sentiment comes from Trustpilot, app stores, and reviews (some vendor-affiliated).
|
||||
|
||||
### Cited Findings
|
||||
- **Klangio Guitar2Tabs ratings**
|
||||
- Trustpilot reviews range from "accuracy ... is legendary" to one-star: "didn't even come close to being correct" for a user's own recording.
|
||||
- Apple App Store: 3.6/5 from 25 ratings.
|
||||
- An aggregator reports 3.4/5 and crashes on MusicXML export.
|
||||
- — [Trustpilot](https://uk.trustpilot.com/review/klang.io); [App Store](https://apps.apple.com/app/id1589660528); [justuseapp](https://justuseapp.com/en/app/1589660528/guitar2tabs-transcribe-riffs/reviews)
|
||||
- **Klangio product details**
|
||||
- Exports Guitar Pro, MIDI, MusicXML, PDF. Has a built-in editor.
|
||||
- Free tier is about a 20-second demo.
|
||||
- One review notes the full transcription not matching the demo.
|
||||
- — [audio2guitar blog (vendor-affiliated)](https://audio2guitar.com/blog/best-ai-guitar-tab-generators); [aipure](https://aipure.ai/products/klangio)
|
||||
- **Independent Klangio test** (Aug 2026): accuracy dropped noticeably with phone-recorded or noisy audio. — [agent-finder](https://agent-finder.co/reviews/klangio)
|
||||
- **Songsterr** advertises "Paste a YouTube link, and AI will generate accurate tabs in a few minutes." No independent quality evaluation was found. — [App Store](https://apps.apple.com/app/id399211291)
|
||||
- **Songscription**: guitar and bass tab generator with Guitar Pro export. Says accuracy is best on clearly audible or solo guitar. — [Songscription](https://www.songscription.ai/guitar-tab-generator)
|
||||
- **audio2guitar** (vendor blog)
|
||||
- Separates the guitar, then tabs, chords, and lyrics. MIDI and PDF export only.
|
||||
- 3 free songs, then $8.99/month.
|
||||
- — [audio2guitar](https://audio2guitar.com/blog/best-ai-guitar-tab-generators)
|
||||
- **Chordify** does chords only, no tabs or fret positions. **Moises** does stems and chords, no tabs. **Ultimate Guitar** and **Songsterr** are libraries of human tabs (about 1.4M on UG; Songsterr has 400k songs and 1.3M tracks). — [audio2guitar](https://audio2guitar.com/blog/best-ai-guitar-tab-generators)
|
||||
|
||||
### Inferences
|
||||
- A differentiator for the project could be published, reproducible benchmark numbers against GP references. No commercial vendor publishes any.
|
||||
|
||||
### Gaps
|
||||
- No info was found on Guitar Pro 8.x "audio import" or transcription features. Its existence or quality is unverified.
|
||||
- No Reddit or r/guitarpro sentiment threads surfaced.
|
||||
- No independent Songsterr AI-tab quality reviews were found.
|
||||
- No numbers were found for Ultimate Guitar AI features.
|
||||
@@ -0,0 +1,216 @@
|
||||
# Open-source automatic transcription of pitched instruments (audio to MIDI/notes), as of October 2026
|
||||
|
||||
Scope: piano/keyboard, bass, vocals, and general multi-instrument transcription. Guitar-tab (string/fret) models and drum transcription are covered by other researchers, so they are only mentioned here in passing.
|
||||
Research date: 2026-10-30. Items marked **[unverified]** come from prior background knowledge and were not re-fetched this session. The report writer should treat them as lower confidence.
|
||||
|
||||
## Q1. Spotify Basic Pitch (Python and basic-pitch-ts in the browser): accuracy, polyphony, pitch bends, complaints
|
||||
|
||||
### Takeaway
|
||||
Basic Pitch is the most permissive and easiest-to-deploy option: Apache-2.0, instrument-agnostic, polyphonic, outputs pitch bends, and runs in the browser through basic-pitch-ts / basicpitch.io. It produces no instrument labels and works best on one instrument at a time. In 2025-26 benchmarks it now sits well behind newer models, both on multi-instrument scoring and on instrument-agnostic frame F1.
|
||||
|
||||
### Cited Findings
|
||||
- **License and platforms.** Apache-2.0 (Spotify AB, 2022). The Python package ships the TF model converted to CoreML, TFLite and ONNX. Defaults are CoreML on macOS, TFLite on Linux and ONNX on Windows. TensorFlow is optional (`basic-pitch[tf]`). Supports Python 3.7–3.11. — [GitHub spotify/basic-pitch](https://github.com/spotify/basic-pitch)
|
||||
- **Capabilities.** "Supports polyphonic instruments" with multipitch support, and outputs MIDI "complete with pitch bends". It is instrument-agnostic, and the README says "Basic pitch works best on one instrument at a time". Stereo is down-mixed to mono and audio is resampled to 22050 Hz. — [GitHub spotify/basic-pitch](https://github.com/spotify/basic-pitch)
|
||||
- **Community and activity.** About 6.0k stars, 546 forks, 49 open issues and 16 open PRs at fetch time. No recent release was visible on the page. — [GitHub spotify/basic-pitch](https://github.com/spotify/basic-pitch)
|
||||
- **Browser version.** basic-pitch-ts is Apache-2.0 and published on npm as `@spotify/basic-pitch`. Usage is browser-style (AudioContext, decodeAudioData), and the web demo is basicpitch.io. The repo has 15 commits, 9 open issues and 4 open PRs. The page does not name the runtime; it is believed to be TF.js **[unverified]**. — [GitHub spotify/basic-pitch-ts](https://github.com/spotify/basic-pitch-ts)
|
||||
- **Model size.** The ICASSP 2022 paper describes a very small model (on the order of ~17K parameters, low memory) **[unverified, not stated on the README]**. — [arXiv 2203.09893](https://arxiv.org/abs/2203.09893)
|
||||
- **2025 AMT Challenge.** Multi-instrument, instrument-aware Onset F1 was **0.0634**, last of 14 entries. MT3 scored 0.3932 and the winner 0.5998. Basic Pitch was also the fastest entry at 3.91 ms runtime. The low score is mostly because it emits no instrument labels, while the metric requires instrument+pitch+onset to match. — [arXiv 2603.27528 (AMT Challenge 2025 results)](https://arxiv.org/html/2603.27528v1)
|
||||
- **Harmonica comparison (Sep 2026, ICASSP 2027 submission).** The Harmonica-nano model (26.3K params) reports frame F1 0.796 on its dev set, "14.6 percentage points higher than Basic Pitch" (which implies about 0.65 for Basic Pitch). It runs at 1,622.5x real time. — [arXiv 2609.04640](https://arxiv.org/abs/2609.04640)
|
||||
- **User complaints.** A search for Reddit/forum complaints returned nothing relevant (see Gaps).
|
||||
|
||||
### Inferences
|
||||
- For a browser-first app that needs unlabeled notes from a single isolated instrument or stem, Basic Pitch is still the default because of its license and footprint. For labeled multi-instrument output it is not competitive.
|
||||
- Harmonica could become a stronger drop-in replacement for Basic Pitch, being instrument-agnostic and even smaller, but its code and license are unknown.
|
||||
|
||||
### Gaps
|
||||
- No primary-source user sentiment was found for Basic Pitch. Commonly reported anecdotes (ghost notes and octave errors on dense mixes, fragmented notes on vibrato or bends) are **[unverified]**.
|
||||
- Exact model file size of basic-pitch-ts, and whether a WebGPU backend exists, are not confirmed.
|
||||
- Harmonica's code, weights and license are not stated on the arXiv abstract.
|
||||
|
||||
## Q2. Multi-instrument models: MT3, YourMT3/YourMT3+, MR-MT3, and 2025-2026 successors (MuScriptor, MIROS, AMT Challenge)
|
||||
|
||||
### Takeaway
|
||||
As of October 2026, the clear quality leader on real mixed recordings is **Kyutai's MuScriptor** (July 2026, ISMIR 2026). Its code is MIT but its **weights are CC BY-NC 4.0** (non-commercial). On a real-music test set it roughly doubles YourMT3+'s Multi F1 (48.2 vs 21.9). YourMT3+ (GPL-3.0 code) remains the strongest commercially-ambiguous or academic baseline and wins on Slakh-like synthetic data. The 2025 AMT Challenge winner, MIROS (a MusicFM encoder plus YourMT3+-style decoders), only narrowly beat YourMT3+ on synthetic data.
|
||||
|
||||
### Cited Findings
|
||||
- **MuScriptor: authorship and training.** By Rouard, Krause, Roebel, Simon-Gabriel and Défossez (Kyutai/Mirelo). arXiv 2607.08168 v1 posted 9 Jul 2026, v2 on 3 Aug 2026, ISMIR 2026 camera-ready. Training runs in three stages: synthetic pre-training, then fine-tuning on real audio, then RL post-training. It adds instrument-presence conditioning. — [arXiv 2607.08168](https://arxiv.org/abs/2607.08168)
|
||||
- **MuScriptor: architecture.** Decoder-only Transformer. Input is a 5 s mono 16 kHz mel-spectrogram (512 mel bins, 100 Hz frames) with an optional instrument list. Output is MT3-style tokens over 36 instrument subgroups, including Voice, Acoustic Piano, E-Piano, E-Bass, Acoustic Bass, Acoustic Guitar, Clean E-Guitar and Distorted E-Guitar. — [arXiv 2607.08168v2 HTML](https://arxiv.org/html/2607.08168v2)
|
||||
- **MuScriptor: data.** About 1.45M MIDI files rendered with more than 250 soundfonts, plus about 170k real recordings (more than 11,000 h) with aligned note annotations. The test set is 372 held-out real tracks. — [arXiv 2607.08168v2](https://arxiv.org/html/2607.08168v2)
|
||||
- **MuScriptor: results on its real test set.** F1 in %:
|
||||
|
||||
| Model | Onset | Frame | Offset | Drums | Multi |
|
||||
|---|---|---|---|---|---|
|
||||
| YourMT3+ | 32.52 | 45.54 | 17.79 | 41.4 | 21.9 |
|
||||
| MuScriptor 1.3B | 60.4 | 73.3 | 49.0 | 50.2 | 48.2 |
|
||||
|
||||
— [arXiv 2607.08168v2](https://arxiv.org/html/2607.08168v2)
|
||||
- **MuScriptor vs YourMT3+ per dataset (Multi F1).**
|
||||
- MuScriptor ahead: Bach10 34.7 vs 26.4; Dagstuhl ChoirSet 11.5 vs 2.6; PHENICX 25.7 vs 12.2; RWC-Pop 25.6 vs 19.1; RWC-Genre 23.7 vs 17.2; RWC-Jazz 31.8 vs 26.4.
|
||||
- YourMT3+ ahead: RWC-Classical 40.5 vs 36.0; RWC-Royalty-free 23.1 vs 20.3.
|
||||
- The paper gives no per-instrument (bass/guitar/vocal) F1 and no comparison with MT3 or Basic Pitch.
|
||||
— [arXiv 2607.08168v2](https://arxiv.org/html/2607.08168v2)
|
||||
- **MuScriptor: release.**
|
||||
- Code is MIT. Weights are CC BY-NC 4.0 (non-commercial) and gated on Hugging Face (accept the license, then log in).
|
||||
- Variants: small 103M (described as "the practical choice on CPU-only machines"), medium 307M (default), and large 1.4B (which "really wants a GPU"). Apple MPS is supported.
|
||||
- Install with `pip install muscriptor` or `uvx muscriptor serve`. The CLI is `muscriptor transcribe`. Sheet output needs MuseScore 4.
|
||||
- Hosted demo at muscriptor.kyutai.org. No ONNX or browser inference is mentioned.
|
||||
— [GitHub muscriptor/muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
- **Discrepancy on parameter counts.** The paper lists about 60M/100M/300M/1.3B. The repo lists 103M/307M/1.4B. — [paper](https://arxiv.org/html/2607.08168v2) vs [repo](https://github.com/muscriptor/muscriptor)
|
||||
- **2025 AMT Challenge: setup.** Accepted to the NeurIPS 2025 AI for Music workshop and posted on arXiv in March 2026. Eight teams took part. The test set is 76 newly composed pieces (about 20 s each, at most 3 instruments) covering piano, violin, viola, cello, flute, bassoon, trombone and oboe. All audio was synthesized with FluidSynth and the FluidR3 GM soundfont. — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **2025 AMT Challenge: results (Multi Onset F1).**
|
||||
|
||||
| Entry | Multi Onset F1 |
|
||||
|---|---|
|
||||
| MIROS (winner) | 0.5998 |
|
||||
| YourMT3-YPTF-MoE-M | 0.5938 |
|
||||
| YourMT3-YPTF-S | 0.5581 |
|
||||
| MT3 baseline | 0.3932 |
|
||||
| ByteDance Piano | 0.1721 |
|
||||
| ReconVAT | 0.1415 |
|
||||
| Basic Pitch | 0.0634 |
|
||||
|
||||
— [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **MIROS (winner).** A MusicFM conformer SSL encoder, a recurrent adapter conditioned on instrument-group embeddings, and parallel T5-style decoders, about 370M params. It scores Slakh2100 multi F 0.83. It is below YourMT3+ on Slakh but slightly better on challenge data, and the authors suggest YourMT3+ may overfit Slakh. — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **Effect of polyphony and common failures.** F1 drops sharply as instrument count grows (1 to 3 instruments): MT3 0.548 to 0.256; MIROS 0.719 to 0.437; YourMT3-MoE 0.759 to 0.392. Common failures are instrument leakage (hallucinated instruments) and confusion in dense polyphony. — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **YourMT3 / YourMT3+.** Repo code is GPL-3.0, and the weights license is not separately stated. YourMT3+ was published at MLSP 2024 and its code is marked "pre-release". A Hugging Face Space demo runs on a free GPU (launched Aug 2024). The newest dated news item is Nov 2024 ("YouTube is blocked again"), there are only 30 commits, and the repo appears lightly maintained. — [GitHub mimbres/YourMT3](https://github.com/mimbres/YourMT3)
|
||||
- **YourMT3+ training data and paper.** Training includes a synthetic dataset of bass-guitar techniques across several basses and pickups, and uses cross-dataset stem augmentation. — [arXiv 2407.04822](https://arxiv.org/pdf/2407.04822); [IEEE](https://ieeexplore.ieee.org/document/10734819/)
|
||||
- **MT3 (Google, 2021).** Original multi-task multitrack T5 transcription, now the baseline. — [arXiv 2111.03017](https://arxiv.org/pdf/2111.03017). Code and checkpoints are Apache-2.0, built on JAX/T5X **[unverified]**.
|
||||
|
||||
### Inferences
|
||||
- **Synthetic vs real.** Slakh and other synthetic results greatly overstate real-world quality. YourMT3+ scores around 0.8+ on Slakh-like data but about 0.22 Multi F1 on MuScriptor's real test set. Even MuScriptor tops out below 0.5 Multi F1 on real music. Instrument labeling, not pitch detection, is the hardest part.
|
||||
- **Commercial use.** MuScriptor's weights are non-commercial and YourMT3 is GPL-3.0 (copyleft, which matters if the code is linked). The only permissive multi-instrument option found is MT3 (Apache **[unverified]**), and it is the weakest of the three.
|
||||
- **Browser.** None of the multi-instrument Transformer models has a browser/ONNX port. The models are 100M–1.4B params, so client-side inference would be heavy. MuScriptor-small at 103M might be feasible with WebGPU, but nobody is documented doing it.
|
||||
|
||||
### Gaps
|
||||
- MR-MT3 (memory-retaining MT3, which reduces instrument leakage) was not re-checked this session. Its license and metrics are unknown.
|
||||
- No MIREX 2025/2026 multi-F0 or note-tracking results were found.
|
||||
- No per-instrument F1 for bass, guitar or vocals from MuScriptor or YourMT3+ on real audio.
|
||||
- Diffusion-based multi-instrument transcription papers were not found in this session's searches.
|
||||
|
||||
## Q3. Piano transcription: Omnizart, Onsets & Frames, ByteDance high-resolution, hFT-Transformer, Transkun, newest SOTA
|
||||
|
||||
### Takeaway
|
||||
Piano is effectively solved on MAESTRO: about 97% note-onset F1 and about 93% note+offset F1 for the best models (the onset figure is **[unverified]**). **Transkun** (about 12.9M params, 93.48% note-with-offset F1 on MAESTRO v3) is the best-documented practical SOTA and is pip-installable. The ByteDance repo is now archived, and newer 2025-26 papers (D3RM diffusion refinement, optimal-transport training, sparse-attention transformers) make only marginal gains.
|
||||
|
||||
### Cited Findings
|
||||
- **Head-to-head on MAESTRO v3, note-with-offset F1.** hFT-Transformer (5.5M params) scores 90.53% (P 92.52, R 88.69). Transkun (12.9M) scores 93.48% (P 94.61, R 92.39). A 2026 optimal-transport model scores 90.78%. That paper attributes offset errors to key release and sustain pedal, and it does not model pedal. — [arXiv 2605.17405](https://arxiv.org/pdf/2605.17405); same numbers in [D3RM arXiv 2501.05068](https://arxiv.org/pdf/2501.05068)
|
||||
- **Transkun V2 checkpoint.** On MAESTRO V3 it reports note onset+offset F1 0.9349. The pip-shipped checkpoint is trained without pedal extension of notes and with data augmentation. — [GitHub Yujia-Yan/Transkun](https://github.com/yujia-yan/transkun)
|
||||
- **Other recent piano papers.** hFT-Transformer (ISMIR 2023) — [ISMIR archive](https://archives.ismir.net/ismir2023/paper/000024.pdf). Non-hierarchical Transformer interval scoring (Transkun v2 paper) — [arXiv 2404.09466](https://arxiv.org/pdf/2404.09466). Efficient sparse-attention piano transcription (Sep 2025) — [arXiv 2509.09318](https://arxiv.org/pdf/2509.09318). D3RM discrete diffusion refinement (Jan 2025) — [arXiv 2501.05068](https://arxiv.org/pdf/2501.05068)
|
||||
- **Robustness on out-of-distribution audio.** Augmentation and re-performance training reached 88.4 note-onset F1 on MAPS without using MAPS training data. — [arXiv 2402.01424](https://arxiv.org/pdf/2402.01424)
|
||||
- **Benchmark ceiling and evaluation.** SOTA systems exceed 90% frame-level and 80% note-level F1 (with offsets). The paper argues IR metrics are not musically informed. — [arXiv 2406.08454](https://arxiv.org/html/2406.08454v3)
|
||||
- **ByteDance piano_transcription (high-resolution, Kong et al.).** Code is Apache 2.0 and the checkpoint license is not stated. The repo was **archived by the owner on Dec 8, 2025 (read-only)**. Inference uses the `piano_transcription_inference` pip package. The README reports test frame_ap 0.9285, onset MAE 0.097 and offset MAE 0.1353, plus pedal transcription. — [GitHub bytedance/piano_transcription](https://github.com/bytedance/piano_transcription)
|
||||
- **ByteDance on non-piano music.** As a piano-only model on the multi-instrument challenge it scored 0.1721. — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **Omnizart.** MIT license. Modules: music (pitched instruments), vocal (note-level), vocal-contour (F0), chord, drum and beat. There is no archived notice, but the README says it is incompatible with ARM macOS and the drum model has unresolved training bugs. 24 open issues and 4 open PRs. No recent release was visible, and it is TensorFlow-based. — [GitHub Omnizart](https://github.com/Music-and-Culture-Technology-Lab/omnizart)
|
||||
- **Onsets & Frames.** Part of Magenta (Apache-2.0), and the MAESTRO paper baseline. — [arXiv 1810.12247](https://arxiv.org/pdf/1810.12247v3). A TF.js Onsets & Frames port exists in magenta-js (@magenta/music) **[unverified]**, which is a historical browser option.
|
||||
|
||||
### Inferences
|
||||
- **Practical choice.** For commercially usable piano transcription, Transkun (12.9M) or ByteDance (archived but Apache code) are the practical picks.
|
||||
- **Maintenance.** Omnizart appears effectively unmaintained (TF-era, ARM-Mac incompatible), though not formally archived.
|
||||
- **Browser.** Piano models are small (5.5–13M params), so ONNX-web/WebGPU ports are feasible in principle. No maintained browser port was found except the old Magenta.js Onsets & Frames.
|
||||
|
||||
### Gaps
|
||||
- Transkun's license was not captured this session (believed MIT **[unverified]**).
|
||||
- hFT-Transformer's license is likewise uncaptured (Sony, believed MIT **[unverified]**).
|
||||
- No Papers-with-Code-style leaderboard for 2026 was found. Papers with Code itself was shut down in 2025 **[unverified]**.
|
||||
- No quantified user sentiment was gathered.
|
||||
|
||||
## Q4. Vocal melody transcription (F0 and notes) and lyrics alignment
|
||||
|
||||
### Takeaway
|
||||
The de facto open pipeline for sung notes is **RMVPE (F0) followed by ROSVOT (notes)**, with SOME as a lighter alternative. ROSVOT's checkpoint is mainly trained on Mandarin. For F0 alone:
|
||||
- **FCPE** is the fast, accurate choice.
|
||||
- **PESTO** is a tiny, CPU-friendly self-supervised choice.
|
||||
- **CREPE/torchcrepe** is accurate but slow.
|
||||
|
||||
MuScriptor also emits a "Voice" track, but vocal-specific metrics are not reported.
|
||||
|
||||
### Cited Findings
|
||||
- **ROSVOT (ACL 2024).** Official PyTorch code provides checkpoints for ROSVOT, RMVPE and a word-boundary model. It handles noisy and accompanied singing, and the paper reports SOTA on clean and noisy input. — [GitHub ROSVOT](https://github.com/RickyL-2000/ROSVOT); [ACL Anthology](https://aclanthology.org/2024.acl-long.526/); [arXiv 2405.09940](https://arxiv.org/html/2405.09940)
|
||||
- **RMVPE-to-ROSVOT pipeline in use.** It was used again in 2025 dataset work (MRSAudio). — [arXiv 2510.10396](https://arxiv.org/html/2510.10396v3)
|
||||
- **TelkNet tool.** It offers SOME (described as fast and lightweight), GAME, ROSVOT and VOCANO side by side, and notes that ROSVOT's released checkpoint is mainly trained on Mandarin singing. — [TelkNet vocal-to-MIDI](https://telknet.cc/en/tools/music-vocal-midi)
|
||||
- **FCPE (arXiv Sep 2025).** 96.79% RPA on MIR-1K, comparable to RMVPE. It is 10.64M params vs 90.42M for RMVPE and 1.06 vs 4.91 GFLOPs. RTF is 0.0062 on an RTX 4090, about 5.3x faster than RMVPE and 77x faster than CREPE. — [arXiv 2509.15140](https://arxiv.org/html/2509.15140v1)
|
||||
- **PESTO (Sony CSL).** Self-supervised. It processes a 2m51s file in about 13 s on a laptop CPU vs about 12 min for CREPE. Accuracy is close to CREPE with about 800x fewer params, and it outperforms other SSL baselines on MIR-1K and MDB-stem-synth. — [GitHub SonyCSLParis/pesto](https://github.com/SonyCSLParis/pesto); [PyPI](https://pypi.org/project/pesto-pitch/0.1.0/)
|
||||
- **Lyrics.** One 2025-26 pipeline (StepAudio 3 Music) separates vocals with Demucs before lyric recognition, saying this "reduces interference from instrumentation and improves the reliability" of lyric predictions. — [arXiv 2609.16034](https://arxiv.org/pdf/2609.16034)
|
||||
|
||||
### Inferences
|
||||
- **Browser.** PESTO (tiny) and FCPE (about 10M) are realistic for browser F0 via ONNX. ROSVOT/SOME note segmentation could follow on stems.
|
||||
- **Vocal transcription is a two-stage problem.** Pitch is near-solved. Note segmentation and quantization (vibrato, slides, melisma) are the weak link.
|
||||
|
||||
### Gaps
|
||||
- Licenses of ROSVOT, SOME, RMVPE, FCPE and PESTO were not captured this session. PESTO is believed LGPL-3.0, SOME MIT, and RMVPE/FCPE MIT/Apache in RVC/DDSP-SVC ecosystems **[all unverified]**.
|
||||
- No head-to-head note-level COnPOff F1 comparison of ROSVOT vs SOME vs VOCANO vs MusicYOLO was found.
|
||||
- Whisper-based lyrics alignment tools were not researched in depth this session. Candidates: WhisperX forced alignment, stable-ts, and lyrics-aligner models **[unverified]**.
|
||||
|
||||
## Q5. Bass transcription and general-model performance on guitar/bass
|
||||
|
||||
### Takeaway
|
||||
No established, maintained, dedicated open bass-to-MIDI model was found. In practice people use (a) Basic Pitch or pitch trackers on a Demucs bass stem (bass is mostly monophonic, which suits this), or (b) the bass track from YourMT3+/MuScriptor, both of which label E-Bass and Acoustic Bass. No published per-instrument bass F1 on real music was found.
|
||||
|
||||
### Cited Findings
|
||||
- **No dedicated electric bass solution.** A 2024 HKU student project notes "a lack of automatic transcription solutions for the electric bass guitar" and builds a monophonic model on synthetic data. — [HKU final paper](https://wp2023.cs.hku.hk/fyp23026/wp-content/uploads/sites/27/final_paper.pdf)
|
||||
- **Double bass.** The FiloBass dataset is in Riley's 2025 QMUL PhD thesis on jazz ensemble transcription. — [Riley PhD thesis](https://webspace.eecs.qmul.ac.uk/s.e.dixon/phd/XavierRiley-PhD-Thesis.pdf)
|
||||
- **YourMT3+ bass training.** It trains on a synthetic bass-technique dataset recorded on multiple basses and pickups. — [arXiv 2407.04822](https://arxiv.org/pdf/2407.04822)
|
||||
- **MuScriptor labels.** Its instrument vocabulary includes E-Bass, Acoustic Bass, Acoustic Guitar, Clean E-Guitar and Distorted E-Guitar, and its demo shows electric guitar and bass with tablature. Per-instrument scores are not given. — [arXiv 2607.08168v2](https://arxiv.org/html/2607.08168v2); [GitHub muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
- **Guitar data.** The GOAT guitar dataset (2025) is relevant to guitar generalization on real recordings. Tab models are out of scope here. — [arXiv 2509.22655](https://arxiv.org/pdf/2509.22655)
|
||||
|
||||
### Inferences
|
||||
- A Demucs bass stem plus a monophonic F0/note tracker (PESTO/FCPE plus note segmentation, or Basic Pitch) is likely the most robust browser-friendly bass approach. The low register (E1 at about 41 Hz) stresses frequency resolution, which is worth testing explicitly.
|
||||
|
||||
### Gaps
|
||||
- No quantitative bass or guitar results were found for Basic Pitch, YourMT3+ or MuScriptor on real stems.
|
||||
- No user reports on bass transcription quality were found.
|
||||
|
||||
## Q6. Does transcribing separated stems beat the mixture? And browser feasibility and model sizes
|
||||
|
||||
### Takeaway
|
||||
No paper was found in this session that directly reports transcription F1 on separated stems vs mixtures. The evidence is qualitative (lyrics pipelines, vendor guidance) plus the structural argument that single-instrument models like Basic Pitch "work best on one instrument at a time". On browser feasibility, only Basic Pitch has an official browser build. Piano models (5–13M) and F0 models (PESTO, FCPE) are small enough to port. Multi-instrument Transformers (100M–1.4B) are not practical client-side today.
|
||||
|
||||
### Cited Findings
|
||||
- **Basic Pitch on single instruments.** "Basic pitch works best on one instrument at a time." — [GitHub spotify/basic-pitch](https://github.com/spotify/basic-pitch)
|
||||
- **Separating vocals before lyric recognition.** Demucs separation before lyrics recognition "reduces interference ... improves the reliability". This is a qualitative claim. — [arXiv 2609.16034](https://arxiv.org/pdf/2609.16034)
|
||||
- **Vendor guidance.** Songscription says stems help mainly when parts mask each other in dense mixes, and that a full mix is otherwise fine. It warns about bleed between separated stems. This is a vendor opinion. — [Songscription blog](https://www.songscription.ai/blog/separate-stems-before-transcribing)
|
||||
- **Demucs.** Separates drums, bass, vocals and other. — [GitHub facebookresearch/demucs](https://github.com/facebookresearch/demucs)
|
||||
- **Polyphony hurts multi-instrument models.** Multi-instrument F1 roughly halves going from 1 to 3 instruments, which indirectly supports separation as preprocessing. — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- **YourMT3+ and stems in training.** YourMT3+ uses "cross-dataset stem augmentation" in training, i.e. stem mixing, not inference-time separation. — [arXiv 2407.04822](https://arxiv.org/pdf/2407.04822)
|
||||
- **Model sizes.**
|
||||
- Harmonica-nano: 26.3K — [arXiv 2609.04640](https://arxiv.org/abs/2609.04640)
|
||||
- hFT-Transformer: 5.5M; Transkun: 12.9M — [arXiv 2605.17405](https://arxiv.org/pdf/2605.17405)
|
||||
- FCPE: 10.64M; RMVPE: 90.42M — [arXiv 2509.15140](https://arxiv.org/html/2509.15140v1)
|
||||
- MIROS: about 370M — [arXiv 2603.27528](https://arxiv.org/html/2603.27528v1)
|
||||
- MuScriptor: 103M / 307M / 1.4B — [GitHub muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
|
||||
### Inferences
|
||||
- **Recommended pipeline.** For mixed recordings, use separation (Demucs or a successor) followed by a per-stem specialist:
|
||||
- Piano stem: Transkun
|
||||
- Vocal stem: RMVPE/FCPE plus ROSVOT/SOME
|
||||
- Bass stem: a monophonic tracker or Basic Pitch
|
||||
|
||||
This is likely the best permissively-licensed option. The quantitative benefit is unproven, and separation artifacts and bleed can create false notes.
|
||||
|
||||
### Gaps
|
||||
- No controlled stems-vs-mix F1 study was found. The 2021 MT3 paper and 2022-23 Demucs+transcription studies may contain such ablations **[unverified]**.
|
||||
- ONNX/WebGPU ports of Transkun, PESTO and FCPE were not verified.
|
||||
|
||||
## Q7. Quantizing performance MIDI into notation (score-like MIDI / MusicXML)
|
||||
|
||||
### Takeaway
|
||||
There are three open research lines:
|
||||
- **PM2S** (Liu et al., ISMIR 2022): CRNN plus neural beat tracking. It has a toolbox and beats commercial software on MV2H.
|
||||
- **End-to-end Transformer PM-to-score** (Beyer and Dai, 2024): RoFormer seq2seq to MusicXML tokens. This is commonly referred to as "MIDI2ScoreTransformer" **[naming unverified]**.
|
||||
- **A 2026 T5-based beat-aware rhythm quantizer** (arXiv 2604.22290).
|
||||
|
||||
MuScriptor sidesteps the problem for end users by exporting sheet music via MuseScore 4.
|
||||
|
||||
### Cited Findings
|
||||
- **PM2S (ISMIR 2022).** CRNN plus beat tracking for rhythm quantization. It is significantly better than commercial software on MV2H. Toolbox: github.com/cheriell/PM2S. — [ISMIR 2022 paper](https://archives.ismir.net/ismir2022/paper/000047.pdf)
|
||||
- **Beyer and Dai (2024).** End-to-end piano performance-MIDI-to-score with Transformers. Frames it as seq2seq producing MusicXML tokens, uses unpaired scores, and uses ASAP data (222 scores, 1068 performances). — [arXiv 2410.00210](https://www.alphaxiv.org/abs/2410.00210)
|
||||
- **Transformer rhythm quantization with beat annotations (Apr 2026).** T5-based, with beat-based pre-quantization and a MIDI tokenizer. It cites the 2024 RoFormer PM-to-score approach. — [arXiv 2604.22290](https://arxiv.org/html/2604.22290v1)
|
||||
- **ScoreTranscriber (KAIST MACLab).** Two stages: rhythm quantization plus hand assignment, then a Transformer that produces MusicXML. It is a research prototype. — [GitHub NZX23/ScoreTranscriber](https://github.com/NZX23/ScoreTranscriber)
|
||||
- **MuScriptor sheet output.** Sheet-music output requires MuseScore 4+. — [GitHub muscriptor](https://github.com/muscriptor/muscriptor)
|
||||
|
||||
### Inferences
|
||||
- **Piano vs everything else.** These tools are almost entirely piano-trained (ASAP). For vocals, bass or multi-instrument output, beat tracking (e.g. Beat This!, madmom; **[unverified]**) plus grid quantization in music21/MuseScore remains the practical route.
|
||||
|
||||
### Gaps
|
||||
- Licenses and maintenance status of PM2S, the Beyer and Dai code, and ScoreTranscriber were not captured.
|
||||
- No browser-capable PM-to-score model was found.
|
||||
@@ -0,0 +1,201 @@
|
||||
# Supporting tooling for an audio-to-sheet-music website: rhythm, harmony, notation, ingestion (as of Oct 2026)
|
||||
|
||||
Scope note: about 17 search/fetch calls. Several sub-questions (Chordino, autochord, music21 quantization, PyGuitarPro, MuseScore CLI, abcjs, VexFlow, flat.io) could not be verified within budget and are listed under Gaps. Where a Gap gives a background-knowledge hint, it is labelled UNVERIFIED and must not be cited as fact.
|
||||
|
||||
## Beat / downbeat / tempo / meter tracking
|
||||
|
||||
### Takeaway
|
||||
madmom is effectively unmaintained on PyPI: the last release was in 2018, and the model files are non-commercial. Beat This! (CPJKU, ISMIR 2024, MIT code and weights, CPU-capable, 8 MB small model) is the best default for a commercial site. allin1 (MIT) adds song structure but pulls in madmom and NATTEN. essentia.js is the only mature in-browser option, but it is AGPLv3.
|
||||
|
||||
### Cited Findings
|
||||
- **madmom status:**
|
||||
- The latest PyPI release is 0.16.1 (Nov 14, 2018). Its classifiers list only Python 2.7 and 3.5–3.7, and it ships only an sdist with no wheels. — [PyPI madmom](https://pypi.org/project/madmom/)
|
||||
- Code is BSD. Model and data files are CC BY-NC-SA 4.0, and commercial use requires contacting Gerhard Widmer (JKU). — [GitHub CPJKU/madmom](https://github.com/CPJKU/madmom)
|
||||
- The repo shows 60 open issues and 22 open PRs, and the README still lists Python 2.7. — [GitHub CPJKU/madmom](https://github.com/CPJKU/madmom)
|
||||
- **madmom on modern Python:**
|
||||
- BeatNet's docs warn that madmom may throw errors on Python ≥3.10 with NumPy ≥1.24. The fixes are a `collections` → `collections.abc` import patch and removing `np.int`/`np.float`. — [BeatNet docs](https://docsearch.algolia.com/mcp/docs/repo/mjhydri/beatnet)
|
||||
- A 2024 student project hit the same failure on Python 3.12. — [Rochester ECE477 project](https://hajim.rochester.edu/ece/sites/zduan/teaching/ece477/projects/2024/JiajunWu_SebastianXu_ProjectPresentation.pdf)
|
||||
- The maintainer historically advised installing from GitHub source instead of PyPI. — [madmom issue mirror](https://git.tdem.in/CPJKU/madmom/issues/478)
|
||||
- **Beat This!:**
|
||||
- A transformer beat and downbeat tracker, "Accurate Beat Tracking Without DBN Postprocessing" (Foscarin, Schlüter, Widmer, ISMIR 2024). — [Zenodo](https://zenodo.org/records/13922116); [beat_this_annotations](https://github.com/CPJKU/beat_this_annotations)
|
||||
- Code and weights are MIT. Some training data has restrictive licenses, and users are told to assess whether that affects them. — [GitHub CPJKU/beat_this](https://github.com/CPJKU/beat_this)
|
||||
- Install is `pip install beat-this` with PyTorch ≥2.0. Main models are about 78 MB and "small" models about 8.1 MB. It falls back to CPU automatically, and madmom is needed only for the optional DBN. — [GitHub CPJKU/beat_this](https://github.com/CPJKU/beat_this)
|
||||
- Third-party leaderboards report downbeat F1 of 96.6 (HJDB), 90.7 (Harmonix) and 88.8 (Beatles). These are aggregator figures that I have not checked against the paper. — [hyper.ai HJDB](https://hyper.ai/ja/sota/tasks/downbeat-tracking/benchmark/downbeat-tracking-on-hjdb)
|
||||
- Community ports exist: a Rust crate (MIT) and a GGUF conversion. — [docs.rs beat-this](https://docs.rs/crate/beat-this/0.1.0); [HF cstr/beat-this-GGUF](https://huggingface.co/cstr/beat-this-GGUF)
|
||||
- A hosted Replicate version exists. — [Replicate xavriley/beat_this](https://replicate.com/xavriley/beat_this/readme)
|
||||
- **allin1 (All-In-One music structure analyzer):**
|
||||
- MIT licensed. It outputs BPM, beats, downbeats, beat positions (1, 2, 3, 4…), and segment boundaries labelled start/end/intro/outro/break/bridge/inst/solo/verse/chorus. — [GitHub mir-aidj/all-in-one](https://github.com/mir-aidj/all-in-one)
|
||||
- Dependencies are PyTorch, NATTEN (built from source on Windows), madmom from GitHub, and Demucs. — [GitHub mir-aidj/all-in-one](https://github.com/mir-aidj/all-in-one)
|
||||
- The README warns that MP3 decoders introduce 20–40 ms offsets and recommends converting to WAV first. — [GitHub mir-aidj/all-in-one](https://github.com/mir-aidj/all-in-one)
|
||||
- Repo stats: 871 stars, 16 open issues, 38 commits, no tagged releases. — [GitHub mir-aidj/all-in-one](https://github.com/mir-aidj/all-in-one)
|
||||
- **BeatNet:**
|
||||
- BeatNet has online (causal) and offline modes, and the offline mode uses a DBN. It depends on madmom, so it inherits madmom's Python 3.12 problems. — [BeatNet docs](https://docsearch.algolia.com/mcp/docs/repo/mjhydri/beatnet)
|
||||
- **essentia / essentia.js:**
|
||||
- Both are AGPLv3, with a commercial dual license available from UPF. — [Essentia licensing](https://essentia.upf.edu/licensing_information.html); [essentia.js README](https://unpkg.com/[email protected]/README.md)
|
||||
- Beat tracking and BPM estimation run in the browser and Node. The README says it is "under rapid development", so APIs may change. — [essentia.upf.edu](https://essentia.upf.edu/); [essentia.js README](https://unpkg.com/[email protected]/README.md)
|
||||
- One production report gives a WASM binary of about 2 MB, a 1–3 s cold init, and KeyExtractor run in Web Workers. — [dev.to TagMyBeat](https://dev.to/tagmybeat/i-built-a-free-music-tool-i4l)
|
||||
|
||||
### Inferences
|
||||
- For a commercial public site, avoid madmom's pretrained models (NC license) and the essentia/essentia.js AGPL unless you buy a commercial license. That leaves Beat This! (MIT) as the safest server-side choice.
|
||||
- allin1's structure labels are attractive, but it depends on madmom, so it inherits both the packaging fragility and the madmom model licence question. Check which madmom components it actually uses before relying on it commercially.
|
||||
- Browser beat tracking is feasible:
|
||||
- essentia.js works directly in the browser, subject to the AGPL.
|
||||
- A small Beat This! model (8 MB) could plausibly run via ONNX Runtime Web after a PyTorch → ONNX export. No official web build was found, so this is an untested inference.
|
||||
- Rubato and odd meters: Beat This! drops the DBN, which in older trackers imposes tempo and meter priors. That plausibly helps with tempo changes. No source in this session quantified accuracy on rubato or odd meters.
|
||||
|
||||
### Gaps
|
||||
- No source found on Beat This! or allin1 accuracy specifically on tempo changes, rubato or odd meters (5/4, 7/8).
|
||||
- Could not confirm whether madmom ever published a newer release (0.17) via GitHub. PyPI shows 0.16.1 only.
|
||||
- No 2025–26 beat-tracking models beyond Beat This! surfaced in searches. No reliable source either way.
|
||||
- librosa was not researched this session. UNVERIFIED background knowledge: it is ISC-licensed and offers a basic DP beat tracker, with no downbeats.
|
||||
|
||||
## Key detection and chord recognition
|
||||
|
||||
### Takeaway
|
||||
The open options are dated but usable. ChordFormer (Feb 2025, conformer, large vocabulary) is the newest academic SOTA claim, but code availability is unconfirmed. BTC (ISMIR 2019) has a usable 170-chord checkpoint. Essentia's key extraction is mature but AGPL.
|
||||
|
||||
### Cited Findings
|
||||
- **ChordFormer (arXiv 2502.11840, Feb 2025):** a conformer-based large-vocabulary chord recognition model. It reports +2% frame-wise accuracy and +6% class-wise accuracy over prior SOTA. — [arXiv](https://arxiv.org/abs/2502.11840v1); [HF papers](https://huggingface.co/papers/2502.11840)
|
||||
- **BTC (bi-directional transformer, ISMIR 2019):**
|
||||
- The large-vocabulary checkpoint covers 170 chord classes, plus a 25-class maj/min model. — [HF puar-playground/btc-chord](https://huggingface.co/puar-playground/btc-chord)
|
||||
- A repo mirror is available. — [GitHub BTC-ISMIR19 mirror](https://github.com/qf6101/BTC-ISMIR19)
|
||||
- **CREMA:** a chord model based on McFee & Bello's ISMIR 2017 structured-prediction model. — [CREMA docs](https://crema.readthedocs.io/en/stable/models.html)
|
||||
- **Extended chord qualities:** 2025 work evaluates a convolutional-transformer ACR model on extended chord qualities. — [SBCM 2025](https://sol.sbc.org.br/index.php/sbcm/article/view/43948)
|
||||
- **LLM refinement:** a 2025 paper uses a 301-class large-vocabulary chord model followed by LLM refinement. — [arXiv 2509.18700](https://www.arxiv.org/pdf/2509.18700)
|
||||
- **Essentia key detection:** KeyExtractor works in the browser via essentia.js under AGPLv3. — [dev.to TagMyBeat](https://dev.to/tagmybeat/i-built-a-free-music-tool-i4l); [Essentia licensing](https://essentia.upf.edu/licensing_information.html)
|
||||
|
||||
### Inferences
|
||||
- For lead-sheet chord symbols, BTC-large (170 classes, Harte-style labels) is the practical open baseline. Its output needs:
|
||||
- a Harte → lead-sheet symbol mapping (e.g., `C:min7` → `Cm7`)
|
||||
- snapping of chord-change times to the beat grid from the beat tracker
|
||||
- Chord accuracy on real pop/rock is typically well below key accuracy. Plan a user chord-editing UI.
|
||||
|
||||
### Gaps
|
||||
- Could not verify ChordFormer code or weights release or licence.
|
||||
- BTC's original repo licence was not verified.
|
||||
- Not researched: Chordino/NNLS-Chroma (Vamp plugin; UNVERIFIED: GPL), autochord, madmom chord models (CC BY-NC-SA, per the madmom model licence above).
|
||||
- No 2026 chord models surfaced.
|
||||
|
||||
## Quantization of onsets to beat grid / time signatures / tuplets
|
||||
|
||||
### Takeaway
|
||||
No dedicated research results were obtained this session. This needs a follow-up.
|
||||
|
||||
### Cited Findings
|
||||
- No sources were retrieved. The one relevant input found is that beat positions (1–4) from allin1 and downbeats from Beat This! provide the bar grid needed for quantization. — [GitHub all-in-one](https://github.com/mir-aidj/all-in-one); [GitHub beat_this](https://github.com/CPJKU/beat_this)
|
||||
|
||||
### Inferences
|
||||
- A practical approach is to use detected beat and downbeat times as a variable-tempo grid, and map each note onset to fractional beat position by interpolating between neighbouring beats. Then snap to the nearest allowed subdivision (16th, 8th triplet) with a cost penalty. This handles rubato better than fixed-BPM quantization.
|
||||
|
||||
### Gaps
|
||||
- music21 `quantize()` was not verified. UNVERIFIED background: music21 is BSD-3 and has `Stream.quantize(quarterLengthDivisors=(4,3))`, which supports triplets but assumes a fixed grid.
|
||||
- MuseScore's MIDI import quantization, partitura and pretty_midi were not verified this session.
|
||||
|
||||
## Notation formats and toolchains (MusicXML, MEI, MIDI, Guitar Pro, alphaTex)
|
||||
|
||||
### Takeaway
|
||||
alphaTab can export Guitar Pro 7 (.gp) since 1.2 and alphaTex since 1.7. That makes it a viable single in-browser model for import and export of GP files. MusicXML and MEI interchange is best handled by Verovio (LGPL) or OSMD (BSD-3).
|
||||
|
||||
### Cited Findings
|
||||
- **alphaTab Guitar Pro export:**
|
||||
- The Gp7Exporter has existed since 1.2.0-alpha.75. — [alphaTab exporter guide](https://next.alphatab.net/docs/guides/exporter); [release 1.2](https://alphatab.net/docs/releases/release1_2)
|
||||
- A Dec 2025 fix addressed exported GP files that had no RSE sound in Guitar Pro (drums in GP8). — [alphaTab PR #2456](https://github.com/CoderLine/alphaTab/pull/2456); [issue #2467](https://github.com/CoderLine/alphaTab/issues/2467)
|
||||
- **alphaTab 1.7** (about six months after 1.6) added:
|
||||
- an alphaTex exporter (whole data model → alphaTex)
|
||||
- an AST-based alphaTex parser, a language server, Monaco integration and a VS Code extension prototype
|
||||
- MIDI bank support for GP, MusicXML and alphaTex
|
||||
- improved SMuFL compliance and custom SMuFL fonts
|
||||
- It also fixed tenor/alto clef mapping. — [alphaTab 1.7 release notes](https://alphatab.net/docs/releases/release1_7)
|
||||
- **alphaTab 1.6** added:
|
||||
- an External Cursor API to sync the score with external audio or video
|
||||
- GP8 backing-track support
|
||||
- an audio export API that returns synthesized audio — [alphaTab 1.6 release notes](https://alphatab.net/docs/releases/release1_6)
|
||||
- **alphaTab inputs and licence:** it loads GP3–5, GP6 and alphaTex, renders standard notation, guitar tab and drum tabs, and is MPL-2.0. — [alphaTab features demo](https://demo.alphatab.net/features.html); [alternativeto](https://alternativeto.net/software/alphatab)
|
||||
- **Verovio input formats:** MEI natively, plus MusicXML, ABC, PAE and Humdrum, rendered to SVG. — [gittrend Verovio](https://gittrend.io/repo/rism-digital/verovio)
|
||||
|
||||
### Inferences
|
||||
- The suggested internal pipeline is:
|
||||
1. Build the internal model from transcription output.
|
||||
2. Emit MusicXML (via music21 or partitura server-side) for standard notation and interchange.
|
||||
3. Emit alphaTex or GP for tab, using alphaTab in the browser.
|
||||
- alphaTex is plain text, which makes it a convenient target to generate server-side and load into alphaTab.
|
||||
|
||||
### Gaps
|
||||
- PyGuitarPro (GP3–5 read/write, LGPL), MuseScore 4 headless CLI conversion (`mscore -o`, GPL), LilyPond (GPL) and partitura (Apache-2.0) were not verified this session. The licences listed here are UNVERIFIED background knowledge.
|
||||
- Whether alphaTab can export MusicXML is not confirmed. No source found.
|
||||
|
||||
## Browser rendering / editing / playback
|
||||
|
||||
### Takeaway
|
||||
- **alphaTab (MPL-2.0)** is the only option with first-class multi-track tab, drums and standard notation plus built-in SoundFont playback and external-audio cursor sync. It fits a guitar/bass/drums-centric product best.
|
||||
- **OSMD (BSD-3)** handles MusicXML well and can render tab from MusicXML, but its playback is sponsor-only early access.
|
||||
- **Verovio (LGPL-3.0, WASM)** is the most rigorous engraver and has had MEI tablature features since 2024–25.
|
||||
|
||||
### Cited Findings
|
||||
- **OSMD:**
|
||||
- BSD-3-Clause; renders MusicXML via VexFlow. — [GitHub OSMD](https://github.com/opensheetmusicdisplay/opensheetmusicdisplay)
|
||||
- "Can display tablature (guitar tabs) from MusicXML", including bends and glissandi. — [GitHub OSMD](https://github.com/opensheetmusicdisplay/opensheetmusicdisplay)
|
||||
- Audio playback is "Soon" and offered as an early-access build for GitHub sponsors. — [GitHub OSMD](https://github.com/opensheetmusicdisplay/opensheetmusicdisplay)
|
||||
- **Verovio:**
|
||||
- The npm WASM build is LGPL-3.0-or-later. Older distro packages list LGPL-3.0-only. — [Verovio changelog (safetycli)](https://data.safetycli.com/changelogs/verovio); [Alpine pkg](https://pkgs.alpinelinux.org/package/v3.20/community/x86_64/verovio)
|
||||
- Tablature: customization arrived in 4.5.0 (Dec 2024) and MEI 5.1 tablature features in 5.2.0 (Apr 2025). A 5.7.0-dev version was dated Sep 2025. — [Verovio changelog (safetycli)](https://data.safetycli.com/changelogs/verovio); [ecosyste.ms](https://science.ecosyste.ms/projects/37342)
|
||||
- **Sentiment and platform notes:** community comments describe Verovio as well supported and frequently updated. One developer noted that it does not use native iOS rendering. — [W3C public-music-notation, Apr 2024](https://lists.w3.org/Archives/Public/public-music-notation/2024Apr/0007.html)
|
||||
- **alphaTab playback and sync:** built-in SoundFont synth, an audio export API (1.6), an External Cursor API to sync with the original audio or YouTube video (1.6), and MIDI bank selection (1.7). — [alphaTab 1.6](https://alphatab.net/docs/releases/release1_6); [alphaTab 1.7](https://alphatab.net/docs/releases/release1_7)
|
||||
- **Obsidian plugin:** an Obsidian plugin uses alphaTab for GP preview, an example of ecosystem adoption. — [Obsidian gtp-preview](https://community.obsidian.md/plugins/gtp-preview)
|
||||
|
||||
### Inferences
|
||||
- Recommended stack:
|
||||
- alphaTab as the primary viewer and player for a multi-track band score (guitar tab + standard, bass tab, drums, vocals melody, keys grand staff).
|
||||
- Optionally, Verovio or OSMD for a "print-quality standard notation" view of MusicXML.
|
||||
- MPL-2.0 (alphaTab) is file-level copyleft: you can use it in a proprietary site, but modifications to alphaTab files themselves must be shared.
|
||||
- LGPL-3.0 (Verovio) loaded as a separate WASM module is generally fine for a proprietary web app.
|
||||
- The External Cursor API directly enables a "score follows the original YouTube/audio" feature.
|
||||
|
||||
### Gaps
|
||||
- Not verified this session: VexFlow (MIT, low-level), abcjs (MIT, ABC with synth playback), flat.io embed (commercial API with paid tiers).
|
||||
- No in-browser editing comparison found. UNVERIFIED: alphaTab and OSMD are primarily renderers, so editing has to be built on top of the data model.
|
||||
- No Reddit or HN sentiment threads were retrieved.
|
||||
|
||||
## YouTube ingestion and audio decoding
|
||||
|
||||
### Takeaway
|
||||
yt-dlp works only server-side. As of 2026 it needs an external JS runtime (Deno) and often a PO-token provider plugin, and it breaks periodically when YouTube changes. A public site that downloads YouTube audio violates YouTube's ToS and carries DMCA §1201 anti-circumvention risk. Safer patterns are user-uploaded files plus an embedded YouTube player synced to the score. In the browser, decodeAudioData covers common formats, and ffmpeg.wasm (~31 MB core, SharedArrayBuffer for the multithreaded build) is the fallback.
|
||||
|
||||
### Cited Findings
|
||||
- **yt-dlp runtime requirements:**
|
||||
- From early Feb 2026, yt-dlp requires an external JS runtime such as Deno for YouTube downloads to work properly. A later release dropped Deno versions below 2.3.0. — [openSUSE yt-dlp update 2026.02.21](https://src.opensuse.org/pool/yt-dlp/pulls/4/files); [SUSE PackageHub 2026.01.31](https://packagehub.suse.com/packages/yt-dlp/2026_01_31-bp160_1_1)
|
||||
- The Deno signature solver has failed against new player scripts ("found 0 sig function possibilities"). — [yt-dlp issue #15897](https://github.com/yt-dlp/yt-dlp/issues/15897)
|
||||
- **yt-dlp PO Token Guide (last edited July 12, 2026):**
|
||||
- yt-dlp cannot generate PO tokens itself. Missing tokens can cause HTTP 403s and account or IP blocks.
|
||||
- The recommended setup is a PO Token Provider plugin with the `mweb` client.
|
||||
- Tokens are bound to the video ID.
|
||||
- YouTube is "rolling out changes to enforce PO Tokens".
|
||||
- Some clients (`tv`, `web_embedded`, `android_vr`) don't need tokens but have limitations: DRM without cookies, embeddable-only, or no made-for-kids videos. — [yt-dlp PO Token Guide](https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide)
|
||||
- **bgutil PO token provider:** its PyPI page warns that PO tokens "no longer bypass the bot check for majority of cases". — [PyPI bgutil-ytdlp-pot-provider](https://pypi.org/project/bgutil-ytdlp-pot-provider); [releases](https://github.com/Brainicism/bgutil-ytdlp-pot-provider/releases)
|
||||
- **Bot detection in practice:**
|
||||
- Users still report "sign in to confirm you're not a bot" in mid-2026 (yt-dlp 2026.07.04). — [Lemmy thread](https://peachpie.theatl.social/c/[email protected]/p/364782/what-are-some-ways-to-torify-debian-13-6-or-yt-dlp-2026-07-04-when-yt-believes)
|
||||
- Developers describe an ongoing cat-and-mouse cycle. — [dev.to](https://dev.to/talha_ramzan_3878156fea8c/building-a-video-downloader-that-doesnt-break-every-time-youtube-sneezes-lpa)
|
||||
- **Legal history:**
|
||||
- The RIAA's 2020 DMCA notice against youtube-dl alleged §1201 circumvention of YouTube's "rolling cipher" and cited a German court decision. — [XDA](https://www.xda-developers.com/youtube-dl-riaa-dmca/); [Law Street Media](https://lawstreetmedia.com/?p=6252)
|
||||
- GitHub reinstated the repo. — [The Next Web](https://thenextweb.com/news/github-took-down-youtube-dl-so-devs-made-more-copies)
|
||||
- No court has ruled on the underlying theory in the sources found.
|
||||
- **ffmpeg.wasm:**
|
||||
- The wrapper is MIT, but the core inherits FFmpeg's LGPL or GPL terms depending on the build. — [jsdelivr ffmpeg README](https://cdn.jsdelivr.net/npm/[email protected]/README.md)
|
||||
- Core is about 31–32 MB (reducible with custom builds), with a 2 GB file limit. — [ffmpeg.wasm issue #623](https://github.com/ffmpegwasm/ffmpeg.wasm/issues/623); [dev.to converter](https://dev.to/felix-tech/building-a-privacy-first-video-to-audio-converter-with-ffmpegwasm-1kd0)
|
||||
- Some builds require SharedArrayBuffer. — [Transloadit, May 2025](https://transloadit.com/devtips/extract-thumbnails-from-videos-in-browsers-with-ffmpeg-wasm/)
|
||||
|
||||
### Inferences
|
||||
- Browser-side YouTube download is not feasible: YouTube media URLs lack CORS headers and need signature, n-challenge and PO-token solving. Any YouTube ingestion must be server-side.
|
||||
- Datacenter IPs are heavily bot-flagged, so expect high operational cost (residential proxies, cookies, ban risk) and frequent breakage.
|
||||
- For a public website, the lower-risk design is:
|
||||
- "Upload your own audio file" as the primary path.
|
||||
- A YouTube URL used only to embed the official IFrame player for sync, not for extraction.
|
||||
- If extraction is offered anyway, put it behind ToS, DMCA-agent and rate-limit measures, after legal review.
|
||||
- For in-browser decoding, the Web Audio `decodeAudioData` path is enough for mp3, wav, aac/m4a and ogg/opus in modern browsers (Safari's ogg support historically lagged). Lazy-load ffmpeg.wasm only for exotic containers, because of its ~31 MB size.
|
||||
- SharedArrayBuffer requires COOP/COEP headers, which can conflict with the YouTube iframe embed. This is an inference to test.
|
||||
|
||||
### Gaps
|
||||
- No official YouTube ToS text was fetched this session. UNVERIFIED: the ToS prohibits downloading except where YouTube provides a download button or link.
|
||||
- yt-dlp breakage frequency in 2025–26 was not quantified, for example as a count of emergency releases.
|
||||
- Alternatives (Invidious/Piped instances, the Cobalt API) were not researched. They face the same bot-detection pressure and legal posture.
|
||||
- Not verified with an MDN fetch: browser codec support for decodeAudioData, specifically Safari ogg/opus support as of 2026.
|
||||
@@ -0,0 +1,362 @@
|
||||
# Open-source music source (stem) separation for downstream transcription (state as of Oct 2026)
|
||||
|
||||
Research date: 2026-10-30. About 20 tool calls. MVSEP SDR figures are MVSEP's own "Multisong" or validation-set numbers. They are not MUSDB18-HQ test numbers, so they can't be compared directly with paper numbers. Old information is flagged.
|
||||
|
||||
## 1. Demucs / HTDemucs: status after the Meta archive, forks, and quality of htdemucs_6s guitar/piano
|
||||
|
||||
### Takeaway
|
||||
facebookresearch/demucs was archived on 2025-01-01. The original author (Défossez, now at Kyutai) keeps a low-activity fork at adefossez/demucs that only takes important bug fixes. Demucs is now well behind the RoFormer family. htdemucs_6s guitar and piano stems are weak: the piano stem is close to unusable (MVSEP piano SDR 2.23). Demucs remains the most widely ported model, with ONNX, C++, WASM, Rust and WebGPU ports.
|
||||
|
||||
### Cited Findings
|
||||
- The facebookresearch/demucs repo was archived 2025-01-01 after the author left Meta. It is "not actively maintained anymore and only important bug fixes will be processed on the new repo" (adefossez/demucs). — [facebookresearch/demucs](https://github.com/facebookresearch/demucs); [DeepWiki](https://deepwiki.com/facebookresearch/demucs)
|
||||
- The author reportedly calls adefossez/demucs "the officially maintained Demucs now that I have left Meta to join Kyutai" and says to "expect slow replies and no new feature for now". A low-quality 2026 blog claims the fork shipped v4.1.0 (unverified). — [aireiter blog (secondary)](https://aireiter.com/blog/demucs-stem-separation-models-flags-cost)
|
||||
- Another fork exists: Ryan5453/demucs, which is also packaged on Replicate. — [Ryan5453/demucs](https://github.com/Ryan5453/demucs); [Replicate](https://replicate.com/ryan5453/demucs)
|
||||
- The official README says htdemucs_6s adds piano and guitar, and that "the piano source is not working great at the moment". The early test found guitar quality OK but "a lot of bleeding and artifacts" on piano. — [Demucs README (jsdelivr mirror)](https://cdn.jsdelivr.net/gh/facebookresearch/demucs@main/README.md)
|
||||
- MVSEP Multisong SDR (bass / drums / other / vocals / instrumental):
|
||||
- htdemucs_ft: 12.05 / 11.24 / 5.74 / 8.33 / 14.63
|
||||
- htdemucs: 11.74 / 10.90 / 5.57 / 8.18 / 14.49
|
||||
- htdemucs_6s: 11.42 / 10.59 / 2.63 / 8.17 / 14.48
|
||||
- On MVSEP's piano validation set, htdemucs_6s piano scores 2.23 SDR, against 7.83 for BS-RoFormer SW. MVSEP describes the 6s guitar and piano quality as "still so-so".
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms); [MVSEP algorithm 17](https://www.mvsep.com/algorithms/17)
|
||||
- ZFTurbo's pretrained list gives htdemucs_6s Multisong SDR of bass 11.22, drums 10.22 and vocals 8.05. Guitar and piano SDR are not reported. — [MSST pretrained_models.md](https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/docs/pretrained_models.md)
|
||||
- Model sizes per demucs-rs: htdemucs 84 MB, htdemucs_6s 84 MB, htdemucs_ft 333 MB. htdemucs_ft is a bag of four fine-tuned models, so it is about 4x slower. — [demucs-rs](https://github.com/nikhilunni/demucs-rs); [search summary of StemSplit/aireiter](https://stemsplit.io/blog/demucs-local-setup-guide)
|
||||
- Demucs code and weights are MIT. Browser ONNX exports state "MIT, matching the upstream demucs project". — [monteslu/htdemucs-web-onnx](https://huggingface.co/monteslu/htdemucs-web-onnx)
|
||||
|
||||
### Inferences
|
||||
- For transcription, Demucs makes sense only as a fallback or where portability matters (browser, C++). It should not be the main separator.
|
||||
- A plausible split is htdemucs_ft for bass and drums (reasonable at about 11-12 SDR) and BS-RoFormer SW for guitar and piano.
|
||||
- Not verified this session (from memory): the HTDemucs paper (arXiv 2211.08553) reports about 9.0 dB average SDR on the MUSDB18-HQ test set, with about 9.2 dB for the fine-tuned / sparse variants.
|
||||
|
||||
### Gaps
|
||||
- I did not verify adefossez/demucs commit activity for 2026 or confirm the v4.1.0 release on its releases page.
|
||||
- I found no published guitar SDR for htdemucs_6s from either MVSEP or ZFTurbo.
|
||||
|
||||
## 2. RoFormer family, SCNet, MDX23C and newer models: which give guitar, piano and bass stems, and how good they are
|
||||
|
||||
### Takeaway
|
||||
The best open-weight multi-stem model as of 2025-26 is BS-RoFormer SW, a 6-stem model (vocals, bass, drums, guitar, piano, other) by viperx/lucidrains, distributed by the community. Its MVSEP SDR:
|
||||
|
||||
| Stem | SDR |
|
||||
|---|---|
|
||||
| Vocals | 11.30 |
|
||||
| Bass | 14.62 |
|
||||
| Drums | 14.11 |
|
||||
| Guitar | 9.05 |
|
||||
| Piano | 7.83 |
|
||||
| Other | 8.71 |
|
||||
|
||||
Guitar and piano stay 3-7 dB behind bass, drums and vocals. For 2-stem vocal separation the leaders are MVSEP's BS-RoFormer 124-band models (12.33 vocal SDR, version 2026.07), but these appear to be MVSEP-hosted only.
|
||||
|
||||
### Cited Findings
|
||||
- BS-RoFormer (ByteDance SAMI, Lu et al. 2023) won the SDX23 Music Separation track "by a large margin". It reaches 9.80 dB average SDR trained on MUSDB18HQ only, and a reported 11.99 dB with 500 extra songs. — [arXiv 2309.02612](https://arxiv.org/abs/2309.02612v2); [ar5iv](https://ar5iv.labs.arxiv.org/html/2309.02612)
|
||||
- Mel-Band RoFormer (arXiv 2310.01809) outperforms BS-RoFormer on vocals, drums and other: vocals 11.21 vs 10.78 dB. It is worse on bass: 9.64 vs 11.43 dB. — [arXiv 2310.01809](https://arxiv.org/html/2310.01809v1)
|
||||
- SCNet (ICASSP 2024, Tsinghua and Kunlun) reaches 9.0 dB average SDR on MUSDB18-HQ without extra data, and SCNet-large reaches 9.69. Its CPU inference time is about 48% of HT Demucs. Official code is at starrytong/SCNet. — [arXiv 2401.13276](https://arxiv.org/abs/2401.13276v1); [hyper.ai listing](https://hyper.ai/en/papers/2401.13276)
|
||||
- BS-RoFormer SW on MVSEP:
|
||||
- It is a "modified version of initial BS Roformer… Modifications were made by lucidrains… 2nd version of weights (with better quality) was prepared by viperx". It became MVSEP's default model for new users.
|
||||
- Multisong SDR: vocals 11.30, instrumental 17.50, bass 14.62, drums 14.11, guitar 9.05, piano 7.83, other 8.71.
|
||||
- MVSEP's standalone guitar and piano models moved to these numbers: guitar 7.51 → 9.05, piano 6.20 → 7.83.
|
||||
— [MVSEP news 64](https://mvsep.com/news/64?lang=en); [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- BS-RoFormer SW weights are distributable:
|
||||
- jarredou's HF repo BS-ROFO-SW-Fixed (Aug 16 2025) was "edited to allow its use with UVR and MSST".
|
||||
- The jarredou account was reportedly deleted. enerjazzer/BS-ROFO-SW-Fixed is cited as the current host, with a checkpoint of about 700 MB.
|
||||
- Community GGUF and ONNX conversions exist: studio409/BSRoformer-SW-6Stem-GGUF, and elicwhite's ONNX export.
|
||||
— [HF jarredou commit](https://huggingface.co/jarredou/BS-ROFO-SW-Fixed/commit/f99fc5c2e4bc201413cfc1ca5767cf62ddb1a9a4); [studio409 GGUF](https://huggingface.co/studio409/BSRoformer-SW-6Stem-GGUF); [openmirlab/bs-roformer-infer](https://github.com/openmirlab/bs-roformer-infer)
|
||||
- MVSEP piano validation SDR:
|
||||
|
||||
| Model | Piano SDR |
|
||||
|---|---|
|
||||
| Demucs4HT 6s | 2.23 |
|
||||
| MDX23C (2023.08) | 4.79 |
|
||||
| MDX23C (2024.09) | 5.59 |
|
||||
| MelRoformer (viperx) | 5.67 |
|
||||
| SCNet Large | 5.89 |
|
||||
| SCNet + Mel ensemble | 6.19 |
|
||||
| BS-RoFormer SW | 7.83 |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- MVSEP bass SDR:
|
||||
|
||||
| Model | Bass SDR |
|
||||
|---|---|
|
||||
| BS-RoFormer | 12.49 |
|
||||
| HTDemucs4 | 12.52 |
|
||||
| SCNet XL | 13.81 |
|
||||
| BS + HTDemucs + SCNet XL ensemble | 14.07 |
|
||||
| BS-RoFormer SW | 14.62 |
|
||||
| SCNet XL + BS-RoFormer SW | 14.87 |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- MVSEP drums SDR:
|
||||
|
||||
| Model | Drums SDR |
|
||||
|---|---|
|
||||
| HTDemucs4 | 12.04 |
|
||||
| MelBand | 12.76 |
|
||||
| SCNet Large | 13.01 |
|
||||
| SCNet XL | 13.42 |
|
||||
| BS-RoFormer SW | 14.11 |
|
||||
| MelBand + SCNet XL + SW ensemble | 14.35 |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- MVSEP's 4/5-stem "Ensemble All-In" (v2025.06.30) scores: bass 14.85, drums 14.33, other 9.00, vocals 11.93, instrumental 18.23. This is an MVSEP service, not open weights. — [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- Top 2-stem vocal models on MVSEP Multisong:
|
||||
|
||||
| Model | Vocals / instrumental SDR |
|
||||
|---|---|
|
||||
| BS-RoFormer 124 bands (ver. 2026.07) | 12.33 / 18.64 |
|
||||
| BS PolarFormer 124 bands | 12.02 / 18.33 |
|
||||
| MelBand "becruily deux" | 11.35 / 17.66 |
|
||||
| SCNet XL IHF | 11.11 / 17.41 |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- MVSEP's guitar model has been through several architectures:
|
||||
- The original (Sep 2023) used MDX23C.
|
||||
- Later versions use MDX23C, Mel-RoFormer and BS-RoFormer, covering acoustic and electric guitar.
|
||||
- In 2025 MVSEP added a separate acoustic guitar model and a guitar leaderboard (electric and acoustic).
|
||||
— [MVSEP news 28](https://www.mvsep.com/news/28?lang=en); [MVSEP news](https://de2.mvsep.com/news); [MVSEP algorithms](https://de.mvsep.com/algorithms)
|
||||
- MVSEP also lists keys, organ and Rhodes models, but publishes no metrics for them. — [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- A 2026 roundup (secondary) says the Roformer family sits "at the top of community quality leaderboards as of mid-2026". — [Futureproof Music School](https://futureproofmusicschool.com/blog/best-ai-stem-separation-tools)
|
||||
|
||||
### Inferences
|
||||
- **Recommended open pipeline for transcription:**
|
||||
1. Run BS-RoFormer SW for 6 stems.
|
||||
2. Optionally replace the vocal stem with a top open vocal model (e.g. KimberleyJensen Mel-RoFormer, 10.98 in MSST).
|
||||
3. Optionally ensemble bass and drums with SCNet XL.
|
||||
- **Guitar stem quality:** at about 9 SDR it is good enough for pitch transcription of a dominant guitar. Expect bleed from keys and synths into the guitar and piano stems.
|
||||
- **Piano stem quality:** at about 7.8 SDR, expect more errors, especially on piano that is quiet in the mix or doubled with synth pads.
|
||||
- **Keys and synth fall into "other":** BS-RoFormer SW has no synth or organ stem, so keys other than piano land in "other".
|
||||
|
||||
### Gaps
|
||||
- No independent (non-MVSEP) benchmark of BS-RoFormer SW guitar or piano on a public test set was found.
|
||||
- The license of the viperx/SW weights is unclear. HF repos found did not show a license in search results; treat it as unlicensed or redistribution-tolerated community weights. This is a commercial-use risk.
|
||||
- No confirmed open-weight 7+ stem music model (e.g. with separate synth, strings or brass) was found. MVSEP has separate single-instrument models (strings, wind, etc.); I could not verify which have downloadable weights.
|
||||
|
||||
## 3. UVR5, python-audio-separator, ZFTurbo MSST and MVSEP: availability and licenses
|
||||
|
||||
### Takeaway
|
||||
- All three open tools are MIT for code, but weights are mixed and often unlicensed community releases.
|
||||
- **ZFTurbo MSST:** the training and inference framework, with the broadest architecture support.
|
||||
- **python-audio-separator:** the most practical pip/CLI wrapper for UVR-style models plus Roformers, with ensembles. Actively maintained.
|
||||
- **UVR5 GUI:** still on v5.6 plus beta Roformer patches; no new official release found for 2025-26.
|
||||
- **MVSEP:** a web service hosting the best models, some of them private.
|
||||
|
||||
### Cited Findings
|
||||
- **ZFTurbo/Music-Source-Separation-Training:**
|
||||
- License: MIT. Installable via `pip install msst`. 634 commits, 72 open issues.
|
||||
- Architectures: mdx23c, htdemucs, segm_models, torchseg, bs_roformer, mel_band_roformer, swin_upernet, bandit, bandit_v2, scnet, apollo, bs_mamba2, conformer, dttnet, bs_conformer, scnet_tran, scnet_masked.
|
||||
- It links an ONNX/TensorRT conversion repo.
|
||||
— [MSST GitHub](https://github.com/ZFTurbo/Music-Source-Separation-Training)
|
||||
- MSST pretrained list, selected entries:
|
||||
- Vocals (Multisong SDR): BS PolarFormer 11.00; MelBand RoFormer (KimberleyJensen) 10.98; BS-RoFormer (viperx) 10.87; MDX23C 10.17.
|
||||
- Bass: HTDemucs4 FT bass 11.96.
|
||||
- DrumSep: models are included (see section 5).
|
||||
- No license is stated for the weights. Newest visible tags are v1.0.22 (DTTNet), v1.0.20 (PolarFormer) and v1.0.19 (BS Mamba2).
|
||||
— [MSST pretrained_models.md](https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/docs/pretrained_models.md)
|
||||
- **nomadkaraoke/python-audio-separator:**
|
||||
- License: MIT.
|
||||
- Architectures: MDX (.onnx), VR (.pth), Demucs (.yaml) and MDXC/RoFormer (.ckpt).
|
||||
- Hardware: CUDA 11.8/12.2; Apple MPS and CoreML; CPU; experimental DirectML (RoFormer falls back to CPU, Demucs unsupported).
|
||||
- Ensembles: 11 algorithms and 9 presets, including karaoke.
|
||||
- In v0.47.0 the RoFormer overlap defaults changed, giving 4-6x more forward passes and slower runs; lowering `overlap` trades seam quality for speed.
|
||||
- The README asks users to credit UVR when using UVR-trained models.
|
||||
— [python-audio-separator](https://github.com/nomadkaraoke/python-audio-separator)
|
||||
- **UVR5 GUI:** the latest version cited is v5.6. Roformer support came via the community beta patch "UVR_Patch_4_14_24_18_7_BETA_full_Roformer". One guide says macOS Apple Silicon users may need to run from source for bs_roformer. No 2025-26 official release was found. — [kocpc guide (secondary)](https://en.kocpc.com.tw/?p=12798); [Bilibili guide](https://www.bilibili.com/read/cv27499700/)
|
||||
- **Community sentiment (secondary 2026 roundup):** the top-voted answer in r/edmproduction's "Favorite Stem Splitter?" was UVR, and by mid-2026 production-subreddit sentiment had "hardened toward MVSEP" for vocal/instrumental splitting. — [Futureproof Music School](https://futureproofmusicschool.com/blog/best-ai-stem-separation-tools); [KVR forum thread](https://www.kvraudio.com/forum/viewtopic.php?p=8906542)
|
||||
- **MVSEP:** a web service with a free tier. Its algorithm list includes several models with no public weights (e.g. "MVSep Team" BS-RoFormer karaoke, the 124-band vocal models, and the "MVSep Lead/Rhythm Guitar (Demo)"). — [MVSEP algorithms](https://mvsep.com/en/algorithms); [MVSEP news](https://de2.mvsep.com/news)
|
||||
|
||||
### Inferences
|
||||
- For an app pipeline, python-audio-separator (MIT, pip) can load BS-RoFormer SW and Mel-RoFormer checkpoints. MSST is the more flexible path if you need SCNet or DrumSep.
|
||||
- **Weight-license caution:** most RoFormer weights (viperx, KimberleyJensen, becruily, gabox, anvuew, jarredou) are community-trained, often on undisclosed or copyrighted data, and usually carry no explicit license. Commercial redistribution is legally unclear. Demucs (MIT) is the cleanest option license-wise.
|
||||
|
||||
### Gaps
|
||||
- I could not access UVR Discord or GitHub issues directly.
|
||||
- I could not verify python-audio-separator's latest release number or date; the page showed no releases list.
|
||||
|
||||
## 4. Multi-guitar separation (lead vs rhythm, L/R panned, acoustic vs electric)
|
||||
|
||||
### Takeaway
|
||||
No open-weight model separating lead from rhythm guitar was found. The lead/rhythm options are commercial:
|
||||
- Moises "Lead & Rhythm Guitar Separation" (Apr 2024, vendor claims only).
|
||||
- MVSEP's "Lead/Rhythm Guitar (Demo)".
|
||||
|
||||
Acoustic-vs-electric splitting exists from Moises, AudioShake and MVSEP. A practical open approach is to separate a single guitar stem, then split it using stereo position (mid/side or pan) or a further acoustic/electric pass.
|
||||
|
||||
### Cited Findings
|
||||
- **Moises (2024-04-24):** "Lead & Rhythm Guitar Separation" and "Acoustic & Electric Guitar Separation". Each yields 2 guitar tracks plus vocals, bass, drums and other. It claims "the first-ever model that accurately separates lead and rhythm guitars", with no metrics or limitations given. Premium/Pro only. — [Moises blog](https://moises.ai/blog/moises-news/new-guitar-separation-models/)
|
||||
- **MVSEP:** lists "MVSep Lead/Rhythm Guitar (Demo)" and an "Electric Guitar" demo with no details. It also has an Acoustic Guitar model (2025). — [MVSEP news](https://de2.mvsep.com/news); [MVSEP news 64](https://mvsep.com/news/64)
|
||||
- **LALAL.AI (Andromeda):** its "Guitars" stem captures all guitars, acoustic and electric, in one file, with no lead/rhythm split. — [LALAL.AI blog](https://www.lalal.ai/blog/andromeda-supports-guitars-updates-piano/); [Magnetic Mag 2026-09](https://magneticmag.com/2026/09/lalal-ai-fixes-a-guitar-stem-problem/)
|
||||
- **AudioShake:** released electric guitar and acoustic guitar stems (commercial). — [AudioShake](https://www.audioshake.ai/post/audioshake-releases-five-new-music-instrument-stems)
|
||||
|
||||
### Inferences
|
||||
- **Practical open workarounds (not benchmarked):**
|
||||
1. Separate a guitar stem with BS-RoFormer SW.
|
||||
2. Split it into L, R, mid (L+R) and side (L−R). Double-tracked rhythm guitars panned hard L/R usually dominate the sides, while a centered lead dominates the mid. Hard-panned parts can also be transcribed per channel.
|
||||
3. Do the pan/mid-side split before separation as well (feed L-only and R-only to the separator) to recover panned parts.
|
||||
4. For transcription, run polyphonic transcription on the guitar stem, then separate voices symbolically: monophonic high line = lead, chordal = rhythm.
|
||||
- These are standard DSP practices inferred from how mixes work. No source in this session benchmarks them.
|
||||
|
||||
### Gaps
|
||||
- No academic paper or open model on lead/rhythm guitar separation was found in this session.
|
||||
- No independent evaluation of Moises's lead/rhythm model was found.
|
||||
|
||||
## 5. Drum sub-stem separation (kick, snare, hats, toms, cymbals)
|
||||
|
||||
### Takeaway
|
||||
Open drum-sub-stem models are mature:
|
||||
- **Best quality:** MVSEP/MSST DrumSep models on Mel-RoFormer, SCNet XL and MDX23C. Kick reaches about 18-22 SDR, snare 13-17 and toms 13-16. Hi-hat, ride and crash are poor at about 3-9.
|
||||
- **LarsNet:** an academic 5-stem model with a VST/AU plug-in (LARS). It is faster than real time but lower quality.
|
||||
|
||||
All of these expect a drum stem as input, so separate drums first.
|
||||
|
||||
### Cited Findings
|
||||
- **LarsNet** (Politecnico di Milano, Pattern Recognition Letters 2024):
|
||||
- Trained on the StemGMD dataset.
|
||||
- Outputs 5 stems: kick, snare, toms, hi-hat and cymbals.
|
||||
- Uses a bank of dedicated U-Nets and runs faster than real time.
|
||||
- The LARS plug-in (VST3/AU) wraps it.
|
||||
— [arXiv 2312.09663 (CC BY 4.0)](https://arxiv.org/html/2312.09663v1); [LARS plug-in](https://github.com/EdoardoMor/LARS); [MVSEP LarsNet](https://mvsep.com/algorithms/31?lang=en)
|
||||
- **MVSEP DrumSep** comes in 4, 5 and 6-stem variants; the 6-stem variant has kick, snare, toms, hh, ride and crash. — [MVSEP DrumSep](https://de2.mvsep.com/algorithms/29)
|
||||
- **MVSEP DrumSep, Multisong SDR:**
|
||||
|
||||
| Model | Kick | Snare | Toms | Cymbals / HH / Ride / Crash |
|
||||
|---|---|---|---|---|
|
||||
| MelBand Roformer 4-stem | 22.22 | 17.09 | 15.86 | cymbals 11.87 |
|
||||
| SCNet XL 4-stem | 20.50 | 14.69 | 15.92 | cymbals 10.08 |
|
||||
| MDX23C 6-stem (aufr33/jarredou) | 18.32 | 13.60 | 13.25 | hh 6.71, ride 5.38, crash 7.56 |
|
||||
| HDemucs (inagoy) 4-stem | 14.13 | 8.42 | 5.67 | cymbals 5.63 |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- **MVSEP DrumSep, 150-track validation set:**
|
||||
|
||||
| Model | Kick | Snare | Toms | Cymbals / HH / other |
|
||||
|---|---|---|---|---|
|
||||
| MelBand 4-stem | 18.67 | 13.55 | 13.60 | cymbals 8.76 |
|
||||
| BS-RoFormer (jarredou) 5-stem | 18.16 | 12.80 | 13.51 | 4.66 / 6.65 |
|
||||
| HDemucs (inagoy) | 10.52 | 6.05 | 4.68 | 5.03 |
|
||||
|
||||
The page scrape had some column ambiguity for the 5- and 6-stem rows. — [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- **MSST DrumSep:**
|
||||
- mdx23c (jarredou) 5-stem: kick 16.66, snare 11.53, toms 12.33, hh 4.04, cymbals 6.36.
|
||||
- mdx23c (aufr33 & jarredou) 6-stem: kick 14.54, snare 9.79, toms 10.63, hh 3.19.
|
||||
- htdemucs (inagoy): included.
|
||||
— [MSST pretrained_models.md](https://github.com/ZFTurbo/Music-Source-Separation-Training/blob/main/docs/pretrained_models.md)
|
||||
- **Other:**
|
||||
- Timbrica drum-part separator (kick/snare/toms, ONNX fp16) on HF. — [HF Faridzar/timbrica](https://huggingface.co/Faridzar/timbrica-drum-separator)
|
||||
- Inference-only `mdxnet-infer` PyPI package for MDX23C drum separation. — [PyPI mdxnet-infer](https://pypi.org/project/mdxnet-infer/)
|
||||
- "Inverse Drum Machine" research (Torres, Peeters, Richard 2025), trained without isolated stems. — [arXiv 2505.03337](https://arxiv.org/pdf/2505.03337v2)
|
||||
|
||||
### Inferences
|
||||
- For drum transcription, kick, snare and toms stems are clean enough to onset-detect reliably.
|
||||
- Hi-hat, ride and crash separation is weak, around 3-9 SDR. A 4-stem model with a combined "cymbals" output (about 9-12 SDR) plus a classifier may beat 6-stem splitting for hat/ride/crash. Dedicated drum-transcription models such as ADT, which operate on the full drum stem, may beat separate-then-detect.
|
||||
|
||||
### Gaps
|
||||
- No LarsNet SDR numbers comparable to MVSEP's were captured this session.
|
||||
- Licenses for the DrumSep weights (jarredou/aufr33/inagoy) were not confirmed.
|
||||
|
||||
## 6. Lead vs backing vocal separation
|
||||
|
||||
### Takeaway
|
||||
"Karaoke" models separate lead from backing+instrumental. The best open-ish weights are Mel/BS-RoFormer karaoke models by becruily, gabox, aufr33/viperx, frazer and anvuew, at about 9.5-10.2 lead SDR. MVSEP's in-house BS-RoFormer (124-band) is best at 10.77 lead and 7.03 back. Backing-vocal stems remain much weaker (about 4.5-7 SDR).
|
||||
|
||||
### Cited Findings
|
||||
- **MVSEP karaoke SDR:**
|
||||
|
||||
| Model | Lead SDR | Back SDR | Other |
|
||||
|---|---|---|---|
|
||||
| UVR HP-KAROKEE-MSB2-3BAND | 6.42 | — | — |
|
||||
| MDX-B Karaoke | 7.42 | — | — |
|
||||
| MDX-B Karaoke, applied to vocals | 8.28 | 4.46 | — |
|
||||
| MelBand (aufr33/viperx) | 9.45 | — | — |
|
||||
| MelBand (becruily) | 9.61 | — | — |
|
||||
| MelBand (gabox) | 9.67 | — | — |
|
||||
| Fused gabox + aufr33/viperx | 9.85 | — | — |
|
||||
| SCNet XL IHF (becruily) | 9.53 | — | — |
|
||||
| BS-RoFormer (frazer & becruily) | 10.10 | — | — |
|
||||
| BS-RoFormer (anvuew) | 10.22 | — | — |
|
||||
| BS-RoFormer (MVSep Team) | 10.41 | 6.61 | — |
|
||||
| BS-RoFormer 124 bands (MVSep Team) | 10.77 | 7.03 | back+instrum 16.10 |
|
||||
| UVR-BVE-4B_SN | — | 0.87 | — |
|
||||
|
||||
— [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- Extracting vocals first and then applying karaoke changes results. With fused MelBand the lead drops to 9.62 but a back stem of 5.63 becomes available. — [MVSEP algorithms](https://mvsep.com/en/algorithms)
|
||||
- python-audio-separator ships karaoke ensemble presets. — [python-audio-separator](https://github.com/nomadkaraoke/python-audio-separator)
|
||||
|
||||
### Inferences
|
||||
- **Recommended open chain:** best vocal model → karaoke Mel/BS-RoFormer on the vocal stem → lead and backing. The backing stem is usable for harmony detection but not clean transcription.
|
||||
|
||||
### Gaps
|
||||
- I did not check whether the MVSep Team karaoke weights are downloadable; they appear to be service-only.
|
||||
|
||||
## 7. In-browser separation (ONNX/WebGPU/WASM)
|
||||
|
||||
### Takeaway
|
||||
Only Demucs (htdemucs, htdemucs_6s, htdemucs_ft) has mature browser ports:
|
||||
- **WebGPU ONNX ports:** about 22x real time on a mid-range GPU.
|
||||
- **demucs-rs:** WASM+WebGPU.
|
||||
- **sevagh's freemusicdemixer:** CPU WASM via demucs.cpp. Very slow (tens of minutes per song single-threaded), and its repo was archived Apr 2025.
|
||||
|
||||
RoFormer-class models have community ONNX exports (including BS-RoFormer SW 6-stem), but no mature browser app with published speed was found.
|
||||
|
||||
### Cited Findings
|
||||
- **monteslu/htdemucs-web-onnx:**
|
||||
- htdemucs 4-stem ONNX for onnxruntime-web WebGPU, split into 21 chained pieces to keep the UI responsive.
|
||||
- About 126 MB in mixed precision (173 MB fp32). Mixed precision is about 2.4x faster than fp32.
|
||||
- About 22x real time back-to-back, or about 8x with UI-friendly pacing, on a Radeon RX 7600.
|
||||
- Full fp16 badly degrades drums and bass, so convolutions are kept in fp32.
|
||||
- License: MIT.
|
||||
— [HF monteslu/htdemucs-web-onnx](https://huggingface.co/monteslu/htdemucs-web-onnx)
|
||||
- monteslu also has an htdemucs_ft WebGPU build. — [HF monteslu/htdemucs-ft-webgpu](https://huggingface.co/monteslu/htdemucs-ft-webgpu)
|
||||
- There are htdemucs_6s WebGPU/ONNX exports with guitar and piano. — [HF kramp/htdemucs-6s-webgpu-onnx](https://huggingface.co/kramp/htdemucs-6s-webgpu-onnx); [HF adowu/htdemucs-6s-onnx](https://huggingface.co/adowu/htdemucs-6s-onnx)
|
||||
- **nikhilunni/demucs-rs:**
|
||||
- Rust port supporting htdemucs, htdemucs_6s and htdemucs_ft.
|
||||
- Browser build is WASM + WebGPU and is "significantly slower than the native CLI".
|
||||
- License: Apache-2.0.
|
||||
— [demucs-rs](https://github.com/nikhilunni/demucs-rs); [Korben article](https://korben.info/en/demucs-rs-separate-tracks-stems-browser.html)
|
||||
- **sevagh/freemusicdemixer.com:**
|
||||
- Demucs v4 transliterated to C++/Eigen (demucs.cpp), compiled to WASM with Emscripten.
|
||||
- Models: htdemucs (about 81 MB fp16) and htdemucs_6s (about 53 MB).
|
||||
- Speed: a 7-min track takes about 41 min single-run or about 9 min with 8 workers. Another track took about 20 min single-run or about 5 min with 8 workers.
|
||||
- WASM/JS is MIT, but "AI model weights and other website assets are proprietary".
|
||||
- Repo archived Apr 26 2025 as an older snapshot; the live site may differ.
|
||||
— [freemusicdemixer.com repo](https://github.com/sevagh/freemusicdemixer.com); [sevagh GitHub](https://www.Github.com/sevagh)
|
||||
- **Other ports:**
|
||||
- demucs-wasm npm, derived from freemusicdemixer code. — [demucs-wasm README](https://cdn.jsdelivr.net/npm/[email protected]/README.md)
|
||||
- timcsy/demucs-web and bakkot/demucs-js. One port notes "if your browser does not support WebGPU it will be much, much slower". — [demucs-web](https://github.com/timcsy/demucs-web); [demucs-js](https://github.com/bakkot/demucs-js)
|
||||
- **RoFormer ONNX exports:**
|
||||
- elicwhite/bs-roformer-sw-6stem-onnx (no model card). — [HF elicwhite](https://huggingface.co/elicwhite)
|
||||
- silverdaw 4-stem BS-RoFormer ONNX. — [HF silverdaw](https://huggingface.co/silverdaw/bs-roformer-rhythm-onnx)
|
||||
- schism-audio mini-BS-RoFormer in MLX and CoreML (Apple). — [HF schism-audio](https://huggingface.co/schism-audio/mini-bs-roformer-v2-coreml)
|
||||
|
||||
### Inferences
|
||||
- **Feasible in browser today:** htdemucs or htdemucs_6s on WebGPU (about 10-30 s for a 4-min song on a decent GPU, based on the 22x figure).
|
||||
- **Fallback:** CPU-WASM fallback is impractical (minutes to tens of minutes).
|
||||
- **BS-RoFormer SW in browser:** its about 700 MB checkpoint and attention-heavy compute make it a heavy browser download. It is feasible only with WebGPU and quantization, and it is unproven.
|
||||
|
||||
### Gaps
|
||||
- No published browser speed or quality numbers for RoFormer ONNX models.
|
||||
- I did not verify whether the live freemusicdemixer.com has moved to WebGPU or added models (it reportedly added MIDI features; not confirmed this session).
|
||||
|
||||
## 8. Typical runtime, CPU vs GPU, for a 4-minute song
|
||||
|
||||
### Takeaway
|
||||
htdemucs takes about 15-30 s on a modern NVIDIA GPU and about 1.5-8 min on CPU. htdemucs_ft is about 4x slower. SCNet runs at about half of HTDemucs CPU time. RoFormer models are heavier, and higher overlap settings multiply their runtime.
|
||||
|
||||
### Cited Findings
|
||||
- **MISST measurement:** htdemucs on a 4:09 track took 2m22s on CPU and 28s on GPU. — [MISST GitHub](https://github.com/Frikallo/MISST)
|
||||
- **LinuxLinks measurement:** a 6:24 FLAC took 15.6 s on an RTX 3060 Ti and 187.8 s on CPU (about 12x). — [LinuxLinks](https://www.linuxlinks.com/machine-learning-linux-demucs-music-source-separation/2/)
|
||||
- **Paper on a Demucs-based tool:** about 5 min on CPU vs under 10 s on GPU (RTX 3060: 7-10 s per track). — [SBrT paper](https://biblioteca.sbrt.org.br/articlefile/4978.pdf)
|
||||
- **Vendor estimates (inconsistent):**
|
||||
- htdemucs on an RTX 3080: 20-45 s; on CPU: 90 s to about 8 min.
|
||||
- htdemucs_ft is about 4x slower.
|
||||
— [StemSplit blog (vendor)](https://stemsplit.io/blog/demucs-local-setup-guide)
|
||||
- **SCNet:** CPU inference time is about 48% of HT Demucs. — [arXiv 2401.13276](https://arxiv.org/abs/2401.13276v1)
|
||||
- **RoFormer in python-audio-separator:** overlap defaults in v0.47.0 cause 4-6x more forward passes, and `torch.compile` speeds up warm runs. — [python-audio-separator](https://github.com/nomadkaraoke/python-audio-separator)
|
||||
|
||||
### Inferences
|
||||
- **Full pipeline estimate on a single consumer GPU:** BS-RoFormer SW, plus karaoke, plus DrumSep, likely totals about 1-3 minutes per 4-min song.
|
||||
- **CPU-only estimate:** RoFormer-class models are likely to take 10+ minutes per pass.
|
||||
- Both are inferences; I found no direct RoFormer timing source.
|
||||
|
||||
### Gaps
|
||||
- No primary-source RoFormer, BS-RoFormer SW or Mel-RoFormer CPU/GPU timing benchmarks were found.
|
||||
- No Hacker News threads were captured this session.
|
||||
@@ -0,0 +1,154 @@
|
||||
# Building / fine-tuning our own transcription and separation models: data, training recipes, compute, compression (as of Oct 2026)
|
||||
|
||||
Scope note: complements `reports/Music transcription tooling survey.md` (not repeated here). That survey listed GuitarSet and DadaGP licenses as "unverified" — both are resolved below (GuitarSet is **NC**; DadaGP is **research-only, by request**).
|
||||
|
||||
## Q1. Datasets and licenses — which permit commercial use?
|
||||
|
||||
### Takeaway
|
||||
Almost every *real-audio* AMT/separation dataset with good annotations is non-commercial (GuitarSet, IDMT-SMT-Guitar, MAESTRO, MoisesDB, MedleyDB, MUSDB18-HQ, M4Singer, MTG-Jamendo). The commercially usable core is small and mostly synthetic or drums: **Slakh2100 (CC BY 4.0), E-GMD and Groove (CC BY 4.0), Guitar-TECHS (CC BY 4.0)**, plus DadaGP *code* (MIT) but not its data. A commercial product will have to lean on synthetic rendering + its own recordings.
|
||||
|
||||
### Cited Findings
|
||||
|
||||
**Guitar**
|
||||
- **GuitarSet**: Zenodo record licensed **CC BY-NC 4.0**; 7.5 GB audio+annotations zip, 37 MB annotation-only zip — [Zenodo 1422265](https://zenodo.org/records/1422265). (mirdata's "MIT" refers to loader code only — [mirdata](https://mirdata.readthedocs.io/en/0.3.8/_modules/mirdata/datasets/guitarset.html).)
|
||||
- **IDMT-SMT-Guitar v2**: **CC BY-NC-ND 4.0** (no derivatives!); 1.3 GB; 4 subsets: ~4,700 technique note events (3 guitars), 400 note events, 5 short pieces, 64 chord/rhythm pieces at two tempi on three guitars — [Zenodo 7544110](https://zenodo.org/records/7544110).
|
||||
- **GOAT**: 5.9 h unique DI electric guitar, re-amped ×5 to 29.5 h; 172 audio files; tabs as .gp/.gp5 + DadaGP tokens + GP-rendered wav + aligned MIDI; guitars: Stratocaster ~235 min, Strandberg ~80 min, Jazzmaster ~26 min, Les Paul ~13 min. **The paper's CC BY 4.0 notice covers the paper, not the data**: data is "available by request ... for research purposes only", and some content is covers whose copyright the authors don't own — [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1). (Correction to prior survey, which listed GOAT as CC BY 4.0.)
|
||||
- **SynthTab**: ~13,113 h, 20,715 tracks, 23 timbres (7 acoustic, 16 electric), rendered from DadaGP with commercial Ample Sound plugins via DawDreamer; no license stated in paper ("All code and data is made available at www.synthtab.dev") — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4). The project site lists ~6,700 h / 15,211 tracks and also no license — [synthtab.dev](https://synthtab.dev). (Hour counts disagree between paper and site.) Because it derives from DadaGP (research-only) and commercial sample libraries, treat as research-only.
|
||||
- **DadaGP**: 26,181 Guitar Pro songs, 739 genres, sourced from user-submitted transcriptions of commercial music — [arXiv 2107.14653](https://arxiv.org/pdf/2107.14653). Access is by emailing the authors "to request access to the dataset for research purposes"; repo code is **MIT**, no dataset license stated — [GitHub dadaGP](https://github.com/dada-bots/dadaGP). A search-summary claimed the paper says CC BY 4.0, but that appears to be the paper's own license, not the data — [arXiv 2107.14653](https://arxiv.org/pdf/2107.14653).
|
||||
- **Guitar-TECHS** (Jan 2025, electric guitar techniques, excerpts, chords, scales across diverse hardware): paper states dataset and code released under **CC BY 4.0** — [arXiv 2501.03720](https://arxiv.org/html/2501.03720v1). Commercially usable.
|
||||
- **GAPS** (classical guitar, ISMIR 2024): paper is CC BY 4.0; no dataset-level license found; described as freely available audio–score pairs — [ISMIR 2024 poster](https://ismir2024program.ismir.net/poster_171.html); [arXiv 2408.08653](https://arxiv.org/html/2408.08653v2).
|
||||
- **Riley et al. jazz guitar** (79 GP scores + ~4 h commercial audio from YouTube): no dataset license; aligned MIDI on request — [arXiv 2402.15258](https://arxiv.org/html/2402.15258v1).
|
||||
- **EGDB**: no license found in search — [paperswithcode EGDB](https://astro.paperswithcode.com/dataset/egdb).
|
||||
|
||||
**Piano / drums / vocals**
|
||||
- **MAESTRO**: **CC BY-NC-SA 4.0** — [Magenta MAESTRO](https://magenta.tensorflow.org/datasets/maestro).
|
||||
- **E-GMD**: **CC BY 4.0** — [Magenta E-GMD](https://magenta.tensorflow.org/datasets/e-gmd); [arXiv 2004.00188](https://arxiv.org/pdf/2004.00188).
|
||||
- **Groove MIDI (GMD)**: CC BY 4.0 per Magenta — [Magenta Groove](https://magenta.tensorflow.org/datasets/groove); mirdata's table shows CC BY-SA — [mirdata quick ref](https://mirdata.readthedocs.io/en/0.3.8/source/quick_reference.html) (conflict; Magenta is authoritative).
|
||||
- **STAR Drums** (TISMIR 2025): built by separating real recordings, annotating drums with an ADT model, resynthesising drums with sample-based instruments, and remixing with the real non-drum stem; uses music whose licenses allow distribution; article is CC BY 4.0 but dataset-file license not confirmed — [TISMIR article 244](https://transactions.ismir.net/articles/244); [AudioLabs ePrint](https://www.audiolabs-erlangen.com/content/05_fau/professor/00_mueller/03_publications/2025_WeberUML_DrumSTAR_TISMIR_ePrint.pdf).
|
||||
- **StemGMD**: listed CC BY 4.0 on Hugging Face — [HF stemgmd](https://huggingface.co/datasets/schismaudio/stemgmd/blob/main/README.md).
|
||||
- **M4Singer**: **CC BY-NC-SA 4.0**, 29.77 h Mandarin a cappella, 20 singers, 700 songs — [NeurIPS 2022](https://papers.neurips.cc/paper_files/paper/2022/file/2de60892dd329683ec21877a4e7c3091-Paper-Datasets_and_Benchmarks.pdf).
|
||||
- **Opencpop**: ~5.25 h, one singer, manual alignment + scores (license not found) — [M4Singer paper table](https://papers.neurips.cc/paper_files/paper/2022/file/2de60892dd329683ec21877a4e7c3091-Paper-Datasets_and_Benchmarks.pdf).
|
||||
- **MIR-ST500**: 160k+ notes from 500 pop songs, labelled by non-experts (license not found) — [ICASSP 2021](https://rc.signalprocessingsociety.org/conferences/icassp-2021/spsicassp21vid0572).
|
||||
|
||||
**Multi-track / separation**
|
||||
- **Slakh2100**: **CC BY 4.0**, 145 h of mixtures, ~105 GB download / ~500 GB as wav — [Zenodo 4599666](https://zenodo.org/records/4599666). Commercially usable (synthetic, includes guitar/bass/piano/drum stems from MIDI).
|
||||
- **MoisesDB**: **CC BY-NC-SA 4.0** per repo — [GitHub moises-db](https://github.com/moises-ai/moises-db); 240 tracks, 45 artists, 12 genres; guitars are the next most common stem after vocals/bass/drums — [arXiv 2307.15913](https://arxiv.org/abs/2307.15913v1). (Search snippets also mention "NC-RCL"; either way non-commercial.)
|
||||
- **MedleyDB**: CC BY-NC-SA 4.0 (non-commercial research), 122 multitracks — [auditory.org 2014](https://auditory.org/postings/2014/614.html); [Zenodo](https://zenodo.org/record/1649325).
|
||||
- **MUSDB18-HQ**: "educational purposes only ... should not be used for any commercial purpose"; 46 tracks from MedleyDB (BY-NC-SA) — [Zenodo MUSDB18-HQ](https://zenodo.org/record/3338373).
|
||||
|
||||
**Unlabelled audio for pseudo-labelling**
|
||||
- **MTG-Jamendo**: CC BY-NC 4.0 dataset terms, research/academic use only, per-track CC licenses set by artists — [UPF repository](https://repositori.upf.edu/items/e48c8c42-dba1-404d-85b7-5a0d6ef99e6d/full). **Jamendo sued NVIDIA (June 2026, N.D. Cal.) alleging commercial training of Fugatto/Audio Flamingo on MTG-Jamendo breached CC BY-NC; ~$20.3M sought; a Ghent court on 11 June 2026 rejected NVIDIA Belgium's jurisdictional objection** — [Music Business Worldwide](https://www.musicbusinessworldwide.com/nvidia-hit-with-us-copyright-suit-claiming-it-trained-ai-models-on-winamp-subsidiary-jamendos-music-without-permission/); [chatgptiseatingtheworld](https://chatgptiseatingtheworld.com/2026/06/23/nvidia-hit-with-8th-copyright-suit-latest-filed-by-s-a-jamendo-over-mtg-jamendo-dataset-of-55000-full-audio-tracks/); refiled Oct 2026 — [aimusicpreneur](https://www.aimusicpreneur.com/ai-music-news/jamendo-refiles-nvidia-ai-training-lawsuit-october-2026/). (Allegations only.)
|
||||
|
||||
### Inferences
|
||||
- Commercial-clean training pool for us: Slakh2100 + E-GMD + Groove + StemGMD + Guitar-TECHS + our own recordings + anything we render ourselves from MIDI/GP that we have rights to. Real-audio guitar transcription data that is commercially clean is essentially just Guitar-TECHS + our own.
|
||||
- NC datasets can still be used for **evaluation/benchmarking** internally in many readings, but training production weights on them is the risk; the Jamendo–NVIDIA case shows NC licensors do litigate.
|
||||
- GuitarSet being NC matters: almost every published guitar model (incl. fine-tunes) is trained on it.
|
||||
|
||||
### Gaps
|
||||
- Licenses not found: EGDB, GAPS dataset files, Opencpop, MIR-ST500, MIR-1K, MAPS, URMP, ADTOF datasets, STAR Drums files, FMA (dataset), Babyslakh, any 2026 bass-specific dataset. Each must be checked on its download page.
|
||||
- No dedicated commercially licensed bass or vocal-note dataset found.
|
||||
|
||||
## Q2. Synthetic data from Guitar Pro / MIDI: renderers, amp sims, mixing, sim-to-real gap
|
||||
|
||||
### Takeaway
|
||||
The proven recipe is: GP → per-string MIDI (keyswitches for techniques) → sampled guitar plugin via DawDreamer → NAM amp + cab IR + random reverb via pedalboard. SynthTab-only pretraining transfers poorly zero-shot (GuitarSet Tab F1 43.1 vs 80.1 in-domain) but improves after fine-tuning on real data; GOAT shows NAM re-amping helps electric-guitar generalisation, while doubling the augmentation doesn't help further. Tooling licenses: DawDreamer and pedalboard are GPL-3.0 (fine for offline data generation, not for shipping), NAM is MIT.
|
||||
|
||||
### Cited Findings
|
||||
- SynthTab pipeline: DadaGP → JAMS → MIDI; each **string rendered separately** "to ensure proper string usage", bends/vibrato via MIDI CC, hammer-ons/pull-offs/slides/palm-mute/harmonics via **keyswitches**; mixes are averages of string signals; Ample Sound plugins (AGL, AGT, AGM, AGSJ, ASH, ALP, ATC, AVC, APF, ASC, AE) hosted in **DawDreamer**; rendering took ~1 week on a 24-core M2 Ultra Mac Studio — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4).
|
||||
- SynthTab sim-to-real: TabCNNx4 trained only on SynthTab scores Tab F1 64.2 on SynthTab val, **43.1 GuitarSet, 13.8 IDMT, 57.0 EGDB**; in-domain TabCNN trained on the real sets reaches 80.1 / 63.1 / 70.5. Pretrain-then-fine-tune "unanimously outperform[s]" single-dataset training, but fine-tuning sometimes *lowered* scores vs the pretrained model; authors blame domain mismatch and propose better humanisation, varied virtual rooms, post-processing and augmentation — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4).
|
||||
- Cross-real-dataset generalisation is also poor: TabCNN trained on IDMT scores 15.3 and 28.8 on other sets — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4).
|
||||
- GOAT amp augmentation: DI re-amped with **Neural Amp Modeler using ~7,000 public profiles from >1,000 amps** (one random per item), cab IRs from NeuralDSP Archetype Nolly with randomised mic/height/distance, room reverb on 25% of items at −10 to 0 dB send, rendered with **pedalboard**, ×5 passes; code public — [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1).
|
||||
- GOAT results (note F1): DI-trained 0.820 DI / **0.714 AMP** / 0.860 GuitarSet; AMP-trained 0.843 / **0.800** / 0.843; AMP-XL (double re-amp) 0.818 / 0.806 / 0.835 — i.e. amp augmentation +8.6 pts on distorted test, more is not better. Whisper-style tab token generation "did not work well" (token acc ~0.68, WER ~0.51–0.55) — [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1).
|
||||
- STAR Drums shows a hybrid approach: real non-drum stems + resynthesised drums from ADT-derived annotations — [TISMIR 244](https://transactions.ismir.net/articles/244).
|
||||
- Licenses: **pedalboard GPL-3.0** (JUCE-derived) — [PyPI pedalboard](https://pypi.org/project/pedalboard/0.9.2); **DawDreamer GPL-3.0** ("to make it easier to comply with all of the dependent projects") — per search result summary of [DawDreamer README](https://github.com/DBraun/DawDreamer) (not fetched directly); **NAM trainer MIT** — [GitHub neural-amp-modeler](https://github.com/sdatkinson/neural-amp-modeler).
|
||||
- Slakh2100 is itself a MIDI-render dataset (145 h) under CC BY 4.0 — [Zenodo 4599666](https://zenodo.org/records/4599666).
|
||||
|
||||
### Inferences
|
||||
- GPL tools used *offline* to generate training audio do not, in the usual reading, impose GPL on the resulting model or audio; they must not be shipped in the browser bundle. Commercial sample libraries (Ample Sound etc.) need their EULA checked for "ML training" use — SynthTab does not address this.
|
||||
- NAM profile licensing varies per uploader (tone3000/ToneHunt); profile-level terms were not verified. Capturing our own NAM profiles from owned amps/plugins is the clean route.
|
||||
- For multiple guitars: render 2–4 guitar parts from multi-track GP files with random pan, distinct amp/IR per part, and mix with Slakh-style bass/drums/keys — no paper found that evaluates this specifically for lead/rhythm separation.
|
||||
- Since DadaGP is research-only, a commercial alternative source of tabs is needed: our own GP files, public-domain/CC tabs, or procedurally generated tabs (random playable riffs/chord progressions).
|
||||
|
||||
### Gaps
|
||||
- No published quantification of sim-to-real gap for *distorted multi-guitar mixes*, nor of physical-model (vs sample) renderers.
|
||||
- FluidSynth / sfizz licenses and quality for guitar were not researched in this pass.
|
||||
- Ample Sound / other plugin EULAs regarding dataset generation not checked.
|
||||
|
||||
## Q3. Fine-tuning on our own GP tabs + real recordings: alignment, few-shot, overfitting
|
||||
|
||||
### Takeaway
|
||||
Score-to-audio alignment via DTW against a pretrained transcriber's activations, then per-note onset snapping, gives near-perfect labels (98% onset F1 @50 ms on resynthesised audio vs 62% for DTW alone), and ~4 h of aligned commercial guitar audio was enough to beat the prior zero-shot GuitarSet SOTA. Exact few-shot curves for AMT were not found.
|
||||
|
||||
### Cited Findings
|
||||
- Riley et al. (high-res guitar via domain adaptation): 79 GP scores + ~4 h commercial jazz guitar audio (33 guitarists); coarse alignment = DTW between score and activations of a pretrained piano model; fine alignment snaps each onset to the local activation maximum. Fine alignment yields **98% onset F1 @50 ms (95% @25 ms) vs 62% / 27% for DTW alone** on resynthesised audio — [arXiv 2402.15258](https://arxiv.org/html/2402.15258v1).
|
||||
- Same paper: GuitarSet **zero-shot onset F1 87.3** vs 82.9 previous best; with GuitarSet added 89.7 (Lu et al. 91.1 higher); own jazz test 84.7 vs 80.0. Fine-tuning recipe: piano-pretrained model, 10 s excerpts, batch 4, LR 1e-5 (searched 1e-3–1e-5), 100K steps (~10 epochs), random transposition — [arXiv 2402.15258](https://arxiv.org/html/2402.15258v1).
|
||||
- GOAT provides aligned MIDI ("fine-aligned MIDI") with GP tabs, a template for packaging our own data — [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1).
|
||||
- A 2022 Fraunhofer study found a piano multi-pitch model "surprisingly robust" to microphone-induced domain shift and that four unsupervised domain-adaptation methods did not notably improve it — [Fraunhofer publica](https://publica.fraunhofer.de/handle/publica/486541) (via search summary).
|
||||
- SynthTab notes fine-tuning on small real sets sometimes *reduced* performance vs the synthetic-pretrained model — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4).
|
||||
|
||||
### Inferences
|
||||
- Our hand-made GP tabs map directly onto the Riley/GOAT recipe: GP → MIDI (keep string/fret) → DTW against Basic Pitch or our model's activations → onset snap → train. The GP tab itself also supplies string/fret labels for a tab-assignment head.
|
||||
- With a small set of songs (tens), use them primarily as a **held-out benchmark**, and fine-tune only with heavy regularisation (low LR ~1e-5, few steps, freeze backbone or LoRA/adapters, mix in synthetic data each batch) and leave-one-song-out evaluation to detect overfitting.
|
||||
- ~4 h of aligned real audio is a demonstrated useful scale for guitar; fewer than ~1 h is unproven.
|
||||
|
||||
### Gaps
|
||||
- No AMT paper found quantifying LoRA/adapter few-shot curves (e.g., 5/10/30 min of target data). synctoolbox and CTC-based alignment for AMT not researched in this pass.
|
||||
|
||||
## Q4. Training recipes and compute costs
|
||||
|
||||
### Takeaway
|
||||
Top-tier separation is expensive: the SDX23-winning BS-RoFormer took **16× A100-80GB for 4 weeks (~10,750 GPU-h, ≈$14–17k at $1.3–1.6/h)** on ~500 in-house songs + MUSDB. Fine-tuning a transcriber is cheap by comparison (Riley's guitar fine-tune: batch 4, 100K steps — a single-GPU job). Exact YourMT3/MT3 training budgets were not found.
|
||||
|
||||
### Cited Findings
|
||||
- BS-RoFormer: "the training process lasted for 4 weeks using 16 Nvidia A100-80GB GPUs", total batch 128 (8/GPU), 8 s segments, **93.4M params** (72.2M for LL=6), trained on 500 in-house songs + MUSDB18HQ; ablation LL=6 models trained on MUSDB18HQ only using 16× V100-32GB; LR ×0.9 every 40k steps — [arXiv 2309.02612](https://arxiv.org/html/2309.02612). Small version reached 9.80 dB avg SDR on MUSDB18HQ without extra data — [arXiv abs](https://arxiv.org/abs/2309.02612).
|
||||
- Cloud A100 prices: Lambda 1× A100 40GB $1.29/GPU-h; Lyceum A100 80GB $1.59/h — [Lambda](https://lambda.ai/welcome/ai-cloud); [Lyceum](https://lyceum.technology/products/training/dedicated/index.html).
|
||||
- Riley et al. guitar fine-tune: batch 4, 100K steps, LR 1e-5; compute not reported — [arXiv 2402.15258](https://arxiv.org/html/2402.15258v1).
|
||||
- SynthTab rendering compute: ~1 week on one M2 Ultra (CPU) for ~13k h audio — [arXiv 2309.09085](https://arxiv.org/html/2309.09085v4).
|
||||
- MSST (ZFTurbo) is the community training framework (17 architectures) — see prior survey; MuScriptor trained on 1.45M rendered MIDI + ~11k h real aligned audio (prior survey) — indicates the scale that beats YourMT3+.
|
||||
|
||||
### Inferences
|
||||
- Rough budget tiers (inference from the above): (a) fine-tune a Basic Pitch / HPPNet / CRNN-scale guitar or bass model on synthetic + ~few h real: tens of single-GPU hours (<$100); (b) train a stem separator from scratch at BS-RoFormer scale: ~10k A100-h; (c) fine-tune an existing *permissively licensed* separator (e.g., Demucs MIT) on a new guitar or lead/rhythm stem: likely 1–2 orders of magnitude less than (b), feasible on rented single GPUs over days.
|
||||
- Free Colab/Kaggle GPUs (session-limited) are viable only for tier (a) experiments.
|
||||
|
||||
### Gaps
|
||||
- No published GPU-hours for YourMT3+/MT3, Basic Pitch, MSST community models (viperx etc.), or Demucs fine-tunes were found; Kaggle/Colab quotas not verified for 2026.
|
||||
|
||||
## Q5. Efficient browser models: distillation, quantization, tiny models
|
||||
|
||||
### Takeaway
|
||||
Tiny transcribers can match or beat Basic Pitch (Harmonica-nano: 26.3K params, frame F1 0.796, +14.6 pts over Basic Pitch, 1,622× real time) but code/weights are not released. No AMT-specific int8 accuracy-loss numbers were found; speech/audio analogues report ~1–2 point drops. Distilling from NC teachers (MuScriptor, ADTOF, MoisesDB-trained models) inherits legal risk.
|
||||
|
||||
### Cited Findings
|
||||
- **Harmonica** (arXiv 2609.04640, submitted ICASSP 2027): "multi-depth harmonic convolution"; nano = **26.3K params, frame F1 0.796, 14.6 pts above Basic Pitch, 1,622.5× real time**; x-large claimed SOTA instrument-agnostic; no code/weights link, training data and causality not stated — [arXiv 2609.04640](https://arxiv.org/abs/2609.04640).
|
||||
- Int8 analogues: distilled-then-8-bit acoustic event detection RNN lost ~2 pts accuracy — [arXiv 2001.10876](https://arxiv.org/pdf/2001.10876); OpenVINO int8 Distil-Whisper 2.1–6.1× faster with 1–2% accuracy drop, reducible via calibration data choice — [OpenVINO blog](https://blog.openvino.ai/blog-posts/optimizing-whisper-and-distil-whisper-for-speech-recognition-with-openvino-and-nncf); outlier activations in the last encoder layer hurt int8 for seq2seq speech models — [arXiv 2406.11022](https://arxiv.org/pdf/2406.11022).
|
||||
- TinyMusician (MusicGen-Small): distillation barely changed quality; mixed-precision quantization had a larger effect — [arXiv 2509.00914](https://arxiv.org/pdf/2509.00914).
|
||||
- Distil-Whisper distinguishes sequence-level (train on teacher pseudo-labels) vs word-level (KL to teacher distribution) distillation — [arXiv 2311.00430](https://arxiv.org/pdf/2311.00430).
|
||||
- MuScriptor weights CC BY-NC 4.0 (prior survey); MoisesDB CC BY-NC-SA — [GitHub moises-db](https://github.com/moises-ai/moises-db).
|
||||
|
||||
### Inferences
|
||||
- Plan: train harmonic-conv / Basic-Pitch-size students per stem; export ONNX fp16 for WebGPU (safest), try int8 weight-only for WASM fallback and measure on our benchmark; expect 1–2 pts loss but verify, since onset precision can be more quantization-sensitive than classification.
|
||||
- Distillation from NC teachers: the CC BY-NC terms restrict *use* of the weights; using NC model outputs to train a commercial student is legally untested and risky — treat NC teachers as benchmark-only unless licensed (e.g., Kyutai commercial licence). Permissive teachers: Demucs (MIT), Basic Pitch (Apache-2.0), ADT_STR (CC BY-SA — share-alike likely attaches to derived weights).
|
||||
- Harmonica's numbers suggest model size is not the browser bottleneck for note transcription; separation (Demucs-scale) is.
|
||||
|
||||
### Gaps
|
||||
- No published AMT int8/pruning accuracy figures; no AMT distillation study (e.g., MT3 → small CNN) with numbers found; PESTO and streaming/causal AMT not researched in this pass.
|
||||
|
||||
## Q6. Pseudo-labelling at scale
|
||||
|
||||
### Takeaway
|
||||
Pseudo-labelling works in MIR (chord recognition with >1,000 h teacher-labelled audio; singing transcription with noisy-student self-training) and at very large scale in speech (Distil-Whisper), but the unlabelled corpus must itself be commercially licensed — MTG-Jamendo is NC and now the subject of litigation.
|
||||
|
||||
### Cited Findings
|
||||
- Chord recognition 2026: BTC teacher labels **>1,000 h** unlabelled audio; student trained on pseudo-labels, then on ground truth with selective KD from teacher as regulariser to limit forgetting; students BTC and 2E1D — [arXiv 2602.19778](https://arxiv.org/html/2602.19778v3).
|
||||
- Singing transcription (Kum et al., ICASSP 2022): frame-level pitch pseudo-labels → note level via pitch/rhythm quantization → teacher-student self-training; noisy-student helped — [arXiv 2203.13422](https://arxiv.org/abs/2203.13422v2).
|
||||
- Pseudo-labelling previously applied in MIR to piano transcription and tagging — [arXiv 2602.19778](https://arxiv.org/html/2602.19778v3).
|
||||
- Distil-Whisper: "Robust Knowledge Distillation via Large-Scale Pseudo Labelling" — [arXiv 2311.00430](https://arxiv.org/pdf/2311.00430).
|
||||
- MTG-Jamendo NC + Jamendo v. NVIDIA — [UPF](https://repositori.upf.edu/items/e48c8c42-dba1-404d-85b7-5a0d6ef99e6d/full); [MBW](https://www.musicbusinessworldwide.com/nvidia-hit-with-us-copyright-suit-claiming-it-trained-ai-models-on-winamp-subsidiary-jamendos-music-without-permission/).
|
||||
- MuScriptor's 11k h of real aligned audio (prior survey) is an example of large-scale real-audio labelling.
|
||||
|
||||
### Inferences
|
||||
- Viable commercial pseudo-label pool: tracks individually licensed CC BY / CC0 (filter per-track licenses from Jamendo/FMA/Free Music Archive by artist license, not dataset license), user-uploaded songs only with explicit ToS consent, and licensed catalogues.
|
||||
- Use confidence filtering and the two-stage (pseudo → ground truth + KD regulariser) recipe from the chord paper; teacher should be a permissive or self-trained large model.
|
||||
|
||||
### Gaps
|
||||
- No pseudo-labelling study specific to guitar/bass/drum transcription found; FMA dataset license and per-track CC-BY share not quantified.
|
||||
@@ -0,0 +1,250 @@
|
||||
# Frontier and unconventional methods for guitar/bass tablature from full-band recordings (as of Oct 2026)
|
||||
|
||||
Scope note: this builds on `/home/user/STSheets/reports/Music transcription tooling survey.md`, which already covers TART (54.08% end-to-end Tab F1, 71.8% from perfect MIDI), Fretting-Transformer, midi2tab, GOAT, GAPS, SynthTab, MuScriptor, and the lack of lead/rhythm separation. That material is not repeated here except where needed. Research budget was limited (about 17 tool calls), so several items are flagged as unverified or gaps. Some arXiv abstract pages gave methods but no numbers; those are marked.
|
||||
|
||||
## 1. Symbolic priors for tabs (tab LMs, rescoring, repetition priors)
|
||||
|
||||
### Takeaway
|
||||
There is a mature family of DadaGP-trained tab language models (GTR-CTRL, ShredGP, ProgGP, all Transformer-XL), but none has been used to rescore audio transcriptions. The 2024–26 work that does use symbolic priors applies them only to string/fret assignment (MIDI-to-Tab MLM, Fretting-Transformer, the 2026 constrained-decoding paper). Using a tab LM as a rescoring prior, as speech recognition does, is an open idea with no published evidence.
|
||||
|
||||
### Cited Findings
|
||||
- DadaGP: about 26,181 Guitar Pro songs, available both as GP files and as a token format designed for GPT-2/Transformer-XL style models. — [DADABOTS science](https://dadabots.com/science/); [GTR-CTRL arXiv 2302.05393](https://arxiv.org/pdf/2302.05393)
|
||||
- GTR-CTRL (2023): a Transformer-XL tab generator conditioned on instrumentation (inst-CTRL) and genre (genre-CTRL). It also introduces GPBERT, a genre classifier over GP token sequences. — [arXiv 2302.05393](https://arxiv.org/pdf/2302.05393)
|
||||
- ShredGP (2023): Transformer tab generation conditioned on the style of individual guitarists. — [arXiv 2307.05324](https://arxiv.org/pdf/2307.05324)
|
||||
- ProgGP (2023): the DadaGP-pretrained Transformer-XL fine-tuned on a 173-song progressive-metal GP corpus. — [arXiv 2307.05328](https://arxiv.org/pdf/2307.05328)
|
||||
- SCORE-SET (2025): a new dataset of Guitar Pro files for phrase generation and sequence learning. It includes technique markings such as bends and palm muting. — [arXiv 2507.18723](https://arxiv.org/html/2507.18723v1)
|
||||
- MIDI-to-Tab (ISMIR 2024; Edwards, Riley, Sarmento, Dixon): an encoder-decoder Transformer trained with masked language modeling. It is pretrained on DadaGP (>25k tabs) and fine-tuned on professionally transcribed performances. It "significantly outperforms competing algorithms", and a guitarist user study rated playability. The abstract gives no numbers. The paper is CC BY-NC-SA 4.0 and no code link was found. — [arXiv 2408.05024](https://arxiv.org/abs/2408.05024)
|
||||
- "Explicit Note-Event Tokenization and Pitch-Validity Constrained Decoding for MIDI-to-Tablature Transcription" (Hsu, Wang, Hong, Chen; July 2026): adds note-event tokens alongside TAB tokens. During generation it masks string/fret candidates that would produce the wrong pitch, which guarantees pitch fidelity, and it preserves the input timing. Trained on DadaGP and the small Francois Leduc set, it improves on Fretting-Transformer, with larger gains on Leduc. The abstract gives no numbers. Code "will be released" at github.com/MusicGuitarTab/GuitarTab. — [arXiv 2607.26440](https://arxiv.org/pdf/2607.26440)
|
||||
- A search for papers that use a tab LM to rescore audio-transcription hypotheses (n-best or lattice rescoring) found none. — search result summary, see [TART 2609.11904](https://arxiv.org/html/2609.11904v1), whose T5 stage conditions on audio plus symbolic input but is a single model, not a rescorer.
|
||||
- Older relevant work: "Automatic composition of guitar tabs by Transformers and groove modeling" (ISMIR 2020) is the origin of the tab-token idea. — [ISMIR 2020](https://archives.ismir.net/ismir2020/paper/000349.pdf)
|
||||
|
||||
### Inferences
|
||||
- **LM rescoring recipe (untested, plausible).** The transcriber emits frame-level pitch posteriors. Generate k-best note/tab hypotheses per bar, for example by varying the onset threshold or the alternative string assignments. Score each hypothesis with log P_audio + λ·log P_tabLM, using a DadaGP-trained GPT on GP tokens. Tune λ on the GP-reference benchmark. This mirrors shallow fusion and n-best rescoring in ASR. The LM should most help with (a) deleting implausible spurious notes (bleed from other stems), (b) filling dropped notes in idiomatic patterns (power-chord fifths, octave doubles), and (c) choosing among string positions.
|
||||
- **Pitch-validity constrained decoding** (2607.26440) is the right pattern for any generative tab model: mask outputs inconsistent with the detected pitch set. It is cheap and removes a whole error class.
|
||||
- **Repetition prior.** Rock and metal guitar parts repeat by riff. No paper quantifying a repetition prior for tab transcription was found. A cheap DIY method: build a bar-level self-similarity matrix on chroma from the guitar stem, cluster repeated bars, and use majority voting or consensus over repetitions to fix per-instance errors. This also forces consistent fingering across repeats, which human tabbers do.
|
||||
- **License risk.** DadaGP derives from community GP tabs. GTR-CTRL/ProgGP weights and DadaGP's license status were not verified in this pass.
|
||||
|
||||
### Gaps
|
||||
- Licenses and availability of GTR-CTRL/ShredGP/ProgGP weights were not checked.
|
||||
- No numeric evidence was found for LM-based correction of audio tab transcription. This would be novel work.
|
||||
- MIDI-to-Tab's numbers and the constrained-decoding paper's numbers were not retrieved (abstract pages only).
|
||||
|
||||
## 2. Playability / fingering models: which matches human tabbers best
|
||||
|
||||
### Takeaway
|
||||
Learned seq2seq assigners pretrained on DadaGP (MIDI-to-Tab MLM, Fretting-Transformer, TART's audio-conditioned T5, the 2026 constrained-decoding model) are the current best at matching human tabs. Classic DP/Viterbi with ergonomic costs remains the license-clean baseline. No head-to-head on high-gain rock tabs was found.
|
||||
|
||||
### Cited Findings
|
||||
- MIDI-to-Tab frames fingering as masked-LM inference rather than the constraint-based DP of prior work. Its premise is that DP methods minimize hand movement, which does not always match what humans write. — [arXiv 2408.05024](https://arxiv.org/abs/2408.05024)
|
||||
- TART stage 3 assigns string/fret with a T5 encoder-decoder conditioned on both the pitch sequence and the original audio. Its symbolic-only T5 variant averages 67.0 across benchmarks and beats prior systems on four zero-shot benchmarks. — [arXiv 2609.11904](https://arxiv.org/html/2609.11904v1) (via search summary)
|
||||
- Classic method: audio tab transcription with multipitch analysis plus playability constraints, using dynamic programming across frames for temporal continuity. — [ResearchGate, Barbancho et al.](https://www.researchgate.net/publication/261282224_Audio-based_guitar_tablature_transcription_using_multipitch_analysis_and_playability_constraints)
|
||||
- Earlier, as noted in the prior survey: Fretting-Transformer vs Guitar Pro vs TuxGuitar vs A*, and the 72–82% exact agreement on hard material. — [arXiv 2506.14223](https://arxiv.org/html/2506.14223v1)
|
||||
|
||||
### Inferences
|
||||
- **Hybrid assigner.** Run a Viterbi with ergonomic costs (fret span, shift distance, open-string bonus, string-skip penalty). Add a **chord-shape dictionary bonus**: power chord root on E/A/D with a +2-fret fifth, CAGED open chords, barre shapes. Add a **riff-consistency term**: the same pitch sequence in repeated bars gets the same fingering. This approximates human-tabber conventions cheaply. Later, replace the emission/transition scores with a learned model's logits.
|
||||
- **Style conditioning.** Genre tags (from GTR-CTRL-like conditioning or a metadata lookup) can switch cost presets. Metal favors low-position power chords and the open low string in drop tunings. Folk/pop favors open chords. Solos favor box positions.
|
||||
- **Audio conditioning** (TART-style) is the principled way to inject string timbre (see section 3) into fingering decisions.
|
||||
- **Lenient Tab F1** (counting any playable alternative as correct) should be reported alongside exact Tab F1, because human references disagree.
|
||||
|
||||
### Gaps
|
||||
- No study was found that directly measures which assigner best matches human tabbers on distorted rock/metal GP references.
|
||||
- No published chord-shape-dictionary assigner with numbers was found in 2023–26.
|
||||
|
||||
## 3. String identity from audio (timbre/inharmonicity)
|
||||
|
||||
### Takeaway
|
||||
String classification works on clean, isolated, monophonic guitar: about 87–90% in the best cases, and about 37–72% on older and harder setups. No evidence was found that it survives heavy distortion or source-separated full mixes. Use it as a soft prior inside the fingering decoder, not as a hard decision.
|
||||
|
||||
### Cited Findings
|
||||
- Fretiq (Garg, July/Aug 2026, CC BY 4.0 paper):
|
||||
- Monophonic electric-guitar string classification from a 26-dimensional engineered feature vector (band energies, spectral statistics, 13 MFCCs), with identical Python and TypeScript feature extraction for browser inference.
|
||||
- Accuracy: 97.25% on shuffled frame validation (which the author says overestimates), **86.53% on held-out recording sessions**, and **87.8% on independent free-play**.
|
||||
- MFCCs are the main driver; removing them drops shuffled accuracy to 92.09%.
|
||||
- "Comparison Training", recording same-pitch pairs on adjacent strings, adds about 25.8 points on matched evaluation.
|
||||
- Single guitar and single player; distortion not addressed; no code link found.
|
||||
— [arXiv 2607.18303](https://arxiv.org/pdf/2607.18303)
|
||||
- Abeßer (CMMR 2012): automatic string detection for bass and electric guitar with inharmonicity-based features. Reported F-measures look low, roughly 0.37–0.46 (snippet garbled), though it beat MFCC baselines. — [CMMR 2012 PDF](https://cmmr2012.eecs.qmul.ac.uk/sites/cmmr2012.eecs.qmul.ac.uk/files/pdf/papers/cmmr2012_submission_70.pdf)
|
||||
- String-Inverse Frequency estimation: SVMs on spectral-envelope features reach about 0.90 F-measure for six-string electric guitar, and earlier SIF features reach up to about 0.72 F1 (per search snippet). — [ResearchGate](https://www.researchgate.net/publication/328602822_Automatic_Guitar_String_Detection_by_String-Inverse_Frequency_Estimation)
|
||||
- Barbancho et al., inharmonicity-based method for automatic tablature generation (classic, pre-2023). — [ResearchGate](https://www.researchgate.net/publication/260691838_Inharmonicity-Based_Method_for_the_Automatic_Generation_of_Guitar_Tablature)
|
||||
- Physics caveat: inharmonicity grows with string core diameter, so an unwound G3 can be **more** inharmonic than a wound D3. Inharmonicity is not monotonic across strings, which complicates simple rules. — [TUM paper](https://www.ce.cit.tum.de/fileadmin/w00cgn/mmk/Lehre/Musikalische_Akustik/PDF/1048.pdf)
|
||||
- Distorted guitar: one 2024 study analyzes inharmonicity, loudness and centroid on Moises-separated distorted guitar. It is descriptive and work-in-progress, not a classifier. Clipping adds many extra harmonics. — [ABRAPEM 2024](https://abrapem.org/wp-content/uploads/2024/09/328.-Distortion-the-sound-of-distorted-electric-guitar-between-perception-spectral-descriptors-and-artificial.pdf)
|
||||
|
||||
### Inferences
|
||||
- Under distortion, the nonlinearity generates intermodulation and harmonic products that swamp the partial-frequency deviations that inharmonicity estimators rely on. String classification on high-gain stems is therefore likely to degrade substantially. This is an inference with no direct measurement found.
|
||||
- Clean and crunch parts (and bass, which is usually less distorted) are where string cues are most likely to help. Bass string ID from inharmonicity was studied by Abeßer.
|
||||
- Practical use: feed a per-note string-probability vector as an emission term into the Viterbi/T5 assigner, with a weight learned on the benchmark. Fretiq-style features are cheap and browser-feasible.
|
||||
|
||||
### Gaps
|
||||
- No paper was found evaluating string classification on distorted guitar or separated stems.
|
||||
- Kehling et al. (DAFx 2014) and the IDMT-SMT-Guitar string-ID results were not verified in this pass.
|
||||
|
||||
## 4. Multi-guitar handling (spatial, timbre, role, symbolic part assignment)
|
||||
|
||||
### Takeaway
|
||||
No research system assigns transcribed notes to multiple guitar parts in full mixes. The usable building blocks are classic panning-based separation (Avendano panning index, ADRess/DUET), mixing conventions (rhythm guitars double-tracked and hard-panned, lead near center), and symbolic voice separation. MVSEP reportedly has a lead/rhythm split model, but no metrics were found for it.
|
||||
|
||||
### Cited Findings
|
||||
- Panning-index separation: Avendano's panning index, then Barry et al.'s ADRess (azimuth discrimination and resynthesis), derived from DUET. ADRess uses gain scaling and phase cancellation to find frequency-dependent nulls in the azimuth domain. It works best on studio mixes where instruments occupy distinct pan positions. — [Overview of lead/accompaniment separation, arXiv 1804.08300](https://arxiv.org/pdf/1804.08300); [Barry et al. DAFx 2004](https://dafx.de/paper-archive/2004/P_240.PDF)
|
||||
- DUET's W-disjoint orthogonality assumption does not hold well for music. — [US8027478B2](https://patents.google.com/patent/US8027478)
|
||||
- ADRess is implemented in Csound as `pvsdemix`, which takes an azimuth target. This is a ready reference implementation. — [Csound manual](https://csound.com/manual/pvsdemix.html)
|
||||
- Practitioner conventions: double-tracked rhythm guitars are hard-panned, and the solo "takes the place of the lead vocal" at or near center. Some engineers place guitars at 10 and 2 o'clock, or the lead around 30% on each side. — [Producer Society](https://producersociety.com/how-to-pan-guitars/); [KVR forum](https://www.kvraudio.com/forum/viewtopic.php?t=546325)
|
||||
- A user on the Steinberg forum says the only way to separate lead from other guitars is via panning. Another mentions that MVSEP has a lead/rhythm guitar model. — [Steinberg forum](https://forums.steinberg.net/t/sl12-pro-for-splitting-lead-rhythm-guitar-advice-and-sourcing-video-tutorials/1015919)
|
||||
|
||||
### Inferences
|
||||
- **Pipeline idea:**
|
||||
1. Run the guitar-stem separator on L and R channels separately, as well as on mid and side. The guitar stem output keeps stereo.
|
||||
2. Compute a per-time-frequency-bin panning index on the guitar stem (ADRess-style) and cluster azimuths per section.
|
||||
3. Transcribe each azimuth cluster separately.
|
||||
4. Merge clusters whose note content is near-identical. A double-tracked rhythm part appears hard L and hard R with the same notes, so it collapses to one GP track, which matches human tabbing.
|
||||
5. Assign the center cluster to lead or other.
|
||||
- **Role classification heuristics:** monophonic, high-register, bend/vibrato-heavy and sparse-repetition material suggests lead. Polyphonic, chordal, palm-muted, riff-repetitive and low-register material suggests rhythm. These features are computable from the transcription itself.
|
||||
- **Symbolic voice separation / part assignment** (e.g. a GPBERT-like classifier over GP tokens, or a model trained on DadaGP multi-track songs to split a merged guitar note stream into tracks) is plausible because DadaGP contains multi-guitar songs with explicit tracks. That makes it free supervised data: merge tracks, then learn to split them. No published model doing this was found.
|
||||
- **Timbre clustering:** amp-tone embeddings per note segment, from any audio embedding model, could separate clean versus distorted guitars even when co-panned. This is untested.
|
||||
|
||||
### Gaps
|
||||
- No academic evaluation of lead/rhythm guitar separation or of symbolic guitar part assignment was found.
|
||||
- MVSEP and Moises lead/rhythm models have no published metrics (consistent with the prior survey).
|
||||
- No study was found measuring how often commercial rock mixes follow the double-track hard-pan convention.
|
||||
|
||||
## 5. Tuning and capo detection
|
||||
|
||||
### Takeaway
|
||||
There is a small amount of evidence that tuning classification from audio is feasible (DLfM 2024 deep-learning classifier; a 2023 Rochester student project). Capo detection is explicitly unaddressed ("future work"). Drop-D versus standard is the hard case.
|
||||
|
||||
### Cited Findings
|
||||
- "Acoustic Classification of Guitar Tunings with Deep Learning" (DLfM 2024, ACM/Cardiff):
|
||||
- Neural networks classify tunings from recordings.
|
||||
- It releases annotated authentic and synthetic guitar datasets.
|
||||
- Five tuning classes, including standard (90 songs), drop D (86) and open D.
|
||||
- Capo and open-string detection is listed as future work, using spectrograms or CQTs.
|
||||
- Exact accuracy figures were not retrieved (the PDF returned 403).
|
||||
— [ACM DL](https://dl.acm.org/doi/fullHtml/10.1145/3660570.3660574); [Cardiff ORCA PDF](https://orca.cardiff.ac.uk/id/eprint/170217/1/dlfm2024-4%20(2).pdf)
|
||||
- Rochester ECE477 project (Chew and Parker, 2023): classifies seven tunings from audio. It notes that Drop D is hard to distinguish from E standard, and it was tested only on monophonic audio. — [Report PDF](https://hajim.rochester.edu/ece/sites/zduan/teaching/ece477/projects/2023/DeclanParker_McCormackChew_FinalReport.pdf)
|
||||
|
||||
### Inferences
|
||||
- **Practical heuristics** (DIY, unvalidated):
|
||||
1. Estimate the global reference deviation from A440 using the pitch-histogram cents offset over the whole song. This catches down-tuned-by-half-step (Eb standard), a common case in rock and metal.
|
||||
2. Take the lowest sustained pitch in the guitar stem. Below E2, this means a drop or down-tuning: D2 gives drop D or D standard, C#2/C2 gives drop C# or drop C, and so on.
|
||||
3. Distinguish drop D from D standard with a playability likelihood: transcribe the notes, then run the fingering decoder under each candidate tuning and pick the tuning that minimizes total ergonomic cost or maximizes tab-LM likelihood. One-finger power chords on the lowest three strings are a strong drop-tuning signature.
|
||||
4. Infer a capo the same way. Open-string resonance pitches (sympathetic ringing, open-string drones) plus a playability minimum under fret offsets give the capo position. Strummed open-chord voicings transposed up are the giveaway.
|
||||
5. Use bass lowest-note evidence to cross-check: for example, a 5-string B0, or a drop-D bass with D1.
|
||||
- Let the user override tuning and capo in the UI. That is cheap and removes a large error source.
|
||||
|
||||
### Gaps
|
||||
- Accuracy numbers for the DLfM 2024 paper and the Rochester project were not retrieved.
|
||||
- No paper on capo detection from audio was found.
|
||||
- No paper on bass tuning detection was found.
|
||||
|
||||
## 6. Retrieval and alignment of existing community tabs
|
||||
|
||||
### Takeaway
|
||||
The closest research is DECIBEL (TISMIR), which aligns crowd-sourced tabs and chord sheets to audio with "Jump Alignment" and uses them to improve chord estimation. No research or product was found that does note-level "tab retrieval + alignment" for GP tabs. Songsterr's internal methods are unknown. Legally this is the riskiest idea.
|
||||
|
||||
### Cited Findings
|
||||
- DECIBEL / "Improving Audio Chord Estimation by Alignment and Integration of Crowd-Sourced Symbolic Music" (TISMIR):
|
||||
- A parser extracts untimed chord sequences from guitar tabs and chord sheets.
|
||||
- "Jump Alignment" aligns them to audio while allowing jumps between tab lines, which handles repeats and missing sections.
|
||||
- Selecting the best tab per song reaches 75.0% WCSR.
|
||||
- The paper notes that tabs carry no timing information.
|
||||
— [TISMIR](https://transactions.ismir.net/articles/10.5334/tismir.81); [arXiv 2002.09748](https://arxiv.org/html/2002.09748v1)
|
||||
- Audio-to-score alignment with DTW on chroma is standard. Recent work refines precision with neural CQT-to-piano-roll alignment. — [arXiv 2605.20014 "Precise and Simple Audio-to-Score Alignment"](https://arxiv.org/pdf/2605.20014); [arXiv 2506.22237](https://arxiv.org/pdf/2506.22237)
|
||||
|
||||
### Inferences
|
||||
- **Technical recipe:**
|
||||
1. Identify the song (fingerprint or metadata).
|
||||
2. Fetch candidate tab(s).
|
||||
3. Synthesize them, or derive chroma and onset features from the score.
|
||||
4. Align with a jump-aware DTW (DECIBEL-style, for repeats and structural differences) anchored on beat-tracker downbeats.
|
||||
5. Use the aligned tab as a strong prior: rescoring, fingering, part assignment, tuning. The audio transcription corrects the tab's errors, and the tab corrects the audio errors.
|
||||
6. Detect version mismatch (live versus studio recording, different key) via alignment cost.
|
||||
- The same machinery is needed anyway for the benchmark (aligning GP references to audio).
|
||||
- **Legal (inference, not legal advice):**
|
||||
- Community tabs are derivative works of copyrighted compositions. Songsterr and Ultimate Guitar license publishers' content, and their ToS typically forbid scraping.
|
||||
- Using DadaGP-style data to train a prior is a different (training-data) risk than serving retrieved tabs.
|
||||
- A safer variant uses only user-supplied tabs ("upload your GP file and we align and fix it"), or retrieval only for internal evaluation.
|
||||
- Get legal review before any scraping.
|
||||
|
||||
### Gaps
|
||||
- No research or product doing note-level GP tab retrieval plus alignment was found.
|
||||
- Songsterr AI's internals (whether it leverages its tab catalogue) are unknown.
|
||||
- Songsterr's and Ultimate Guitar's ToS were not fetched in this pass.
|
||||
|
||||
## 7. Technique detection on (distorted) electric guitar
|
||||
|
||||
### Takeaway
|
||||
Datasets now exist: Guitar-TECHS 2025, GOAT 2025 with bends and palm mute in GP format, the Magcil set cited by TART, and classic IDMT-SMT-Guitar. TART reports high per-note classification on curated data (95.9% macro F1 per the prior survey). Evidence for robust technique detection in distorted full-mix stems is still missing.
|
||||
|
||||
### Cited Findings
|
||||
- TART (earlier version, arXiv 2510.02597) targets slides, bends and percussive hits with a per-note technique classifier stage. It cites Magcil: 549 electric guitar recordings with technique labels including bends and palm mutes. — [arXiv 2510.02597](https://arxiv.org/html/2510.02597)
|
||||
- Guitar-TECHS (2025):
|
||||
- Electric guitar recorded on diverse hardware (guitars, amps, mics, environments).
|
||||
- Six single-note techniques: alternate picking, palm mute, vibrato, harmonics, pinch harmonics, bends.
|
||||
- Also chords (maj, min, aug, dim, maj7, dom7, min7, m7b5; no power chords listed), scales and excerpts.
|
||||
- Hosted on Zenodo.
|
||||
— [arXiv 2501.03720](https://arxiv.org/html/2501.03720v1); [Zenodo](https://zenodo.org/records/14963133)
|
||||
- GOAT pairs audio with GP tabs including bend and palm-mute annotations, and names technique identification as a use case. — [arXiv 2509.22655](https://arxiv.org/html/2509.22655v1)
|
||||
- Classic: ISMIR 2015 electric-guitar technique detection in solo tracks, including bends, with accuracies from about 58% to 88% depending on technique (per search snippet). — [ISMIR 2015](https://www.ismir2015.uma.es/articles/119_Paper.pdf)
|
||||
|
||||
### Inferences
|
||||
- **Bends and vibrato** are best detected from the F0 contour (Basic Pitch pitch-bend output, or a monophonic F0 tracker on the lead cluster). Contour features survive distortion better than spectral-shape features, because pitch is preserved even when partials are smeared. This is an inference.
|
||||
- **Palm muting** under high gain correlates with short decay and reduced high-frequency energy. Train on GOAT's amp-augmented audio, which already has GP-format palm-mute labels.
|
||||
- **Pinch harmonics** produce a sudden high-pitch onset with a fundamental 2–4 octaves above the fretted note. Treat them as "pitch far above the playable fret plus a lead context" and assign them via the fingering decoder.
|
||||
|
||||
### Gaps
|
||||
- No technique-detection results were found on separated full-mix distorted stems.
|
||||
- No open technique-detection weights were confirmed.
|
||||
|
||||
## 8. Bass-specific
|
||||
|
||||
### Takeaway
|
||||
No dedicated research bass-tab or slap/pop detection model was found for 2023–26. Notation conventions (T/S for slap, P for pop; slaps on low strings, pops on high) can drive heuristic string assignment and technique marking.
|
||||
|
||||
### Cited Findings
|
||||
- Slap is notated as T (thumb) or S, and pop as P. P can conflict with pull-off notation. — [BassBuzz](https://www.bassbuzz.com/lessons/how-to-read-bass-tabs); [Soundslice](https://www.soundslice.com/help/en/creating/tablature/130/bass-popping-and-slapping/)
|
||||
- Slaps are typically on the lower E/A strings and pops on the higher D/G strings. — [Wikipedia: Slapping](https://en.wikipedia.org/wiki/Slapping_(music))
|
||||
- Guitar Pro 8 supports slap/pop marks and a "Slap Bass" playing style. — [Guitar Pro blog](https://www.guitar-pro.com/blog/p/53111-tutorial-how-to-create-a-slap-bass-tab-in-guitar-pro-8)
|
||||
- Abeßer (CMMR 2012) covers string detection for bass as well as guitar. — [CMMR 2012](https://cmmr2012.eecs.qmul.ac.uk/sites/cmmr2012.eecs.qmul.ac.uk/files/pdf/papers/cmmr2012_submission_70.pdf)
|
||||
|
||||
### Inferences
|
||||
- **Bass fingering:** a Viterbi with bass-specific costs (one-finger-per-fret span of about 4 frets at low positions, strong preference for staying in position, open-string use in rock). 4-, 5- or 6-string choice comes from the lowest pitch: below E1 means a 5-string or drop tuning.
|
||||
- **Slap/pop detection:** percussive high-frequency transient onsets on bass notes. Pitch and register can assign the thumb (low string) or the pop (high string). Training data could come from DadaGP bass tracks with slap/pop marks rendered via SynthTab-style synthesis. This is untested.
|
||||
|
||||
### Gaps
|
||||
- No 2023–26 bass-tab or slap/pop detection paper was found, so a targeted search is still needed.
|
||||
- IDMT-SMT-Bass and related classic datasets were not re-verified.
|
||||
|
||||
## 9. Heavily distorted guitar, power chords, chord-level transcription
|
||||
|
||||
### Takeaway
|
||||
Distortion is the hardest condition: GOAT, two-thirds distorted, is called the hardest dataset in a 2026 benchmark. Mitigations are amp/effect augmentation in training (2024), de-distortion preprocessing (2022–24), and treating rhythm parts at the chord level (root plus power chord or quality) rather than the note level, then expanding to a canonical voicing via a shape dictionary.
|
||||
|
||||
### Cited Findings
|
||||
- Harmonica (Sept 2026, submitted to ICASSP 2027):
|
||||
- Instrument-agnostic transcription with multi-depth harmonic convolution.
|
||||
- Nano model: 26.3K parameters, frame F1 0.796 on its development set, 1,622.5x real time, beating Basic Pitch by 14.6 points.
|
||||
- Per the search summary, it describes GOAT as the hardest dataset in its benchmark, with about two-thirds of its recordings distorted.
|
||||
- The abstract page did not show distorted-versus-clean numbers or a code link.
|
||||
— [arXiv 2609.04640](https://arxiv.org/pdf/2609.04640)
|
||||
- "Leveraging Real Electric Guitar Tones and Effects to Improve Robustness in Guitar Tablature Transcription Modeling" (Pedroza, Abreu, Corey, Roman, 2024):
|
||||
- Adds synthetic training data made from real guitar tones played through various effects.
|
||||
- Evaluates on a new professional solo-guitar set with many tones.
|
||||
- It reports improved robustness, but no numbers appear on the abstract page.
|
||||
— [arXiv 2405.14679](https://arxiv.org/pdf/2405.14679)
|
||||
- Distortion removal and recovery as preprocessing: "Distortion Audio Effects: Learning How to Recover the Clean Signal" (2022) and "Distortion Recovery: A Two-Stage Method for Guitar Effect Removal" (2024). — [arXiv 2202.01664](https://arxiv.org/pdf/2202.01664); [arXiv 2407.16639](https://arxiv.org/pdf/2407.16639)
|
||||
- EGDB-related: "Towards automatic transcription of polyphonic electric guitar music" (2022) introduced the EGDB dataset of DI guitar re-amped through multiple amp tones (classic, still relevant). — [arXiv 2202.09907](https://arxiv.org/pdf/2202.09907)
|
||||
- ACR (GitHub, bachelor's thesis): chord recognition aimed at heavily distorted electric guitar, with templates for major, minor and **power chord** for all 12 roots. — [GitHub Driemtax/ACR](https://github.com/Driemtax/ACR)
|
||||
- Guitar-TECHS chord set has no power chords. — [Zenodo](https://zenodo.org/records/14963133)
|
||||
|
||||
### Inferences
|
||||
- **Chord-level fallback for rhythm parts:**
|
||||
1. Detect frame-level "root + type" with types {5 (power), maj, min, …} on the rhythm-guitar cluster, together with onsets and palm-mute flags.
|
||||
2. Render each chord as its canonical shape, such as a 2- or 3-note power chord on the string set whose root matches the lowest detected pitch.
|
||||
3. Do this under the detected tuning.
|
||||
This exploits the fact that high-gain rhythm guitar is dominated by a few shapes. It also makes errors "musically plausible", which human-judged playability favors.
|
||||
- **Augmentation** is the cheapest win for distorted guitar: GOAT DI plus amp-sim and IR chains, EGDB-style re-amping, and the real-tone approach of 2405.14679.
|
||||
- **Bass as an anchor:** in metal, the bass usually doubles the rhythm-guitar root, and the bass stem is cleaner (about 14.6 SDR per the prior survey). The bass transcription can constrain power-chord roots in the guitar part.
|
||||
|
||||
### Gaps
|
||||
- No benchmark of high-gain full-mix tab transcription was found (consistent with the prior survey).
|
||||
- Harmonica's GOAT numbers and the 2405.14679 numbers were not retrieved.
|
||||
- No published comparison of chord-level versus note-level tab accuracy on distorted rhythm guitar was found.
|
||||
@@ -0,0 +1,242 @@
|
||||
# New and Emerging AMT Paradigms (2024 – Oct 2026) Beyond "Separate Stems, Then Transcribe Per Instrument"
|
||||
|
||||
Scope note: the prior survey (`reports/Music transcription tooling survey.md`) already covers Demucs/BS-RoFormer, Basic Pitch, MuScriptor, YourMT3+, TART (arXiv 2609.11904), GOAT, SynthTab, midi2tab_Deconvolutions, and Noise-to-Notes (drums, MERT encoder, code not found). These notes add things that survey does not include. The research used about 17 search/fetch calls, so several sub-questions are only partly answered. See the Gaps sections.
|
||||
|
||||
## 1. Foundation-model / self-supervised audio features for transcription
|
||||
|
||||
### Takeaway
|
||||
The clearest 2025–26 evidence that self-supervised music encoders help transcription comes from the 2025 AMT Challenge. Its winner (MIROS) put a **MusicFM** encoder into YourMT3+ and scored 0.600 multi-instrument note-onset F1, against 0.393 for the MT3 baseline. However, the paper does not separate the encoder's contribution from the other changes. Guitar-specific probing evidence is thin: MERT has been used for guitar *strum/rhythm* detection, but I found no published MERT-vs-MuQ comparison for guitar or bass note transcription.
|
||||
|
||||
### Cited Findings
|
||||
- **2025 AMT Challenge** (NeurIPS 2025 AI for Music Workshop; arXiv 2603.27528, CC BY 4.0). Eight teams submitted valid solutions and two beat MT3. The test set was 76 newly composed pieces of about 20 s each, with 8 orchestral/piano instruments and at most 3 instruments per piece, rendered with FluidSynth/FluidR3 GM. The metric is multi-instrument note-onset F1 (instrument + pitch + onset within ±50 ms) — [arXiv HTML](https://arxiv.org/html/2603.27528v1)
|
||||
- Winner **MIROS**: YourMT3+ encoder-decoder with a **MusicFM** encoder (conformer pretrained with BEST-RQ masked-token modelling), a recurrent adapter conditioned on learned instrument-group embeddings, and **parallel per-instrument-group T5-style decoders**. It reported 0.83 F on Slakh2100 at about 370M params. Challenge scores: MIROS F1 0.5998 (P 0.656 / R 0.572); YourMT3-YPTF-MoE-M 0.5938; YourMT3-YPTF-S 0.5581; MT3 baseline 0.3932 — [arXiv HTML](https://arxiv.org/html/2603.27528v1)
|
||||
- Other lessons from the challenge: performance dropped significantly as instrument count rose, and MIROS precision fell from 0.907 on solo pieces to 0.464 on 3-instrument pieces. Models hallucinated instruments that were not present ("instrument leakage"). Cross-dataset augmentation gave only marginal gains. Common techniques were MoE, hierarchical time-frequency attention, random-projection quantizers, and auxiliary onset/offset losses. The page gave no team code links — [arXiv HTML](https://arxiv.org/html/2603.27528v1)
|
||||
- Guitar rhythm/strum transcription in polyphonic music (arXiv 2510.05756) chose **MERT** for its frame-level strength, especially on beat tracking. It compared frozen probing (a lightweight head on a learnable weighted average of all layers) with fine-tuning — [arXiv 2510.05756](https://arxiv.org/html/2510.05756v1)
|
||||
- MARBLE benchmarks MERT, MusicFM and MuQ. All three use masked-LM pretraining with different tokenisation — [search result summarising arXiv 2608.14819](https://arxiv.org/pdf/2608.14819)
|
||||
- A 2026 arXiv paper, "What Makes a Good Layer?", assesses layer-wise properties of music foundation models. I could not verify its transcription findings — [arXiv 2608.14819](https://arxiv.org/pdf/2608.14819)
|
||||
- Counterpoint, small specialised models: **Harmonica** (arXiv 2609.04640, submitted to ICASSP 2027) uses multi-depth harmonic convolution for instrument-agnostic transcription. Its *nano* model (26.3K params, 1,622× real-time) reaches 0.796 frame F1 on its dev set, which it says is +14.6 pp over Basic Pitch. The x-large model claims SOTA instrument-agnostic results, with no numbers in the abstract. No code or weights were stated — [arXiv 2609.04640](https://arxiv.org/abs/2609.04640)
|
||||
- From the prior survey: Noise-to-Notes (drums) uses a 330M MERT encoder.
|
||||
|
||||
### Inferences
|
||||
- The most practical experiment for us is a **MusicFM or MERT encoder with a lightweight onset/frame/pitch head fine-tuned on guitar and bass stems** (GOAT, GuitarSet, EGDB, plus synthetic DadaGP renders), compared against Basic Pitch on our GP-reference benchmark. MIROS shows the encoder plugs into YourMT3+, which we already use.
|
||||
- Harmonica suggests a well-designed harmonic inductive bias can beat Basic Pitch at tiny size. That makes it a candidate replacement for the *browser draft* tier if weights are released. The SSL-encoder route suits the *server quality* tier.
|
||||
- The challenge's precision collapse with polyphony supports keeping the separation front-end. Separation reduces the per-model polyphony and instrument count, which is where all models fail.
|
||||
|
||||
### Gaps
|
||||
- No published numbers for MERT/MuQ/MusicFM/BEATs/Jukebox probing on **guitar or bass note transcription**. I found nothing that answers whether they help low-data instruments, beyond the indirect MIROS evidence.
|
||||
- MIROS code and weights not found. MusicFM, MERT and MuQ licenses not verified in this pass. From memory, MERT checkpoints are CC BY-NC 4.0, but this needs verifying before any commercial use.
|
||||
- Jukebox-embedding transcription probes (e.g. Castellon et al. 2021) are pre-2024 and were not re-checked.
|
||||
|
||||
## 2. Large audio-language models (LALMs) and LLM-based transcription
|
||||
|
||||
### Takeaway
|
||||
Every source found says general LALMs (GPT-4o, Gemini 2.5, Qwen2-Audio) are **far from usable for note-level transcription**. They handle music given as symbolic input (MIDI/ABC) well, but perception from the waveform is the bottleneck. One audio-to-score benchmark reports 41% accuracy for a GPT-4o pipeline. I found no 2025–26 paper showing a LALM competitive with MT3-class models on standard transcription metrics.
|
||||
|
||||
### Cited Findings
|
||||
- A 2025 study of core music perception tasks found Gemini models near ceiling with **MIDI input**, but "audio reduced accuracy across tasks", so perception from the waveform is the main bottleneck. Gemini 2.5 Pro and Flash both scored significantly higher on MIDI than on audio — [arXiv 2510.22455](https://arxiv.org/pdf/2510.22455)
|
||||
- MuseAgent-1: GPT-4o (55.9%) and Gemini 2.5 Pro (53.1%) are only marginally above random on its performance-audio reasoning benchmark. The authors conclude raw audio is not enough for fine-grained music understanding — [arXiv 2601.11968](https://arxiv.org/pdf/2601.11968)
|
||||
- A music-to-score transcription benchmark (reported via search snippet) puts an aggregated GPT-4o system at 41.0% accuracy, and its authors say low audio pitch-recognition accuracy makes audio-to-score transcription unlikely to succeed — [search summary; source among arXiv 2510.22455 / 2607.06015](https://arxiv.org/html/2607.06015v1) (attribution unverified; treat as indicative)
|
||||
- **HMU-Bench** (arXiv 2610.15773, omni models) has an ABC part and an audio part (460 question pairs on 92 recordings). The best models do well on notation but are uniformly low on audio. Rankings across the two parts barely correlate (Spearman −0.04). Its audio questions cover chords and sections, not full transcription — [arXiv 2610.15773](https://arxiv.org/html/2610.15773)
|
||||
- **ABC-Eval** (ICASSP 2026): text-only LLMs on ABC scores across 10 sub-tasks and 1,086 samples. Seven SOTA LLMs show "notable limitations" in symbolic processing — [arXiv 2509.23350](https://arxiv.org/pdf/2509.23350)
|
||||
- Qwen2-Audio uses a Whisper-large-v3 encoder (one frame per ~40 ms) and a 7B Qwen LM. An informal (non-peer-reviewed) blog test found it weak at basic music tasks like tempo and instrument detection — [Qwen2-Audio report](https://arxiv.org/pdf/2407.10759); [TDS blog](https://towardsdatascience.com/exploring-music-transcription-with-multi-modal-language-models-af352105db56/)
|
||||
- Seq2seq piano transcription with a pretrained roll-based encoder and a hierarchical LM decoder (arXiv 2501.03038) is the "LM-decoder" paradigm that does work, but it is trained specifically for transcription — [arXiv 2501.03038](https://arxiv.org/pdf/2501.03038)
|
||||
|
||||
### Inferences
|
||||
- A ~40 ms encoder frame (Whisper-style) is too coarse for ±50 ms onset accuracy with dense notes. That is a structural reason general LALMs underperform.
|
||||
- LLMs are better used **downstream of transcription**, on the symbolic side, for:
|
||||
1. cleaning MIDI into readable notation (key/meter/spelling)
|
||||
2. ranking alternative tab fingerings
|
||||
3. explaining low-confidence regions to users
|
||||
|
||||
Even there, ABC-Eval shows text LLMs make symbolic errors, so their outputs need rule-based validation.
|
||||
- Do not invest in LALM-based transcription now. Revisit if a dedicated audio→MIDI/ABC LALM is published with MAESTRO/Slakh/GuitarSet numbers.
|
||||
|
||||
### Gaps
|
||||
- I found no current numbers for SALMONN, MuMu-LLaMA, LLark or Qwen2.5-Omni on note-level transcription. LLark (2023) is known for music captioning/QA, not transcription.
|
||||
- No head-to-head of GPT-4o and Gemini on audio→MIDI with mir_eval metrics was found.
|
||||
|
||||
## 3. Generative / diffusion approaches to transcription
|
||||
|
||||
### Takeaway
|
||||
**Noise2Fret** (Aug 2026) is the most relevant new result for us. It is a diffusion audio-to-tab model with **public code and trained models**. On GOAT (electric guitar) it lifts Tab F1 from about 0.66 (TabCNN/FretNet) to **0.795**, and on GuitarSet from about 0.72 to **0.845**. It is also ~10–50× cheaper in FLOPs than those baselines. Generative models seem to win because they model the joint distribution of string/fret choices, which is multi-modal because the same pitch can be played in several positions, and because training adds musical/playability priors through auxiliary losses.
|
||||
|
||||
### Cited Findings
|
||||
- **Noise2Fret** — Simionato and Bigo (LaBRI, Bordeaux), arXiv 2608.30854, 31 Aug 2026, preprint. It diffuses in a continuous embedding of discrete string-fret targets using a 1D conv U-Net with a self-attention bottleneck. Conditioning: STFT magnitude, spectral flux, brightness and raw audio. It adds five auxiliary losses: pitch-class distance, circle-of-fifths distance, positional distance, string similarity and hand-span feasibility — [arXiv](https://arxiv.org/html/2608.30854)
|
||||
- Results (pitch F / tab F / TDR) — [arXiv](https://arxiv.org/html/2608.30854):
|
||||
|
||||
| Dataset | Noise2Fret (+aux) | TabCNN | FretNet |
|
||||
|---|---|---|---|
|
||||
| GuitarSet | 0.856 / 0.845 / 0.988 | 0.820 / 0.717 / 0.860 | 0.818 / 0.727 / 0.871 |
|
||||
| GOAT | 0.800 / 0.795 / 0.993 | 0.664 / 0.656 | 0.669 / 0.662 |
|
||||
|
||||
On GOAT, Noise2Fret's FNR is 0.165 against 0.293 for TabCNN.
|
||||
- Cost: ~15M params, ~341M FLOPs per pass (TabCNN 3.4B, FretNet 17.4B). RTF 0.67 on an RTX 3090. Code and trained models are at github.com/RiccardoVib/Noise2Fret, with code license not stated (paper text is CC BY-SA 4.0). No synthetic data and no MERT were used — [arXiv](https://arxiv.org/html/2608.30854); [GitHub](https://github.com/RiccardoVib/Noise2Fret); [Zenodo](https://zenodo.org/records/23037260)
|
||||
- Hand-span feasibility is a training loss only. No playability metric is reported — [arXiv](https://arxiv.org/html/2608.30854)
|
||||
- Third-party projects are already reusing Noise2Fret playability costs in a fingering engine with AlphaTex export (unreviewed open-source PR) — [GitHub PR](https://github.com/quantumq1981/Tab-Translator-Pro/pull/116)
|
||||
- **GuitarFlow** (arXiv 2510.21872) is the reverse direction: flow-matching synthesis of realistic electric guitar audio from tablature with style transfer. It is a candidate renderer for synthetic training data and analysis-by-synthesis — [arXiv 2510.21872](https://arxiv.org/pdf/2510.21872)
|
||||
- Joint piano rendering and transcription by disentangling score content from performance style (arXiv 2509.23878) is a generative, bidirectional model of audio and score — [arXiv 2509.23878](https://arxiv.org/pdf/2509.23878)
|
||||
|
||||
### Inferences
|
||||
- **Top experiment:** run Noise2Fret on our separated guitar stems and compare against "Basic Pitch → Viterbi assigner" on the GP-reference benchmark. Check the code license first. Caveat: GuitarSet and GOAT are solo/DI-ish recordings, so real gains on separated, distorted mix stems are unknown. We would likely need to fine-tune on amp-rendered or stem-like audio.
|
||||
- Diffusion sampling gives multiple candidate tabs per segment for free. Their disagreement is a natural **uncertainty signal** (see section 8).
|
||||
- The auxiliary-loss idea transfers to bass (4/5-string) with retargeted string counts.
|
||||
|
||||
### Gaps
|
||||
- DiffRoll (2022, piano, pre-2024) was not re-checked. Noise-to-Notes is already covered in the prior survey.
|
||||
- No Noise2Fret numbers exist on polyphonic mixes, separated stems or high-gain distortion.
|
||||
|
||||
## 4. Joint separation + transcription / multi-task models
|
||||
|
||||
### Takeaway
|
||||
Few new 2024–26 joint separation+transcription models appeared. The active directions are:
|
||||
1. timbre-clustered "transcribe and separate at the note level" (arXiv 2509.12712)
|
||||
2. analysis-by-synthesis drum models that learn separation from transcription labels alone (Inverse Drum Machine, 2025)
|
||||
3. multi-decoder MT3 descendants that do instrument-grouped decoding (MIROS)
|
||||
|
||||
Classic joint models (Jointist, Cerberus, Lin et al.) are 2021–23.
|
||||
|
||||
### Cited Findings
|
||||
- **Jointist** (2023, older) jointly trains instrument recognition, transcription and separation, and reports that each task improves the others — [arXiv 2302.00286](https://arxiv.org/html/2302.00286)
|
||||
- Lin et al. 2021 (older): a unified zero-shot separation/transcription/synthesis model. The transcribed score is a "powerful auxiliary" for separation — [arXiv 2108.03456](https://arxiv.org/abs/2108.03456)
|
||||
- **Lightweight multi-instrument transcription** (arXiv 2509.12712, 2025) extends a timbre-agnostic backbone with a timbre encoder and **note-level deep clustering**. This gives joint transcription and dynamic separation of arbitrary instruments ("timbre-separated transcription" at the symbolic level) — [arXiv 2509.12712](https://arxiv.org/html/2509.12712v3)
|
||||
- **Inverse Drum Machine** (arXiv 2505.03337, v2 Sep 2025) jointly optimises drum transcription and one-shot sample synthesis. It convolves onsets with synthesised one-shots to rebuild stems and trains on mixture reconstruction using only transcription labels (StemGMD). It claims separation "comparable to SOTA supervised methods" that need isolated stems. No numbers or code on the abstract page — [arXiv](https://arxiv.org/abs/2505.03337)
|
||||
|
||||
### Inferences
|
||||
- Note-level timbre clustering (2509.12712) directly targets one of our hard problems: **splitting multiple guitars** that stem separators lump into one "guitar/other" stem. Worth evaluating as a post-separation step that clusters transcribed notes into Guitar 1 and Guitar 2.
|
||||
- For drums, Inverse-Drum-Machine-style reconstruction gives a self-supervised consistency check. Resynthesise the predicted hits and compare with the drum stem.
|
||||
|
||||
### Gaps
|
||||
- No 2024–26 "transcription-informed separation" result with SDR+F1 on rock/pop guitar was found.
|
||||
- Code availability for 2509.12712 was not checked.
|
||||
|
||||
## 5. Synthetic-data strategies and the sim-to-real gap
|
||||
|
||||
### Takeaway
|
||||
Synthetic rendering (SynthTab, already in the prior survey) remains the main way to scale guitar data. New pieces are:
|
||||
- amp/effects augmentation frameworks such as Open-Amp
|
||||
- adversarial domain-confusion sim-to-real training (Sato and Akama, ICME 2024)
|
||||
- a WASPAA 2025 synthetic-vs-real guitar comparison (Kusaka and Maezawa, findings not retrieved)
|
||||
- flow-matching guitar renderers (GuitarFlow) that could close the timbre gap
|
||||
|
||||
I found no paper that uses NAM captures specifically to train a transcriber.
|
||||
|
||||
### Cited Findings
|
||||
- **Open-Amp** (arXiv 2411.14972): a synthetic data framework that renders guitar (IDMT-Guitar DI) through amp and pedal models on the fly, for audio-effect foundation models. It is applied to distortion-effect classification and effect emulation — [arXiv](https://arxiv.org/html/2411.14972v1)
|
||||
- **EGDB**: electric guitar processed by various digital amp models, made for transcription — [search summary](https://arxiv.org/html/2411.14972v1)
|
||||
- Sato and Akama (ICME 2024): annotation-free transcription from scalable synthetic data plus **adversarial domain confusion** — [cited in search results; TART reference list](https://arxiv.org/pdf/2609.11904)
|
||||
- Kusaka and Maezawa (WASPAA 2025): compares guitar transcription trained on synthetic vs. real audio. Only the title and venue were retrieved, not the results — [TART reference list](https://arxiv.org/pdf/2609.11904)
|
||||
- GOAT amp-augments 5.9 h of DI to 29.5 h (prior survey). Noise2Fret on GOAT reached Tab F1 0.795 — [arXiv 2608.30854](https://arxiv.org/html/2608.30854)
|
||||
- **GuitarFlow** (arXiv 2510.21872): realistic electric-guitar synthesis from tabs via flow matching and style transfer — [arXiv](https://arxiv.org/pdf/2510.21872)
|
||||
- DDSP string-wise polyphonic guitar synthesis from string-wise MIDI (arXiv 2309.07658, 2023): six per-string DDSP voices summed through a trainable reverb — [arXiv](https://arxiv.org/pdf/2309.07658)
|
||||
- Velocity estimation across instruments via **differentiable SoundFont proxies** (arXiv 2608.08985) — [arXiv](https://arxiv.org/pdf/2608.08985)
|
||||
|
||||
### Inferences
|
||||
Proposed pipeline:
|
||||
1. DadaGP/GP files → per-string render, using a DDSP or GuitarFlow-style renderer or sampled guitars.
|
||||
2. Randomised NAM captures, cab IRs and pedals (Open-Amp-style).
|
||||
3. **Mix with other DadaGP tracks, then run through our actual separator.**
|
||||
4. Train on the *separated* output.
|
||||
|
||||
Step 4 teaches the transcriber the separator's artefacts, which is probably the largest sim-to-real gap for our pipeline. Add a domain-confusion loss using unlabeled real stems.
|
||||
|
||||
### Gaps
|
||||
- No quantitative sim-to-real numbers for guitar were retrieved (the Kusaka and Maezawa results are missing). No study uses NAM models for transcription data. Licenses for NAM community captures vary per capture.
|
||||
|
||||
## 6. Weak / noisy supervision and alignment-based training
|
||||
|
||||
### Takeaway
|
||||
This is the most mature route to large real-audio training sets, and it fits us well because we can pair user .gp files and public tabs with recordings:
|
||||
- **NoteEM** (2022): alignment plus EM pseudo-labels
|
||||
- **Riley et al. 2024**: aligning tabs to piano-model activations, then training a high-res guitar model, with zero-shot GuitarSet SOTA
|
||||
- **CountEM** (ISMIR 2025): supervision from note *histograms* only, no alignment. Fine-tuning on GAPS raised GuitarSet accuracy from 66.2% to 84.6%.
|
||||
- Cycle-consistent training with almost no supervision (2026): 64.81 GuitarSet frame F1 from unlabeled guitar audio
|
||||
|
||||
### Cited Findings
|
||||
- **NoteEM** (Maman and Bermano, ICML 2022, older) trains a transcriber while aligning unaligned scores to performances. It adds pseudo-labels and pitch-shift augmentation, and its guitar setup used ~5 h of self-collected recordings evaluated on GuitarSet — [PMLR](https://proceedings.mlr.press/v162/maman22a/maman22a.pdf); [arXiv](https://ar5iv.labs.arxiv.org/html/2204.13668)
|
||||
- **CountEM / "Count The Notes"** (Yaffe, Maman, Müller, Bermano; ISMIR 2025; arXiv 2511.14250, CC BY 4.0):
|
||||
- Method: uses **note-occurrence histograms** as the training signal in EM, with no local alignment (no DTW or soft-DTW). It matches or beats prior weakly supervised methods on piano, guitar and multi-instrument data — [arXiv abs](https://arxiv.org/abs/2511.14250)
|
||||
- Search snippet: fine-tuning on GAPS with whole-track histogram supervision improved GuitarSet accuracy from 66.2% to 84.6% — [arXiv PDF](https://arxiv.org/pdf/2511.14250)
|
||||
- Project page: [yoni-yaffe.github.io/count-the-notes](https://yoni-yaffe.github.io/count-the-notes)
|
||||
- **High-resolution guitar transcription via domain adaptation** (Riley, Edwards, Dixon; arXiv 2402.15258, 2024): adapts Kong et al.'s high-res piano model to guitar by aligning professional transcriptions (~60 train pieces by F. Leduc) to the piano model's activations, NoteEM-style. It claims SOTA zero-shot on all of GuitarSet (no figure on the page). The dataset is on Zenodo; no code or weights are linked — [project page](https://xavriley.github.io/HighResolutionGuitarTranscription/)
|
||||
- **Music Transcription with (Almost) No Supervision** (arXiv 2605.24193, 2026): cycle-consistent learning on unpaired audio and scores reaches **64.81 frame F1 on GuitarSet** from unlabeled guitar audio plus minimal piano supervision. Without any paired data, training admits pitch-shifted degenerate solutions — [arXiv](https://arxiv.org/html/2605.24193v1)
|
||||
- "Just Label the Repeats" (arXiv 2411.07428): low-effort annotation for in-the-wild audio-to-score alignment — [arXiv](https://arxiv.org/pdf/2411.07428)
|
||||
|
||||
### Inferences
|
||||
- **Highest-leverage data strategy for us:** gather (recording, Guitar Pro tab) pairs, from user uploads with consent or from licensed tab sites. Run our separator, then train with CountEM (robust to structural mismatch such as repeats and tab errors) or NoteEM alignment on the *separated stems*. This produces real-audio, real-mix training data for guitar, bass and drums, and directly targets the gap between solo-guitar test sets and dense mixes.
|
||||
- CountEM's histogram loss tolerates imperfect user tabs (wrong repeats, missing sections) better than strict DTW alignment.
|
||||
|
||||
### Gaps
|
||||
- Code release for CountEM and 2605.24193 not confirmed. No per-instrument (bass, drums) results for these methods in mixes.
|
||||
|
||||
## 7. Test-time adaptation, domain adaptation, pseudo-labelling
|
||||
|
||||
### Takeaway
|
||||
I found **no published test-time adaptation (TTA) method specifically for AMT**. Pseudo-labelling exists only at training time (NoteEM, teacher-student vocal transcription). General TTA work warns about confirmation bias from self-generated pseudo-labels.
|
||||
|
||||
### Cited Findings
|
||||
- NoteEM uses network self-predictions as pseudo-labels alongside alignment, at training time only — [arXiv 2204.13668](https://arxiv.org/abs/2204.13668)
|
||||
- Teacher-student vocal transcription converts frame-level pseudo-labels from pitch estimators into note-level labels (2022, older) — [arXiv 2203.13422](https://arxiv.org/pdf/2203.13422)
|
||||
- Improved self-training for TTA (CVPR 2024) fixes unreliable pseudo-labels with multiple augmented views, feature-similarity graphs and parameter moving averages — [CVF](https://openaccess.thecvf.com/content/CVPR2024/papers/Ma_Improved_Self-Training_for_Test-Time_Adaptation_CVPR_2024_paper.pdf)
|
||||
- A 2026 survey of self-improving test-time methods notes confirmation-bias risk when a model trains on its own predictions — [arXiv 2609.01679](https://arxiv.org/pdf/2609.01679)
|
||||
- Audio TTA in wild acoustic conditions (2023) — [arXiv 2310.09505](https://arxiv.org/pdf/2310.09505)
|
||||
|
||||
### Inferences
|
||||
- A cheap per-song TTA to try: adapt only BatchNorm/LayerNorm statistics, or a small adapter, on the song's stem. Use high-confidence pseudo-labels with augmentation agreement (pitch-shift and EQ views must agree), and stop early. This is an open research opportunity rather than an established recipe, so its benefit must be measured.
|
||||
- Song-level self-consistency (repeated riffs and choruses should transcribe identically) is a music-specific pseudo-label filter not seen in the literature found.
|
||||
|
||||
### Gaps
|
||||
- No AMT-specific TTA papers or numbers were found.
|
||||
|
||||
## 8. Analysis-by-synthesis / render-and-compare
|
||||
|
||||
### Takeaway
|
||||
Render-and-compare has working 2023–25 examples:
|
||||
- the Drexel semi-supervised tab work: predicted tab → differentiable guitar synth → reconstruction loss on 27 h of unlabeled YouTube guitar
|
||||
- the Inverse Drum Machine for drums
|
||||
- DDSP mixture-model parameter fitting
|
||||
|
||||
None reports a large accuracy jump on real mixes. It is most useful as a training signal on unlabeled data and as a verification/uncertainty signal.
|
||||
|
||||
### Cited Findings
|
||||
- Drexel (Cwitkowitz et al. line) "Leveraging unlabeled data to improve guitar tablature transcription": predicted tabs drive a differentiable guitar synthesiser, training minimises reconstruction error, and it introduces 27+ h of unlabeled YouTube solo guitar. This is a thesis, and the snippet was partly garbled — [Drexel](https://researchdiscovery.drexel.edu/view/pdfCoverPage?instCode=01DRXU_INST&filePid=13549920770004721&download=true)
|
||||
- Differentiable acoustic guitar model for string-specific polyphonic synthesis (IEEE ICASSP 2023) — [IEEE](https://ieeexplore.ieee.org/document/10248159/); [arXiv 2309.07658](https://arxiv.org/pdf/2309.07658)
|
||||
- DDSP mixture model fits pretrained source synths to an observed mixture by gradient descent (2022, older) — [arXiv 2202.00200](https://arxiv.org/pdf/2202.00200)
|
||||
- Inverse Drum Machine (2025): see section 4 — [arXiv 2505.03337](https://arxiv.org/abs/2505.03337)
|
||||
- Cross-instrument velocity via differentiable SoundFont proxies (2026) — [arXiv 2608.08985](https://arxiv.org/pdf/2608.08985)
|
||||
|
||||
### Inferences
|
||||
- A practical non-differentiable version for the product: resynthesise each transcribed bar (sampled or DDSP guitar/bass, drum one-shots) and compute a spectral/chroma/onset distance to the stem per bar. Bars with high residual are flagged for review or re-decoded, for example by sampling more Noise2Fret candidates and keeping the best match. No gradient training is needed, and it doubles as a confidence score.
|
||||
|
||||
### Gaps
|
||||
- No quantitative gain figures were retrieved for render-and-compare refinement on real recordings.
|
||||
|
||||
## 9. Confidence / uncertainty estimation for flagging dubious notes
|
||||
|
||||
### Takeaway
|
||||
I found **almost no dedicated work on calibrated note-level confidence in AMT**. Older work notes miscalibration: models are overconfident on sustain and underconfident on off frames. A 2026 MIR paper (TCP_α) offers post-hoc confidence heads for frozen MIR classifiers, framed as failure prediction under domain shift. That framing fits a "flag for human review" feature directly.
|
||||
|
||||
### Cited Findings
|
||||
- **TCP_α** (arXiv 2608.20326): a margin-controlled confidence target for post-hoc confidence heads (ConfidNet-style) on frozen MIR classifiers, with theoretical guarantees, framed as failure prediction and examined under domain shift — [arXiv](https://arxiv.org/html/2608.20326v1)
|
||||
- Autoregressive multi-state piano model (2020, older): overconfident on sustain, underconfident on off frames. The mismatch hurt beam search, which did worse than greedy — [arXiv 2010.01104](https://arxiv.org/pdf/2010.01104)
|
||||
- Audio captioning (arXiv 2512.10170): a learned confidence head plus confidence-aware beam reranking improved quality and calibration (lower ECE and Brier score) — [arXiv](https://arxiv.org/pdf/2512.10170)
|
||||
|
||||
### Inferences
|
||||
Candidate signals for a "dubious note" score, combined into a per-note confidence, then calibrated (temperature scaling or isotonic regression) on our GP-reference benchmark:
|
||||
1. onset/frame posterior from the transcriber
|
||||
2. disagreement between diffusion samples (Noise2Fret) or between models (Basic Pitch vs. YourMT3+ vs. fine-tuned SSL model)
|
||||
3. render-and-compare residual (section 8)
|
||||
4. separation leakage estimate (energy of the same pitch in other stems)
|
||||
5. a TCP_α-style post-hoc head trained to predict errors on our reference set
|
||||
|
||||
### Gaps
|
||||
- No AMT calibration benchmarks (ECE for notes) were found. No user studies on review UIs driven by transcription confidence.
|
||||
|
||||
## Recommended experiment shortlist (ranked by expected value per effort; inference)
|
||||
|
||||
1. **Noise2Fret on separated guitar stems** vs. current Basic Pitch + Viterbi, on the GP benchmark. Then fine-tune on amp-rendered DadaGP/GOAT passed through our separator. Code exists; verify the license.
|
||||
2. **Weakly supervised real-data fine-tuning** (CountEM histograms or NoteEM alignment) using paired recordings and GP tabs, run on separated stems, for guitar and bass.
|
||||
3. **SSL encoder (MusicFM/MERT) with a light transcription head** for guitar and bass, following the MIROS precedent. Check encoder licenses.
|
||||
4. **Separator-in-the-loop synthetic data** (render → NAM/IR randomisation → mix → separate → train), plus a domain-confusion loss.
|
||||
5. **Note-level timbre clustering** (arXiv 2509.12712) to split multiple guitars.
|
||||
6. **Render-and-compare and multi-model disagreement** for per-note confidence and review flags. Calibrate on the benchmark.
|
||||
7. Skip LALM-based transcription for now. Use LLMs only for symbolic post-processing, with validation.
|
||||
@@ -0,0 +1,259 @@
|
||||
# Post-processing, ensembling, refinement, human-in-the-loop and evaluation methods for transcription accuracy and notation quality (as of Oct 2026)
|
||||
|
||||
Scope note: this builds on `/home/user/STSheets/reports/Music transcription tooling survey.md` and does not repeat it. That survey already covers the baseline chain, Beat This!, BTC chords, PM2S, alphaTab, Tab F1/TDR, the Hungarian track matching, and the "mix vs stem" ablation idea. Research budget was about 15 web calls, so several questions end in explicit gaps. Items marked **[older]** are from before 2023. Items marked **[unverified this session]** come from background knowledge and were not re-checked against a fetched page in this session; verify them before relying on them.
|
||||
|
||||
## 1. Separation ensembles and tricks; does better SDR mean better transcription?
|
||||
|
||||
### Takeaway
|
||||
Ensembling and overlap tricks add very little SDR: tenths of a dB at most, and about 0.02 dB for high overlap on a modern RoFormer. No 2023–2026 paper was found that measures note-F1 for guitar, bass or keys transcribed from separated stems against the mix. The nearest evidence (vocals → Whisper lyrics) is mixed: better separation helped on one dataset and made no difference on another. SDR also correlates only weakly with perceived quality, and artifact-focused metrics (SI-SAR) predict perception better for bass and drums. The project must measure stem-vs-mix itself rather than assume that SDR gains turn into note gains.
|
||||
|
||||
### Cited Findings
|
||||
- **Ensemble methods in the MSST / MVSEP ecosystem.** The community `ensemble.py` defines `avg_wave`, `median_wave`, `min_wave`, `max_wave` and FFT-domain variants (`avg_fft`, `min_fft`, `max_fft`, …). `avg_wave` is a weighted average: each prediction is multiplied by its weight and the sum is divided by the total weight. Newer commits choose min/max by absolute value. — [HF ensemble.py (MSST copy)](https://huggingface.co/spaces/TGPro1/YingMusic-SingerGPU/blob/main/src/third_party/MusicSourceSeparationTraining/ensemble.py); [SESA commit](https://huggingface.co/spaces/NeoPy/SESA_Fast_Separation/commit/dc08c30578ba6a4589e070d5ade1b7a39010c2d5)
|
||||
- **MVSEP API** offers an option to "include intermediate results and max_fft, min_fft". In community practice, max_fft is used for a "fuller" stem with more bleed and min_fft for a "cleaner" stem with more loss. — [MVSEP full API](https://hk.mvsep.com/full_api)
|
||||
- **Ensemble gains are small.** MVSEP's Ensemble (2025.06) is listed at vocals SDR 11.93 and instrumental 18.23 on its Multisong set. The API's default ensemble reports an "SDR average: 13.67 (v. 2025.06.30)". — [MVSEP algorithms](https://de.mvsep.com/algorithms); [MVSEP full API](https://hk.mvsep.com/full_api)
|
||||
- **Overlap barely matters.** BS PolarFormer vocals go from 11.75 SDR at overlap 2 to 11.77 at overlap 8. MVSEP itself says high overlap "significantly increases system load while providing only a minor quality improvement". The API offers 50% (default) and 87.5% overlap. — [MVSEP algorithm page 128](https://hk.mvsep.com/algorithms/128)
|
||||
- **Demucs "shift trick".** This test-time augmentation averages predictions over random time shifts of the input (`--shifts N`). The Demucs authors describe it as giving roughly +0.2 dB SDR at N× the compute. **[older, 2019; unverified this session]** — [Demucs paper arXiv 1911.13254](https://arxiv.org/pdf/1911.13254)
|
||||
- **SDR vs perception.** The 2025 "Musical Source Separation Bake-Off" listening study found that SDR-family metrics track human judgments poorly and that no single metric works for every stem type. SI-SAR (an artifacts measure) better predicts listener ratings for drums and bass. — [arXiv 2507.06917](https://arxiv.org/abs/2507.06917)
|
||||
- **Separation → downstream transcription (lyrics, the closest evidence found).** For Whisper lyrics transcription, WER "consistently decreases as separation quality improves" on MUSDB-ALT. On Jam-ALT, WER stays essentially the same for original and separated audio. — [arXiv 2506.15514](https://arxiv.org/pdf/2506.15514)
|
||||
- **Some systems skip separation.** A 2024 model transcribes vocals directly from the mix, with no separation front-end, and beats MT3 and PerceiverTF on Slakh and URMP. — [arXiv 2407.04822](https://arxiv.org/pdf/2407.04822)
|
||||
- **Joint separation and transcription models** (Jointist, which conditions separation on predicted piano rolls; Cerberus) exist as an alternative to running the two stages in sequence. **[older]** — [Jointist arXiv 2206.10805](https://arxiv.org/pdf/2206.10805); [MT3 arXiv 2111.03017](https://arxiv.org/pdf/2111.03017)
|
||||
- **Guitar duets.** GuitarDuets (2025) offers a classical guitar duet dataset with note-level annotations of synthesized duets, plus a joint transcription and separation framework. It is relevant to the multi-guitar problem. — [arXiv 2507.01172](https://www.arxiv.org/pdf/2507.01172)
|
||||
- **Analysis-by-synthesis for drums.** The Inverse Drum Machine (2025) does joint drum transcription and separation via analysis-by-synthesis. It reports both transcription and separation metrics per class. — [arXiv 2505.03337](https://arxiv.org/pdf/2505.03337)
|
||||
- **User-guided generative separation** (2025) is a human-in-the-loop separation direction. — [arXiv 2507.01339](https://arxiv.org/abs/2507.01339)
|
||||
|
||||
### Inferences
|
||||
- An ensemble or high-overlap pass costs 2–4× the compute for at most a few tenths of a dB. That is not worth it for the browser tier. In the server tier it only makes sense for stems where the transcriber is artifact-sensitive (keys and guitar inside "other").
|
||||
- For transcription, artifacts (the SAR side) probably matter more than interference (the SIR side). Spurious harmonics and bleed show up as ghost notes, while slight interference is often filtered by note thresholds. That would argue for `min_fft`-style "clean" ensembles before transcription and `max_fft` "full" stems for listening. This is a hypothesis to test, not a finding.
|
||||
- **Multi-pass "residual" separation** has community precedent but no measured transcription benefit was found. In this approach, vocals are removed first (best vocal model), then a guitar or piano model runs on the instrumental. Treat it as an A/B in the benchmark.
|
||||
|
||||
### Gaps
|
||||
- No 2023–2026 paper was found that reports note-F1 for guitar, bass or piano transcribed from separated stems vs the mix. The 2024 "Source Separation & Automatic Transcription" paper (arXiv 2412.06703) is a pipeline description, with no stem-vs-mix numbers in its abstract.
|
||||
- No published numbers were found for polarity or channel-flip TTA in RoFormer models, or for the SDR gain of residual multi-pass separation.
|
||||
- MVSEP forum discussions of ensemble recipes were not fetched.
|
||||
|
||||
## 2. Transcription ensembles (voting / ROVER-style / confidence-weighted / mix-vs-stem cross-check)
|
||||
|
||||
### Takeaway
|
||||
No published note-level multi-model voting or ROVER-style fusion for music transcription was found in 2023–2026. Fusion in the literature happens inside a single network (multi-feature or multi-spectrogram), or is human-machine (the ENSEMBLE crowd system, 2016). This is an open, low-risk engineering opportunity: align the note lists from several transcribers and vote.
|
||||
|
||||
### Cited Findings
|
||||
- **Fusion inside one network.** A 2025 piano model fuses Mel-cyclic and STFT spectrogram features in one network, which is not cross-model voting. — [MDPI Electronics 14(23):4720](https://www.mdpi.com/2079-9292/14/23/4720)
|
||||
- **Decision-level voting exists for instrument recognition, not notes.** A Swin-T voting ensemble of three transformers reaches micro/macro F1 0.66/0.62. **[older, 2022]** — [EURASIP 2022](https://link.springer.com/article/10.1186/s13636-022-00245-8)
|
||||
- **"Ensemble" (ISMIR 2016)** is a hybrid human-machine melody transcription system. It combines algorithm output with crowd edits in a web note editor and logged 53,411 user actions, about 60 per 3-second clip. **[older]** — [ISMIR 2016](https://archives.ismir.net/ismir2016/paper/000072.pdf)
|
||||
- **Note-level output in most AMT models** still comes from threshold and peak-picking over frame, onset and offset activations. That makes probability-level averaging across models feasible whenever activations are exposed. — [TASLP 2024 audio-visual piano AMT](https://dl.acm.org/doi/pdf/10.1109/TASLP.2024.3426303)
|
||||
- **ROVER background.** NIST ROVER (Fiscus 1997) aligns several ASR hypotheses into a word transition network and votes by frequency and/or confidence. **[older; unverified this session]** — [NIST ROVER paper](https://ieeexplore.ieee.org/document/659110)
|
||||
|
||||
### Inferences
|
||||
- **A practical "note-ROVER":**
|
||||
1. Run 2–3 transcribers, for example Basic Pitch on the stem, YourMT3+ or MuScriptor on the mix (NC licence, so reference only), and a specialist such as Transkun on the piano stem.
|
||||
2. Match notes across outputs with mir_eval-style bipartite matching (±50 ms onset, same pitch).
|
||||
3. Keep notes supported by ≥2 systems, or by 1 system with confidence above a threshold.
|
||||
4. Feed the agreement count into the UI as a confidence value.
|
||||
- **Mix-vs-stem cross-check as a filter.** A note found on the guitar stem but absent from a mix-level multi-instrument transcription, while present in another stem at the same pitch and time, is likely bleed. Assign it to the stem with higher energy at that pitch.
|
||||
- **Multi-resolution:** run the same model at different hop sizes or on time-stretched input (TTA) and average activations. This is cheap to test, but no published numbers were found.
|
||||
|
||||
### Gaps
|
||||
- No paper measuring the F1 gain from combining Basic Pitch, MT3-family and specialist models was found. The gain has to be measured in-house.
|
||||
|
||||
## 3. Musically-informed post-processing (key/harmony, octave, overtone ghosts, durations, structure)
|
||||
|
||||
### Takeaway
|
||||
Most of the literature here is older (2007–2021). It shows that priors help: music language models with error models for melody, repetition-aware LMs for drums, and self-similarity for meter and bar errors. Recent (2023–26) work is sparse, so these heuristics are mostly in-house engineering. Structure-aware consolidation of repeated sections has precedent (chorus pooling for lyrics, non-local statistics for piano) but no off-the-shelf tool.
|
||||
|
||||
### Cited Findings
|
||||
- **Statistical melody correction (ICASSP 2021).** Notes produced by a singing transcriber are corrected by combining a music LM with an explicit transcription-error model (insertions, deletions, substitutions). Viterbi decoding with an LSTM LM picks the most plausible output. **[older]** — [Hiramatsu et al. ICASSP 2021](https://eita-nakamura.github.io/articles/Hiramatsu_etal_TranscriptionCorrection_ICASSP2021.pdf); [IEEE](https://ieeexplore.ieee.org/document/9414249/)
|
||||
- **Non-local musical statistics (piano, 2020).** Self-similarity of pitch and rhythm between score segments, and how well a score repeats at different bar lengths, are used to fix tempo, meter and bar-line errors. The authors name these as the most common failure modes of audio-to-score transcription. **[older]** — [arXiv 2008.12710](https://arxiv.org/pdf/2008.12710)
|
||||
- **Chorus pooling for lyrics.** Pooling repeated chorus sections at feature or hypothesis level gives more accurate and more consistent transcription than transcribing each chorus alone. **[older, ICASSP 2014]** — [McVicar, Ellis, Goto](https://staff.aist.go.jp/m.goto/PAPER/ICASSP2014mcvicar.pdf)
|
||||
- **Transposition-invariant self-similarity** finds repeated sections in both symbolic and audio music. **[older, 2018]** — [arXiv 1806.08236](https://arxiv.org/abs/1806.08236v2)
|
||||
- **Repetition priors for drums** are covered in section 7.
|
||||
- **Score-informed velocity refinement (2025).** A lightweight Transformer correction module refines AMT velocity estimates, reaching SOTA velocity with HPT. It needs a score as side information. — [arXiv 2508.07757](https://arxiv.org/abs/2508.07757)
|
||||
- **Rule-based MusicXML error detection (2025).** "Detecting Notational Errors in Digital Music Scores" localizes rhythm and notation errors in MusicXML. It is usable as a post-export validator. — [arXiv 2510.02746](https://arxiv.org/pdf/2510.02746)
|
||||
- **Gains from MLMs are historically small.** The MIDI Degradation Toolkit authors note that music LMs have so far given only small gains in AMT systems. **[older, 2020]** — [arXiv 2010.00059](https://arxiv.org/pdf/2010.00059)
|
||||
|
||||
### Inferences
|
||||
These are concrete, cheap heuristics to implement and A/B test in-house. None comes from a single cited benchmark.
|
||||
- **Overtone ghosts:** delete a note at pitch p+12, p+19 or p+24 when a simultaneous note at p is much stronger, the candidate is short or quiet, and it sits on a mono instrument (bass, lead vocal).
|
||||
- **Octave correction for bass and vocal:** enforce instrument range (for example, 4-string E1–G4 when the user declares the tuning). Smooth octave jumps with a Viterbi over an octave-shift penalty.
|
||||
- **Out-of-key filter:** using BTC chords and a key estimate, down-weight short, low-confidence non-chord, non-scale notes. Never hard-delete them, because chromatic passing tones are real; flag them in the UI instead.
|
||||
- **Duration cleanup:** minimum note length (for example, a 64th note at the local tempo, or about 40–60 ms). Merge same-pitch fragments separated by short gaps (legato repair). Trim offsets to the next onset on mono instruments.
|
||||
- **Repetition consolidation:**
|
||||
1. Use allin1 sections or a beat-synchronous SSM to find repeated bars.
|
||||
2. Align symbolic content across repeats and do a majority vote per note slot.
|
||||
3. Optionally emit repeat signs or "x2" to improve readability.
|
||||
|
||||
This is the most promising underused idea. It mirrors chorus pooling, and its value for readability is high.
|
||||
|
||||
### Gaps
|
||||
- No 2023–2026 paper was found that quantifies key-aware filtering, ghost-note removal or minimum-duration rules on modern transcribers (Basic Pitch, YourMT3).
|
||||
|
||||
## 4. Symbolic music LMs as correctors or priors (AMT, MusicLang, NotaGen, CLaMP, etc.)
|
||||
|
||||
### Takeaway
|
||||
No 2023–2026 work was found that uses Anticipatory Music Transformer, MusicLang, NotaGen or ChatMusician to correct transcription errors. The infilling models (AMT, Composer's Assistant 2, MIDI-RWKV) are technically suited to "mask low-confidence region → infill conditioned on surrounding notes". Two pieces of evidence exist: BERT-APC uses a symbolic music LM for vocal pitch correction, and CLaMP 3 embeddings can work as a quality metric.
|
||||
|
||||
### Cited Findings
|
||||
- **Anticipatory Music Transformer (TMLR 2024).** Trained on Lakh MIDI, it infills a sequence given fixed control events and can anticipate controls up to 5 s ahead. Transcription correction is not addressed. — [arXiv 2306.08620](https://arxiv.org/abs/2306.08620)
|
||||
- **Composer's Assistant 2 (2024)** does interactive multi-track MIDI infilling with fine-grained user control. — [arXiv 2407.14700](https://arxiv.org/pdf/2407.14700)
|
||||
- **MIDI-RWKV (2025)** is an adaptable symbolic infilling model. — [arXiv 2506.13001](https://arxiv.org/pdf/2506.13001)
|
||||
- **BERT-APC (2025)** is a reference-free automatic pitch correction method that infers the intended pitch of detuned vocals from musical context with a symbolic music LM. It is the closest recent example of an LM-as-corrector. — [arXiv 2511.20006](https://arxiv.org/pdf/2511.20006)
|
||||
- **MIDI Degradation Toolkit** synthesizes errors (pitch shifts, onset shifts, inserted and removed notes) in clean MIDI so that models can detect, classify, locate and correct them. The authors propose this as AMT post-processing. **[older, ISMIR 2020]** — [arXiv 2010.00059](https://arxiv.org/pdf/2010.00059)
|
||||
- **Ycart's thesis** gives a taxonomy: transduction (which needs paired data) vs MLM-decoding (scoring candidate outputs with a separately trained symbolic model). **[older]** — [Ycart thesis](https://theses.eurasip.org/media/theses/documents/ycart-adrien-music-language-models-for-automatic-music-transcription.pdf)
|
||||
- **CLaMP 3 as an evaluator.** Its audio→ABC cross-modal similarity correlates with human preference for transcriptions at Spearman ρ=0.879 (see section 9). The same embedding can therefore act as a reranker. — [arXiv 2608.04511](https://arxiv.org/html/2608.04511)
|
||||
|
||||
### Inferences
|
||||
- A cheap, licence-friendly route is to train a small corrector on synthetic degradations, MDTK-style, applied to DadaGP or GOAT tabs. Pick degradations that match the transcriber's error profile, measured on the benchmark (octave errors, ghost harmonics, split notes). This avoids NC weights.
|
||||
- Use LM infilling only on low-confidence spans and show the result as a suggestion in the editor, not as an automatic replacement.
|
||||
- Check licences before production use. AMT weights are Apache-2.0 per the Stanford CRFM release **[unverified this session]**. MusicLang, NotaGen and ChatMusician licences were not checked.
|
||||
|
||||
### Gaps
|
||||
- No benchmark of symbolic-LM correction on real AMT output from 2023–26 was found. No licence check was done for MusicLang, NotaGen or ChatMusician.
|
||||
|
||||
## 5. General LLMs for notation cleanup
|
||||
|
||||
### Takeaway
|
||||
LLM benchmarks for symbolic music (2025–26) test understanding and QA, mainly in ABC. None was found for *correcting* notation (time signature, spelling, beaming). LLMs are also weak judges of octave errors. LLMs are therefore better used as orchestrators that call deterministic tools (music21, a pitch speller, a validator) than as direct MusicXML editors.
|
||||
|
||||
### Cited Findings
|
||||
- **ABC-Eval (2025)** has 10 sub-tasks and 1,086 samples, including error *detection*. Its samples were hand-picked to be free of syntax errors, so correction is not tested. — [arXiv 2509.23350](https://arxiv.org/pdf/2509.23350)
|
||||
- **MSU-Bench (ACL 2026)** has 1,800 QA pairs over real scores. Representing scores as ABC "substantially helps" LLMs compared with other encodings. — [ACL 2026](https://aclanthology.org/2026.acl-long.493.pdf)
|
||||
- **HMU-Bench (Oct 2026)** found that LLM-based grading can miss octave errors in symbolic output, and recommends event-level checks. — [arXiv 2610.15773](https://arxiv.org/html/2610.15773v1)
|
||||
- **MusICA-MetaBench (2026)** generates questions from MusicXML via music21 in ABC, audio and image forms. — [arXiv 2607.06015](https://arxiv.org/html/2607.06015v1)
|
||||
- **SSMR-Bench (2025)** is a synthetic sheet-music reasoning benchmark in which DeepSeek-R1 beats DeepSeek-V3. — [arXiv 2509.04059](https://arxiv.org/pdf/2509.04059)
|
||||
- **CoComposer (2025)** uses LLM agents to fix timing and formatting errors in ABC. It is a composition system, so the evidence that agents can repair bar-duration errors is anecdotal. — [arXiv 2509.00132](https://arxiv.org/pdf/2509.00132)
|
||||
- **Gemini 3.1 Pro as a judge** of transcription playback correlated with humans at ρ=0.970. The authors still advise against AudioLM judges: they cost about 13× more and depend on model version and prompt. — [arXiv 2608.04511](https://arxiv.org/html/2608.04511)
|
||||
|
||||
### Inferences
|
||||
- An LLM pass should output *edit operations* (for example "bar 12: change time signature to 6/8", "respell G# as Ab in bars 20–24"). A deterministic validator applies them and re-checks bar durations, using arXiv 2510.02746-style rules. Free-form generation of MusicXML is risky.
|
||||
|
||||
### Gaps
|
||||
- No benchmark was found for LLM enharmonic spelling, beaming or time-signature inference.
|
||||
|
||||
## 6. Rhythm quantization and score typesetting (quantizers, tuplets, meter, spelling, voice/staff)
|
||||
|
||||
### Takeaway
|
||||
Two newer options beat PM2S on parts of the problem. The beat-conditioned T5 quantizer (Apr 2026, piano plus jazz guitar) takes beats as input, which fits the Beat This! pipeline. MIDI2ScoreTransformer (ISMIR 2024, piano, MusicXML out) gives the best notation among modular pipelines in a 2026 human-anchored study. Pitch spelling is essentially solved for tonal music (ps13 ≈ 99.3–99.4%; PKSpell is better on MuseData). Voice/staff separation has a 2024 GNN. Nearly all of this is piano- or classical-trained.
|
||||
|
||||
### Cited Findings
|
||||
- **Beat-conditioned T5 quantizer (Wachter, Murgul, Heizmann; arXiv 2604.22290, Apr 2026).**
|
||||
- Input is performance MIDI plus beat and downbeat annotations. Tokens are pitch, onset and note value, plus a new-measure token. It works on two-bar segments with beam width 5.
|
||||
- The beat grid is interpolated to 12 ticks per beat, covering straight values and triplets from 16th-triplets up. It handles 6/8 with 18 ticks.
|
||||
- ASAP: onset F1 97.3%, note-value accuracy 83.3%. Leduc jazz guitar (GP files converted to MIDI and aligned): onset F1 92.1%, NV acc 90.2%.
|
||||
- MUSTER onset/offset error: 12.30/28.30. Baselines: transformer "End-to-End" [18] 15.55/23.84, HMM+heuristics 22.58–25.02 / 29.21–29.84, Finale 31.85/45.34, MuseScore 47.90/49.44.
|
||||
- It assumes a one-to-one note correspondence, so it cannot add or remove notes. It has no voices, no 32nd notes and no irregular meters, and it was not tested with noisy beats.
|
||||
- No code release was found.
|
||||
|
||||
— [arXiv 2604.22290](https://arxiv.org/html/2604.22290v1)
|
||||
- **Earlier version (arXiv 2508.19262, 2025)** was trained on piano and guitar performances and claims SOTA on MUSTER. — [arXiv 2508.19262](https://arxiv.org/pdf/2508.19262)
|
||||
- **MIDI2ScoreTransformer (Beyer & Dai, ISMIR 2024)** is an end-to-end RoFormer encoder-decoder from performance MIDI to MusicXML tokens (11 score attributes). It implicitly does beat tracking and quantization, and it beats HMM and earlier DL pipelines on MUSTER. Training data is ASAP only (piano). — [arXiv 2410.00210](https://arxiv.org/html/2410.00210v1); [ISMIR 2024 poster](https://ismir2024program.ismir.net/poster_158.html)
|
||||
- **MIDI-to-score converter choice drives notation quality.** In a 2026 evaluation of 24 pipelines (8 AMT models × music21 / MuseScore / M2ST):
|
||||
- Transkun+M2ST had the best OMR-NED among modular pipelines (86.61).
|
||||
- The end-to-end audio-to-score system Rubato had the lowest OMR-NED overall (72.30) but ranked 6th on playback.
|
||||
- MuseScore import was best for playback-faithful output.
|
||||
|
||||
— [arXiv 2608.04511](https://arxiv.org/html/2608.04511)
|
||||
- **Beat/downbeat tracking from performance MIDI** with an end-to-end transformer (2025). Multi-instrument support is listed as future work. — [arXiv 2507.00466](https://arxiv.org/pdf/2507.00466)
|
||||
- **Pitch spelling.**
|
||||
- ps13 spelt 99.31% of a 195,972-note corpus correctly (99.43% for a variant). **[older]** — [ps13, Goldsmiths](https://research.gold.ac.uk/id/eprint/1007)
|
||||
- PKSpell (2021, an RNN for spelling plus key signature) made 256 errors on MuseData, against 343 for CIV and 1,064 for ps13. The caveat is that ps13 and CIV were tuned on test data. PKSpell got 97% of global key signatures right. **[older]** — [arXiv 2107.14009](https://arxiv.org/pdf/2107.14009)
|
||||
- No newer (2024–26) deep pitch-spelling SOTA was found.
|
||||
- **Voice and staff separation.** "Cluster and Separate" (2024) is a GNN that assigns notes to voices and staves for engraving, with staff as a per-note binary label and voice as edge prediction. **[code licence not checked]** — [arXiv 2407.21030](https://arxiv.org/html/2407.21030v1)
|
||||
- **Classical hand split:** apply voice separation per hand, at most two voices per hand. **[older]** — [arXiv 2008.12710](https://arxiv.org/pdf/2008.12710)
|
||||
|
||||
### Inferences
|
||||
- For guitar and bass tabs, the T5-quantizer recipe fits the pipeline (Beat This! beats → quantize) and was shown on GP-derived jazz guitar data. Retraining it on DadaGP/GOAT aligned to synth audio is feasible. Its one-to-one constraint means it must run *after* note cleanup.
|
||||
- Swing: the 12-tick grid captures triplet swing. A per-section swing detector (the ratio of off-beat 8th positions) can decide whether to notate straight 8ths with a "swing" marking instead of triplets. That is a readability win. No cited benchmark exists.
|
||||
- For pickup bars and time signatures, use Beat This! downbeats. When the first downbeat comes after the first onset, emit an anacrusis bar. Validate bar lengths with arXiv 2510.02746-style rules.
|
||||
- Spelling: music21's built-in spelling and key analysis are fine for pop and rock. Spell relative to BTC chord roots for non-diatonic passages.
|
||||
|
||||
### Gaps
|
||||
- No public code or licence was found for the 2604.22290 quantizer. The licences of M2ST and the GNN voice-separation code were not checked. No guitar-specific tuplet or swing detection paper was found.
|
||||
|
||||
## 7. Drums post-processing
|
||||
|
||||
### Takeaway
|
||||
Repetition-aware drum language models are the main published post-processing approach (2007–2021). They regularize tatum-level outputs toward bar-periodic patterns, and the reported improvements are on ENST and RWC. No 2023–26 work on cymbal disambiguation via drum sub-stems was found.
|
||||
|
||||
### Cited Findings
|
||||
- **Tatum-level CRNN with LM-regularized training (2020).**
|
||||
- A skip bi-gram compares each tatum with the tatum 16 positions earlier (one 4/4 bar of 16ths); a GRU LM captures longer structure.
|
||||
- The pretrained LM regularizes training by penalizing "unnatural" binarized outputs.
|
||||
- **[older]** — [arXiv 2010.03749](https://arxiv.org/pdf/2010.03749)
|
||||
- **Global structure-aware drum transcription (2021)** uses self-attention plus a masked-LM regularizer. **[older]** — [arXiv 2105.05791](https://arxiv.org/pdf/2105.05791)
|
||||
- **Gillet & Richard (ISMIR 2007)** use N-gram and complexity-criterion post-processing, picking the most regular sequence consistent with acoustic scores. They report gains on ENST-drums. **[older]** — [ISMIR 2007](https://archives.ismir.net/ismir2007/paper/000219.pdf)
|
||||
- **Symbolic drum-score engraving (Digard & Jacquemard 2022)** turns MIDI drum input into notated drum scores via parsing and term rewriting, evaluated on Groove MIDI. **[older]** — [WoRMS 2022 proceedings](https://arxiv.org/pdf/2211.13285)
|
||||
- **Joint drum transcription and separation (Inverse Drum Machine, 2025).** — [arXiv 2505.03337](https://arxiv.org/pdf/2505.03337)
|
||||
|
||||
### Inferences
|
||||
- Implement bar-level pattern consolidation:
|
||||
1. Quantize drum onsets to a tatum grid (Beat This!) with a swing ratio.
|
||||
2. Cluster bars by Hamming distance.
|
||||
3. Within each cluster, majority-vote the hits and flag the deviating bars as fills to keep, rather than overwriting them.
|
||||
- For cymbal disambiguation, the cheap approach uses a DrumSep/LarsNet sub-stem energy ratio (hi-hat vs ride vs crash) as a tie-breaker feature. The previous survey found no evidence that this helps, so it stays a hypothesis.
|
||||
|
||||
### Gaps
|
||||
- No modern (2023–26) drum-specific post-processing evaluation was found. Groove/swing quantization specific to drums is unpublished beyond the generic quantizers.
|
||||
|
||||
## 8. Human-in-the-loop, confidence highlighting, learning from corrections, editing cost
|
||||
|
||||
### Takeaway
|
||||
Evidence that editing beats transcribing from scratch exists but is modest and dated. Holzapfel & Benetos (2019) found time and effort savings for some participants, and quality gains mainly for less-skilled users. No 2023–26 "time-to-correct" study for AI music transcription was found. The ASR literature (confidence-based error markup) is the best design analogue.
|
||||
|
||||
### Cited Findings
|
||||
- **Holzapfel & Benetos (ISMIR 2019).** Editing AMT output generally took less time and lower self-rated effort than manual transcription for some participants. In the follow-up, less-skilled transcribers produced higher-rated transcriptions when editing AMT output, while skilled ones showed no significant difference. **[older]** — [ISMIR 2019](https://archives.ismir.net/ismir2019/paper/000082.pdf); [DSH 2022](https://academic.oup.com/dsh/article/37/3/747/6408955)
|
||||
- **OMR-assisted transcription (ISMIR 2021)** measures correction time against time saved compared with manual work, and calls error correction "slow and tedious". **[older]** — [ISMIR 2021](https://archives.ismir.net/ismir2021/paper/000003.pdf)
|
||||
- **Tony (TENOR 2015),** a melody annotation tool, found that editing of any kind adds annotation time. **[older]** — [Tony paper](https://www.tenor-conference.org/proceedings/2015/04-Mauch-Tony.pdf)
|
||||
- **ENSEMBLE (ISMIR 2016)** is a crowd correction web editor with about 60 actions per 3-second clip. This shows that crowd correction is laborious at note level. **[older]** — [ISMIR 2016](https://archives.ismir.net/ismir2016/paper/000072.pdf)
|
||||
- **ASR analogue (2025).** Confidence-based error highlighting in a correction UI, evaluated with keystroke and time logs and UEQ-S. — [arXiv 2503.15124](https://arxiv.org/pdf/2503.15124)
|
||||
- **OMR-NED** is argued to be the metric that best reflects human correction effort: insert/delete edit distance over notation symbols. The 2026 dual-evaluation paper counters that it "may not reflect the effort to read or edit" and was not validated against human reading. — [Sheet Music Benchmark arXiv 2506.10488](https://arxiv.org/html/2506.10488v1); [arXiv 2608.04511](https://arxiv.org/html/2608.04511)
|
||||
- **User-guided generative separation (2025):** users steer separation interactively. — [arXiv 2507.01339](https://arxiv.org/abs/2507.01339)
|
||||
|
||||
### Inferences
|
||||
- Log every editor action in alphaTab (note add, delete, move, string change) with timestamps. This gives the in-house "time-to-correct" and "edits per bar" metrics, and builds a correction dataset for the MDTK-style corrector in section 4 and for confidence calibration.
|
||||
- Highlight notes by confidence, using ensemble agreement plus model probability plus rule flags (out-of-key, isolated short notes).
|
||||
- Offer bulk actions that exploit structure: "apply this fix to all repeats of this chorus" uses the repetition clusters from section 3.
|
||||
|
||||
### Gaps
|
||||
- No 2023–26 user study of AI-transcription editing time was found. Published systems that learn from user corrections (active learning) in AMT were not found.
|
||||
|
||||
## 9. Evaluation metrics that track human judgement; evaluating against non-aligned GP tabs
|
||||
|
||||
### Takeaway
|
||||
The strongest 2026 evidence (Wang et al., 106 listeners, 3,180 ABX judgments) is that notation-side and playback-side metrics favour *different* systems. Playback-side, CLEWS embeddings (ρ=0.971) and MFCC-TWED (ρ=0.924) track human preference best. Notation-side, MUSTER MeanER and OnsetER (ρ≈0.79–0.84) track humans far better than OMR-NED (ρ≈0.38–0.50). MV2H's non-aligned variant and DTW alignment handle GP references without audio timestamps.
|
||||
|
||||
### Cited Findings
|
||||
- **"A Dual Evaluation for Music Transcription" (Wang, Yang, Tamer, Ebert, Smith; UW/AI2; arXiv 2608.04511; under review at TISMIR).**
|
||||
- Data: 230 ATEPP piano recordings; 24 pipelines (8 AMT models × 3 MIDI-to-score converters) plus Rubato.
|
||||
- Human study: 106 participants, 3,180 ABX judgments, aggregated with Bradley–Terry.
|
||||
- Spearman ρ / Kendall τ against human preference:
|
||||
|
||||
| Metric | ρ | τ |
|
||||
|---|---|---|
|
||||
| CLEWS | 0.971 | 0.891 |
|
||||
| Gemini 3.1 Pro judge | 0.970 | 0.870 |
|
||||
| TWED-MFCC | 0.924 | 0.790 |
|
||||
| DTW-Chroma CENS | 0.891 | 0.761 |
|
||||
| CLaMP 3 audio | 0.884 | 0.710 |
|
||||
| CLaMP 3 audio→ABC | 0.879 | 0.710 |
|
||||
|
||||
- MUSTER MeanER and OnsetER: ρ=0.79 and 0.84, against 0.38 and 0.50 for OMR-NED.
|
||||
- The structure-oriented error rates (VoiceER, HandER, ScaleErr) track OMR-NED instead.
|
||||
- Code: MIT, [AMT-Dual-Eval](https://github.com/pingw220/AMT-Dual-Eval). Data tables are CC BY 4.0.
|
||||
|
||||
— [arXiv 2608.04511](https://arxiv.org/html/2608.04511)
|
||||
|
||||
(A search summary gave a differing figure, "global cosine ρ=0.824 vs OMR-NED ρ=0.681". The fetched full text above is the authoritative reading; the discrepancy is unresolved.)
|
||||
- **OMR-NED (Sheet Music Benchmark, 2025)** builds on musicdiff and is computed over visual notation symbols, normalized insert/delete. — [arXiv 2506.10488](https://arxiv.org/html/2506.10488v1)
|
||||
- **MV2H** has multi-pitch, voice, meter, note value and harmony sub-scores. The 2019 extension evaluates *non-aligned* transcriptions via DTW-style alignment, and a later version handles key changes per section. **[older]** — [arXiv 1906.00566](https://arxiv.org/html/1906.00566); [ISMIR 2018](https://ismir2018.ircam.fr/doc/pdfs/148_Paper.pdf)
|
||||
- **MUSTER** is used as the comparison metric for quantizers (2604.22290) and M2ST. — [arXiv 2604.22290](https://arxiv.org/html/2604.22290v1)
|
||||
- **"ScoreSimilarity" by that name** was not found. The nearest equivalents are OMR-NED/musicdiff and the dual-eval cosine/embedding scores.
|
||||
|
||||
### Inferences
|
||||
- The benchmark should report at least three families:
|
||||
1. Note- and tab-level F1 (already planned).
|
||||
2. A notation metric. MUSTER-style error rates beat OMR-NED as a human proxy, but OMR-NED/musicdiff approximates edit effort.
|
||||
3. A playback metric: render the GP output with the alphaTab synth and compare it to the original stem with CLEWS or DTW-chroma. This needs no alignment of the reference GP, so it can sanity-check songs where GP-to-audio alignment fails.
|
||||
- For GP references with imperfect tempo alignment, compare in *score time*: bar and beat positions after a DTW map. MV2H-non-aligned is an existing tool for exactly this, but check its piano-voice assumptions before applying it to tab. Report alignment failure separately, as the survey already proposes.
|
||||
- Make editor logs (section 8) the ultimate metric: "edits per minute of music to reach the reference".
|
||||
|
||||
### Gaps
|
||||
- Tab-specific human-correlation studies (does Tab F1 track guitarist judgments?) were not found. The MV2H licence and maintenance status were not checked. The dual-eval study is piano-only; its transfer to guitar, bass and drums is unknown.
|
||||
Reference in New Issue
Block a user