Compare commits
25
Commits
@@ -35,3 +35,48 @@ The baseline is in `results/baseline-2026-10-30.md` (tier summaries plus a per-s
|
||||
- `oracle-stems` mode feeds the true per-track stems, which reveals the track count and roles, so part-split F1 under oracle-stems (and the oracle modes built on it) is not meaningful.
|
||||
- The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
|
||||
- Fingering (string agreement) is informational, not pass/fail.
|
||||
|
||||
## Aligning reference songs
|
||||
|
||||
Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored `reference/` directory (never commit them; `*.flac`, `*.mid` and `runs/` are ignored too). Each song gets `reference/<id>/` with its audio, MIDI and `alignment.json`, and `reference/songs.json` lists the registered songs.
|
||||
|
||||
Register a song (guesses a scoring role per MIDI track; unmapped tracks are `null`):
|
||||
|
||||
```sh
|
||||
bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]
|
||||
```
|
||||
|
||||
The `--capo` track name must match the MIDI track name exactly, including any trailing space (e.g. `"Acoustic Guitar =9"`).
|
||||
|
||||
Compute and save an automatic alignment (all anchors `auto`; manual anchors from an earlier save are kept) and print its fit summary:
|
||||
|
||||
```sh
|
||||
bun run align auto <id>
|
||||
```
|
||||
|
||||
Launch the alignment page. A bare `bun run align` starts a long-running server; the key in the printed URL is required and changes on every start. For LAN use:
|
||||
|
||||
```sh
|
||||
bun run align --host 0.0.0.0 --url-host 10.20.20.2
|
||||
```
|
||||
|
||||
Page workflow:
|
||||
|
||||
1. Pick a song and add anchors where the MIDI and the recording visibly disagree (click the waveform/roll to pin a MIDI time to an audio time).
|
||||
2. Play and mix: toggle the recording and the rendered MIDI, and per-role levels, to listen for sync.
|
||||
3. Run auto-align around anchors; manual anchors are kept fixed and the rest is recomputed.
|
||||
4. Save to write `alignment.json`.
|
||||
|
||||
Benchmark against the real songs (uses each saved `alignment.json`; `--auto` computes one on the fly for songs without it):
|
||||
|
||||
```sh
|
||||
bun run bench real [--auto] --run-id <id>
|
||||
```
|
||||
|
||||
This writes `results/real-<id>.md` and `.json` (committed) and per-song output under `runs/real-<id>/` (ignored).
|
||||
|
||||
Caveats:
|
||||
|
||||
- The fit summary is a consistency measure between the MIDI and the audio, not ground truth: *precision* (share of the recording's detected attacks with a warped MIDI onset within ±70 ms; a 30 s window is flagged below 60 %), *lift* (precision minus its off-grid chance level; flagged below 0.25), a chroma check that flags suspected whole-beat/bar shifts, "MIDI onsets audible as attacks" (information only: a full mix exposes far fewer attacks than the score has notes), and the number of warned stretches (> 5 % and > 15 % tempo change). A high precision does not prove the alignment is right.
|
||||
- `align auto` also reports a constant-offset baseline (best single offset, −2..+2 s); with no manual anchors it keeps the baseline (one auto anchor) unless the DTW alignment beats its precision by 2 points.
|
||||
- The page plays Ogg Opus; Safari needs 18.4 or later.
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Accuracy experiments — 2026-10-31
|
||||
|
||||
Branch `exp/accuracy`. Every experiment is judged on **both** the synthetic tiers
|
||||
(median merged-guitar pitch F1, ±50 ms) and the real reference song
|
||||
*Last Night on Earth* (`lnoe`, the only reference with a trustworthy alignment:
|
||||
constant +0.03 s, precision 93%). Swimming/Holes alignments are still flagged and
|
||||
are not used for decisions.
|
||||
|
||||
Baseline before these experiments: synthetic full-mode guitar F1
|
||||
t1a 0.43 · t1b 0.51 · t1c 0.39 · t1d 0.60; lnoe guitar F1 0.390.
|
||||
|
||||
| # | Experiment | Synthetic (median guitar F1) | lnoe guitar F1 | Verdict |
|
||||
|---|---|---|---|---|
|
||||
| E1 | Transcribe guitar from Demucs guitar + piano + other (instead of guitar stem only) | t1a 0.87 · t1b 0.81 · t1c 0.73 · t1d 0.72 | 0.403 | **Adopted** (commit 0f14fbc). Big synthetic gain is mostly a SONiVOX-timbre artefact (Demucs routes the synth guitar to piano/other); real gain is small but positive. t1a now passes. |
|
||||
| E2 | Basic Pitch thresholds grid (onset 0.3–0.6, frame 0.2–0.4, min note 58/100 ms) | best ≈ default (0.5/0.3/58) | best 0.411 vs 0.403 | Not adopted — defaults are at the optimum on both. |
|
||||
| E3 | Error taxonomy | — | fp: 46% no reference note near, 16% octave, 21% 50–150 ms timing; fn: 47% different pitch at the onset (partial chords), 25% octave, 24% missed | Octave fixing upper bound only 0.40 → 0.46 (pitch-class F1) with no systematic direction. |
|
||||
| E4 | Complete partial strummed chords to standard guitar voicings (open + E/A-shape barres; maj/min/5/7/m7/sus/maj7) | −0.03…+0.03 | +0.016 (min 3 notes) | Not adopted yet — small, single-song evidence. |
|
||||
| E5 | Repetition voting on the guitar transcription (`src/pipeline/repeat.ts`): bars of the estimated grid are repeats when their 16th-slot × pitch occupancy is similar (cosine ≥ 0.6, smoothed ±1 bar along the diagonal) **and** ≥ 30% of notes match exactly (pitch, ±50 ms); per bar, drop notes found in < 15% of its repeats, add notes found in ≥ 25% and ≥ 3 repeats | t1a 0.885 · t1b 0.840 · t1c 0.752 · t1d 0.737 (+0.016…+0.026) | **0.507** (P 0.45→0.53, R 0.37→0.49) | **Adopted** (commit 132659a), default on for guitar (`createPipeline(models, { repeatVote })`). |
|
||||
|
||||
## Learnings
|
||||
|
||||
- Synthetic gains do not transfer 1:1 to real recordings while the synthetic guitar
|
||||
is the SONiVOX GM patch; a more realistic guitar sound in the renderer would make
|
||||
the synthetic benchmark more predictive.
|
||||
- On real audio the limits are Basic Pitch's partial-chord recall on strummed
|
||||
rhythm guitar and octave confusions, not thresholds.
|
||||
- An undocumented MT3 ONNX export exists (HF `ModernMube/HTDemucs_onnx`, ~300 MB);
|
||||
not pursued — input/token format would need reverse-engineering and MT3 is weak
|
||||
on real recordings.
|
||||
- E5 repetition voting: the two halves behave differently. Dropping non-recurring notes is what
|
||||
helps synthetic (precision; t1 A-sections have only 2 repeats, so the add rule — which needs ≥ 3 — never
|
||||
fires there). On lnoe adding is the big win (add-only, before the exact-match gate: recall 0.37 → 0.56, F1 0.500): Basic Pitch drops *different*
|
||||
chord tones in each repeat of a strummed part, so the union across repeats completes chords (cf. E3/E4).
|
||||
The add thresholds are therefore tuned on one real song — re-check when another trustworthy alignment exists.
|
||||
- Controls: treating every bar as a repeat (no repetition detection) gives lnoe 0.440 and wrecks synthetic
|
||||
(t1a 0.48), so the gain comes from the detected repeats, not from song-level note priors.
|
||||
- Voting needs an accurate grid. With only the coarse (slot ±1) similarity, a 1% tempo error made distant bars
|
||||
"similar" but 40 ms/bar misaligned and dropped lnoe to 0.23; the exact-match gate (≥ 0.3) turns that case into
|
||||
a no-op (0.403). A per-pair time-shift search (±½ beat) to absorb drift picked spurious shifts and cost
|
||||
0.03 on lnoe — not used. Half/double/1.5× tempo errors and ¼–½ beat downbeat errors still gain (lnoe 0.44–0.51).
|
||||
- Evaluated on cached transcriptions + estimated grids (`.scratch/exp/cases_full.ts`, baseline reproduces E1: 0.866 · 0.814 · 0.734 · 0.721); full pipeline spot check t1c/1 0.708 → 0.751 matches.
|
||||
- Per-song effect is not uniformly positive (t1a/2 −0.011, t1d/5 −0.026); medians and lnoe improve.
|
||||
|
||||
Experiment scripts live in the git-ignored `.scratch/exp/`.
|
||||
@@ -0,0 +1,183 @@
|
||||
# Reference Alignment Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Align team-supplied GP-exported MIDI to real recordings (automatic pass + manual anchor editing in a local web page) and score the pipeline on real songs against the aligned MIDI.
|
||||
|
||||
**Architecture:** Pure alignment logic in `src/align/` (SMF parsing, warp model, MIDI rendering via alphaTab, features, DTW auto-aligner, fit quality). File/manifest I/O in `src/ref/`. A token-protected `Bun.serve` app (`src/align/server.ts` + HTML import page) exposes songs, waveform peaks, audio, warped MIDI renders, auto-align and save. `src/bench/real.ts` scores real songs.
|
||||
|
||||
**Tech Stack:** Bun 1.3 + TypeScript, alphaTab 1.8.4 (synth), existing `src/dsp`, `src/pipeline/run.ts`, `src/eval/*`, ffmpeg CLI (transcode only), Web Audio API in the page.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-30-reference-alignment-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports, no vite/express, `bun test`, `Bun.file`/`Bun.write`, `Bun.$`).
|
||||
- `src/align/**` except `server.ts`, `cli.ts` and `page/**` must not use `node:fs`/`Bun.*`/`process` (browser-portable); I/O lives in `src/ref/**`, `src/align/server.ts`, `src/bench/**`.
|
||||
- Never commit reference audio, MIDI, renders, `reference/**`, or `*.flac/*.mid`. Stage files by explicit path only; check `git status --short` before committing.
|
||||
- Manifest: `reference/songs.json`; per song dir `reference/<songId>/` holds `alignment.json`, cached transcoded audio, peaks.
|
||||
- Alignment = anchors `{ midiSec, audioSec, kind: "manual" | "auto" }`, monotonic piecewise-linear MIDI→audio map; implicit anchors at start (0→0 unless an anchor exists) and end (extrapolate with the last stretch's slope).
|
||||
- Roles scored: guitar (merged), vocals, bass, drums (by class, existing heuristic transcriber), keys. Unmapped tracks excluded and listed.
|
||||
- Fit quality: recording onsets from raw-audio onset flux (never from the pipeline's transcription); MIDI onsets = unique warped onset times of scored-role notes; match ±70 ms; report median |residual| (ms) and match rate, overall and per 30 s window; window flagged when match rate < 0.5 or median residual > 60 ms.
|
||||
- Editing model: adding/moving a manual anchor deletes auto anchors in the two affected stretches; "Auto-align around my anchors" re-solves autos between consecutive manual anchors; auto never runs implicitly on drag.
|
||||
- Server binds configurable `--host` (default 127.0.0.1) / `--port`; prints URL with random token; rejects requests without token (query `?key=` once, then cookie); file access only for manifest entries.
|
||||
- `bench real`: report-only; skips songs without `alignment.json` unless `--auto`; per-run artefacts `runs/real-<runId>/` (ignored); summary `results/real-<runId>.{md,json}` (titles, scores, alignment source + fit only) committed.
|
||||
- Commits end with the Co-Authored-By line from your own system reminder.
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. **Anchors at or beyond the ends / out of order** — warp must stay monotonic: reject (throw) non-increasing anchors; the page prevents dragging past neighbours. Test in Task 2.
|
||||
2. **Rendered MIDI clipping** — SMF renders peak > 1 (Holes peaked 2.13); render must peak-normalise to −1 dBFS. Test in Task 3.
|
||||
3. **Silence / no onsets in a 30 s window** — fit quality reports `null` for that window (not NaN) and doesn't flag it. Test in Task 5.
|
||||
4. **Path traversal / unknown song ids** on the server — 404, never read outside manifest paths. Test in Task 6.
|
||||
5. **Huge DTW matrices** — coarse pass at ≈10 Hz over a 5-min song is ~3000×3000; must stay under ~200 MB and ~5 s CPU. Test timing in Task 5.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
src/align/smf.ts parse SMF → { durationSec, tracks: RefTrack[], tempoMap }
|
||||
src/align/roles.ts guessRole(track) → Role | null
|
||||
src/align/warp.ts anchors model: makeWarp, invert, stretches, edit ops
|
||||
src/align/render.ts renderMidi(tracks, warp, sf2) → Audio (alphaTab synth)
|
||||
src/align/features.ts onsetEnvelope, pickOnsets, chroma (frame-rate param)
|
||||
src/align/dtw.ts banded/unconstrained DTW
|
||||
src/align/auto.ts autoAlign(audio, tracks, manualAnchors) → Anchor[]
|
||||
src/align/fit.ts fitQuality(audioOnsets, warpedMidiOnsets) → FitReport
|
||||
src/align/peaks.ts waveform peaks (min/max per bucket)
|
||||
src/align/server.ts Bun.serve app (token, API, HTML import)
|
||||
src/align/page/index.html, app.ts, style.css layout-A page
|
||||
src/align/cli.ts `bun run align [serve|add|auto]`
|
||||
src/ref/manifest.ts read/write reference/songs.json, song dirs, transcode cache
|
||||
src/bench/real.ts real-song scoring + report
|
||||
tests/align/**, tests/ref/**, tests/bench/real.test.ts
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 1: SMF parser, role guessing, manifest + `align add`
|
||||
|
||||
**Files:** Create `src/align/smf.ts`, `src/align/roles.ts`, `src/ref/manifest.ts`, `src/align/cli.ts`; modify `package.json` (`"align": "bun src/align/cli.ts"`). Tests: `tests/align/smf.test.ts`, `tests/ref/manifest.test.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
```ts
|
||||
// smf.ts
|
||||
export interface RefNote { onset: number; offset: number; pitch: number; velocity: number; channel: number }
|
||||
export interface RefTrack { index: number; name: string; program: number; isDrums: boolean; notes: RefNote[] }
|
||||
export interface TempoPoint { sec: number; bpm: number }
|
||||
export interface Smf { durationSec: number; tracks: RefTrack[]; tempoMap: TempoPoint[]; timeSigs: { sec: number; num: number; den: number }[] }
|
||||
export function parseSmf(bytes: Uint8Array): Smf
|
||||
// roles.ts
|
||||
export function guessRole(t: RefTrack): Role | null // drums (ch 10) / vocal / bass / guitar|delay / piano|key|synth; else null
|
||||
// manifest.ts
|
||||
export interface SongEntry { id: string; title: string; audio: string; midi: string; roles: Record<string, Role | null>; capo: Record<string, number>; tuning?: Record<string, number[]> }
|
||||
export async function readManifest(root = "reference"): Promise<SongEntry[]>
|
||||
export async function addSong(e: Omit<SongEntry, "roles"> & { roles?: SongEntry["roles"] }, root = "reference"): Promise<SongEntry> // fills roles via guessRole, keyed by track name
|
||||
export function songDir(id: string, root = "reference"): string
|
||||
```
|
||||
CLI: `bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]`.
|
||||
|
||||
- [ ] **Step 1: Write failing tests** — build a tiny SMF in the test (format 1, ppq 480, tempo 120 then 60 at tick 1920, a guitar track program 25 with two notes, a ch-10 drum note): assert note onsets in seconds (0.0, 0.5 → second-tempo section at 2.0 s + 1.0 s), `tempoMap` has 2 points, drums `isDrums`, `guessRole` returns "guitar"/"drums"; running-status and note-on velocity 0 = note-off handled. Manifest: `addSong` in a temp root writes `songs.json`, guesses roles, rejects duplicate ids.
|
||||
- [ ] **Step 2:** FAIL. **Step 3:** implement (port the parser in `.scratch/realsong.ts` but clean). **Step 4:** PASS + typecheck.
|
||||
- [ ] **Step 5:** Commit `feat(align): SMF parser, role guess, reference manifest`.
|
||||
|
||||
### Task 2: Warp model
|
||||
|
||||
**Files:** Create `src/align/warp.ts`; test `tests/align/warp.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
```ts
|
||||
export type AnchorKind = "manual" | "auto";
|
||||
export interface Anchor { midiSec: number; audioSec: number; kind: AnchorKind }
|
||||
export interface Warp { toAudio(midiSec: number): number; toMidi(audioSec: number): number; anchors: Anchor[] }
|
||||
export function makeWarp(anchors: Anchor[]): Warp // sorts by midiSec; throws if audioSec not strictly increasing; 0 anchors = identity; 1 anchor = pure offset
|
||||
export interface Stretch { fromMidi: number; toMidi: number; ratio: number; warn: boolean } // ratio = audio dur / midi dur; warn when |1/ratio − 1| > 0.05
|
||||
export function stretches(anchors: Anchor[], midiDur: number): Stretch[]
|
||||
export function addManual(anchors: Anchor[], a: Omit<Anchor, "kind">): Anchor[] // removes autos in the two stretches adjacent to the new anchor
|
||||
export function moveAnchor(anchors: Anchor[], i: number, to: Omit<Anchor, "kind">): Anchor[] // becomes manual; same auto-removal; throws if it would cross a neighbour
|
||||
export function deleteAnchor(anchors: Anchor[], i: number): Anchor[]
|
||||
```
|
||||
- [ ] **Step 1: Failing tests:** identity; single anchor = offset; two anchors interpolate and extrapolate with end slopes; `toMidi(toAudio(x)) ≈ x`; non-monotonic throws; `stretches` ratio + warn at 7%; `addManual` between autos removes only autos in adjacent stretches; `moveAnchor` crossing throws.
|
||||
- [ ] **Step 2–4:** FAIL → implement → PASS. **Step 5:** Commit `feat(align): warp model`.
|
||||
|
||||
### Task 3: Warped MIDI rendering + waveform peaks
|
||||
|
||||
**Files:** Create `src/align/render.ts`, `src/align/peaks.ts`; tests `tests/align/render.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
```ts
|
||||
export function renderMidi(tracks: RefTrack[], warp: Warp, sf2: Uint8Array, opts?: { sampleRate?: number; mono?: boolean; durationSec?: number }): Audio
|
||||
export function peaks(x: Float32Array, buckets: number): { min: Float32Array; max: Float32Array }
|
||||
```
|
||||
Approach (verified by controller spike): build `at.midi.MidiFile` (division 960) at a fixed 120 bpm (`TempoChangeEvent(0, 500000)` → 1920 ticks/s); place each note at `round(warp.toAudio(onset) * 1920)`; drums on channel 9, others on channels 0–8, 10–15 assigned per track with a `ProgramChangeEvent`; render with `AlphaSynth.exportAudio` (dummy output, `useSyncPoints = false`) as in `src/audio/render.ts` (reuse its dummy output helper if exported; otherwise export it from there); peak-normalise to −1 dBFS; pad/trim to `durationSec` when given.
|
||||
- [ ] **Step 1: Failing tests:** one guitar note at MIDI 1.0 s with warp offset +0.5 s → envelope onset in [1.48, 1.52] s; peak ≤ 10^(−1/20)+1e-3 on a dense 20-note chord; drums track audible; `peaks` of a known sine.
|
||||
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): warped MIDI rendering and peaks`.
|
||||
|
||||
### Task 4: Features + DTW
|
||||
|
||||
**Files:** Create `src/align/features.ts`, `src/align/dtw.ts`; tests `tests/align/features.test.ts`, `tests/align/dtw.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
```ts
|
||||
export interface Features { rate: number; onset: Float32Array; chroma: Float32Array[] } // per frame; chroma 12-dim L2-normalised
|
||||
export function features(audio: Audio, rate: number): Features
|
||||
export function pickOnsets(onset: Float32Array, rate: number): number[] // seconds; adaptive threshold (median + k·MAD), 50 ms min gap
|
||||
export function dtw(costAt: (i: number, j: number) => number, n: number, m: number, band?: { center: (i: number) => number; width: number }): [number, number][] // path start (0,0) → end (n-1,m-1), steps (1,0),(0,1),(1,1)
|
||||
```
|
||||
Reuse `src/dsp/fft.ts` `stft`, `src/dsp/mono.ts`, `src/dsp/resample.ts`. Cost for alignment = (1 − chroma cosine) + λ·|onset_a − onset_b| (λ = 1, onsets z-normalised).
|
||||
- [ ] **Step 1: Failing tests:** DTW recovers a known diagonal-with-kink path on synthetic cost; banded DTW equals unbanded when band covers the path; features rate/length correct; `pickOnsets` on click train ±10 ms.
|
||||
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): features and DTW`.
|
||||
|
||||
### Task 5: Auto aligner + fit quality
|
||||
|
||||
**Files:** Create `src/align/auto.ts`, `src/align/fit.ts`; tests `tests/align/auto.test.ts`, `tests/align/fit.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
```ts
|
||||
export interface FitWindow { startSec: number; matchRate: number | null; medianResidualMs: number | null; flagged: boolean }
|
||||
export interface FitReport { matchRate: number | null; medianResidualMs: number | null; windows: FitWindow[] }
|
||||
export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number): FitReport // ±70 ms, 30 s windows, flag rule per Global Constraints
|
||||
export function autoAlign(audio: Audio, smf: Smf, roles: Record<string, Role | null>, manual: Anchor[], sf2: Uint8Array): { anchors: Anchor[]; fit: FitReport }
|
||||
```
|
||||
Approach: render the (manual-anchored) warped MIDI with `renderMidi`; coarse DTW at 10 Hz per segment between consecutive manual anchors (+ start/end); refine at 50 Hz in a band (±0.5 s) around the coarse path; convert path to MIDI→audio pairs; reduce to one `auto` anchor per bar using `smf.tempoMap`/`timeSigs` bar times (skip bars within 1 s of a manual anchor); compose with manual anchors; compute fit with `pickOnsets` on the recording and warped scored-role MIDI onsets.
|
||||
- [ ] **Step 1: Failing tests:** synthetic known warp — take generated song `t1c/1` (`buildSong`), write its truth as an SMF-like `Smf` object (no file needed), distort MIDI times with a known piecewise map (offset +0.4 s, section 2 ×1.06 tempo, section 3 −0.8 s) and align against the original mix: median |recovered − true| < 40 ms, overall fit matchRate > 0.7. With one correct manual anchor mid-song the result honours it exactly. `fitQuality`: empty window → nulls, not flagged; timing: 5-min synthetic input completes < 15 s.
|
||||
- [ ] **Step 2–4.** **Step 5:** Commit `feat(align): automatic alignment and fit quality`.
|
||||
|
||||
### Task 6: Alignment server + page (layout A)
|
||||
|
||||
**Files:** Create `src/align/server.ts`, `src/align/page/index.html`, `src/align/page/app.ts`, `src/align/page/style.css`; extend `src/align/cli.ts` (`serve` default: `bun run align [--host] [--port]`; `bun run align auto <id>` computes autoAlign and saves `alignment.json`, printing the fit summary), `src/ref/manifest.ts` (transcode cache: `reference/<id>/audio.ogg` via ffmpeg Opus 160k + `recording.wav` 44.1 kHz for analysis; `alignment.json` read/write). Test: `tests/align/server.test.ts`.
|
||||
|
||||
**API (all require token):**
|
||||
- `GET /` → page (HTML import). `GET /api/songs` → `[{ id, title, hasAlignment }]`.
|
||||
- `GET /api/song/:id` → `{ id, title, audioDur, midiDur, tracks: [{name, role}], recordingPeaks, anchors, stretches, fit, midiTempo: tempoMap }` (auto-aligns and returns unsaved anchors if no `alignment.json`).
|
||||
- `GET /api/song/:id/audio` → cached Opus.
|
||||
- `POST /api/song/:id/render` body `{ anchors }` → WAV (mono 22.05 kHz) of warped MIDI + header-free JSON peaks via `GET /api/song/:id/render-peaks?hash=` (or return both; builder's call).
|
||||
- `POST /api/song/:id/auto` body `{ anchors }` (manual ones used) → `{ anchors, stretches, fit }`.
|
||||
- `PUT /api/song/:id/alignment` body `{ anchors }` → saves `{ anchors, savedAt, fit }`. `POST /api/song/:id/reset` → raw MIDI (no anchors).
|
||||
|
||||
**Page behaviour (layout A, from the approved mockup):** song picker; two lanes on a shared zoom/scroll timeline (wheel = zoom, drag background = scroll); recording peaks on top, warped-MIDI peaks below; anchors as green lines spanning both lanes, draggable at the recording end or the MIDI end (prevented from crossing neighbours), click-select + Delete key; "+ Anchor at cursor" (maps cursor recording time → `warp.toMidi` for the MIDI side); stretch labels "MIDI bpm → effective bpm (±%)" with ⚠ when warned; fit-flagged 30 s windows shaded red; click timeline to set cursor; Play/Pause from cursor; mix select (both / recording / MIDI / rec L + MIDI R) via Web Audio gain + stereo panner; MIDI render re-requested 400 ms after the last edit, old buffer keeps playing until replaced, both sources started at the same `AudioContext.currentTime`; buttons: Auto-align around my anchors, Save, Reset to raw MIDI; unsaved-changes indicator.
|
||||
- [ ] **Step 1: Failing tests (API):** start server on port 0 with a temp reference root containing a fixture song built from `data/t1a/1` (mix.wav as audio, an SMF written from its truth — add `writeSmf(tracks)` helper in `src/align/smf.ts` for tests); no token → 401; `/api/songs` lists it; unknown id and `..` id → 404; `/api/song/:id` returns peaks + anchors; `PUT` then `GET` round-trips anchors; `/render` returns RIFF WAV.
|
||||
- [ ] **Step 2–4:** FAIL → implement → PASS. Smoke: `bun run align --host 0.0.0.0` prints a tokenised URL; load it with `curl` and confirm HTML + bundle are served.
|
||||
- [ ] **Step 5:** Commit `feat(align): alignment web page`.
|
||||
|
||||
### Task 7: Real-song benchmark
|
||||
|
||||
**Files:** Create `src/bench/real.ts`; modify `src/bench/cli.ts` (`real` subcommand); test `tests/bench/real.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
```ts
|
||||
export interface RealSongResult { id: string; title: string; alignment: { source: "manual" | "auto"; manual: number; auto: number; savedAt: string | null }; fit: FitReport; perRole: Partial<Record<Role, MatchResult>>; drumsPerClass: Record<DrumClass, MatchResult> | null; excludedTracks: string[]; seconds: number }
|
||||
export async function runReal(opts: { runId: string; auto: boolean; root?: string; out?: string }): Promise<RealSongResult[]>
|
||||
export function renderRealReport(runId: string, results: RealSongResult[]): string
|
||||
```
|
||||
Behaviour: for each manifest song: load `alignment.json` (or `autoAlign` when `--auto`, else skip with a note); run `createPipeline` full mode on `recording.wav`; warp each scored-role MIDI note's onset/offset with `makeWarp(anchors).toAudio`; per role `matchNotes` (guitar merged; drums by class like `scoreSong`); write `runs/real-<runId>/<id>/{notes.json,out.gp,score.json}` and `results/real-<runId>.{md,json}` (strip `matches`). Report: per song alignment source/fit, role table (guitar first), excluded tracks.
|
||||
- [ ] **Step 1: Failing tests:** `renderRealReport` fixture (guitar column first, fit shown, "auto" labelled); `runReal` on a temp root with the t1a fixture song + an identity alignment → guitar F1 equals the synthetic full-mode score for t1a/1 within 0.02; real `reference/` test `skipIf` absent.
|
||||
- [ ] **Step 2–4.** **Step 5:** Commit `feat(bench): real-song benchmark`.
|
||||
|
||||
### Task 8: Seed songs, auto-align, first real benchmark
|
||||
|
||||
- [ ] **Step 1:** `bun run align add` the three songs (ids `lnoe`, `swimming`, `holes`; titles from file names; Holes `--capo "Acoustic Guitar =9"` — match the exact track name from the MIDI). Verify roles in `reference/songs.json` (Holes brass/crowd → null).
|
||||
- [ ] **Step 2:** For each song run `bun run align auto <id>` (CLI: compute auto alignment and save it as `alignment.json` with all anchors `auto`), print fit summaries.
|
||||
- [ ] **Step 3:** `bun run bench real --run-id real-auto-2026-10-30` (uses saved auto alignments) → results files.
|
||||
- [ ] **Step 4:** README section "Aligning reference songs" (commands, where files live, the page workflow). Commit README + `results/real-auto-2026-10-30.{md,json}` by explicit path; push.
|
||||
@@ -0,0 +1,202 @@
|
||||
# Reference Song Alignment — Design
|
||||
|
||||
Date: 2026-10-30
|
||||
Status: approved by user 2026-10-30
|
||||
Builds on: `docs/superpowers/specs/2026-10-30-synthetic-benchmark-design.md` (branch `feat/alignment`, based on `feat/synthetic-benchmark`).
|
||||
|
||||
## Intent
|
||||
|
||||
The team has real recordings (FLAC) with hand-made Guitar Pro tabs exported as MIDI.
|
||||
For two of the first three songs (*Swimming*, *Holes*) the MIDI does not follow the
|
||||
recording's timing: the best-fitting offset drifts from about −2 s to +2 s across the song,
|
||||
and it drifts differently per section, so a single stretch can't fix it. Until the
|
||||
MIDI follows the recording, real-song scores mostly measure misalignment, not
|
||||
transcription. (*Last Night on Earth* fits with one ~60 ms offset.)
|
||||
|
||||
We want (1) an automatic alignment step, (2) a local web page where a person can
|
||||
correct it by placing a few anchors (the user expects ~3 phrases of ~20 s per song
|
||||
to be enough), and (3) a real-song benchmark that scores the pipeline against the
|
||||
aligned MIDI.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Alignment model.** A song's alignment is a monotonic piecewise-linear map from
|
||||
MIDI time to recording time, defined by anchors (`{ midiSec, audioSec }`).
|
||||
Between two anchors, the MIDI's own tempo map is scaled uniformly so the
|
||||
stretch fits. Start and end are implicit anchors.
|
||||
2. **Automatic first pass.** Computes an alignment from the audio and the MIDI
|
||||
alone, honouring any manual anchors as fixed points (the automatic part only
|
||||
fills in between them). Reports a fit-quality number so we know whether to trust it.
|
||||
3. **Alignment web page (layout A, chosen by the user):**
|
||||
- Two lanes on one shared, zoomable timeline: recording waveform on top, the
|
||||
MIDI rendered to audio (with the current alignment applied) underneath.
|
||||
- Anchors drawn as lines across both lanes; the user can add one at the
|
||||
playback cursor, drag either end (recording side / MIDI side), and delete it.
|
||||
- Each stretch between anchors shows its resulting tempo vs. the MIDI's own
|
||||
(e.g. "55.0 → 58.9 bpm (+7%)"), with a warning when the change is large
|
||||
(likely an anchor on the wrong bar).
|
||||
- Playback from the cursor with a mix choice: both, recording only, MIDI only,
|
||||
recording left / MIDI right. Playback reflects the current anchors.
|
||||
- Save writes the alignment to disk; reopening the page loads it.
|
||||
- Song picker for the songs listed in the reference manifest.
|
||||
4. **Real-song benchmark.** A command runs the pipeline (full mode) on each
|
||||
reference recording, warps the reference MIDI's note times through the saved
|
||||
alignment, and scores per role (guitar tracks merged; same pitch metric as the
|
||||
synthetic benchmark: ±50 ms onset, offsets ignored), plus the alignment's fit
|
||||
quality. Writes a markdown + JSON summary.
|
||||
5. **Per-song metadata** in the manifest: title, audio path, MIDI path, MIDI
|
||||
track → role mapping (auto-guessed from track names/programs, overridable),
|
||||
and per-guitar-track capo (Holes: acoustic guitar, capo 9) and tuning.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Editing notes, pitches, or instruments in the MIDI — only timing changes.
|
||||
- Multi-user or hosted use; this is a local team tool.
|
||||
- Rhythm/fingering scoring for real songs (needs 12/8, tempo changes, capo in
|
||||
the tab assigner — later work). Capo is *recorded* now, not yet used.
|
||||
- Changing the synthetic benchmark.
|
||||
|
||||
## Anti-goals
|
||||
|
||||
- Committing any reference audio, MIDI, rendered audio, or the manifest to git.
|
||||
- An alignment that silently fits wrong (e.g. a chorus shifted by a whole bar)
|
||||
without the fit-quality number or the page showing it.
|
||||
- Scores reported for a real song without saying which alignment (auto/manual,
|
||||
fit quality) they were computed against.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Bun + TypeScript; follow `CLAUDE.md` (Bun.serve with HTML imports for the
|
||||
page, no vite/express; `bun test`).
|
||||
- Reuse existing code: MIDI note/tempo extraction, alphaTab synth rendering,
|
||||
`src/dsp` (STFT, mono, resample), `src/pipeline/run.ts`, `src/eval/notes.ts`.
|
||||
Pure alignment logic lives outside `src/bench`/server code so it stays
|
||||
browser-portable (same rule as the pipeline).
|
||||
- All reference material lives in the git-ignored `reference/` folder (and the
|
||||
FLAC/MIDI files in the repo root, also ignored). The manifest is
|
||||
`reference/songs.json` (git-ignored — coordinator's call, to keep song lists
|
||||
out of VCS). Alignments are saved as `reference/<songId>/alignment.json`.
|
||||
- Served on the user's network: the server binds a configurable host (the user
|
||||
reaches this machine at `10.20.20.2`); `bun run align --host 0.0.0.0`.
|
||||
- CPU only.
|
||||
|
||||
## How it works (decided parts)
|
||||
|
||||
- **Features for automatic alignment (coordinator's call):** onset strength and
|
||||
chroma from the recording and from the MIDI rendered to audio, compared with
|
||||
dynamic time warping restricted to a band around the current path, solved
|
||||
separately between consecutive manual anchors (so anchors are hard
|
||||
constraints). The resulting path is reduced to anchors (about one per bar)
|
||||
marked `auto`; manual anchors are marked `manual` and always win.
|
||||
- **Default flow (coordinator's call):** opening a song with no saved alignment
|
||||
runs the automatic pass first, so the user corrects a guess rather than
|
||||
starting from the raw MIDI; a "reset to raw MIDI" action exists.
|
||||
- **Fit quality:** median absolute onset residual (ms) between matched recording
|
||||
onsets and warped MIDI onsets, overall and per 30 s window; shown on the page
|
||||
and in benchmark output. (Amended 2026-10-30: see "Fit quality — Amended" below.)
|
||||
- **Playback of the warped MIDI:** the server re-renders the warped MIDI audio
|
||||
when anchors change (alphaTab renders a 4-minute song in ~1–2 s); the page
|
||||
requests it on Play. Waveforms are sent as precomputed peaks, not full audio.
|
||||
|
||||
|
||||
## Clarifications (coordinator's calls after builder review)
|
||||
|
||||
- **Roles scored on real songs:** guitar (all guitar tracks merged), vocals (lead +
|
||||
backing), bass, drums (existing heuristic transcriber, compared by drum class as
|
||||
in the synthetic benchmark), keys (piano/keyboard/synth tracks). MIDI tracks with
|
||||
no role (e.g. brass, crowd) are excluded and listed in the report. The
|
||||
track→role mapping is auto-guessed from track names/programs when a song is
|
||||
added and can be overridden by editing `reference/songs.json` (no UI for it).
|
||||
- **Automatic aligner, coarse-to-fine:** a first unconstrained DTW pass at a low
|
||||
frame rate (≈10 Hz) over the whole song (between manual anchors when present),
|
||||
then refinement in a band around that path at full rate (≈50 Hz). The bar for
|
||||
"auto" is *good enough to correct by hand*, not perfect.
|
||||
- **Fit quality (not circular):** recording onsets come from the raw audio (onset
|
||||
flux peak-picking on the mix), never from the pipeline's transcription. MIDI
|
||||
onsets = unique onset times of all scored-role notes after warping. Matching
|
||||
within ±70 ms. Report (a) median absolute residual of matched onsets and (b)
|
||||
match rate = share of warped MIDI onsets with a recording onset within ±70 ms,
|
||||
overall and per 30 s window. A window is flagged "don't trust" when its match
|
||||
rate < 0.5 or median residual > 60 ms; the page highlights flagged windows.
|
||||
*(Superseded by the amendment below.)*
|
||||
- **Fit quality — Amended 2026-10-30 after real-song evidence (rulings A7, A8).**
|
||||
On real full mixes the onset picker finds ~1.4 attacks/s while a scored MIDI
|
||||
has ~13 unique onsets/s, so the match rate above caps near 40 % even when
|
||||
perfectly aligned (every window was flagged on every song), and onset metrics
|
||||
cannot see whole-beat shifts (they alias onto the grid).
|
||||
- **Flag metric = precision:** share of recording onsets (onset flux, k = 4,
|
||||
parabolic sub-frame peak times) with a warped scored-role MIDI onset within
|
||||
±70 ms; per 30 s window by recording time. A window with < 8 recording
|
||||
onsets is null and never flagged; otherwise flagged when precision < 0.6.
|
||||
- **Chance-corrected lift:** precision minus the mean precision with the
|
||||
warped MIDI shifted by +0.11, +0.23 and +0.37 s (off-grid); reported per
|
||||
window and overall; a window is also flagged when lift < 0.25.
|
||||
- **Beat/bar-shift check:** per window, mean chroma cosine between the
|
||||
recording (features chroma, 10 Hz) and a symbolic MIDI chroma (piano roll of
|
||||
the warped pitched scored-role notes, no render) at the current alignment
|
||||
vs the MIDI moved by ±1, ±2 beats and ±1 bar along the MIDI's own
|
||||
tempo/time-signature grid. Flagged "beat shift suspected: ±k beat(s)/bar"
|
||||
when a shift beats the current alignment by > 0.05 (the margin keeps
|
||||
repetitive harmony, where shifts tie, from flagging). "+1 beat" means the
|
||||
MIDI content belongs one beat later in the recording.
|
||||
- **Information only:** the old match rate, renamed "MIDI onsets audible as
|
||||
attacks" (JSON `audibleRate`), and the median residual (recording onsets →
|
||||
nearest warped MIDI onset).
|
||||
- **Stretch warnings:** the fit summary (CLI, page, real report) includes the
|
||||
number of warned stretches (> 5 % tempo change) and of stretches with
|
||||
|tempo change| > 15 %.
|
||||
- **Constant-offset baseline:** grid search −2..+2 s in 10 ms steps for the
|
||||
offset maximising overall precision; precision peaks within 0.1 of the best
|
||||
are told apart by the symbolic chroma cosine (beat aliasing), and within a
|
||||
peak the offset with the smallest mean residual is taken. Reported with its
|
||||
precision. Without manual anchors, `autoAlign` returns the baseline as one
|
||||
auto anchor (`chosen: "baseline"`) unless the DTW alignment's precision
|
||||
beats it by ≥ 0.02 (`chosen: "dtw"`); with manual anchors there is never a
|
||||
substitution.
|
||||
- **Aligner guards:** both DTW passes use slope-limited steps (Sakoe–Chiba
|
||||
P = 1: (1,1), (1,2), (2,1)), so the path cannot jump; the first segment may
|
||||
start late in the recording (open begin) and the final segment uses an open
|
||||
end (the path may stop before the recording ends, or before the MIDI ends,
|
||||
each unconsumed MIDI frame costing 1), so a length mismatch is not absorbed
|
||||
as a tempo ramp. Per-bar offsets more than ¼ beat from the median of their
|
||||
±3 neighbouring bars are rejected (iterated until stable), and no auto
|
||||
anchor is placed in the MIDI's final partial bar.
|
||||
- **Editing model:** dragging an anchor only re-fits its two neighbouring
|
||||
stretches. Adding or moving a manual anchor deletes the auto anchors in the two
|
||||
affected stretches (those stretches become plain uniform scaling) until the user
|
||||
presses "Auto-align around my anchors", which re-solves auto anchors in every
|
||||
stretch between consecutive manual anchors. Auto never runs implicitly on drag.
|
||||
- **MIDI rendering:** build an alphaTab `MidiFile` from the parsed SMF events
|
||||
(programs, channel 10 drums) and render with the existing SONiVOX SoundFont.
|
||||
If that proves impossible, a small built-in additive synth is an acceptable
|
||||
fallback, labelled in the UI. MIDI pitches already include capo, so capo does
|
||||
not affect rendering.
|
||||
- **Manifest seeding:** `bun run align add <audio> <midi> [--id] [--capo track=N]`
|
||||
adds a song and auto-guesses roles; a one-off seeding of the three current songs
|
||||
(Holes: acoustic guitar capo 9) is part of the work.
|
||||
- **Playback/transport:** the recording is transcoded once (ffmpeg) to a
|
||||
browser-friendly cached file under `reference/<songId>/`; waveforms are sent as
|
||||
precomputed peaks; the warped MIDI render is re-requested after edits
|
||||
(debounced), the previous render keeps playing until the new one arrives, and
|
||||
both buffers start at the same AudioContext time.
|
||||
- **Access:** the server prints a URL with a random access token (like the
|
||||
mockup server) and rejects requests without it; file access is limited to
|
||||
manifest entries.
|
||||
- **`bench real`:** report-only (no pass bar). Songs without `alignment.json` are
|
||||
skipped unless `--auto` (then the automatic alignment is computed and used, and
|
||||
labelled). The report states, per song, alignment source (manual/auto anchor
|
||||
counts, file timestamp) and fit quality. Per-run artefacts go to
|
||||
`runs/real-<runId>/` (git-ignored); the summary `results/real-<runId>.{md,json}` (song titles, scores, alignment quality only — no audio/MIDI) is committed (user decision).
|
||||
- **Tests:** the aligner is tested on generated songs with a known synthetic warp
|
||||
(piecewise tempo changes and offsets applied to the MIDI); tests touching real
|
||||
reference files skip when `reference/` is absent.
|
||||
|
||||
## Left to the builder
|
||||
|
||||
- Page structure, styling, zoom/scroll details, keyboard shortcuts.
|
||||
- Exact DTW band width, feature frame rate, and bar-anchor reduction rule.
|
||||
- How the MIDI is fed to alphaTab's synth (constructing a `MidiFile` from the
|
||||
parsed SMF events is expected to work; report if not).
|
||||
- Audio transport format to the browser (e.g. FLAC/WAV/Opus) as long as
|
||||
playback works in current Chrome/Firefox/Safari.
|
||||
- CLI shape (`bun run align`, `bun run bench real` are suggestions).
|
||||
+2
-1
@@ -6,7 +6,8 @@
|
||||
"test": "bun test",
|
||||
"typecheck": "bunx tsc --noEmit",
|
||||
"fetch-models": "bun scripts/fetch-models.ts",
|
||||
"bench": "bun src/bench/cli.ts"
|
||||
"bench": "bun src/bench/cli.ts",
|
||||
"align": "bun src/align/cli.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/bun": "latest"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
# STSheets benchmark report: exp-gpo-2026-10-31
|
||||
|
||||
- Run id: exp-gpo-2026-10-31
|
||||
- Date: 2026-10-30T19:46:32.213Z
|
||||
- Machine: AMD Ryzen 9 9950X 16-Core Processor x16, linux x64, bun 1.3.14
|
||||
- Models (sha256): htdemucs_6s.onnx 35cb61d4d35173ff1d16df7393f41f05651ffdfe13d3c10e06abc26708bd6933; nmp.onnx 2c3c1d144bfa61ad236e92e169c13535c880469a12a047d4e73451f2c059a0ec
|
||||
- SoundFont: alphaTab bundled sonivox.sf2 (Apache-2.0)
|
||||
- Modes: full (stems=false, tempo=false, notes=false)
|
||||
|
||||
> Caveats: vocals are rendered with a GM "Voice Oohs" patch, not real singing, so separation/transcription numbers for vocals are indicative only. Drums use a heuristic baseline transcriber. The SONiVOX guitar is partly routed to Demucs' piano stem (known synthetic-timbre gap). Pitch is pass/fail (song passes at merged-guitar pitch F1 >= 0.85); rhythm is secondary; fingering (string agreement) is informational. Cells are median (min–max).
|
||||
|
||||
## Tier t1a
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.87 (0.64–0.93) ✓ 4/5 | 0.0 (0.0–0.0) | 0.19 (0.13–0.53) | n/a | n/a (424 spurious) | n/a (1260 spurious) | n/a (80 spurious) | n/a (70 spurious) | n/a (200 spurious) | n/a (1592 spurious) | 0.97 (0.52–1.00) | 0.92 (0.08–0.94) | 1.00 (1.00–1.00) | 0 (-3–0) | 0.85 (0.71–0.92) | 0.89 (0.71–0.92) | 0 (0–1) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1a: PASS** (median merged-guitar F1 0.87 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1b
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.81 (0.72–0.93) ✗ 2/5 | 0.0 (0.0–2.0) | 0.22 (0.12–0.43) | n/a | n/a (371 spurious) | 0.37 (0.22–0.38) | 0.98 (0.95–1.00) | 0.49 (0.16–0.76) | 1.00 (0.99–1.00) | n/a (1774 spurious) | 0.99 (0.98–1.00) | 0.88 (0.75–0.96) | 1.00 (1.00–1.00) | 0 (0–0) | 0.95 (0.80–0.99) | 0.83 (0.75–0.94) | 0 (0–0) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1b: FAIL** (median merged-guitar F1 0.81 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1c
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.73 (0.55–0.77) ✗ 0/5 | 1.0 (0.0–8.0) | 0.38 (0.35–0.60) | n/a | 0.21 (0.05–0.24) | 0.32 (0.18–0.43) | 1.00 (0.92–1.00) | 0.75 (0.37–0.92) | 0.99 (0.99–1.00) | 0.11 (0.06–0.18) | 0.96 (0.95–0.98) | 0.74 (0.59–0.80) | 1.00 (1.00–1.00) | 0 (-2–0) | 0.63 (0.27–0.89) | 0.75 (0.58–0.76) | 0 (0–1) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1c: FAIL** (median merged-guitar F1 0.73 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1d
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.72 (0.68–0.76) ✗ 0/5 | 8.0 (0.0–12.0) | 0.28 (0.26–0.35) | 0.68 (0.59–0.70) | 0.29 (0.13–0.52) | 0.37 (0.34–0.45) | 1.00 (0.98–1.00) | 0.41 (0.07–0.58) | 0.99 (0.89–1.00) | 0.10 (0.07–0.15) | 0.93 (0.85–0.96) | 0.66 (0.48–0.78) | 1.00 (1.00–1.00) | 0 (0–0) | 0.78 (0.36–0.96) | 0.74 (0.72–0.78) | 0 (0–0) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1d: FAIL** (median merged-guitar F1 0.72 in `full` mode, threshold 0.85)
|
||||
|
||||
## Per-song appendix
|
||||
|
||||
| Tier | Seed | Mode | Guitar pass | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques | Seconds |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| t1a | 1 | full | ✓ | 0.87 | 0.0 | 0.19 | n/a | n/a (77 spurious) | n/a (320 spurious) | n/a (38 spurious) | n/a (2 spurious) | n/a (3 spurious) | n/a (349 spurious) | 1.00 | 0.93 | 1.00 | 0 | 0.88 | 0.85 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.5 |
|
||||
| t1a | 2 | full | ✗ | 0.64 | 0.0 | 0.53 | n/a | n/a (94 spurious) | n/a (125 spurious) | n/a | n/a (16 spurious) | n/a (84 spurious) | n/a (223 spurious) | 0.56 | 0.25 | 1.00 | -1 | 0.71 | 0.71 | 0 | 1 | n/a (tier 1 has no technique marks) | 21.7 |
|
||||
| t1a | 3 | full | ✓ | 0.86 | 0.0 | 0.21 | n/a | n/a (96 spurious) | n/a (250 spurious) | n/a (42 spurious) | n/a (1 spurious) | n/a (15 spurious) | n/a (258 spurious) | 0.97 | 0.94 | 1.00 | 0 | 0.85 | 0.89 | 0 | 1 | n/a (tier 1 has no technique marks) | 13.1 |
|
||||
| t1a | 4 | full | ✓ | 0.90 | 0.0 | 0.18 | n/a | n/a (77 spurious) | n/a (225 spurious) | n/a | n/a (19 spurious) | n/a (64 spurious) | n/a (309 spurious) | 1.00 | 0.92 | 1.00 | 0 | 0.84 | 0.92 | 0 | 1 | n/a (tier 1 has no technique marks) | 14.7 |
|
||||
| t1a | 5 | full | ✓ | 0.93 | 0.0 | 0.13 | n/a | n/a (80 spurious) | n/a (340 spurious) | n/a | n/a (32 spurious) | n/a (34 spurious) | n/a (453 spurious) | 0.52 | 0.08 | 1.00 | -3 | 0.92 | 0.91 | 1 | 1 | n/a (tier 1 has no technique marks) | 11.9 |
|
||||
| t1b | 1 | full | ✗ | 0.81 | 0.0 | 0.24 | n/a | n/a (91 spurious) | 0.23 | 0.95 | 0.25 | 1.00 | n/a (411 spurious) | 0.99 | 0.75 | 1.00 | 0 | 0.99 | 0.81 | 0 | 1 | n/a (tier 1 has no technique marks) | 26.3 |
|
||||
| t1b | 2 | full | ✗ | 0.72 | 2.0 | 0.43 | n/a | n/a (77 spurious) | 0.38 | 1.00 | 0.54 | 0.99 | n/a (301 spurious) | 0.99 | 0.76 | 1.00 | 0 | 0.80 | 0.75 | 0 | 1 | n/a (tier 1 has no technique marks) | 18.3 |
|
||||
| t1b | 3 | full | ✗ | 0.81 | 0.0 | 0.22 | n/a | n/a (80 spurious) | 0.22 | 0.96 | 0.16 | 1.00 | n/a (307 spurious) | 0.98 | 0.88 | 1.00 | 0 | 0.92 | 0.83 | 0 | 1 | n/a (tier 1 has no technique marks) | 14.4 |
|
||||
| t1b | 4 | full | ✓ | 0.89 | 1.0 | 0.18 | n/a | n/a (60 spurious) | 0.37 | 0.98 | 0.76 | 1.00 | n/a (373 spurious) | 0.99 | 0.88 | 1.00 | 0 | 0.95 | 0.90 | 0 | 1 | n/a (tier 1 has no technique marks) | 22.1 |
|
||||
| t1b | 5 | full | ✓ | 0.93 | 0.0 | 0.12 | n/a | n/a (63 spurious) | 0.38 | 1.00 | 0.49 | 1.00 | n/a (382 spurious) | 1.00 | 0.96 | 1.00 | 0 | 0.95 | 0.94 | 0 | 1 | n/a (tier 1 has no technique marks) | 25.8 |
|
||||
| t1c | 1 | full | ✗ | 0.71 | 0.0 | 0.41 | n/a | 0.21 | 0.18 | 0.92 | 0.37 | 1.00 | 0.11 | 0.96 | 0.59 | 1.00 | -2 | 0.89 | 0.70 | 1 | 1 | n/a (tier 1 has no technique marks) | 28.4 |
|
||||
| t1c | 2 | full | ✗ | 0.55 | 8.0 | 0.60 | n/a | 0.24 | 0.32 | 1.00 | 0.86 | 0.99 | 0.15 | 0.96 | 0.61 | 1.00 | 0 | 0.62 | 0.58 | 0 | 1 | n/a (tier 1 has no technique marks) | 25.9 |
|
||||
| t1c | 3 | full | ✗ | 0.73 | 2.0 | 0.38 | n/a | 0.05 | 0.18 | 0.99 | 0.53 | 1.00 | 0.09 | 0.95 | 0.80 | 1.00 | 0 | 0.69 | 0.75 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.1 |
|
||||
| t1c | 4 | full | ✗ | 0.76 | 1.0 | 0.37 | n/a | 0.09 | 0.36 | 1.00 | 0.92 | 0.99 | 0.18 | 0.98 | 0.74 | 1.00 | 0 | 0.27 | 0.76 | 0 | 1 | n/a (tier 1 has no technique marks) | 32.7 |
|
||||
| t1c | 5 | full | ✗ | 0.77 | 0.0 | 0.35 | n/a | 0.24 | 0.43 | 1.00 | 0.75 | 0.99 | 0.06 | 0.97 | 0.74 | 1.00 | 0 | 0.63 | 0.76 | 0 | 1 | n/a (tier 1 has no technique marks) | 19.8 |
|
||||
| t1d | 1 | full | ✗ | 0.72 | 2.0 | 0.26 | 0.69 | 0.38 | 0.45 | 1.00 | 0.41 | 1.00 | 0.07 | 0.94 | 0.78 | 1.00 | 0 | 0.76 | 0.72 | 0 | 1 | n/a (tier 1 has no technique marks) | 27.7 |
|
||||
| t1d | 2 | full | ✗ | 0.76 | 8.0 | 0.28 | 0.70 | 0.52 | 0.34 | 1.00 | 0.50 | 0.99 | 0.15 | 0.93 | 0.75 | 1.00 | 0 | 0.78 | 0.78 | 0 | 1 | n/a (tier 1 has no technique marks) | 27.8 |
|
||||
| t1d | 3 | full | ✗ | 0.68 | 12.0 | 0.35 | 0.61 | 0.22 | 0.40 | 0.98 | 0.07 | 0.89 | 0.11 | 0.88 | 0.57 | 1.00 | 0 | 0.82 | 0.74 | 0 | 1 | n/a (tier 1 has no technique marks) | 25.0 |
|
||||
| t1d | 4 | full | ✗ | 0.76 | 8.0 | 0.26 | 0.68 | 0.13 | 0.37 | 0.98 | 0.58 | 0.99 | 0.10 | 0.96 | 0.66 | 1.00 | 0 | 0.36 | 0.78 | 0 | 1 | n/a (tier 1 has no technique marks) | 17.5 |
|
||||
| t1d | 5 | full | ✗ | 0.68 | 0.0 | 0.29 | 0.59 | 0.29 | 0.34 | 1.00 | 0.11 | 0.96 | 0.10 | 0.85 | 0.48 | 1.00 | 0 | 0.96 | 0.72 | 0 | 1 | n/a (tier 1 has no technique marks) | 15.3 |
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
# STSheets benchmark report: exp-repeat-2026-10-31
|
||||
|
||||
- Run id: exp-repeat-2026-10-31
|
||||
- Date: 2026-10-30T20:14:56.152Z
|
||||
- Machine: AMD Ryzen 9 9950X 16-Core Processor x16, linux x64, bun 1.3.14
|
||||
- Models (sha256): htdemucs_6s.onnx 35cb61d4d35173ff1d16df7393f41f05651ffdfe13d3c10e06abc26708bd6933; nmp.onnx 2c3c1d144bfa61ad236e92e169c13535c880469a12a047d4e73451f2c059a0ec
|
||||
- SoundFont: alphaTab bundled sonivox.sf2 (Apache-2.0)
|
||||
- Modes: full (stems=false, tempo=false, notes=false)
|
||||
|
||||
> Caveats: vocals are rendered with a GM "Voice Oohs" patch, not real singing, so separation/transcription numbers for vocals are indicative only. Drums use a heuristic baseline transcriber. The SONiVOX guitar is partly routed to Demucs' piano stem (known synthetic-timbre gap). Pitch is pass/fail (song passes at merged-guitar pitch F1 >= 0.85); rhythm is secondary; fingering (string agreement) is informational. Cells are median (min–max).
|
||||
|
||||
## Tier t1a
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.89 (0.63–0.94) ✓ 4/5 | 0.0 (0.0–0.0) | 0.15 (0.12–0.53) | n/a | n/a (424 spurious) | n/a (1260 spurious) | n/a (80 spurious) | n/a (70 spurious) | n/a (200 spurious) | n/a (1592 spurious) | 0.98 (0.52–1.00) | 0.92 (0.08–0.95) | 1.00 (1.00–1.00) | 0 (-3–0) | 0.88 (0.76–0.95) | 0.90 (0.69–0.93) | 0 (0–1) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1a: PASS** (median merged-guitar F1 0.89 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1b
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.84 (0.75–0.95) ✗ 2/5 | 0.0 (0.0–2.0) | 0.18 (0.09–0.39) | n/a | n/a (371 spurious) | 0.37 (0.22–0.38) | 0.98 (0.95–1.00) | 0.49 (0.16–0.76) | 1.00 (0.99–1.00) | n/a (1774 spurious) | 1.00 (0.99–1.00) | 0.89 (0.78–0.96) | 1.00 (1.00–1.00) | 0 (0–0) | 0.95 (0.80–1.00) | 0.84 (0.77–0.96) | 0 (0–0) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1b: FAIL** (median merged-guitar F1 0.84 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1c
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.75 (0.56–0.80) ✗ 0/5 | 1.0 (0.0–8.0) | 0.34 (0.30–0.59) | n/a | 0.21 (0.05–0.24) | 0.32 (0.18–0.43) | 1.00 (0.92–1.00) | 0.75 (0.37–0.92) | 0.99 (0.99–1.00) | 0.11 (0.06–0.18) | 0.98 (0.96–1.00) | 0.75 (0.60–0.80) | 1.00 (1.00–1.00) | 0 (-2–0) | 0.64 (0.34–0.94) | 0.76 (0.60–0.79) | 0 (0–1) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1c: FAIL** (median merged-guitar F1 0.75 in `full` mode, threshold 0.85)
|
||||
|
||||
## Tier t1d
|
||||
|
||||
| Mode | Songs | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| full | 5 | 0.74 (0.65–0.78) ✗ 0/5 | 4.0 (2.0–13.0) | 0.23 (0.22–0.33) | 0.68 (0.57–0.72) | 0.29 (0.13–0.52) | 0.37 (0.34–0.45) | 1.00 (0.98–1.00) | 0.41 (0.07–0.58) | 0.99 (0.89–1.00) | 0.10 (0.07–0.15) | 0.94 (0.85–0.97) | 0.66 (0.48–0.78) | 1.00 (1.00–1.00) | 0 (0–0) | 0.82 (0.39–0.97) | 0.75 (0.69–0.80) | 0 (0–0) | 1 (1–1) | n/a (tier 1 has no technique marks) |
|
||||
|
||||
**Tier t1d: FAIL** (median merged-guitar F1 0.74 in `full` mode, threshold 0.85)
|
||||
|
||||
## Per-song appendix
|
||||
|
||||
| Tier | Seed | Mode | Guitar pass | Guitar merged F1 | Guitar octave errors | Spurious rate | Part-split F1 (t1d) | Vocals F1 | Bass F1 | Drums kick F1 | Drums snare F1 | Drums hihat F1 | Keys F1 | Rhythm onBeat | Rhythm durationOk | Tempo factor | Beat offset | String agreement | GP-file guitar F1 | Bar count match | Time sig match | Techniques | Seconds |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
| t1a | 1 | full | ✓ | 0.89 | 0.0 | 0.15 | n/a | n/a (77 spurious) | n/a (320 spurious) | n/a (38 spurious) | n/a (2 spurious) | n/a (3 spurious) | n/a (349 spurious) | 1.00 | 0.94 | 1.00 | 0 | 0.88 | 0.87 | 0 | 1 | n/a (tier 1 has no technique marks) | 24.2 |
|
||||
| t1a | 2 | full | ✗ | 0.63 | 0.0 | 0.53 | n/a | n/a (94 spurious) | n/a (125 spurious) | n/a | n/a (16 spurious) | n/a (84 spurious) | n/a (223 spurious) | 0.56 | 0.24 | 1.00 | -1 | 0.76 | 0.69 | 0 | 1 | n/a (tier 1 has no technique marks) | 23.5 |
|
||||
| t1a | 3 | full | ✓ | 0.87 | 0.0 | 0.20 | n/a | n/a (96 spurious) | n/a (250 spurious) | n/a (42 spurious) | n/a (1 spurious) | n/a (15 spurious) | n/a (258 spurious) | 0.98 | 0.95 | 1.00 | 0 | 0.85 | 0.90 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.7 |
|
||||
| t1a | 4 | full | ✓ | 0.92 | 0.0 | 0.15 | n/a | n/a (77 spurious) | n/a (225 spurious) | n/a | n/a (19 spurious) | n/a (64 spurious) | n/a (309 spurious) | 1.00 | 0.92 | 1.00 | 0 | 0.94 | 0.93 | 0 | 1 | n/a (tier 1 has no technique marks) | 21.5 |
|
||||
| t1a | 5 | full | ✓ | 0.94 | 0.0 | 0.12 | n/a | n/a (80 spurious) | n/a (340 spurious) | n/a | n/a (32 spurious) | n/a (34 spurious) | n/a (453 spurious) | 0.52 | 0.08 | 1.00 | -3 | 0.95 | 0.91 | 1 | 1 | n/a (tier 1 has no technique marks) | 19.5 |
|
||||
| t1b | 1 | full | ✗ | 0.84 | 0.0 | 0.18 | n/a | n/a (91 spurious) | 0.23 | 0.95 | 0.25 | 1.00 | n/a (411 spurious) | 1.00 | 0.78 | 1.00 | 0 | 0.99 | 0.84 | 0 | 1 | n/a (tier 1 has no technique marks) | 25.3 |
|
||||
| t1b | 2 | full | ✗ | 0.75 | 2.0 | 0.39 | n/a | n/a (77 spurious) | 0.38 | 1.00 | 0.54 | 0.99 | n/a (301 spurious) | 0.99 | 0.78 | 1.00 | 0 | 0.80 | 0.77 | 0 | 1 | n/a (tier 1 has no technique marks) | 26.3 |
|
||||
| t1b | 3 | full | ✗ | 0.82 | 0.0 | 0.19 | n/a | n/a (80 spurious) | 0.22 | 0.96 | 0.16 | 1.00 | n/a (307 spurious) | 0.99 | 0.91 | 1.00 | 0 | 0.93 | 0.83 | 0 | 1 | n/a (tier 1 has no technique marks) | 17.2 |
|
||||
| t1b | 4 | full | ✓ | 0.90 | 1.0 | 0.15 | n/a | n/a (60 spurious) | 0.37 | 0.98 | 0.76 | 1.00 | n/a (373 spurious) | 1.00 | 0.89 | 1.00 | 0 | 0.95 | 0.91 | 0 | 1 | n/a (tier 1 has no technique marks) | 43.1 |
|
||||
| t1b | 5 | full | ✓ | 0.95 | 0.0 | 0.09 | n/a | n/a (63 spurious) | 0.38 | 1.00 | 0.49 | 1.00 | n/a (382 spurious) | 1.00 | 0.96 | 1.00 | 0 | 1.00 | 0.96 | 0 | 1 | n/a (tier 1 has no technique marks) | 31.0 |
|
||||
| t1c | 1 | full | ✗ | 0.75 | 0.0 | 0.34 | n/a | 0.21 | 0.18 | 0.92 | 0.37 | 1.00 | 0.11 | 0.98 | 0.63 | 1.00 | -2 | 0.94 | 0.74 | 1 | 1 | n/a (tier 1 has no technique marks) | 41.9 |
|
||||
| t1c | 2 | full | ✗ | 0.56 | 8.0 | 0.59 | n/a | 0.24 | 0.32 | 1.00 | 0.86 | 0.99 | 0.15 | 0.96 | 0.60 | 1.00 | 0 | 0.63 | 0.60 | 0 | 1 | n/a (tier 1 has no technique marks) | 47.0 |
|
||||
| t1c | 3 | full | ✗ | 0.75 | 1.0 | 0.35 | n/a | 0.05 | 0.18 | 0.99 | 0.53 | 1.00 | 0.09 | 0.96 | 0.80 | 1.00 | 0 | 0.74 | 0.76 | 0 | 1 | n/a (tier 1 has no technique marks) | 23.0 |
|
||||
| t1c | 4 | full | ✗ | 0.79 | 1.0 | 0.33 | n/a | 0.09 | 0.36 | 1.00 | 0.92 | 0.99 | 0.18 | 1.00 | 0.76 | 1.00 | 0 | 0.34 | 0.79 | 0 | 1 | n/a (tier 1 has no technique marks) | 16.1 |
|
||||
| t1c | 5 | full | ✗ | 0.80 | 0.0 | 0.30 | n/a | 0.24 | 0.43 | 1.00 | 0.75 | 0.99 | 0.06 | 0.98 | 0.75 | 1.00 | 0 | 0.64 | 0.78 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.0 |
|
||||
| t1d | 1 | full | ✗ | 0.74 | 2.0 | 0.22 | 0.71 | 0.38 | 0.45 | 1.00 | 0.41 | 1.00 | 0.07 | 0.94 | 0.78 | 1.00 | 0 | 0.82 | 0.74 | 0 | 1 | n/a (tier 1 has no technique marks) | 25.3 |
|
||||
| t1d | 2 | full | ✗ | 0.78 | 7.0 | 0.23 | 0.72 | 0.52 | 0.34 | 1.00 | 0.50 | 0.99 | 0.15 | 0.95 | 0.77 | 1.00 | 0 | 0.80 | 0.80 | 0 | 1 | n/a (tier 1 has no technique marks) | 29.2 |
|
||||
| t1d | 3 | full | ✗ | 0.69 | 13.0 | 0.33 | 0.61 | 0.22 | 0.40 | 0.98 | 0.07 | 0.89 | 0.11 | 0.89 | 0.58 | 1.00 | 0 | 0.87 | 0.75 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.1 |
|
||||
| t1d | 4 | full | ✗ | 0.77 | 4.0 | 0.22 | 0.68 | 0.13 | 0.37 | 0.98 | 0.58 | 0.99 | 0.10 | 0.97 | 0.66 | 1.00 | 0 | 0.39 | 0.78 | 0 | 1 | n/a (tier 1 has no technique marks) | 21.7 |
|
||||
| t1d | 5 | full | ✗ | 0.65 | 4.0 | 0.28 | 0.57 | 0.29 | 0.34 | 1.00 | 0.11 | 0.96 | 0.10 | 0.85 | 0.48 | 1.00 | 0 | 0.97 | 0.69 | 0 | 1 | n/a (tier 1 has no technique marks) | 20.1 |
|
||||
@@ -0,0 +1,728 @@
|
||||
[
|
||||
{
|
||||
"id": "lnoe",
|
||||
"title": "Last Night on Earth (Green Day)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:04.512Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": 0.03,
|
||||
"precision": 0.9296187683284457,
|
||||
"cosine": 0.7456730914857798
|
||||
},
|
||||
"dtwPrecision": 0.9237536656891495
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.9296187683284457,
|
||||
"lift": 0.5562072336265884,
|
||||
"medianResidualMs": 1.6048588715733558,
|
||||
"audibleRate": 0.4055299539170507,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 28,
|
||||
"precision": 0.8928571428571429,
|
||||
"lift": 0.4761904761904762,
|
||||
"medianResidualMs": 1.455394808147048,
|
||||
"audibleRate": 0.3359683794466403,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 43,
|
||||
"precision": 0.9302325581395349,
|
||||
"lift": 0.6046511627906976,
|
||||
"medianResidualMs": 1.1654104135097043,
|
||||
"audibleRate": 0.3695150115473441,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9583333333333334,
|
||||
"lift": 0.5069444444444444,
|
||||
"medianResidualMs": 1.1717814137455207,
|
||||
"audibleRate": 0.3784860557768924,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 54,
|
||||
"precision": 0.9074074074074074,
|
||||
"lift": 0.5061728395061729,
|
||||
"medianResidualMs": 1.803905423017227,
|
||||
"audibleRate": 0.42736842105263156,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9166666666666666,
|
||||
"lift": 0.5625,
|
||||
"medianResidualMs": 1.4688324335949687,
|
||||
"audibleRate": 0.41435185185185186,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 1,
|
||||
"lift": 0.6848484848484848,
|
||||
"medianResidualMs": 1.8973122094507744,
|
||||
"audibleRate": 0.4789356984478936,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.9122807017543859,
|
||||
"lift": 0.5438596491228069,
|
||||
"medianResidualMs": 2.066004036294089,
|
||||
"audibleRate": 0.43863636363636366,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 8,
|
||||
"precision": 0.75,
|
||||
"lift": 0.375,
|
||||
"medianResidualMs": 10.922815385200124,
|
||||
"audibleRate": 0.11538461538461539,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 995,
|
||||
"fp": 1043,
|
||||
"fn": 2068,
|
||||
"precision": 0.48822374877330715,
|
||||
"recall": 0.3248449232778322,
|
||||
"f1": 0.3901195843952166,
|
||||
"octaveErrors": 104
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 108,
|
||||
"fp": 388,
|
||||
"fn": 131,
|
||||
"precision": 0.21774193548387097,
|
||||
"recall": 0.45188284518828453,
|
||||
"f1": 0.2938775510204082,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 84,
|
||||
"fp": 312,
|
||||
"fn": 79,
|
||||
"precision": 0.21212121212121213,
|
||||
"recall": 0.5153374233128835,
|
||||
"f1": 0.3005366726296959,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"drums": {
|
||||
"tp": 417,
|
||||
"fp": 427,
|
||||
"fn": 159,
|
||||
"precision": 0.4940758293838863,
|
||||
"recall": 0.7239583333333334,
|
||||
"f1": 0.5873239436619718,
|
||||
"octaveErrors": 16
|
||||
},
|
||||
"keys": {
|
||||
"tp": 329,
|
||||
"fp": 1553,
|
||||
"fn": 853,
|
||||
"precision": 0.17481402763018067,
|
||||
"recall": 0.27834179357022,
|
||||
"f1": 0.21475195822454307,
|
||||
"octaveErrors": 94
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.4665751813369928,
|
||||
"vocals": 0.39183673469387753,
|
||||
"bass": 0.35062611806797855,
|
||||
"drums": 0.5985915492957746,
|
||||
"keys": 0.25718015665796345
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 98,
|
||||
"fp": 53,
|
||||
"fn": 0,
|
||||
"precision": 0.6490066225165563,
|
||||
"recall": 1,
|
||||
"f1": 0.7871485943775101,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 90,
|
||||
"fp": 211,
|
||||
"fn": 8,
|
||||
"precision": 0.29900332225913623,
|
||||
"recall": 0.9183673469387755,
|
||||
"f1": 0.45112781954887216,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 217,
|
||||
"fp": 156,
|
||||
"fn": 3,
|
||||
"precision": 0.5817694369973191,
|
||||
"recall": 0.9863636363636363,
|
||||
"f1": 0.7318718381112985,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 12,
|
||||
"fp": 7,
|
||||
"fn": 142,
|
||||
"precision": 0.631578947368421,
|
||||
"recall": 0.07792207792207792,
|
||||
"f1": 0.13872832369942195,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 6,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 89.209386787
|
||||
},
|
||||
{
|
||||
"id": "swimming",
|
||||
"title": "Swimming (Lunar Vacation)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 24,
|
||||
"savedAt": "2026-10-30T18:50:09.863Z",
|
||||
"chosen": "dtw",
|
||||
"baseline": {
|
||||
"offsetSec": 0.23,
|
||||
"precision": 0.6965944272445821,
|
||||
"cosine": 0.6263260975223001
|
||||
},
|
||||
"dtwPrecision": 0.7956656346749226
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7956656346749226,
|
||||
"lift": 0.14138286893704843,
|
||||
"medianResidualMs": 15.842554880606485,
|
||||
"audibleRate": 0.2524440582228981,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.9,
|
||||
"lift": 0.5166666666666667,
|
||||
"medianResidualMs": 7.930399045602776,
|
||||
"audibleRate": 0.4263157894736842,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 38,
|
||||
"precision": 0.8421052631578947,
|
||||
"lift": 0.4561403508771929,
|
||||
"medianResidualMs": 8.935720572655015,
|
||||
"audibleRate": 0.4797047970479705,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 30,
|
||||
"precision": 0.9666666666666667,
|
||||
"lift": 0.5,
|
||||
"medianResidualMs": 8.143065510864744,
|
||||
"audibleRate": 0.361244019138756,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 27,
|
||||
"precision": 0.7777777777777778,
|
||||
"lift": 0.39506172839506176,
|
||||
"medianResidualMs": 15.735118333282117,
|
||||
"audibleRate": 0.41919191919191917,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 45,
|
||||
"precision": 0.4666666666666667,
|
||||
"lift": -0.2666666666666667,
|
||||
"medianResidualMs": 2.775248941929931,
|
||||
"audibleRate": 0.1555944055944056,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 47%",
|
||||
"lift -0.27"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.9454545454545454,
|
||||
"lift": 0.16363636363636358,
|
||||
"medianResidualMs": 24.587909988653678,
|
||||
"audibleRate": 0.3125763125763126,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.16"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 52,
|
||||
"precision": 0.6923076923076923,
|
||||
"lift": -0.11538461538461553,
|
||||
"medianResidualMs": 33.98722951001787,
|
||||
"audibleRate": 0.19047619047619047,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.12"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 47,
|
||||
"precision": 0.8297872340425532,
|
||||
"lift": -1.1102230246251565e-16,
|
||||
"medianResidualMs": 23.422793802808428,
|
||||
"audibleRate": 0.21311475409836064,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.00"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 240,
|
||||
"recOnsets": 9,
|
||||
"precision": 1,
|
||||
"lift": 0.14814814814814825,
|
||||
"medianResidualMs": 1.804172777298163,
|
||||
"audibleRate": 0.09375,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.15"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 270,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 25,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 1043,
|
||||
"fp": 2521,
|
||||
"fn": 3949,
|
||||
"precision": 0.29264870931537595,
|
||||
"recall": 0.20893429487179488,
|
||||
"f1": 0.24380551659654043,
|
||||
"octaveErrors": 157
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 35,
|
||||
"fp": 571,
|
||||
"fn": 166,
|
||||
"precision": 0.057755775577557754,
|
||||
"recall": 0.17412935323383086,
|
||||
"f1": 0.08674101610904585,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 185,
|
||||
"fp": 716,
|
||||
"fn": 345,
|
||||
"precision": 0.20532741398446172,
|
||||
"recall": 0.3490566037735849,
|
||||
"f1": 0.2585604472396925,
|
||||
"octaveErrors": 9
|
||||
},
|
||||
"drums": {
|
||||
"tp": 392,
|
||||
"fp": 1408,
|
||||
"fn": 856,
|
||||
"precision": 0.21777777777777776,
|
||||
"recall": 0.3141025641025641,
|
||||
"f1": 0.2572178477690289,
|
||||
"octaveErrors": 142
|
||||
},
|
||||
"keys": {
|
||||
"tp": 0,
|
||||
"fp": 2032,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.3751753155680224,
|
||||
"vocals": 0.12391573729863693,
|
||||
"bass": 0.3535988819007687,
|
||||
"drums": 0.3353018372703412,
|
||||
"keys": 0
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 175,
|
||||
"fp": 393,
|
||||
"fn": 163,
|
||||
"precision": 0.30809859154929575,
|
||||
"recall": 0.5177514792899408,
|
||||
"f1": 0.38631346578366443,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 97,
|
||||
"fp": 363,
|
||||
"fn": 145,
|
||||
"precision": 0.2108695652173913,
|
||||
"recall": 0.40082644628099173,
|
||||
"f1": 0.27635327635327633,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 0,
|
||||
"fp": 465,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 120,
|
||||
"fp": 187,
|
||||
"fn": 510,
|
||||
"precision": 0.39087947882736157,
|
||||
"recall": 0.19047619047619047,
|
||||
"f1": 0.256136606189968,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 38,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 110.59234721600001
|
||||
},
|
||||
{
|
||||
"id": "holes",
|
||||
"title": "Holes (Passenger)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:13.626Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": -0.62,
|
||||
"precision": 0.7642585551330798,
|
||||
"cosine": 0.5150959661213577
|
||||
},
|
||||
"dtwPrecision": 0.7452471482889734
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7642585551330798,
|
||||
"lift": 0.026615969581749055,
|
||||
"medianResidualMs": 26.02722634637189,
|
||||
"audibleRate": 0.2326544701415257,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.8596491228070176,
|
||||
"lift": 0.04093567251462005,
|
||||
"medianResidualMs": 29.71729302428372,
|
||||
"audibleRate": 0.24890190336749635,
|
||||
"beatShift": {
|
||||
"label": "+1 beat",
|
||||
"gain": 0.07068144786351394
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.04",
|
||||
"beat shift suspected: +1 beat"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 53,
|
||||
"precision": 0.8867924528301887,
|
||||
"lift": 0.07547169811320753,
|
||||
"medianResidualMs": 22.94817039821595,
|
||||
"audibleRate": 0.29635499207606975,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.8727272727272727,
|
||||
"lift": 0.07272727272727275,
|
||||
"medianResidualMs": 20.39969875089298,
|
||||
"audibleRate": 0.2913793103448276,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.07"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 36,
|
||||
"precision": 0.5833333333333334,
|
||||
"lift": -0.08333333333333326,
|
||||
"medianResidualMs": 26.357498430371606,
|
||||
"audibleRate": 0.24187725631768953,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 58%",
|
||||
"lift -0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 7,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": 0.05371900826446281,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 35,
|
||||
"precision": 0.6571428571428571,
|
||||
"lift": -0.10476190476190472,
|
||||
"medianResidualMs": 26.66406226160234,
|
||||
"audibleRate": 0.13317191283292978,
|
||||
"beatShift": {
|
||||
"label": "+1 bar",
|
||||
"gain": 0.09281354253241003
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.10",
|
||||
"beat shift suspected: +1 bar"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.5,
|
||||
"lift": 0.2,
|
||||
"medianResidualMs": 35.25757102913474,
|
||||
"audibleRate": 0.18309859154929578,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 50%",
|
||||
"lift 0.20"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 479,
|
||||
"fp": 2547,
|
||||
"fn": 2185,
|
||||
"precision": 0.15829477858559154,
|
||||
"recall": 0.17980480480480482,
|
||||
"f1": 0.16836555360281194,
|
||||
"octaveErrors": 195
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 57,
|
||||
"fp": 623,
|
||||
"fn": 387,
|
||||
"precision": 0.0838235294117647,
|
||||
"recall": 0.12837837837837837,
|
||||
"f1": 0.10142348754448399,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"bass": {
|
||||
"tp": 15,
|
||||
"fp": 508,
|
||||
"fn": 110,
|
||||
"precision": 0.028680688336520075,
|
||||
"recall": 0.12,
|
||||
"f1": 0.046296296296296294,
|
||||
"octaveErrors": 2
|
||||
},
|
||||
"drums": {
|
||||
"tp": 192,
|
||||
"fp": 1431,
|
||||
"fn": 534,
|
||||
"precision": 0.11829944547134935,
|
||||
"recall": 0.2644628099173554,
|
||||
"f1": 0.16347381864623245,
|
||||
"octaveErrors": 29
|
||||
},
|
||||
"keys": {
|
||||
"tp": 34,
|
||||
"fp": 1057,
|
||||
"fn": 520,
|
||||
"precision": 0.031164069660861594,
|
||||
"recall": 0.061371841155234655,
|
||||
"f1": 0.04133738601823708,
|
||||
"octaveErrors": 18
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.30509666080843584,
|
||||
"vocals": 0.15480427046263345,
|
||||
"bass": 0.07407407407407407,
|
||||
"drums": 0.2835249042145594,
|
||||
"keys": 0.0729483282674772
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 21,
|
||||
"fp": 223,
|
||||
"fn": 99,
|
||||
"precision": 0.0860655737704918,
|
||||
"recall": 0.175,
|
||||
"f1": 0.11538461538461539,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 69,
|
||||
"fp": 261,
|
||||
"fn": 183,
|
||||
"precision": 0.20909090909090908,
|
||||
"recall": 0.27380952380952384,
|
||||
"f1": 0.23711340206185566,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 101,
|
||||
"fp": 603,
|
||||
"fn": 194,
|
||||
"precision": 0.1434659090909091,
|
||||
"recall": 0.3423728813559322,
|
||||
"f1": 0.2022022022022022,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 1,
|
||||
"fp": 344,
|
||||
"fn": 5,
|
||||
"precision": 0.002898550724637681,
|
||||
"recall": 0.16666666666666666,
|
||||
"f1": 0.005698005698005698,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 1,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [
|
||||
"Brass",
|
||||
"Crowd"
|
||||
],
|
||||
"capo": {
|
||||
"Acoustic Guitar ": 9
|
||||
},
|
||||
"seconds": 116.92060425800004
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,78 @@
|
||||
# STSheets real-song benchmark: auto-2026-10-30
|
||||
|
||||
> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). The fit line reports the *consistency of the alignment*, not an independent validation of it: precision = share of the recording's onset-flux peaks with a warped MIDI onset within ±70 ms (flag < 60% per 30 s window), lift = precision minus its value with the MIDI shifted off-grid (flag < 0.25), a symbolic-chroma check for whole-beat/bar shifts, and "MIDI onsets audible as attacks" (information only; a full mix exposes far fewer attacks than the score has notes). A poor alignment lowers every score below. Capo is informational and not applied.
|
||||
|
||||
## Last Night on Earth (Green Day) (lnoe)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:04.512Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 92.4% not better by ≥ 2 points) · constant-offset baseline +0.03 s: precision 93.0%
|
||||
- Consistency of the alignment: precision 93.0% (lift +0.56) · MIDI onsets audible as attacks 40.6% · median residual 1.6 ms · 0/8 windows flagged · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 89.2 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 3063 | 2038 | 0.49 | 0.32 | 0.39 | 0.47 |
|
||||
| vocals | 239 | 496 | 0.22 | 0.45 | 0.29 | 0.39 |
|
||||
| bass | 163 | 396 | 0.21 | 0.52 | 0.30 | 0.35 |
|
||||
| drums | 576 | 844 | 0.49 | 0.72 | 0.59 | 0.60 |
|
||||
| keys | 1182 | 1882 | 0.17 | 0.28 | 0.21 | 0.26 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 98 | 151 | 0.79 |
|
||||
| snare | 98 | 301 | 0.45 |
|
||||
| hihat | 220 | 373 | 0.73 |
|
||||
| cymbal | 154 | 19 | 0.14 |
|
||||
| tom | 6 | 0 | 0.00 |
|
||||
|
||||
## Swimming (Lunar Vacation) (swimming)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:09.863Z (0 manual / 24 auto anchors)
|
||||
- Method: chosen: DTW (precision 79.6%) · constant-offset baseline +0.23 s: precision 69.7%
|
||||
- Consistency of the alignment: precision 79.6% (lift +0.14) · MIDI onsets audible as attacks 25.2% · median residual 15.8 ms · 5/10 windows flagged (2:00 precision 47%, lift -0.27; 2:30 lift 0.16; 3:00 lift -0.12; 3:30 lift -0.00; 4:00 lift 0.15) · stretches: 0/25 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 110.6 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 4992 | 3564 | 0.29 | 0.21 | 0.24 | 0.38 |
|
||||
| vocals | 201 | 606 | 0.06 | 0.17 | 0.09 | 0.12 |
|
||||
| bass | 530 | 901 | 0.21 | 0.35 | 0.26 | 0.35 |
|
||||
| drums | 1248 | 1800 | 0.22 | 0.31 | 0.26 | 0.34 |
|
||||
| keys | 0 | 2032 | 0.00 | n/a | 0.00 | 0.00 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 338 | 568 | 0.39 |
|
||||
| snare | 242 | 460 | 0.28 |
|
||||
| hihat | 0 | 465 | 0.00 |
|
||||
| cymbal | 630 | 307 | 0.26 |
|
||||
| tom | 38 | 0 | 0.00 |
|
||||
|
||||
## Holes (Passenger) (holes)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:13.626Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 74.5% not better by ≥ 2 points) · constant-offset baseline −0.62 s: precision 76.4%
|
||||
- Consistency of the alignment: precision 76.4% (lift +0.03) · MIDI onsets audible as attacks 23.3% · median residual 26.0 ms · 6/8 windows flagged (0:00 lift 0.04, beat shift suspected: +1 beat; 0:30 lift 0.08; 1:00 lift 0.07; 1:30 precision 58%, lift -0.08; 2:30 lift -0.10, beat shift suspected: +1 bar; 3:00 precision 50%, lift 0.20) · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: Acoustic Guitar : 9
|
||||
- Excluded tracks (no role): Brass, Crowd
|
||||
- Pipeline + scoring: 116.9 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 2664 | 3026 | 0.16 | 0.18 | 0.17 | 0.31 |
|
||||
| vocals | 444 | 680 | 0.08 | 0.13 | 0.10 | 0.15 |
|
||||
| bass | 125 | 523 | 0.03 | 0.12 | 0.05 | 0.07 |
|
||||
| drums | 726 | 1623 | 0.12 | 0.26 | 0.16 | 0.28 |
|
||||
| keys | 554 | 1091 | 0.03 | 0.06 | 0.04 | 0.07 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 120 | 244 | 0.12 |
|
||||
| snare | 252 | 330 | 0.24 |
|
||||
| hihat | 295 | 704 | 0.20 |
|
||||
| cymbal | 6 | 345 | 0.01 |
|
||||
| tom | 1 | 0 | 0.00 |
|
||||
@@ -0,0 +1,728 @@
|
||||
[
|
||||
{
|
||||
"id": "lnoe",
|
||||
"title": "Last Night on Earth (Green Day)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:04.512Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": 0.03,
|
||||
"precision": 0.9296187683284457,
|
||||
"cosine": 0.7456730914857798
|
||||
},
|
||||
"dtwPrecision": 0.9237536656891495
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.9296187683284457,
|
||||
"lift": 0.5562072336265884,
|
||||
"medianResidualMs": 1.6048588715733558,
|
||||
"audibleRate": 0.4055299539170507,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 28,
|
||||
"precision": 0.8928571428571429,
|
||||
"lift": 0.4761904761904762,
|
||||
"medianResidualMs": 1.455394808147048,
|
||||
"audibleRate": 0.3359683794466403,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 43,
|
||||
"precision": 0.9302325581395349,
|
||||
"lift": 0.6046511627906976,
|
||||
"medianResidualMs": 1.1654104135097043,
|
||||
"audibleRate": 0.3695150115473441,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9583333333333334,
|
||||
"lift": 0.5069444444444444,
|
||||
"medianResidualMs": 1.1717814137455207,
|
||||
"audibleRate": 0.3784860557768924,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 54,
|
||||
"precision": 0.9074074074074074,
|
||||
"lift": 0.5061728395061729,
|
||||
"medianResidualMs": 1.803905423017227,
|
||||
"audibleRate": 0.42736842105263156,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9166666666666666,
|
||||
"lift": 0.5625,
|
||||
"medianResidualMs": 1.4688324335949687,
|
||||
"audibleRate": 0.41435185185185186,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 1,
|
||||
"lift": 0.6848484848484848,
|
||||
"medianResidualMs": 1.8973122094507744,
|
||||
"audibleRate": 0.4789356984478936,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.9122807017543859,
|
||||
"lift": 0.5438596491228069,
|
||||
"medianResidualMs": 2.066004036294089,
|
||||
"audibleRate": 0.43863636363636366,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 8,
|
||||
"precision": 0.75,
|
||||
"lift": 0.375,
|
||||
"medianResidualMs": 10.922815385200124,
|
||||
"audibleRate": 0.11538461538461539,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 1123,
|
||||
"fp": 1392,
|
||||
"fn": 1940,
|
||||
"precision": 0.44652087475149105,
|
||||
"recall": 0.366634018935684,
|
||||
"f1": 0.40265328074578705,
|
||||
"octaveErrors": 122
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 108,
|
||||
"fp": 388,
|
||||
"fn": 131,
|
||||
"precision": 0.21774193548387097,
|
||||
"recall": 0.45188284518828453,
|
||||
"f1": 0.2938775510204082,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 84,
|
||||
"fp": 312,
|
||||
"fn": 79,
|
||||
"precision": 0.21212121212121213,
|
||||
"recall": 0.5153374233128835,
|
||||
"f1": 0.3005366726296959,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"drums": {
|
||||
"tp": 417,
|
||||
"fp": 427,
|
||||
"fn": 159,
|
||||
"precision": 0.4940758293838863,
|
||||
"recall": 0.7239583333333334,
|
||||
"f1": 0.5873239436619718,
|
||||
"octaveErrors": 16
|
||||
},
|
||||
"keys": {
|
||||
"tp": 329,
|
||||
"fp": 1553,
|
||||
"fn": 853,
|
||||
"precision": 0.17481402763018067,
|
||||
"recall": 0.27834179357022,
|
||||
"f1": 0.21475195822454307,
|
||||
"octaveErrors": 94
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.47615632843313016,
|
||||
"vocals": 0.39183673469387753,
|
||||
"bass": 0.35062611806797855,
|
||||
"drums": 0.5985915492957746,
|
||||
"keys": 0.25718015665796345
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 98,
|
||||
"fp": 53,
|
||||
"fn": 0,
|
||||
"precision": 0.6490066225165563,
|
||||
"recall": 1,
|
||||
"f1": 0.7871485943775101,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 90,
|
||||
"fp": 211,
|
||||
"fn": 8,
|
||||
"precision": 0.29900332225913623,
|
||||
"recall": 0.9183673469387755,
|
||||
"f1": 0.45112781954887216,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 217,
|
||||
"fp": 156,
|
||||
"fn": 3,
|
||||
"precision": 0.5817694369973191,
|
||||
"recall": 0.9863636363636363,
|
||||
"f1": 0.7318718381112985,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 12,
|
||||
"fp": 7,
|
||||
"fn": 142,
|
||||
"precision": 0.631578947368421,
|
||||
"recall": 0.07792207792207792,
|
||||
"f1": 0.13872832369942195,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 6,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 108.26291943300001
|
||||
},
|
||||
{
|
||||
"id": "swimming",
|
||||
"title": "Swimming (Lunar Vacation)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 24,
|
||||
"savedAt": "2026-10-30T18:50:09.863Z",
|
||||
"chosen": "dtw",
|
||||
"baseline": {
|
||||
"offsetSec": 0.23,
|
||||
"precision": 0.6965944272445821,
|
||||
"cosine": 0.6263260975223001
|
||||
},
|
||||
"dtwPrecision": 0.7956656346749226
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7956656346749226,
|
||||
"lift": 0.14138286893704843,
|
||||
"medianResidualMs": 15.842554880606485,
|
||||
"audibleRate": 0.2524440582228981,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.9,
|
||||
"lift": 0.5166666666666667,
|
||||
"medianResidualMs": 7.930399045602776,
|
||||
"audibleRate": 0.4263157894736842,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 38,
|
||||
"precision": 0.8421052631578947,
|
||||
"lift": 0.4561403508771929,
|
||||
"medianResidualMs": 8.935720572655015,
|
||||
"audibleRate": 0.4797047970479705,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 30,
|
||||
"precision": 0.9666666666666667,
|
||||
"lift": 0.5,
|
||||
"medianResidualMs": 8.143065510864744,
|
||||
"audibleRate": 0.361244019138756,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 27,
|
||||
"precision": 0.7777777777777778,
|
||||
"lift": 0.39506172839506176,
|
||||
"medianResidualMs": 15.735118333282117,
|
||||
"audibleRate": 0.41919191919191917,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 45,
|
||||
"precision": 0.4666666666666667,
|
||||
"lift": -0.2666666666666667,
|
||||
"medianResidualMs": 2.775248941929931,
|
||||
"audibleRate": 0.1555944055944056,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 47%",
|
||||
"lift -0.27"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.9454545454545454,
|
||||
"lift": 0.16363636363636358,
|
||||
"medianResidualMs": 24.587909988653678,
|
||||
"audibleRate": 0.3125763125763126,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.16"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 52,
|
||||
"precision": 0.6923076923076923,
|
||||
"lift": -0.11538461538461553,
|
||||
"medianResidualMs": 33.98722951001787,
|
||||
"audibleRate": 0.19047619047619047,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.12"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 47,
|
||||
"precision": 0.8297872340425532,
|
||||
"lift": -1.1102230246251565e-16,
|
||||
"medianResidualMs": 23.422793802808428,
|
||||
"audibleRate": 0.21311475409836064,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.00"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 240,
|
||||
"recOnsets": 9,
|
||||
"precision": 1,
|
||||
"lift": 0.14814814814814825,
|
||||
"medianResidualMs": 1.804172777298163,
|
||||
"audibleRate": 0.09375,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.15"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 270,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 25,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 1169,
|
||||
"fp": 3386,
|
||||
"fn": 3823,
|
||||
"precision": 0.2566410537870472,
|
||||
"recall": 0.2341746794871795,
|
||||
"f1": 0.2448936838797528,
|
||||
"octaveErrors": 250
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 35,
|
||||
"fp": 571,
|
||||
"fn": 166,
|
||||
"precision": 0.057755775577557754,
|
||||
"recall": 0.17412935323383086,
|
||||
"f1": 0.08674101610904585,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 185,
|
||||
"fp": 716,
|
||||
"fn": 345,
|
||||
"precision": 0.20532741398446172,
|
||||
"recall": 0.3490566037735849,
|
||||
"f1": 0.2585604472396925,
|
||||
"octaveErrors": 9
|
||||
},
|
||||
"drums": {
|
||||
"tp": 392,
|
||||
"fp": 1408,
|
||||
"fn": 856,
|
||||
"precision": 0.21777777777777776,
|
||||
"recall": 0.3141025641025641,
|
||||
"f1": 0.2572178477690289,
|
||||
"octaveErrors": 142
|
||||
},
|
||||
"keys": {
|
||||
"tp": 0,
|
||||
"fp": 2032,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.3762438462344192,
|
||||
"vocals": 0.12391573729863693,
|
||||
"bass": 0.3535988819007687,
|
||||
"drums": 0.3353018372703412,
|
||||
"keys": 0
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 175,
|
||||
"fp": 393,
|
||||
"fn": 163,
|
||||
"precision": 0.30809859154929575,
|
||||
"recall": 0.5177514792899408,
|
||||
"f1": 0.38631346578366443,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 97,
|
||||
"fp": 363,
|
||||
"fn": 145,
|
||||
"precision": 0.2108695652173913,
|
||||
"recall": 0.40082644628099173,
|
||||
"f1": 0.27635327635327633,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 0,
|
||||
"fp": 465,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 120,
|
||||
"fp": 187,
|
||||
"fn": 510,
|
||||
"precision": 0.39087947882736157,
|
||||
"recall": 0.19047619047619047,
|
||||
"f1": 0.256136606189968,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 38,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 155.53003568399998
|
||||
},
|
||||
{
|
||||
"id": "holes",
|
||||
"title": "Holes (Passenger)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:13.626Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": -0.62,
|
||||
"precision": 0.7642585551330798,
|
||||
"cosine": 0.5150959661213577
|
||||
},
|
||||
"dtwPrecision": 0.7452471482889734
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7642585551330798,
|
||||
"lift": 0.026615969581749055,
|
||||
"medianResidualMs": 26.02722634637189,
|
||||
"audibleRate": 0.2326544701415257,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.8596491228070176,
|
||||
"lift": 0.04093567251462005,
|
||||
"medianResidualMs": 29.71729302428372,
|
||||
"audibleRate": 0.24890190336749635,
|
||||
"beatShift": {
|
||||
"label": "+1 beat",
|
||||
"gain": 0.07068144786351394
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.04",
|
||||
"beat shift suspected: +1 beat"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 53,
|
||||
"precision": 0.8867924528301887,
|
||||
"lift": 0.07547169811320753,
|
||||
"medianResidualMs": 22.94817039821595,
|
||||
"audibleRate": 0.29635499207606975,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.8727272727272727,
|
||||
"lift": 0.07272727272727275,
|
||||
"medianResidualMs": 20.39969875089298,
|
||||
"audibleRate": 0.2913793103448276,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.07"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 36,
|
||||
"precision": 0.5833333333333334,
|
||||
"lift": -0.08333333333333326,
|
||||
"medianResidualMs": 26.357498430371606,
|
||||
"audibleRate": 0.24187725631768953,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 58%",
|
||||
"lift -0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 7,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": 0.05371900826446281,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 35,
|
||||
"precision": 0.6571428571428571,
|
||||
"lift": -0.10476190476190472,
|
||||
"medianResidualMs": 26.66406226160234,
|
||||
"audibleRate": 0.13317191283292978,
|
||||
"beatShift": {
|
||||
"label": "+1 bar",
|
||||
"gain": 0.09281354253241003
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.10",
|
||||
"beat shift suspected: +1 bar"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.5,
|
||||
"lift": 0.2,
|
||||
"medianResidualMs": 35.25757102913474,
|
||||
"audibleRate": 0.18309859154929578,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 50%",
|
||||
"lift 0.20"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 481,
|
||||
"fp": 2673,
|
||||
"fn": 2183,
|
||||
"precision": 0.15250475586556753,
|
||||
"recall": 0.18055555555555555,
|
||||
"f1": 0.16534891715366104,
|
||||
"octaveErrors": 189
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 57,
|
||||
"fp": 623,
|
||||
"fn": 387,
|
||||
"precision": 0.0838235294117647,
|
||||
"recall": 0.12837837837837837,
|
||||
"f1": 0.10142348754448399,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"bass": {
|
||||
"tp": 15,
|
||||
"fp": 508,
|
||||
"fn": 110,
|
||||
"precision": 0.028680688336520075,
|
||||
"recall": 0.12,
|
||||
"f1": 0.046296296296296294,
|
||||
"octaveErrors": 2
|
||||
},
|
||||
"drums": {
|
||||
"tp": 192,
|
||||
"fp": 1431,
|
||||
"fn": 534,
|
||||
"precision": 0.11829944547134935,
|
||||
"recall": 0.2644628099173554,
|
||||
"f1": 0.16347381864623245,
|
||||
"octaveErrors": 29
|
||||
},
|
||||
"keys": {
|
||||
"tp": 34,
|
||||
"fp": 1057,
|
||||
"fn": 520,
|
||||
"precision": 0.031164069660861594,
|
||||
"recall": 0.061371841155234655,
|
||||
"f1": 0.04133738601823708,
|
||||
"octaveErrors": 18
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.2911653489171537,
|
||||
"vocals": 0.15480427046263345,
|
||||
"bass": 0.07407407407407407,
|
||||
"drums": 0.2835249042145594,
|
||||
"keys": 0.0729483282674772
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 21,
|
||||
"fp": 223,
|
||||
"fn": 99,
|
||||
"precision": 0.0860655737704918,
|
||||
"recall": 0.175,
|
||||
"f1": 0.11538461538461539,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 69,
|
||||
"fp": 261,
|
||||
"fn": 183,
|
||||
"precision": 0.20909090909090908,
|
||||
"recall": 0.27380952380952384,
|
||||
"f1": 0.23711340206185566,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 101,
|
||||
"fp": 603,
|
||||
"fn": 194,
|
||||
"precision": 0.1434659090909091,
|
||||
"recall": 0.3423728813559322,
|
||||
"f1": 0.2022022022022022,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 1,
|
||||
"fp": 344,
|
||||
"fn": 5,
|
||||
"precision": 0.002898550724637681,
|
||||
"recall": 0.16666666666666666,
|
||||
"f1": 0.005698005698005698,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 1,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [
|
||||
"Brass",
|
||||
"Crowd"
|
||||
],
|
||||
"capo": {
|
||||
"Acoustic Guitar ": 9
|
||||
},
|
||||
"seconds": 127.737292572
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,78 @@
|
||||
# STSheets real-song benchmark: gpo-2026-10-31
|
||||
|
||||
> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). The fit line reports the *consistency of the alignment*, not an independent validation of it: precision = share of the recording's onset-flux peaks with a warped MIDI onset within ±70 ms (flag < 60% per 30 s window), lift = precision minus its value with the MIDI shifted off-grid (flag < 0.25), a symbolic-chroma check for whole-beat/bar shifts, and "MIDI onsets audible as attacks" (information only; a full mix exposes far fewer attacks than the score has notes). A poor alignment lowers every score below. Capo is informational and not applied.
|
||||
|
||||
## Last Night on Earth (Green Day) (lnoe)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:04.512Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 92.4% not better by ≥ 2 points) · constant-offset baseline +0.03 s: precision 93.0%
|
||||
- Consistency of the alignment: precision 93.0% (lift +0.56) · MIDI onsets audible as attacks 40.6% · median residual 1.6 ms · 0/8 windows flagged · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 108.3 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 3063 | 2515 | 0.45 | 0.37 | 0.40 | 0.48 |
|
||||
| vocals | 239 | 496 | 0.22 | 0.45 | 0.29 | 0.39 |
|
||||
| bass | 163 | 396 | 0.21 | 0.52 | 0.30 | 0.35 |
|
||||
| drums | 576 | 844 | 0.49 | 0.72 | 0.59 | 0.60 |
|
||||
| keys | 1182 | 1882 | 0.17 | 0.28 | 0.21 | 0.26 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 98 | 151 | 0.79 |
|
||||
| snare | 98 | 301 | 0.45 |
|
||||
| hihat | 220 | 373 | 0.73 |
|
||||
| cymbal | 154 | 19 | 0.14 |
|
||||
| tom | 6 | 0 | 0.00 |
|
||||
|
||||
## Swimming (Lunar Vacation) (swimming)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:09.863Z (0 manual / 24 auto anchors)
|
||||
- Method: chosen: DTW (precision 79.6%) · constant-offset baseline +0.23 s: precision 69.7%
|
||||
- Consistency of the alignment: precision 79.6% (lift +0.14) · MIDI onsets audible as attacks 25.2% · median residual 15.8 ms · 5/10 windows flagged (2:00 precision 47%, lift -0.27; 2:30 lift 0.16; 3:00 lift -0.12; 3:30 lift -0.00; 4:00 lift 0.15) · stretches: 0/25 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 155.5 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 4992 | 4555 | 0.26 | 0.23 | 0.24 | 0.38 |
|
||||
| vocals | 201 | 606 | 0.06 | 0.17 | 0.09 | 0.12 |
|
||||
| bass | 530 | 901 | 0.21 | 0.35 | 0.26 | 0.35 |
|
||||
| drums | 1248 | 1800 | 0.22 | 0.31 | 0.26 | 0.34 |
|
||||
| keys | 0 | 2032 | 0.00 | n/a | 0.00 | 0.00 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 338 | 568 | 0.39 |
|
||||
| snare | 242 | 460 | 0.28 |
|
||||
| hihat | 0 | 465 | 0.00 |
|
||||
| cymbal | 630 | 307 | 0.26 |
|
||||
| tom | 38 | 0 | 0.00 |
|
||||
|
||||
## Holes (Passenger) (holes)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:13.626Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 74.5% not better by ≥ 2 points) · constant-offset baseline −0.62 s: precision 76.4%
|
||||
- Consistency of the alignment: precision 76.4% (lift +0.03) · MIDI onsets audible as attacks 23.3% · median residual 26.0 ms · 6/8 windows flagged (0:00 lift 0.04, beat shift suspected: +1 beat; 0:30 lift 0.08; 1:00 lift 0.07; 1:30 precision 58%, lift -0.08; 2:30 lift -0.10, beat shift suspected: +1 bar; 3:00 precision 50%, lift 0.20) · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: Acoustic Guitar : 9
|
||||
- Excluded tracks (no role): Brass, Crowd
|
||||
- Pipeline + scoring: 127.7 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 2664 | 3154 | 0.15 | 0.18 | 0.17 | 0.29 |
|
||||
| vocals | 444 | 680 | 0.08 | 0.13 | 0.10 | 0.15 |
|
||||
| bass | 125 | 523 | 0.03 | 0.12 | 0.05 | 0.07 |
|
||||
| drums | 726 | 1623 | 0.12 | 0.26 | 0.16 | 0.28 |
|
||||
| keys | 554 | 1091 | 0.03 | 0.06 | 0.04 | 0.07 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 120 | 244 | 0.12 |
|
||||
| snare | 252 | 330 | 0.24 |
|
||||
| hihat | 295 | 704 | 0.20 |
|
||||
| cymbal | 6 | 345 | 0.01 |
|
||||
| tom | 1 | 0 | 0.00 |
|
||||
@@ -0,0 +1,728 @@
|
||||
[
|
||||
{
|
||||
"id": "lnoe",
|
||||
"title": "Last Night on Earth (Green Day)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:04.512Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": 0.03,
|
||||
"precision": 0.9296187683284457,
|
||||
"cosine": 0.7456730914857798
|
||||
},
|
||||
"dtwPrecision": 0.9237536656891495
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.9296187683284457,
|
||||
"lift": 0.5562072336265884,
|
||||
"medianResidualMs": 1.6048588715733558,
|
||||
"audibleRate": 0.4055299539170507,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 28,
|
||||
"precision": 0.8928571428571429,
|
||||
"lift": 0.4761904761904762,
|
||||
"medianResidualMs": 1.455394808147048,
|
||||
"audibleRate": 0.3359683794466403,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 43,
|
||||
"precision": 0.9302325581395349,
|
||||
"lift": 0.6046511627906976,
|
||||
"medianResidualMs": 1.1654104135097043,
|
||||
"audibleRate": 0.3695150115473441,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9583333333333334,
|
||||
"lift": 0.5069444444444444,
|
||||
"medianResidualMs": 1.1717814137455207,
|
||||
"audibleRate": 0.3784860557768924,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 54,
|
||||
"precision": 0.9074074074074074,
|
||||
"lift": 0.5061728395061729,
|
||||
"medianResidualMs": 1.803905423017227,
|
||||
"audibleRate": 0.42736842105263156,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 48,
|
||||
"precision": 0.9166666666666666,
|
||||
"lift": 0.5625,
|
||||
"medianResidualMs": 1.4688324335949687,
|
||||
"audibleRate": 0.41435185185185186,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 1,
|
||||
"lift": 0.6848484848484848,
|
||||
"medianResidualMs": 1.8973122094507744,
|
||||
"audibleRate": 0.4789356984478936,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.9122807017543859,
|
||||
"lift": 0.5438596491228069,
|
||||
"medianResidualMs": 2.066004036294089,
|
||||
"audibleRate": 0.43863636363636366,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 8,
|
||||
"precision": 0.75,
|
||||
"lift": 0.375,
|
||||
"medianResidualMs": 10.922815385200124,
|
||||
"audibleRate": 0.11538461538461539,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 1485,
|
||||
"fp": 1314,
|
||||
"fn": 1578,
|
||||
"precision": 0.5305466237942122,
|
||||
"recall": 0.48481880509304603,
|
||||
"f1": 0.5066530194472876,
|
||||
"octaveErrors": 118
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 108,
|
||||
"fp": 388,
|
||||
"fn": 131,
|
||||
"precision": 0.21774193548387097,
|
||||
"recall": 0.45188284518828453,
|
||||
"f1": 0.2938775510204082,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 84,
|
||||
"fp": 312,
|
||||
"fn": 79,
|
||||
"precision": 0.21212121212121213,
|
||||
"recall": 0.5153374233128835,
|
||||
"f1": 0.3005366726296959,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"drums": {
|
||||
"tp": 417,
|
||||
"fp": 427,
|
||||
"fn": 159,
|
||||
"precision": 0.4940758293838863,
|
||||
"recall": 0.7239583333333334,
|
||||
"f1": 0.5873239436619718,
|
||||
"octaveErrors": 16
|
||||
},
|
||||
"keys": {
|
||||
"tp": 329,
|
||||
"fp": 1553,
|
||||
"fn": 853,
|
||||
"precision": 0.17481402763018067,
|
||||
"recall": 0.27834179357022,
|
||||
"f1": 0.21475195822454307,
|
||||
"octaveErrors": 94
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.539065165472535,
|
||||
"vocals": 0.39183673469387753,
|
||||
"bass": 0.35062611806797855,
|
||||
"drums": 0.5985915492957746,
|
||||
"keys": 0.25718015665796345
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 98,
|
||||
"fp": 53,
|
||||
"fn": 0,
|
||||
"precision": 0.6490066225165563,
|
||||
"recall": 1,
|
||||
"f1": 0.7871485943775101,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 90,
|
||||
"fp": 211,
|
||||
"fn": 8,
|
||||
"precision": 0.29900332225913623,
|
||||
"recall": 0.9183673469387755,
|
||||
"f1": 0.45112781954887216,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 217,
|
||||
"fp": 156,
|
||||
"fn": 3,
|
||||
"precision": 0.5817694369973191,
|
||||
"recall": 0.9863636363636363,
|
||||
"f1": 0.7318718381112985,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 12,
|
||||
"fp": 7,
|
||||
"fn": 142,
|
||||
"precision": 0.631578947368421,
|
||||
"recall": 0.07792207792207792,
|
||||
"f1": 0.13872832369942195,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 6,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 99.844065053
|
||||
},
|
||||
{
|
||||
"id": "swimming",
|
||||
"title": "Swimming (Lunar Vacation)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 24,
|
||||
"savedAt": "2026-10-30T18:50:09.863Z",
|
||||
"chosen": "dtw",
|
||||
"baseline": {
|
||||
"offsetSec": 0.23,
|
||||
"precision": 0.6965944272445821,
|
||||
"cosine": 0.6263260975223001
|
||||
},
|
||||
"dtwPrecision": 0.7956656346749226
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7956656346749226,
|
||||
"lift": 0.14138286893704843,
|
||||
"medianResidualMs": 15.842554880606485,
|
||||
"audibleRate": 0.2524440582228981,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.9,
|
||||
"lift": 0.5166666666666667,
|
||||
"medianResidualMs": 7.930399045602776,
|
||||
"audibleRate": 0.4263157894736842,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 38,
|
||||
"precision": 0.8421052631578947,
|
||||
"lift": 0.4561403508771929,
|
||||
"medianResidualMs": 8.935720572655015,
|
||||
"audibleRate": 0.4797047970479705,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 30,
|
||||
"precision": 0.9666666666666667,
|
||||
"lift": 0.5,
|
||||
"medianResidualMs": 8.143065510864744,
|
||||
"audibleRate": 0.361244019138756,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 27,
|
||||
"precision": 0.7777777777777778,
|
||||
"lift": 0.39506172839506176,
|
||||
"medianResidualMs": 15.735118333282117,
|
||||
"audibleRate": 0.41919191919191917,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 45,
|
||||
"precision": 0.4666666666666667,
|
||||
"lift": -0.2666666666666667,
|
||||
"medianResidualMs": 2.775248941929931,
|
||||
"audibleRate": 0.1555944055944056,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 47%",
|
||||
"lift -0.27"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.9454545454545454,
|
||||
"lift": 0.16363636363636358,
|
||||
"medianResidualMs": 24.587909988653678,
|
||||
"audibleRate": 0.3125763125763126,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.16"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 52,
|
||||
"precision": 0.6923076923076923,
|
||||
"lift": -0.11538461538461553,
|
||||
"medianResidualMs": 33.98722951001787,
|
||||
"audibleRate": 0.19047619047619047,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.12"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 47,
|
||||
"precision": 0.8297872340425532,
|
||||
"lift": -1.1102230246251565e-16,
|
||||
"medianResidualMs": 23.422793802808428,
|
||||
"audibleRate": 0.21311475409836064,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.00"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 240,
|
||||
"recOnsets": 9,
|
||||
"precision": 1,
|
||||
"lift": 0.14814814814814825,
|
||||
"medianResidualMs": 1.804172777298163,
|
||||
"audibleRate": 0.09375,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.15"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 270,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 25,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 1219,
|
||||
"fp": 3001,
|
||||
"fn": 3773,
|
||||
"precision": 0.2888625592417062,
|
||||
"recall": 0.24419070512820512,
|
||||
"f1": 0.26465479808944853,
|
||||
"octaveErrors": 208
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 35,
|
||||
"fp": 571,
|
||||
"fn": 166,
|
||||
"precision": 0.057755775577557754,
|
||||
"recall": 0.17412935323383086,
|
||||
"f1": 0.08674101610904585,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"bass": {
|
||||
"tp": 185,
|
||||
"fp": 716,
|
||||
"fn": 345,
|
||||
"precision": 0.20532741398446172,
|
||||
"recall": 0.3490566037735849,
|
||||
"f1": 0.2585604472396925,
|
||||
"octaveErrors": 9
|
||||
},
|
||||
"drums": {
|
||||
"tp": 392,
|
||||
"fp": 1408,
|
||||
"fn": 856,
|
||||
"precision": 0.21777777777777776,
|
||||
"recall": 0.3141025641025641,
|
||||
"f1": 0.2572178477690289,
|
||||
"octaveErrors": 142
|
||||
},
|
||||
"keys": {
|
||||
"tp": 0,
|
||||
"fp": 2032,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.3864524533217542,
|
||||
"vocals": 0.12391573729863693,
|
||||
"bass": 0.3535988819007687,
|
||||
"drums": 0.3353018372703412,
|
||||
"keys": 0
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 175,
|
||||
"fp": 393,
|
||||
"fn": 163,
|
||||
"precision": 0.30809859154929575,
|
||||
"recall": 0.5177514792899408,
|
||||
"f1": 0.38631346578366443,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 97,
|
||||
"fp": 363,
|
||||
"fn": 145,
|
||||
"precision": 0.2108695652173913,
|
||||
"recall": 0.40082644628099173,
|
||||
"f1": 0.27635327635327633,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 0,
|
||||
"fp": 465,
|
||||
"fn": 0,
|
||||
"precision": 0,
|
||||
"recall": null,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 120,
|
||||
"fp": 187,
|
||||
"fn": 510,
|
||||
"precision": 0.39087947882736157,
|
||||
"recall": 0.19047619047619047,
|
||||
"f1": 0.256136606189968,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 38,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [],
|
||||
"capo": {},
|
||||
"seconds": 76.04858104300001
|
||||
},
|
||||
{
|
||||
"id": "holes",
|
||||
"title": "Holes (Passenger)",
|
||||
"alignment": {
|
||||
"source": "saved",
|
||||
"manual": 0,
|
||||
"auto": 1,
|
||||
"savedAt": "2026-10-30T18:50:13.626Z",
|
||||
"chosen": "baseline",
|
||||
"baseline": {
|
||||
"offsetSec": -0.62,
|
||||
"precision": 0.7642585551330798,
|
||||
"cosine": 0.5150959661213577
|
||||
},
|
||||
"dtwPrecision": 0.7452471482889734
|
||||
},
|
||||
"fit": {
|
||||
"precision": 0.7642585551330798,
|
||||
"lift": 0.026615969581749055,
|
||||
"medianResidualMs": 26.02722634637189,
|
||||
"audibleRate": 0.2326544701415257,
|
||||
"windows": [
|
||||
{
|
||||
"startSec": 0,
|
||||
"recOnsets": 57,
|
||||
"precision": 0.8596491228070176,
|
||||
"lift": 0.04093567251462005,
|
||||
"medianResidualMs": 29.71729302428372,
|
||||
"audibleRate": 0.24890190336749635,
|
||||
"beatShift": {
|
||||
"label": "+1 beat",
|
||||
"gain": 0.07068144786351394
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.04",
|
||||
"beat shift suspected: +1 beat"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 30,
|
||||
"recOnsets": 53,
|
||||
"precision": 0.8867924528301887,
|
||||
"lift": 0.07547169811320753,
|
||||
"medianResidualMs": 22.94817039821595,
|
||||
"audibleRate": 0.29635499207606975,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 60,
|
||||
"recOnsets": 55,
|
||||
"precision": 0.8727272727272727,
|
||||
"lift": 0.07272727272727275,
|
||||
"medianResidualMs": 20.39969875089298,
|
||||
"audibleRate": 0.2913793103448276,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift 0.07"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 90,
|
||||
"recOnsets": 36,
|
||||
"precision": 0.5833333333333334,
|
||||
"lift": -0.08333333333333326,
|
||||
"medianResidualMs": 26.357498430371606,
|
||||
"audibleRate": 0.24187725631768953,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 58%",
|
||||
"lift -0.08"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 120,
|
||||
"recOnsets": 7,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": 0.05371900826446281,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
},
|
||||
{
|
||||
"startSec": 150,
|
||||
"recOnsets": 35,
|
||||
"precision": 0.6571428571428571,
|
||||
"lift": -0.10476190476190472,
|
||||
"medianResidualMs": 26.66406226160234,
|
||||
"audibleRate": 0.13317191283292978,
|
||||
"beatShift": {
|
||||
"label": "+1 bar",
|
||||
"gain": 0.09281354253241003
|
||||
},
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"lift -0.10",
|
||||
"beat shift suspected: +1 bar"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 180,
|
||||
"recOnsets": 20,
|
||||
"precision": 0.5,
|
||||
"lift": 0.2,
|
||||
"medianResidualMs": 35.25757102913474,
|
||||
"audibleRate": 0.18309859154929578,
|
||||
"beatShift": null,
|
||||
"flagged": true,
|
||||
"reasons": [
|
||||
"precision 50%",
|
||||
"lift 0.20"
|
||||
]
|
||||
},
|
||||
{
|
||||
"startSec": 210,
|
||||
"recOnsets": 0,
|
||||
"precision": null,
|
||||
"lift": null,
|
||||
"medianResidualMs": null,
|
||||
"audibleRate": null,
|
||||
"beatShift": null,
|
||||
"flagged": false,
|
||||
"reasons": []
|
||||
}
|
||||
],
|
||||
"stretches": {
|
||||
"total": 2,
|
||||
"warned": 0,
|
||||
"over15": 0
|
||||
}
|
||||
},
|
||||
"perRole": {
|
||||
"guitar": {
|
||||
"tp": 540,
|
||||
"fp": 2916,
|
||||
"fn": 2124,
|
||||
"precision": 0.15625,
|
||||
"recall": 0.20270270270270271,
|
||||
"f1": 0.17647058823529413,
|
||||
"octaveErrors": 207
|
||||
},
|
||||
"vocals": {
|
||||
"tp": 57,
|
||||
"fp": 623,
|
||||
"fn": 387,
|
||||
"precision": 0.0838235294117647,
|
||||
"recall": 0.12837837837837837,
|
||||
"f1": 0.10142348754448399,
|
||||
"octaveErrors": 1
|
||||
},
|
||||
"bass": {
|
||||
"tp": 15,
|
||||
"fp": 508,
|
||||
"fn": 110,
|
||||
"precision": 0.028680688336520075,
|
||||
"recall": 0.12,
|
||||
"f1": 0.046296296296296294,
|
||||
"octaveErrors": 2
|
||||
},
|
||||
"drums": {
|
||||
"tp": 192,
|
||||
"fp": 1431,
|
||||
"fn": 534,
|
||||
"precision": 0.11829944547134935,
|
||||
"recall": 0.2644628099173554,
|
||||
"f1": 0.16347381864623245,
|
||||
"octaveErrors": 29
|
||||
},
|
||||
"keys": {
|
||||
"tp": 34,
|
||||
"fp": 1057,
|
||||
"fn": 520,
|
||||
"precision": 0.031164069660861594,
|
||||
"recall": 0.061371841155234655,
|
||||
"f1": 0.04133738601823708,
|
||||
"octaveErrors": 18
|
||||
}
|
||||
},
|
||||
"lenient": {
|
||||
"guitar": 0.3055555555555556,
|
||||
"vocals": 0.15480427046263345,
|
||||
"bass": 0.07407407407407407,
|
||||
"drums": 0.2835249042145594,
|
||||
"keys": 0.0729483282674772
|
||||
},
|
||||
"drumsPerClass": {
|
||||
"kick": {
|
||||
"tp": 21,
|
||||
"fp": 223,
|
||||
"fn": 99,
|
||||
"precision": 0.0860655737704918,
|
||||
"recall": 0.175,
|
||||
"f1": 0.11538461538461539,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"snare": {
|
||||
"tp": 69,
|
||||
"fp": 261,
|
||||
"fn": 183,
|
||||
"precision": 0.20909090909090908,
|
||||
"recall": 0.27380952380952384,
|
||||
"f1": 0.23711340206185566,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"hihat": {
|
||||
"tp": 101,
|
||||
"fp": 603,
|
||||
"fn": 194,
|
||||
"precision": 0.1434659090909091,
|
||||
"recall": 0.3423728813559322,
|
||||
"f1": 0.2022022022022022,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"cymbal": {
|
||||
"tp": 1,
|
||||
"fp": 344,
|
||||
"fn": 5,
|
||||
"precision": 0.002898550724637681,
|
||||
"recall": 0.16666666666666666,
|
||||
"f1": 0.005698005698005698,
|
||||
"octaveErrors": 0
|
||||
},
|
||||
"tom": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 1,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": 0,
|
||||
"octaveErrors": 0
|
||||
}
|
||||
},
|
||||
"excludedTracks": [
|
||||
"Brass",
|
||||
"Crowd"
|
||||
],
|
||||
"capo": {
|
||||
"Acoustic Guitar ": 9
|
||||
},
|
||||
"seconds": 58.31966715700002
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,78 @@
|
||||
# STSheets real-song benchmark: repeat-2026-10-31
|
||||
|
||||
> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). The fit line reports the *consistency of the alignment*, not an independent validation of it: precision = share of the recording's onset-flux peaks with a warped MIDI onset within ±70 ms (flag < 60% per 30 s window), lift = precision minus its value with the MIDI shifted off-grid (flag < 0.25), a symbolic-chroma check for whole-beat/bar shifts, and "MIDI onsets audible as attacks" (information only; a full mix exposes far fewer attacks than the score has notes). A poor alignment lowers every score below. Capo is informational and not applied.
|
||||
|
||||
## Last Night on Earth (Green Day) (lnoe)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:04.512Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 92.4% not better by ≥ 2 points) · constant-offset baseline +0.03 s: precision 93.0%
|
||||
- Consistency of the alignment: precision 93.0% (lift +0.56) · MIDI onsets audible as attacks 40.6% · median residual 1.6 ms · 0/8 windows flagged · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 99.8 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 3063 | 2799 | 0.53 | 0.48 | 0.51 | 0.54 |
|
||||
| vocals | 239 | 496 | 0.22 | 0.45 | 0.29 | 0.39 |
|
||||
| bass | 163 | 396 | 0.21 | 0.52 | 0.30 | 0.35 |
|
||||
| drums | 576 | 844 | 0.49 | 0.72 | 0.59 | 0.60 |
|
||||
| keys | 1182 | 1882 | 0.17 | 0.28 | 0.21 | 0.26 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 98 | 151 | 0.79 |
|
||||
| snare | 98 | 301 | 0.45 |
|
||||
| hihat | 220 | 373 | 0.73 |
|
||||
| cymbal | 154 | 19 | 0.14 |
|
||||
| tom | 6 | 0 | 0.00 |
|
||||
|
||||
## Swimming (Lunar Vacation) (swimming)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:09.863Z (0 manual / 24 auto anchors)
|
||||
- Method: chosen: DTW (precision 79.6%) · constant-offset baseline +0.23 s: precision 69.7%
|
||||
- Consistency of the alignment: precision 79.6% (lift +0.14) · MIDI onsets audible as attacks 25.2% · median residual 15.8 ms · 5/10 windows flagged (2:00 precision 47%, lift -0.27; 2:30 lift 0.16; 3:00 lift -0.12; 3:30 lift -0.00; 4:00 lift 0.15) · stretches: 0/25 warned, 0 > 15%
|
||||
- Capo: none listed
|
||||
- Excluded tracks (no role): none
|
||||
- Pipeline + scoring: 76.0 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 4992 | 4220 | 0.29 | 0.24 | 0.26 | 0.39 |
|
||||
| vocals | 201 | 606 | 0.06 | 0.17 | 0.09 | 0.12 |
|
||||
| bass | 530 | 901 | 0.21 | 0.35 | 0.26 | 0.35 |
|
||||
| drums | 1248 | 1800 | 0.22 | 0.31 | 0.26 | 0.34 |
|
||||
| keys | 0 | 2032 | 0.00 | n/a | 0.00 | 0.00 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 338 | 568 | 0.39 |
|
||||
| snare | 242 | 460 | 0.28 |
|
||||
| hihat | 0 | 465 | 0.00 |
|
||||
| cymbal | 630 | 307 | 0.26 |
|
||||
| tom | 38 | 0 | 0.00 |
|
||||
|
||||
## Holes (Passenger) (holes)
|
||||
|
||||
- Alignment: saved 2026-10-30T18:50:13.626Z (0 manual / 1 auto anchors)
|
||||
- Method: chosen: baseline (DTW precision 74.5% not better by ≥ 2 points) · constant-offset baseline −0.62 s: precision 76.4%
|
||||
- Consistency of the alignment: precision 76.4% (lift +0.03) · MIDI onsets audible as attacks 23.3% · median residual 26.0 ms · 6/8 windows flagged (0:00 lift 0.04, beat shift suspected: +1 beat; 0:30 lift 0.08; 1:00 lift 0.07; 1:30 precision 58%, lift -0.08; 2:30 lift -0.10, beat shift suspected: +1 bar; 3:00 precision 50%, lift 0.20) · stretches: 0/2 warned, 0 > 15%
|
||||
- Capo: Acoustic Guitar : 9
|
||||
- Excluded tracks (no role): Brass, Crowd
|
||||
- Pipeline + scoring: 58.3 s
|
||||
|
||||
| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| guitar | 2664 | 3456 | 0.16 | 0.20 | 0.18 | 0.31 |
|
||||
| vocals | 444 | 680 | 0.08 | 0.13 | 0.10 | 0.15 |
|
||||
| bass | 125 | 523 | 0.03 | 0.12 | 0.05 | 0.07 |
|
||||
| drums | 726 | 1623 | 0.12 | 0.26 | 0.16 | 0.28 |
|
||||
| keys | 554 | 1091 | 0.03 | 0.06 | 0.04 | 0.07 |
|
||||
|
||||
| Drum class | Ref | Est | F1 ±50 ms |
|
||||
| --- | --- | --- | --- |
|
||||
| kick | 120 | 244 | 0.12 |
|
||||
| snare | 252 | 330 | 0.24 |
|
||||
| hihat | 295 | 704 | 0.20 |
|
||||
| cymbal | 6 | 345 | 0.01 |
|
||||
| tom | 1 | 0 | 0.00 |
|
||||
@@ -0,0 +1,495 @@
|
||||
// Automatic MIDI -> recording alignment (coarse-to-fine DTW on chroma + onset features) and its fit quality.
|
||||
// Pure: no I/O, no Bun/process/fs. The SoundFont bytes are passed in.
|
||||
import type { Audio, Role } from "../core/types.ts";
|
||||
import type { Smf } from "./smf.ts";
|
||||
import { makeWarp, stretches, type Anchor, type Warp } from "./warp.ts";
|
||||
import { renderMidi } from "./render.ts";
|
||||
import { features, pickOnsets, type Features } from "./features.ts";
|
||||
import { dtw, type DtwOptions } from "./dtw.ts";
|
||||
import { trackKeys } from "./roles.ts";
|
||||
import { MATCH_TOL_SEC, fitQuality, meanCosine, precisionOf, uniqueSorted, type FitReport, type ShiftInputs, type StretchCounts } from "./fit.ts";
|
||||
|
||||
const FINE_RATE = 50;
|
||||
const COARSE_FACTOR = 5; // coarse pass at 10 Hz
|
||||
const LAMBDA = 1; // weight of the onset-strength term in the frame cost
|
||||
const FINE_BANDS_SEC = [0.5, 1, 2]; // fine pass band half-widths tried in turn before falling back to the coarse path
|
||||
const ANCHOR_HALF_WIN_SEC = 1; // path points within ±1 s (warped time) of a bar estimate its offset
|
||||
const MIN_POINTS = 10; // diagonal path points required to place a bar anchor
|
||||
const MANUAL_CLEARANCE_SEC = 1; // no auto anchor within 1 s (MIDI time) of a manual anchor
|
||||
const OUTLIER_BEATS = 0.25; // per-bar offsets further than ¼ beat from the robust median of ±3 neighbours are rejected
|
||||
const OUTLIER_NEIGHBOURS = 3;
|
||||
const BASELINE_RANGE_SEC = 2, BASELINE_STEP_SEC = 0.01; // constant-offset baseline grid
|
||||
const BASELINE_MARGIN = 0.02; // DTW must beat the baseline's precision by this much to be kept (no manual anchors)
|
||||
const BASELINE_PLATEAU = 0.05; // within a precision peak, offsets this close to its best are refined by residual
|
||||
const BASELINE_TIE = 0.1; // precision peaks this close to the best are told apart by symbolic chroma (beat aliasing)
|
||||
const OVER_TEMPO = 0.15; // stretches with |tempo change| above this are counted separately in the fit
|
||||
const ZERO = 1e-9;
|
||||
|
||||
export interface MeterGrid { beats: number[]; bars: number[]; /** end of the last bar */ end: number; /** the MIDI ends before its last bar does */ lastBarPartial: boolean }
|
||||
|
||||
/**
|
||||
* Beat and bar start times (MIDI seconds) from the tempo map and time signatures (4/4 when none; a beat is one
|
||||
* denominator note), for bars starting before `untilSec` (default: the MIDI duration). A signature change mid-bar
|
||||
* restarts the bar there.
|
||||
*/
|
||||
export function meterGrid(smf: Smf, untilSec = smf.durationSec): MeterGrid {
|
||||
const tempos = smf.tempoMap.length ? [...smf.tempoMap].sort((a, b) => a.sec - b.sec) : [{ sec: 0, bpm: 120 }];
|
||||
const sigs = [...smf.timeSigs].sort((a, b) => a.sec - b.sec);
|
||||
const bpmAt = (t: number) => { let b = tempos[0]!.bpm; for (const p of tempos) if (p.sec <= t + ZERO) b = p.bpm; return b; };
|
||||
// time after advancing `q` quarter notes from `t`, honouring tempo changes on the way
|
||||
const advance = (t: number, q: number): number => {
|
||||
while (q > ZERO) {
|
||||
const spq = 60 / bpmAt(t);
|
||||
const next = tempos.find((p) => p.sec > t + ZERO);
|
||||
const span = next ? (next.sec - t) / spq : Infinity;
|
||||
if (q <= span) return t + q * spq;
|
||||
q -= span; t = next!.sec;
|
||||
}
|
||||
return t;
|
||||
};
|
||||
const beats: number[] = [], bars: number[] = [];
|
||||
let t = 0;
|
||||
while (t < untilSec - ZERO && bars.length < 100000) {
|
||||
bars.push(t);
|
||||
let num = 4, den = 4;
|
||||
for (const s of sigs) if (s.sec <= t + ZERO) { num = s.num; den = s.den; }
|
||||
const change = sigs.find((s) => s.sec > t + ZERO)?.sec ?? Infinity;
|
||||
let b = t;
|
||||
for (let k = 0; k < num; k++) {
|
||||
beats.push(b);
|
||||
const nb = advance(b, 4 / den);
|
||||
if (nb >= change - ZERO) { b = change; break; } // signature change mid-bar: restart there
|
||||
b = nb;
|
||||
}
|
||||
if (!(b > t)) break;
|
||||
t = b;
|
||||
}
|
||||
return { beats, bars, end: t, lastBarPartial: bars.length > 0 && t > smf.durationSec + 1e-3 };
|
||||
}
|
||||
|
||||
/** Bar start times (MIDI seconds), up to durationSec. */
|
||||
export const barTimes = (smf: Smf): number[] => meterGrid(smf).bars;
|
||||
/** Beat start times (MIDI seconds) of the bars starting before durationSec. */
|
||||
export const beatTimes = (smf: Smf): number[] => meterGrid(smf).beats;
|
||||
|
||||
/** Time `k` grid steps after `t` on a sorted grid (fractional positions; linear extrapolation outside it). */
|
||||
function gridShift(grid: number[], t: number, k: number): number {
|
||||
const n = grid.length;
|
||||
if (n < 2) return t;
|
||||
const posOf = (x: number): number => {
|
||||
if (x <= grid[0]!) return (x - grid[0]!) / (grid[1]! - grid[0]!);
|
||||
if (x >= grid[n - 1]!) return n - 1 + (x - grid[n - 1]!) / (grid[n - 1]! - grid[n - 2]!);
|
||||
let lo = 0, hi = n - 1;
|
||||
while (hi - lo > 1) { const m = (lo + hi) >> 1; if (grid[m]! <= x) lo = m; else hi = m; }
|
||||
return lo + (x - grid[lo]!) / (grid[hi]! - grid[lo]!);
|
||||
};
|
||||
const timeOf = (p: number): number => {
|
||||
if (p <= 0) return grid[0]! + p * (grid[1]! - grid[0]!);
|
||||
if (p >= n - 1) return grid[n - 1]! + (p - n + 1) * (grid[n - 1]! - grid[n - 2]!);
|
||||
const i = Math.floor(p);
|
||||
return grid[i]! + (p - i) * (grid[i + 1]! - grid[i]!);
|
||||
};
|
||||
return timeOf(posOf(t) + k);
|
||||
}
|
||||
|
||||
const med = (xs: number[]): number => { const s = [...xs].sort((a, b) => a - b), h = s.length >> 1; return s.length % 2 ? s[h]! : (s[h - 1]! + s[h]!) / 2; };
|
||||
|
||||
/**
|
||||
* Per-bar offsets (recording − warped, s) at MIDI time `midiSec`, with each bar's beat length (s): iteratively drops
|
||||
* the bar deviating most from the robust median of its ±3 kept neighbours' offsets, while that deviation exceeds ¼
|
||||
* beat. The neighbours' offsets are transported to the bar along their median pairwise (Theil–Sen) slope, so a smooth
|
||||
* ramp (a genuine tempo difference) is kept whole instead of being eroded from its ends, where a plain one-sided median
|
||||
* lags the ramp; beat/bar jumps are still rejected.
|
||||
*/
|
||||
export function rejectOffsetOutliers<T extends { midiSec: number; off: number; beatSec: number }>(bars: T[]): T[] {
|
||||
const kept = [...bars];
|
||||
for (;;) {
|
||||
let worst = -1, worstDev = 0;
|
||||
for (let k = 0; k < kept.length; k++) {
|
||||
const nb = [...kept.slice(Math.max(0, k - OUTLIER_NEIGHBOURS), k), ...kept.slice(k + 1, k + 1 + OUTLIER_NEIGHBOURS)];
|
||||
if (!nb.length) continue;
|
||||
const slopes: number[] = [];
|
||||
for (let a = 0; a < nb.length; a++) for (let b = a + 1; b < nb.length; b++) slopes.push((nb[b]!.off - nb[a]!.off) / (nb[b]!.midiSec - nb[a]!.midiSec));
|
||||
const slope = slopes.length ? med(slopes) : 0;
|
||||
const x = kept[k]!.midiSec;
|
||||
const pred = med(nb.map((b) => b.off + slope * (x - b.midiSec)));
|
||||
const dev = Math.abs(kept[k]!.off - pred) / kept[k]!.beatSec;
|
||||
if (dev > OUTLIER_BEATS && dev > worstDev) { worst = k; worstDev = dev; }
|
||||
}
|
||||
if (worst < 0) return kept;
|
||||
kept.splice(worst, 1);
|
||||
}
|
||||
}
|
||||
|
||||
/** Aggregate 50 Hz features by `k` frames (centred): chroma energy summed and re-normalised, onset max, re-z-scored. */
|
||||
function downsample(f: Features, k: number): Features {
|
||||
const n = Math.max(1, Math.floor((f.chroma.length - 1) / k) + 1);
|
||||
const chroma: Float32Array[] = [], onset = new Float32Array(n);
|
||||
for (let o = 0; o < n; o++) {
|
||||
const c = new Float32Array(12);
|
||||
let fl = -Infinity;
|
||||
for (let t = Math.max(0, o * k - (k >> 1)); t <= Math.min(f.chroma.length - 1, o * k + (k >> 1)); t++) {
|
||||
const s = f.chroma[t]!;
|
||||
for (let q = 0; q < 12; q++) c[q]! += s[q]! * s[q]!;
|
||||
fl = Math.max(fl, f.onset[t]!);
|
||||
}
|
||||
let norm = 0;
|
||||
for (let q = 0; q < 12; q++) { c[q] = Math.sqrt(c[q]!); norm += c[q]! * c[q]!; }
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 1e-6) for (let q = 0; q < 12; q++) c[q]! /= norm; else c.fill(0);
|
||||
chroma.push(c); onset[o] = fl;
|
||||
}
|
||||
let mean = 0; for (const v of onset) mean += v; mean /= n;
|
||||
let v2 = 0; for (const v of onset) v2 += (v - mean) ** 2;
|
||||
const sd = Math.sqrt(v2 / n) || 1;
|
||||
for (let o = 0; o < n; o++) onset[o] = (onset[o]! - mean) / sd;
|
||||
return { rate: f.rate / k, onset, chroma };
|
||||
}
|
||||
|
||||
/** Frame cost (1 − cos) + λ|Δonset|. Two silent (all-zero chroma) frames match perfectly; silence vs sound costs 1. */
|
||||
function costFn(A: Features, B: Features): (i: number, j: number) => number {
|
||||
const flat = (f: Features) => { const x = new Float32Array(f.chroma.length * 12); f.chroma.forEach((c, i) => x.set(c, i * 12)); return x; };
|
||||
const zero = (f: Features) => Uint8Array.from(f.chroma, (c) => (c.every((v) => v === 0) ? 1 : 0));
|
||||
const ca = flat(A), cb = flat(B), za = zero(A), zb = zero(B), oa = A.onset, ob = B.onset;
|
||||
return (i, j) => {
|
||||
let c: number;
|
||||
if (za[i] || zb[j]) c = za[i] && zb[j] ? 0 : 1;
|
||||
else {
|
||||
let d = 0;
|
||||
for (let q = 0, p = i * 12, r = j * 12; q < 12; q++) d += ca[p + q]! * cb[r + q]!;
|
||||
c = 1 - d;
|
||||
}
|
||||
return c + LAMBDA * Math.abs(oa[i]! - ob[j]!);
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
type Path = [number, number][];
|
||||
type Open = { openBegin: boolean; openEnd: boolean };
|
||||
|
||||
/** Slope-limited DTW; falls back to the unconstrained symmetric steps when the slope limit makes the end unreachable. */
|
||||
function dtwSlope(costAt: (i: number, j: number) => number, n: number, m: number, open: Open, band?: { center: (i: number) => number; width: number }): Path {
|
||||
const opts: DtwOptions = { steps: "slope", ...open };
|
||||
try {
|
||||
return dtw(costAt, n, m, band, opts);
|
||||
} catch (e) {
|
||||
if (band || !(e instanceof Error) || !/unreachable/.test(e.message)) throw e;
|
||||
return dtw(costAt, n, m, undefined, { ...open, steps: "symmetric" });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Align rendered frames [i0, i1] with recording frames [j0, j1] (inclusive, fine rate), coarse then banded fine, both
|
||||
* slope-limited. `open.openBegin`/`openEnd` let the path start after j0 / stop before j1 (or before i1).
|
||||
*/
|
||||
function alignSegment(fine: [Features, Features], coarse: [Features, Features], i0: number, i1: number, j0: number, j1: number, bandsSec: number[], open: Open): Path {
|
||||
const K = COARSE_FACTOR;
|
||||
const ci0 = Math.round(i0 / K), ci1 = Math.max(ci0, Math.min(coarse[0].chroma.length - 1, Math.round(i1 / K)));
|
||||
const cj0 = Math.round(j0 / K), cj1 = Math.max(cj0, Math.min(coarse[1].chroma.length - 1, Math.round(j1 / K)));
|
||||
const cc = costFn(coarse[0], coarse[1]);
|
||||
const cp = dtwSlope((i, j) => cc(ci0 + i, cj0 + j), ci1 - ci0 + 1, cj1 - cj0 + 1, open);
|
||||
// coarse map: mean recording frame per rendered frame, in segment-relative coarse units; rows the path does not
|
||||
// reach (open end on the last column) continue at slope 1
|
||||
const nRows = ci1 - ci0 + 1;
|
||||
const sum = new Float64Array(nRows), cnt = new Float64Array(nRows);
|
||||
for (const [i, j] of cp) { sum[i]! += j; cnt[i]! += 1; }
|
||||
const cmap = Array.from(sum, (v, i) => (cnt[i] ? v / cnt[i]! : NaN));
|
||||
for (let i = 1; i < nRows; i++) if (Number.isNaN(cmap[i]!)) cmap[i] = cmap[i - 1]! + 1;
|
||||
const centre = (i: number): number => { // fine row (segment-relative) -> fine column (segment-relative)
|
||||
const x = Math.min(cmap.length - 1, Math.max(0, (i0 + i) / K - ci0));
|
||||
const a = Math.floor(x), b = Math.min(cmap.length - 1, a + 1);
|
||||
const y = cmap[a]! + (x - a) * (cmap[b]! - cmap[a]!);
|
||||
return (y + cj0) * K - j0;
|
||||
};
|
||||
const fc = costFn(fine[0], fine[1]);
|
||||
const p = bandedWithRetry((i, j) => fc(i0 + i, j0 + j), i1 - i0 + 1, j1 - j0 + 1, centre, bandsSec, open);
|
||||
if (p) return p.map(([i, j]) => [i0 + i, j0 + j]);
|
||||
return densify(cp, ci0, cj0, i0, i1, j0, j1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fallback: the coarse path expanded to a contiguous fine path from (i0, j0) to (i1, j1). Each coarse step becomes K
|
||||
* fine steps in the same direction (so coarse diagonals stay diagonal and still yield anchors), clamped to the
|
||||
* segment; the lead-in to the coarse path's first cell and any remainder at the end use horizontal/vertical steps
|
||||
* (which carry no timing).
|
||||
*/
|
||||
function densify(cp: Path, ci0: number, cj0: number, i0: number, i1: number, j0: number, j1: number): Path {
|
||||
const K = COARSE_FACTOR;
|
||||
const out: Path = [[i0, j0]];
|
||||
let i = i0, j = j0;
|
||||
const step = (di: number, dj: number) => {
|
||||
const ni = Math.min(i1, i + di), nj = Math.min(j1, j + dj);
|
||||
if (ni !== i || nj !== j) { i = ni; j = nj; out.push([i, j]); }
|
||||
};
|
||||
const [si, sj] = cp[0] ?? [0, 0]; // the coarse path may start off the corner (open begin)
|
||||
while (i < Math.min(i1, (ci0 + si) * K)) step(1, 0);
|
||||
while (j < Math.min(j1, (cj0 + sj) * K)) step(0, 1);
|
||||
for (let k = 1; k < cp.length; k++) {
|
||||
const di = cp[k]![0] - cp[k - 1]![0], dj = cp[k]![1] - cp[k - 1]![1];
|
||||
for (let q = 0; q < K; q++) step(di, dj);
|
||||
}
|
||||
while (i < i1 || j < j1) step(i < i1 ? 1 : 0, j < j1 ? 1 : 0);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Banded fine DTW (slope-limited), widening the band (±0.5 s, ±1 s, ±2 s) when the end is unreachable; null if all fail. */
|
||||
export function bandedWithRetry(
|
||||
costAt: (i: number, j: number) => number, n: number, m: number, centre: (i: number) => number, bandsSec = FINE_BANDS_SEC,
|
||||
open: Open = { openBegin: false, openEnd: false },
|
||||
): Path | null {
|
||||
for (const sec of bandsSec) {
|
||||
try {
|
||||
return dtw(costAt, n, m, { center: centre, width: Math.round(sec * FINE_RATE) }, { steps: "slope", ...open });
|
||||
} catch (e) {
|
||||
if (!(e instanceof Error) || !/unreachable/.test(e.message)) throw e;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Offset (recording − warped) at warped time `w`: median over diagonal path steps within ±1 s, clipped to the
|
||||
* segment [segLo, segHi] (warped seconds); null if too few points. */
|
||||
function offsetAt(diag: { w: number; off: number }[], w: number, segLo: number, segHi: number): number | null {
|
||||
const from = Math.max(segLo, w - ANCHOR_HALF_WIN_SEC), to = Math.min(segHi, w + ANCHOR_HALF_WIN_SEC);
|
||||
let lo = 0, hi = diag.length;
|
||||
while (lo < hi) { const mid = (lo + hi) >> 1; if (diag[mid]!.w < from) lo = mid + 1; else hi = mid; }
|
||||
const offs: number[] = [];
|
||||
for (let k = lo; k < diag.length && diag[k]!.w <= to; k++) offs.push(diag[k]!.off);
|
||||
if (offs.length < MIN_POINTS) return null;
|
||||
offs.sort((a, b) => a - b);
|
||||
const h = offs.length >> 1;
|
||||
return offs.length % 2 ? offs[h]! : (offs[h - 1]! + offs[h]!) / 2;
|
||||
}
|
||||
|
||||
/** Keep manual anchors; keep an auto anchor only if it is strictly monotone against its kept neighbours. */
|
||||
function compose(manual: Anchor[], auto: Anchor[]): Anchor[] {
|
||||
const all = [...manual, ...auto].sort((a, b) => a.midiSec - b.midiSec || (a.kind === "manual" ? -1 : 1));
|
||||
const out: Anchor[] = [];
|
||||
for (let k = 0; k < all.length; k++) {
|
||||
const a = all[k]!;
|
||||
if (a.kind === "manual") {
|
||||
while (out.length && out[out.length - 1]!.kind === "auto" && !(out[out.length - 1]!.audioSec < a.audioSec && out[out.length - 1]!.midiSec < a.midiSec)) out.pop();
|
||||
out.push(a); continue;
|
||||
}
|
||||
const prev = out[out.length - 1];
|
||||
if (prev && !(a.midiSec > prev.midiSec && a.audioSec > prev.audioSec)) continue;
|
||||
const nextManual = all.slice(k + 1).find((x) => x.kind === "manual");
|
||||
if (nextManual && !(a.audioSec < nextManual.audioSec)) continue;
|
||||
out.push(a);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const durationOf = (a: Audio) => (a.channels[0]?.length ?? 0) / a.sampleRate;
|
||||
|
||||
/** What the fit needs from the recording: onset-flux peaks (s) and 10 Hz chroma. Compute once per song. */
|
||||
export interface RecordingCues { onsets: number[]; chroma: Float32Array[]; chromaRate: number; durSec: number }
|
||||
|
||||
function cuesFrom(fA: Features, cA: Features, durSec: number): RecordingCues {
|
||||
return { onsets: pickOnsets(fA.onset, FINE_RATE), chroma: cA.chroma, chromaRate: cA.rate, durSec };
|
||||
}
|
||||
|
||||
/** Onset-flux peaks and chroma of the raw recording. Compute once and reuse with `fitFromCues` to re-score edits cheaply. */
|
||||
export function recordingCues(audio: Audio): RecordingCues {
|
||||
const fA = features(audio, FINE_RATE);
|
||||
return cuesFrom(fA, downsample(fA, COARSE_FACTOR), durationOf(audio));
|
||||
}
|
||||
|
||||
/** Scored-role tracks (by `trackKeys`). */
|
||||
function scoredTracks(smf: Smf, roles: Record<string, Role | null>) {
|
||||
const keys = trackKeys(smf.tracks);
|
||||
return smf.tracks.filter((_, i) => roles[keys[i]!]);
|
||||
}
|
||||
|
||||
/** Piano-roll chroma (pitch classes of sounding pitched notes, 1 per note) with MIDI time placed by `place`. */
|
||||
function symbolicChroma(tracks: Smf["tracks"], place: (m: number) => number, nFrames: number, rate: number): Float32Array[] {
|
||||
const out = Array.from({ length: nFrames }, () => new Float32Array(12));
|
||||
for (const t of tracks) {
|
||||
if (t.isDrums) continue;
|
||||
for (const n of t.notes) {
|
||||
const a = Math.max(0, Math.round(place(n.onset) * rate));
|
||||
const b = Math.min(nFrames, Math.max(a + 1, Math.round(place(n.offset) * rate)));
|
||||
for (let f = a; f < b; f++) out[f]![n.pitch % 12]! += 1;
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Warped MIDI onsets of scored-role notes. */
|
||||
const scoredOnsets = (tracks: Smf["tracks"], place: (m: number) => number) => tracks.flatMap((t) => t.notes.map((n) => place(n.onset)));
|
||||
|
||||
/** Fit of `anchors`: precision/lift/audible rate, the beat/bar-shift chroma check and stretch-warning counts. */
|
||||
export function fitFromCues(cues: RecordingCues, smf: Smf, roles: Record<string, Role | null>, anchors: Anchor[]): FitReport {
|
||||
const warp = makeWarp(anchors);
|
||||
const tracks = scoredTracks(smf, roles);
|
||||
const nFrames = cues.chroma.length, rate = cues.chromaRate;
|
||||
const g = meterGrid(smf, smf.durationSec + 30);
|
||||
const barSteps = (m: number, k: number) => gridShift(g.bars, m, k);
|
||||
const beatSteps = (m: number, k: number) => gridShift(g.beats, m, k);
|
||||
// "+k beat(s)": the MIDI content belongs k beats later in the recording than the current alignment puts it
|
||||
const cands: [string, (m: number) => number][] = [
|
||||
["+1 beat", (m) => beatSteps(m, 1)], ["−1 beat", (m) => beatSteps(m, -1)],
|
||||
["+2 beats", (m) => beatSteps(m, 2)], ["−2 beats", (m) => beatSteps(m, -2)],
|
||||
["+1 bar", (m) => barSteps(m, 1)], ["−1 bar", (m) => barSteps(m, -1)],
|
||||
];
|
||||
const shift: ShiftInputs = {
|
||||
rate, recording: cues.chroma,
|
||||
current: symbolicChroma(tracks, warp.toAudio, nFrames, rate),
|
||||
shifts: cands.map(([label, f]) => ({ label, chroma: symbolicChroma(tracks, (m) => warp.toAudio(f(m)), nFrames, rate) })),
|
||||
};
|
||||
const st = stretches(anchors, smf.durationSec);
|
||||
const counts: StretchCounts = { total: st.length, warned: st.filter((s) => s.warn).length, over15: st.filter((s) => Math.abs(1 / s.ratio - 1) > OVER_TEMPO).length };
|
||||
return fitQuality(cues.onsets, scoredOnsets(tracks, warp.toAudio), cues.durSec, { shift, stretches: counts });
|
||||
}
|
||||
|
||||
/** Fit quality of any alignment (e.g. a hand-made one) against the raw recording. */
|
||||
export function alignmentFit(audio: Audio, smf: Smf, roles: Record<string, Role | null>, anchors: Anchor[]): FitReport {
|
||||
return fitFromCues(recordingCues(audio), smf, roles, anchors);
|
||||
}
|
||||
|
||||
/** Best constant offset (recording − MIDI, s) and its precision; `cosine` = mean symbolic-chroma cosine there. */
|
||||
export interface Baseline { offsetSec: number; precision: number | null; cosine: number | null }
|
||||
|
||||
/**
|
||||
* Constant-offset baseline: grid search of −2..+2 s in 10 ms steps maximising precision. Whole-beat shifts alias onto
|
||||
* the onset grid (equal precision), so the precision peaks within 0.1 of the best are told apart by the mean chroma
|
||||
* cosine of the symbolic MIDI vs the recording. Within a peak precision is flat across the ±70 ms tolerance, so the
|
||||
* offset with the smallest mean residual (among those within 0.05 of the peak's precision) is reported.
|
||||
*/
|
||||
export function constantOffsetBaseline(cues: RecordingCues, smf: Smf, roles: Record<string, Role | null>): Baseline {
|
||||
const tracks = scoredTracks(smf, roles);
|
||||
const midi = uniqueSorted(scoredOnsets(tracks, (m) => m));
|
||||
const steps = Math.round(BASELINE_RANGE_SEC / BASELINE_STEP_SEC);
|
||||
const offs = Array.from({ length: 2 * steps + 1 }, (_, k) => Math.round((k - steps) * BASELINE_STEP_SEC * 100) / 100);
|
||||
const prec = offs.map((o) => precisionOf(cues.onsets, midi, o));
|
||||
if (prec.every((p) => p === null)) return { offsetSec: 0, precision: null, cosine: null };
|
||||
const best = Math.max(...prec.map((p) => p ?? -1));
|
||||
// peaks: runs of consecutive offsets within the tie margin
|
||||
const runs: number[][] = [];
|
||||
for (let k = 0; k < offs.length; k++) {
|
||||
if ((prec[k] ?? -1) < best - BASELINE_TIE) continue;
|
||||
if (k > 0 && (prec[k - 1] ?? -1) >= best - BASELINE_TIE) runs[runs.length - 1]!.push(k); else runs.push([k]);
|
||||
}
|
||||
// within a run, precision is flat across the ±70 ms tolerance (with spikes at its edges, where neighbouring notes
|
||||
// start to match): take the offset with the smallest mean residual among those within 0.05 of the run's best
|
||||
const refine = (run: number[]): number => {
|
||||
const top = Math.max(...run.map((k) => prec[k]!));
|
||||
let pick = run[0]!, bestRes = Infinity;
|
||||
for (const k of run) {
|
||||
if (prec[k]! < top - BASELINE_PLATEAU) continue;
|
||||
const r = meanResidual(cues.onsets, midi, offs[k]!);
|
||||
if (r < bestRes) { bestRes = r; pick = k; }
|
||||
}
|
||||
return pick;
|
||||
};
|
||||
const cosAt = (o: number) => meanCosine(cues.chroma, symbolicChroma(tracks, (m) => m + o, cues.chroma.length, cues.chromaRate), 0, cues.chroma.length, 0);
|
||||
let pick = refine(runs[0]!), pickCos = runs.length > 1 ? cosAt(offs[pick]!) : null;
|
||||
for (const run of runs.slice(1)) {
|
||||
const k = refine(run), c = cosAt(offs[k]!);
|
||||
if (c !== null && (pickCos === null || c > pickCos)) { pick = k; pickCos = c; }
|
||||
}
|
||||
return { offsetSec: offs[pick]!, precision: prec[pick]!, cosine: pickCos ?? cosAt(offs[pick]!) };
|
||||
}
|
||||
|
||||
/** Mean |recording onset − nearest (MIDI + shift)| over recording onsets matched within ±70 ms. */
|
||||
function meanResidual(rec: number[], midi: number[], shift: number): number {
|
||||
let s = 0, n = 0;
|
||||
for (const t of rec) {
|
||||
const x = t - shift;
|
||||
let lo = 0, hi = midi.length;
|
||||
while (lo < hi) { const m = (lo + hi) >> 1; if (midi[m]! < x) lo = m + 1; else hi = m; }
|
||||
const d = Math.min(lo < midi.length ? Math.abs(midi[lo]! - x) : Infinity, lo > 0 ? Math.abs(midi[lo - 1]! - x) : Infinity);
|
||||
if (d <= MATCH_TOL_SEC + 1e-9) { s += d; n++; }
|
||||
}
|
||||
return n ? s / n : Infinity;
|
||||
}
|
||||
|
||||
/** A single auto anchor realising a constant offset: at the bar nearest the middle of the MIDI (positive audio time). */
|
||||
function baselineAnchors(smf: Smf, offsetSec: number): Anchor[] {
|
||||
const bars = barTimes(smf).filter((b) => b + offsetSec > 0.01);
|
||||
const mid = smf.durationSec / 2;
|
||||
const m = bars.length ? bars.reduce((a, b) => (Math.abs(b - mid) < Math.abs(a - mid) ? b : a)) : Math.max(0, -offsetSec) + 1;
|
||||
return [{ midiSec: m, audioSec: m + offsetSec, kind: "auto" }];
|
||||
}
|
||||
|
||||
/** `dtwAnchors`/`dtwPrecision`: the DTW alignment and its precision, also when the baseline replaced it. */
|
||||
export interface AutoResult { anchors: Anchor[]; fit: FitReport; baseline: Baseline; chosen: "baseline" | "dtw"; dtwAnchors: Anchor[]; dtwPrecision: number | null }
|
||||
|
||||
/**
|
||||
* Align `smf` to `audio`, honouring `manual` anchors exactly. The MIDI is rendered under the manual warp, then each
|
||||
* stretch between consecutive manual anchors (plus start/end) is aligned by a slope-limited 10 Hz DTW refined at 50 Hz
|
||||
* in a band around it; the first stretch may start late in the recording (open begin) and the last one may end early
|
||||
* on either axis (open end), so extra intro/outro material does not force the path. One auto anchor is placed per bar
|
||||
* (skipping bars within 1 s of a manual anchor and the final partial bar); per-bar offsets more than ¼ beat from the
|
||||
* median of their ±3 neighbours are rejected. Without manual anchors the result is replaced by the constant-offset
|
||||
* baseline (one auto anchor) unless it beats the baseline's precision by ≥ 0.02. `roles` is keyed by `trackKeys`;
|
||||
* tracks without a role are rendered but excluded from the fit.
|
||||
*/
|
||||
export function autoAlign(
|
||||
audio: Audio, smf: Smf, roles: Record<string, Role | null>, manual: Anchor[], sf2: Uint8Array,
|
||||
opts: { fineBandsSec?: number[] } = {}, // band half-widths for the fine pass; [] forces the coarse-path fallback
|
||||
): AutoResult {
|
||||
const man = makeWarp(manual).anchors.map((a) => ({ ...a, kind: "manual" as const }));
|
||||
const W0 = makeWarp(man);
|
||||
const audioDur = durationOf(audio);
|
||||
const renderDur = Math.max(audioDur, W0.toAudio(smf.durationSec));
|
||||
const rendered = renderMidi(smf.tracks, W0, sf2, { mono: true, durationSec: renderDur });
|
||||
const fR = features(rendered, FINE_RATE), fA = features(audio, FINE_RATE);
|
||||
const cR = downsample(fR, COARSE_FACTOR), cA = downsample(fA, COARSE_FACTOR);
|
||||
const lastR = fR.chroma.length - 1, lastA = fA.chroma.length - 1;
|
||||
const midiEnd = Math.max(1, Math.min(lastR, Math.round(W0.toAudio(smf.durationSec) * FINE_RATE))); // last rendered frame with MIDI content
|
||||
|
||||
// segment boundaries: manual anchors map warped time == recording time, so both axes split at the same instants
|
||||
const cuts = man.map((a) => Math.round(a.audioSec * FINE_RATE)).filter((f) => f > 0 && f < Math.min(midiEnd, lastA));
|
||||
const bounds = [...new Set(cuts)].sort((a, b) => a - b);
|
||||
// per segment: warped-time span and diagonal path steps (they carry timing; horizontal/vertical steps do not)
|
||||
const segs: { lo: number; hi: number; diag: { w: number; off: number }[] }[] = [];
|
||||
let prev = 0;
|
||||
for (const b of [...bounds, -1]) {
|
||||
const i1 = b < 0 ? midiEnd : b, j1 = b < 0 ? lastA : b;
|
||||
const open = { openBegin: prev === 0, openEnd: b < 0 };
|
||||
const path = alignSegment([fR, fA], [cR, cA], prev, i1, prev, j1, opts.fineBandsSec ?? FINE_BANDS_SEC, open);
|
||||
const diag: { w: number; off: number }[] = [];
|
||||
for (let k = 1; k < path.length; k++) {
|
||||
const [i, j] = path[k]!, [pi, pj] = path[k - 1]!;
|
||||
if (i === pi + 1 && j === pj + 1) diag.push({ w: i / FINE_RATE, off: (j - i) / FINE_RATE });
|
||||
}
|
||||
segs.push({ lo: prev / FINE_RATE, hi: b < 0 ? Infinity : i1 / FINE_RATE, diag });
|
||||
prev = b;
|
||||
}
|
||||
|
||||
const grid = meterGrid(smf);
|
||||
const lastBar = grid.lastBarPartial ? grid.bars[grid.bars.length - 1] : undefined;
|
||||
const beatAt = (m: number) => { const k = grid.beats.findIndex((b) => b > m + ZERO); return k > 0 ? grid.beats[k]! - grid.beats[k - 1]! : 0.5; };
|
||||
const perBar: { midiSec: number; w: number; off: number; beatSec: number }[] = [];
|
||||
for (const m of grid.bars) {
|
||||
if (m === lastBar) continue; // never anchor the final partial bar
|
||||
if (man.some((a) => Math.abs(a.midiSec - m) < MANUAL_CLEARANCE_SEC)) continue;
|
||||
const w = W0.toAudio(m);
|
||||
const seg = segs.find((s) => w < s.hi) ?? segs[segs.length - 1]!;
|
||||
const off = offsetAt(seg.diag, w, seg.lo, seg.hi);
|
||||
if (off === null) continue;
|
||||
const a = w + off;
|
||||
if ((m > 0 ? a <= 0 : a < 0) || a > audioDur) continue; // no degenerate first stretch
|
||||
perBar.push({ midiSec: m, w, off, beatSec: beatAt(m) * (W0.toAudio(m + 1) - w) });
|
||||
}
|
||||
const auto = rejectOffsetOutliers(perBar).map((p): Anchor => ({ midiSec: p.midiSec, audioSec: p.w + p.off, kind: "auto" }));
|
||||
const anchors = compose(man, auto);
|
||||
const cues = cuesFrom(fA, cA, audioDur);
|
||||
const fit = fitFromCues(cues, smf, roles, anchors);
|
||||
const baseline = constantOffsetBaseline(cues, smf, roles);
|
||||
const beats = fit.precision !== null && baseline.precision !== null && fit.precision >= baseline.precision + BASELINE_MARGIN;
|
||||
if (man.length || baseline.precision === null || beats) return { anchors, fit, baseline, chosen: "dtw", dtwAnchors: anchors, dtwPrecision: fit.precision };
|
||||
const b = baselineAnchors(smf, baseline.offsetSec);
|
||||
return { anchors: b, fit: fitFromCues(cues, smf, roles, b), baseline, chosen: "baseline", dtwAnchors: anchors, dtwPrecision: fit.precision };
|
||||
}
|
||||
|
||||
/** e.g. "DTW (precision 93.5%) · constant-offset baseline +0.06 s: precision 88.0%"; with "chosen: baseline" the
|
||||
* single-anchor constant offset replaced a DTW alignment that was not better by 2 points. */
|
||||
export function autoSummary(r: { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null }): string {
|
||||
const pc = (x: number | null) => (x === null ? "n/a" : `${(x * 100).toFixed(1)}%`);
|
||||
const off = `${r.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(r.baseline.offsetSec).toFixed(2)} s`;
|
||||
const base = `constant-offset baseline ${off}: precision ${pc(r.baseline.precision)}`;
|
||||
return r.chosen === "dtw" ? `chosen: DTW (precision ${pc(r.dtwPrecision)}) · ${base}`
|
||||
: `chosen: baseline (DTW precision ${pc(r.dtwPrecision)} not better by ≥ 2 points) · ${base}`;
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { parseArgs } from "node:util";
|
||||
import { addSong, loadSong, readAlignment, trackKeys, writeAlignment } from "../ref/manifest.ts";
|
||||
import { parseSmf } from "./smf.ts";
|
||||
import { autoAlign, autoSummary } from "./auto.ts";
|
||||
import { fitSummary } from "./fit.ts";
|
||||
import { SF2_PATH, startServer } from "./server.ts";
|
||||
|
||||
const USAGE = `usage:
|
||||
bun run align [serve] [--host 127.0.0.1] [--port 5180] [--url-host <host shown in the URL>] [--root reference]
|
||||
bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]...
|
||||
bun run align auto <id> [--root reference]`;
|
||||
|
||||
async function add(argv: string[]) {
|
||||
const { values, positionals } = parseArgs({ args: argv, allowPositionals: true, options: { id: { type: "string" }, title: { type: "string" }, capo: { type: "string", multiple: true } } });
|
||||
const [audio, midi] = positionals;
|
||||
if (!audio || !midi || !values.id) throw new Error(USAGE);
|
||||
const smf = parseSmf(new Uint8Array(await Bun.file(midi).arrayBuffer()));
|
||||
const keys = new Set(trackKeys(smf.tracks));
|
||||
const capo: Record<string, number> = {};
|
||||
for (const c of values.capo ?? []) {
|
||||
const i = c.lastIndexOf("=");
|
||||
const k = c.slice(0, i), n = Number(c.slice(i + 1));
|
||||
if (i < 0 || !Number.isInteger(n) || n < 0) throw new Error(`bad --capo ${JSON.stringify(c)}; expected "<track name>=<fret>"`);
|
||||
if (!keys.has(k)) throw new Error(`--capo: no track named ${JSON.stringify(k)}; tracks: ${[...keys].map((x) => JSON.stringify(x)).join(", ")}`);
|
||||
capo[k] = n;
|
||||
}
|
||||
const e = await addSong({ id: values.id, title: values.title ?? values.id, audio, midi, capo });
|
||||
console.log(`added ${e.id} (${e.title})`);
|
||||
for (const [k, r] of Object.entries(e.roles)) console.log(` ${JSON.stringify(k)} -> ${r ?? "unmapped"}`);
|
||||
}
|
||||
|
||||
async function serve(argv: string[]) {
|
||||
const { values } = parseArgs({ args: argv, options: { host: { type: "string" }, port: { type: "string" }, "url-host": { type: "string" }, root: { type: "string" } } });
|
||||
const port = values.port === undefined ? 5180 : Number(values.port);
|
||||
if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error(`bad --port ${values.port}`);
|
||||
const s = await startServer({ host: values.host ?? "127.0.0.1", port, urlHost: values["url-host"], root: values.root ?? "reference" });
|
||||
console.log(`Alignment page: ${s.url}`);
|
||||
console.log(`(listening on ${values.host ?? "127.0.0.1"}:${s.port}; the key in the URL is required and changes on every start)`);
|
||||
}
|
||||
|
||||
async function auto(argv: string[]) {
|
||||
const { values, positionals } = parseArgs({ args: argv, allowPositionals: true, options: { root: { type: "string" } } });
|
||||
const root = values.root ?? "reference";
|
||||
const id = positionals[0];
|
||||
if (!id) throw new Error(USAGE);
|
||||
const s = await loadSong(id, root);
|
||||
const prev = await readAlignment(id, root);
|
||||
const manual = (prev?.anchors ?? []).filter((a) => a.kind === "manual");
|
||||
const keys = trackKeys(s.smf.tracks);
|
||||
const roles = Object.fromEntries(keys.map((k) => [k, s.entry.roles[k] ?? null]));
|
||||
const sf2 = new Uint8Array(await Bun.file(SF2_PATH).arrayBuffer());
|
||||
const t0 = performance.now();
|
||||
const r = autoAlign(s.audio, s.smf, roles, manual, sf2);
|
||||
await writeAlignment(id, { anchors: r.anchors, savedAt: new Date().toISOString(), fit: r.fit, auto: { chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision } }, root);
|
||||
console.log(`${id}: ${r.anchors.length} anchors (${manual.length} manual kept) in ${((performance.now() - t0) / 1000).toFixed(1)} s -> ${root}/${id}/alignment.json`);
|
||||
console.log(`method: ${autoSummary(r)}`);
|
||||
console.log(`fit: ${fitSummary(r.fit)}`);
|
||||
}
|
||||
|
||||
const [cmd, ...rest] = process.argv.slice(2);
|
||||
try {
|
||||
if (cmd === "add") await add(rest);
|
||||
else if (cmd === "auto") await auto(rest);
|
||||
else if (cmd === "serve") await serve(rest);
|
||||
else if (cmd === undefined || cmd.startsWith("--")) await serve(process.argv.slice(2));
|
||||
else throw new Error(USAGE);
|
||||
} catch (err) {
|
||||
console.error((err as Error).message);
|
||||
process.exit(1);
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
// Pure (banded) dynamic time warping. Only the latest accumulated-cost rows are kept; backpointers are one byte per
|
||||
// cell (O(n·band) when banded), so a 3000×3000 unbanded run uses ≈ 9 MB.
|
||||
|
||||
export interface DtwBand { center: (i: number) => number; width: number }
|
||||
/**
|
||||
* steps: "symmetric" (default) — steps (1,0), (0,1), (1,1), each paying the cost of the cell entered.
|
||||
* "slope" — slope-limited (Sakoe–Chiba P = 1, asymmetric weights): (1,1), (1,2) via (i, j−1), (2,1) via
|
||||
* (i−1, j). The local slope stays within [1/2, 2], so the path cannot make abrupt jumps; each row (query frame)
|
||||
* carries weight 1, so paths that consume all rows are comparable whatever columns they cover.
|
||||
* openBegin: the path may start anywhere on row 0 (the target may have extra material before the query).
|
||||
* openEnd: the path may end anywhere on the last row or the last column (target or query has extra material at the
|
||||
* end); candidates are compared by mean cost per query row, each query row left unconsumed costing 1 (the
|
||||
* cost of sound vs silence), so a trivial path into a cheap corner cannot win.
|
||||
*/
|
||||
export interface DtwOptions { steps?: "symmetric" | "slope"; openBegin?: boolean; openEnd?: boolean }
|
||||
|
||||
const DIAG = 0, UP = 1, LEFT = 2, S12 = 3, S21 = 4; // from (i-1,j-1), (i-1,j), (i,j-1), (i-1,j-2) via (i,j-1), (i-2,j-1) via (i-1,j)
|
||||
const START = 5;
|
||||
const UNCONSUMED_ROW_COST = 1;
|
||||
|
||||
/** Minimum-cost monotone path from (0,0) to (n-1,m-1) (or open ends, see `DtwOptions`). */
|
||||
export function dtw(
|
||||
costAt: (i: number, j: number) => number,
|
||||
n: number,
|
||||
m: number,
|
||||
band?: DtwBand,
|
||||
opts: DtwOptions = {},
|
||||
): [number, number][] {
|
||||
if (n <= 0 || m <= 0) return [];
|
||||
const slope = opts.steps === "slope", openBegin = !!opts.openBegin, openEnd = !!opts.openEnd;
|
||||
const lo = new Int32Array(n);
|
||||
const hi = new Int32Array(n);
|
||||
const off = new Int32Array(n + 1);
|
||||
for (let i = 0; i < n; i++) {
|
||||
let a = 0, b = m - 1;
|
||||
if (band) {
|
||||
const c = Math.round(band.center(i));
|
||||
a = Math.max(0, c - band.width);
|
||||
b = Math.min(m - 1, c + band.width);
|
||||
if (i === 0 && !openBegin) a = 0;
|
||||
if (i === n - 1 && !openEnd) b = m - 1;
|
||||
if (a > b) { // centre outside the matrix: clamp to the nearest edge
|
||||
a = b = Math.min(m - 1, Math.max(0, c));
|
||||
}
|
||||
}
|
||||
lo[i] = a; hi[i] = b;
|
||||
off[i + 1] = off[i]! + (b - a + 1);
|
||||
}
|
||||
const bp = new Uint8Array(off[n]!);
|
||||
// accumulated cost rows (row i in D[i % 3]) and cell-cost rows (row i in C[i % 2]); full width, cleared by range
|
||||
const D = [0, 1, 2].map(() => new Float64Array(m).fill(Infinity));
|
||||
const C = [0, 1].map(() => new Float64Array(m));
|
||||
const inRow = (r: number, j: number) => r >= 0 && j >= lo[r]! && j <= hi[r]!;
|
||||
const acc = (r: number, j: number) => (inRow(r, j) ? D[r % 3]![j]! : Infinity);
|
||||
// open-end candidates: mean cost per query row, each row left unconsumed (ending on the last column) costing 1
|
||||
let endScore = Infinity, endI = n - 1, endJ = m - 1;
|
||||
const consider = (i: number, v: number, j: number) => {
|
||||
const score = (v + (n - 1 - i) * UNCONSUMED_ROW_COST) / n;
|
||||
if (v < Infinity && score < endScore - 1e-12) { endScore = score; endI = i; endJ = j; }
|
||||
};
|
||||
for (let i = 0; i < n; i++) {
|
||||
const a = lo[i]!, b = hi[i]!, base = off[i]! - a;
|
||||
const cur = D[i % 3]!, cc = C[i % 2]!, pc = C[(i + 1) % 2]!;
|
||||
if (i >= 3) cur.fill(Infinity, lo[i - 3]!, hi[i - 3]! + 1);
|
||||
for (let j = a; j <= b; j++) cc[j] = costAt(i, j);
|
||||
for (let j = a; j <= b; j++) {
|
||||
let best = Infinity, dir = DIAG;
|
||||
const d = cc[j]!;
|
||||
if (i === 0 && (j === 0 || openBegin)) { best = 0; dir = START; }
|
||||
else if (slope) {
|
||||
const dg = acc(i - 1, j - 1);
|
||||
if (dg + d < best) { best = dg + d; dir = DIAG; }
|
||||
const h = acc(i - 1, j - 2);
|
||||
if (h < Infinity) {
|
||||
const v = h + ((j - 1 >= a ? cc[j - 1]! : costAt(i, j - 1)) + d) / 2;
|
||||
if (v < best) { best = v; dir = S12; }
|
||||
}
|
||||
const u = acc(i - 2, j - 1);
|
||||
if (u < Infinity) {
|
||||
const v = u + (inRow(i - 1, j) ? pc[j]! : costAt(i - 1, j)) + d;
|
||||
if (v < best) { best = v; dir = S21; }
|
||||
}
|
||||
} else {
|
||||
const dg = acc(i - 1, j - 1);
|
||||
if (dg < best) { best = dg; dir = DIAG; }
|
||||
const up = acc(i - 1, j);
|
||||
if (up < best) { best = up; dir = UP; }
|
||||
if (j > a && cur[j - 1]! < best) { best = cur[j - 1]!; dir = LEFT; }
|
||||
if (best < Infinity) best += d;
|
||||
}
|
||||
if (best === Infinity) { cur[j] = Infinity; continue; }
|
||||
cur[j] = dir === START ? d : best;
|
||||
bp[base + j] = dir;
|
||||
}
|
||||
if (openEnd && i < n - 1 && b === m - 1) consider(i, cur[m - 1]!, m - 1);
|
||||
}
|
||||
// choose the end cell (last-column candidates were collected during the sweep)
|
||||
let ei = n - 1, ej = m - 1;
|
||||
if (openEnd) {
|
||||
for (let j = lo[n - 1]!; j <= hi[n - 1]!; j++) consider(n - 1, acc(n - 1, j), j);
|
||||
if (!(endScore < Infinity)) throw new Error("dtw: end cell unreachable within band");
|
||||
ei = endI; ej = endJ;
|
||||
} else if (acc(n - 1, m - 1) === Infinity) throw new Error("dtw: end cell unreachable within band");
|
||||
const path: [number, number][] = [];
|
||||
let i = ei, j = ej;
|
||||
for (;;) {
|
||||
path.push([i, j]);
|
||||
const d = bp[off[i]! - lo[i]! + j]!;
|
||||
if (d === START) break;
|
||||
if (d === DIAG) { i--; j--; }
|
||||
else if (d === UP) i--;
|
||||
else if (d === LEFT) j--;
|
||||
else if (d === S12) { path.push([i, j - 1]); i--; j -= 2; }
|
||||
else { path.push([i - 1, j]); i -= 2; j--; }
|
||||
}
|
||||
return path.reverse();
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
// Pure feature extraction for alignment: 12-d chroma + spectral-flux onset strength at a chosen frame rate, onset
|
||||
// picking. The spectra are computed once on a fixed 50 Hz grid and aggregated to the requested rate.
|
||||
import type { Audio } from "../core/types.ts";
|
||||
import { fft, hann } from "../dsp/fft.ts";
|
||||
import { toMono } from "../dsp/mono.ts";
|
||||
import { resample } from "../dsp/resample.ts";
|
||||
|
||||
export interface Features { rate: number; onset: Float32Array; chroma: Float32Array[] }
|
||||
|
||||
const SR = 22050;
|
||||
const N_CHROMA = 4096; // chroma window (≈186 ms)
|
||||
const N_ONSET = 1024; // onset window (≈46 ms) for sharper timing
|
||||
const BASE_HOP = 441; // 50 Hz base grid
|
||||
const BASE_RATE = SR / BASE_HOP;
|
||||
const F_MIN = 27.5, F_MAX = 5000;
|
||||
|
||||
/** Magnitude frame at `centre` (reflect-free: zero padded) written into re/im, windowed. */
|
||||
function frameMag(x: Float32Array, centre: number, n: number, w: Float64Array, re: Float64Array, im: Float64Array, mag: Float64Array): void {
|
||||
const start = centre - (n >> 1);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const j = start + i;
|
||||
re[i] = j >= 0 && j < x.length ? x[j]! * w[i]! : 0;
|
||||
im[i] = 0;
|
||||
}
|
||||
fft(re, im);
|
||||
const scale = 1 / Math.sqrt(n);
|
||||
for (let k = 0; k <= n >> 1; k++) mag[k] = Math.hypot(re[k]!, im[k]!) * scale;
|
||||
}
|
||||
|
||||
export function features(audio: Audio, rate: number): Features {
|
||||
let x = toMono(audio);
|
||||
if (audio.sampleRate !== SR) {
|
||||
const k = audio.sampleRate / SR;
|
||||
if (Number.isInteger(k)) { // fast path (e.g. 44.1k): boxcar average + decimate; the sinc resampler is ~9 s per 5 min
|
||||
const y = new Float32Array(Math.floor(x.length / k));
|
||||
for (let i = 0; i < y.length; i++) { let s = 0; for (let q = 0; q < k; q++) s += x[i * k + q]!; y[i] = s / k; }
|
||||
x = y;
|
||||
} else x = resample(x, audio.sampleRate, SR);
|
||||
}
|
||||
const nOut = Math.max(1, 1 + Math.floor((x.length / SR) * rate));
|
||||
const nBase = 1 + Math.floor(x.length / BASE_HOP);
|
||||
|
||||
// pitch-class map for chroma bins
|
||||
const binPc = new Int8Array((N_CHROMA >> 1) + 1).fill(-1);
|
||||
for (let k = 1; k < binPc.length; k++) {
|
||||
const f = (k * SR) / N_CHROMA;
|
||||
if (f < F_MIN || f > F_MAX) continue;
|
||||
const midi = 69 + 12 * Math.log2(f / 440);
|
||||
binPc[k] = (((Math.round(midi) % 12) + 12) % 12);
|
||||
}
|
||||
|
||||
const wc = hann(N_CHROMA), wo = hann(N_ONSET);
|
||||
const re = new Float64Array(N_CHROMA), im = new Float64Array(N_CHROMA), mag = new Float64Array((N_CHROMA >> 1) + 1);
|
||||
const baseChroma: Float32Array[] = [];
|
||||
for (let t = 0; t < nBase; t++) {
|
||||
frameMag(x, t * BASE_HOP, N_CHROMA, wc, re, im, mag);
|
||||
const c = new Float32Array(12);
|
||||
for (let k = 1; k < binPc.length; k++) { const pc = binPc[k]!; if (pc >= 0) c[pc]! += mag[k]! * mag[k]!; }
|
||||
baseChroma.push(c);
|
||||
}
|
||||
|
||||
// onset strength: half-wave rectified log-magnitude flux on the short window
|
||||
const baseFlux = new Float32Array(nBase);
|
||||
let prevMag = new Float64Array((N_ONSET >> 1) + 1);
|
||||
const re2 = new Float64Array(N_ONSET), im2 = new Float64Array(N_ONSET);
|
||||
let curMag = new Float64Array((N_ONSET >> 1) + 1);
|
||||
for (let t = 0; t < nBase; t++) {
|
||||
frameMag(x, t * BASE_HOP, N_ONSET, wo, re2, im2, curMag);
|
||||
let s = 0;
|
||||
for (let k = 0; k < curMag.length; k++) {
|
||||
const d = Math.log1p(100 * curMag[k]!) - Math.log1p(100 * prevMag[k]!);
|
||||
if (t > 0 && d > 0) s += d;
|
||||
}
|
||||
baseFlux[t] = s;
|
||||
[prevMag, curMag] = [curMag, prevMag];
|
||||
}
|
||||
|
||||
// aggregate base grid -> output grid: mean of chroma energy, max of flux over each output frame's span
|
||||
const chroma: Float32Array[] = [];
|
||||
const onset = new Float32Array(nOut);
|
||||
for (let o = 0; o < nOut; o++) {
|
||||
const t0 = (o - 0.5) / rate * BASE_RATE, t1 = (o + 0.5) / rate * BASE_RATE;
|
||||
let a = Math.max(0, Math.ceil(t0 - 1e-9)), b = Math.min(nBase - 1, Math.ceil(t1 - 1e-9) - 1);
|
||||
if (b < a) a = b = Math.min(nBase - 1, Math.max(0, Math.round(o / rate * BASE_RATE)));
|
||||
const c = new Float32Array(12);
|
||||
let fl = 0;
|
||||
for (let t = a; t <= b; t++) {
|
||||
const bc = baseChroma[t]!;
|
||||
for (let k = 0; k < 12; k++) c[k]! += bc[k]!;
|
||||
if (baseFlux[t]! > fl) fl = baseFlux[t]!;
|
||||
}
|
||||
let norm = 0;
|
||||
for (let k = 0; k < 12; k++) { c[k] = Math.sqrt(c[k]!); norm += c[k]! * c[k]!; }
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm > 1e-6) for (let k = 0; k < 12; k++) c[k]! /= norm; else c.fill(0);
|
||||
chroma.push(c);
|
||||
onset[o] = fl;
|
||||
}
|
||||
|
||||
// z-normalise onset strength
|
||||
let mean = 0;
|
||||
for (const v of onset) mean += v;
|
||||
mean /= nOut;
|
||||
let varr = 0;
|
||||
for (const v of onset) varr += (v - mean) ** 2;
|
||||
const sd = Math.sqrt(varr / nOut) || 1;
|
||||
for (let i = 0; i < nOut; i++) onset[i] = (onset[i]! - mean) / sd;
|
||||
return { rate, onset, chroma };
|
||||
}
|
||||
|
||||
/**
|
||||
* Peak-pick an onset-strength curve (adaptive threshold median + k·MAD, 50 ms minimum gap, the stronger peak wins).
|
||||
* Peak times are refined to sub-frame precision by parabolic interpolation over the peak and its two neighbours.
|
||||
* Times in seconds.
|
||||
*/
|
||||
export function pickOnsets(onset: Float32Array, rate: number, k = 4): number[] {
|
||||
const n = onset.length;
|
||||
if (n === 0) return [];
|
||||
const sorted = Float32Array.from(onset).sort();
|
||||
const med = sorted[n >> 1]!;
|
||||
const dev = Float32Array.from(onset, (v) => Math.abs(v - med)).sort();
|
||||
const mad = dev[n >> 1]! * 1.4826;
|
||||
const thr = med + k * Math.max(mad, 1e-3);
|
||||
const minGap = 0.05;
|
||||
const frames: number[] = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
const v = onset[i]!;
|
||||
if (v <= thr || v < (onset[i - 1] ?? -Infinity) || v <= (onset[i + 1] ?? -Infinity)) continue;
|
||||
const last = frames.length - 1;
|
||||
if (last >= 0 && (i - frames[last]!) / rate < minGap - 1e-9) {
|
||||
if (v > onset[frames[last]!]!) frames[last] = i;
|
||||
} else frames.push(i);
|
||||
}
|
||||
return frames.map((i) => {
|
||||
if (i === 0 || i === n - 1) return i / rate;
|
||||
const a = onset[i - 1]!, b = onset[i]!, c = onset[i + 1]!;
|
||||
const den = a - 2 * b + c;
|
||||
const d = den < 0 ? Math.max(-0.5, Math.min(0.5, (0.5 * (a - c)) / den)) : 0;
|
||||
return (i + d) / rate;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
// Fit quality of an alignment: how well onsets picked from the raw recording land on the warped MIDI onsets.
|
||||
// Pure: no I/O. Recording onsets must come from the audio itself (onset flux), never from transcription.
|
||||
//
|
||||
// Calibrated on real mixes (ruling A7, spec "Fit quality", amended 2026-10-30): the onset picker finds ~1.4 attacks/s
|
||||
// in a full mix while a scored MIDI has ~13 unique onsets/s, so the share of MIDI onsets that are heard ("audible
|
||||
// rate", the old match rate) is capped near 40% even when perfectly aligned and carries no information. The flag
|
||||
// metric is therefore *precision*: the share of recording onsets with a warped MIDI onset within ±70 ms. Because a
|
||||
// dense score matches almost anything, precision is chance-corrected by the precision at off-grid shifts (lift), and a
|
||||
// symbolic chroma check catches whole-beat / whole-bar shifts that onset metrics cannot see (they alias onto the grid).
|
||||
|
||||
/** A beat/bar shift that explains the recording's chroma better than the current alignment. */
|
||||
export interface BeatShift { label: string; gain: number }
|
||||
export interface FitWindow {
|
||||
startSec: number;
|
||||
/** Recording onsets in this window (by recording time). */
|
||||
recOnsets: number;
|
||||
/** Share of this window's recording onsets with a warped MIDI onset within ±70 ms; null with < 8 recording onsets. */
|
||||
precision: number | null;
|
||||
/** precision − mean precision at the off-grid shifts (chance-corrected); null when precision is null. */
|
||||
lift: number | null;
|
||||
/** Information only (JSON `audibleRate`): share of unique MIDI onsets (in this window by warped time) heard as attacks. */
|
||||
audibleRate: number | null;
|
||||
/** Median |recording onset − nearest warped MIDI onset| over matched recording onsets (ms). */
|
||||
medianResidualMs: number | null;
|
||||
beatShift: BeatShift | null;
|
||||
flagged: boolean;
|
||||
/** Why the window is flagged, e.g. "precision 41%", "lift 0.12", "beat shift suspected: +1 beat". */
|
||||
reasons: string[];
|
||||
}
|
||||
export interface StretchCounts { total: number; warned: number; over15: number }
|
||||
export interface FitReport {
|
||||
precision: number | null;
|
||||
lift: number | null;
|
||||
/** Information only: share of unique MIDI onsets heard as attacks ("MIDI onsets audible as attacks"). */
|
||||
audibleRate: number | null;
|
||||
medianResidualMs: number | null;
|
||||
windows: FitWindow[];
|
||||
/** Stretches of the alignment: total, warned (> 5% tempo change) and with |tempo change| > 15%. */
|
||||
stretches?: StretchCounts;
|
||||
}
|
||||
|
||||
/** Symbolic-chroma inputs for the beat/bar-shift check: frames at `rate` Hz in recording time, 12-d (any scale). */
|
||||
export interface ShiftInputs { rate: number; recording: Float32Array[]; current: Float32Array[]; shifts: { label: string; chroma: Float32Array[] }[] }
|
||||
|
||||
export const MATCH_TOL_SEC = 0.07;
|
||||
export const WINDOW_SEC = 30;
|
||||
export const FLAG_MIN_PRECISION = 0.6;
|
||||
export const FLAG_MIN_LIFT = 0.25;
|
||||
export const MIN_REC_ONSETS = 8;
|
||||
/** Off-grid shifts (s) of the warped MIDI used as the chance level of precision. */
|
||||
export const CHANCE_SHIFTS_SEC = [0.11, 0.23, 0.37];
|
||||
/** A shifted alignment must beat the current one by this much mean chroma cosine to flag the window. */
|
||||
export const SHIFT_MARGIN = 0.05;
|
||||
/** The current symbolic chroma must cover at least this share of the window's sounding frames for the check to run. */
|
||||
const SHIFT_MIN_COVER = 0.25;
|
||||
const UNIQUE_EPS = 1e-6;
|
||||
|
||||
function median(xs: number[]): number | null {
|
||||
if (!xs.length) return null;
|
||||
const s = [...xs].sort((a, b) => a - b), m = s.length >> 1;
|
||||
return s.length % 2 ? s[m]! : (s[m - 1]! + s[m]!) / 2;
|
||||
}
|
||||
|
||||
/** |distance| from `t` to the nearest value in `sorted` (Infinity when empty). */
|
||||
function nearestDist(sorted: number[], t: number): number {
|
||||
let lo = 0, hi = sorted.length;
|
||||
while (lo < hi) { const m = (lo + hi) >> 1; if (sorted[m]! < t) lo = m + 1; else hi = m; }
|
||||
let best = Infinity;
|
||||
if (lo < sorted.length) best = Math.min(best, Math.abs(sorted[lo]! - t));
|
||||
if (lo > 0) best = Math.min(best, Math.abs(sorted[lo - 1]! - t));
|
||||
return best;
|
||||
}
|
||||
const hit = (d: number) => d <= MATCH_TOL_SEC + 1e-9;
|
||||
|
||||
/** Sorted, de-duplicated copy. */
|
||||
export function uniqueSorted(xs: number[]): number[] {
|
||||
const out: number[] = [];
|
||||
for (const t of [...xs].sort((a, b) => a - b)) if (!out.length || t - out[out.length - 1]! > UNIQUE_EPS) out.push(t);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Share of `rec` (any order) with a value of sorted `midi` + `shift` within ±70 ms; null when `rec` or `midi` is empty. */
|
||||
export function precisionOf(rec: number[], midi: number[], shift = 0): number | null {
|
||||
if (!rec.length || !midi.length) return null;
|
||||
let n = 0;
|
||||
for (const t of rec) if (hit(nearestDist(midi, t - shift))) n++;
|
||||
return n / rec.length;
|
||||
}
|
||||
|
||||
function cos(a: Float32Array, b: Float32Array): number {
|
||||
let d = 0, na = 0, nb = 0;
|
||||
for (let q = 0; q < 12; q++) { d += a[q]! * b[q]!; na += a[q]! * a[q]!; nb += b[q]! * b[q]!; }
|
||||
return na > 1e-12 && nb > 1e-12 ? d / Math.sqrt(na * nb) : 0;
|
||||
}
|
||||
|
||||
/** Mean chroma cosine over the sounding recording frames of [f0, f1); null when the candidate covers too little. */
|
||||
export function meanCosine(rec: Float32Array[], cand: Float32Array[], f0: number, f1: number, minCover = SHIFT_MIN_COVER): number | null {
|
||||
let sum = 0, sounding = 0, covered = 0;
|
||||
for (let t = f0; t < Math.min(f1, rec.length); t++) {
|
||||
const r = rec[t]!;
|
||||
if (!r.some((v) => v !== 0)) continue;
|
||||
sounding++;
|
||||
const c = cand[t];
|
||||
if (!c || !c.some((v) => v !== 0)) continue;
|
||||
covered++;
|
||||
sum += cos(r, c);
|
||||
}
|
||||
if (!sounding || covered / sounding < minCover) return null;
|
||||
return sum / sounding;
|
||||
}
|
||||
|
||||
function shiftCheck(s: ShiftInputs, f0: number, f1: number): BeatShift | null {
|
||||
const cur = meanCosine(s.recording, s.current, f0, f1);
|
||||
if (cur === null) return null;
|
||||
let best: BeatShift | null = null;
|
||||
for (const c of s.shifts) {
|
||||
const v = meanCosine(s.recording, c.chroma, f0, f1, 0);
|
||||
if (v !== null && v - cur > SHIFT_MARGIN && (!best || v - cur > best.gain)) best = { label: c.label, gain: v - cur };
|
||||
}
|
||||
return best;
|
||||
}
|
||||
|
||||
const pct = (x: number) => `${Math.round(x * 100)}%`;
|
||||
|
||||
/**
|
||||
* Recording onsets are matched to the nearest warped MIDI onset within ±70 ms (non-exclusive). Overall and per 30 s
|
||||
* window (by recording onset time): precision, lift (precision − mean precision with the MIDI shifted by +0.11, +0.23,
|
||||
* +0.37 s), median residual; plus the audible rate (info) and, with `extra.shift`, the beat/bar-shift chroma check.
|
||||
* A window with < 8 recording onsets is null and unflagged; otherwise it is flagged when precision < 0.6, lift < 0.25,
|
||||
* or a shifted alignment beats the current one by > 0.05 mean chroma cosine.
|
||||
*/
|
||||
export function fitQuality(audioOnsets: number[], midiOnsets: number[], durSec: number, extra: { shift?: ShiftInputs; stretches?: StretchCounts } = {}): FitReport {
|
||||
const rec = [...audioOnsets].sort((a, b) => a - b);
|
||||
const midi = uniqueSorted(midiOnsets);
|
||||
const nWin = Math.max(1, Math.ceil(Math.max(durSec, (rec[rec.length - 1] ?? 0) + 1e-9) / WINDOW_SEC));
|
||||
const winOf = (t: number) => Math.min(nWin - 1, Math.max(0, Math.floor(t / WINDOW_SEC)));
|
||||
const recWin: number[][] = Array.from({ length: nWin }, () => []);
|
||||
for (const t of rec) recWin[winOf(t)]!.push(t);
|
||||
const midiHit = midi.map((t) => hit(nearestDist(rec, t)));
|
||||
|
||||
const stats = (r: number[]) => {
|
||||
const precision = midi.length ? precisionOf(r, midi) : null;
|
||||
if (precision === null) return { precision: null, lift: null, medianResidualMs: null };
|
||||
const chance = CHANCE_SHIFTS_SEC.map((s) => precisionOf(r, midi, s)!);
|
||||
const dists = r.map((t) => nearestDist(midi, t)).filter(hit);
|
||||
const med = median(dists);
|
||||
return { precision, lift: precision - chance.reduce((a, b) => a + b, 0) / chance.length, medianResidualMs: med === null ? null : med * 1000 };
|
||||
};
|
||||
const audible = (lo: number, hi: number) => {
|
||||
let n = 0, h = 0;
|
||||
midi.forEach((t, k) => { if (t >= lo && t < hi) { n++; if (midiHit[k]) h++; } });
|
||||
return n ? h / n : null;
|
||||
};
|
||||
|
||||
const windows = recWin.map((r, w): FitWindow => {
|
||||
const startSec = w * WINDOW_SEC;
|
||||
const lo = w === 0 ? -Infinity : startSec, hi = w === nWin - 1 ? Infinity : startSec + WINDOW_SEC;
|
||||
const s = r.length >= MIN_REC_ONSETS ? stats(r) : { precision: null, lift: null, medianResidualMs: null };
|
||||
const beatShift = extra.shift && s.precision !== null
|
||||
? shiftCheck(extra.shift, Math.round(startSec * extra.shift.rate), w === nWin - 1 ? Infinity : Math.round((startSec + WINDOW_SEC) * extra.shift.rate))
|
||||
: null;
|
||||
const reasons: string[] = [];
|
||||
if (s.precision !== null && s.precision < FLAG_MIN_PRECISION) reasons.push(`precision ${pct(s.precision)}`);
|
||||
if (s.lift !== null && s.lift < FLAG_MIN_LIFT) reasons.push(`lift ${s.lift.toFixed(2)}`);
|
||||
if (beatShift) reasons.push(`beat shift suspected: ${beatShift.label}`);
|
||||
return { startSec, recOnsets: r.length, ...s, audibleRate: audible(lo, hi), beatShift, flagged: reasons.length > 0, reasons };
|
||||
});
|
||||
const all = rec.length && midi.length ? stats(rec) : { precision: null, lift: null, medianResidualMs: null };
|
||||
return { ...all, audibleRate: audible(-Infinity, Infinity), windows, ...(extra.stretches ? { stretches: extra.stretches } : {}) };
|
||||
}
|
||||
|
||||
/**
|
||||
* One line, e.g. "precision 91.2% (lift +0.70) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms ·
|
||||
* 1/8 windows flagged (0:30 beat shift suspected: +1 beat) · stretches: 3/40 warned, 1 > 15%".
|
||||
*/
|
||||
export function fitSummary(f: FitReport): string {
|
||||
if (f.precision === null) return f.audibleRate === null ? "no scored MIDI onsets" : "no recording onsets";
|
||||
const mmss = (s: number) => `${Math.floor(s / 60)}:${String(Math.floor(s % 60)).padStart(2, "0")}`;
|
||||
const flagged = f.windows.filter((w) => w.flagged);
|
||||
const med = f.medianResidualMs === null ? "n/a" : `${f.medianResidualMs.toFixed(1)} ms`;
|
||||
const lift = f.lift === null ? "" : ` (lift ${f.lift >= 0 ? "+" : "−"}${Math.abs(f.lift).toFixed(2)})`;
|
||||
const aud = f.audibleRate === null ? "n/a" : `${(f.audibleRate * 100).toFixed(1)}%`;
|
||||
const list = flagged.length ? ` (${flagged.map((w) => `${mmss(w.startSec)} ${w.reasons.join(", ")}`).join("; ")})` : "";
|
||||
const st = f.stretches ? ` · stretches: ${f.stretches.warned}/${f.stretches.total} warned, ${f.stretches.over15} > 15%` : "";
|
||||
return `precision ${(f.precision * 100).toFixed(1)}%${lift} · MIDI onsets audible as attacks ${aud} · median residual ${med} · ${flagged.length}/${f.windows.length} windows flagged${list}${st}`;
|
||||
}
|
||||
@@ -0,0 +1,647 @@
|
||||
/// <reference lib="dom" />
|
||||
/// <reference lib="dom.iterable" />
|
||||
// Alignment page (layout A): two lanes (recording, warped MIDI audio) on one zoomable timeline, anchors spanning both,
|
||||
// stretch tempo labels, fit-flagged windows, Web Audio playback with a mix choice. Geometry/edit logic is in the pure
|
||||
// modules ./timeline.ts and ./edit.ts; warp maths come from ../warp.ts so every edit redraws instantly.
|
||||
import { AUTO_CLEAR_SEC, deleteAnchor, makeWarp, stretches, type Anchor, type Warp } from "../warp.ts";
|
||||
import { stretchInfo, stretchLabel, type StretchInfo } from "../tempo.ts";
|
||||
import { fitSummary, WINDOW_SEC, type FitReport } from "../fit.ts";
|
||||
import type { TempoPoint } from "../smf.ts";
|
||||
import {
|
||||
buildPyramid, clampView, columnPeaks, fitView, follow, formatTime, pyramidFromPeaks, rulerStep, scrollBy, timeToX, xToTime, zoomAt,
|
||||
type PeakLevel, type View,
|
||||
} from "./timeline.ts";
|
||||
import { anchorAtCursor, dragAudio, dragMidi, hitAnchor, type Edit } from "./edit.ts";
|
||||
|
||||
// ---------- layout (CSS px) ----------
|
||||
const GUTTER = 92, RULER_H = 22, LABEL_H = 18, LANE_H = 118, LANE_GAP = 8, STRETCH_H = 26;
|
||||
const REC_Y = RULER_H + LABEL_H, MIDI_Y = REC_Y + LANE_H + LANE_GAP, STRETCH_Y = MIDI_Y + LANE_H, HEIGHT = STRETCH_Y + STRETCH_H;
|
||||
const C = { rec: "#5b8def", midi: "#e2873b", anchor: "#2bb673", anchorAuto: "#86cfa9", warn: "#d9534f", ink: "#222", muted: "#888", lane: "#f4f5f7", flag: "rgba(217,83,79,.13)", play: "#7a3cff" };
|
||||
const HIT_PX = 7, DRAG_PX = 3, RENDER_DEBOUNCE_MS = 400;
|
||||
|
||||
// ---------- DOM ----------
|
||||
const $ = <T extends HTMLElement>(id: string) => document.getElementById(id) as T;
|
||||
const canvas = $<HTMLCanvasElement>("tl"), g = canvas.getContext("2d")!;
|
||||
const wrap = $("wrap"), overlay = $("overlay"), statusEl = $("status"), fitEl = $("fit"), countsEl = $("counts");
|
||||
const songSel = $<HTMLSelectElement>("song"), songInfo = $("songinfo"), saveState = $("savestate");
|
||||
const btn = {
|
||||
play: $<HTMLButtonElement>("play"), add: $<HTMLButtonElement>("add"), del: $<HTMLButtonElement>("del"), undo: $<HTMLButtonElement>("undo"),
|
||||
auto: $<HTMLButtonElement>("auto"), save: $<HTMLButtonElement>("save"), reset: $<HTMLButtonElement>("reset"),
|
||||
};
|
||||
const mixButtons = [...document.querySelectorAll<HTMLButtonElement>(".mix button")];
|
||||
|
||||
// ---------- state ----------
|
||||
interface SongInfo {
|
||||
id: string; title: string; audioDur: number; midiDur: number; tracks: { name: string; role: string | null }[];
|
||||
recordingPeaks: { bucketSec: number; min: number[]; max: number[] }; anchors: Anchor[]; fit: FitReport;
|
||||
midiTempo: TempoPoint[]; source: "saved" | "auto"; savedAt: string | null; warning?: string;
|
||||
}
|
||||
interface Wave { levels: PeakLevel[]; rate: number; norm: number }
|
||||
type Mix = "both" | "rec" | "midi" | "split";
|
||||
|
||||
let song: SongInfo | null = null;
|
||||
let anchors: Anchor[] = [];
|
||||
let warp: Warp = makeWarp([]);
|
||||
let info: StretchInfo[] = [];
|
||||
let fit: FitReport | null = null, fitKey = "";
|
||||
let savedKey: string | null = null, savedAt: string | null = null;
|
||||
let view: View = { start: 0, pxPerSec: 10, width: 800 };
|
||||
let cursor = 0, selected = -1, hover = -1, hoverLane: "rec" | "midi" | null = null;
|
||||
let recWave: Wave | null = null, midiWave: Wave | null = null, midiKey = "";
|
||||
let recBuf: AudioBuffer | null = null, midiBuf: AudioBuffer | null = null;
|
||||
let busy = false, rendering = false;
|
||||
const undoStack: Anchor[][] = [], redoStack: Anchor[][] = [];
|
||||
let mix: Mix = "both";
|
||||
let loadGen = 0;
|
||||
|
||||
const keyOf = (A: Anchor[]) => JSON.stringify(A.map((a) => [a.midiSec, a.audioSec, a.kind]));
|
||||
const dur = () => song?.audioDur ?? 1;
|
||||
const isDirty = () => song !== null && keyOf(anchors) !== savedKey;
|
||||
|
||||
// ---------- HTTP ----------
|
||||
async function http<T = unknown>(path: string, init: RequestInit = {}): Promise<T> {
|
||||
const r = await fetch(path, init);
|
||||
if (r.status === 401) throw new Error("Not authorised (server restarted?): open the URL printed by the server again.");
|
||||
if (!r.ok) {
|
||||
let msg = `${r.status} ${r.statusText}`;
|
||||
try { msg = ((await r.json()) as { error?: string }).error ?? msg; } catch { /* not JSON */ }
|
||||
throw new Error(msg);
|
||||
}
|
||||
return (r.headers.get("content-type") ?? "").includes("json") ? ((await r.json()) as T) : ((await r.arrayBuffer()) as T);
|
||||
}
|
||||
const send = (method: string, body: unknown, signal?: AbortSignal): RequestInit => ({ method, body: JSON.stringify(body), headers: { "content-type": "application/json" }, signal });
|
||||
const songUrl = (rest = "") => `/api/song/${encodeURIComponent(song!.id)}${rest}`;
|
||||
|
||||
function setStatus(msg: string, error = false) { statusEl.textContent = msg; statusEl.classList.toggle("error", error); }
|
||||
function showOverlay(msg: string | null) { overlay.hidden = msg === null; overlay.textContent = msg ?? ""; }
|
||||
|
||||
// ---------- audio ----------
|
||||
const actx = new AudioContext();
|
||||
const recGain = actx.createGain(), midiGain = actx.createGain();
|
||||
const recPan = actx.createStereoPanner(), midiPan = actx.createStereoPanner();
|
||||
recGain.connect(recPan).connect(actx.destination);
|
||||
midiGain.connect(midiPan).connect(actx.destination);
|
||||
let recSrc: AudioBufferSourceNode | null = null, midiSrc: AudioBufferSourceNode | null = null;
|
||||
let playing = false, playCtx0 = 0, playPos0 = 0;
|
||||
|
||||
function applyMix() {
|
||||
const set = (p: AudioParam, v: number) => p.setTargetAtTime(v, actx.currentTime, 0.015);
|
||||
set(recGain.gain, mix === "midi" ? 0 : 1);
|
||||
set(midiGain.gain, mix === "rec" ? 0 : mix === "both" ? 0.8 : 1);
|
||||
set(recPan.pan, mix === "split" ? -1 : 0);
|
||||
set(midiPan.pan, mix === "split" ? 1 : 0);
|
||||
mixButtons.forEach((b) => { b.classList.toggle("on", b.dataset.mix === mix); b.setAttribute("aria-checked", String(b.dataset.mix === mix)); });
|
||||
}
|
||||
|
||||
function startSource(buf: AudioBuffer, dest: AudioNode, when: number, offset: number): AudioBufferSourceNode | null {
|
||||
if (offset >= buf.duration) return null;
|
||||
const s = actx.createBufferSource();
|
||||
s.buffer = buf;
|
||||
s.connect(dest);
|
||||
s.start(when, Math.max(0, offset));
|
||||
return s;
|
||||
}
|
||||
|
||||
const playPos = () => playPos0 + Math.max(0, actx.currentTime - playCtx0);
|
||||
|
||||
async function play() {
|
||||
if (!recBuf && !midiBuf) { setStatus("Audio is still loading…"); return; }
|
||||
await actx.resume();
|
||||
stop();
|
||||
const t0 = actx.currentTime + 0.06; // both sources start on the same context tick
|
||||
playCtx0 = t0; playPos0 = cursor;
|
||||
recSrc = recBuf ? startSource(recBuf, recGain, t0, cursor) : null;
|
||||
midiSrc = midiBuf ? startSource(midiBuf, midiGain, t0, cursor) : null;
|
||||
playing = true;
|
||||
if (!recBuf) setStatus("Recording still loading: playing MIDI only.");
|
||||
else if (!midiBuf) setStatus("MIDI still rendering: playing the recording only.");
|
||||
btn.play.textContent = "■ Stop";
|
||||
requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
function stop() {
|
||||
for (const s of [recSrc, midiSrc]) { try { s?.stop(); } catch { /* already stopped */ } s?.disconnect(); }
|
||||
recSrc = midiSrc = null;
|
||||
if (playing) { playing = false; btn.play.textContent = "▶ Play from cursor"; draw(); }
|
||||
}
|
||||
|
||||
/** Replace the MIDI buffer; if playing, the new one takes over seamlessly at the current position. */
|
||||
function swapMidi(buf: AudioBuffer) {
|
||||
midiBuf = buf;
|
||||
if (!playing) return;
|
||||
const t = actx.currentTime + 0.03;
|
||||
const next = startSource(buf, midiGain, t, playPos0 + (t - playCtx0));
|
||||
try { midiSrc?.stop(t); } catch { /* ended */ }
|
||||
midiSrc = next;
|
||||
}
|
||||
|
||||
function tick() {
|
||||
if (!playing) return;
|
||||
const p = playPos();
|
||||
if (p >= Math.max(recBuf?.duration ?? 0, midiBuf?.duration ?? 0)) { stop(); return; }
|
||||
view = follow(view, p, dur());
|
||||
draw();
|
||||
requestAnimationFrame(tick);
|
||||
}
|
||||
|
||||
function waveFrom(buf: AudioBuffer): Wave {
|
||||
const n = buf.length, mono = new Float32Array(n);
|
||||
for (let c = 0; c < buf.numberOfChannels; c++) { const d = buf.getChannelData(c); for (let i = 0; i < n; i++) mono[i]! += d[i]! / buf.numberOfChannels; }
|
||||
const levels = buildPyramid(mono, 64);
|
||||
return { levels, rate: buf.sampleRate, norm: peakNorm(levels) };
|
||||
}
|
||||
function peakNorm(levels: PeakLevel[]): number {
|
||||
const top = levels[levels.length - 1]!;
|
||||
let m = 0;
|
||||
for (let i = 0; i < top.min.length; i++) m = Math.max(m, -top.min[i]!, top.max[i]!);
|
||||
return m > 1e-4 ? 1 / m : 1;
|
||||
}
|
||||
|
||||
// ---------- edits ----------
|
||||
function setAnchors(next: Anchor[]) {
|
||||
anchors = next;
|
||||
warp = makeWarp(anchors);
|
||||
info = song ? stretchInfo(stretches(anchors, song.midiDur), song.midiTempo) : [];
|
||||
if (selected >= anchors.length) selected = -1;
|
||||
updateUi();
|
||||
draw();
|
||||
}
|
||||
|
||||
/** A finished edit: undoable, marks unsaved, and (debounced) re-renders the MIDI and re-scores the fit. */
|
||||
function commit(next: Anchor[], before: Anchor[], refreshNow = false) {
|
||||
if (keyOf(next) === keyOf(before)) { setAnchors(next); return; }
|
||||
undoStack.push(before);
|
||||
if (undoStack.length > 200) undoStack.shift();
|
||||
redoStack.length = 0;
|
||||
setAnchors(next);
|
||||
scheduleRefresh(refreshNow);
|
||||
}
|
||||
|
||||
function tryEdit(f: () => Edit, before = anchors): boolean {
|
||||
try {
|
||||
const r = f();
|
||||
selected = r.index;
|
||||
commit(r.anchors, before);
|
||||
return true;
|
||||
} catch (e) { setStatus((e as Error).message, true); return false; }
|
||||
}
|
||||
|
||||
let refreshTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
let renderAbort: AbortController | null = null;
|
||||
let fitGen = 0;
|
||||
function scheduleRefresh(now = false) {
|
||||
clearTimeout(refreshTimer);
|
||||
fitEl.classList.add("stale");
|
||||
refreshTimer = setTimeout(refresh, now ? 0 : RENDER_DEBOUNCE_MS);
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
if (!song) return;
|
||||
const A = anchors, key = keyOf(A), id = song.id, gen = ++fitGen;
|
||||
http<{ fit: FitReport }>(songUrl("/fit"), send("POST", { anchors: A }))
|
||||
.then((j) => { if (gen === fitGen && song?.id === id) { fit = j.fit; fitKey = key; updateUi(); draw(); } })
|
||||
.catch((e) => setStatus(`Fit update failed: ${(e as Error).message}`, true));
|
||||
if (key === midiKey) return;
|
||||
renderAbort?.abort();
|
||||
const ac = (renderAbort = new AbortController());
|
||||
rendering = true; draw();
|
||||
try {
|
||||
const wav = await http<ArrayBuffer>(songUrl("/render"), send("POST", { anchors: A }, ac.signal));
|
||||
const buf = await actx.decodeAudioData(wav);
|
||||
if (ac.signal.aborted || song?.id !== id) return;
|
||||
midiWave = waveFrom(buf); midiKey = key;
|
||||
swapMidi(buf);
|
||||
} catch (e) {
|
||||
if (!ac.signal.aborted) setStatus(`MIDI render failed: ${(e as Error).message}`, true);
|
||||
} finally {
|
||||
if (renderAbort === ac) { rendering = false; renderAbort = null; draw(); }
|
||||
}
|
||||
}
|
||||
|
||||
function undo(redo = false) {
|
||||
const from = redo ? redoStack : undoStack, to = redo ? undoStack : redoStack;
|
||||
const prev = from.pop();
|
||||
if (!prev) return;
|
||||
to.push(anchors);
|
||||
selected = -1;
|
||||
setAnchors(prev);
|
||||
scheduleRefresh();
|
||||
}
|
||||
|
||||
function addAtCursor() {
|
||||
if (!song) return;
|
||||
const autosBefore = anchors.filter((a) => a.kind === "auto").length;
|
||||
if (!tryEdit(() => anchorAtCursor(anchors, cursor))) return;
|
||||
const removed = autosBefore - anchors.filter((a) => a.kind === "auto").length;
|
||||
setStatus(`Anchor added at ${formatTime(cursor, 2)}. Drag its MIDI end (lower lane) until the MIDI lines up with the recording.`
|
||||
+ (removed > 0 ? ` ${removed} automatic anchors within ${AUTO_CLEAR_SEC} s of it were removed; "Auto-align around my anchors" fills them in again.` : ""));
|
||||
}
|
||||
|
||||
function deleteSelected() {
|
||||
if (selected < 0) return;
|
||||
const k = selected;
|
||||
selected = -1;
|
||||
commit(deleteAnchor(anchors, k), anchors);
|
||||
}
|
||||
|
||||
function nudge(delta: number) {
|
||||
if (selected < 0) { cursor = Math.min(dur(), Math.max(0, cursor + delta * 100)); view = follow(view, cursor, dur()); draw(); return; }
|
||||
tryEdit(() => dragAudio(anchors, selected, anchors[selected]!.audioSec + delta));
|
||||
}
|
||||
|
||||
async function runBusy(label: string, f: () => Promise<void>) {
|
||||
if (busy || !song) return;
|
||||
busy = true; updateUi(); showOverlay(label); setStatus(label);
|
||||
try { await f(); } catch (e) { setStatus((e as Error).message, true); } finally { busy = false; showOverlay(null); updateUi(); }
|
||||
}
|
||||
|
||||
interface AutoInfo { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null }
|
||||
const pctOf = (x: number | null) => (x === null ? "n/a" : `${(x * 100).toFixed(1)}%`);
|
||||
/** "DTW, precision 91.0% vs constant offset +0.06 s 88.0%" or "constant offset +0.06 s (precision 91.0%; DTW 89.5% was not better)". */
|
||||
function autoNote(a: AutoInfo): string {
|
||||
const off = `${a.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(a.baseline.offsetSec).toFixed(2)} s`;
|
||||
return a.chosen === "dtw" ? `DTW, precision ${pctOf(a.dtwPrecision)} vs constant offset ${off} ${pctOf(a.baseline.precision)}`
|
||||
: `constant offset ${off} (precision ${pctOf(a.baseline.precision)}; DTW ${pctOf(a.dtwPrecision)} was not better by 2 points)`;
|
||||
}
|
||||
|
||||
const autoAlign = () => runBusy("Auto-aligning around your anchors… (a few seconds to half a minute)", async () => {
|
||||
const manual = anchors.filter((a) => a.kind === "manual");
|
||||
const j = await http<{ anchors: Anchor[]; fit: FitReport; auto: AutoInfo }>(songUrl("/auto"), send("POST", { anchors: manual }));
|
||||
selected = -1;
|
||||
commit(j.anchors, anchors, true);
|
||||
fit = j.fit; fitKey = keyOf(j.anchors);
|
||||
updateUi();
|
||||
setStatus(`Auto-aligned: ${j.anchors.length - manual.length} automatic anchors around your ${manual.length} (${autoNote(j.auto)}). Not saved yet.`);
|
||||
});
|
||||
|
||||
const save = () => runBusy("Saving…", async () => {
|
||||
const A = anchors;
|
||||
const j = await http<{ savedAt: string; fit: FitReport }>(songUrl("/alignment"), send("PUT", { anchors: A }));
|
||||
savedKey = keyOf(A); savedAt = j.savedAt; fit = j.fit; fitKey = savedKey;
|
||||
const opt = [...songSel.options].find((o) => o.value === song!.id);
|
||||
if (opt) opt.textContent = `✓ ${song!.title}`;
|
||||
setStatus(`Saved ${A.length} anchors.`);
|
||||
});
|
||||
|
||||
function reset() {
|
||||
if (!song || !anchors.length) return;
|
||||
if (!confirm("Remove all anchors and show the raw MIDI timing?\nNothing is written until you press Save (Undo also works).")) return;
|
||||
selected = -1;
|
||||
commit([], anchors, true);
|
||||
setStatus("Reset to raw MIDI timing (not saved).");
|
||||
}
|
||||
|
||||
// ---------- song loading ----------
|
||||
async function loadSongs() {
|
||||
const list = await http<{ id: string; title: string; hasAlignment: boolean }[]>("/api/songs");
|
||||
songSel.innerHTML = "";
|
||||
for (const s of list) songSel.add(new Option(`${s.hasAlignment ? "✓ " : ""}${s.title}`, s.id));
|
||||
if (!list.length) {
|
||||
showOverlay("No songs yet. Add one on the server with:\nbun run align add <recording> <midi> --id <id> --title <title>\nthen reload this page.");
|
||||
return;
|
||||
}
|
||||
const want = decodeURIComponent(location.hash.slice(1));
|
||||
await openSong(list.some((s) => s.id === want) ? want : list[0]!.id);
|
||||
}
|
||||
|
||||
async function openSong(id: string) {
|
||||
if (song && isDirty() && !confirm(`"${song.title}" has unsaved changes. Discard them?`)) { songSel.value = song.id; return; }
|
||||
const gen = ++loadGen;
|
||||
stop();
|
||||
renderAbort?.abort();
|
||||
clearTimeout(refreshTimer);
|
||||
songSel.value = id;
|
||||
history.replaceState(null, "", `#${encodeURIComponent(id)}`);
|
||||
song = null; recBuf = midiBuf = null; recWave = midiWave = null; midiKey = ""; fit = null; undoStack.length = redoStack.length = 0;
|
||||
selected = hover = -1;
|
||||
setAnchors([]);
|
||||
showOverlay("Loading song… The first time a song is opened the server converts the recording and runs the automatic alignment (up to ~30 s).");
|
||||
setStatus("");
|
||||
try {
|
||||
const s = await http<SongInfo>(`/api/song/${encodeURIComponent(id)}`);
|
||||
if (gen !== loadGen) return;
|
||||
song = s;
|
||||
savedKey = s.source === "saved" ? keyOf(makeWarp(s.anchors).anchors) : null;
|
||||
savedAt = s.savedAt;
|
||||
fit = s.fit;
|
||||
cursor = 0;
|
||||
view = fitView(s.audioDur, view.width);
|
||||
recWave = { levels: pyramidFromPeaks(s.recordingPeaks.min, s.recordingPeaks.max), rate: 1 / s.recordingPeaks.bucketSec, norm: 1 };
|
||||
recWave.norm = peakNorm(recWave.levels);
|
||||
setAnchors(makeWarp(s.anchors).anchors);
|
||||
fitKey = keyOf(anchors);
|
||||
const roles = s.tracks.map((t) => `${t.name}: ${t.role ?? "unmapped"}`).join("\n");
|
||||
const bpms = [...new Set(s.midiTempo.map((t) => t.bpm.toFixed(1)))];
|
||||
songInfo.textContent = `${formatTime(s.audioDur, 0)} · MIDI ${bpms.length > 1 ? `${bpms.length} tempos` : `${bpms[0]} bpm`} · ${s.tracks.length} tracks`;
|
||||
songInfo.title = roles;
|
||||
showOverlay(null);
|
||||
if (s.warning) setStatus(s.warning, true);
|
||||
else setStatus(s.source === "auto" ? "This is the automatic alignment (not saved yet). Play a few spots, fix anchors where it drifts, then Save." : `Loaded the saved alignment (${anchors.length} anchors).`);
|
||||
scheduleRefresh(true); // render the MIDI for the current anchors
|
||||
http<ArrayBuffer>(songUrl("/audio"))
|
||||
.then((b) => actx.decodeAudioData(b))
|
||||
.then((buf) => { if (gen !== loadGen) return; recBuf = buf; recWave = waveFrom(buf); draw(); })
|
||||
.catch((e) => setStatus(`Could not load the recording audio: ${(e as Error).message}`, true));
|
||||
} catch (e) {
|
||||
if (gen !== loadGen) return;
|
||||
showOverlay(`Could not open the song: ${(e as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- UI state ----------
|
||||
function updateUi() {
|
||||
const has = song !== null;
|
||||
btn.add.disabled = !has || busy;
|
||||
btn.del.disabled = !has || busy || selected < 0;
|
||||
btn.undo.disabled = busy || !undoStack.length;
|
||||
btn.auto.disabled = btn.save.disabled = btn.reset.disabled = !has || busy;
|
||||
btn.play.disabled = !has;
|
||||
if (!has) { saveState.textContent = ""; fitEl.textContent = "Fit: –"; countsEl.textContent = ""; return; }
|
||||
const dirty = isDirty();
|
||||
saveState.className = `savestate ${dirty ? "dirty" : "clean"}`;
|
||||
saveState.textContent = dirty ? (savedKey === null && !undoStack.length ? "● Not saved yet (automatic alignment)" : "● Unsaved changes")
|
||||
: `✓ Saved${savedAt ? ` ${new Date(savedAt).toLocaleString()}` : ""}`;
|
||||
btn.save.textContent = dirty ? "Save alignment" : "Saved";
|
||||
const stale = fitKey !== keyOf(anchors);
|
||||
fitEl.classList.toggle("stale", stale);
|
||||
if (fit) {
|
||||
fitEl.textContent = `Fit${stale ? " (updating…)" : ""}: ${fitSummary(fit)}`;
|
||||
fitEl.classList.toggle("bad", fit.windows.some((w) => w.flagged));
|
||||
}
|
||||
const man = anchors.filter((a) => a.kind === "manual").length;
|
||||
countsEl.textContent = `· anchors: ${man} yours, ${anchors.length - man} automatic`;
|
||||
}
|
||||
|
||||
// ---------- drawing ----------
|
||||
let cssW = 800;
|
||||
function resize() {
|
||||
cssW = wrap.clientWidth;
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
canvas.width = Math.round(cssW * dpr); canvas.height = Math.round(HEIGHT * dpr);
|
||||
canvas.style.height = `${HEIGHT}px`;
|
||||
g.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||
view = { ...view, width: Math.max(50, cssW - GUTTER) };
|
||||
if (song) view = clampView(view, dur());
|
||||
draw();
|
||||
}
|
||||
|
||||
let drawQueued = false;
|
||||
function draw() {
|
||||
if (drawQueued) return;
|
||||
drawQueued = true;
|
||||
requestAnimationFrame(() => { drawQueued = false; paint(); });
|
||||
}
|
||||
|
||||
const X = (t: number) => GUTTER + timeToX(view, t);
|
||||
|
||||
function paintWave(w: Wave | null, y: number, color: string, alpha: number) {
|
||||
g.fillStyle = C.lane; g.fillRect(GUTTER, y, view.width, LANE_H);
|
||||
if (!w) return;
|
||||
const { min, max } = columnPeaks(w.levels, w.rate, view);
|
||||
const mid = y + LANE_H / 2, half = (LANE_H / 2 - 4) * w.norm;
|
||||
g.globalAlpha = alpha; g.fillStyle = color;
|
||||
for (let c = 0; c < min.length; c++) {
|
||||
if (min[c] === 0 && max[c] === 0) continue;
|
||||
const top = mid - max[c]! * half, bot = mid - min[c]! * half;
|
||||
g.fillRect(GUTTER + c, top, 1, Math.max(1, bot - top));
|
||||
}
|
||||
g.globalAlpha = 1;
|
||||
}
|
||||
|
||||
function paint() {
|
||||
const W = cssW;
|
||||
g.clearRect(0, 0, W, HEIGHT);
|
||||
g.font = "11px system-ui, sans-serif"; g.textBaseline = "middle";
|
||||
// ruler
|
||||
const step = rulerStep(view.pxPerSec), dec = step < 0.1 ? 2 : step < 1 ? 1 : 0;
|
||||
const t0 = Math.floor(xToTime(view, 0) / step) * step, t1 = xToTime(view, view.width);
|
||||
g.strokeStyle = "#ccc"; g.fillStyle = C.muted; g.textAlign = "left";
|
||||
for (let t = t0; t <= t1; t += step) {
|
||||
const x = Math.round(X(t)) + 0.5;
|
||||
if (x < GUTTER) continue;
|
||||
g.beginPath(); g.moveTo(x, RULER_H - 7); g.lineTo(x, RULER_H); g.stroke();
|
||||
g.fillText(formatTime(t, dec), x + 3, RULER_H / 2 - 1);
|
||||
}
|
||||
g.beginPath(); g.moveTo(GUTTER, RULER_H + 0.5); g.lineTo(W, RULER_H + 0.5); g.stroke();
|
||||
if (!song) return;
|
||||
|
||||
// lanes and flagged windows
|
||||
paintWave(recWave, REC_Y, C.rec, 0.85);
|
||||
paintWave(midiWave, MIDI_Y, C.midi, rendering || midiKey !== keyOf(anchors) ? 0.35 : 0.85);
|
||||
const outside = (a: number, b: number) => { const x0 = Math.max(GUTTER, X(a)), x1 = Math.min(W, X(b)); if (x1 > x0) { g.fillRect(x0, REC_Y, x1 - x0, LANE_H); g.fillRect(x0, MIDI_Y, x1 - x0, LANE_H); } };
|
||||
g.fillStyle = "rgba(0,0,0,.05)"; outside(-1e6, 0); outside(dur(), 1e6);
|
||||
if (fit) {
|
||||
g.fillStyle = C.flag;
|
||||
for (const w of fit.windows) if (w.flagged) outside(w.startSec, w.startSec + WINDOW_SEC);
|
||||
}
|
||||
if (rendering || midiKey !== keyOf(anchors)) { g.fillStyle = C.muted; g.textAlign = "right"; g.fillText(midiWave ? "re-rendering MIDI…" : "rendering MIDI…", W - 8, MIDI_Y + 10); }
|
||||
|
||||
// stretch labels (MIDI bpm -> effective bpm), between consecutive anchors
|
||||
g.textAlign = "center";
|
||||
for (const s of info) {
|
||||
const xa = X(warp.toAudio(s.fromMidi)), xb = X(warp.toAudio(s.toMidi));
|
||||
if (xb < GUTTER || xa > W) continue;
|
||||
const label = stretchLabel(s), tw = g.measureText(label).width;
|
||||
const cx = (Math.max(xa, GUTTER) + Math.min(xb, W)) / 2, room = Math.min(xb, W) - Math.max(xa, GUTTER);
|
||||
if (s.warn) { g.fillStyle = "rgba(217,83,79,.5)"; g.fillRect(Math.max(xa, GUTTER), STRETCH_Y + 2, Math.min(xb, W) - Math.max(xa, GUTTER), 2); }
|
||||
g.fillStyle = s.warn ? C.warn : "#555";
|
||||
if (tw + 10 <= room) g.fillText(label, cx, STRETCH_Y + 14);
|
||||
else if (s.warn && room >= 12) g.fillText("⚠", cx, STRETCH_Y + 14);
|
||||
}
|
||||
|
||||
// anchors
|
||||
let lastLabelEnd = -Infinity;
|
||||
anchors.forEach((a, k) => {
|
||||
const x = Math.round(X(a.audioSec)) + 0.5;
|
||||
if (x < GUTTER - 2 || x > W + 2) return;
|
||||
const manual = a.kind === "manual", sel = k === selected, hov = k === hover;
|
||||
g.strokeStyle = sel ? "#16804c" : manual || hov ? C.anchor : C.anchorAuto;
|
||||
g.lineWidth = sel ? 3 : manual ? 2 : hov ? 2 : 1;
|
||||
g.beginPath(); g.moveTo(x, REC_Y - 2); g.lineTo(x, MIDI_Y + LANE_H); g.stroke();
|
||||
if (manual || sel || hov) { // grab handles at the recording (top) and MIDI (bottom) ends
|
||||
g.fillStyle = sel ? "#16804c" : C.anchor;
|
||||
g.fillRect(x - 4, REC_Y - 2, 8, 6);
|
||||
g.fillRect(x - 4, MIDI_Y + LANE_H - 6, 8, 6);
|
||||
}
|
||||
const label = formatTime(a.audioSec, view.pxPerSec > 200 ? 2 : 1), tw = g.measureText(label).width;
|
||||
if ((manual || sel || hov || spacingOk(k)) && x - tw / 2 > lastLabelEnd + 6) {
|
||||
g.fillStyle = manual || sel ? "#1e8a57" : "#6fbf97"; g.textAlign = "center";
|
||||
g.fillText(label, x, RULER_H + LABEL_H / 2);
|
||||
lastLabelEnd = x + tw / 2;
|
||||
}
|
||||
});
|
||||
g.lineWidth = 1;
|
||||
|
||||
// hover tooltip
|
||||
if (hover >= 0 && hover < anchors.length) {
|
||||
const a = anchors[hover]!;
|
||||
const txt = `${a.kind === "manual" ? "Your anchor" : "Automatic anchor"} · recording ${formatTime(a.audioSec, 2)} = MIDI ${formatTime(a.midiSec, 2)} · drag in the ${hoverLane === "midi" ? "MIDI lane to slide the MIDI" : "recording lane to move it"}`;
|
||||
const tw = g.measureText(txt).width + 12;
|
||||
const x = Math.min(W - tw - 4, Math.max(GUTTER, X(a.audioSec) + 8));
|
||||
const y = hoverLane === "midi" ? MIDI_Y + 6 : REC_Y + 6;
|
||||
g.fillStyle = "rgba(34,34,34,.85)"; g.fillRect(x, y, tw, 18);
|
||||
g.fillStyle = "#fff"; g.textAlign = "left"; g.fillText(txt, x + 6, y + 9);
|
||||
}
|
||||
|
||||
// cursor and playhead
|
||||
const line = (t: number, color: string, w: number) => {
|
||||
const x = Math.round(X(t)) + 0.5;
|
||||
if (x < GUTTER || x > W) return;
|
||||
g.strokeStyle = color; g.lineWidth = w;
|
||||
g.beginPath(); g.moveTo(x, 0); g.lineTo(x, MIDI_Y + LANE_H); g.stroke();
|
||||
g.lineWidth = 1;
|
||||
};
|
||||
line(cursor, C.ink, 1);
|
||||
{ const x = X(cursor); if (x >= GUTTER) { g.fillStyle = C.ink; g.beginPath(); g.moveTo(x - 5, 0); g.lineTo(x + 5, 0); g.lineTo(x, 7); g.fill(); } }
|
||||
if (playing) line(playPos(), C.play, 2);
|
||||
|
||||
// gutter with lane tags (drawn last so it masks anything scrolled under it)
|
||||
g.fillStyle = "#fff"; g.fillRect(0, 0, GUTTER, HEIGHT);
|
||||
g.textAlign = "left"; g.fillStyle = "#555"; g.font = "12px system-ui, sans-serif";
|
||||
g.fillText("Recording", 10, REC_Y + 16);
|
||||
g.fillText("MIDI audio", 10, MIDI_Y + 16);
|
||||
g.font = "11px system-ui, sans-serif"; g.fillStyle = C.muted;
|
||||
g.fillText("drag = move", 10, REC_Y + 32);
|
||||
g.fillText("drag = slide", 10, MIDI_Y + 32);
|
||||
g.fillText("tempo", 10, STRETCH_Y + 14);
|
||||
g.fillText(formatTime(cursor, 2), 10, RULER_H / 2);
|
||||
}
|
||||
|
||||
function spacingOk(k: number): boolean {
|
||||
const px = (i: number) => (anchors[i] ? Math.abs(timeToX(view, anchors[i]!.audioSec) - timeToX(view, anchors[k]!.audioSec)) : Infinity);
|
||||
return Math.min(px(k - 1), px(k + 1)) >= 70;
|
||||
}
|
||||
|
||||
// ---------- pointer ----------
|
||||
type Drag =
|
||||
| { kind: "pan"; x0: number; lastX: number; moved: boolean }
|
||||
| { kind: "cursor" }
|
||||
| { kind: "rec" | "midi"; index: number; start: Anchor[]; grabTime: number; grabMidi: number; offset: number; x0: number; moved: boolean };
|
||||
let drag: Drag | null = null;
|
||||
|
||||
const local = (e: PointerEvent | WheelEvent) => { const r = canvas.getBoundingClientRect(); return { x: e.clientX - r.left, y: e.clientY - r.top }; };
|
||||
const laneAt = (y: number): "rec" | "midi" | null => (y >= REC_Y - 4 && y < MIDI_Y - LANE_GAP / 2 ? "rec" : y >= MIDI_Y - LANE_GAP / 2 && y <= MIDI_Y + LANE_H + 2 ? "midi" : null);
|
||||
|
||||
canvas.addEventListener("pointerdown", (e) => {
|
||||
if (!song || busy || e.button !== 0) return;
|
||||
const { x, y } = local(e);
|
||||
if (x < GUTTER) return;
|
||||
canvas.setPointerCapture(e.pointerId);
|
||||
const lx = x - GUTTER, lane = laneAt(y);
|
||||
if (y < REC_Y - 4) { drag = { kind: "cursor" }; setCursor(xToTime(view, lx), false); return; }
|
||||
const hit = lane ? hitAnchor(anchors, view, lx, HIT_PX) : -1;
|
||||
if (hit >= 0 && lane) {
|
||||
selected = hit;
|
||||
const t = xToTime(view, lx);
|
||||
drag = { kind: lane, index: hit, start: anchors, grabTime: t, grabMidi: warp.toMidi(t), offset: anchors[hit]!.audioSec - t, x0: x, moved: false };
|
||||
updateUi(); draw();
|
||||
return;
|
||||
}
|
||||
drag = { kind: "pan", x0: x, lastX: x, moved: false };
|
||||
canvas.style.cursor = "grabbing";
|
||||
});
|
||||
|
||||
canvas.addEventListener("pointermove", (e) => {
|
||||
if (!song) return;
|
||||
const { x, y } = local(e), lx = x - GUTTER;
|
||||
if (!drag) {
|
||||
const lane = x >= GUTTER ? laneAt(y) : null;
|
||||
const h = lane ? hitAnchor(anchors, view, lx, HIT_PX) : -1;
|
||||
if (h !== hover || lane !== hoverLane) { hover = h; hoverLane = lane; draw(); }
|
||||
canvas.style.cursor = h >= 0 ? "ew-resize" : y < REC_Y - 4 && x >= GUTTER ? "text" : "default";
|
||||
return;
|
||||
}
|
||||
if (drag.kind === "cursor") { setCursor(xToTime(view, lx), false); return; }
|
||||
if (drag.kind === "pan") {
|
||||
if (Math.abs(x - drag.x0) > DRAG_PX) drag.moved = true;
|
||||
if (drag.moved) { view = scrollBy(view, x - drag.lastX, dur()); drag.lastX = x; draw(); }
|
||||
return;
|
||||
}
|
||||
if (!drag.moved && Math.abs(x - drag.x0) <= DRAG_PX) return;
|
||||
drag.moved = true;
|
||||
hover = -1;
|
||||
const t = xToTime(view, lx);
|
||||
try {
|
||||
const r = drag.kind === "rec" ? dragAudio(drag.start, drag.index, t + drag.offset) : dragMidi(drag.start, drag.index, drag.grabMidi, t);
|
||||
selected = r.index;
|
||||
setAnchors(r.anchors); // live preview; committed on release
|
||||
} catch (err) { setStatus((err as Error).message, true); }
|
||||
});
|
||||
|
||||
function endDrag(e: PointerEvent) {
|
||||
if (!drag) return;
|
||||
const d = drag;
|
||||
drag = null;
|
||||
canvas.style.cursor = "default";
|
||||
const { x } = local(e);
|
||||
if (d.kind === "cursor" && playing) void play(); // scrubbing seeks playback once, on release
|
||||
else if (d.kind === "pan" && !d.moved && x >= GUTTER) { selected = -1; setCursor(xToTime(view, x - GUTTER)); updateUi(); }
|
||||
else if ((d.kind === "rec" || d.kind === "midi") && d.moved) {
|
||||
const now = anchors;
|
||||
anchors = d.start; // commit from the drag-start state so undo returns there
|
||||
commit(now, d.start);
|
||||
const removedAutos = d.start.filter((a) => a.kind === "auto").length - now.filter((a) => a.kind === "auto").length
|
||||
- (d.start[d.index]!.kind === "auto" ? 1 : 0); // the dragged auto itself became manual
|
||||
if (removedAutos > 0) setStatus(`Moved the anchor; ${removedAutos} automatic anchors within ${AUTO_CLEAR_SEC} s of it were removed. Use "Auto-align around my anchors" to fill them in again.`);
|
||||
}
|
||||
}
|
||||
canvas.addEventListener("pointerup", endDrag);
|
||||
canvas.addEventListener("pointercancel", endDrag);
|
||||
canvas.addEventListener("pointerleave", () => { if (!drag && hover >= 0) { hover = -1; draw(); } });
|
||||
|
||||
function setCursor(t: number, seek = true) {
|
||||
cursor = Math.min(dur(), Math.max(0, t));
|
||||
if (seek && playing) void play(); // jump playback to the new cursor
|
||||
draw();
|
||||
}
|
||||
|
||||
canvas.addEventListener("wheel", (e) => {
|
||||
if (!song) return;
|
||||
e.preventDefault();
|
||||
const scale = e.deltaMode === 1 ? 33 : e.deltaMode === 2 ? 400 : 1;
|
||||
const dx = e.deltaX * scale, dy = e.deltaY * scale;
|
||||
const { x } = local(e);
|
||||
if (e.shiftKey || Math.abs(dx) > Math.abs(dy)) view = scrollBy(view, -(e.shiftKey ? dy || dx : dx), dur());
|
||||
else view = zoomAt(view, Math.max(0, x - GUTTER), Math.exp(-dy * (e.ctrlKey ? 0.01 : 0.0015)), dur());
|
||||
draw();
|
||||
}, { passive: false });
|
||||
|
||||
// ---------- keyboard & buttons ----------
|
||||
window.addEventListener("keydown", (e) => {
|
||||
const tag = (e.target as HTMLElement).tagName;
|
||||
if (tag === "SELECT" || tag === "INPUT" || tag === "TEXTAREA" || !song) return;
|
||||
const mod = e.ctrlKey || e.metaKey;
|
||||
if (mod && e.key.toLowerCase() === "z") { e.preventDefault(); undo(e.shiftKey); return; }
|
||||
if (mod && e.key.toLowerCase() === "y") { e.preventDefault(); undo(true); return; }
|
||||
if (mod && e.key.toLowerCase() === "s") { e.preventDefault(); void save(); return; }
|
||||
if (mod || e.altKey || busy) return;
|
||||
switch (e.key) {
|
||||
case " ": e.preventDefault(); playing ? stop() : void play(); break;
|
||||
case "Delete": case "Backspace": e.preventDefault(); deleteSelected(); break;
|
||||
case "a": case "A": addAtCursor(); break;
|
||||
case "ArrowLeft": e.preventDefault(); nudge(e.shiftKey ? -0.1 : -0.01); break;
|
||||
case "ArrowRight": e.preventDefault(); nudge(e.shiftKey ? 0.1 : 0.01); break;
|
||||
case "Escape": selected = -1; updateUi(); draw(); break;
|
||||
case "0": view = fitView(dur(), view.width); draw(); break;
|
||||
case "1": case "2": case "3": case "4": mix = (["both", "rec", "midi", "split"] as const)[Number(e.key) - 1]!; applyMix(); break;
|
||||
}
|
||||
});
|
||||
|
||||
btn.play.addEventListener("click", () => (playing ? stop() : void play()));
|
||||
btn.add.addEventListener("click", addAtCursor);
|
||||
btn.del.addEventListener("click", deleteSelected);
|
||||
btn.undo.addEventListener("click", () => undo());
|
||||
btn.auto.addEventListener("click", () => void autoAlign());
|
||||
btn.save.addEventListener("click", () => void save());
|
||||
btn.reset.addEventListener("click", reset);
|
||||
for (const b of mixButtons) b.addEventListener("click", () => { mix = b.dataset.mix as Mix; applyMix(); });
|
||||
// keep Space/Enter from re-triggering the last clicked button
|
||||
for (const b of document.querySelectorAll("button")) b.addEventListener("keydown", (e) => { if (e.key === " ") e.preventDefault(); });
|
||||
songSel.addEventListener("change", () => void openSong(songSel.value));
|
||||
window.addEventListener("beforeunload", (e) => { if (isDirty() && (undoStack.length || savedKey !== null)) e.preventDefault(); });
|
||||
new ResizeObserver(resize).observe(wrap);
|
||||
|
||||
applyMix();
|
||||
resize();
|
||||
updateUi();
|
||||
loadSongs().catch((e) => showOverlay(`Could not load the song list: ${(e as Error).message}`));
|
||||
@@ -0,0 +1,65 @@
|
||||
// Anchor editing for the alignment page: drags, hit-testing, add-at-cursor. Pure (no DOM); wraps src/align/warp.ts.
|
||||
import { addManual, makeWarp, moveAnchor, type Anchor } from "../warp.ts";
|
||||
import { timeToX, type View } from "./timeline.ts";
|
||||
|
||||
/** Closest a dragged anchor may get to a manual neighbour (seconds, both axes). */
|
||||
export const MIN_GAP_SEC = 0.05;
|
||||
|
||||
export interface Edit { anchors: Anchor[]; index: number }
|
||||
|
||||
/** Nearest manual anchors left/right of index `i` (autos are dropped by the edit, so only manuals constrain it). */
|
||||
function manualNeighbours(anchors: Anchor[], i: number): { left?: Anchor; right?: Anchor } {
|
||||
let left: Anchor | undefined, right: Anchor | undefined;
|
||||
for (let k = i - 1; k >= 0 && !left; k--) if (anchors[k]!.kind === "manual") left = anchors[k];
|
||||
for (let k = i + 1; k < anchors.length && !right; k++) if (anchors[k]!.kind === "manual") right = anchors[k];
|
||||
return { left, right };
|
||||
}
|
||||
|
||||
const clamp = (x: number, lo: number, hi: number) => Math.min(hi, Math.max(lo, x));
|
||||
|
||||
function moved(anchors: Anchor[], i: number, midiSec: number, audioSec: number): Edit {
|
||||
const out = moveAnchor(anchors, i, { midiSec, audioSec });
|
||||
return { anchors: out, index: out.findIndex((a) => a.kind === "manual" && a.midiSec === midiSec && a.audioSec === audioSec) };
|
||||
}
|
||||
|
||||
/** Recording end: anchor `i` of `start` (the list at drag start) to `audioSec`, MIDI time unchanged; clamped between manual neighbours and at >= 0. */
|
||||
export function dragAudio(start: Anchor[], i: number, audioSec: number): Edit {
|
||||
const A = makeWarp(start).anchors;
|
||||
const { left, right } = manualNeighbours(A, i);
|
||||
const a = clamp(audioSec, left ? left.audioSec + MIN_GAP_SEC : 0, right ? right.audioSec - MIN_GAP_SEC : Infinity);
|
||||
return moved(A, i, A[i]!.midiSec, a);
|
||||
}
|
||||
|
||||
/**
|
||||
* MIDI end: the anchor keeps its recording time; its MIDI time is chosen so that the MIDI moment `grabMidi` (what was
|
||||
* under the pointer on the warped MIDI lane at drag start) is drawn at `toAudio` (where the pointer is now). The
|
||||
* grabbed point therefore follows the mouse. Solved by bisection (the drawn position falls as the anchor's MIDI time
|
||||
* rises); clamped between the manual neighbours' MIDI times.
|
||||
*/
|
||||
export function dragMidi(start: Anchor[], i: number, grabMidi: number, toAudio: number): Edit {
|
||||
const A = makeWarp(start).anchors;
|
||||
const { left, right } = manualNeighbours(A, i);
|
||||
const a = A[i]!.audioSec, SPAN = 3600;
|
||||
const lo0 = left ? left.midiSec + MIN_GAP_SEC : A[i]!.midiSec - SPAN;
|
||||
const hi0 = right ? right.midiSec - MIN_GAP_SEC : A[i]!.midiSec + SPAN;
|
||||
const drawnAt = (m: number) => makeWarp(moveAnchor(A, i, { midiSec: m, audioSec: a })).toAudio(grabMidi);
|
||||
let lo = lo0, hi = hi0;
|
||||
if (drawnAt(lo) <= toAudio) hi = lo;
|
||||
else if (drawnAt(hi) >= toAudio) lo = hi;
|
||||
else for (let k = 0; k < 60 && hi - lo > 1e-9; k++) { const mid = (lo + hi) / 2; if (drawnAt(mid) > toAudio) lo = mid; else hi = mid; }
|
||||
return moved(A, i, (lo + hi) / 2, a);
|
||||
}
|
||||
|
||||
/** Index of the anchor line nearest pixel `x` within `tolPx`, or -1. */
|
||||
export function hitAnchor(anchors: Anchor[], v: View, x: number, tolPx: number): number {
|
||||
let best = -1, bestD = tolPx;
|
||||
anchors.forEach((a, k) => { const d = Math.abs(timeToX(v, a.audioSec) - x); if (d <= bestD) { best = k; bestD = d; } });
|
||||
return best;
|
||||
}
|
||||
|
||||
/** New manual anchor at recording time `cursorSec`, its MIDI time from the current warp (so nothing jumps there). */
|
||||
export function anchorAtCursor(anchors: Anchor[], cursorSec: number): Edit {
|
||||
const midiSec = makeWarp(anchors).toMidi(cursorSec);
|
||||
const out = addManual(anchors, { midiSec, audioSec: cursorSec });
|
||||
return { anchors: out, index: out.findIndex((a) => a.audioSec === cursorSec && a.kind === "manual") };
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>STSheets · MIDI alignment</title>
|
||||
<link rel="stylesheet" href="./style.css" />
|
||||
</head>
|
||||
<body>
|
||||
<header class="bar">
|
||||
<h1>MIDI alignment</h1>
|
||||
<label class="picker">Song <select id="song" aria-label="Song"></select></label>
|
||||
<span id="songinfo" class="muted"></span>
|
||||
<span class="grow"></span>
|
||||
<span id="savestate" class="savestate"></span>
|
||||
</header>
|
||||
|
||||
<section class="fitbar" aria-live="polite">
|
||||
<span id="fit" class="fit">Fit: –</span>
|
||||
<span id="counts" class="muted"></span>
|
||||
</section>
|
||||
|
||||
<div id="wrap" class="wrap">
|
||||
<canvas id="tl" aria-label="Recording and MIDI timeline"></canvas>
|
||||
<div id="overlay" class="overlay" hidden></div>
|
||||
</div>
|
||||
|
||||
<div class="row controls">
|
||||
<button id="play" title="Space">▶ Play from cursor</button>
|
||||
<div class="mix" role="radiogroup" aria-label="Mix">
|
||||
<span class="mixlabel">Mix:</span>
|
||||
<button data-mix="both" class="on" title="1">both</button>
|
||||
<button data-mix="rec" title="2">rec only</button>
|
||||
<button data-mix="midi" title="3">MIDI only</button>
|
||||
<button data-mix="split" title="4">rec L / MIDI R</button>
|
||||
</div>
|
||||
<button id="add" title="A">+ Anchor at cursor</button>
|
||||
<button id="del" title="Delete" disabled>Delete anchor</button>
|
||||
<button id="undo" title="Ctrl+Z" disabled>Undo</button>
|
||||
<span class="grow"></span>
|
||||
<button id="auto">Auto-align around my anchors</button>
|
||||
<button id="save" class="primary" title="Ctrl+S">Save alignment</button>
|
||||
<button id="reset" class="quiet">Reset to raw MIDI</button>
|
||||
</div>
|
||||
|
||||
<p id="status" class="status" role="status"></p>
|
||||
|
||||
<details class="help">
|
||||
<summary>How it works · keyboard</summary>
|
||||
<p>Green lines are <b>anchors</b>: "this MIDI moment = this recording moment". Between two anchors the MIDI's tempo is scaled so it fits;
|
||||
the label under the lanes shows MIDI bpm → resulting bpm. <b>⚠</b> = a stretch changed by more than 5%; that usually means an anchor sits on the wrong bar.
|
||||
Thick lines are your (manual) anchors, thin ones were placed automatically. Red-shaded 30 s windows fit poorly.</p>
|
||||
<ul>
|
||||
<li><b>Click</b> the lanes to set the cursor (black line); click an anchor to select it. <b>Wheel</b> zooms around the mouse (Shift+wheel or a sideways swipe scrolls); <b>drag the background</b> to scroll.</li>
|
||||
<li>Drag an anchor in the <b>Recording</b> lane to move it along the recording (the MIDI moment stays attached to it).
|
||||
Drag it in the <b>MIDI audio</b> lane to slide the MIDI under it: the bit of MIDI you grabbed follows the mouse while the anchor stays at the same place in the recording.</li>
|
||||
<li>Moving or adding an anchor removes the automatic anchors within 10 s of it (never past your other anchors), plus any that would now be out of order; <b>Auto-align around my anchors</b> fills them in again.</li>
|
||||
<li>Keys: <kbd>Space</kbd> play/stop · <kbd>A</kbd> anchor at cursor · <kbd>Delete</kbd> delete selected · <kbd>←</kbd>/<kbd>→</kbd> nudge selected anchor 10 ms (<kbd>Shift</kbd> 100 ms; nothing selected: move cursor 1 s) ·
|
||||
<kbd>1</kbd>–<kbd>4</kbd> mix · <kbd>Ctrl</kbd>+<kbd>Z</kbd> undo / <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>Z</kbd> redo · <kbd>Ctrl</kbd>+<kbd>S</kbd> save · <kbd>0</kbd> show whole song.</li>
|
||||
</ul>
|
||||
</details>
|
||||
|
||||
<script type="module" src="./app.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,42 @@
|
||||
:root {
|
||||
--rec: #5b8def; --midi: #e2873b; --anchor: #2bb673; --warn: #d9534f; --ink: #222; --muted: #777;
|
||||
--bg: #fafafa; --panel: #fff; --line: #ddd;
|
||||
font-family: system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; color: var(--ink); background: var(--bg);
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; padding: 12px 16px 24px; }
|
||||
h1 { font-size: 16px; margin: 0; font-weight: 600; }
|
||||
.bar, .row, .fitbar { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
|
||||
.bar { padding-bottom: 8px; border-bottom: 1px solid var(--line); }
|
||||
.grow { flex: 1; }
|
||||
.muted { color: var(--muted); font-size: 13px; }
|
||||
.picker { font-size: 13px; color: var(--muted); display: flex; align-items: center; gap: 6px; }
|
||||
select { font: inherit; font-size: 14px; padding: 3px 6px; color: var(--ink); max-width: 40ch; }
|
||||
.savestate { font-size: 13px; padding: 2px 8px; border-radius: 10px; }
|
||||
.savestate.dirty { background: #fff3e0; color: #b45309; }
|
||||
.savestate.clean { color: #2f7d4f; }
|
||||
.fitbar { margin: 8px 0 6px; font-size: 13px; min-height: 20px; }
|
||||
.fit.bad { color: var(--warn); }
|
||||
.fit.stale { opacity: .55; }
|
||||
.wrap { position: relative; background: var(--panel); border: 1px solid var(--line); border-radius: 6px; overflow: hidden; user-select: none; }
|
||||
canvas { display: block; width: 100%; touch-action: none; }
|
||||
.overlay { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; background: rgba(255,255,255,.75); font-size: 14px; color: #333; text-align: center; padding: 20px; }
|
||||
.overlay[hidden] { display: none; }
|
||||
.controls { margin-top: 10px; }
|
||||
button { font: inherit; font-size: 13px; padding: 5px 10px; border: 1px solid #c8c8c8; background: #fff; border-radius: 5px; cursor: pointer; color: var(--ink); }
|
||||
button:hover:not(:disabled) { background: #f1f1f1; }
|
||||
button:disabled { opacity: .45; cursor: default; }
|
||||
button.primary { background: var(--anchor); border-color: #22975f; color: #fff; }
|
||||
button.primary:hover:not(:disabled) { background: #26a467; }
|
||||
button.quiet { color: #a33; }
|
||||
button:focus-visible { outline: 2px solid var(--rec); outline-offset: 1px; }
|
||||
.mix { display: inline-flex; align-items: center; border: 1px solid #c8c8c8; border-radius: 5px; overflow: hidden; background: #fff; }
|
||||
.mixlabel { font-size: 13px; padding: 0 6px 0 8px; color: var(--muted); }
|
||||
.mix button { border: 0; border-left: 1px solid #e3e3e3; border-radius: 0; }
|
||||
.mix button.on { background: #2b2b2b; color: #fff; }
|
||||
.status { min-height: 18px; font-size: 13px; margin: 8px 0 0; color: #444; }
|
||||
.status.error { color: var(--warn); }
|
||||
.help { margin-top: 10px; font-size: 13px; color: #555; max-width: 110ch; }
|
||||
.help summary { cursor: pointer; color: var(--muted); }
|
||||
.help li { margin: 4px 0; }
|
||||
kbd { font: 11px ui-monospace, monospace; border: 1px solid #ccc; border-bottom-width: 2px; border-radius: 3px; padding: 0 4px; background: #fff; }
|
||||
@@ -0,0 +1,121 @@
|
||||
// Timeline geometry for the alignment page: time <-> pixel mapping, zoom/scroll, ruler, waveform peaks.
|
||||
// Pure (no DOM), unit-tested in tests/align/page/timeline.test.ts.
|
||||
|
||||
/** `start` = time (s) at x = 0 of the lane area; `width` = lane area width in CSS px. */
|
||||
export interface View { start: number; pxPerSec: number; width: number }
|
||||
|
||||
export const MAX_PX_PER_SEC = 4000; // ~11 samples per px at 44.1 kHz
|
||||
const MARGIN = 0.01; // fraction of the song shown beyond either end
|
||||
|
||||
export const timeToX = (v: View, t: number) => (t - v.start) * v.pxPerSec;
|
||||
export const xToTime = (v: View, x: number) => v.start + x / v.pxPerSec;
|
||||
|
||||
const minPxPerSec = (dur: number, width: number) => width / (Math.max(dur, 1e-3) * (1 + 2 * MARGIN));
|
||||
|
||||
/** The whole song, with a small margin either side. */
|
||||
export function fitView(dur: number, width: number): View {
|
||||
return { start: -dur * MARGIN, pxPerSec: minPxPerSec(dur, width), width };
|
||||
}
|
||||
|
||||
/** Clamps zoom to [whole song, MAX_PX_PER_SEC] and scroll so the view stays within the song (plus margin). */
|
||||
export function clampView(v: View, dur: number): View {
|
||||
const lo = minPxPerSec(dur, v.width);
|
||||
const pxPerSec = Math.min(MAX_PX_PER_SEC, Math.max(lo, v.pxPerSec));
|
||||
const visible = v.width / pxPerSec, m = dur * MARGIN;
|
||||
const start = Math.min(Math.max(v.start, -m), Math.max(-m, dur + m - visible));
|
||||
return start === v.start && pxPerSec === v.pxPerSec ? v : { start, pxPerSec, width: v.width };
|
||||
}
|
||||
|
||||
/** Zoom by `factor` keeping the time under pixel `x` fixed (as far as clamping allows). */
|
||||
export function zoomAt(v: View, x: number, factor: number, dur: number): View {
|
||||
const t = xToTime(v, x);
|
||||
const pxPerSec = Math.min(MAX_PX_PER_SEC, Math.max(minPxPerSec(dur, v.width), v.pxPerSec * factor));
|
||||
return clampView({ start: t - x / pxPerSec, pxPerSec, width: v.width }, dur);
|
||||
}
|
||||
|
||||
/** Drag-scroll: content follows the pointer, so dragging right by dx shows earlier time. */
|
||||
export function scrollBy(v: View, dx: number, dur: number): View {
|
||||
return clampView({ ...v, start: v.start - dx / v.pxPerSec }, dur);
|
||||
}
|
||||
|
||||
/** Pages the view when `t` (e.g. the playhead) leaves it; unchanged otherwise. */
|
||||
export function follow(v: View, t: number, dur: number): View {
|
||||
const visible = v.width / v.pxPerSec;
|
||||
if (t >= v.start && t <= v.start + visible * 0.95) return v;
|
||||
return clampView({ ...v, start: t - visible * 0.05 }, dur);
|
||||
}
|
||||
|
||||
/** "m:ss.s" (decimals configurable). */
|
||||
export function formatTime(sec: number, decimals = 1): string {
|
||||
const p = 10 ** decimals;
|
||||
const total = Math.round(Math.abs(sec) * p) / p;
|
||||
const mins = Math.floor(total / 60);
|
||||
const s = (total - mins * 60).toFixed(decimals).padStart(decimals ? 3 + decimals : 2, "0");
|
||||
return `${sec < 0 && total > 0 ? "-" : ""}${mins}:${s}`;
|
||||
}
|
||||
|
||||
const STEPS = [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10, 15, 30, 60, 120, 300, 600, 1200];
|
||||
/** Seconds between labelled ruler ticks so labels are at least 80 px apart. */
|
||||
export function rulerStep(pxPerSec: number): number {
|
||||
return STEPS.find((s) => s * pxPerSec >= 80) ?? STEPS[STEPS.length - 1]!;
|
||||
}
|
||||
|
||||
/** One level of a min/max pyramid: `bucket` = input samples per entry. */
|
||||
export interface PeakLevel { bucket: number; min: Float32Array; max: Float32Array }
|
||||
|
||||
function levelsFrom(level: PeakLevel): PeakLevel[] {
|
||||
const out = [level];
|
||||
let cur = level;
|
||||
while (cur.min.length > 2) {
|
||||
const n = Math.ceil(cur.min.length / 2);
|
||||
const min = new Float32Array(n), max = new Float32Array(n);
|
||||
for (let i = 0; i < n; i++) {
|
||||
const a = 2 * i, b = Math.min(2 * i + 1, cur.min.length - 1);
|
||||
min[i] = Math.min(cur.min[a]!, cur.min[b]!);
|
||||
max[i] = Math.max(cur.max[a]!, cur.max[b]!);
|
||||
}
|
||||
cur = { bucket: cur.bucket * 2, min, max };
|
||||
out.push(cur);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Min/max pyramid of a waveform: level 0 has `base` samples per bucket, each next level doubles it. */
|
||||
export function buildPyramid(x: Float32Array, base = 64): PeakLevel[] {
|
||||
const n = Math.max(1, Math.ceil(x.length / base));
|
||||
const min = new Float32Array(n), max = new Float32Array(n);
|
||||
for (let b = 0; b < n; b++) {
|
||||
let lo = Infinity, hi = -Infinity;
|
||||
for (let i = b * base, e = Math.min(x.length, i + base); i < e; i++) { const s = x[i]!; if (s < lo) lo = s; if (s > hi) hi = s; }
|
||||
min[b] = lo === Infinity ? 0 : lo; max[b] = hi === -Infinity ? 0 : hi;
|
||||
}
|
||||
return levelsFrom({ bucket: base, min, max });
|
||||
}
|
||||
|
||||
/** Pyramid over precomputed peaks (e.g. from the server): each entry counts as one "sample". */
|
||||
export function pyramidFromPeaks(min: ArrayLike<number>, max: ArrayLike<number>): PeakLevel[] {
|
||||
return levelsFrom({ bucket: 1, min: Float32Array.from(min), max: Float32Array.from(max) });
|
||||
}
|
||||
|
||||
/** Per pixel column of `v`: min/max of the waveform (sampled at `sampleRate`) under that column; 0 outside it. */
|
||||
export function columnPeaks(levels: PeakLevel[], sampleRate: number, v: View): { min: Float32Array; max: Float32Array } {
|
||||
const w = Math.max(0, Math.round(v.width));
|
||||
const min = new Float32Array(w), max = new Float32Array(w);
|
||||
const spp = sampleRate / v.pxPerSec;
|
||||
let L = levels[0]!;
|
||||
for (const l of levels) if (l.bucket <= spp) L = l;
|
||||
const n = L.min.length;
|
||||
for (let c = 0; c < w; c++) {
|
||||
const s0 = (v.start + c / v.pxPerSec) * sampleRate;
|
||||
const b0 = Math.floor(s0 / L.bucket);
|
||||
const b1 = Math.max(b0, Math.ceil((s0 + spp) / L.bucket) - 1);
|
||||
if (b1 < 0 || b0 >= n) continue;
|
||||
let lo = Infinity, hi = -Infinity;
|
||||
for (let b = Math.max(0, b0), e = Math.min(n - 1, b1); b <= e; b++) {
|
||||
if (L.min[b]! < lo) lo = L.min[b]!;
|
||||
if (L.max[b]! > hi) hi = L.max[b]!;
|
||||
}
|
||||
min[c] = lo; max[c] = hi;
|
||||
}
|
||||
return { min, max };
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
/** Per-bucket min/max of a waveform, for drawing. Pure. */
|
||||
export function peaks(x: Float32Array, buckets: number): { min: Float32Array; max: Float32Array } {
|
||||
const min = new Float32Array(buckets), max = new Float32Array(buckets);
|
||||
const n = x.length;
|
||||
for (let b = 0; b < buckets; b++) {
|
||||
const from = Math.floor((b * n) / buckets);
|
||||
const to = Math.max(from + 1, Math.floor(((b + 1) * n) / buckets));
|
||||
let lo = Infinity, hi = -Infinity;
|
||||
for (let i = from; i < to && i < n; i++) { const v = x[i]!; if (v < lo) lo = v; if (v > hi) hi = v; }
|
||||
min[b] = lo === Infinity ? 0 : lo;
|
||||
max[b] = hi === -Infinity ? 0 : hi;
|
||||
}
|
||||
return { min, max };
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import * as at from "@coderline/alphatab";
|
||||
import type { Audio } from "../core/types.ts";
|
||||
import { renderMidiFile } from "../audio/render.ts";
|
||||
import { resample } from "../dsp/resample.ts";
|
||||
import type { RefTrack } from "./smf.ts";
|
||||
import type { Warp } from "./warp.ts";
|
||||
|
||||
const RENDER_SR = 44100; // alphaSynth yields NaN at other rates (e.g. 22050), so always render here and resample
|
||||
const TPS = 1920; // division 960 at a fixed 120 bpm
|
||||
const PEAK = Math.pow(10, -1 / 20);
|
||||
const MELODIC_CHANNELS = [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15];
|
||||
|
||||
/**
|
||||
* Render reference tracks to audio with `warp` applied (MIDI time -> audio time). Notes are placed on a fixed-tempo
|
||||
* tick grid so alphaSynth output time equals audio time. Peak-normalised to -1 dBFS; `durationSec` pads/trims.
|
||||
* Pure: no I/O. Notes landing before audio t=0 are clamped to 0 (dropped if they end before it).
|
||||
*/
|
||||
export function renderMidi(
|
||||
tracks: RefTrack[], warp: Warp, sf2: Uint8Array,
|
||||
opts: { sampleRate?: number; mono?: boolean; durationSec?: number } = {},
|
||||
): Audio {
|
||||
const sampleRate = opts.sampleRate ?? RENDER_SR;
|
||||
const midi = new at.midi.MidiFile();
|
||||
midi.division = 960;
|
||||
midi.addEvent(new at.midi.TempoChangeEvent(0, 500000));
|
||||
const seen = new Set<number>();
|
||||
let next = 0;
|
||||
for (const t of tracks) {
|
||||
// Beyond 15 melodic tracks channels wrap and are shared; the first track's program wins on a shared channel.
|
||||
const ch = t.isDrums ? 9 : MELODIC_CHANNELS[next++ % MELODIC_CHANNELS.length]!;
|
||||
if (!seen.has(ch)) { seen.add(ch); midi.addEvent(new at.midi.ProgramChangeEvent(0, 0, ch, t.isDrums ? 0 : t.program)); }
|
||||
for (const n of t.notes) {
|
||||
const offSec = warp.toAudio(n.offset);
|
||||
if (offSec <= 0) continue;
|
||||
const on = Math.round(Math.max(0, warp.toAudio(n.onset)) * TPS);
|
||||
const off = Math.max(on + 1, Math.round(offSec * TPS));
|
||||
midi.addEvent(new at.midi.NoteOnEvent(0, on, ch, n.pitch, n.velocity));
|
||||
midi.addEvent(new at.midi.NoteOffEvent(0, off, ch, n.pitch, 0));
|
||||
}
|
||||
}
|
||||
const out = renderMidiFile(midi, sf2, RENDER_SR, { frames: opts.durationSec === undefined ? undefined : Math.round(opts.durationSec * RENDER_SR) });
|
||||
let chans = out.channels;
|
||||
if (sampleRate !== RENDER_SR) {
|
||||
chans = chans.map((c) => resample(c, RENDER_SR, sampleRate));
|
||||
if (opts.durationSec !== undefined) { const n = Math.round(opts.durationSec * sampleRate); chans = chans.map((c) => { const o = new Float32Array(n); o.set(c.subarray(0, n)); return o; }); }
|
||||
}
|
||||
if (opts.mono) {
|
||||
const [L, R] = chans as [Float32Array, Float32Array];
|
||||
chans = [L.map((v, i) => (v + R[i]!) / 2)];
|
||||
}
|
||||
for (const c of chans) for (let i = 0; i < c.length; i++) if (!Number.isFinite(c[i]!)) throw new Error("renderMidi: non-finite samples from synth");
|
||||
let peak = 0;
|
||||
for (const c of chans) for (let i = 0; i < c.length; i++) { const a = Math.abs(c[i]!); if (a > peak) peak = a; }
|
||||
if (peak > 0) { const g = PEAK / peak; for (const c of chans) for (let i = 0; i < c.length; i++) c[i]! *= g; }
|
||||
return { sampleRate, channels: chans };
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
import type { Role } from "../core/types.ts";
|
||||
import type { RefTrack } from "./smf.ts";
|
||||
|
||||
// Best-effort role for a reference MIDI track; null = unmapped (excluded from scoring until the user maps it).
|
||||
export function guessRole(t: RefTrack): Role | null {
|
||||
if (t.isDrums) return "drums";
|
||||
const n = t.name.trim().toLowerCase();
|
||||
if (/vocal|voice|vox|sing/.test(n)) return "vocals";
|
||||
if (/bass/.test(n)) return "bass";
|
||||
if (/guitar|delay/.test(n)) return "guitar";
|
||||
if (/piano|key|synth|organ/.test(n)) return "keys";
|
||||
const p = t.program;
|
||||
if (p >= 32 && p <= 39) return "bass";
|
||||
if (p >= 24 && p <= 31) return "guitar";
|
||||
if ((p >= 0 && p <= 7) || (p >= 16 && p <= 23) || (p >= 88 && p <= 95)) return "keys";
|
||||
return null;
|
||||
}
|
||||
|
||||
// Role-map key for a track: its name, or "track N" when unnamed; disambiguated on collision.
|
||||
export function trackKeys(tracks: { index: number; name: string }[]): string[] {
|
||||
const used = new Set<string>();
|
||||
return tracks.map((t) => {
|
||||
let k = t.name || `track ${t.index}`;
|
||||
if (used.has(k)) k = `${k} #${t.index}`;
|
||||
used.add(k);
|
||||
return k;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,281 @@
|
||||
// Alignment web server: serves the page (HTML import) and the alignment API behind a random access token.
|
||||
// The page bundle lives on a loopback-only inner server; the public server checks the token and proxies to it,
|
||||
// because Bun serves HTML-import assets directly from `routes`, bypassing any auth in `fetch`.
|
||||
import { join } from "node:path";
|
||||
import { timingSafeEqual } from "node:crypto";
|
||||
import index from "./page/index.html";
|
||||
import type { Role } from "../core/types.ts";
|
||||
import { audioOgg, loadSong, readAlignment, readManifest, writeAlignment, type AlignmentFile, type LoadedSong } from "../ref/manifest.ts";
|
||||
import { trackKeys } from "./roles.ts";
|
||||
import { makeWarp, stretches, type Anchor } from "./warp.ts";
|
||||
import { autoAlign, fitFromCues, recordingCues, type AutoResult, type RecordingCues } from "./auto.ts";
|
||||
import { renderMidi } from "./render.ts";
|
||||
import { peaks } from "./peaks.ts";
|
||||
import { stretchInfo } from "./tempo.ts";
|
||||
import type { FitReport } from "./fit.ts";
|
||||
import { encodeWav, toMono } from "../audio/wav.ts";
|
||||
|
||||
export const SF2_PATH = join(import.meta.dir, "../../node_modules/@coderline/alphatab/dist/soundfont/sonivox.sf2");
|
||||
// Cookies are not port-scoped: two align servers on one host share this name, so opening one server's URL replaces
|
||||
// the other's cookie (the other then answers 401 until its own URL is opened again). Harmless; tokens never match.
|
||||
const COOKIE = "stsheets_align";
|
||||
const MAX_ANCHORS = 10_000;
|
||||
const MAX_BODY_BYTES = 1 << 20;
|
||||
const PEAKS_PER_SEC = 2000 / 60;
|
||||
const PEAKS_VERSION = 1;
|
||||
const RENDER_SR = 22050;
|
||||
const RENDER_CACHE = 6;
|
||||
const SONG_CACHE = 2;
|
||||
|
||||
export interface ServeOptions { root?: string; host?: string; port?: number; urlHost?: string; quiet?: boolean; sf2Path?: string }
|
||||
export interface AlignServer { url: string; origin: string; token: string; port: number; stop(): Promise<void> }
|
||||
|
||||
interface SongState extends LoadedSong {
|
||||
roles: Record<string, Role | null>;
|
||||
cues: RecordingCues;
|
||||
peaks: { bucketSec: number; min: number[]; max: number[] };
|
||||
auto?: AutoResult; // first automatic pass, reused while no alignment.json exists
|
||||
}
|
||||
|
||||
class HttpError extends Error { constructor(readonly status: number, msg: string) { super(msg); } }
|
||||
|
||||
/** Small insertion-ordered LRU. */
|
||||
class Lru<V> {
|
||||
private m = new Map<string, V>();
|
||||
constructor(private cap: number) {}
|
||||
get(k: string): V | undefined { const v = this.m.get(k); if (v !== undefined) { this.m.delete(k); this.m.set(k, v); } return v; }
|
||||
set(k: string, v: V) { this.m.delete(k); this.m.set(k, v); while (this.m.size > this.cap) this.m.delete(this.m.keys().next().value!); }
|
||||
delete(k: string) { this.m.delete(k); }
|
||||
}
|
||||
|
||||
/** Validates `{ anchors }` from a request body; throws HttpError(400). */
|
||||
function parseAnchors(body: unknown): Anchor[] {
|
||||
const list = (body as { anchors?: unknown } | null)?.anchors;
|
||||
if (!Array.isArray(list)) throw new HttpError(400, "body must be { anchors: [...] }");
|
||||
if (list.length > MAX_ANCHORS) throw new HttpError(400, `too many anchors (${list.length} > ${MAX_ANCHORS})`);
|
||||
const out = list.map((a: any): Anchor => {
|
||||
if (!a || typeof a.midiSec !== "number" || typeof a.audioSec !== "number" || (a.kind !== "manual" && a.kind !== "auto")) {
|
||||
throw new HttpError(400, "each anchor needs numeric midiSec, audioSec and kind 'manual' | 'auto'");
|
||||
}
|
||||
return { midiSec: a.midiSec, audioSec: a.audioSec, kind: a.kind };
|
||||
});
|
||||
try { return makeWarp(out).anchors; } catch (e) { throw new HttpError(400, (e as Error).message); }
|
||||
}
|
||||
|
||||
async function readJson(req: Request): Promise<unknown> {
|
||||
try { return await req.json(); } catch { throw new HttpError(400, "invalid JSON body"); }
|
||||
}
|
||||
|
||||
function sameToken(a: string, b: string): boolean {
|
||||
const x = Buffer.from(a), y = Buffer.from(b);
|
||||
return x.length === y.length && timingSafeEqual(x, y);
|
||||
}
|
||||
|
||||
function cookieValue(req: Request, name: string): string | null {
|
||||
for (const part of (req.headers.get("cookie") ?? "").split(";")) {
|
||||
const i = part.indexOf("=");
|
||||
if (i > 0 && part.slice(0, i).trim() === name) return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
export async function startServer(opts: ServeOptions = {}): Promise<AlignServer> {
|
||||
const root = opts.root ?? "reference";
|
||||
const host = opts.host ?? "127.0.0.1";
|
||||
const log = (msg: string) => { if (!opts.quiet) console.log(msg); };
|
||||
const token = Buffer.from(crypto.getRandomValues(new Uint8Array(18))).toString("base64url");
|
||||
const sf2 = new Uint8Array(await Bun.file(opts.sf2Path ?? SF2_PATH).arrayBuffer());
|
||||
|
||||
const songs = new Lru<Promise<SongState>>(SONG_CACHE);
|
||||
const renders = new Lru<Uint8Array<ArrayBuffer>>(RENDER_CACHE);
|
||||
|
||||
async function entryExists(id: string): Promise<boolean> {
|
||||
return (await readManifest(root)).some((s) => s.id === id);
|
||||
}
|
||||
|
||||
async function buildState(id: string): Promise<SongState> {
|
||||
const t0 = performance.now();
|
||||
const s = await loadSong(id, root);
|
||||
const keys = trackKeys(s.smf.tracks);
|
||||
const roles = Object.fromEntries(keys.map((k) => [k, s.entry.roles[k] ?? null]));
|
||||
const cues = recordingCues(s.audio);
|
||||
const peaksFile = Bun.file(join(root, id, "peaks.json"));
|
||||
const recWav = Bun.file(join(root, id, "recording.wav"));
|
||||
let pk: SongState["peaks"] | null = null;
|
||||
if (await peaksFile.exists()) {
|
||||
const j = await peaksFile.json();
|
||||
if (j.version === PEAKS_VERSION && peaksFile.lastModified >= recWav.lastModified) pk = j.peaks;
|
||||
}
|
||||
if (!pk) {
|
||||
const n = Math.max(1, Math.round(s.audioDur * PEAKS_PER_SEC));
|
||||
const p = peaks(toMono(s.audio), n);
|
||||
const r = (v: number) => Math.round(v * 1000) / 1000;
|
||||
pk = { bucketSec: s.audioDur / n, min: Array.from(p.min, r), max: Array.from(p.max, r) };
|
||||
await Bun.write(peaksFile, JSON.stringify({ version: PEAKS_VERSION, peaks: pk }));
|
||||
}
|
||||
log(`loaded ${id} (${s.audioDur.toFixed(1)} s) in ${((performance.now() - t0) / 1000).toFixed(1)} s`);
|
||||
return { ...s, roles, cues, peaks: pk };
|
||||
}
|
||||
|
||||
function song(id: string): Promise<SongState> {
|
||||
let p = songs.get(id);
|
||||
if (!p) {
|
||||
p = buildState(id);
|
||||
p.catch(() => songs.delete(id));
|
||||
songs.set(id, p);
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
const fitOf = (s: SongState, anchors: Anchor[]) => fitFromCues(s.cues, s.smf, s.roles, anchors);
|
||||
const stretchesOf = (s: SongState, anchors: Anchor[]) => stretchInfo(stretches(anchors, s.smf.durationSec), s.smf.tempoMap);
|
||||
const state = (s: SongState, anchors: Anchor[], fit = fitOf(s, anchors)) => ({ anchors, stretches: stretchesOf(s, anchors), fit });
|
||||
|
||||
function runAuto(s: SongState, manual: Anchor[]) {
|
||||
const t0 = performance.now();
|
||||
const r = autoAlign(s.audio, s.smf, s.roles, manual, sf2);
|
||||
log(`auto-aligned ${s.entry.id}: ${r.anchors.length} anchors (${manual.length} manual, ${r.chosen}) in ${((performance.now() - t0) / 1000).toFixed(1)} s`);
|
||||
return r;
|
||||
}
|
||||
|
||||
const autoInfo = (r: AutoResult) => ({ chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision });
|
||||
|
||||
/** Saved alignment with validated anchors; an unreadable/invalid file yields `error` instead of failing the request. */
|
||||
async function savedAlignment(id: string): Promise<{ file: AlignmentFile | null; error?: string }> {
|
||||
try {
|
||||
const f = await readAlignment(id, root);
|
||||
if (!f) return { file: null };
|
||||
return { file: { ...f, anchors: parseAnchors(f) } };
|
||||
} catch (e) {
|
||||
return { file: null, error: `${id}/alignment.json is unreadable (${(e as Error).message})` };
|
||||
}
|
||||
}
|
||||
|
||||
async function api(req: Request, path: string): Promise<Response> {
|
||||
const method = req.method;
|
||||
if (path === "/api/songs" && method === "GET") {
|
||||
const all = await readManifest(root);
|
||||
return Response.json(await Promise.all(all.map(async (e) => {
|
||||
const a = await savedAlignment(e.id);
|
||||
return { id: e.id, title: e.title, hasAlignment: a.file !== null, ...(a.error ? { alignmentError: a.error } : {}) };
|
||||
})));
|
||||
}
|
||||
const m = path.match(/^\/api\/song\/([^/]+)(?:\/(audio|render|fit|auto|alignment|reset))?$/);
|
||||
if (!m) throw new HttpError(404, "not found");
|
||||
let id: string;
|
||||
try { id = decodeURIComponent(m[1]!); } catch { throw new HttpError(404, "not found"); }
|
||||
if (!(await entryExists(id))) throw new HttpError(404, `unknown song: ${id}`);
|
||||
const action = m[2] ?? "", route = `${method} ${action}`;
|
||||
|
||||
if (route === "GET ") {
|
||||
const s = await song(id);
|
||||
const { file: saved, error } = await savedAlignment(id);
|
||||
let anchors: Anchor[], fit: FitReport | undefined;
|
||||
if (saved) anchors = saved.anchors;
|
||||
else { s.auto ??= runAuto(s, []); anchors = s.auto.anchors; fit = s.auto.fit; }
|
||||
return Response.json({
|
||||
id, title: s.entry.title, audioDur: s.audioDur, midiDur: s.smf.durationSec,
|
||||
tracks: trackKeys(s.smf.tracks).map((name) => ({ name, role: s.roles[name] ?? null })),
|
||||
recordingPeaks: s.peaks, ...state(s, anchors, fit), midiTempo: s.smf.tempoMap,
|
||||
source: saved ? "saved" : "auto", savedAt: saved?.savedAt ?? null, ...(saved ? {} : { auto: autoInfo(s.auto!) }),
|
||||
...(error ? { warning: `${error}. Showing the automatic alignment instead; saving will overwrite the file.` } : {}),
|
||||
});
|
||||
}
|
||||
if (route === "GET audio") {
|
||||
const e = (await readManifest(root)).find((x) => x.id === id)!;
|
||||
return new Response(Bun.file(await audioOgg(e, root)), { headers: { "content-type": "audio/ogg", "cache-control": "private, max-age=3600" } });
|
||||
}
|
||||
if (route === "POST render") {
|
||||
const anchors = parseAnchors(await readJson(req));
|
||||
const s = await song(id);
|
||||
const hash = Bun.hash(JSON.stringify(anchors)).toString(16);
|
||||
const key = `${id}:${hash}`;
|
||||
let wav = renders.get(key);
|
||||
const hit = wav !== undefined;
|
||||
if (!wav) {
|
||||
const t0 = performance.now();
|
||||
const warp = makeWarp(anchors);
|
||||
const dur = Math.max(s.audioDur, Math.min(warp.toAudio(s.smf.durationSec) + 1, s.audioDur + 60));
|
||||
wav = encodeWav(renderMidi(s.smf.tracks, warp, sf2, { sampleRate: RENDER_SR, mono: true, durationSec: dur }));
|
||||
renders.set(key, wav);
|
||||
log(`rendered ${id} (${anchors.length} anchors) in ${((performance.now() - t0) / 1000).toFixed(1)} s`);
|
||||
}
|
||||
return new Response(wav, { headers: { "content-type": "audio/wav", "x-anchors-hash": hash, "x-cache": hit ? "hit" : "miss" } });
|
||||
}
|
||||
if (route === "POST fit") {
|
||||
const anchors = parseAnchors(await readJson(req));
|
||||
return Response.json(state(await song(id), anchors));
|
||||
}
|
||||
if (route === "POST auto") {
|
||||
const manual = parseAnchors(await readJson(req)).filter((a) => a.kind === "manual");
|
||||
const s = await song(id);
|
||||
const r = runAuto(s, manual);
|
||||
return Response.json({ ...state(s, r.anchors, r.fit), auto: autoInfo(r) });
|
||||
}
|
||||
if (route === "POST reset") return Response.json(state(await song(id), []));
|
||||
if (route === "PUT alignment") {
|
||||
const anchors = parseAnchors(await readJson(req));
|
||||
const s = await song(id);
|
||||
const file = { anchors, savedAt: new Date().toISOString(), fit: fitOf(s, anchors) };
|
||||
await writeAlignment(id, file, root);
|
||||
log(`saved ${id}: ${anchors.length} anchors`);
|
||||
return Response.json({ savedAt: file.savedAt, fit: file.fit });
|
||||
}
|
||||
throw new HttpError(405, "method not allowed");
|
||||
}
|
||||
|
||||
// Inner server: Bun bundles and serves the page and its assets. It deliberately has no token check: it binds
|
||||
// loopback only on a random port, serves nothing but the static page bundle (no song data), and the public server
|
||||
// below authenticates every request before proxying to it.
|
||||
const inner = Bun.serve({
|
||||
hostname: "127.0.0.1", port: 0, development: false,
|
||||
routes: { "/": index },
|
||||
fetch: () => new Response("not found", { status: 404 }),
|
||||
});
|
||||
|
||||
async function proxyPage(req: Request, path: string): Promise<Response> {
|
||||
if (req.method !== "GET" && req.method !== "HEAD") return new Response("method not allowed", { status: 405 });
|
||||
const r = await fetch(inner.url.origin + path, { method: req.method }); // `path` is "/..." with one leading slash
|
||||
|
||||
const headers = new Headers(r.headers);
|
||||
for (const h of ["content-encoding", "content-length", "transfer-encoding", "connection"]) headers.delete(h);
|
||||
return new Response(r.body, { status: r.status, headers });
|
||||
}
|
||||
|
||||
const outer = Bun.serve({
|
||||
hostname: host, port: opts.port ?? 5180, maxRequestBodySize: MAX_BODY_BYTES,
|
||||
idleTimeout: 255, // first open of a song transcodes and auto-aligns; renders take seconds
|
||||
async fetch(req) {
|
||||
const url = new URL(req.url);
|
||||
// "//host/x" must never become a protocol-relative URL (off-host proxy target or redirect): collapse slashes
|
||||
const path = "/" + url.pathname.replace(/^\/+/, "");
|
||||
const key = url.searchParams.get("key");
|
||||
const viaKey = key !== null && sameToken(key, token);
|
||||
const viaCookie = !viaKey && sameToken(cookieValue(req, COOKIE) ?? "", token);
|
||||
if (!viaKey && !viaCookie) {
|
||||
return new Response("Unauthorized: open the URL printed by `bun run align` (it contains ?key=...).\n", { status: 401, headers: { "content-type": "text/plain; charset=utf-8" } });
|
||||
}
|
||||
const setCookie = `${COOKIE}=${token}; HttpOnly; SameSite=Strict; Path=/`;
|
||||
if (viaKey && req.method === "GET" && !path.startsWith("/api/")) {
|
||||
url.searchParams.delete("key"); // drop the token from the address bar
|
||||
return new Response(null, { status: 302, headers: { location: path + url.search, "set-cookie": setCookie } });
|
||||
}
|
||||
let res: Response;
|
||||
try {
|
||||
res = path.startsWith("/api/") ? await api(req, path) : await proxyPage(req, path);
|
||||
} catch (e) {
|
||||
const status = e instanceof HttpError ? e.status : 500;
|
||||
if (status === 500) console.error(`${req.method} ${url.pathname}:`, e);
|
||||
res = Response.json({ error: (e as Error).message }, { status });
|
||||
}
|
||||
if (viaKey) res.headers.append("set-cookie", setCookie);
|
||||
return res;
|
||||
},
|
||||
});
|
||||
|
||||
const origin = `http://${opts.urlHost ?? (host === "0.0.0.0" || host === "::" ? "localhost" : host)}:${outer.port}`;
|
||||
return {
|
||||
url: `${origin}/?key=${token}`, origin, token, port: outer.port!,
|
||||
async stop() { await outer.stop(true); await inner.stop(true); },
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
// Standard MIDI File parser. Pure: bytes in, seconds-based notes out (no I/O).
|
||||
|
||||
export interface RefNote { onset: number; offset: number; pitch: number; velocity: number; channel: number }
|
||||
export interface RefTrack { index: number; name: string; program: number; isDrums: boolean; notes: RefNote[] }
|
||||
export interface TempoPoint { sec: number; bpm: number }
|
||||
export interface Smf { durationSec: number; tracks: RefTrack[]; tempoMap: TempoPoint[]; timeSigs: { sec: number; num: number; den: number }[] }
|
||||
|
||||
const DEFAULT_US_PER_BEAT = 500000;
|
||||
const DRUM_CHANNEL = 9;
|
||||
|
||||
interface RawNote { onTick: number; offTick: number; pitch: number; velocity: number; channel: number }
|
||||
interface RawTrack { index: number; name: string; program: number; notes: RawNote[]; endTick: number }
|
||||
|
||||
class Reader {
|
||||
p = 0;
|
||||
constructor(readonly b: Uint8Array) {}
|
||||
u8(): number {
|
||||
if (this.p >= this.b.length) throw new Error("SMF: unexpected end of data");
|
||||
return this.b[this.p++]!;
|
||||
}
|
||||
u16(): number { return (this.u8() << 8) | this.u8(); }
|
||||
u32(): number { return ((this.u8() << 24) | (this.u8() << 16) | (this.u8() << 8) | this.u8()) >>> 0; }
|
||||
vlq(): number {
|
||||
let v = 0;
|
||||
for (let i = 0; i < 4; i++) {
|
||||
const c = this.u8();
|
||||
v = (v << 7) | (c & 0x7f);
|
||||
if (!(c & 0x80)) return v;
|
||||
}
|
||||
throw new Error("SMF: malformed variable-length quantity");
|
||||
}
|
||||
skip(n: number): void {
|
||||
if (this.p + n > this.b.length) throw new Error("SMF: unexpected end of data");
|
||||
this.p += n;
|
||||
}
|
||||
tag(): string { return String.fromCharCode(this.u8(), this.u8(), this.u8(), this.u8()); }
|
||||
}
|
||||
|
||||
interface TempoEvent { tick: number; usPerBeat: number }
|
||||
interface SigEvent { tick: number; num: number; den: number }
|
||||
|
||||
function parseTrack(r: Reader, index: number, end: number, tempos: TempoEvent[], sigs: SigEvent[]): RawTrack {
|
||||
const tr: RawTrack = { index, name: "", program: -1, notes: [], endTick: 0 };
|
||||
const open = new Map<number, RawNote[]>(); // key = channel*128+pitch, FIFO queue of sounding notes
|
||||
let tick = 0, running = 0;
|
||||
const close = (ch: number, pitch: number, at: number) => {
|
||||
const q = open.get(ch * 128 + pitch);
|
||||
const n = q?.shift();
|
||||
if (n) n.offTick = at;
|
||||
};
|
||||
while (r.p < end) {
|
||||
tick += r.vlq();
|
||||
let status = r.b[r.p]!;
|
||||
if (status & 0x80) r.p++;
|
||||
else if (running) status = running;
|
||||
else throw new Error("SMF: data byte without running status");
|
||||
if (status === 0xff) {
|
||||
const type = r.u8(), len = r.vlq(), start = r.p;
|
||||
r.skip(len);
|
||||
const d = r.b.subarray(start, start + len);
|
||||
if (type === 0x03 && !tr.name) tr.name = new TextDecoder("latin1").decode(d);
|
||||
else if (type === 0x51 && len === 3) tempos.push({ tick, usPerBeat: (d[0]! << 16) | (d[1]! << 8) | d[2]! });
|
||||
else if (type === 0x58 && len >= 2) sigs.push({ tick, num: d[0]!, den: 2 ** d[1]! });
|
||||
else if (type === 0x2f) break;
|
||||
} else if (status === 0xf0 || status === 0xf7) {
|
||||
r.skip(r.vlq()); // sysex: skipped; cancels running status per spec
|
||||
running = 0;
|
||||
} else if (status >= 0xf1) {
|
||||
running = 0; // other system messages carry no data we use
|
||||
} else {
|
||||
running = status;
|
||||
const hi = status & 0xf0, ch = status & 15;
|
||||
const a = r.u8();
|
||||
const b = hi === 0xc0 || hi === 0xd0 ? 0 : r.u8();
|
||||
if (hi === 0xc0 && tr.program < 0) tr.program = a;
|
||||
else if (hi === 0x90 && b > 0) {
|
||||
const n: RawNote = { onTick: tick, offTick: -1, pitch: a, velocity: b, channel: ch };
|
||||
tr.notes.push(n);
|
||||
const k = ch * 128 + a;
|
||||
const q = open.get(k);
|
||||
if (q) q.push(n); else open.set(k, [n]);
|
||||
} else if (hi === 0x80 || hi === 0x90) close(ch, a, tick);
|
||||
}
|
||||
}
|
||||
r.p = end;
|
||||
tr.endTick = tick;
|
||||
for (const n of tr.notes) if (n.offTick < 0) n.offTick = tick; // unterminated: ends with the track
|
||||
return tr;
|
||||
}
|
||||
|
||||
export function parseSmf(bytes: Uint8Array): Smf {
|
||||
const r = new Reader(bytes);
|
||||
if (r.tag() !== "MThd") throw new Error("SMF: missing MThd header");
|
||||
const hlen = r.u32();
|
||||
r.u16(); // format: 0/1/2 all handled identically (tracks merged on one timeline)
|
||||
const ntrks = r.u16(), div = r.u16();
|
||||
r.skip(Math.max(0, hlen - 6));
|
||||
if (div & 0x8000) throw new Error("SMF: SMPTE time division is not supported");
|
||||
if (div === 0) throw new Error("SMF: zero ticks per quarter note");
|
||||
|
||||
const tempos: TempoEvent[] = [], sigs: SigEvent[] = [], raw: RawTrack[] = [];
|
||||
for (let i = 0; i < ntrks && r.p < bytes.length; i++) {
|
||||
const id = r.tag(), len = r.u32();
|
||||
const end = Math.min(r.p + len, bytes.length);
|
||||
if (id !== "MTrk") { r.p = end; continue; }
|
||||
raw.push(parseTrack(r, i, end, tempos, sigs));
|
||||
}
|
||||
|
||||
// tempo map: sorted by tick, later event wins on equal tick; implicit 120 bpm until the first change
|
||||
tempos.sort((a, b) => a.tick - b.tick);
|
||||
const segs: { tick: number; sec: number; us: number }[] = [{ tick: 0, sec: 0, us: DEFAULT_US_PER_BEAT }];
|
||||
for (const t of tempos) {
|
||||
const last = segs[segs.length - 1]!;
|
||||
if (t.tick === last.tick) { last.us = t.usPerBeat; continue; }
|
||||
segs.push({ tick: t.tick, sec: last.sec + ((t.tick - last.tick) / div) * (last.us / 1e6), us: t.usPerBeat });
|
||||
}
|
||||
const toSec = (tick: number): number => {
|
||||
let lo = 0, hi = segs.length - 1;
|
||||
while (lo < hi) { const m = (lo + hi + 1) >> 1; if (segs[m]!.tick <= tick) lo = m; else hi = m - 1; }
|
||||
const s = segs[lo]!;
|
||||
return s.sec + ((tick - s.tick) / div) * (s.us / 1e6);
|
||||
};
|
||||
|
||||
let durationSec = 0;
|
||||
const tracks: RefTrack[] = [];
|
||||
for (const t of raw) {
|
||||
durationSec = Math.max(durationSec, toSec(t.endTick));
|
||||
if (!t.notes.length) continue; // conductor / meta-only tracks
|
||||
const notes: RefNote[] = t.notes.map((n) => ({ onset: toSec(n.onTick), offset: toSec(n.offTick), pitch: n.pitch, velocity: n.velocity, channel: n.channel }));
|
||||
notes.sort((a, b) => a.onset - b.onset || a.pitch - b.pitch);
|
||||
for (const n of notes) durationSec = Math.max(durationSec, n.offset);
|
||||
tracks.push({ index: t.index, name: t.name, program: t.program, isDrums: notes.some((n) => n.channel === DRUM_CHANNEL), notes });
|
||||
}
|
||||
|
||||
sigs.sort((a, b) => a.tick - b.tick);
|
||||
return {
|
||||
durationSec,
|
||||
tracks,
|
||||
tempoMap: segs.map((s) => ({ sec: s.sec, bpm: 60e6 / s.us })),
|
||||
timeSigs: sigs.map((s) => ({ sec: toSec(s.tick), num: s.num, den: s.den })),
|
||||
};
|
||||
}
|
||||
|
||||
export interface SmfTrackIn { name: string; program: number; isDrums: boolean; notes: { onset: number; offset: number; pitch: number; velocity?: number }[] }
|
||||
|
||||
/**
|
||||
* Write a format-1 SMF at a single fixed tempo (PPQ 960): a conductor track plus one track per input track.
|
||||
* Melodic tracks take channels 0..15 skipping 9; drums use channel 9. Inverse of parseSmf up to tick rounding.
|
||||
* Intended for tests and fixtures.
|
||||
*/
|
||||
export function writeSmf(tracks: SmfTrackIn[], tempoBpm = 120): Uint8Array {
|
||||
const PPQ = 960;
|
||||
const us = Math.round(60e6 / tempoBpm);
|
||||
const tickOf = (sec: number) => Math.max(0, Math.round((sec * 1e6 * PPQ) / us));
|
||||
const vlq = (v: number): number[] => { const o = [v & 0x7f]; while ((v >>>= 7) > 0) o.unshift((v & 0x7f) | 0x80); return o; };
|
||||
const u32 = (v: number) => [(v >>> 24) & 255, (v >>> 16) & 255, (v >>> 8) & 255, v & 255];
|
||||
const text = (s: string) => [...new TextEncoder().encode(s)];
|
||||
const chunk = (id: string, body: number[]) => [...text(id), ...u32(body.length), ...body];
|
||||
const conductor = [0, 0xff, 0x51, 3, (us >> 16) & 255, (us >> 8) & 255, us & 255, 0, 0xff, 0x58, 4, 4, 2, 24, 8, 0, 0xff, 0x2f, 0];
|
||||
const melodic = [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15];
|
||||
let next = 0;
|
||||
const bodies = tracks.map((t) => {
|
||||
const ch = t.isDrums ? 9 : melodic[next++ % melodic.length]!;
|
||||
// offs sort before ons at equal ticks so a repeated pitch re-triggers cleanly
|
||||
const ev: { tick: number; off: boolean; bytes: number[] }[] = [];
|
||||
for (const n of t.notes) {
|
||||
const on = tickOf(n.onset), off = Math.max(on + 1, tickOf(n.offset));
|
||||
ev.push({ tick: on, off: false, bytes: [0x90 | ch, n.pitch & 127, Math.max(1, Math.min(127, Math.round(n.velocity ?? 96)))] });
|
||||
ev.push({ tick: off, off: true, bytes: [0x80 | ch, n.pitch & 127, 0] });
|
||||
}
|
||||
ev.sort((a, b) => a.tick - b.tick || Number(b.off) - Number(a.off));
|
||||
const name = text(t.name);
|
||||
const out = [0, 0xff, 0x03, ...vlq(name.length), ...name, 0, 0xc0 | ch, Math.max(0, t.program) & 127];
|
||||
let at = 0;
|
||||
for (const e of ev) { out.push(...vlq(e.tick - at), ...e.bytes); at = e.tick; }
|
||||
out.push(0, 0xff, 0x2f, 0);
|
||||
return out;
|
||||
});
|
||||
const header = chunk("MThd", [0, 1, (tracks.length + 1) >> 8, (tracks.length + 1) & 255, PPQ >> 8, PPQ & 255]);
|
||||
return new Uint8Array([...header, ...chunk("MTrk", conductor), ...bodies.flatMap((b) => chunk("MTrk", b))]);
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
// Tempo of alignment stretches: the MIDI's own tempo vs the tempo after warping. Pure (shared by server and page).
|
||||
import type { TempoPoint } from "./smf.ts";
|
||||
import type { Stretch } from "./warp.ts";
|
||||
|
||||
export interface StretchInfo extends Stretch { midiBpm: number; bpm: number; pct: number }
|
||||
|
||||
/** Average MIDI tempo over [from, to] (MIDI seconds) = beats / minutes; 120 bpm before the first tempo point. */
|
||||
export function avgBpm(tempoMap: TempoPoint[], from: number, to: number): number {
|
||||
const pts = [...tempoMap].sort((a, b) => a.sec - b.sec);
|
||||
const bpmAt = (t: number) => { let b = 120; for (const p of pts) if (p.sec <= t) b = p.bpm; else break; return b; };
|
||||
if (!(to > from)) return bpmAt(from);
|
||||
let beats = 0, t = from;
|
||||
for (const p of pts) {
|
||||
if (p.sec <= t) continue;
|
||||
if (p.sec >= to) break;
|
||||
beats += ((p.sec - t) * bpmAt(t)) / 60;
|
||||
t = p.sec;
|
||||
}
|
||||
beats += ((to - t) * bpmAt(t)) / 60;
|
||||
return (beats * 60) / (to - from);
|
||||
}
|
||||
|
||||
/** Adds the MIDI tempo, the effective (warped) tempo = MIDI bpm / ratio, and the change in percent. */
|
||||
export function stretchInfo(stretches: Stretch[], tempoMap: TempoPoint[]): StretchInfo[] {
|
||||
return stretches.map((s) => {
|
||||
const midiBpm = avgBpm(tempoMap, s.fromMidi, s.toMidi);
|
||||
return { ...s, midiBpm, bpm: midiBpm / s.ratio, pct: (1 / s.ratio - 1) * 100 };
|
||||
});
|
||||
}
|
||||
|
||||
/** "78.0 → 79.1 bpm (+1.4%)", with " ⚠" when the stretch is warned; the percent is omitted below 0.05%. */
|
||||
export function stretchLabel(s: StretchInfo): string {
|
||||
const pct = Math.abs(s.pct) < 0.05 ? "" : ` (${s.pct > 0 ? "+" : "−"}${Math.abs(s.pct).toFixed(1)}%)`;
|
||||
return `${s.midiBpm.toFixed(1)} → ${s.bpm.toFixed(1)} bpm${pct}${s.warn ? " ⚠" : ""}`;
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
// Anchor model for MIDI-time -> recording-time alignment. Pure: no I/O, no Bun/process/fs.
|
||||
|
||||
export type AnchorKind = "manual" | "auto";
|
||||
export interface Anchor { midiSec: number; audioSec: number; kind: AnchorKind }
|
||||
export interface Warp { toAudio(midiSec: number): number; toMidi(audioSec: number): number; anchors: Anchor[] }
|
||||
// ratio = audio duration / midi duration of the stretch; warn when the tempo change exceeds 5%.
|
||||
export interface Stretch { fromMidi: number; toMidi: number; ratio: number; warn: boolean }
|
||||
|
||||
const WARN_TOL = 0.05;
|
||||
|
||||
function sortedCopy(anchors: Anchor[]): Anchor[] {
|
||||
const s = anchors.map((a) => ({ midiSec: a.midiSec, audioSec: a.audioSec, kind: a.kind })).sort((x, y) => x.midiSec - y.midiSec);
|
||||
for (const a of s) {
|
||||
if (!Number.isFinite(a.midiSec) || !Number.isFinite(a.audioSec)) throw new Error("anchors: times must be finite");
|
||||
}
|
||||
for (let i = 1; i < s.length; i++) {
|
||||
const p = s[i - 1]!, c = s[i]!;
|
||||
if (!(c.midiSec > p.midiSec)) throw new Error(`anchors: midiSec must be strictly increasing (${p.midiSec}, ${c.midiSec})`);
|
||||
if (!(c.audioSec > p.audioSec)) throw new Error(`anchors: audioSec must be strictly increasing (${p.audioSec}, ${c.audioSec})`);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
const ratioOf = (a: Anchor, b: Anchor) => (b.audioSec - a.audioSec) / (b.midiSec - a.midiSec);
|
||||
|
||||
// Slopes used beyond the outermost anchors: the nearest interior stretch, or 1 with fewer than two anchors. So
|
||||
// before the first anchor the map extrapolates with the *first* stretch's slope (there is no implicit 0 -> 0 anchor:
|
||||
// a warp whose first anchor is (m0, a0) maps MIDI 0 to a0 − m0·slope₀), and after the last anchor with the last one's.
|
||||
function edgeRatios(A: Anchor[]): { first: number; last: number } {
|
||||
if (A.length < 2) return { first: 1, last: 1 };
|
||||
return { first: ratioOf(A[0]!, A[1]!), last: ratioOf(A[A.length - 2]!, A[A.length - 1]!) };
|
||||
}
|
||||
|
||||
// Piecewise-linear map over the sorted keys X -> Y; extrapolates with `first` / `last` slopes.
|
||||
function piecewise(X: number[], Y: number[], x: number, first: number, last: number): number {
|
||||
const n = X.length;
|
||||
if (n === 0) return x;
|
||||
if (x <= X[0]!) return Y[0]! + (x - X[0]!) * first;
|
||||
if (x >= X[n - 1]!) return Y[n - 1]! + (x - X[n - 1]!) * last;
|
||||
let lo = 0, hi = n - 1;
|
||||
while (hi - lo > 1) {
|
||||
const mid = (lo + hi) >> 1;
|
||||
if (X[mid]! <= x) lo = mid; else hi = mid;
|
||||
}
|
||||
return Y[lo]! + ((x - X[lo]!) * (Y[hi]! - Y[lo]!)) / (X[hi]! - X[lo]!);
|
||||
}
|
||||
|
||||
/**
|
||||
* Monotonic piecewise-linear MIDI <-> recording map through the anchors. Outside the anchors it extrapolates linearly:
|
||||
* before the first anchor with the first stretch's slope, after the last anchor with the last stretch's slope (slope
|
||||
* 1 with a single anchor, identity with none).
|
||||
*/
|
||||
export function makeWarp(anchors: Anchor[]): Warp {
|
||||
const A = sortedCopy(anchors);
|
||||
const midi = A.map((a) => a.midiSec), audio = A.map((a) => a.audioSec);
|
||||
const { first, last } = edgeRatios(A);
|
||||
return {
|
||||
toAudio: (x) => piecewise(midi, audio, x, first, last),
|
||||
toMidi: (y) => piecewise(audio, midi, y, 1 / first, 1 / last),
|
||||
anchors: A.map((a) => ({ ...a })),
|
||||
};
|
||||
}
|
||||
|
||||
export function stretches(anchors: Anchor[], midiDur: number): Stretch[] {
|
||||
const A = sortedCopy(anchors);
|
||||
const mk = (fromMidi: number, toMidi: number, ratio: number): Stretch => ({
|
||||
fromMidi, toMidi, ratio, warn: Math.abs(1 / ratio - 1) > WARN_TOL,
|
||||
});
|
||||
if (A.length === 0) return [mk(0, midiDur, 1)];
|
||||
const { first, last } = edgeRatios(A);
|
||||
const segs: Stretch[] = [mk(0, A[0]!.midiSec, first)];
|
||||
for (let k = 0; k + 1 < A.length; k++) {
|
||||
segs.push(mk(A[k]!.midiSec, A[k + 1]!.midiSec, ratioOf(A[k]!, A[k + 1]!)));
|
||||
}
|
||||
segs.push(mk(A[A.length - 1]!.midiSec, midiDur, last));
|
||||
return segs.filter((s) => s.toMidi > s.fromMidi);
|
||||
}
|
||||
|
||||
// Throws unless `a` fits strictly between the given neighbours in both midi and audio time.
|
||||
function checkSlot(left: Anchor | undefined, right: Anchor | undefined, a: Omit<Anchor, "kind">): void {
|
||||
if (left && !(a.midiSec > left.midiSec && a.audioSec > left.audioSec)) throw new Error("anchor would cross its left neighbour");
|
||||
if (right && !(a.midiSec < right.midiSec && a.audioSec < right.audioSec)) throw new Error("anchor would cross its right neighbour");
|
||||
}
|
||||
|
||||
/** Autos within this many MIDI seconds of an edited anchor (new position, and old position for a move) are removed. */
|
||||
export const AUTO_CLEAR_SEC = 10;
|
||||
|
||||
// Inserts a manual anchor at index `at` of an already-sorted list. Removes the auto anchors within AUTO_CLEAR_SEC (MIDI
|
||||
// time) of the new position, and of `oldMidi` for a move, but never past the nearest manual anchor on either side
|
||||
// (manual neighbours around `at`). Then removes any remaining auto whose audio time no longer lies strictly between the
|
||||
// manual anchors around it, so the result stays strictly monotonic. Crossing checks apply only against the manual
|
||||
// neighbours.
|
||||
function insertManual(list: Anchor[], at: number, a: Omit<Anchor, "kind">, oldMidi?: number): Anchor[] {
|
||||
let l = at - 1;
|
||||
while (l >= 0 && list[l]!.kind === "auto") l--;
|
||||
let r = at;
|
||||
while (r < list.length && list[r]!.kind === "auto") r++;
|
||||
checkSlot(list[l], list[r], a);
|
||||
const near = (m: number) => Math.abs(m - a.midiSec) <= AUTO_CLEAR_SEC || (oldMidi !== undefined && Math.abs(m - oldMidi) <= AUTO_CLEAR_SEC);
|
||||
const kept = list.filter((x, k) => !(x.kind === "auto" && k > l && k < r && (near(x.midiSec) || x.midiSec === a.midiSec)));
|
||||
const all = [...kept, { midiSec: a.midiSec, audioSec: a.audioSec, kind: "manual" as const }].sort((x, y) => x.midiSec - y.midiSec);
|
||||
// audio time of the nearest manual before / after each position
|
||||
const before: number[] = [], after: number[] = [];
|
||||
let lo = -Infinity;
|
||||
for (let k = 0; k < all.length; k++) { before[k] = lo; if (all[k]!.kind === "manual") lo = all[k]!.audioSec; }
|
||||
let hi = Infinity;
|
||||
for (let k = all.length - 1; k >= 0; k--) { after[k] = hi; if (all[k]!.kind === "manual") hi = all[k]!.audioSec; }
|
||||
return sortedCopy(all.filter((x, k) => x.kind === "manual" || (x.audioSec > before[k]! && x.audioSec < after[k]!)));
|
||||
}
|
||||
|
||||
export function addManual(anchors: Anchor[], a: Omit<Anchor, "kind">): Anchor[] {
|
||||
const A = sortedCopy(anchors);
|
||||
const idx = A.findIndex((x) => x.midiSec > a.midiSec);
|
||||
return insertManual(A, idx < 0 ? A.length : idx, a);
|
||||
}
|
||||
|
||||
export function moveAnchor(anchors: Anchor[], i: number, to: Omit<Anchor, "kind">): Anchor[] {
|
||||
const A = sortedCopy(anchors);
|
||||
if (!Number.isInteger(i) || i < 0 || i >= A.length) throw new RangeError(`moveAnchor: index ${i} out of range`);
|
||||
const rest = [...A.slice(0, i), ...A.slice(i + 1)];
|
||||
return insertManual(rest, i, to, A[i]!.midiSec);
|
||||
}
|
||||
|
||||
export function deleteAnchor(anchors: Anchor[], i: number): Anchor[] {
|
||||
const A = sortedCopy(anchors);
|
||||
if (!Number.isInteger(i) || i < 0 || i >= A.length) throw new RangeError(`deleteAnchor: index ${i} out of range`);
|
||||
return A.filter((_, k) => k !== i);
|
||||
}
|
||||
+36
-22
@@ -5,7 +5,7 @@ import { songToScore } from "../score/gp.ts";
|
||||
const TAIL_SECONDS = 2;
|
||||
|
||||
/** Offline-render sink: AlphaSynth requires an ISynthOutput but exportAudio never uses it. */
|
||||
function dummyOutput(sampleRate: number): at.synth.ISynthOutput {
|
||||
export function dummyOutput(sampleRate: number): at.synth.ISynthOutput {
|
||||
const ev = { on() {}, off() {} };
|
||||
return {
|
||||
sampleRate, open() {}, play() {}, destroy() {}, pause() {}, addSamples() {}, resetSamples() {}, activate() {},
|
||||
@@ -18,6 +18,39 @@ function songSeconds(song: Song): number {
|
||||
return (song.bars * song.timeSig[0] * (4 / song.timeSig[1]) * 60) / song.bpm;
|
||||
}
|
||||
|
||||
/** Render a MidiFile offline. With `frames`, output is exactly that long (zero-padded or truncated); else full length. */
|
||||
export function renderMidiFile(
|
||||
midi: at.midi.MidiFile, sf2: Uint8Array, sampleRate: number,
|
||||
opts: { trackVolume?: Map<number, number>; frames?: number } = {},
|
||||
): Audio {
|
||||
const synth = new at.synth.AlphaSynth(dummyOutput(sampleRate), 500);
|
||||
const eo = new at.synth.AudioExportOptions();
|
||||
eo.soundFonts = [sf2];
|
||||
eo.sampleRate = sampleRate;
|
||||
eo.useSyncPoints = false;
|
||||
if (opts.trackVolume) eo.trackVolume = opts.trackVolume;
|
||||
const exporter = synth.exportAudio(eo, midi, [], new Map());
|
||||
const cap = opts.frames ?? Infinity;
|
||||
const Ls: Float32Array[] = [], Rs: Float32Array[] = [];
|
||||
let pos = 0;
|
||||
while (pos < cap) {
|
||||
const chunk = exporter.render(500);
|
||||
if (!chunk) break;
|
||||
const s = chunk.samples;
|
||||
const frames = Math.min(s.length >> 1, cap - pos);
|
||||
const l = new Float32Array(frames), r = new Float32Array(frames);
|
||||
for (let i = 0; i < frames; i++) { l[i] = s[2 * i]!; r[i] = s[2 * i + 1]!; }
|
||||
Ls.push(l); Rs.push(r);
|
||||
pos += frames;
|
||||
}
|
||||
synth.destroy();
|
||||
const n = opts.frames ?? pos;
|
||||
const L = new Float32Array(n), R = new Float32Array(n);
|
||||
let o = 0;
|
||||
for (let k = 0; k < Ls.length; k++) { L.set(Ls[k]!, o); R.set(Rs[k]!, o); o += Ls[k]!.length; }
|
||||
return { sampleRate, channels: [L, R] };
|
||||
}
|
||||
|
||||
/**
|
||||
* Render each track in isolation with alphaSynth. Isolation: `AudioExportOptions.trackVolume` is keyed by MIDI
|
||||
* channel (despite the doc's "track index"), so every channel not owned by the target track is muted (volume 0).
|
||||
@@ -36,27 +69,8 @@ export function renderStems(song: Song, sf2: Uint8Array, sampleRate = 44100): Ma
|
||||
const trackVolume = new Map<number, number>();
|
||||
for (let ch = 0; ch < 16; ch++) if (!own.has(ch)) trackVolume.set(ch, 0);
|
||||
|
||||
const synth = new at.synth.AlphaSynth(dummyOutput(sampleRate), 500);
|
||||
const opts = new at.synth.AudioExportOptions();
|
||||
opts.soundFonts = [sf2];
|
||||
opts.sampleRate = sampleRate;
|
||||
opts.useSyncPoints = false;
|
||||
opts.trackVolume = trackVolume;
|
||||
const exporter = synth.exportAudio(opts, midi, [], new Map());
|
||||
|
||||
const L = new Float32Array(total), R = new Float32Array(total);
|
||||
let pos = 0;
|
||||
for (;;) {
|
||||
const chunk = exporter.render(500);
|
||||
if (!chunk) break;
|
||||
const s = chunk.samples;
|
||||
const frames = Math.min(s.length >> 1, total - pos);
|
||||
for (let i = 0; i < frames; i++) { L[pos + i] = s[2 * i]!; R[pos + i] = s[2 * i + 1]!; }
|
||||
pos += frames;
|
||||
if (pos >= total) break;
|
||||
}
|
||||
synth.destroy();
|
||||
stems.set(track.name, { sampleRate, channels: [L, R] });
|
||||
const audio = renderMidiFile(midi, sf2, sampleRate, { trackVolume, frames: total });
|
||||
stems.set(track.name, audio);
|
||||
}
|
||||
return stems;
|
||||
}
|
||||
|
||||
+6
-1
@@ -45,6 +45,11 @@ export async function readWav(path: string): Promise<Audio> {
|
||||
|
||||
/** Write 16-bit PCM WAV (clipped to [-1, 1]). */
|
||||
export async function writeWav(path: string, a: Audio): Promise<void> {
|
||||
await Bun.write(path, encodeWav(a));
|
||||
}
|
||||
|
||||
/** Encode 16-bit PCM WAV bytes (clipped to [-1, 1]). */
|
||||
export function encodeWav(a: Audio): Uint8Array<ArrayBuffer> {
|
||||
const nch = a.channels.length;
|
||||
const frames = a.channels[0]?.length ?? 0;
|
||||
const dataLen = frames * nch * 2;
|
||||
@@ -64,7 +69,7 @@ export async function writeWav(path: string, a: Audio): Promise<void> {
|
||||
o += 2;
|
||||
}
|
||||
}
|
||||
await Bun.write(path, out);
|
||||
return out;
|
||||
}
|
||||
|
||||
export { toMono } from "../dsp/mono.ts";
|
||||
|
||||
+9
-2
@@ -1,4 +1,5 @@
|
||||
import { runBench } from "./runner.ts";
|
||||
import { runReal } from "./real.ts";
|
||||
import { buildSong } from "./dataset.ts";
|
||||
import type { TierId } from "../gen/tiers.ts";
|
||||
import { TIERS } from "../gen/tiers.ts";
|
||||
@@ -43,12 +44,18 @@ async function main() {
|
||||
const seeds = parseSeeds(arg(rest, "seeds", "1-5"));
|
||||
const modes = arg(rest, "modes", "full,oracle-stems,oracle-tempo,oracle-notes").split(",");
|
||||
const runId = arg(rest, "run-id", new Date().toISOString().replace(/[:.]/g, "-"));
|
||||
await runBench({ tiers, seeds, modes, runId, root: arg(rest, "root", "data"), out: arg(rest, "out", "runs") });
|
||||
await runBench({ tiers, seeds, modes, runId, root: arg(rest, "root", "data"), out: arg(rest, "out", "runs"), repeatVote: !rest.includes("--no-repeat-vote") });
|
||||
console.log(`results/${runId}.md`);
|
||||
break;
|
||||
}
|
||||
case "real": {
|
||||
const runId = arg(rest, "run-id", new Date().toISOString().replace(/[:.]/g, "-"));
|
||||
await runReal({ runId, auto: rest.includes("--auto"), root: arg(rest, "root", "reference"), out: arg(rest, "out", "runs"), repeatVote: !rest.includes("--no-repeat-vote") });
|
||||
console.log(`results/real-${runId}.md`);
|
||||
break;
|
||||
}
|
||||
default:
|
||||
console.error("usage: bun run bench gen [--tiers t1a,t1b] [--seeds 1-5] [--root data]\n bun run bench run [--tiers ..] [--seeds ..] [--modes full,oracle-stems,oracle-tempo,oracle-notes] [--run-id id] [--root data] [--out runs]");
|
||||
console.error("usage: bun run bench gen [--tiers t1a,t1b] [--seeds 1-5] [--root data]\n bun run bench run [--tiers ..] [--seeds ..] [--modes full,oracle-stems,oracle-tempo,oracle-notes] [--run-id id] [--root data] [--out runs] [--no-repeat-vote]\n bun run bench real [--auto] [--run-id id] [--root reference] [--out runs] [--no-repeat-vote]");
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
// Real-song benchmark: full pipeline on a team recording, scored per role against the GP-exported MIDI warped through the saved alignment.
|
||||
import { join } from "node:path";
|
||||
import { mkdir } from "node:fs/promises";
|
||||
import type { Role, TimedNote } from "../core/types.ts";
|
||||
import { createPipeline } from "../pipeline/run.ts";
|
||||
import { matchNotes, type MatchResult } from "../eval/notes.ts";
|
||||
import { DRUM_CLASSES, ROLE_ORDER, drumClass, toClassPitch, type DrumClass } from "../eval/score.ts";
|
||||
import { readManifest, readAlignment, loadSong, trackKeys } from "../ref/manifest.ts";
|
||||
import { makeWarp, type Anchor } from "../align/warp.ts";
|
||||
import { autoAlign, autoSummary, constantOffsetBaseline, fitFromCues, recordingCues, type Baseline } from "../align/auto.ts";
|
||||
import { fitSummary, type FitReport } from "../align/fit.ts";
|
||||
|
||||
export interface RealSongResult {
|
||||
id: string; title: string;
|
||||
/**
|
||||
* source: "saved" = reference/<id>/alignment.json, "computed" = auto-aligned this run (--auto). chosen: how the auto
|
||||
* alignment was made ("dtw" or the constant-offset "baseline"), null when unknown (saved from the page).
|
||||
* baseline: the constant-offset baseline computed this run, for comparison.
|
||||
*/
|
||||
alignment: {
|
||||
source: "saved" | "computed"; manual: number; auto: number; savedAt: string | null;
|
||||
chosen: "baseline" | "dtw" | null; baseline: Baseline | { offsetSec: number; precision: number | null }; dtwPrecision: number | null;
|
||||
};
|
||||
fit: FitReport;
|
||||
perRole: Partial<Record<Role, MatchResult>>;
|
||||
/** F1 at the lenient +-100 ms tolerance, per role (null when the role has no notes on either side). */
|
||||
lenient: Partial<Record<Role, number | null>>;
|
||||
drumsPerClass: Record<DrumClass, MatchResult> | null;
|
||||
excludedTracks: string[];
|
||||
/** Capo per track from the manifest; informational only (not applied to scoring). */
|
||||
capo: Record<string, number>;
|
||||
seconds: number;
|
||||
}
|
||||
|
||||
const MODELS = { demucs: "models/demucs/htdemucs_6s.onnx", basicPitch: "models/basic-pitch/nmp.onnx" };
|
||||
const SF2 = join(import.meta.dir, "../../node_modules/@coderline/alphatab/dist/soundfont/sonivox.sf2");
|
||||
const TOL = 0.05, LENIENT_TOL = 0.1;
|
||||
|
||||
export async function runReal(opts: { runId: string; auto: boolean; root?: string; out?: string; resultsDir?: string; repeatVote?: boolean }): Promise<RealSongResult[]> {
|
||||
const root = opts.root ?? "reference", out = opts.out ?? "runs", rdir = opts.resultsDir ?? "results";
|
||||
const songs = await readManifest(root);
|
||||
const pipeline = await createPipeline({
|
||||
demucs: new Uint8Array(await Bun.file(MODELS.demucs).arrayBuffer()),
|
||||
basicPitch: new Uint8Array(await Bun.file(MODELS.basicPitch).arrayBuffer()),
|
||||
}, { repeatVote: opts.repeatVote ?? true });
|
||||
const sf2 = opts.auto ? new Uint8Array(await Bun.file(SF2).arrayBuffer()) : null;
|
||||
const results: RealSongResult[] = [];
|
||||
const skipped: SkippedSong[] = [];
|
||||
const skip = (id: string, reason: string) => { skipped.push({ id, reason }); console.log(`${id}: skipped (${reason})`); };
|
||||
for (const entry of songs) {
|
||||
let saved: Awaited<ReturnType<typeof readAlignment>>;
|
||||
try {
|
||||
saved = await readAlignment(entry.id, root);
|
||||
if (saved) makeWarp(saved.anchors); // validates the anchors
|
||||
} catch (e) { skip(entry.id, (e as Error).message); continue; }
|
||||
if (!saved && !opts.auto) { skip(entry.id, "no alignment.json; pass --auto to compute one"); continue; }
|
||||
const t0 = performance.now();
|
||||
const { smf, audio } = await loadSong(entry.id, root);
|
||||
const keys = trackKeys(smf.tracks);
|
||||
const roleOf = (i: number): Role | null => entry.roles[keys[i]!] ?? null;
|
||||
const roles = Object.fromEntries(keys.map((k) => [k, entry.roles[k] ?? null]));
|
||||
|
||||
const cues = recordingCues(audio);
|
||||
let anchors: Anchor[], alignment: RealSongResult["alignment"];
|
||||
if (saved) {
|
||||
anchors = saved.anchors;
|
||||
alignment = {
|
||||
source: "saved", manual: 0, auto: 0, savedAt: saved.savedAt ?? null,
|
||||
chosen: saved.auto?.chosen ?? null, baseline: constantOffsetBaseline(cues, smf, roles), dtwPrecision: saved.auto?.dtwPrecision ?? null,
|
||||
};
|
||||
} else {
|
||||
const r = autoAlign(audio, smf, roles, [], sf2!);
|
||||
anchors = r.anchors;
|
||||
alignment = { source: "computed", manual: 0, auto: 0, savedAt: null, chosen: r.chosen, baseline: r.baseline, dtwPrecision: r.dtwPrecision };
|
||||
}
|
||||
alignment.manual = anchors.filter((a) => a.kind === "manual").length;
|
||||
alignment.auto = anchors.filter((a) => a.kind === "auto").length;
|
||||
const fit = fitFromCues(cues, smf, roles, anchors);
|
||||
const warp = makeWarp(anchors);
|
||||
|
||||
const res = await pipeline({ mix: audio }, { oracleStems: false, oracleTempo: false, oracleNotes: false });
|
||||
const refOf = (role: Role): TimedNote[] => smf.tracks.flatMap((t, i) => roleOf(i) === role
|
||||
? t.notes.map((n) => ({ onset: warp.toAudio(n.onset), offset: warp.toAudio(n.offset), pitch: n.pitch })) : []);
|
||||
const estOf = (role: Role): TimedNote[] => res.output.tracks.filter((t) => t.role === role).flatMap((t) => t.notes);
|
||||
|
||||
const perRole: RealSongResult["perRole"] = {}, lenient: RealSongResult["lenient"] = {};
|
||||
let drumsPerClass: RealSongResult["drumsPerClass"] = null;
|
||||
for (const role of ROLE_ORDER) {
|
||||
let r = refOf(role), e = estOf(role);
|
||||
if (role === "drums") {
|
||||
const rawR = r, rawE = e;
|
||||
r = r.map(toClassPitch); e = e.map(toClassPitch);
|
||||
if (rawR.length + rawE.length > 0) {
|
||||
drumsPerClass = {} as Record<DrumClass, MatchResult>;
|
||||
for (const c of Object.keys(DRUM_CLASSES) as DrumClass[]) {
|
||||
drumsPerClass[c] = matchNotes(rawR.filter((n) => drumClass(n.pitch) === c).map(toClassPitch), rawE.filter((n) => drumClass(n.pitch) === c).map(toClassPitch), { onsetTol: TOL });
|
||||
}
|
||||
}
|
||||
}
|
||||
perRole[role] = matchNotes(r, e, { onsetTol: TOL });
|
||||
lenient[role] = matchNotes(r, e, { onsetTol: LENIENT_TOL }).f1;
|
||||
}
|
||||
const seconds = (performance.now() - t0) / 1000;
|
||||
const result: RealSongResult = {
|
||||
id: entry.id, title: entry.title,
|
||||
alignment,
|
||||
fit, perRole, lenient, drumsPerClass,
|
||||
excludedTracks: smf.tracks.map((t, i) => (roleOf(i) ? null : keys[i]!)).filter((k): k is string => k !== null),
|
||||
capo: entry.capo ?? {}, seconds,
|
||||
};
|
||||
const d = join(out, `real-${opts.runId}`, entry.id);
|
||||
await mkdir(d, { recursive: true });
|
||||
await Bun.write(join(d, "notes.json"), JSON.stringify({ grid: res.output.grid, tracks: res.output.tracks, warnings: res.warnings, anchors, seconds }));
|
||||
await Bun.write(join(d, "out.gp"), res.gp);
|
||||
await Bun.write(join(d, "score.json"), JSON.stringify(result, null, 1));
|
||||
results.push(result);
|
||||
console.log(`${entry.id}: guitar F1 ${fmt(perRole.guitar?.f1)} (${seconds.toFixed(1)}s)`);
|
||||
}
|
||||
await mkdir(rdir, { recursive: true });
|
||||
await Bun.write(join(rdir, `real-${opts.runId}.json`), JSON.stringify(results, (k, v) => (k === "matches" ? undefined : v), 1));
|
||||
await Bun.write(join(rdir, `real-${opts.runId}.md`), renderRealReport(opts.runId, results, skipped));
|
||||
return results;
|
||||
}
|
||||
|
||||
const NA = "n/a";
|
||||
const fmt = (x: number | null | undefined) => (x == null ? NA : x.toFixed(2));
|
||||
const row = (cells: string[]) => `| ${cells.join(" | ")} |`;
|
||||
|
||||
export interface SkippedSong { id: string; reason: string }
|
||||
|
||||
export function renderRealReport(runId: string, results: RealSongResult[], skipped: SkippedSong[] = []): string {
|
||||
const o: string[] = [`# STSheets real-song benchmark: ${runId}`, ""];
|
||||
o.push("> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). " +
|
||||
"The fit line reports the *consistency of the alignment*, not an independent validation of it: precision = share of the recording's onset-flux peaks with a warped MIDI onset within ±70 ms " +
|
||||
"(flag < 60% per 30 s window), lift = precision minus its value with the MIDI shifted off-grid (flag < 0.25), a symbolic-chroma check for whole-beat/bar shifts, " +
|
||||
"and \"MIDI onsets audible as attacks\" (information only; a full mix exposes far fewer attacks than the score has notes). " +
|
||||
"A poor alignment lowers every score below. Capo is informational and not applied.", "");
|
||||
for (const r of results) {
|
||||
const a = r.alignment;
|
||||
const counts = `${a.manual} manual / ${a.auto} auto anchors`;
|
||||
const src = a.source === "computed" ? `computed this run (${counts})` : `saved ${a.savedAt ?? "(time unknown)"} (${counts})`;
|
||||
const pc = (x: number | null) => (x === null ? NA : `${(x * 100).toFixed(1)}%`);
|
||||
const off = `${a.baseline.offsetSec >= 0 ? "+" : "−"}${Math.abs(a.baseline.offsetSec).toFixed(2)} s`;
|
||||
const method = a.chosen ? autoSummary({ chosen: a.chosen, baseline: a.baseline, dtwPrecision: a.dtwPrecision })
|
||||
: `method unknown (saved from the page) · constant-offset baseline ${off}: precision ${pc(a.baseline.precision)}`;
|
||||
o.push(`## ${r.title} (${r.id})`, "");
|
||||
o.push(`- Alignment: ${src}`, `- Method: ${method}`, `- Consistency of the alignment: ${fitSummary(r.fit)}`);
|
||||
const capos = Object.entries(r.capo);
|
||||
o.push(`- Capo: ${capos.length ? capos.map(([k, v]) => `${k}: ${v}`).join(", ") : "none listed"}`);
|
||||
o.push(`- Excluded tracks (no role): ${r.excludedTracks.length ? r.excludedTracks.join(", ") : "none"}`, `- Pipeline + scoring: ${r.seconds.toFixed(1)} s`, "");
|
||||
const head = ["Role", "Ref notes", "Est notes", "P ±50 ms", "R ±50 ms", "F1 ±50 ms", "F1 ±100 ms"];
|
||||
o.push(row(head), row(head.map(() => "---")));
|
||||
for (const role of ROLE_ORDER) {
|
||||
const m = r.perRole[role];
|
||||
if (!m) continue;
|
||||
o.push(row([role, String(m.tp + m.fn), String(m.tp + m.fp), fmt(m.precision), fmt(m.recall), fmt(m.f1), fmt(r.lenient[role])]));
|
||||
}
|
||||
o.push("");
|
||||
if (r.drumsPerClass) {
|
||||
const dh = ["Drum class", "Ref", "Est", "F1 ±50 ms"];
|
||||
o.push(row(dh), row(dh.map(() => "---")));
|
||||
for (const [c, m] of Object.entries(r.drumsPerClass)) o.push(row([c, String(m.tp + m.fn), String(m.tp + m.fp), fmt(m.f1)]));
|
||||
o.push("");
|
||||
}
|
||||
}
|
||||
if (skipped.length) o.push(`Skipped: ${skipped.map((s) => `${s.id} (${s.reason})`).join("; ")}`, "");
|
||||
return o.join("\n");
|
||||
}
|
||||
+3
-1
@@ -25,11 +25,13 @@ async function sha256(bytes: Uint8Array): Promise<string> {
|
||||
|
||||
export async function runBench(opts: {
|
||||
tiers: TierId[]; seeds: number[]; modes: string[]; runId: string; root: string; out: string; resultsDir?: string;
|
||||
/** Guitar repetition voting (default on); false = --no-repeat-vote. */
|
||||
repeatVote?: boolean;
|
||||
}): Promise<BenchRun> {
|
||||
for (const m of opts.modes) if (!Object.hasOwn(MODES, m)) throw new Error(`unknown mode: ${m}`);
|
||||
const demucs = new Uint8Array(await Bun.file(MODEL_PATHS.demucs).arrayBuffer());
|
||||
const basicPitch = new Uint8Array(await Bun.file(MODEL_PATHS.basicPitch).arrayBuffer());
|
||||
const pipeline = await createPipeline({ demucs, basicPitch }); // model load once per run
|
||||
const pipeline = await createPipeline({ demucs, basicPitch }, { repeatVote: opts.repeatVote ?? true }); // model load once per run
|
||||
const run: BenchRun = {
|
||||
runId: opts.runId,
|
||||
date: new Date().toISOString(),
|
||||
|
||||
+21
-6
@@ -10,16 +10,31 @@ export function resample(x: Float32Array, from: number, to: number): Float32Arra
|
||||
const out = new Float32Array(outLen);
|
||||
const cutoff = Math.min(1, ratio); // in units of input Nyquist
|
||||
const half = 16 / cutoff; // kernel half-width in input samples (16 zero-crossings)
|
||||
const tap = (d: number) => {
|
||||
const w = 0.42 + 0.5 * Math.cos((Math.PI * d) / half) + 0.08 * Math.cos((2 * Math.PI * d) / half);
|
||||
return cutoff * sinc(cutoff * d) * w;
|
||||
};
|
||||
// When an output lands exactly on an input sample (e.g. 2:1 decimation) the kernel depends only on the integer
|
||||
// offset d = i - t, so the taps are computed once; the arithmetic is identical to the direct path.
|
||||
const kLo = Math.ceil(-half), kHi = Math.floor(half);
|
||||
let table: Float64Array | null = null, tableSum = 0;
|
||||
for (let j = 0; j < outLen; j++) {
|
||||
const t = j / ratio;
|
||||
const lo = Math.max(0, Math.ceil(t - half)), hi = Math.min(x.length - 1, Math.floor(t + half));
|
||||
let acc = 0, wsum = 0;
|
||||
for (let i = lo; i <= hi; i++) {
|
||||
const d = i - t;
|
||||
const w = 0.42 + 0.5 * Math.cos((Math.PI * d) / half) + 0.08 * Math.cos((2 * Math.PI * d) / half);
|
||||
const k = cutoff * sinc(cutoff * d) * w;
|
||||
acc += x[i]! * k;
|
||||
wsum += k;
|
||||
if (Number.isInteger(t) && lo === t + kLo && hi === t + kHi) {
|
||||
if (!table) {
|
||||
table = new Float64Array(kHi - kLo + 1);
|
||||
for (let d = kLo; d <= kHi; d++) { table[d - kLo] = tap(d); tableSum += table[d - kLo]!; }
|
||||
}
|
||||
for (let i = lo, k = 0; i <= hi; i++, k++) acc += x[i]! * table[k]!;
|
||||
wsum = tableSum;
|
||||
} else {
|
||||
for (let i = lo; i <= hi; i++) {
|
||||
const k = tap(i - t);
|
||||
acc += x[i]! * k;
|
||||
wsum += k;
|
||||
}
|
||||
}
|
||||
// Normalise to unity DC gain so edges and non-integer ratios stay calibrated.
|
||||
out[j] = wsum !== 0 ? acc / wsum : 0;
|
||||
|
||||
+2
-2
@@ -30,12 +30,12 @@ export const DRUM_CLASSES: Record<DrumClass, number[]> = {
|
||||
};
|
||||
const PASS_F1 = 0.85;
|
||||
|
||||
function drumClass(pitch: number): DrumClass | null {
|
||||
export function drumClass(pitch: number): DrumClass | null {
|
||||
for (const [c, ps] of Object.entries(DRUM_CLASSES)) if (ps.includes(pitch)) return c as DrumClass;
|
||||
return null;
|
||||
}
|
||||
// Drum pitches are compared by class: map to the class's first GM number; unknown pitches stay distinct.
|
||||
const toClassPitch = (n: TimedNote): TimedNote => {
|
||||
export const toClassPitch = (n: TimedNote): TimedNote => {
|
||||
const c = drumClass(n.pitch);
|
||||
return { ...n, pitch: c ? DRUM_CLASSES[c][0]! : 1000 + n.pitch };
|
||||
};
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
// Repetition voting. Songs repeat bars (riffs, verses, choruses); a transcriber misses or invents different notes
|
||||
// in each repeat. Bars whose transcriptions are similar are treated as repeats of one another, and notes are voted
|
||||
// per (pitch, position in bar): notes that almost never recur across a bar's repeats are dropped (precision) and
|
||||
// notes present in enough repeats but missing from the bar are added (recall). Repetition is estimated from the
|
||||
// transcription itself under the estimated constant-tempo grid. Pure: no I/O.
|
||||
import type { Grid, TimedNote } from "../core/types.ts";
|
||||
|
||||
export interface RepeatOpts {
|
||||
/** Minimum coarse similarity (16th-slot occupancy cosine, smoothed along the diagonal) for two bars to be repeats. */
|
||||
minSim?: number;
|
||||
/** Minimum exact similarity (equal pitch, onset within tol) for two bars to be repeats. Guards against tempo
|
||||
* drift: a slightly wrong grid makes distant bars coarse-similar but misaligned, and voting must not run then. */
|
||||
minExact?: number;
|
||||
/** Bars of context on each side used to smooth coarse similarity along the repetition diagonal. */
|
||||
context?: number;
|
||||
/** Minimum number of repeats (other bars) a bar needs before voting is applied. */
|
||||
minRepeats?: number;
|
||||
/** Drop a note when it recurs in fewer than this fraction of the bar's repeats. */
|
||||
keepFrac?: number;
|
||||
/** Add a note when it occurs in at least this fraction of the bar's repeats ... */
|
||||
addFrac?: number;
|
||||
/** ... and in at least this many repeats. */
|
||||
minAddSupport?: number;
|
||||
/** Onset tolerance (s) when matching notes across repeats. */
|
||||
tol?: number;
|
||||
drop?: boolean;
|
||||
add?: boolean;
|
||||
}
|
||||
|
||||
/** Defaults chosen in experiment E5 (docs/experiments/2026-10-31-accuracy.md). */
|
||||
export const REPEAT_DEFAULTS: Required<RepeatOpts> = {
|
||||
minSim: 0.6, minExact: 0.3, context: 1, minRepeats: 2, keepFrac: 0.15, addFrac: 0.25, minAddSupport: 3, tol: 0.05,
|
||||
drop: true, add: true,
|
||||
};
|
||||
|
||||
const SLOTS = 16;
|
||||
|
||||
interface Placed { n: TimedNote; rel: number }
|
||||
|
||||
/** Group notes into bars of the grid; rel = onset relative to the bar's downbeat (s). bar0 = time of bar 0. */
|
||||
function barsOf(notes: TimedNote[], grid: Grid): { byBar: Placed[][]; barSec: number; bar0: number } {
|
||||
const barSec = (60 / grid.bpm) * grid.beatsPerBar;
|
||||
const half = barSec / SLOTS / 2; // a note just before a downbeat belongs to that bar
|
||||
const startBar = notes.reduce((m, n) => Math.min(m, Math.floor((n.onset - grid.firstDownbeat + half) / barSec)), 0);
|
||||
const bar0 = grid.firstDownbeat + startBar * barSec;
|
||||
const byBar: Placed[][] = [];
|
||||
for (const n of notes) {
|
||||
const b = Math.floor((n.onset - bar0 + half) / barSec);
|
||||
(byBar[b] ??= []).push({ n, rel: n.onset - (bar0 + b * barSec) });
|
||||
}
|
||||
for (let b = 0; b < byBar.length; b++) byBar[b] ??= [];
|
||||
return { byBar, barSec, bar0 };
|
||||
}
|
||||
|
||||
/**
|
||||
* Bar x bar similarities. coarse: cosine of (16th slot, pitch) occupancy, a key counting as shared at the same or an
|
||||
* adjacent slot, averaged along the diagonal over +-context bars. exact: fraction of notes with an equal-pitch note
|
||||
* within tol at the same position, normalised by sqrt(|i| |j|).
|
||||
*/
|
||||
export function barSimilarity(
|
||||
byBar: { pitch: number; rel: number }[][], barSec: number, tol: number, context: number,
|
||||
): { coarse: Float64Array[]; exact: Float64Array[] } {
|
||||
const nB = byBar.length;
|
||||
const keys = byBar.map((ps) => new Set(ps.map((p) => Math.min(SLOTS - 1, Math.max(0, Math.round((p.rel / barSec) * SLOTS))) * 128 + p.pitch)));
|
||||
const raw = keys.map((ki) => Float64Array.from(keys, (kj) => {
|
||||
if (!ki.size || !kj.size) return 0;
|
||||
let dot = 0;
|
||||
for (const k of ki) if (kj.has(k) || kj.has(k - 128) || kj.has(k + 128)) dot++;
|
||||
return dot / Math.sqrt(ki.size * kj.size);
|
||||
}));
|
||||
const coarse = raw.map((_, i) => Float64Array.from(raw, (__, j) => {
|
||||
let s = 0, c = 0;
|
||||
for (let k = -context; k <= context; k++) {
|
||||
const a = i + k, b = j + k;
|
||||
if (a >= 0 && b >= 0 && a < nB && b < nB) { s += raw[a]![b]!; c++; }
|
||||
}
|
||||
return c ? s / c : 0;
|
||||
}));
|
||||
const exact = byBar.map((A) => Float64Array.from(byBar, (B) => {
|
||||
if (!A.length || !B.length) return 0;
|
||||
let m = 0;
|
||||
for (const p of A) if (B.some((q) => q.pitch === p.pitch && Math.abs(q.rel - p.rel) <= tol)) m++;
|
||||
return m / Math.sqrt(A.length * B.length);
|
||||
}));
|
||||
return { coarse, exact };
|
||||
}
|
||||
|
||||
/** Vote notes across repeated bars. Returns a new, onset-sorted note list; input notes are not modified. */
|
||||
export function repeatVote(notes: TimedNote[], grid: Grid, opts: RepeatOpts = {}): TimedNote[] {
|
||||
const o = { ...REPEAT_DEFAULTS, ...opts };
|
||||
const finite = Number.isFinite(grid.bpm) && Number.isFinite(grid.firstDownbeat) && Number.isFinite(grid.beatsPerBar);
|
||||
if (notes.length === 0 || !finite || grid.bpm <= 0 || grid.beatsPerBar <= 0) return notes.slice();
|
||||
const { byBar, barSec, bar0 } = barsOf(notes, grid);
|
||||
const nB = byBar.length;
|
||||
const { coarse, exact } = barSimilarity(byBar.map((ps) => ps.map((p) => ({ pitch: p.n.pitch, rel: p.rel }))), barSec, o.tol, o.context);
|
||||
const has = (bar: number, pitch: number, rel: number) => byBar[bar]!.some((q) => q.n.pitch === pitch && Math.abs(q.rel - rel) <= o.tol);
|
||||
|
||||
const out: TimedNote[] = [];
|
||||
for (let i = 0; i < nB; i++) {
|
||||
const bar = byBar[i]!;
|
||||
const reps: number[] = [];
|
||||
for (let j = 0; j < nB; j++) if (j !== i && coarse[i]![j]! >= o.minSim && exact[i]![j]! >= o.minExact) reps.push(j);
|
||||
if (reps.length < o.minRepeats) { for (const p of bar) out.push(p.n); continue; }
|
||||
|
||||
for (const p of bar) {
|
||||
if (!o.drop) { out.push(p.n); continue; }
|
||||
const support = reps.filter((j) => has(j, p.n.pitch, p.rel)).length;
|
||||
if (support / reps.length >= o.keepFrac) out.push(p.n);
|
||||
}
|
||||
if (!o.add) continue;
|
||||
|
||||
// Candidates from the repeats, clustered by pitch and position; one vote per repeat.
|
||||
const cands: { pitch: number; rel: number; rels: number[]; durs: number[]; bars: Set<number> }[] = [];
|
||||
for (const j of reps) for (const q of byBar[j]!) {
|
||||
if (has(i, q.n.pitch, q.rel)) continue;
|
||||
let c = cands.find((x) => x.pitch === q.n.pitch && Math.abs(x.rel - q.rel) <= o.tol);
|
||||
if (!c) cands.push((c = { pitch: q.n.pitch, rel: q.rel, rels: [], durs: [], bars: new Set() }));
|
||||
if (c.bars.has(j)) continue;
|
||||
c.bars.add(j); c.rels.push(q.rel); c.durs.push(q.n.offset - q.n.onset);
|
||||
}
|
||||
const added: TimedNote[] = [];
|
||||
// Near-duplicate guard: no added note within tol of a same-pitch note already in the bar (or its neighbours,
|
||||
// or already added) - an onset just outside tol in the bar itself would otherwise get a second copy.
|
||||
const near = (onset: number, pitch: number) => [...(byBar[i - 1] ?? []), ...bar, ...(byBar[i + 1] ?? [])]
|
||||
.some((q) => q.n.pitch === pitch && Math.abs(q.n.onset - onset) <= o.tol) ||
|
||||
added.some((a) => a.pitch === pitch && Math.abs(a.onset - onset) <= o.tol);
|
||||
for (const c of cands) {
|
||||
if (c.bars.size < o.minAddSupport || c.bars.size / reps.length < o.addFrac) continue;
|
||||
const onset = bar0 + i * barSec + median(c.rels);
|
||||
if (onset < 0 || near(onset, c.pitch)) continue;
|
||||
added.push({ onset, offset: onset + median(c.durs), pitch: c.pitch });
|
||||
}
|
||||
out.push(...added);
|
||||
}
|
||||
return out.sort((a, b) => a.onset - b.onset || a.pitch - b.pitch);
|
||||
}
|
||||
|
||||
function median(a: number[]): number {
|
||||
const s = [...a].sort((x, y) => x - y);
|
||||
const m = s.length >> 1;
|
||||
return s.length % 2 ? s[m]! : (s[m - 1]! + s[m]!) / 2;
|
||||
}
|
||||
+27
-3
@@ -10,12 +10,18 @@ import { transcribeDrums } from "./drums.ts";
|
||||
import { estimateGrid, gridFromTruth } from "./tempo.ts";
|
||||
import { quantize, snapTicks } from "./quantize.ts";
|
||||
import { assignTab } from "./tab.ts";
|
||||
import { repeatVote } from "./repeat.ts";
|
||||
|
||||
export interface PipelineInput {
|
||||
mix: Audio;
|
||||
oracle?: { stems: Map<string, Audio>; truth: { bpm: number; timeSig: [number, number]; tracks: TruthTrack[] } };
|
||||
}
|
||||
export interface PipelineResult { output: PipelineOutput; gp: Uint8Array; warnings: string[] }
|
||||
export interface PipelineOpts {
|
||||
/** Vote guitar notes across repeated bars (see repeat.ts). Default on: experiment E5 raised guitar F1 on
|
||||
* synthetic t1a–t1d (+0.02 median) and on "Last Night on Earth" (0.403 → 0.507). */
|
||||
repeatVote?: boolean;
|
||||
}
|
||||
|
||||
const PROGRAM: Record<Role, number> = { guitar: 27, bass: 33, keys: 0, vocals: 53, drums: 0 };
|
||||
const RANGE: Record<Exclude<Role, "drums">, TranscribeOpts> = {
|
||||
@@ -40,8 +46,22 @@ export function copyFingering(timed: TimedNote[], assigned: ScoreNote[], grid: G
|
||||
});
|
||||
}
|
||||
|
||||
/** Sample-wise sum of equally shaped audio buffers. */
|
||||
export function sumAudio(...as: Audio[]): Audio {
|
||||
const first = as[0]!;
|
||||
return {
|
||||
sampleRate: first.sampleRate,
|
||||
channels: first.channels.map((c, k) => {
|
||||
const o = new Float32Array(c.length);
|
||||
for (const a of as) { const x = a.channels[k]!; for (let i = 0; i < o.length; i++) o[i]! += x[i]!; }
|
||||
return o;
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
export async function createPipeline(
|
||||
models: { demucs: Uint8Array; basicPitch: Uint8Array },
|
||||
opts: PipelineOpts = {},
|
||||
): Promise<(input: PipelineInput, flags: OracleFlags) => Promise<PipelineResult>> {
|
||||
const separate = await createSeparator(models.demucs);
|
||||
const transcribe = await createTranscriber(models.basicPitch);
|
||||
@@ -63,9 +83,10 @@ export async function createPipeline(
|
||||
}
|
||||
} else {
|
||||
const s = await separate(input.mix);
|
||||
warnings.push('demucs "other" stem unused');
|
||||
const mk = (name: string, role: Role, audio: Audio): Job => ({ name, role, audio, truthNotes: null });
|
||||
jobs.push(mk("Guitar", "guitar", s.guitar), mk("Bass", "bass", s.bass), mk("Keys", "keys", s.piano),
|
||||
// Guitar is transcribed from guitar + piano + other: Demucs-6s scatters guitar energy across all three
|
||||
// (synthetic t1 median guitar F1 0.39–0.60 → 0.72–0.87; real "Last Night on Earth" 0.390 → 0.403).
|
||||
jobs.push(mk("Guitar", "guitar", sumAudio(s.guitar, s.piano, s.other)), mk("Bass", "bass", s.bass), mk("Keys", "keys", s.piano),
|
||||
mk("Vocals", "vocals", s.vocals), mk("Drums", "drums", s.drums));
|
||||
}
|
||||
|
||||
@@ -89,7 +110,10 @@ export async function createPipeline(
|
||||
else if (flags.oracleNotes) {
|
||||
const src = job.truthNotes ?? oracle!.truth.tracks.filter((t) => t.role === job.role).flatMap((t) => t.notes);
|
||||
notes = src.map((n) => ({ onset: n.onset, offset: n.offset, pitch: n.pitch, velocity: n.velocity }));
|
||||
} else notes = await transcribe(job.audio!, RANGE[job.role]);
|
||||
} else {
|
||||
notes = await transcribe(job.audio!, RANGE[job.role]);
|
||||
if (job.role === "guitar" && (opts.repeatVote ?? true)) notes = repeatVote(notes, grid);
|
||||
}
|
||||
|
||||
let scoreNotes = quantize(notes, grid);
|
||||
const tuning = TUNING[job.role];
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
import { join, resolve } from "node:path";
|
||||
import { mkdir, rename, rm } from "node:fs/promises";
|
||||
import type { Audio, Role } from "../core/types.ts";
|
||||
import { parseSmf, type Smf } from "../align/smf.ts";
|
||||
import type { Anchor } from "../align/warp.ts";
|
||||
import type { FitReport } from "../align/fit.ts";
|
||||
import { readWav } from "../audio/wav.ts";
|
||||
import { guessRole, trackKeys } from "../align/roles.ts";
|
||||
|
||||
export interface SongEntry {
|
||||
id: string; title: string; audio: string; midi: string;
|
||||
roles: Record<string, Role | null>; capo: Record<string, number>; tuning?: Record<string, number[]>;
|
||||
}
|
||||
|
||||
const ID_RE = /^[A-Za-z0-9][A-Za-z0-9_-]*$/;
|
||||
|
||||
export function songDir(id: string, root = "reference"): string {
|
||||
if (!ID_RE.test(id)) throw new Error(`invalid song id: ${JSON.stringify(id)}`);
|
||||
return join(root, id);
|
||||
}
|
||||
|
||||
export async function readManifest(root = "reference"): Promise<SongEntry[]> {
|
||||
const f = Bun.file(join(root, "songs.json"));
|
||||
if (!(await f.exists())) return [];
|
||||
return (await f.json()) as SongEntry[];
|
||||
}
|
||||
|
||||
export { trackKeys };
|
||||
|
||||
export async function addSong(e: Omit<SongEntry, "roles"> & { roles?: SongEntry["roles"] }, root = "reference"): Promise<SongEntry> {
|
||||
const dir = songDir(e.id, root);
|
||||
const all = await readManifest(root);
|
||||
if (all.some((s) => s.id === e.id)) throw new Error(`duplicate song id: ${e.id}`);
|
||||
let roles = e.roles;
|
||||
if (!roles) {
|
||||
const tracks = parseSmf(new Uint8Array(await Bun.file(e.midi).arrayBuffer())).tracks;
|
||||
const keys = trackKeys(tracks);
|
||||
roles = Object.fromEntries(tracks.map((t, i) => [keys[i]!, guessRole(t)]));
|
||||
}
|
||||
const entry: SongEntry = { ...e, audio: resolve(e.audio), midi: resolve(e.midi), roles };
|
||||
await mkdir(dir, { recursive: true });
|
||||
await Bun.write(join(root, "songs.json"), JSON.stringify([...all, entry], null, 2) + "\n");
|
||||
return entry;
|
||||
}
|
||||
|
||||
// ---- per-song caches and alignment file (reference/<id>/) ----
|
||||
|
||||
/** `auto`: how `bun run align auto` produced the anchors (absent for alignments saved from the page). */
|
||||
export interface AlignmentFile { anchors: Anchor[]; savedAt: string; fit: FitReport | null; auto?: AutoInfo }
|
||||
export interface AutoInfo { chosen: "baseline" | "dtw"; baseline: { offsetSec: number; precision: number | null }; dtwPrecision: number | null }
|
||||
|
||||
const inflight = new Map<string, Promise<string>>();
|
||||
|
||||
/** Runs `make(tmp)` to build `out` when it is missing or older than `src`; atomic via rename; concurrent calls share one run. */
|
||||
function cached(out: string, src: string, make: (tmp: string) => Promise<unknown>): Promise<string> {
|
||||
const running = inflight.get(out);
|
||||
if (running) return running;
|
||||
const p = (async () => {
|
||||
const o = Bun.file(out);
|
||||
if ((await o.exists()) && o.lastModified >= Bun.file(src).lastModified) return out;
|
||||
const tmp = `${out}.${process.pid}.${Date.now()}.tmp`;
|
||||
try {
|
||||
await make(tmp);
|
||||
await rename(tmp, out);
|
||||
} catch (e) {
|
||||
await rm(tmp, { force: true });
|
||||
throw e;
|
||||
}
|
||||
return out;
|
||||
})().finally(() => inflight.delete(out));
|
||||
inflight.set(out, p);
|
||||
return p;
|
||||
}
|
||||
|
||||
async function ffmpeg(args: string[]): Promise<void> {
|
||||
const r = await Bun.$`ffmpeg -nostdin -y -loglevel error ${args}`.nothrow().quiet();
|
||||
if (r.exitCode !== 0) throw new Error(`ffmpeg failed (${r.exitCode}): ${r.stderr.toString().trim()}`);
|
||||
}
|
||||
|
||||
/** `reference/<id>/recording.wav`: the recording as 44.1 kHz stereo PCM16 (analysis input), transcoded lazily. */
|
||||
export async function recordingWav(e: SongEntry, root = "reference"): Promise<string> {
|
||||
const dir = songDir(e.id, root);
|
||||
await mkdir(dir, { recursive: true });
|
||||
return cached(join(dir, "recording.wav"), e.audio, (tmp) => ffmpeg(["-i", e.audio, "-vn", "-ar", "44100", "-ac", "2", "-c:a", "pcm_s16le", "-f", "wav", tmp]));
|
||||
}
|
||||
|
||||
/** `reference/<id>/audio.ogg`: the recording as Opus 160 kbit/s for the browser, transcoded lazily. */
|
||||
export async function audioOgg(e: SongEntry, root = "reference"): Promise<string> {
|
||||
const dir = songDir(e.id, root);
|
||||
await mkdir(dir, { recursive: true });
|
||||
return cached(join(dir, "audio.ogg"), e.audio, (tmp) => ffmpeg(["-i", e.audio, "-vn", "-c:a", "libopus", "-b:a", "160k", "-f", "ogg", tmp]));
|
||||
}
|
||||
|
||||
/** The saved alignment, or null when none exists. Throws a message naming the file when it is unreadable. */
|
||||
export async function readAlignment(id: string, root = "reference"): Promise<AlignmentFile | null> {
|
||||
const path = join(songDir(id, root), "alignment.json");
|
||||
const f = Bun.file(path);
|
||||
if (!(await f.exists())) return null;
|
||||
let a: AlignmentFile;
|
||||
try { a = (await f.json()) as AlignmentFile; } catch (e) { throw new Error(`${path}: not valid JSON (${(e as Error).message})`); }
|
||||
if (!a || !Array.isArray(a.anchors)) throw new Error(`${path}: missing "anchors" array`);
|
||||
return a;
|
||||
}
|
||||
|
||||
let tmpCounter = 0;
|
||||
|
||||
export async function writeAlignment(id: string, a: AlignmentFile, root = "reference"): Promise<void> {
|
||||
const dir = songDir(id, root);
|
||||
await mkdir(dir, { recursive: true });
|
||||
const tmp = join(dir, `alignment.json.${process.pid}.${Date.now()}.${tmpCounter++}.tmp`);
|
||||
await Bun.write(tmp, JSON.stringify(a, null, 1) + "\n");
|
||||
await rename(tmp, join(dir, "alignment.json"));
|
||||
}
|
||||
|
||||
export interface LoadedSong { entry: SongEntry; smf: Smf; audio: Audio; audioDur: number }
|
||||
|
||||
/** Manifest entry, parsed MIDI and the 44.1 kHz recording (transcoded on first use). */
|
||||
export async function loadSong(id: string, root = "reference"): Promise<LoadedSong> {
|
||||
const entry = (await readManifest(root)).find((s) => s.id === id);
|
||||
if (!entry) throw new Error(`unknown song id: ${id}`);
|
||||
const smf = parseSmf(new Uint8Array(await Bun.file(entry.midi).arrayBuffer()));
|
||||
const audio = await readWav(await recordingWav(entry, root));
|
||||
return { entry, smf, audio, audioDur: (audio.channels[0]?.length ?? 0) / audio.sampleRate };
|
||||
}
|
||||
@@ -0,0 +1,289 @@
|
||||
import { test, expect, beforeAll } from "bun:test";
|
||||
import { alignmentFit, autoAlign, bandedWithRetry, barTimes, beatTimes, constantOffsetBaseline, fitFromCues, meterGrid, recordingCues, rejectOffsetOutliers } from "../../src/align/auto.ts";
|
||||
import { makeWarp, type Anchor, type Warp } from "../../src/align/warp.ts";
|
||||
import type { RefTrack, Smf } from "../../src/align/smf.ts";
|
||||
import type { Audio, Role, TruthTrack } from "../../src/core/types.ts";
|
||||
import { buildSong, loadSongDir } from "../../src/bench/dataset.ts";
|
||||
|
||||
const SF2 = new Uint8Array(await Bun.file("node_modules/@coderline/alphatab/dist/soundfont/sonivox.sf2").arrayBuffer());
|
||||
const BIG = 60_000;
|
||||
|
||||
let song: Awaited<ReturnType<typeof loadSongDir>>;
|
||||
beforeAll(async () => { song = await loadSongDir(await buildSong("t1c", 1)); }, BIG);
|
||||
|
||||
/** In-memory Smf from truth tracks: `map` turns truth (= recording) time into MIDI time; `reps` tiles the song. */
|
||||
function smfFrom(truth: TruthTrack[], programs: number[], bpm: number, map: (t: number) => number, reps = 1, period = 0): Smf {
|
||||
const tracks: RefTrack[] = truth.map((t, i) => ({
|
||||
index: i + 1, name: t.name, program: programs[i]!, isDrums: t.role === "drums",
|
||||
notes: Array.from({ length: reps }, (_, r) => t.notes.map((n) => ({
|
||||
onset: map(n.onset + r * period), offset: map(n.offset + r * period), pitch: n.pitch, velocity: n.velocity ?? 96,
|
||||
channel: t.role === "drums" ? 9 : i,
|
||||
}))).flat(),
|
||||
}));
|
||||
const durationSec = Math.max(...tracks.flatMap((t) => t.notes.map((n) => n.offset)));
|
||||
return { durationSec, tracks, tempoMap: [{ sec: 0, bpm }], timeSigs: [{ sec: 0, num: 4, den: 4 }] };
|
||||
}
|
||||
const rolesOf = (truth: TruthTrack[]): Record<string, Role | null> => Object.fromEntries(truth.map((t) => [t.name, t.role]));
|
||||
const durOf = (a: Audio) => a.channels[0]!.length / a.sampleRate;
|
||||
const median = (xs: number[]) => { const s = [...xs].sort((a, b) => a - b); return s[s.length >> 1]!; };
|
||||
const p90 = (xs: number[]) => { const s = [...xs].sort((a, b) => a - b); return s[Math.floor(s.length * 0.9)]!; };
|
||||
const debug = (msg: string) => { if (process.env.ALIGN_DEBUG) console.log(msg); };
|
||||
const strictlyMonotone = (A: Anchor[]) => A.every((a, k) => k === 0 || (a.midiSec > A[k - 1]!.midiSec && a.audioSec > A[k - 1]!.audioSec));
|
||||
|
||||
/** Known distortion recording time -> MIDI time: +0.4 s, then bars 4–10 at ×1.06 tempo, then ≈ −0.8 s (continuous). */
|
||||
function distortion(bpm: number): Warp {
|
||||
const bar = (4 * 60) / bpm, b1 = 4 * bar, b2 = 10 * bar;
|
||||
const m2 = b1 + 0.4 + (b2 - b1) / 1.06;
|
||||
return makeWarp([
|
||||
{ midiSec: 0, audioSec: 0.4, kind: "manual" },
|
||||
{ midiSec: b1, audioSec: b1 + 0.4, kind: "manual" },
|
||||
{ midiSec: b2, audioSec: m2, kind: "manual" },
|
||||
{ midiSec: b2 + 100, audioSec: m2 + 100, kind: "manual" },
|
||||
]); // toAudio: recording -> MIDI, toMidi: MIDI -> recording (the truth the aligner must recover)
|
||||
}
|
||||
|
||||
function recoveryErrors(smf: Smf, got: Anchor[], trueMidiToAudio: (m: number) => number, audioDur: number): number[] {
|
||||
const w = makeWarp(got);
|
||||
const onsets = [...new Set(smf.tracks.flatMap((t) => t.notes.map((n) => n.onset)))];
|
||||
return onsets.filter((m) => { const a = trueMidiToAudio(m); return a >= 0 && a <= audioDur; })
|
||||
.map((m) => Math.abs(w.toAudio(m) - trueMidiToAudio(m)));
|
||||
}
|
||||
|
||||
test("recovers a known piecewise warp: median error < 40 ms, fit precision ≥ 0.8, DTW beats the constant baseline", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const t0 = performance.now();
|
||||
const { anchors, fit, baseline, chosen } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [], SF2);
|
||||
const ms = performance.now() - t0;
|
||||
const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix));
|
||||
debug(`synthetic warp: ${anchors.length} anchors, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.precision?.toFixed(3)} lift ${fit.lift?.toFixed(3)} / ${fit.medianResidualMs?.toFixed(1)} ms, baseline ${baseline.offsetSec} s ${baseline.precision?.toFixed(3)}, ${ms.toFixed(0)} ms`);
|
||||
expect(chosen).toBe("dtw");
|
||||
expect(anchors.every((a) => a.kind === "auto")).toBe(true);
|
||||
expect(median(err)).toBeLessThan(0.04);
|
||||
expect(p90(err)).toBeLessThan(0.08);
|
||||
expect(fit.precision!).toBeGreaterThanOrEqual(0.8);
|
||||
expect(fit.precision!).toBeGreaterThanOrEqual(baseline.precision! + 0.02);
|
||||
expect(fit.windows.some((w) => w.flagged)).toBe(false);
|
||||
expect(fit.stretches!.total).toBe(anchors.length + 1);
|
||||
expect(fit.windows.length).toBe(2);
|
||||
expect(anchors.every((a) => a.audioSec > 0 || a.midiSec === 0)).toBe(true);
|
||||
}, BIG);
|
||||
|
||||
test("a correct manual anchor mid-song is honoured exactly", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const manual: Anchor = { midiSec: D.toAudio(30), audioSec: 30, kind: "manual" };
|
||||
const { anchors } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [manual], SF2);
|
||||
expect(anchors.filter((a) => a.kind === "manual")).toEqual([manual]);
|
||||
expect(makeWarp(anchors).toAudio(manual.midiSec)).toBe(30);
|
||||
// no auto anchor within 1 s (MIDI time) of the manual one
|
||||
expect(anchors.some((a) => a.kind === "auto" && Math.abs(a.midiSec - manual.midiSec) < 1)).toBe(false);
|
||||
expect(anchors.length).toBeGreaterThan(5);
|
||||
const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix));
|
||||
expect(median(err)).toBeLessThan(0.04);
|
||||
}, BIG);
|
||||
|
||||
test("a silent intro in the recording does not drag the path", () => {
|
||||
const pad = 4 * song.mix.sampleRate;
|
||||
const mix: Audio = { sampleRate: song.mix.sampleRate, channels: song.mix.channels.map((c) => { const o = new Float32Array(c.length + pad); o.set(c, pad); return o; }) };
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t);
|
||||
const { anchors, fit } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2);
|
||||
const err = recoveryErrors(smf, anchors, (m) => m + 4, durOf(mix));
|
||||
expect(median(err)).toBeLessThan(0.04);
|
||||
expect(fit.precision!).toBeGreaterThanOrEqual(0.8);
|
||||
}, BIG);
|
||||
|
||||
test("5-minute input aligns in < 15 s", () => {
|
||||
const reps = 5, period = durOf(song.mix);
|
||||
const mix: Audio = { sampleRate: song.mix.sampleRate, channels: song.mix.channels.map((c) => { const o = new Float32Array(c.length * reps); for (let r = 0; r < reps; r++) o.set(c, r * c.length); return o; }) };
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t + 0.3, reps, period);
|
||||
const t0 = performance.now();
|
||||
const { anchors, fit, baseline, chosen } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2);
|
||||
const sec = (performance.now() - t0) / 1000;
|
||||
debug(`5-min (${durOf(mix).toFixed(0)} s) input: ${sec.toFixed(2)} s, ${anchors.length} anchors, fit ${fit.precision?.toFixed(3)}, ${chosen}, baseline ${baseline.offsetSec} ${baseline.precision?.toFixed(3)}`);
|
||||
expect(durOf(mix)).toBeGreaterThan(290);
|
||||
expect(sec).toBeLessThan(15);
|
||||
expect(fit.windows.length).toBe(10);
|
||||
// a constant offset: DTW cannot beat the baseline by 0.02, so the single-anchor baseline is returned
|
||||
expect(baseline.offsetSec).toBeCloseTo(-0.3, 1);
|
||||
expect(chosen).toBe("baseline");
|
||||
expect(anchors).toHaveLength(1);
|
||||
expect(anchors[0]!.kind).toBe("auto");
|
||||
expect(anchors[0]!.audioSec - anchors[0]!.midiSec).toBeCloseTo(baseline.offsetSec, 6);
|
||||
}, BIG);
|
||||
|
||||
test("barTimes follows tempo and time-signature changes", () => {
|
||||
const smf: Smf = {
|
||||
durationSec: 10, tracks: [],
|
||||
tempoMap: [{ sec: 0, bpm: 120 }, { sec: 4, bpm: 60 }],
|
||||
timeSigs: [{ sec: 0, num: 4, den: 4 }, { sec: 4, num: 3, den: 4 }],
|
||||
};
|
||||
// 4/4 @120: bars of 2 s at 0, 2; then 3/4 @60: bars of 3 s at 4, 7
|
||||
expect(barTimes(smf)).toEqual([0, 2, 4, 7]);
|
||||
expect(barTimes({ durationSec: 3, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }], timeSigs: [] })).toEqual([0, 2]); // default 4/4
|
||||
expect(barTimes({ durationSec: 4, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }], timeSigs: [{ sec: 0, num: 6, den: 8 }] })).toEqual([0, 1.5, 3]);
|
||||
});
|
||||
|
||||
test("fine pass widens the band when the end cell is unreachable, then gives up with null", () => {
|
||||
const cost = (i: number, j: number) => Math.abs(i - j);
|
||||
// centre jumps by 120 frames mid-way: ±25 and ±50 leave a gap between rows, ±100 bridges it
|
||||
const jump = (i: number) => (i < 200 ? i : i + 120);
|
||||
const p = bandedWithRetry(cost, 400, 600, jump);
|
||||
expect(p).not.toBeNull();
|
||||
expect(p![p!.length - 1]).toEqual([399, 599]);
|
||||
// a 400-frame jump defeats even ±2 s (±100 frames)
|
||||
expect(bandedWithRetry(cost, 400, 1000, (i) => (i < 200 ? i : i + 400))).toBeNull();
|
||||
});
|
||||
|
||||
test("fit exposes a bad alignment instead of hiding it", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const roles = rolesOf(song.truth.tracks);
|
||||
const bad = alignmentFit(song.mix, smf, roles, []); // identity: off by 0.4–0.8 s throughout
|
||||
expect(bad.precision!).toBeLessThan(0.6);
|
||||
// the −0.8 s section is exposed; the +0.4 s section is not, as 0.4 s is ≈ 40 ms from two 16ths at 68 bpm (grid alias)
|
||||
expect(bad.windows[1]!.flagged).toBe(true);
|
||||
const good = alignmentFit(song.mix, smf, roles, autoAlign(song.mix, smf, roles, [], SF2).anchors);
|
||||
expect(good.windows.some((w) => w.flagged)).toBe(false);
|
||||
// unmapped tracks are excluded: with no scored roles there is nothing to measure
|
||||
expect(alignmentFit(song.mix, smf, Object.fromEntries(Object.keys(roles).map((k) => [k, null])), []).precision).toBeNull();
|
||||
}, BIG);
|
||||
|
||||
test("a deliberately wrong manual anchor (+0.3 s) is kept exactly and the result stays monotonic", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const manual: Anchor = { midiSec: D.toAudio(30), audioSec: 30.3, kind: "manual" };
|
||||
const { anchors } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [manual], SF2);
|
||||
expect(anchors.filter((a) => a.kind === "manual")).toEqual([manual]);
|
||||
expect(makeWarp(anchors).toAudio(manual.midiSec)).toBe(30.3);
|
||||
expect(strictlyMonotone(anchors)).toBe(true);
|
||||
expect(anchors.every((a) => a.audioSec > 0 || a.midiSec === 0)).toBe(true);
|
||||
// away from the bad anchor (> 4 s), the autos still follow the truth
|
||||
const far = anchors.filter((a) => a.kind === "auto" && Math.abs(a.audioSec - 30) > 4);
|
||||
expect(far.length).toBeGreaterThan(5);
|
||||
expect(median(far.map((a) => Math.abs(a.audioSec - D.toMidi(a.midiSec))))).toBeLessThan(0.04);
|
||||
}, BIG);
|
||||
|
||||
test("when every fine band fails, the coarse-path fallback still yields auto anchors", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const manual: Anchor = { midiSec: D.toAudio(30), audioSec: 30, kind: "manual" };
|
||||
const { anchors, fit, chosen } = autoAlign(song.mix, smf, rolesOf(song.truth.tracks), [manual], SF2, { fineBandsSec: [] });
|
||||
expect(chosen).toBe("dtw"); // manual anchors present: never substituted by the baseline
|
||||
const autos = anchors.filter((a) => a.kind === "auto");
|
||||
// both segments (before and after the manual anchor) get anchors
|
||||
expect(autos.filter((a) => a.audioSec < 30).length).toBeGreaterThan(3);
|
||||
expect(autos.filter((a) => a.audioSec > 30).length).toBeGreaterThan(3);
|
||||
expect(strictlyMonotone(anchors)).toBe(true);
|
||||
const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix));
|
||||
debug(`coarse fallback: ${autos.length} autos, median err ${(median(err) * 1000).toFixed(1)} ms, p90 ${(p90(err) * 1000).toFixed(1)} ms, fit ${fit.precision?.toFixed(3)}`);
|
||||
expect(median(err)).toBeLessThan(0.06); // 10 Hz resolution: coarser than the fine pass, still hand-correctable
|
||||
expect(p90(err)).toBeLessThan(0.12);
|
||||
}, BIG);
|
||||
|
||||
test("fitFromCues with cached recordingCues equals alignmentFit", () => {
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t);
|
||||
const roles = rolesOf(song.truth.tracks);
|
||||
const anchors: Anchor[] = [{ midiSec: 1, audioSec: 1.05, kind: "manual" }, { midiSec: 20, audioSec: 20.1, kind: "manual" }];
|
||||
const cues = recordingCues(song.mix);
|
||||
expect(cues.onsets.length).toBeGreaterThan(10);
|
||||
expect(cues.chroma.length).toBeGreaterThan(cues.durSec * cues.chromaRate - 2);
|
||||
expect(fitFromCues(cues, smf, roles, anchors)).toEqual(alignmentFit(song.mix, smf, roles, anchors));
|
||||
});
|
||||
|
||||
// ---- ruling A7: calibrated fit, beat-shift check, constant-offset baseline ----
|
||||
|
||||
test("fit precision: correct alignment ≥ 0.8, a 0.12 s off-grid shift < 0.5 (and flagged)", () => {
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t);
|
||||
const roles = rolesOf(song.truth.tracks);
|
||||
const good = alignmentFit(song.mix, smf, roles, []);
|
||||
const off = alignmentFit(song.mix, smf, roles, [{ midiSec: 10, audioSec: 10.12, kind: "manual" }]);
|
||||
debug(`precision: correct ${good.precision?.toFixed(3)} (lift ${good.lift?.toFixed(3)}, audible ${good.audibleRate?.toFixed(3)}), +0.12 s ${off.precision?.toFixed(3)} (lift ${off.lift?.toFixed(3)})`);
|
||||
expect(good.precision!).toBeGreaterThanOrEqual(0.8);
|
||||
expect(good.lift!).toBeGreaterThanOrEqual(0.25);
|
||||
expect(good.windows.some((w) => w.flagged)).toBe(false);
|
||||
expect(off.precision!).toBeLessThan(0.5);
|
||||
expect(off.windows.filter((w) => w.precision !== null).every((w) => w.flagged)).toBe(true);
|
||||
});
|
||||
|
||||
test("beat-shift check flags an alignment that is one beat late, though onset precision cannot see it", () => {
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t);
|
||||
const roles = rolesOf(song.truth.tracks);
|
||||
const beat = 60 / song.song.bpm;
|
||||
const late = alignmentFit(song.mix, smf, roles, [{ midiSec: 10, audioSec: 10 + beat, kind: "manual" }]);
|
||||
const good = alignmentFit(song.mix, smf, roles, []);
|
||||
debug(`beat shift: late precision ${late.precision?.toFixed(3)}, windows ${JSON.stringify(late.windows.map((w) => [w.precision?.toFixed(2), w.lift?.toFixed(2), w.beatShift]))}`);
|
||||
expect(good.windows.some((w) => w.beatShift)).toBe(false);
|
||||
const shifted = late.windows.filter((w) => w.beatShift);
|
||||
expect(shifted.length).toBeGreaterThan(0);
|
||||
expect(shifted.every((w) => w.flagged && w.reasons.some((r) => r.startsWith("beat shift suspected: −")))).toBe(true);
|
||||
});
|
||||
|
||||
test("constantOffsetBaseline grid-searches −2..+2 s for the offset with the best precision", () => {
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, (t) => t + 0.27);
|
||||
const b = constantOffsetBaseline(recordingCues(song.mix), smf, rolesOf(song.truth.tracks));
|
||||
expect(b.offsetSec).toBeCloseTo(-0.27, 1);
|
||||
expect(Math.round(b.offsetSec * 100)).toBe(b.offsetSec * 100); // 10 ms grid
|
||||
expect(b.precision!).toBeGreaterThanOrEqual(0.8);
|
||||
});
|
||||
|
||||
// ---- ruling A8: aligner guards ----
|
||||
|
||||
test("rejectOffsetOutliers drops bars > ¼ beat from the robust median of ±3 neighbours (slope-transported), iterating until stable", () => {
|
||||
const bars = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: 0.1 + k * 0.004, beatSec: 0.5 }));
|
||||
bars[5]!.off = 0.6; // a beat out
|
||||
bars[6]!.off = 0.55; // its neighbour, also out
|
||||
bars[15]!.off = -0.3;
|
||||
const kept = rejectOffsetOutliers(bars).map((b) => b.midiSec / 2);
|
||||
expect(kept).toEqual(Array.from({ length: 20 }, (_, k) => k).filter((k) => k !== 5 && k !== 6 && k !== 15));
|
||||
// a smooth ramp (genuine tempo change) is kept whole
|
||||
const ramp = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: k * 0.05, beatSec: 0.5 }));
|
||||
expect(rejectOffsetOutliers(ramp)).toHaveLength(20);
|
||||
// ... also a steep one (10% tempo difference: 0.2 s per 2 s bar), whose ends a plain one-sided median would erode
|
||||
// bar by bar; neighbours are transported along their median (Theil–Sen) slope
|
||||
const steep = Array.from({ length: 20 }, (_, k) => ({ midiSec: k * 2, off: k * 0.2, beatSec: 0.5 }));
|
||||
expect(rejectOffsetOutliers(steep)).toHaveLength(20);
|
||||
// a beat jump inside the steep ramp is still rejected
|
||||
steep[10]!.off += 0.5;
|
||||
expect(rejectOffsetOutliers(steep).map((b) => b.midiSec / 2)).toEqual(Array.from({ length: 20 }, (_, k) => k).filter((k) => k !== 10));
|
||||
});
|
||||
|
||||
test("meterGrid: beats and bars from tempo/time signatures; the final bar is partial when the MIDI ends mid-bar", () => {
|
||||
const smf: Smf = { durationSec: 9, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }, { sec: 4, bpm: 60 }], timeSigs: [{ sec: 0, num: 4, den: 4 }, { sec: 4, num: 3, den: 4 }] };
|
||||
const g = meterGrid(smf);
|
||||
expect(g.bars).toEqual([0, 2, 4, 7]);
|
||||
expect(g.beats).toEqual([0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 5, 6, 7, 8, 9]);
|
||||
expect(g.lastBarPartial).toBe(true); // bar 7..10 ends after 9
|
||||
expect(meterGrid({ ...smf, durationSec: 10 }).lastBarPartial).toBe(false);
|
||||
expect(beatTimes({ durationSec: 3, tracks: [], tempoMap: [{ sec: 0, bpm: 120 }], timeSigs: [{ sec: 0, num: 6, den: 8 }] })).toEqual([0, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75]);
|
||||
});
|
||||
|
||||
test("no auto anchor in the final partial bar", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const g = meterGrid(smf);
|
||||
const last = g.bars[g.bars.length - 1]!;
|
||||
const cut: Smf = { ...smf, durationSec: last + 0.5 * (last - g.bars[g.bars.length - 2]!) }; // ends half-way into the last bar
|
||||
expect(meterGrid(cut).lastBarPartial).toBe(true);
|
||||
const { anchors } = autoAlign(song.mix, cut, rolesOf(song.truth.tracks), [], SF2);
|
||||
expect(anchors.some((a) => a.midiSec >= last - 1e-9)).toBe(false);
|
||||
}, BIG);
|
||||
|
||||
test("12 s of extra outro in the recording: open-end DTW, no ramp at the end", () => {
|
||||
const D = distortion(song.song.bpm);
|
||||
const smf = smfFrom(song.truth.tracks, song.song.tracks.map((t) => t.program), song.song.bpm, D.toAudio);
|
||||
const sr = song.mix.sampleRate, extra = 12 * sr;
|
||||
// the outro repeats the first 12 s of the song (material that DTW could be tempted to match)
|
||||
const mix: Audio = { sampleRate: sr, channels: song.mix.channels.map((c) => { const o = new Float32Array(c.length + extra); o.set(c); o.set(c.subarray(0, extra), c.length); return o; }) };
|
||||
const { anchors, chosen } = autoAlign(mix, smf, rolesOf(song.truth.tracks), [], SF2);
|
||||
expect(chosen).toBe("dtw");
|
||||
const autos = anchors.filter((a) => a.kind === "auto");
|
||||
const tail = autos.slice(-4).map((a) => a.audioSec - D.toMidi(a.midiSec));
|
||||
debug(`outro: last anchor offsets vs truth ${tail.map((e) => (e * 1000).toFixed(0)).join(", ")} ms`);
|
||||
expect(tail.every((e) => Math.abs(e) < 0.06)).toBe(true);
|
||||
const w = makeWarp(anchors);
|
||||
const endErr = Math.abs(w.toAudio(smf.durationSec) - D.toMidi(smf.durationSec));
|
||||
expect(endErr).toBeLessThan(0.1);
|
||||
const err = recoveryErrors(smf, anchors, D.toMidi, durOf(song.mix));
|
||||
expect(median(err)).toBeLessThan(0.04);
|
||||
}, BIG);
|
||||
@@ -0,0 +1,111 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { dtw } from "../../src/align/dtw.ts";
|
||||
|
||||
// Ground-truth path: diagonal with a kink (j runs ahead by 10 after i=50).
|
||||
function truthJ(i: number): number { return i < 50 ? i : i + 10; }
|
||||
const N = 100, M = 110;
|
||||
// Path cells cost 0, others grow with distance from the true path.
|
||||
const cost = (i: number, j: number) => {
|
||||
const t = truthJ(i);
|
||||
return Math.min(1, Math.abs(j - t) * 0.2);
|
||||
};
|
||||
|
||||
test("dtw recovers diagonal-with-kink path", () => {
|
||||
const p = dtw(cost, N, M);
|
||||
expect(p[0]).toEqual([0, 0]);
|
||||
expect(p[p.length - 1]).toEqual([N - 1, M - 1]);
|
||||
for (const [i, j] of p) if (i !== 49 && i !== 50) expect(Math.abs(j - truthJ(i))).toBeLessThanOrEqual(1);
|
||||
for (let k = 1; k < p.length; k++) {
|
||||
const di = p[k]![0] - p[k - 1]![0], dj = p[k]![1] - p[k - 1]![1];
|
||||
expect(di >= 0 && di <= 1 && dj >= 0 && dj <= 1 && di + dj >= 1).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
test("banded dtw equals unbanded when the band covers the path", () => {
|
||||
const full = dtw(cost, N, M);
|
||||
const banded = dtw(cost, N, M, { center: (i) => truthJ(i), width: 15 });
|
||||
expect(banded).toEqual(full);
|
||||
});
|
||||
|
||||
test("banded dtw only visits cells in the band", () => {
|
||||
let calls = 0;
|
||||
const p = dtw((i, j) => { calls++; return Math.abs(i - j) * 0.1; }, 200, 200, { center: (i) => i, width: 5 });
|
||||
expect(calls).toBeLessThanOrEqual(200 * 11);
|
||||
expect(p[p.length - 1]).toEqual([199, 199]);
|
||||
});
|
||||
|
||||
test("3000x3000 unbanded dtw finishes in < 5 s", () => {
|
||||
const n = 3000;
|
||||
const t0 = performance.now();
|
||||
const p = dtw((i, j) => Math.abs(i - j) * 0.001 + ((i * 7 + j * 13) % 5) * 0.0001, n, n);
|
||||
const ms = performance.now() - t0;
|
||||
console.log(`dtw 3000x3000: ${ms.toFixed(0)} ms`);
|
||||
expect(p[p.length - 1]).toEqual([n - 1, n - 1]);
|
||||
expect(ms).toBeLessThan(5000);
|
||||
});
|
||||
|
||||
// ---- slope-limited steps and open begin/end (ruling A8) ----
|
||||
const stepsOf = (p: [number, number][]) => p.slice(1).map(([i, j], k) => [i - p[k]![0], j - p[k]![1]] as const);
|
||||
/** No two consecutive non-diagonal steps in the same direction: local slope stays within [1/2, 2]. */
|
||||
function slopeLimited(p: [number, number][]): boolean {
|
||||
const s = stepsOf(p);
|
||||
return s.every(([di, dj], k) => {
|
||||
if (!((di === 1 && dj === 1) || (di === 1 && dj === 0) || (di === 0 && dj === 1))) return false;
|
||||
const prev = s[k - 1];
|
||||
return !(k > 0 && di + dj === 1 && prev![0] === di && prev![1] === dj);
|
||||
});
|
||||
}
|
||||
|
||||
test("slope-limited steps: path never makes an abrupt jump even when the cost invites one", () => {
|
||||
// the cost is 0 on a path that jumps 20 columns at i = 50; symmetric steps follow the jump, slope-limited ones cannot
|
||||
const jumpCost = (i: number, j: number) => Math.min(1, Math.abs(j - (i < 50 ? i : i + 20)) * 0.2);
|
||||
const sym = dtw(jumpCost, 100, 120);
|
||||
expect(stepsOf(sym).some(([di, dj], k, s) => di === 0 && dj === 1 && s[k + 1]?.[0] === 0)).toBe(true); // a horizontal run
|
||||
const p = dtw(jumpCost, 100, 120, undefined, { steps: "slope" });
|
||||
expect(p[0]).toEqual([0, 0]);
|
||||
expect(p[p.length - 1]).toEqual([99, 119]);
|
||||
expect(slopeLimited(p)).toBe(true);
|
||||
});
|
||||
|
||||
test("slope-limited steps follow a smooth diagonal-with-kink path exactly like the default (banded too)", () => {
|
||||
const p = dtw(cost, N, M, undefined, { steps: "slope" });
|
||||
expect(slopeLimited(p)).toBe(true);
|
||||
for (const [i, j] of p) if (i < 45 || i > 60) expect(Math.abs(j - truthJ(i))).toBeLessThanOrEqual(1);
|
||||
expect(dtw(cost, N, M, { center: truthJ, width: 15 }, { steps: "slope" })).toEqual(p);
|
||||
});
|
||||
|
||||
test("slope-limited steps throw 'unreachable' when the aspect ratio exceeds 2", () => {
|
||||
expect(() => dtw(cost, 10, 30, undefined, { steps: "slope" })).toThrow(/unreachable/);
|
||||
});
|
||||
|
||||
test("open end: the path may stop before the recording ends (subsequence end)", () => {
|
||||
// the query (rows) matches columns 0..99 diagonally; the last 40 columns are extra material
|
||||
const c = (i: number, j: number) => (j < 100 ? Math.min(1, Math.abs(i - j) * 0.2) : 0.5);
|
||||
const closed = dtw(c, 100, 140, undefined, { steps: "slope" });
|
||||
expect(closed[closed.length - 1]).toEqual([99, 139]);
|
||||
const open = dtw(c, 100, 140, undefined, { steps: "slope", openEnd: true });
|
||||
expect(open[open.length - 1]).toEqual([99, 99]);
|
||||
for (const [i, j] of open) expect(Math.abs(i - j)).toBeLessThanOrEqual(1);
|
||||
// the query being longer than the target (its extra rows mismatch everything, costing more than an unconsumed row):
|
||||
// the path may end on the last column
|
||||
const longer = dtw((i, j) => (i < 60 ? Math.min(1, Math.abs(i - j) * 0.2) : 1.5), 100, 60, undefined, { steps: "slope", openEnd: true });
|
||||
expect(longer[longer.length - 1]).toEqual([59, 59]);
|
||||
});
|
||||
|
||||
test("open begin: the path may start after the beginning of the target", () => {
|
||||
const c = (i: number, j: number) => (j >= 30 ? Math.min(1, Math.abs(i + 30 - j) * 0.2) : 0.5);
|
||||
const p = dtw(c, 100, 130, undefined, { steps: "slope", openBegin: true });
|
||||
expect(p[0]).toEqual([0, 30]);
|
||||
expect(p[p.length - 1]).toEqual([99, 129]);
|
||||
for (const [i, j] of p) expect(Math.abs(i + 30 - j)).toBeLessThanOrEqual(1);
|
||||
// banded with a band that does not include column 0 at row 0
|
||||
expect(dtw(c, 100, 130, { center: (i) => i + 30, width: 10 }, { steps: "slope", openBegin: true })).toEqual(p);
|
||||
});
|
||||
|
||||
test("open begin + end: a trivial path in a zero-cost corner does not win (unconsumed rows cost 1 each)", () => {
|
||||
// the last column is free everywhere (e.g. silence vs silence), the real match is the diagonal of the first 100 columns
|
||||
const c = (i: number, j: number) => (j === 119 ? 0 : j < 100 ? Math.min(1, Math.abs(i - j) * 0.2) : 0.6);
|
||||
const p = dtw(c, 100, 120, undefined, { steps: "slope", openBegin: true, openEnd: true });
|
||||
expect(p.length).toBeGreaterThan(90);
|
||||
expect(p[p.length - 1]).toEqual([99, 99]);
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { features, pickOnsets } from "../../src/align/features.ts";
|
||||
import type { Audio } from "../../src/core/types.ts";
|
||||
|
||||
const SR = 44100;
|
||||
function tone(freqs: number[], sec: number): Audio {
|
||||
const x = new Float32Array(Math.round(sec * SR));
|
||||
for (let i = 0; i < x.length; i++) for (const f of freqs) x[i]! += 0.3 * Math.sin((2 * Math.PI * f * i) / SR);
|
||||
return { sampleRate: SR, channels: [x] };
|
||||
}
|
||||
|
||||
test("features rate and length", () => {
|
||||
const f = features(tone([440], 3), 10);
|
||||
expect(f.rate).toBe(10);
|
||||
expect(Math.abs(f.chroma.length - 31)).toBeLessThanOrEqual(1);
|
||||
expect(f.onset.length).toBe(f.chroma.length);
|
||||
expect(f.chroma[0]!.length).toBe(12);
|
||||
const g = features(tone([440], 3), 50);
|
||||
expect(Math.abs(g.chroma.length - 151)).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test("chroma peaks at the right pitch class and is unit-norm", () => {
|
||||
const f = features(tone([440], 2), 10); // A -> pitch class 9
|
||||
const c = f.chroma[10]!;
|
||||
let best = 0;
|
||||
for (let k = 1; k < 12; k++) if (c[k]! > c[best]!) best = k;
|
||||
expect(best).toBe(9);
|
||||
expect(Math.hypot(...c)).toBeCloseTo(1, 4);
|
||||
const fe = features(tone([261.63, 329.63, 392], 2), 10).chroma[10]!; // C E G
|
||||
const top = [...fe].map((v, k) => [v, k] as const).sort((a, b) => b[0] - a[0]).slice(0, 3).map((e) => e[1]).sort((a, b) => a - b);
|
||||
expect(top).toEqual([0, 4, 7]);
|
||||
});
|
||||
|
||||
test("silent frames stay zero", () => {
|
||||
const f = features({ sampleRate: SR, channels: [new Float32Array(SR * 2)] }, 10);
|
||||
for (const c of f.chroma) expect(c.every((v) => v === 0)).toBe(true);
|
||||
});
|
||||
|
||||
test("pickOnsets finds a click train within 10 ms", () => {
|
||||
const x = new Float32Array(SR * 4);
|
||||
const times = [0.5, 1.2, 2.0, 2.7, 3.3];
|
||||
for (const t of times) {
|
||||
const s = Math.round(t * SR);
|
||||
for (let k = 0; k < 200; k++) x[s + k] = Math.sin(k * 0.8) * Math.exp(-k / 40) * 0.8;
|
||||
}
|
||||
const f = features({ sampleRate: SR, channels: [x] }, 50);
|
||||
const got = pickOnsets(f.onset, f.rate);
|
||||
expect(got.length).toBe(times.length);
|
||||
times.forEach((t, k) => expect(Math.abs(got[k]! - t)).toBeLessThanOrEqual(0.01));
|
||||
});
|
||||
|
||||
test("pickOnsets enforces 50 ms min gap and ignores flat input", () => {
|
||||
const o = new Float32Array(100);
|
||||
o[20] = 5; o[22] = 6; o[60] = 5;
|
||||
expect(pickOnsets(o, 100)).toEqual([0.22, 0.6]);
|
||||
expect(pickOnsets(new Float32Array(100), 100)).toEqual([]);
|
||||
});
|
||||
|
||||
test("pickOnsets refines peaks to sub-frame times by parabolic interpolation", () => {
|
||||
const o = new Float32Array(100);
|
||||
o[30] = 4; o[31] = 8; o[32] = 6; // peak leans right: δ = ½(4 − 6)/(4 − 16 + 6) = 1/6 frame
|
||||
const got = pickOnsets(o, 50);
|
||||
expect(got).toHaveLength(1);
|
||||
expect(got[0]!).toBeCloseTo((31 + 1 / 6) / 50, 9);
|
||||
// min-gap competition still compares the peak heights, not the interpolated times
|
||||
const p = new Float32Array(100);
|
||||
p[40] = 5; p[41] = 3; p[42] = 7; p[43] = 1;
|
||||
expect(pickOnsets(p, 100)).toHaveLength(1);
|
||||
expect(pickOnsets(p, 100)[0]!).toBeCloseTo((42 + 0.5 * (3 - 1) / (3 - 14 + 1)) / 100, 9);
|
||||
});
|
||||
@@ -0,0 +1,106 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { fitQuality, fitSummary, FLAG_MIN_PRECISION, FLAG_MIN_LIFT, MIN_REC_ONSETS, SHIFT_MARGIN, type FitReport, type FitWindow } from "../../src/align/fit.ts";
|
||||
|
||||
const grid = (from: number, to: number, step: number) => { const o: number[] = []; for (let t = from; t < to - 1e-9; t += step) o.push(Math.round(t * 1000) / 1000); return o; };
|
||||
|
||||
test("constants match ruling A7", () => {
|
||||
expect(FLAG_MIN_PRECISION).toBe(0.6);
|
||||
expect(FLAG_MIN_LIFT).toBe(0.25);
|
||||
expect(MIN_REC_ONSETS).toBe(8);
|
||||
expect(SHIFT_MARGIN).toBe(0.05);
|
||||
});
|
||||
|
||||
test("precision = share of recording onsets with a MIDI onset within ±70 ms; audibleRate = the reverse (info)", () => {
|
||||
// recording hears every 2nd MIDI onset (dense score, sparse attacks) plus 2 extra attacks out of the score
|
||||
const midi = grid(0.5, 20.5, 0.25); // 80 onsets
|
||||
const rec = [...midi.filter((_, k) => k % 2 === 0), 3.13, 7.63]; // 40 + 2
|
||||
const r = fitQuality(rec, midi, 20.5);
|
||||
expect(r.precision).toBeCloseTo(40 / 42, 6);
|
||||
expect(r.audibleRate).toBeCloseTo(0.5, 6);
|
||||
expect(r.medianResidualMs).toBe(0);
|
||||
expect(r.windows).toHaveLength(1);
|
||||
expect(r.windows[0]!.recOnsets).toBe(42);
|
||||
// low audibleRate alone does not flag (that was the old, miscalibrated rule)
|
||||
expect(r.windows[0]!.flagged).toBe(false);
|
||||
});
|
||||
|
||||
test("lift: precision minus mean precision at off-grid shifts (+0.11, +0.23, +0.37 s)", () => {
|
||||
const midi = grid(1, 29, 1); // one onset per second: shifts of 0.11–0.37 s never land
|
||||
const r = fitQuality(midi, midi, 30);
|
||||
expect(r.precision).toBe(1);
|
||||
expect(r.lift).toBeCloseTo(1, 6);
|
||||
expect(r.windows[0]!.lift).toBeCloseTo(1, 6);
|
||||
// a dense MIDI grid (every 0.12 s) matches anything: precision 1 but no lift -> flagged
|
||||
const dense = grid(0, 30, 0.12);
|
||||
const rec = grid(0.5, 29.5, 0.5).map((t) => t + 0.031);
|
||||
const d = fitQuality(rec, dense, 30);
|
||||
expect(d.precision).toBe(1);
|
||||
expect(d.lift!).toBeLessThan(FLAG_MIN_LIFT);
|
||||
expect(d.windows[0]!.flagged).toBe(true);
|
||||
expect(d.windows[0]!.reasons.join(" ")).toContain("lift");
|
||||
});
|
||||
|
||||
test("window flagged when precision < 0.6; null and unflagged with < 8 recording onsets", () => {
|
||||
const midi = grid(1, 89, 1);
|
||||
const rec = [
|
||||
...grid(1, 11, 1), // window 0: 10 onsets, all on the score
|
||||
...grid(31.5, 41.5, 1), // window 1: 10 onsets, all half-way between score onsets
|
||||
61, 62, 63, // window 2: only 3 onsets
|
||||
];
|
||||
const r = fitQuality(rec, midi, 90);
|
||||
expect(r.windows.map((w) => w.startSec)).toEqual([0, 30, 60]);
|
||||
expect(r.windows[0]!.precision).toBe(1);
|
||||
expect(r.windows[0]!.flagged).toBe(false);
|
||||
expect(r.windows[1]!.precision).toBe(0);
|
||||
expect(r.windows[1]!.flagged).toBe(true);
|
||||
expect(r.windows[1]!.reasons[0]).toContain("precision");
|
||||
const w2 = r.windows[2]!;
|
||||
expect(w2.recOnsets).toBe(3);
|
||||
expect(w2.precision).toBeNull();
|
||||
expect(w2.lift).toBeNull();
|
||||
expect(w2.flagged).toBe(false);
|
||||
});
|
||||
|
||||
test("no MIDI onsets: overall nulls, nothing flagged", () => {
|
||||
const r = fitQuality([1, 2], [], 10);
|
||||
expect(r.precision).toBeNull();
|
||||
expect(r.audibleRate).toBeNull();
|
||||
expect(r.lift).toBeNull();
|
||||
expect(r.windows.every((w) => !w.flagged)).toBe(true);
|
||||
});
|
||||
|
||||
test("residual: median |recording − nearest MIDI| over matched recording onsets", () => {
|
||||
const midi = grid(1, 11, 1);
|
||||
const r = fitQuality(midi.map((t, k) => t + (k % 2 ? 0.02 : 0.04)), midi, 12);
|
||||
expect(r.medianResidualMs!).toBeCloseTo(30, 6);
|
||||
});
|
||||
|
||||
test("beat-shift check: flags a window where a shifted symbolic chroma beats the current one by the margin", () => {
|
||||
const rate = 10, n = 600; // 60 s, two windows
|
||||
const pc = (k: number) => { const c = new Float32Array(12); c[k % 12] = 1; return c; };
|
||||
// chord changes every second (one "beat"); recording = sequence s(t); current MIDI = s shifted by one beat in window 1
|
||||
const seq = (t: number) => pc(Math.floor(t / rate) * 5);
|
||||
const recording = Array.from({ length: n }, (_, t) => seq(t));
|
||||
const current = Array.from({ length: n }, (_, t) => (t < 300 ? seq(t) : seq(t - rate)));
|
||||
const plus1 = Array.from({ length: n }, (_, t) => seq(t - rate)), minus1 = Array.from({ length: n }, (_, t) => (t < 300 ? seq(t - rate) : seq(t)));
|
||||
const onsets = grid(0.5, 59.5, 0.5);
|
||||
const r = fitQuality(onsets, onsets, 60, { shift: { rate, recording, current, shifts: [{ label: "+1 beat", chroma: plus1 }, { label: "−1 beat", chroma: minus1 }] } });
|
||||
expect(r.windows[0]!.beatShift).toBeNull();
|
||||
expect(r.windows[0]!.flagged).toBe(false);
|
||||
expect(r.windows[1]!.beatShift?.label).toBe("−1 beat");
|
||||
expect(r.windows[1]!.beatShift!.gain).toBeGreaterThan(SHIFT_MARGIN);
|
||||
expect(r.windows[1]!.flagged).toBe(true);
|
||||
expect(r.windows[1]!.reasons).toContain("beat shift suspected: −1 beat");
|
||||
});
|
||||
|
||||
test("fitSummary: precision, lift, audible rate, residual, flagged windows with reasons, stretch warnings", () => {
|
||||
const w = (startSec: number, reasons: string[]): FitWindow => ({ startSec, recOnsets: 20, precision: 0.9, lift: 0.7, audibleRate: 0.4, medianResidualMs: 10, beatShift: null, flagged: reasons.length > 0, reasons });
|
||||
const f: FitReport = {
|
||||
precision: 0.8234, lift: 0.6512, audibleRate: 0.3811, medianResidualMs: 14.24,
|
||||
windows: [w(0, []), w(30, ["beat shift suspected: +1 beat"]), w(60, []), w(90, ["precision 41%"])],
|
||||
stretches: { total: 40, warned: 3, over15: 1 },
|
||||
};
|
||||
expect(fitSummary(f)).toBe("precision 82.3% (lift +0.65) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms · 2/4 windows flagged (0:30 beat shift suspected: +1 beat; 1:30 precision 41%) · stretches: 3/40 warned, 1 > 15%");
|
||||
expect(fitSummary({ ...f, windows: [w(0, [])], stretches: undefined })).toBe("precision 82.3% (lift +0.65) · MIDI onsets audible as attacks 38.1% · median residual 14.2 ms · 0/1 windows flagged");
|
||||
expect(fitSummary({ precision: null, lift: null, audibleRate: null, medianResidualMs: null, windows: [] })).toBe("no scored MIDI onsets");
|
||||
});
|
||||
@@ -0,0 +1,71 @@
|
||||
import { test, expect, describe } from "bun:test";
|
||||
import { dragAudio, dragMidi, hitAnchor, anchorAtCursor, MIN_GAP_SEC } from "../../../src/align/page/edit.ts";
|
||||
import { makeWarp, type Anchor } from "../../../src/align/warp.ts";
|
||||
|
||||
const m = (midiSec: number, audioSec: number): Anchor => ({ midiSec, audioSec, kind: "manual" });
|
||||
const a = (midiSec: number, audioSec: number): Anchor => ({ midiSec, audioSec, kind: "auto" });
|
||||
const base = [m(10, 11), m(20, 22), m(30, 32)];
|
||||
|
||||
describe("dragAudio (recording end)", () => {
|
||||
test("moves audioSec, keeps midiSec, returns the anchor's new index", () => {
|
||||
const r = dragAudio(base, 1, 23);
|
||||
expect(r.anchors[r.index]).toEqual(m(20, 23));
|
||||
expect(r.anchors).toHaveLength(3);
|
||||
});
|
||||
test("cannot cross manual neighbours", () => {
|
||||
expect(dragAudio(base, 1, 5).anchors[1]!.audioSec).toBeCloseTo(11 + MIN_GAP_SEC, 9);
|
||||
expect(dragAudio(base, 1, 99).anchors[1]!.audioSec).toBeCloseTo(32 - MIN_GAP_SEC, 9);
|
||||
});
|
||||
test("the first anchor cannot be dragged before the start of the recording", () => {
|
||||
expect(dragAudio(base, 0, -3).anchors[0]!.audioSec).toBe(0);
|
||||
});
|
||||
test("dragging an auto anchor makes it manual and drops the autos in its stretches", () => {
|
||||
const list = [m(0, 0), a(5, 5.2), a(10, 10.3), a(15, 15.1), m(20, 20)];
|
||||
const r = dragAudio(list, 2, 10.5);
|
||||
expect(r.anchors).toEqual([m(0, 0), m(10, 10.5), m(20, 20)]);
|
||||
expect(r.index).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("dragMidi (MIDI end): the grabbed MIDI moment follows the mouse, audioSec stays", () => {
|
||||
for (const [grab, to] of [[22, 23.5], [22, 20.5], [21, 25], [24, 21]] as const) {
|
||||
test(`grab at ${grab}s, drag to ${to}s`, () => {
|
||||
const m0 = makeWarp(base).toMidi(grab);
|
||||
const r = dragMidi(base, 1, m0, to);
|
||||
expect(r.anchors[r.index]!.audioSec).toBe(22);
|
||||
expect(makeWarp(r.anchors).toAudio(m0)).toBeCloseTo(to, 6);
|
||||
});
|
||||
}
|
||||
test("no drag = no change", () => {
|
||||
const r = dragMidi(base, 1, 20, 22);
|
||||
expect(r.anchors[1]!.midiSec).toBeCloseTo(20, 6);
|
||||
});
|
||||
test("clamped between manual neighbours' MIDI times", () => {
|
||||
const r = dragMidi(base, 1, 20, 31.9);
|
||||
expect(r.anchors[1]!.midiSec).toBeGreaterThanOrEqual(10 + MIN_GAP_SEC - 1e-9);
|
||||
const s = dragMidi(base, 1, 20, 11.1);
|
||||
expect(s.anchors[1]!.midiSec).toBeLessThanOrEqual(30 - MIN_GAP_SEC + 1e-9);
|
||||
});
|
||||
test("works for the first and only anchor (edge extrapolation)", () => {
|
||||
const r = dragMidi([m(10, 11)], 0, 10, 12);
|
||||
expect(makeWarp(r.anchors).toAudio(10)).toBeCloseTo(12, 6);
|
||||
expect(r.anchors[0]!.audioSec).toBe(11);
|
||||
const f = dragMidi(base, 0, 10, 12);
|
||||
expect(makeWarp(f.anchors).toAudio(10)).toBeCloseTo(12, 6);
|
||||
});
|
||||
});
|
||||
|
||||
test("hitAnchor picks the nearest anchor line within tolerance", () => {
|
||||
const v = { start: 0, pxPerSec: 10, width: 500 };
|
||||
expect(hitAnchor(base, v, 221, 6)).toBe(1);
|
||||
expect(hitAnchor(base, v, 113, 6)).toBe(0);
|
||||
expect(hitAnchor(base, v, 160, 6)).toBe(-1);
|
||||
});
|
||||
|
||||
test("anchorAtCursor adds a manual anchor where the current warp maps the cursor", () => {
|
||||
const r = anchorAtCursor(base, 27);
|
||||
const k = r.anchors.findIndex((x) => x.audioSec === 27);
|
||||
expect(r.index).toBe(k);
|
||||
expect(r.anchors[k]).toEqual(m(makeWarp(base).toMidi(27), 27));
|
||||
expect(() => anchorAtCursor(base, 22)).toThrow(); // already an anchor there
|
||||
});
|
||||
@@ -0,0 +1,91 @@
|
||||
import { test, expect, describe } from "bun:test";
|
||||
import { timeToX, xToTime, fitView, zoomAt, scrollBy, clampView, follow, formatTime, rulerStep, buildPyramid, columnPeaks, MAX_PX_PER_SEC, type View } from "../../../src/align/page/timeline.ts";
|
||||
|
||||
const DUR = 200;
|
||||
|
||||
describe("view mapping", () => {
|
||||
const v: View = { start: 10, pxPerSec: 50, width: 1000 };
|
||||
test("time <-> x are inverse", () => {
|
||||
expect(timeToX(v, 12)).toBe(100);
|
||||
expect(xToTime(v, 100)).toBe(12);
|
||||
for (const t of [0, 7.3, 31]) expect(xToTime(v, timeToX(v, t))).toBeCloseTo(t, 12);
|
||||
});
|
||||
test("fitView shows the whole song", () => {
|
||||
const f = fitView(DUR, 1000);
|
||||
expect(xToTime(f, 0)).toBeLessThanOrEqual(0);
|
||||
expect(xToTime(f, 1000)).toBeGreaterThanOrEqual(DUR);
|
||||
expect(xToTime(f, 1000)).toBeLessThan(DUR * 1.05);
|
||||
});
|
||||
});
|
||||
|
||||
describe("zoom and scroll", () => {
|
||||
const v: View = { start: 50, pxPerSec: 20, width: 1000 };
|
||||
test("zoomAt keeps the time under the mouse fixed", () => {
|
||||
const t = xToTime(v, 300);
|
||||
const z = zoomAt(v, 300, 2, DUR);
|
||||
expect(z.pxPerSec).toBe(40);
|
||||
expect(xToTime(z, 300)).toBeCloseTo(t, 9);
|
||||
const o = zoomAt(v, 700, 0.8, DUR);
|
||||
expect(xToTime(o, 700)).toBeCloseTo(xToTime(v, 700), 9);
|
||||
});
|
||||
test("zoom is clamped: cannot zoom out past the whole song or in past the maximum", () => {
|
||||
const out = zoomAt(v, 500, 1e-6, DUR);
|
||||
expect(out.pxPerSec).toBeCloseTo(fitView(DUR, 1000).pxPerSec, 9);
|
||||
const inn = zoomAt(v, 500, 1e9, DUR);
|
||||
expect(inn.pxPerSec).toBe(MAX_PX_PER_SEC);
|
||||
});
|
||||
test("scrollBy moves by pixels (drag right = earlier time) and is clamped to the song", () => {
|
||||
expect(scrollBy(v, -100, DUR).start).toBeCloseTo(55, 9);
|
||||
expect(scrollBy(v, 100, DUR).start).toBeCloseTo(45, 9);
|
||||
expect(xToTime(scrollBy(v, 1e7, DUR), 0)).toBeGreaterThanOrEqual(-DUR * 0.05);
|
||||
expect(xToTime(scrollBy(v, -1e7, DUR), 1000)).toBeLessThanOrEqual(DUR * 1.05 + 1e-9);
|
||||
});
|
||||
test("clampView leaves a valid view unchanged", () => {
|
||||
expect(clampView(v, DUR)).toEqual(v);
|
||||
});
|
||||
test("follow pages the view when the playhead leaves it, and not otherwise", () => {
|
||||
expect(follow(v, 60, DUR)).toEqual(v);
|
||||
const f = follow(v, 120, DUR); // visible range is 50..100
|
||||
expect(f.start).toBeLessThan(120);
|
||||
expect(xToTime(f, 1000)).toBeGreaterThan(120);
|
||||
expect(f.pxPerSec).toBe(v.pxPerSec);
|
||||
});
|
||||
});
|
||||
|
||||
test("formatTime and ruler step", () => {
|
||||
expect(formatTime(58.24)).toBe("0:58.2");
|
||||
expect(formatTime(130)).toBe("2:10.0");
|
||||
expect(formatTime(3.04, 2)).toBe("0:03.04");
|
||||
expect(formatTime(-1.5)).toBe("-0:01.5");
|
||||
// labelled ticks at least ~80 px apart
|
||||
for (const pps of [0.5, 3, 20, 100, 1000]) expect(rulerStep(pps) * pps).toBeGreaterThanOrEqual(80);
|
||||
expect(rulerStep(20)).toBe(5);
|
||||
});
|
||||
|
||||
describe("peaks pyramid", () => {
|
||||
const sr = 1000;
|
||||
const x = new Float32Array(10 * sr);
|
||||
for (let i = 0; i < x.length; i++) x[i] = i >= 2000 && i < 3000 ? (i % 2 ? 0.8 : -0.5) : 0.01;
|
||||
const levels = buildPyramid(x, 16);
|
||||
test("each level halves the previous", () => {
|
||||
expect(levels[0]!.bucket).toBe(16);
|
||||
for (let k = 1; k < levels.length; k++) expect(levels[k]!.bucket).toBe(levels[k - 1]!.bucket * 2);
|
||||
expect(levels.at(-1)!.min.length).toBeLessThanOrEqual(2);
|
||||
});
|
||||
test("columnPeaks finds the loud second (2..3 s) at any zoom", () => {
|
||||
for (const pxPerSec of [2, 30, 400, 5000]) {
|
||||
const v: View = { start: 0, pxPerSec, width: Math.ceil(10 * pxPerSec) };
|
||||
const { min, max } = columnPeaks(levels, sr, v);
|
||||
expect(min.length).toBe(v.width);
|
||||
const col = Math.floor(2.5 * pxPerSec);
|
||||
expect(max[col]).toBeCloseTo(0.8, 5);
|
||||
expect(min[col]).toBeCloseTo(-0.5, 5);
|
||||
expect(max[Math.floor(6 * pxPerSec)]).toBeCloseTo(0.01, 5);
|
||||
}
|
||||
});
|
||||
test("columns outside the audio are silent", () => {
|
||||
const { min, max } = columnPeaks(levels, sr, { start: -5, pxPerSec: 10, width: 20 });
|
||||
expect(max[5]).toBe(0);
|
||||
expect(min[5]).toBe(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,77 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { renderMidi } from "../../src/align/render.ts";
|
||||
import { peaks } from "../../src/align/peaks.ts";
|
||||
import { makeWarp } from "../../src/align/warp.ts";
|
||||
import type { RefTrack, RefNote } from "../../src/align/smf.ts";
|
||||
|
||||
const SF2 = new Uint8Array(await Bun.file("node_modules/@coderline/alphatab/dist/soundfont/sonivox.sf2").arrayBuffer());
|
||||
const note = (onset: number, offset: number, pitch: number): RefNote => ({ onset, offset, pitch, velocity: 100, channel: 0 });
|
||||
const track = (notes: RefNote[], isDrums = false): RefTrack => ({ index: 0, name: "t", program: 25, isDrums, notes });
|
||||
const SR = 44100;
|
||||
|
||||
function firstAbove(x: Float32Array, thr: number): number {
|
||||
for (let i = 0; i < x.length; i++) if (Math.abs(x[i]!) > thr) return i;
|
||||
return -1;
|
||||
}
|
||||
|
||||
test("warp offset moves the note onset", () => {
|
||||
const a = renderMidi([track([note(1, 2, 60)])], makeWarp([{ midiSec: 0, audioSec: 0.5, kind: "manual" }]), SF2, { sampleRate: SR, mono: true, durationSec: 4 });
|
||||
expect(a.channels.length).toBe(1);
|
||||
expect(a.channels[0]!.length).toBe(4 * SR);
|
||||
const t = firstAbove(a.channels[0]!, 0.02) / SR;
|
||||
expect(t).toBeGreaterThan(1.48);
|
||||
expect(t).toBeLessThan(1.52);
|
||||
});
|
||||
|
||||
test("dense chord is peak-normalised to -1 dBFS", () => {
|
||||
const notes = Array.from({ length: 20 }, (_, i) => note(0, 1, 40 + i * 2));
|
||||
const a = renderMidi([track(notes)], makeWarp([]), SF2, { sampleRate: SR });
|
||||
let p = 0;
|
||||
for (const c of a.channels) for (const v of c) p = Math.max(p, Math.abs(v));
|
||||
expect(p).toBeLessThanOrEqual(Math.pow(10, -1 / 20) + 1e-3);
|
||||
expect(p).toBeGreaterThan(0.85);
|
||||
});
|
||||
|
||||
test("drums track is audible", () => {
|
||||
const a = renderMidi([track([note(0.2, 0.3, 38)], true)], makeWarp([]), SF2, { sampleRate: SR, durationSec: 1 });
|
||||
expect(a.channels[0]!.some((v) => Math.abs(v) > 0.1)).toBe(true);
|
||||
});
|
||||
|
||||
test("peaks of a known sine", () => {
|
||||
const x = new Float32Array(1000).map((_, i) => Math.sin((2 * Math.PI * i) / 100));
|
||||
const p = peaks(x, 10);
|
||||
expect(p.max.length).toBe(10);
|
||||
for (let b = 0; b < 10; b++) {
|
||||
expect(p.max[b]!).toBeCloseTo(1, 1);
|
||||
expect(p.min[b]!).toBeCloseTo(-1, 1);
|
||||
}
|
||||
const q = peaks(new Float32Array([0.5, 0.25, -0.75, 0]), 2);
|
||||
expect([...q.max]).toEqual([0.5, 0]);
|
||||
expect([...q.min]).toEqual([0.25, -0.75]);
|
||||
});
|
||||
|
||||
test("sampleRate 22050 is finite, at 22050 Hz, onset preserved", () => {
|
||||
const a = renderMidi([track([note(1, 2, 60)])], makeWarp([{ midiSec: 0, audioSec: 0.5, kind: "manual" }]), SF2, { sampleRate: 22050, mono: true, durationSec: 4 });
|
||||
expect(a.sampleRate).toBe(22050);
|
||||
expect(a.channels[0]!.length).toBe(4 * 22050);
|
||||
expect(a.channels[0]!.every(Number.isFinite)).toBe(true);
|
||||
const t = firstAbove(a.channels[0]!, 0.02) / 22050;
|
||||
expect(t).toBeGreaterThan(1.48);
|
||||
expect(t).toBeLessThan(1.52);
|
||||
});
|
||||
|
||||
test("notes before audio zero: clamped when overlapping, dropped when entirely before", () => {
|
||||
const w = makeWarp([{ midiSec: 1, audioSec: 0, kind: "manual" }]); // audio = midi - 1
|
||||
const dropped = renderMidi([track([note(0, 0.5, 60)]), track([note(2, 2.5, 64)])], w, SF2, { durationSec: 3 });
|
||||
const t = firstAbove(dropped.channels[0]!, 0.02) / 44100;
|
||||
expect(t).toBeGreaterThan(0.98);
|
||||
const clamped = renderMidi([track([note(0.5, 2, 60)])], w, SF2, { durationSec: 3 });
|
||||
expect(firstAbove(clamped.channels[0]!, 0.02) / 44100).toBeLessThan(0.05);
|
||||
});
|
||||
|
||||
test("stereo with durationSec has two channels of exact length", () => {
|
||||
const a = renderMidi([track([note(0, 0.5, 60)])], makeWarp([]), SF2, { durationSec: 2 });
|
||||
expect(a.channels.length).toBe(2);
|
||||
expect(a.channels[0]!.length).toBe(88200);
|
||||
expect(a.channels[1]!.length).toBe(88200);
|
||||
});
|
||||
@@ -0,0 +1,229 @@
|
||||
import { test, expect, beforeAll, afterAll, describe } from "bun:test";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { startServer, type AlignServer } from "../../src/align/server.ts";
|
||||
import { addSong } from "../../src/ref/manifest.ts";
|
||||
import { writeSmf } from "../../src/align/smf.ts";
|
||||
import { buildSong, loadSongDir } from "../../src/bench/dataset.ts";
|
||||
import type { Anchor } from "../../src/align/warp.ts";
|
||||
|
||||
const BIG = 120_000;
|
||||
let root = "", srv: AlignServer, cookie = "";
|
||||
const url = (p: string) => new URL(p, srv.origin).toString();
|
||||
const api = (p: string, init: RequestInit = {}) => fetch(url(p), { ...init, headers: { cookie, ...(init.headers ?? {}) }, redirect: "manual" });
|
||||
const json = (body: unknown, method = "POST"): RequestInit => ({ method, body: JSON.stringify(body), headers: { "content-type": "application/json" } });
|
||||
|
||||
beforeAll(async () => {
|
||||
root = await mkdtemp(join(tmpdir(), "stsheets-align-"));
|
||||
const dir = await buildSong("t1a", 1);
|
||||
const { song, truth } = await loadSongDir(dir);
|
||||
const midi = join(root, "fixture.mid");
|
||||
await Bun.write(midi, writeSmf(truth.tracks.map((t, i) => ({ name: t.name, program: song.tracks[i]!.program, isDrums: t.role === "drums", notes: t.notes })), song.bpm));
|
||||
await addSong({ id: "fix", title: "Fixture Song", audio: join(dir, "mix.wav"), midi, capo: {} }, root);
|
||||
srv = await startServer({ root, host: "127.0.0.1", port: 0, quiet: true });
|
||||
}, BIG);
|
||||
afterAll(async () => { await srv?.stop(); if (root) await rm(root, { recursive: true, force: true }); });
|
||||
|
||||
describe("access token", () => {
|
||||
test("requests without the token get 401", async () => {
|
||||
for (const p of ["/", "/api/songs", "/api/song/fix", "/api/songs?key=wrong"]) expect((await fetch(url(p), { redirect: "manual" })).status).toBe(401);
|
||||
expect((await fetch(url("/api/songs"), { headers: { cookie: "stsheets_align=nope" } })).status).toBe(401);
|
||||
});
|
||||
test("printed URL carries the token; ?key= sets an HttpOnly cookie and redirects to a clean URL", async () => {
|
||||
expect(srv.url).toContain(`?key=${srv.token}`);
|
||||
const r = await fetch(url(`/?key=${srv.token}`), { redirect: "manual" });
|
||||
expect(r.status).toBe(302);
|
||||
expect(r.headers.get("location")).toBe("/");
|
||||
const sc = r.headers.get("set-cookie") ?? "";
|
||||
expect(sc).toContain("HttpOnly");
|
||||
cookie = sc.split(";")[0]!;
|
||||
expect(cookie).toBe(`stsheets_align=${srv.token}`);
|
||||
});
|
||||
});
|
||||
|
||||
/** Raw HTTP/1.1 GET (Bun's fetch normalises "//" in paths, which would hide the bug under test). */
|
||||
async function rawGet(path: string, cookieHeader = ""): Promise<{ status: number; headers: Record<string, string> }> {
|
||||
const text = await new Promise<string>((resolve, reject) => {
|
||||
let buf = "";
|
||||
Bun.connect({
|
||||
hostname: "127.0.0.1", port: srv.port,
|
||||
socket: {
|
||||
open(sk) { sk.write(`GET ${path} HTTP/1.1\r\nHost: 127.0.0.1\r\n${cookieHeader ? `Cookie: ${cookieHeader}\r\n` : ""}Connection: close\r\n\r\n`); },
|
||||
data(sk, d) {
|
||||
buf += d.toString();
|
||||
const end = buf.indexOf("\r\n\r\n"); // complete once headers and Content-Length bytes have arrived
|
||||
const len = Number(buf.slice(0, end).match(/content-length:\s*(\d+)/i)?.[1] ?? 0);
|
||||
if (end >= 0 && buf.length - end - 4 >= len) { sk.end(); resolve(buf); }
|
||||
},
|
||||
close() { resolve(buf); },
|
||||
error(_, e) { reject(e); },
|
||||
},
|
||||
}).catch(reject);
|
||||
});
|
||||
const [head] = text.split("\r\n\r\n");
|
||||
const [statusLine, ...lines] = head!.split("\r\n");
|
||||
const headers = Object.fromEntries(lines.map((l) => { const i = l.indexOf(":"); return [l.slice(0, i).toLowerCase(), l.slice(i + 1).trim()]; }));
|
||||
return { status: Number(statusLine!.split(" ")[1]), headers };
|
||||
}
|
||||
|
||||
describe("paths with leading double slashes stay on this server", () => {
|
||||
test("GET //example.invalid/ is not proxied off-host (served locally: 404)", async () => {
|
||||
const r = await rawGet("//example.invalid/", cookie);
|
||||
expect(r.status).toBe(404);
|
||||
expect(r.headers.location).toBeUndefined();
|
||||
});
|
||||
test("GET //<this server>/ serves the local page, not a proxied fetch", async () => {
|
||||
const r = await rawGet(`//127.0.0.1:${srv.port}/`, cookie);
|
||||
expect(r.status).toBe(404); // collapsed to /127.0.0.1:<port>/ on the inner server
|
||||
});
|
||||
test("the ?key= redirect keeps a same-origin Location", async () => {
|
||||
const r = await rawGet(`//example.invalid/x?key=${srv.token}`);
|
||||
expect(r.status).toBe(302);
|
||||
const loc = r.headers.location!;
|
||||
expect(loc.startsWith("/")).toBe(true);
|
||||
expect(loc.startsWith("//")).toBe(false);
|
||||
expect(new URL(loc, srv.origin).origin).toBe(new URL(srv.origin).origin);
|
||||
});
|
||||
});
|
||||
|
||||
describe("page", () => {
|
||||
test("GET / serves the HTML page and its bundled script", async () => {
|
||||
const r = await api("/");
|
||||
expect(r.status).toBe(200);
|
||||
const html = await r.text();
|
||||
expect(html).toContain("<canvas");
|
||||
const src = html.match(/<script[^>]+src="([^"]+)"/)?.[1];
|
||||
expect(src).toBeTruthy();
|
||||
const js = await api(src!);
|
||||
expect(js.status).toBe(200);
|
||||
expect((await js.text()).length).toBeGreaterThan(1000);
|
||||
expect((await fetch(url(src!))).status).toBe(401); // assets are behind the token too
|
||||
});
|
||||
});
|
||||
|
||||
describe("API", () => {
|
||||
test("songs list", async () => {
|
||||
expect(await (await api("/api/songs")).json()).toEqual([{ id: "fix", title: "Fixture Song", hasAlignment: false }]);
|
||||
});
|
||||
test("unknown and path-traversal ids are 404", async () => {
|
||||
for (const p of ["/api/song/nope", "/api/song/%2E%2E", "/api/song/..%2F..%2Fetc", "/api/song/%2E%2E/audio", "/api/song/nope/render"]) {
|
||||
expect((await api(p, p.endsWith("render") ? json({ anchors: [] }) : {})).status).toBe(404);
|
||||
}
|
||||
});
|
||||
let first: any;
|
||||
test("song: peaks, auto anchors (unsaved), stretches with MIDI and effective bpm, fit, tempo map", async () => {
|
||||
const r = await api("/api/song/fix");
|
||||
expect(r.status).toBe(200);
|
||||
first = await r.json();
|
||||
expect(first.id).toBe("fix");
|
||||
expect(first.title).toBe("Fixture Song");
|
||||
expect(first.audioDur).toBeGreaterThan(20);
|
||||
expect(first.midiDur).toBeGreaterThan(20);
|
||||
expect(first.tracks.length).toBeGreaterThan(0);
|
||||
expect(first.tracks.every((t: any) => typeof t.name === "string" && "role" in t)).toBe(true);
|
||||
expect(first.recordingPeaks.bucketSec).toBeGreaterThan(0);
|
||||
expect(first.recordingPeaks.max.length).toBeGreaterThan(first.audioDur * 20);
|
||||
expect(Math.max(...first.recordingPeaks.max)).toBeGreaterThan(0.05);
|
||||
expect(first.source).toBe("auto");
|
||||
expect(first.savedAt).toBeNull();
|
||||
// the fixture's MIDI is the recording's own score (identity alignment): the constant-offset baseline wins
|
||||
expect(first.auto.chosen).toBe("baseline");
|
||||
expect(first.anchors).toHaveLength(1);
|
||||
expect(Math.abs(first.auto.baseline.offsetSec)).toBeLessThan(0.05);
|
||||
expect(first.anchors.every((a: Anchor) => a.kind === "auto")).toBe(true);
|
||||
expect(first.stretches.length).toBeGreaterThanOrEqual(first.anchors.length); // +1, minus an empty edge stretch
|
||||
expect(first.stretches[1].midiBpm).toBeGreaterThan(30);
|
||||
expect(first.stretches[1].bpm).toBeCloseTo(first.stretches[1].midiBpm / first.stretches[1].ratio, 6);
|
||||
expect(first.fit.precision).toBeGreaterThan(0.6);
|
||||
expect(first.fit.stretches.total).toBe(first.stretches.length);
|
||||
expect(["dtw", "baseline"]).toContain(first.auto.chosen);
|
||||
expect(typeof first.auto.baseline.offsetSec).toBe("number");
|
||||
expect(first.midiTempo[0].bpm).toBeGreaterThan(30);
|
||||
expect(await Bun.file(join(root, "fix", "recording.wav")).exists()).toBe(true);
|
||||
expect(await Bun.file(join(root, "fix", "peaks.json")).exists()).toBe(true);
|
||||
}, BIG);
|
||||
test("audio: cached Ogg Opus", async () => {
|
||||
const r = await api("/api/song/fix/audio");
|
||||
expect(r.status).toBe(200);
|
||||
expect(r.headers.get("content-type")).toContain("audio/ogg");
|
||||
const b = new Uint8Array(await r.arrayBuffer());
|
||||
expect(String.fromCharCode(...b.subarray(0, 4))).toBe("OggS");
|
||||
expect(await Bun.file(join(root, "fix", "audio.ogg")).exists()).toBe(true);
|
||||
}, BIG);
|
||||
test("render: RIFF WAV, mono 22.05 kHz, cached by anchors", async () => {
|
||||
const body = json({ anchors: first.anchors });
|
||||
const r = await api("/api/song/fix/render", body);
|
||||
expect(r.status).toBe(200);
|
||||
expect(r.headers.get("content-type")).toBe("audio/wav");
|
||||
const b = new Uint8Array(await r.arrayBuffer());
|
||||
const dv = new DataView(b.buffer);
|
||||
expect(String.fromCharCode(...b.subarray(0, 4))).toBe("RIFF");
|
||||
expect(dv.getUint16(22, true)).toBe(1);
|
||||
expect(dv.getUint32(24, true)).toBe(22050);
|
||||
expect(b.length).toBeGreaterThan(first.audioDur * 22050 * 2);
|
||||
const again = await api("/api/song/fix/render", body);
|
||||
expect(again.headers.get("x-cache")).toBe("hit");
|
||||
expect(again.headers.get("x-anchors-hash")).toBe(r.headers.get("x-anchors-hash"));
|
||||
}, BIG);
|
||||
test("fit: re-scores arbitrary anchors", async () => {
|
||||
const r = await api("/api/song/fix/fit", json({ anchors: [{ midiSec: 1, audioSec: 1.4, kind: "manual" }] }));
|
||||
const j = await r.json();
|
||||
expect(j.fit.windows.length).toBeGreaterThan(0);
|
||||
expect(j.fit.precision).toBeLessThan(first.fit.precision);
|
||||
expect(j.stretches.length).toBe(2);
|
||||
});
|
||||
test("bad anchors are rejected with 400", async () => {
|
||||
const bad = [{ anchors: "x" }, { anchors: [{ midiSec: 1, audioSec: 2, kind: "manual" }, { midiSec: 2, audioSec: 1, kind: "manual" }] }, { anchors: [{ midiSec: "1", audioSec: 2, kind: "manual" }] }, { anchors: [{ midiSec: 1, audioSec: 2, kind: "weird" }] }];
|
||||
for (const b of bad) expect((await api("/api/song/fix/fit", json(b))).status).toBe(400);
|
||||
expect((await api("/api/song/fix/fit", { method: "POST", body: "{not json" })).status).toBe(400);
|
||||
});
|
||||
test("auto: keeps manual anchors and re-solves autos around them", async () => {
|
||||
const manual: Anchor = { midiSec: 10, audioSec: 10, kind: "manual" };
|
||||
const r = await api("/api/song/fix/auto", json({ anchors: [manual, { midiSec: 15, audioSec: 15.5, kind: "auto" }] }));
|
||||
const j = await r.json();
|
||||
expect(j.anchors.filter((a: Anchor) => a.kind === "manual")).toEqual([manual]);
|
||||
expect(j.anchors.filter((a: Anchor) => a.kind === "auto").length).toBeGreaterThan(2);
|
||||
expect(j.fit.precision).toBeGreaterThan(0.6);
|
||||
expect(j.auto.chosen).toBe("dtw"); // manual anchors present: never the baseline
|
||||
expect(j.stretches.length).toBeGreaterThanOrEqual(j.anchors.length);
|
||||
}, BIG);
|
||||
test("reset: raw MIDI (no anchors)", async () => {
|
||||
const j = await (await api("/api/song/fix/reset", { method: "POST" })).json();
|
||||
expect(j.anchors).toEqual([]);
|
||||
expect(j.stretches).toHaveLength(1);
|
||||
expect(j.fit.windows.length).toBeGreaterThan(0);
|
||||
});
|
||||
test("PUT alignment then GET round-trips anchors; file holds anchors, savedAt, fit", async () => {
|
||||
const anchors: Anchor[] = [{ midiSec: 0.5, audioSec: 0.52, kind: "manual" }, { midiSec: 10, audioSec: 10.05, kind: "auto" }];
|
||||
const p = await api("/api/song/fix/alignment", json({ anchors }, "PUT"));
|
||||
expect(p.status).toBe(200);
|
||||
const saved = await p.json();
|
||||
expect(typeof saved.savedAt).toBe("string");
|
||||
const file = await Bun.file(join(root, "fix", "alignment.json")).json();
|
||||
expect(file.anchors).toEqual(anchors);
|
||||
expect(file.savedAt).toBe(saved.savedAt);
|
||||
expect(file.fit.windows.length).toBeGreaterThan(0);
|
||||
const g = await (await api("/api/song/fix")).json();
|
||||
expect(g.anchors).toEqual(anchors);
|
||||
expect(g.source).toBe("saved");
|
||||
expect(g.savedAt).toBe(saved.savedAt);
|
||||
expect(await (await api("/api/songs")).json()).toEqual([{ id: "fix", title: "Fixture Song", hasAlignment: true }]);
|
||||
});
|
||||
test("more than 10k anchors is rejected with 400", async () => {
|
||||
const many = Array.from({ length: 10_001 }, (_, k) => ({ midiSec: k * 0.01, audioSec: k * 0.01, kind: "auto" }));
|
||||
expect((await api("/api/song/fix/fit", json({ anchors: many }))).status).toBe(400);
|
||||
});
|
||||
test("a corrupt alignment.json does not break the song list; the song falls back to auto with a warning", async () => {
|
||||
await Bun.write(join(root, "fix", "alignment.json"), "{ not json");
|
||||
const list = await api("/api/songs");
|
||||
expect(list.status).toBe(200);
|
||||
expect(await list.json()).toEqual([{ id: "fix", title: "Fixture Song", hasAlignment: false, alignmentError: expect.stringContaining("alignment.json") }]);
|
||||
const g = await api("/api/song/fix");
|
||||
expect(g.status).toBe(200);
|
||||
const j = await g.json();
|
||||
expect(j.source).toBe("auto");
|
||||
expect(j.warning).toContain("alignment.json");
|
||||
expect(j.anchors.length).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,137 @@
|
||||
import { test, expect, describe } from "bun:test";
|
||||
import { existsSync } from "node:fs";
|
||||
import { parseSmf, writeSmf } from "../../src/align/smf.ts";
|
||||
import { guessRole } from "../../src/align/roles.ts";
|
||||
import type { RefTrack } from "../../src/align/smf.ts";
|
||||
|
||||
const vlq = (v: number): number[] => {
|
||||
const out = [v & 0x7f];
|
||||
while ((v >>= 7) > 0) out.unshift((v & 0x7f) | 0x80);
|
||||
return out;
|
||||
};
|
||||
const u32 = (v: number) => [(v >>> 24) & 255, (v >>> 16) & 255, (v >>> 8) & 255, v & 255];
|
||||
const u16 = (v: number) => [(v >> 8) & 255, v & 255];
|
||||
const ascii = (s: string) => [...s].map((c) => c.charCodeAt(0));
|
||||
const chunk = (id: string, body: number[]) => [...ascii(id), ...u32(body.length), ...body];
|
||||
const meta = (dt: number, type: number, data: number[]) => [...vlq(dt), 0xff, type, ...vlq(data.length), ...data];
|
||||
const name = (n: string) => meta(0, 3, ascii(n));
|
||||
const tempo = (dt: number, bpm: number) => { const us = Math.round(60e6 / bpm); return meta(dt, 0x51, [(us >> 16) & 255, (us >> 8) & 255, us & 255]); };
|
||||
const end = meta(0, 0x2f, []);
|
||||
const smf = (tracks: number[][], ppq = 480) => new Uint8Array([...chunk("MThd", [...u16(1), ...u16(tracks.length), ...u16(ppq)]), ...tracks.flatMap((t) => chunk("MTrk", t))]);
|
||||
|
||||
function build() {
|
||||
const conductor = [...name("tempo"), ...meta(0, 0x58, [3, 2, 24, 8]), ...tempo(0, 120), ...tempo(1920, 60), ...end];
|
||||
// guitar: program 25; note 60 at tick 0 for 480 ticks, then running-status note 62 using velocity-0 note-off
|
||||
const guitar = [
|
||||
...name("Guitar 1"), ...vlq(0), 0xc0, 25,
|
||||
...vlq(0), 0x90, 60, 100,
|
||||
...vlq(480), 0x80, 60, 0,
|
||||
...vlq(0), 0x90, 62, 90, // new status
|
||||
...vlq(480), 62, 0, // running status note-on vel 0 = off, at tick 960
|
||||
...vlq(0), 0xf0, 3, 1, 2, 0xf7, // sysex skipped
|
||||
...vlq(960), 0x90, 64, 80, // tick 1920 (tempo change point) -> 2.0 s
|
||||
...vlq(480), 64, 0, // tick 2400 -> 2.0 + 1.0 s
|
||||
...end,
|
||||
];
|
||||
const drums = [...name("Drums"), ...vlq(0), 0x99, 36, 100, ...vlq(240), 0x99, 36, 0, ...end];
|
||||
return smf([conductor, guitar, drums]);
|
||||
}
|
||||
|
||||
describe("parseSmf", () => {
|
||||
const s = parseSmf(build());
|
||||
test("tempo map and time signature", () => {
|
||||
expect(s.tempoMap).toEqual([{ sec: 0, bpm: 120 }, { sec: 2, bpm: 60 }]);
|
||||
expect(s.timeSigs).toEqual([{ sec: 0, num: 3, den: 4 }]);
|
||||
});
|
||||
test("note times in seconds across tempo change", () => {
|
||||
const g = s.tracks.find((t) => t.name === "Guitar 1")!;
|
||||
expect(g.program).toBe(25);
|
||||
expect(g.isDrums).toBe(false);
|
||||
expect(g.notes.map((n) => [n.onset, n.offset, n.pitch, n.velocity])).toEqual([
|
||||
[0, 0.5, 60, 100], [0.5, 1, 62, 90], [2, 3, 64, 80],
|
||||
]);
|
||||
expect(s.durationSec).toBeCloseTo(3, 9);
|
||||
});
|
||||
test("drums on channel 10", () => {
|
||||
const d = s.tracks.find((t) => t.name === "Drums")!;
|
||||
expect(d.isDrums).toBe(true);
|
||||
expect(d.notes).toHaveLength(1);
|
||||
expect(d.notes[0]!.channel).toBe(9);
|
||||
});
|
||||
test("conductor track without notes is dropped; index is SMF track order", () => {
|
||||
expect(s.tracks.map((t) => t.index)).toEqual([1, 2]);
|
||||
});
|
||||
test("overlapping same-pitch notes on one channel pair FIFO", () => {
|
||||
const t = [...name("x"), 0, 0x90, 60, 50, ...vlq(100), 0x90, 60, 60, ...vlq(100), 0x80, 60, 0, ...vlq(100), 0x80, 60, 0, ...end];
|
||||
const n = parseSmf(smf([t], 480)).tracks[0]!.notes;
|
||||
expect(n.map((x) => [x.velocity, x.offset > x.onset])).toEqual([[50, true], [60, true]]);
|
||||
expect(n[0]!.offset).toBeLessThan(n[1]!.offset);
|
||||
expect(n[0]!.onset).toBeLessThan(n[1]!.onset);
|
||||
});
|
||||
test("same pitch on different channels are independent", () => {
|
||||
const t = [0, 0x90, 60, 50, 0, 0x91, 60, 50, ...vlq(480), 0x81, 60, 0, ...vlq(480), 0x80, 60, 0, ...end];
|
||||
const n = parseSmf(smf([t])).tracks[0]!.notes;
|
||||
expect(n.map((x) => [x.channel, x.offset]).sort()).toEqual([[0, 1], [1, 0.5]]);
|
||||
});
|
||||
test("rejects non-SMF", () => {
|
||||
expect(() => parseSmf(new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]))).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("guessRole", () => {
|
||||
const t = (o: Partial<RefTrack>): RefTrack => ({ index: 0, name: "", program: -1, isDrums: false, notes: [], ...o });
|
||||
test("by drums / name / program", () => {
|
||||
expect(guessRole(t({ isDrums: true }))).toBe("drums");
|
||||
expect(guessRole(t({ name: "Lead Vocals" }))).toBe("vocals");
|
||||
expect(guessRole(t({ name: "Bass" }))).toBe("bass");
|
||||
expect(guessRole(t({ name: "Billie Joe Armstrong | Rhythm Guitar" }))).toBe("guitar");
|
||||
expect(guessRole(t({ name: "Acoustic Guitar " }))).toBe("guitar");
|
||||
expect(guessRole(t({ name: "Delay Lead" }))).toBe("guitar");
|
||||
expect(guessRole(t({ name: "Piano" }))).toBe("keys");
|
||||
expect(guessRole(t({ name: "Synth Pad" }))).toBe("keys");
|
||||
expect(guessRole(t({ name: "", program: 25 }))).toBe("guitar");
|
||||
expect(guessRole(t({ name: "", program: 33 }))).toBe("bass");
|
||||
});
|
||||
test("unknown -> null", () => {
|
||||
expect(guessRole(t({ name: "Brass", program: 61 }))).toBeNull();
|
||||
expect(guessRole(t({ name: "Crowd", program: 126 }))).toBeNull();
|
||||
expect(guessRole(t({ name: "" }))).toBeNull();
|
||||
});
|
||||
test("integration with parsed SMF", () => {
|
||||
const s = parseSmf(build());
|
||||
expect(s.tracks.map(guessRole)).toEqual(["guitar", "drums"]);
|
||||
});
|
||||
});
|
||||
|
||||
const REAL = "Passenger-Holes-10-24-2016.mid";
|
||||
describe("real GP export", () => {
|
||||
test.skipIf(!existsSync(REAL))("Holes parses; Brass/Crowd unmapped", async () => {
|
||||
const s = parseSmf(new Uint8Array(await Bun.file(REAL).arrayBuffer()));
|
||||
expect(s.tracks).toHaveLength(7);
|
||||
for (const t of s.tracks) {
|
||||
if (/^(brass|crowd)/i.test(t.name.trim())) expect(guessRole(t)).toBeNull();
|
||||
}
|
||||
expect(s.durationSec).toBeGreaterThan(60);
|
||||
});
|
||||
});
|
||||
|
||||
describe("writeSmf", () => {
|
||||
test("round-trips tracks, programs, drums, tempo and note times through parseSmf", () => {
|
||||
const tracks = [
|
||||
{ name: "Guitar", program: 25, isDrums: false, notes: [{ onset: 0, offset: 0.5, pitch: 60, velocity: 100 }, { onset: 0.25, offset: 1.5, pitch: 64, velocity: 80 }] },
|
||||
{ name: "Drums", program: 0, isDrums: true, notes: [{ onset: 1, offset: 1.1, pitch: 36, velocity: 110 }] },
|
||||
];
|
||||
const s = parseSmf(writeSmf(tracks, 90));
|
||||
expect(s.tempoMap).toEqual([{ sec: 0, bpm: expect.closeTo(90, 3) }]);
|
||||
expect(s.tracks.map((t) => [t.name, t.program, t.isDrums])).toEqual([["Guitar", 25, false], ["Drums", 0, true]]);
|
||||
const g = s.tracks[0]!.notes;
|
||||
expect(g.map((n) => n.pitch)).toEqual([60, 64]);
|
||||
for (const [k, n] of tracks[0]!.notes.entries()) {
|
||||
expect(g[k]!.onset).toBeCloseTo(n.onset, 3);
|
||||
expect(g[k]!.offset).toBeCloseTo(n.offset, 3);
|
||||
expect(g[k]!.velocity).toBe(n.velocity);
|
||||
}
|
||||
expect(s.tracks[1]!.notes[0]!.channel).toBe(9);
|
||||
expect(s.durationSec).toBeCloseTo(1.5, 3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,25 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { avgBpm, stretchInfo, stretchLabel } from "../../src/align/tempo.ts";
|
||||
import type { Stretch } from "../../src/align/warp.ts";
|
||||
|
||||
const map = [{ sec: 0, bpm: 120 }, { sec: 10, bpm: 60 }];
|
||||
|
||||
test("avgBpm: beats over seconds across tempo changes", () => {
|
||||
expect(avgBpm(map, 0, 10)).toBeCloseTo(120, 9);
|
||||
expect(avgBpm(map, 10, 20)).toBeCloseTo(60, 9);
|
||||
expect(avgBpm(map, 5, 15)).toBeCloseTo((10 + 5) / 10 * 60, 9); // 10 beats + 5 beats in 10 s = 90 bpm
|
||||
expect(avgBpm([], 0, 4)).toBeCloseTo(120, 9); // implicit 120
|
||||
expect(avgBpm(map, 3, 3)).toBeCloseTo(120, 9); // empty span: tempo at that point
|
||||
});
|
||||
|
||||
test("stretchInfo: effective bpm = MIDI bpm / ratio, signed percent", () => {
|
||||
const s: Stretch[] = [{ fromMidi: 0, toMidi: 10, ratio: 120 / 121.68, warn: false }, { fromMidi: 10, toMidi: 20, ratio: 60 / 55, warn: true }];
|
||||
const [a, b] = stretchInfo(s, map);
|
||||
expect(a!.midiBpm).toBeCloseTo(120, 9);
|
||||
expect(a!.bpm).toBeCloseTo(121.68, 6);
|
||||
expect(a!.pct).toBeCloseTo(1.4, 6);
|
||||
expect(b!.bpm).toBeCloseTo(55, 6);
|
||||
expect(stretchLabel(a!)).toBe("120.0 → 121.7 bpm (+1.4%)");
|
||||
expect(stretchLabel(b!)).toBe("60.0 → 55.0 bpm (−8.3%) ⚠");
|
||||
expect(stretchLabel({ ...a!, bpm: 120, pct: 0.01 })).toBe("120.0 → 120.0 bpm");
|
||||
});
|
||||
@@ -0,0 +1,239 @@
|
||||
import { test, expect, describe } from "bun:test";
|
||||
import { makeWarp, stretches, addManual, moveAnchor, deleteAnchor, type Anchor } from "../../src/align/warp.ts";
|
||||
|
||||
const M = (midiSec: number, audioSec: number): Anchor => ({ midiSec, audioSec, kind: "manual" });
|
||||
const A = (midiSec: number, audioSec: number): Anchor => ({ midiSec, audioSec, kind: "auto" });
|
||||
const close = (a: number, b: number, eps = 1e-9) => expect(Math.abs(a - b)).toBeLessThan(eps);
|
||||
|
||||
describe("makeWarp", () => {
|
||||
test("zero anchors is the identity", () => {
|
||||
const w = makeWarp([]);
|
||||
expect(w.anchors).toEqual([]);
|
||||
for (const x of [-2, 0, 3.5, 100]) {
|
||||
expect(w.toAudio(x)).toBe(x);
|
||||
expect(w.toMidi(x)).toBe(x);
|
||||
}
|
||||
});
|
||||
|
||||
test("single anchor is a pure offset", () => {
|
||||
const w = makeWarp([M(1, 1.5)]);
|
||||
expect(w.toAudio(1)).toBe(1.5);
|
||||
expect(w.toAudio(3)).toBe(3.5);
|
||||
expect(w.toAudio(-1)).toBe(-0.5);
|
||||
expect(w.toMidi(1.5)).toBe(1);
|
||||
expect(w.toMidi(0)).toBe(-0.5);
|
||||
});
|
||||
|
||||
test("two anchors interpolate linearly and extrapolate with the same slope", () => {
|
||||
const w = makeWarp([M(1, 1), M(3, 5)]); // slope 2
|
||||
expect(w.toAudio(2)).toBe(3);
|
||||
expect(w.toAudio(0)).toBe(-1);
|
||||
expect(w.toAudio(5)).toBe(9);
|
||||
expect(w.toMidi(3)).toBe(2);
|
||||
});
|
||||
|
||||
test("three anchors: outer segments use the nearest interior slope", () => {
|
||||
const w = makeWarp([M(1, 1), M(3, 5), M(5, 6)]); // slopes 2, then 0.5
|
||||
expect(w.toAudio(4)).toBe(5.5); // interior
|
||||
expect(w.toAudio(0)).toBe(-1); // before first: slope 2
|
||||
expect(w.toAudio(7)).toBe(7); // after last: slope 0.5 -> 6 + 2*0.5
|
||||
});
|
||||
|
||||
test("toMidi(toAudio(x)) round-trips", () => {
|
||||
const w = makeWarp([M(0.5, 0.2), A(2, 2.9), M(4, 4.1), A(9, 8.0)]);
|
||||
for (let x = -3; x <= 12; x += 0.37) close(w.toMidi(w.toAudio(x)), x);
|
||||
for (let y = -3; y <= 12; y += 0.41) close(w.toAudio(w.toMidi(y)), y);
|
||||
});
|
||||
|
||||
test("sorts anchors by midiSec and keeps them as copies", () => {
|
||||
const input = [M(3, 5), M(1, 1)];
|
||||
const w = makeWarp(input);
|
||||
expect(w.anchors.map((a) => a.midiSec)).toEqual([1, 3]);
|
||||
expect(w.toAudio(2)).toBe(3);
|
||||
input[0]!.audioSec = 999;
|
||||
expect(w.toAudio(3)).toBe(5);
|
||||
});
|
||||
|
||||
test("throws when audioSec is not strictly increasing with midiSec", () => {
|
||||
expect(() => makeWarp([M(1, 2), M(3, 1)])).toThrow();
|
||||
expect(() => makeWarp([M(1, 2), M(3, 2)])).toThrow();
|
||||
});
|
||||
|
||||
test("throws on duplicate midiSec", () => {
|
||||
expect(() => makeWarp([M(1, 1), M(1, 2)])).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("stretches", () => {
|
||||
test("no anchors: one stretch with ratio 1", () => {
|
||||
expect(stretches([], 10)).toEqual([{ fromMidi: 0, toMidi: 10, ratio: 1, warn: false }]);
|
||||
});
|
||||
|
||||
test("single anchor: two ratio-1 edge stretches split at the anchor", () => {
|
||||
expect(stretches([M(4, 4.2)], 10)).toEqual([
|
||||
{ fromMidi: 0, toMidi: 4, ratio: 1, warn: false },
|
||||
{ fromMidi: 4, toMidi: 10, ratio: 1, warn: false },
|
||||
]);
|
||||
});
|
||||
|
||||
test("interior stretches get their ratio; edges take the nearest interior ratio", () => {
|
||||
const s = stretches([M(1, 1), M(3, 5), M(5, 6)], 7);
|
||||
expect(s.map((x) => [x.fromMidi, x.toMidi])).toEqual([
|
||||
[0, 1],
|
||||
[1, 3],
|
||||
[3, 5],
|
||||
[5, 7],
|
||||
]);
|
||||
expect(s.map((x) => x.ratio)).toEqual([2, 2, 0.5, 0.5]);
|
||||
expect(s.map((x) => x.warn)).toEqual([true, true, true, true]);
|
||||
});
|
||||
|
||||
test("warn when tempo changes by more than 5%", () => {
|
||||
const hot = stretches([M(0, 0), M(10, 10.7)], 10); // ratio 1.07 -> 1/1.07-1 = -6.5%
|
||||
expect(hot.length).toBe(1);
|
||||
close(hot[0]!.ratio, 1.07);
|
||||
expect(hot[0]!.warn).toBe(true);
|
||||
const mild = stretches([M(0, 0), M(10, 10.4)], 10); // ratio 1.04 -> -3.8%
|
||||
expect(mild[0]!.warn).toBe(false);
|
||||
});
|
||||
|
||||
test("drops zero-length edge stretches when an anchor sits at 0 or at the end", () => {
|
||||
const s = stretches([M(0, 0.5), M(10, 10.5)], 10);
|
||||
expect(s).toEqual([{ fromMidi: 0, toMidi: 10, ratio: 1, warn: false }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("addManual", () => {
|
||||
test("adds a manual anchor into an empty list", () => {
|
||||
expect(addManual([], { midiSec: 2, audioSec: 2.5 })).toEqual([M(2, 2.5)]);
|
||||
});
|
||||
|
||||
test("removes autos near the new anchor (both within 10 s here), never past a manual", () => {
|
||||
// M0 | A2 | A4 | M8 ; insert manual at 6 -> both autos between M0 and M8 go
|
||||
const base = [M(0, 0), A(2, 2), A(4, 4.5), M(8, 8)];
|
||||
expect(addManual(base, { midiSec: 6, audioSec: 6.2 })).toEqual([M(0, 0), M(6, 6.2), M(8, 8)]);
|
||||
});
|
||||
|
||||
test("autos every 2 s: adding a manual at 20 s removes autos between manuals at 10 s and 30 s only", () => {
|
||||
const autos = [2, 4, 6, 8, 12, 14, 16, 18, 22, 24, 26, 28].map((t) => A(t, t));
|
||||
const base = [...autos.slice(0, 4), M(10, 10), M(30, 30), ...autos.slice(4)].sort((x, y) => x.midiSec - y.midiSec);
|
||||
// autos 12..28 sit between the manuals at 10 and 30 and go; 2..8 (before 10) are kept
|
||||
expect(addManual(base, { midiSec: 20, audioSec: 20 })).toEqual([
|
||||
A(2, 2), A(4, 4), A(6, 6), A(8, 8), M(10, 10), M(20, 20), M(30, 30),
|
||||
]);
|
||||
});
|
||||
|
||||
test("moving a manual anchor across several autos succeeds and removes the crossed autos", () => {
|
||||
// M0 | A10 | A20 | M30 | A40 | M60 ; move M30 -> midi 45 (crosses A40, stays before M60)
|
||||
const base = [M(0, 0), A(10, 10), A(20, 20), M(30, 30), A(40, 40), M(60, 60)];
|
||||
// A40 is within 10 s of the new position, A20 within 10 s of the old one; A10 is further than 10 s from both and kept
|
||||
expect(moveAnchor(base, 3, { midiSec: 45, audioSec: 45 })).toEqual([M(0, 0), A(10, 10), M(45, 45), M(60, 60)]);
|
||||
});
|
||||
|
||||
test("autos every 2 s, no manuals: a manual at 60 s removes only the autos within 10 s (50..70)", () => {
|
||||
const base = Array.from({ length: 60 }, (_, k) => A(2 * k + 1, 2 * k + 1)); // 1, 3, ..., 119
|
||||
const out = addManual(base, { midiSec: 60, audioSec: 60 });
|
||||
expect(out.filter((a) => a.kind === "auto").map((a) => a.midiSec)).toEqual(base.map((a) => a.midiSec).filter((m) => m < 50 || m > 70));
|
||||
expect(out.find((a) => a.kind === "manual")).toEqual(M(60, 60));
|
||||
});
|
||||
|
||||
test("autos outside the 10 s window that would break audio monotonicity are removed too", () => {
|
||||
const base = Array.from({ length: 60 }, (_, k) => A(2 * k + 1, 2 * k + 1));
|
||||
// a manual at midi 60 placed at audio 75: autos at midi 71, 73 (outside the window) sit at audio < 75 after it
|
||||
const out = addManual(base, { midiSec: 60, audioSec: 75 });
|
||||
const autos = out.filter((a) => a.kind === "auto").map((a) => a.midiSec);
|
||||
expect(autos).not.toContain(71);
|
||||
expect(autos).not.toContain(73);
|
||||
expect(autos).toContain(75 + 2); // 77 > 75 stays
|
||||
expect(autos).toContain(49);
|
||||
expect(() => makeWarp(out)).not.toThrow();
|
||||
});
|
||||
|
||||
test("a drag that crosses an auto outside the window removes it via the monotonicity rule", () => {
|
||||
const base = [...Array.from({ length: 30 }, (_, k) => A(2 * k + 1, 2 * k + 1)).filter((a) => a.midiSec !== 31), M(30, 30)].sort((x, y) => x.midiSec - y.midiSec);
|
||||
const i = base.findIndex((a) => a.kind === "manual");
|
||||
// keep MIDI 30, drag the recording end to 45 s: autos at midi 41, 43 (window is 20..40) now lie before it in audio
|
||||
const out = moveAnchor(base, i, { midiSec: 30, audioSec: 45 });
|
||||
const autos = out.filter((a) => a.kind === "auto").map((a) => a.midiSec);
|
||||
expect(autos.filter((m) => m >= 20 && m <= 44)).toEqual([]);
|
||||
expect(autos).toContain(47);
|
||||
expect(autos).toContain(19);
|
||||
expect(() => makeWarp(out)).not.toThrow();
|
||||
});
|
||||
|
||||
test("auto runs are removed even when they are the only anchors besides the moved one", () => {
|
||||
const base = [A(2, 2), A(4, 4), A(6, 6)];
|
||||
expect(addManual(base, { midiSec: 5, audioSec: 5 })).toEqual([M(5, 5)]);
|
||||
});
|
||||
|
||||
test("keeps manual neighbours untouched", () => {
|
||||
const base = [M(0, 0), M(8, 8)];
|
||||
expect(addManual(base, { midiSec: 4, audioSec: 4 })).toEqual([M(0, 0), M(4, 4), M(8, 8)]);
|
||||
});
|
||||
|
||||
test("removes auto neighbours on both sides", () => {
|
||||
const base = [M(0, 0), A(2, 2), A(6, 6.5), M(10, 10)];
|
||||
expect(addManual(base, { midiSec: 4, audioSec: 4.1 })).toEqual([M(0, 0), M(4, 4.1), M(10, 10)]);
|
||||
});
|
||||
|
||||
test("throws if the new anchor would cross a neighbour's audio time", () => {
|
||||
const base = [M(0, 0), M(8, 8)];
|
||||
expect(() => addManual(base, { midiSec: 4, audioSec: 9 })).toThrow();
|
||||
expect(() => addManual(base, { midiSec: 4, audioSec: -1 })).toThrow();
|
||||
});
|
||||
|
||||
test("throws on an existing midi position", () => {
|
||||
expect(() => addManual([M(4, 4)], { midiSec: 4, audioSec: 4.5 })).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("moveAnchor", () => {
|
||||
test("moves an anchor, makes it manual and removes adjacent autos", () => {
|
||||
const base = [M(0, 0), A(3, 3), M(6, 6), A(9, 9)];
|
||||
// move M(6) to midi 5 audio 5.5; immediate auto neighbours: A(3) left (removed), A(9) right (removed)
|
||||
expect(moveAnchor(base, 2, { midiSec: 5, audioSec: 5.5 })).toEqual([M(0, 0), M(5, 5.5)]);
|
||||
});
|
||||
|
||||
test("turns an auto anchor into a manual one", () => {
|
||||
const base = [M(0, 0), A(4, 4.5), M(8, 8)];
|
||||
expect(moveAnchor(base, 1, { midiSec: 4.2, audioSec: 4.6 })).toEqual([M(0, 0), M(4.2, 4.6), M(8, 8)]);
|
||||
});
|
||||
|
||||
test("throws when moved past a manual neighbour", () => {
|
||||
const base = [M(0, 0), M(4, 4), M(8, 8)];
|
||||
expect(() => moveAnchor(base, 1, { midiSec: 9, audioSec: 4 })).toThrow();
|
||||
expect(() => moveAnchor(base, 1, { midiSec: 4, audioSec: 8.5 })).toThrow();
|
||||
});
|
||||
|
||||
test("moving past an auto neighbour is allowed; the crossed auto is removed", () => {
|
||||
const base = [M(0, 0), A(4, 4), M(8, 8)];
|
||||
expect(moveAnchor(base, 2, { midiSec: 3, audioSec: 3 })).toEqual([M(0, 0), M(3, 3)]);
|
||||
});
|
||||
|
||||
test("crossing a manual neighbour in audio time throws even with an auto between", () => {
|
||||
const base = [M(0, 0), A(4, 4), M(8, 8), A(12, 12), M(16, 16)];
|
||||
// Moving M(8) to audio 17 puts it past its right manual neighbour M(16) (audio 16); autos between are ignored.
|
||||
expect(() => moveAnchor(base, 2, { midiSec: 10, audioSec: 17 })).toThrow();
|
||||
});
|
||||
|
||||
test("throws on a bad index", () => {
|
||||
expect(() => moveAnchor([M(1, 1)], 3, { midiSec: 2, audioSec: 2 })).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe("deleteAnchor", () => {
|
||||
test("removes the anchor at index", () => {
|
||||
expect(deleteAnchor([M(1, 1), A(2, 2), M(3, 3)], 1)).toEqual([M(1, 1), M(3, 3)]);
|
||||
});
|
||||
|
||||
test("throws on a bad index", () => {
|
||||
expect(() => deleteAnchor([M(1, 1)], 1)).toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
test("extrapolation before the first anchor uses the first stretch's slope (no implicit 0 -> 0 anchor)", () => {
|
||||
const w = makeWarp([{ midiSec: 10, audioSec: 12, kind: "auto" }, { midiSec: 20, audioSec: 23, kind: "auto" }]); // slope 1.1
|
||||
expect(w.toAudio(0)).toBeCloseTo(12 - 10 * 1.1, 9);
|
||||
expect(w.toMidi(12 - 10 * 1.1)).toBeCloseTo(0, 9);
|
||||
expect(makeWarp([{ midiSec: 10, audioSec: 12, kind: "auto" }]).toAudio(0)).toBe(2); // single anchor: slope 1
|
||||
});
|
||||
@@ -0,0 +1,113 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { renderRealReport, runReal, type RealSongResult } from "../../src/bench/real.ts";
|
||||
import { buildSong, loadSongDir } from "../../src/bench/dataset.ts";
|
||||
import { writeSmf } from "../../src/align/smf.ts";
|
||||
import { writeAlignment, type SongEntry } from "../../src/ref/manifest.ts";
|
||||
import { createPipeline } from "../../src/pipeline/run.ts";
|
||||
import { scoreSong } from "../../src/eval/score.ts";
|
||||
import type { Role } from "../../src/core/types.ts";
|
||||
import type { FitReport, FitWindow } from "../../src/align/fit.ts";
|
||||
|
||||
const hasReference = await Bun.file("reference/songs.json").exists();
|
||||
const mr = (f1: number | null, tp = 8, fp = 2, fn = 2) => ({ tp, fp, fn, precision: f1, recall: f1, f1, octaveErrors: 0, matches: [] as [number, number][] });
|
||||
const win = (startSec: number, precision: number, reasons: string[]): FitWindow => ({ startSec, recOnsets: 40, precision, lift: 0.6, audibleRate: 0.4, medianResidualMs: 12.3, beatShift: null, flagged: reasons.length > 0, reasons });
|
||||
const fit = (p: number): FitReport => ({
|
||||
precision: p, lift: 0.62, audibleRate: 0.38, medianResidualMs: 12.3,
|
||||
windows: [win(0, p, []), win(30, 0.5, ["precision 50%"])], stretches: { total: 41, warned: 4, over15: 2 },
|
||||
});
|
||||
const base = { offsetSec: 0.06, precision: 0.88 };
|
||||
const fixture = (): RealSongResult[] => [
|
||||
{
|
||||
id: "a", title: "Song A", alignment: { source: "computed", manual: 0, auto: 40, savedAt: null, chosen: "dtw", baseline: base, dtwPrecision: 0.8 }, fit: fit(0.8),
|
||||
perRole: { bass: mr(0.5), guitar: mr(0.9), keys: mr(null, 0, 0, 0) }, lenient: { bass: 0.6, guitar: 0.95, keys: null },
|
||||
drumsPerClass: null, excludedTracks: ["Click", "Pad"], capo: { Lead: 2 }, seconds: 3,
|
||||
},
|
||||
{
|
||||
id: "b", title: "Song B", alignment: { source: "saved", manual: 0, auto: 1, savedAt: "2026-10-29T10:00:00Z", chosen: "baseline", baseline: base, dtwPrecision: 0.85 }, fit: fit(0.6),
|
||||
perRole: { guitar: mr(0.3) }, lenient: { guitar: 0.4 }, drumsPerClass: null, excludedTracks: [], capo: {}, seconds: 4,
|
||||
},
|
||||
];
|
||||
|
||||
describe("renderRealReport", () => {
|
||||
const md = renderRealReport("fx", fixture(), [{ id: "c", reason: "reference/c/alignment.json: not valid JSON" }, { id: "d", reason: "no alignment.json" }]);
|
||||
test("guitar row precedes the other roles; source, method, baseline, fit and stretch warnings shown", () => {
|
||||
expect(md).toContain("Song A");
|
||||
expect(md).toContain("computed this run");
|
||||
expect(md).toContain("saved 2026-10-29T10:00:00Z");
|
||||
expect(md).toContain("0 manual / 40 auto");
|
||||
expect(md).not.toMatch(/manual \(/);
|
||||
expect(md).toContain("chosen: DTW (precision 80.0%)");
|
||||
expect(md).toContain("chosen: baseline (DTW precision 85.0% not better");
|
||||
expect(md).toContain("constant-offset baseline +0.06 s: precision 88.0%");
|
||||
expect(md).toContain("precision 80.0% (lift +0.62)");
|
||||
expect(md).toContain("MIDI onsets audible as attacks 38.0%");
|
||||
expect(md).toContain("1/2 windows flagged (0:30 precision 50%)");
|
||||
expect(md).toContain("stretches: 4/41 warned, 2 > 15%");
|
||||
expect(md).toContain("consistency of the alignment");
|
||||
expect(md).toContain("c (reference/c/alignment.json: not valid JSON)");
|
||||
expect(md).toContain("d (no alignment.json)");
|
||||
const a = md.slice(md.indexOf("Song A"), md.indexOf("Song B"));
|
||||
expect(a.indexOf("| guitar")).toBeGreaterThan(-1);
|
||||
expect(a.indexOf("| guitar")).toBeLessThan(a.indexOf("| bass"));
|
||||
expect(a.indexOf("| bass")).toBeLessThan(a.indexOf("| keys"));
|
||||
expect(a).toContain("0.90");
|
||||
expect(a).toContain("0.95");
|
||||
expect(a).toContain("Click, Pad");
|
||||
expect(a).toContain("Lead: 2");
|
||||
});
|
||||
});
|
||||
|
||||
describe("runReal", () => {
|
||||
test("t1a fixture with identity alignment matches the synthetic full-mode score", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "stsheets-real-"));
|
||||
try {
|
||||
const dir = await buildSong("t1a", 1, join(root, "data"));
|
||||
const { truth } = await loadSongDir(dir);
|
||||
const prog: Record<Role, number> = { guitar: 25, bass: 33, keys: 0, vocals: 53, drums: 0 };
|
||||
const tracks = truth.tracks.map((t) => ({ name: t.name, program: prog[t.role], isDrums: t.role === "drums", notes: t.notes }));
|
||||
const ref = join(root, "ref");
|
||||
await Bun.write(join(ref, "t1a-1", "song.mid"), writeSmf(tracks, truth.bpm));
|
||||
const entry: SongEntry = {
|
||||
id: "t1a-1", title: "Synthetic t1a/1", audio: join(dir, "mix.wav"), midi: join(ref, "t1a-1", "song.mid"),
|
||||
roles: Object.fromEntries(truth.tracks.map((t) => [t.name, t.role])), capo: {},
|
||||
};
|
||||
await Bun.write(join(ref, "songs.json"), JSON.stringify([entry, { ...entry, id: "noalign", title: "No alignment" }, { ...entry, id: "corrupt", title: "Corrupt alignment" }]));
|
||||
await Bun.write(join(ref, "corrupt", "alignment.json"), "{ not json");
|
||||
const dur = 600;
|
||||
await writeAlignment("t1a-1", { anchors: [{ midiSec: 0, audioSec: 0, kind: "manual" }, { midiSec: dur, audioSec: dur, kind: "manual" }], savedAt: "2026-10-30T00:00:00Z", fit: null }, ref);
|
||||
|
||||
const results = await runReal({ runId: "t", auto: false, root: ref, out: join(root, "runs"), resultsDir: join(root, "results") });
|
||||
expect(results.map((r) => r.id)).toEqual(["t1a-1"]);
|
||||
const r = results[0]!;
|
||||
expect(r.alignment).toEqual({ source: "saved", manual: 2, auto: 0, savedAt: "2026-10-30T00:00:00Z", chosen: null, baseline: expect.any(Object), dtwPrecision: null });
|
||||
expect(Math.abs(r.alignment.baseline.offsetSec)).toBeLessThan(0.05); // identity alignment
|
||||
expect(r.fit.precision).not.toBeNull();
|
||||
expect(r.fit.stretches).toBeDefined();
|
||||
|
||||
const pipe = await createPipeline({
|
||||
demucs: new Uint8Array(await Bun.file("models/demucs/htdemucs_6s.onnx").arrayBuffer()),
|
||||
basicPitch: new Uint8Array(await Bun.file("models/basic-pitch/nmp.onnx").arrayBuffer()),
|
||||
});
|
||||
const flags = { oracleStems: false, oracleTempo: false, oracleNotes: false };
|
||||
const loaded = await loadSongDir(dir);
|
||||
const synth = scoreSong(truth, (await pipe({ mix: loaded.mix }, flags)).output, flags);
|
||||
expect(Math.abs((r.perRole.guitar!.f1 ?? 0) - (synth.guitarMerged.f1 ?? 0))).toBeLessThan(0.02);
|
||||
|
||||
for (const f of ["notes.json", "out.gp", "score.json"]) expect(await Bun.file(join(root, "runs", "real-t", "t1a-1", f)).exists()).toBe(true);
|
||||
const json = await Bun.file(join(root, "results", "real-t.json")).text();
|
||||
expect(json).not.toContain('"matches"');
|
||||
const md = await Bun.file(join(root, "results", "real-t.md")).text();
|
||||
expect(md).toContain("Synthetic t1a/1");
|
||||
expect(md).toMatch(/corrupt \(.*alignment\.json.*not valid JSON/);
|
||||
expect(md).toContain("noalign (no alignment.json; pass --auto to compute one)");
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
}, 600_000);
|
||||
|
||||
test.skipIf(!hasReference)("real reference manifest loads", async () => {
|
||||
const { readManifest } = await import("../../src/ref/manifest.ts");
|
||||
expect((await readManifest()).length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
import { test, expect } from "bun:test";
|
||||
import { resample } from "../../src/dsp/resample.ts";
|
||||
|
||||
// Reference: the direct windowed-sinc evaluation (the original implementation), used to check the cached path is exact.
|
||||
function reference(x: Float32Array, from: number, to: number): Float32Array {
|
||||
const sinc = (v: number) => (v === 0 ? 1 : Math.sin(Math.PI * v) / (Math.PI * v));
|
||||
const ratio = to / from, outLen = Math.round(x.length * ratio), out = new Float32Array(outLen);
|
||||
const cutoff = Math.min(1, ratio), half = 16 / cutoff;
|
||||
for (let j = 0; j < outLen; j++) {
|
||||
const t = j / ratio;
|
||||
const lo = Math.max(0, Math.ceil(t - half)), hi = Math.min(x.length - 1, Math.floor(t + half));
|
||||
let acc = 0, wsum = 0;
|
||||
for (let i = lo; i <= hi; i++) {
|
||||
const d = i - t;
|
||||
const w = 0.42 + 0.5 * Math.cos((Math.PI * d) / half) + 0.08 * Math.cos((2 * Math.PI * d) / half);
|
||||
const k = cutoff * sinc(cutoff * d) * w;
|
||||
acc += x[i]! * k; wsum += k;
|
||||
}
|
||||
out[j] = wsum !== 0 ? acc / wsum : 0;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const noise = (n: number) => { let s = 1; return Float32Array.from({ length: n }, () => ((s = (s * 16807) % 2147483647) / 2147483647) * 2 - 1); };
|
||||
|
||||
test("2:1 decimation is bit-identical to the direct evaluation and much faster", () => {
|
||||
const x = noise(44100);
|
||||
expect(Array.from(resample(x, 44100, 22050))).toEqual(Array.from(reference(x, 44100, 22050)));
|
||||
const big = noise(44100 * 10);
|
||||
const time = (f: () => unknown) => { const t0 = performance.now(); f(); return performance.now() - t0; };
|
||||
time(() => resample(big, 44100, 22050)); // warm-up (JIT)
|
||||
const fast = time(() => resample(big, 44100, 22050)), slow = time(() => reference(big, 44100, 22050));
|
||||
// measured ~40x; a 4x margin keeps this robust on a loaded machine
|
||||
expect(fast * 4).toBeLessThan(slow);
|
||||
});
|
||||
|
||||
test("non-integer ratios are unchanged", () => {
|
||||
const x = noise(4800);
|
||||
expect(Array.from(resample(x, 48000, 44100))).toEqual(Array.from(reference(x, 48000, 44100)));
|
||||
expect(Array.from(resample(x, 22050, 44100))).toEqual(Array.from(reference(x, 22050, 44100)));
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import type { Grid, TimedNote } from "../../src/core/types.ts";
|
||||
import { repeatVote } from "../../src/pipeline/repeat.ts";
|
||||
|
||||
const grid: Grid = { bpm: 120, firstDownbeat: 0, beatsPerBar: 4 }; // 2 s bars
|
||||
const riff: [number, number][] = [[0, 52], [0.5, 55], [1, 57], [1.5, 59], [1.75, 60]]; // [pos s, pitch]
|
||||
|
||||
function bars(n: number, f: (bar: number, pos: number, pitch: number) => boolean = () => true): TimedNote[] {
|
||||
const out: TimedNote[] = [];
|
||||
for (let b = 0; b < n; b++) for (const [pos, pitch] of riff) {
|
||||
if (f(b, pos, pitch)) out.push({ onset: b * 2 + pos, offset: b * 2 + pos + 0.2, pitch });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
const key = (n: TimedNote) => `${n.onset.toFixed(3)}:${n.pitch}`;
|
||||
|
||||
test("adds a note missing from one repeat", () => {
|
||||
const notes = bars(6, (b, pos) => !(b === 3 && pos === 1));
|
||||
const out = repeatVote(notes, grid);
|
||||
expect(out.length).toBe(30);
|
||||
const added = out.find((n) => Math.abs(n.onset - 7) < 1e-6 && n.pitch === 57);
|
||||
expect(added).toBeDefined();
|
||||
expect(added!.offset - added!.onset).toBeCloseTo(0.2, 6);
|
||||
});
|
||||
|
||||
test("drops a note that never recurs in a bar's repeats", () => {
|
||||
const notes = [...bars(6), { onset: 4.25, offset: 4.4, pitch: 70 }];
|
||||
const out = repeatVote(notes, grid);
|
||||
expect(out.map(key)).toEqual(bars(6).map(key));
|
||||
});
|
||||
|
||||
test("leaves non-repeating material unchanged", () => {
|
||||
const notes: TimedNote[] = [];
|
||||
for (let b = 0; b < 6; b++) notes.push({ onset: b * 2 + 0.3 * (b % 4), offset: b * 2 + 1, pitch: 45 + 3 * b });
|
||||
expect(repeatVote(notes, grid).map(key)).toEqual(notes.map(key));
|
||||
});
|
||||
|
||||
test("add/drop can be disabled and empty input is handled", () => {
|
||||
const notes = [...bars(6, (b, pos) => !(b === 3 && pos === 1)), { onset: 4.25, offset: 4.4, pitch: 70 }];
|
||||
expect(repeatVote(notes, grid, { add: false, drop: false }).length).toBe(notes.length);
|
||||
expect(repeatVote([], grid)).toEqual([]);
|
||||
});
|
||||
|
||||
test("notes before a late first downbeat are voted too", () => {
|
||||
// Grid downbeat at 4.3 s; the riff starts at 0.3 s, so bars -2 and -1 precede the first downbeat.
|
||||
const g: Grid = { ...grid, firstDownbeat: 4.3 };
|
||||
const notes = bars(6, (b, pos) => !(b === 0 && pos === 0.5)).map((n) => ({ ...n, onset: n.onset + 0.3, offset: n.offset + 0.3 }));
|
||||
expect(notes[0]!.onset).toBeLessThan(g.firstDownbeat);
|
||||
const out = repeatVote(notes, g);
|
||||
expect(out.length).toBe(30);
|
||||
expect(out.some((n) => Math.abs(n.onset - 0.8) < 1e-6 && n.pitch === 55)).toBe(true);
|
||||
});
|
||||
|
||||
test("non-finite or zero grid returns an unchanged copy", () => {
|
||||
const notes = bars(6, (b, pos) => !(b === 3 && pos === 1));
|
||||
for (const g of [
|
||||
{ ...grid, bpm: 0 }, { ...grid, bpm: NaN }, { ...grid, bpm: Infinity },
|
||||
{ ...grid, firstDownbeat: NaN }, { ...grid, firstDownbeat: -Infinity }, { ...grid, beatsPerBar: NaN },
|
||||
]) {
|
||||
const out = repeatVote(notes, g);
|
||||
expect(out).toEqual(notes);
|
||||
expect(out).not.toBe(notes);
|
||||
}
|
||||
});
|
||||
|
||||
test("notes after the last full bar are kept and voted", () => {
|
||||
// 6 full bars plus a trailing partial bar holding only the riff's first note.
|
||||
const notes = [...bars(6), { onset: 12, offset: 12.2, pitch: 52 }];
|
||||
const out = repeatVote(notes, grid);
|
||||
// Every input note survives (the trailing one is supported by the riff's downbeat note in all repeats) ...
|
||||
for (const n of notes) expect(out.some((m) => Math.abs(m.onset - n.onset) < 1e-6 && m.pitch === n.pitch)).toBe(true);
|
||||
// ... and anything added there stays inside that bar.
|
||||
expect(out.every((n) => n.onset < 14)).toBe(true);
|
||||
});
|
||||
|
||||
test("half-tempo grid (riff twice per bar) still fills a missing note", () => {
|
||||
const g: Grid = { ...grid, bpm: 60 }; // 4 s bars
|
||||
const notes = bars(8, (b, pos) => !(b === 5 && pos === 1));
|
||||
const out = repeatVote(notes, g);
|
||||
expect(out.some((n) => Math.abs(n.onset - 11) < 1e-6 && n.pitch === 57)).toBe(true);
|
||||
});
|
||||
|
||||
test("added notes do not duplicate a same-pitch note within tol in a neighbouring bar", () => {
|
||||
// The riff ends with a note 80 ms before the next downbeat. In bar 3 that note is 40 ms late, which moves it
|
||||
// across the bar line into bar 4: bar 3 looks like it misses the note, but adding it would duplicate the late one.
|
||||
const notes = bars(6).flatMap((n) => n.pitch === 52 ? [n, { onset: n.onset + 1.92, offset: n.onset + 2, pitch: 64 }] : [n])
|
||||
.map((n) => (n.pitch === 64 && Math.abs(n.onset - 7.92) < 1e-9 ? { ...n, onset: 7.96, offset: 8.04 } : n));
|
||||
const out = repeatVote(notes, grid, { drop: false });
|
||||
expect(out.filter((n) => n.pitch === 64 && Math.abs(n.onset - 7.94) < 0.1).length).toBe(1);
|
||||
// Without the late note, the gap is filled.
|
||||
const gap = notes.filter((n) => !(n.pitch === 64 && n.onset === 7.96));
|
||||
expect(repeatVote(gap, grid, { drop: false }).some((n) => n.pitch === 64 && Math.abs(n.onset - 7.92) < 1e-6)).toBe(true);
|
||||
});
|
||||
@@ -1,7 +1,9 @@
|
||||
import { test, expect, beforeAll } from "bun:test";
|
||||
import { join } from "node:path";
|
||||
import { buildSong, loadSongDir } from "../../src/bench/dataset.ts";
|
||||
import { createPipeline, copyFingering } from "../../src/pipeline/run.ts";
|
||||
import { createPipeline, copyFingering, sumAudio } from "../../src/pipeline/run.ts";
|
||||
import { createTranscriber } from "../../src/pipeline/transcribe.ts";
|
||||
import { repeatVote } from "../../src/pipeline/repeat.ts";
|
||||
import { snapTicks, quantize } from "../../src/pipeline/quantize.ts";
|
||||
import type { TimedNote } from "../../src/core/types.ts";
|
||||
import { truthFromGp } from "../../src/score/gp.ts";
|
||||
@@ -47,9 +49,9 @@ test("t1c/1 full pipeline yields all roles", async () => {
|
||||
const roles = new Set(output.tracks.map((t) => t.role));
|
||||
for (const r of ["guitar", "bass", "keys", "vocals", "drums"]) expect(roles.has(r as never)).toBe(true);
|
||||
expect(gp.length).toBeGreaterThan(0);
|
||||
expect(warnings.some((w) => w.includes("other"))).toBe(true);
|
||||
expect(warnings.some((w) => w.includes("other"))).toBe(false);
|
||||
expect(Math.abs(output.grid.bpm - d.truth.bpm)).toBeLessThan(2);
|
||||
expect(s.guitarMerged.f1!).toBeGreaterThan(0.3);
|
||||
expect(s.guitarMerged.f1!).toBeGreaterThan(0.6); // guitar from guitar+piano+other stems (was 0.57 from the guitar stem alone)
|
||||
}, SLOW);
|
||||
|
||||
test("t1d/1 all oracle flags: two guitar tracks, partSplit and merged F1 === 1", async () => {
|
||||
@@ -90,3 +92,30 @@ test("copyFingering matches by snapped tick + pitch (merged cluster, adjacent re
|
||||
copyFingering(timed, assigned, grid);
|
||||
expect(timed.map((x) => [x.string, x.fret])).toEqual([[2, 0], [2, 0], [3, 240], [1, 0]]);
|
||||
});
|
||||
|
||||
test("sumAudio adds buffers sample-wise", () => {
|
||||
const a = { sampleRate: 10, channels: [new Float32Array([1, 2]), new Float32Array([0, 1])] };
|
||||
const b = { sampleRate: 10, channels: [new Float32Array([3, -2]), new Float32Array([1, 1])] };
|
||||
const s = sumAudio(a, b);
|
||||
expect([...s.channels[0]!]).toEqual([4, 0]);
|
||||
expect([...s.channels[1]!]).toEqual([1, 2]);
|
||||
expect(s.sampleRate).toBe(10);
|
||||
});
|
||||
|
||||
test("repeatVote option: off = raw transcription, on (default) = repeat-voted", async () => {
|
||||
// Cheap fixture: oracle stems + oracle tempo (no separation), t1a/1 guitar stem.
|
||||
const bytes = async (p: string) => new Uint8Array(await Bun.file(join(root, p)).arrayBuffer());
|
||||
const off = await createPipeline({ demucs: await bytes("models/demucs/htdemucs_6s.onnx"), basicPitch: await bytes("models/basic-pitch/nmp.onnx") }, { repeatVote: false });
|
||||
const transcribe = await createTranscriber(await bytes("models/basic-pitch/nmp.onnx"));
|
||||
const d = await loadSongDir(join(root, "data/t1a/1"));
|
||||
const flags = { oracleStems: true, oracleTempo: true, oracleNotes: false };
|
||||
const input = { mix: d.mix, oracle: { stems: d.stems, truth: d.truth } };
|
||||
const gt = d.truth.tracks.find((t) => t.role === "guitar")!;
|
||||
const raw = await transcribe(d.stems.get(gt.name)!, { minPitch: 40, maxPitch: 88 });
|
||||
const strip = (ns: TimedNote[]) => ns.map((n) => [n.onset, n.offset, n.pitch]);
|
||||
const offNotes = (await off(input, flags)).output.tracks.find((t) => t.role === "guitar")!.notes;
|
||||
expect(strip(offNotes)).toEqual(strip(raw));
|
||||
const onRes = await run(input, flags);
|
||||
const onNotes = onRes.output.tracks.find((t) => t.role === "guitar")!.notes;
|
||||
expect(strip(onNotes)).toEqual(strip(repeatVote(raw, onRes.output.grid)));
|
||||
}, SLOW);
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
import { test, expect, beforeEach, afterEach } from "bun:test";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { addSong, readManifest, songDir } from "../../src/ref/manifest.ts";
|
||||
|
||||
let root = "";
|
||||
beforeEach(async () => { root = await mkdtemp(join(tmpdir(), "ref-")); });
|
||||
afterEach(async () => { await rm(root, { recursive: true, force: true }); });
|
||||
|
||||
const base = { id: "s1", title: "Song", audio: "a.flac", midi: "a.mid", capo: {} };
|
||||
|
||||
test("readManifest on empty root returns []", async () => {
|
||||
expect(await readManifest(root)).toEqual([]);
|
||||
});
|
||||
|
||||
test("addSong writes songs.json with given roles", async () => {
|
||||
const e = await addSong({ ...base, roles: { Gtr: "guitar", Crowd: null } }, root);
|
||||
expect(e.roles).toEqual({ Gtr: "guitar", Crowd: null });
|
||||
expect(await Bun.file(join(root, "songs.json")).exists()).toBe(true);
|
||||
expect(await readManifest(root)).toEqual([e]);
|
||||
});
|
||||
|
||||
test("addSong guesses roles from the midi, keyed by track name", async () => {
|
||||
const ascii = (x: string) => [...x].map((c) => c.charCodeAt(0));
|
||||
const trk = (n: string, prog: number) => {
|
||||
const body = [0, 0xff, 3, n.length, ...ascii(n), 0, 0xc0, prog, 0, 0x90, 60, 90, 0x60, 0x80, 60, 0, 0, 0xff, 0x2f, 0];
|
||||
return [...ascii("MTrk"), 0, 0, 0, body.length, ...body];
|
||||
};
|
||||
const mid = new Uint8Array([...ascii("MThd"), 0, 0, 0, 6, 0, 1, 0, 2, 0x01, 0xe0, ...trk("Acoustic Guitar ", 25), ...trk("Brass", 61)]);
|
||||
const midi = join(root, "in.mid");
|
||||
await Bun.write(midi, mid);
|
||||
const e = await addSong({ ...base, midi }, root);
|
||||
expect(e.roles).toEqual({ "Acoustic Guitar ": "guitar", Brass: null });
|
||||
});
|
||||
|
||||
test("addSong rejects duplicate ids", async () => {
|
||||
await addSong({ ...base, roles: {} }, root);
|
||||
await expect(addSong({ ...base, roles: {} }, root)).rejects.toThrow(/duplicate|exists/i);
|
||||
expect(await readManifest(root)).toHaveLength(1);
|
||||
});
|
||||
|
||||
test("rejects unsafe ids", async () => {
|
||||
await expect(addSong({ ...base, id: "../x", roles: {} }, root)).rejects.toThrow();
|
||||
});
|
||||
|
||||
test("songDir", () => {
|
||||
expect(songDir("s1", root)).toBe(join(root, "s1"));
|
||||
});
|
||||
Reference in New Issue
Block a user