docs+bench: README aligning section; first real-song benchmark (auto alignments)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
2026-10-30 16:03:50 +01:00
parent d623026f6d
commit 3b002059e9
3 changed files with 640 additions and 3 deletions
@@ -36,7 +36,46 @@ The baseline is in `results/baseline-2026-10-30.md` (tier summaries plus a per-s
 - The SONiVOX guitar is partly routed to Demucs' piano stem, a synthetic-timbre gap that lowers full-mode guitar scores.
 - Fingering (string agreement) is informational, not pass/fail.

-## Reference songs (alignment)
+## Aligning reference songs

-Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored `reference/` directory (never committed).
-Register a song with `bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]`; this writes `reference/songs.json` and guesses a scoring role per MIDI track (unmapped tracks are `null`).
+Team-supplied real recordings and Guitar Pro-exported MIDI live in the git-ignored `reference/` directory (never commit them; `*.flac`, `*.mid` and `runs/` are ignored too). Each song gets `reference/<id>/` with its audio, MIDI and `alignment.json`, and `reference/songs.json` lists the registered songs.
+
+Register a song (guesses a scoring role per MIDI track; unmapped tracks are `null`):
+
+```sh
+bun run align add <audio> <midi> --id <id> [--title <t>] [--capo "<track name>=9"]
+```
+
+The `--capo` track name must match the MIDI track name exactly, including any trailing space (e.g. `"Acoustic Guitar =9"`).
+
+Compute and save an automatic alignment (all anchors `auto`; manual anchors from an earlier save are kept) and print its fit summary:
+
+```sh
+bun run align auto <id>
+```
+
+Launch the alignment page. A bare `bun run align` starts a long-running server; the key in the printed URL is required and changes on every start. For LAN use:
+
+```sh
+bun run align --host 0.0.0.0 --url-host 10.20.20.2
+```
+
+Page workflow:
+
+1. Pick a song and add anchors where the MIDI and the recording visibly disagree (click the waveform/roll to pin a MIDI time to an audio time).
+2. Play and mix: toggle the recording and the rendered MIDI, and per-role levels, to listen for sync.
+3. Run auto-align around anchors; manual anchors are kept fixed and the rest is recomputed.
+4. Save to write `alignment.json`.
+
+Benchmark against the real songs (uses each saved `alignment.json`; `--auto` computes one on the fly for songs without it):
+
+```sh
+bun run bench real [--auto] --run-id <id>
+```
+
+This writes `results/real-<id>.md` and `.json` (committed) and per-song output under `runs/real-<id>/` (ignored).
+
+Caveats:
+
+- The fit summary (matched MIDI onsets, median residual, flagged windows) is a consistency measure between the MIDI and the audio, not ground truth; a high match rate does not prove the alignment is right.
+- The page plays Ogg Opus; Safari needs 18.4 or later.
@@ -0,0 +1,523 @@
+[
+ {
+  "id": "lnoe",
+  "title": "Last Night on Earth (Green Day)",
+  "alignment": {
+   "source": "manual",
+   "manual": 0,
+   "auto": 57,
+   "savedAt": "2026-10-30T14:19:32.084Z"
+  },
+  "fit": {
+   "matchRate": 0.36537195523370636,
+   "medianResidualMs": 18.229166666657193,
+   "windows": [
+    {
+     "startSec": 0,
+     "matchRate": 0.3387096774193548,
+     "medianResidualMs": 18.160807291668135,
+     "flagged": true
+    },
+    {
+     "startSec": 30,
+     "matchRate": 0.2853881278538813,
+     "medianResidualMs": 17.207031250002558,
+     "flagged": true
+    },
+    {
+     "startSec": 60,
+     "matchRate": 0.3106212424849699,
+     "medianResidualMs": 16.458333333318365,
+     "flagged": true
+    },
+    {
+     "startSec": 90,
+     "matchRate": 0.2907949790794979,
+     "medianResidualMs": 32.67578125000625,
+     "flagged": true
+    },
+    {
+     "startSec": 120,
+     "matchRate": 0.42592592592592593,
+     "medianResidualMs": 17.275390625002274,
+     "flagged": true
+    },
+    {
+     "startSec": 150,
+     "matchRate": 0.49002217294900224,
+     "medianResidualMs": 18.229166666657193,
+     "flagged": true
+    },
+    {
+     "startSec": 180,
+     "matchRate": 0.4409090909090909,
+     "medianResidualMs": 18.229166666657193,
+     "flagged": true
+    },
+    {
+     "startSec": 210,
+     "matchRate": 0.15384615384615385,
+     "medianResidualMs": 35.62500000001023,
+     "flagged": true
+    }
+   ]
+  },
+  "perRole": {
+   "guitar": {
+    "tp": 848,
+    "fp": 1190,
+    "fn": 2215,
+    "precision": 0.41609421000981356,
+    "recall": 0.27685275873326803,
+    "f1": 0.3324838267006469,
+    "octaveErrors": 88
+   },
+   "vocals": {
+    "tp": 80,
+    "fp": 416,
+    "fn": 159,
+    "precision": 0.16129032258064516,
+    "recall": 0.33472803347280333,
+    "f1": 0.21768707482993196,
+    "octaveErrors": 0
+   },
+   "bass": {
+    "tp": 73,
+    "fp": 323,
+    "fn": 90,
+    "precision": 0.18434343434343434,
+    "recall": 0.44785276073619634,
+    "f1": 0.26118067978533094,
+    "octaveErrors": 0
+   },
+   "drums": {
+    "tp": 319,
+    "fp": 525,
+    "fn": 257,
+    "precision": 0.37796208530805686,
+    "recall": 0.5538194444444444,
+    "f1": 0.44929577464788734,
+    "octaveErrors": 17
+   },
+   "keys": {
+    "tp": 281,
+    "fp": 1601,
+    "fn": 901,
+    "precision": 0.14930924548352817,
+    "recall": 0.2377326565143824,
+    "f1": 0.18342036553524804,
+    "octaveErrors": 85
+   }
+  },
+  "lenient": {
+   "guitar": 0.4328563026857479,
+   "vocals": 0.34285714285714286,
+   "bass": 0.3041144901610018,
+   "drums": 0.5,
+   "keys": 0.239556135770235
+  },
+  "drumsPerClass": {
+   "kick": {
+    "tp": 76,
+    "fp": 75,
+    "fn": 22,
+    "precision": 0.5033112582781457,
+    "recall": 0.7755102040816326,
+    "f1": 0.6104417670682731,
+    "octaveErrors": 0
+   },
+   "snare": {
+    "tp": 78,
+    "fp": 223,
+    "fn": 20,
+    "precision": 0.2591362126245847,
+    "recall": 0.7959183673469388,
+    "f1": 0.39097744360902253,
+    "octaveErrors": 0
+   },
+   "hihat": {
+    "tp": 154,
+    "fp": 219,
+    "fn": 66,
+    "precision": 0.4128686327077748,
+    "recall": 0.7,
+    "f1": 0.5193929173693086,
+    "octaveErrors": 0
+   },
+   "cymbal": {
+    "tp": 11,
+    "fp": 8,
+    "fn": 143,
+    "precision": 0.5789473684210527,
+    "recall": 0.07142857142857142,
+    "f1": 0.12716763005780346,
+    "octaveErrors": 0
+   },
+   "tom": {
+    "tp": 0,
+    "fp": 0,
+    "fn": 6,
+    "precision": null,
+    "recall": 0,
+    "f1": 0,
+    "octaveErrors": 0
+   }
+  },
+  "excludedTracks": [],
+  "capo": {},
+  "seconds": 206.677340181
+ },
+ {
+  "id": "swimming",
+  "title": "Swimming (Lunar Vacation)",
+  "alignment": {
+   "source": "manual",
+   "manual": 0,
+   "auto": 58,
+   "savedAt": "2026-10-30T14:19:38.759Z"
+  },
+  "fit": {
+   "matchRate": 0.2324570932000869,
+   "medianResidualMs": 23.892940234389926,
+   "windows": [
+    {
+     "startSec": 0,
+     "matchRate": 0.37623762376237624,
+     "medianResidualMs": 14.82047643229123,
+     "flagged": true
+    },
+    {
+     "startSec": 30,
+     "matchRate": 0.23826714801444043,
+     "medianResidualMs": 22.7287138020813,
+     "flagged": true
+    },
+    {
+     "startSec": 60,
+     "matchRate": 0.28329297820823246,
+     "medianResidualMs": 17.78587610677107,
+     "flagged": true
+    },
+    {
+     "startSec": 90,
+     "matchRate": 0.29533678756476683,
+     "medianResidualMs": 14.576658246525653,
+     "flagged": true
+    },
+    {
+     "startSec": 120,
+     "matchRate": 0.25,
+     "medianResidualMs": 28.613346408427276,
+     "flagged": true
+    },
+    {
+     "startSec": 150,
+     "matchRate": 0.21959858323494688,
+     "medianResidualMs": 29.938027777788534,
+     "flagged": true
+    },
+    {
+     "startSec": 180,
+     "matchRate": 0.2471042471042471,
+     "medianResidualMs": 30.590373632819023,
+     "flagged": true
+    },
+    {
+     "startSec": 210,
+     "matchRate": 0.2388419782870929,
+     "medianResidualMs": 29.840034038628005,
+     "flagged": true
+    },
+    {
+     "startSec": 240,
+     "matchRate": 0.08235294117647059,
+     "medianResidualMs": 10.238290299454889,
+     "flagged": true
+    },
+    {
+     "startSec": 270,
+     "matchRate": 0,
+     "medianResidualMs": null,
+     "flagged": true
+    }
+   ]
+  },
+  "perRole": {
+   "guitar": {
+    "tp": 1010,
+    "fp": 2554,
+    "fn": 3982,
+    "precision": 0.2833894500561167,
+    "recall": 0.20232371794871795,
+    "f1": 0.23609163160355306,
+    "octaveErrors": 152
+   },
+   "vocals": {
+    "tp": 32,
+    "fp": 574,
+    "fn": 169,
+    "precision": 0.052805280528052806,
+    "recall": 0.15920398009950248,
+    "f1": 0.07930607187112763,
+    "octaveErrors": 0
+   },
+   "bass": {
+    "tp": 153,
+    "fp": 748,
+    "fn": 377,
+    "precision": 0.16981132075471697,
+    "recall": 0.28867924528301886,
+    "f1": 0.2138364779874214,
+    "octaveErrors": 9
+   },
+   "drums": {
+    "tp": 338,
+    "fp": 1462,
+    "fn": 910,
+    "precision": 0.18777777777777777,
+    "recall": 0.2708333333333333,
+    "f1": 0.22178477690288714,
+    "octaveErrors": 117
+   },
+   "keys": {
+    "tp": 0,
+    "fp": 2032,
+    "fn": 0,
+    "precision": 0,
+    "recall": null,
+    "f1": 0,
+    "octaveErrors": 0
+   }
+  },
+  "lenient": {
+   "guitar": 0.36605890603085556,
+   "vocals": 0.13135068153655513,
+   "bass": 0.2921034241788959,
+   "drums": 0.3064304461942257,
+   "keys": 0
+  },
+  "drumsPerClass": {
+   "kick": {
+    "tp": 137,
+    "fp": 431,
+    "fn": 201,
+    "precision": 0.24119718309859156,
+    "recall": 0.40532544378698226,
+    "f1": 0.30242825607064017,
+    "octaveErrors": 0
+   },
+   "snare": {
+    "tp": 86,
+    "fp": 374,
+    "fn": 156,
+    "precision": 0.18695652173913044,
+    "recall": 0.35537190082644626,
+    "f1": 0.245014245014245,
+    "octaveErrors": 0
+   },
+   "hihat": {
+    "tp": 0,
+    "fp": 465,
+    "fn": 0,
+    "precision": 0,
+    "recall": null,
+    "f1": 0,
+    "octaveErrors": 0
+   },
+   "cymbal": {
+    "tp": 115,
+    "fp": 192,
+    "fn": 515,
+    "precision": 0.3745928338762215,
+    "recall": 0.18253968253968253,
+    "f1": 0.24546424759871932,
+    "octaveErrors": 0
+   },
+   "tom": {
+    "tp": 0,
+    "fp": 0,
+    "fn": 38,
+    "precision": null,
+    "recall": 0,
+    "f1": 0,
+    "octaveErrors": 0
+   }
+  },
+  "excludedTracks": [],
+  "capo": {},
+  "seconds": 141.90737477700003
+ },
+ {
+  "id": "holes",
+  "title": "Holes (Passenger)",
+  "alignment": {
+   "source": "manual",
+   "manual": 0,
+   "auto": 127,
+   "savedAt": "2026-10-30T14:19:43.699Z"
+  },
+  "fit": {
+   "matchRate": 0.29754918881601655,
+   "medianResidualMs": 20.573708593755846,
+   "windows": [
+    {
+     "startSec": 0,
+     "matchRate": 0.2949852507374631,
+     "medianResidualMs": 26.468118229166393,
+     "flagged": true
+    },
+    {
+     "startSec": 30,
+     "matchRate": 0.36477987421383645,
+     "medianResidualMs": 14.129902864585375,
+     "flagged": true
+    },
+    {
+     "startSec": 60,
+     "matchRate": 0.34183673469387754,
+     "medianResidualMs": 21.174896093739903,
+     "flagged": true
+    },
+    {
+     "startSec": 90,
+     "matchRate": 0.31226765799256506,
+     "medianResidualMs": 26.190420182309992,
+     "flagged": true
+    },
+    {
+     "startSec": 120,
+     "matchRate": 0.07906976744186046,
+     "medianResidualMs": 20.52699999995866,
+     "flagged": true
+    },
+    {
+     "startSec": 150,
+     "matchRate": 0.22043010752688172,
+     "medianResidualMs": 19.901970572888672,
+     "flagged": true
+    },
+    {
+     "startSec": 180,
+     "matchRate": 0.33093525179856115,
+     "medianResidualMs": 25.241090234416674,
+     "flagged": true
+    },
+    {
+     "startSec": 210,
+     "matchRate": null,
+     "medianResidualMs": null,
+     "flagged": false
+    }
+   ]
+  },
+  "perRole": {
+   "guitar": {
+    "tp": 817,
+    "fp": 2209,
+    "fn": 1847,
+    "precision": 0.2699933906146728,
+    "recall": 0.3066816816816817,
+    "f1": 0.287170474516696,
+    "octaveErrors": 224
+   },
+   "vocals": {
+    "tp": 66,
+    "fp": 614,
+    "fn": 378,
+    "precision": 0.09705882352941177,
+    "recall": 0.14864864864864866,
+    "f1": 0.11743772241992882,
+    "octaveErrors": 0
+   },
+   "bass": {
+    "tp": 54,
+    "fp": 469,
+    "fn": 71,
+    "precision": 0.10325047801147227,
+    "recall": 0.432,
+    "f1": 0.16666666666666666,
+    "octaveErrors": 12
+   },
+   "drums": {
+    "tp": 391,
+    "fp": 1232,
+    "fn": 335,
+    "precision": 0.24091189155884166,
+    "recall": 0.5385674931129476,
+    "f1": 0.3329076202639421,
+    "octaveErrors": 10
+   },
+   "keys": {
+    "tp": 80,
+    "fp": 1011,
+    "fn": 474,
+    "precision": 0.07332722273143905,
+    "recall": 0.1444043321299639,
+    "f1": 0.0972644376899696,
+    "octaveErrors": 30
+   }
+  },
+  "lenient": {
+   "guitar": 0.38242530755711773,
+   "vocals": 0.23487544483985764,
+   "bass": 0.23148148148148148,
+   "drums": 0.4240102171136654,
+   "keys": 0.14832826747720365
+  },
+  "drumsPerClass": {
+   "kick": {
+    "tp": 77,
+    "fp": 167,
+    "fn": 43,
+    "precision": 0.3155737704918033,
+    "recall": 0.6416666666666667,
+    "f1": 0.4230769230769231,
+    "octaveErrors": 0
+   },
+   "snare": {
+    "tp": 162,
+    "fp": 168,
+    "fn": 90,
+    "precision": 0.4909090909090909,
+    "recall": 0.6428571428571429,
+    "f1": 0.5567010309278351,
+    "octaveErrors": 0
+   },
+   "hihat": {
+    "tp": 151,
+    "fp": 553,
+    "fn": 144,
+    "precision": 0.21448863636363635,
+    "recall": 0.511864406779661,
+    "f1": 0.3023023023023023,
+    "octaveErrors": 0
+   },
+   "cymbal": {
+    "tp": 1,
+    "fp": 344,
+    "fn": 5,
+    "precision": 0.002898550724637681,
+    "recall": 0.16666666666666666,
+    "f1": 0.005698005698005698,
+    "octaveErrors": 0
+   },
+   "tom": {
+    "tp": 0,
+    "fp": 0,
+    "fn": 1,
+    "precision": null,
+    "recall": 0,
+    "f1": 0,
+    "octaveErrors": 0
+   }
+  },
+  "excludedTracks": [
+   "Brass",
+   "Crowd"
+  ],
+  "capo": {
+   "Acoustic Guitar ": 9
+  },
+  "seconds": 60.68984948400001
+ }
+]
@@ -0,0 +1,75 @@
+# STSheets real-song benchmark: auto-2026-10-30
+
+> Reference = the team's Guitar Pro-exported MIDI warped onto the recording by the saved alignment; scores are note-onset matches (pitch must equal; drums compared by class). The fit line reports the *consistency of the alignment* (warped MIDI onsets vs onset-flux peaks of the recording), not an independent validation of it: a poor alignment lowers every score below. Capo is informational and not applied.
+
+## Last Night on Earth (Green Day) (lnoe)
+
+- Alignment: saved (0 manual / 57 auto anchors, saved 2026-10-30T14:19:32.084Z)
+- Consistency of the alignment: 36.5% of MIDI onsets matched · median residual 18.2 ms · 8/8 windows flagged (0:00, 0:30, 1:00, 1:30, 2:00, 2:30, 3:00, 3:30)
+- Capo: none listed
+- Excluded tracks (no role): none
+- Pipeline + scoring: 206.7 s
+
+| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
+| --- | --- | --- | --- | --- | --- | --- |
+| guitar | 3063 | 2038 | 0.42 | 0.28 | 0.33 | 0.43 |
+| vocals | 239 | 496 | 0.16 | 0.33 | 0.22 | 0.34 |
+| bass | 163 | 396 | 0.18 | 0.45 | 0.26 | 0.30 |
+| drums | 576 | 844 | 0.38 | 0.55 | 0.45 | 0.50 |
+| keys | 1182 | 1882 | 0.15 | 0.24 | 0.18 | 0.24 |
+
+| Drum class | Ref | Est | F1 ±50 ms |
+| --- | --- | --- | --- |
+| kick | 98 | 151 | 0.61 |
+| snare | 98 | 301 | 0.39 |
+| hihat | 220 | 373 | 0.52 |
+| cymbal | 154 | 19 | 0.13 |
+| tom | 6 | 0 | 0.00 |
+
+## Swimming (Lunar Vacation) (swimming)
+
+- Alignment: saved (0 manual / 58 auto anchors, saved 2026-10-30T14:19:38.759Z)
+- Consistency of the alignment: 23.2% of MIDI onsets matched · median residual 23.9 ms · 10/10 windows flagged (0:00, 0:30, 1:00, 1:30, 2:00, 2:30, 3:00, 3:30, 4:00, 4:30)
+- Capo: none listed
+- Excluded tracks (no role): none
+- Pipeline + scoring: 141.9 s
+
+| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
+| --- | --- | --- | --- | --- | --- | --- |
+| guitar | 4992 | 3564 | 0.28 | 0.20 | 0.24 | 0.37 |
+| vocals | 201 | 606 | 0.05 | 0.16 | 0.08 | 0.13 |
+| bass | 530 | 901 | 0.17 | 0.29 | 0.21 | 0.29 |
+| drums | 1248 | 1800 | 0.19 | 0.27 | 0.22 | 0.31 |
+| keys | 0 | 2032 | 0.00 | n/a | 0.00 | 0.00 |
+
+| Drum class | Ref | Est | F1 ±50 ms |
+| --- | --- | --- | --- |
+| kick | 338 | 568 | 0.30 |
+| snare | 242 | 460 | 0.25 |
+| hihat | 0 | 465 | 0.00 |
+| cymbal | 630 | 307 | 0.25 |
+| tom | 38 | 0 | 0.00 |
+
+## Holes (Passenger) (holes)
+
+- Alignment: saved (0 manual / 127 auto anchors, saved 2026-10-30T14:19:43.699Z)
+- Consistency of the alignment: 29.8% of MIDI onsets matched · median residual 20.6 ms · 7/8 windows flagged (0:00, 0:30, 1:00, 1:30, 2:00, 2:30, 3:00)
+- Capo: Acoustic Guitar : 9
+- Excluded tracks (no role): Brass, Crowd
+- Pipeline + scoring: 60.7 s
+
+| Role | Ref notes | Est notes | P ±50 ms | R ±50 ms | F1 ±50 ms | F1 ±100 ms |
+| --- | --- | --- | --- | --- | --- | --- |
+| guitar | 2664 | 3026 | 0.27 | 0.31 | 0.29 | 0.38 |
+| vocals | 444 | 680 | 0.10 | 0.15 | 0.12 | 0.23 |
+| bass | 125 | 523 | 0.10 | 0.43 | 0.17 | 0.23 |
+| drums | 726 | 1623 | 0.24 | 0.54 | 0.33 | 0.42 |
+| keys | 554 | 1091 | 0.07 | 0.14 | 0.10 | 0.15 |
+
+| Drum class | Ref | Est | F1 ±50 ms |
+| --- | --- | --- | --- |
+| kick | 120 | 244 | 0.42 |
+| snare | 252 | 330 | 0.56 |
+| hihat | 295 | 704 | 0.30 |
+| cymbal | 6 | 345 | 0.01 |
+| tom | 1 | 0 | 0.00 |