From 378aff6a7b0e77239949866a31bd9c2ceca5c998 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Thu, 29 Oct 2026 23:52:29 +0100 Subject: [PATCH] Record D-WS2a: hint measurements, the 120-token truncation rule, no faster-whisper hotwords Co-Authored-By: Claude Opus 5.5 --- docs/Decisions.md | 84 +++++++++++++++++++++++++++++ docs/WorkBreakdown.md | 10 ++++ docs/plans/001-2-vocabulary-hint.md | 8 +-- 3 files changed, 98 insertions(+), 4 deletions(-) diff --git a/docs/Decisions.md b/docs/Decisions.md index 5dc8f08..0571a2d 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1224,3 +1224,87 @@ whole-file output is equally bad, which is exactly what the absolute bar exists to catch. **Consequence:** full `bun test` ≈ 19 min on CPU (en_alien ≈ 20 min of it on a cold run); the fast suite excludes this file. + +## D-WS2a — "Names and terms" vocabulary hint as a Whisper decoder prompt, cut to 120 tokens +**Status:** Accepted (2026-10-29) — spec 001, Goal 2. Placeholder id; the +integrator renumbers it. +**Context:** users want names and jargon spelled their way. Whisper takes a +"previous text" prompt (`<|startofprev|> … <|startoftranscript|>`). +Transformers.js 4.2.0 — and 4.3.2, checked in workstream 1's worktree — +declare `prompt_ids` in the Whisper generation config but never use it (the +parameter is commented out in `WhisperForConditionalGeneration.generate`), +and the pipeline's `_decode_asr` prints every non-special decoder token, so +a prompt passed as `decoder_input_ids` would appear in the transcript. +**Decision:** +- UI: an optional "Names and terms" textarea for every model + (`VocabularyHint`), raw text kept in localStorage (`whisper-vocabulary-hint`), + ≤ 1000 characters, sent as `prompt` on the worker's transcribe message, on + the server engine's query string, and on the live WebSocket URL. +- `src/lib/hint.ts` normalizes (one term per line or comma-separated → + `"a, b, c"`, control characters dropped) and owns the **truncation rule**: + prompt text is `" " + hint` ending in a full stop (unless it ends in + `.?!`); over budget → the longest prefix of whole space-separated words + that fits (dangling `,`/`;` removed, full stop re-added) — the start of the + list is kept, the end dropped; if not even the first word fits, its first + `budget` tokens. Budget **120 tokens** (Whisper allows 223). The Python twin + `fit_prompt_ids` in `server-py/sidecar.py` is parity-tested. +- Browser: `src/worker/prompt.ts` builds `<|startofprev|> + ids` once per pass + and passes it through the ASR pipeline as a private generate kwarg; a + one-time wrapper around `model.generate` prepends it to the init tokens of + every 30 s window, caps `max_new_tokens` so prompt + init + new ≤ 448 + decoder positions (D-021's 440 otherwise), and slices it off the output. + Live passes use the same path. +- Server: `parseLanguageAndTask` validates `prompt` (400 above 1000 chars) for + the HTTP and WebSocket routes; the sidecar passes the cut token ids as + faster-whisper's `initial_prompt`. **No `hotwords`.** +**Measurements** (en_alien, 888 s gaming livestream, whole file; WER vs +`tests/audio_files/en_alien.txt`, single runs; hinted terms counted as +maxwell / voices of the void / slenderman / runescape, reference 9/1/1/1; +hint "Maxwell, Voices of the Void, Slenderman, RuneScape, UFO" = 21 tokens): +- Browser path, **onnxruntime-node CPU**, catalog WASM dtypes (turbo q4, + base.en q8), the worker's prompt code: + + | model | hint | prompt tokens | WER | terms | + |---|---|---|---|---| + | large-v3-turbo | none | 0 | 0.255 | 9/1/0/1 | + | large-v3-turbo | short list | 21 | **0.120** | 9/1/1/1 | + | large-v3-turbo | 54-name realistic list | 120 | 0.253 | 9/1/1/1 | + | large-v3-turbo | 54-name realistic list | 221 | 0.205 | 10/1/1/1 | + | large-v3-turbo | 300 synthetic "TermN" | 120 | 0.266 | 9/1/1/1 | + | base.en | none | 0 | 0.339 | 7/1/0/1 | + | base.en | short list, no final full stop | 21 | 0.566 | 4/1/1/0 | + | base.en | short list | 21 | 0.319 | 7/1/1/1 | + | base.en | realistic list | 32 / 64 / 96 / 120 | 0.286 / 0.306 / 0.305 / 0.284 | 9/1/1/1 at 32 and 120 | + | base.en | realistic list | 160 / 218 | 0.386 / **0.476** | 8/1/1/1, 6/1/1/0 | + | base.en | 300 synthetic "TermN" | 96 / 120 / 223 | 0.530 / 0.474 / **1.050** | | + + Wall-clock per whole-file pass (turbo 141 s without a hint, 129–206 s + with) varied with machine load more than with the hint; the extra cost is + only the prompt's decoder positions per window. +- Without a final full stop the prompt made base.en drop punctuation and loop + (punctuation 239 → 183) — hence the full stop. +- Short windows (live): 10 cases × 2 models (silence 1/3 s, noise 1–3 s, + 2 s speech snippets from en_aiproblem and da_festuge), with and without the + short hint: **0 / 20** outputs contained a hint word the no-hint output + lacked. One 2 s Danish snippet on turbo (language pinned to `da`) came out + as English words with the English hint ("I have been at an order June + August." vs "at den orde tøde i august."). +- Server path, faster-whisper 1.2.1 large-v3-turbo int8 CPU, short hint: + none 0.118 (9/1/0/1), `initial_prompt` 0.085, `hotwords` 0.077, both 0.112 + — all with 9/1/1/1. Hotwords did not improve hinted-word recall over + `initial_prompt` and combining them lost WER, so the plan's bar ("only if it + measurably helps") is not met. +**Why 120 tokens:** with prompts near Whisper's 223-token cap the small +model degraded badly (base.en 0.339 → 0.476 realistic, 1.05 synthetic — +repetition and dropped text), while at ≤ 120 tokens a realistic list stayed +at or below the no-hint WER on both models. A junk list still hurts base.en +at 96–120 tokens (0.47–0.53); the UI says "put the most important names +first" once the text passes ~300 characters (≈ 120 tokens of names), and +short lists measured best everywhere. +**Rejected:** Transformers.js `prompt_ids` (unused in 4.2.0/4.3.2); +patching `_decode_asr` (a wrapper avoids touching the transformers patch); +the full 223-token budget (above); per-model budgets (one rule, one parity +test; revisit if a larger list is wanted on turbo). +**Consequence:** every window pays up to 121 extra decoder positions with a +long hint. WebGPU behaviour of the prompt path is unmeasured here (CPU +only) — needs the owner's browser check. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 336b0b4..caa888d 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -339,6 +339,12 @@ cached. `language`/`task` the same way the HTTP route does, and `tests/server-live.test.ts` covers the bridge against a fake sidecar. +## Spec 001 — Transcription quality and speed +- [x] Vocabulary hint "Names and terms" (D-WS2a) — optional field for every + model, remembered in localStorage, sent as `prompt` on file and live + passes (browser and server engines); cut to 120 tokens at a whole word. + faster-whisper gets `initial_prompt` only (hotwords measured, no gain). + --- ## Deliberately out of scope (for now) @@ -372,6 +378,10 @@ Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29. — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the Backend select overflowed its panel on 390 px-wide phones. +- [ ] An English "Names and terms" hint can pull a short Danish live window + toward English words (one 2 s da_festuge snippet on turbo, D-WS2a). + Not addressed: suggest same-language hints if it shows up in use. + ## Recurring - [ ] Maintain `AGENTS.md`: - **Add a rule only in response to a real failure.** When an agent repeatedly diff --git a/docs/plans/001-2-vocabulary-hint.md b/docs/plans/001-2-vocabulary-hint.md index 9805c05..a27a2c5 100644 --- a/docs/plans/001-2-vocabulary-hint.md +++ b/docs/plans/001-2-vocabulary-hint.md @@ -46,7 +46,7 @@ sidecar applies the same truncation rule (Python twin, parity-tested) and passes above it — URL-safe since it travels as a query param). - `normalizeHint`: drop control chars, one term per line or comma-separated → lines trimmed, empty lines dropped, joined with `", "`, whitespace collapsed. Empty → `""` (= no prompt). -- `PROMPT_TOKEN_BUDGET = 223` (= 448/2 − 1, the openai/faster-whisper prompt cap). +- `PROMPT_TOKEN_BUDGET = 120` (measured; was 223 = 448/2 − 1, see D-WS2a). - **Truncation rule:** encode `" " + hint`; if > budget, keep the longest prefix of whole (space-separated) words that fits, trailing `,;` stripped from the kept text; if not even the first word fits, its first 223 tokens. The start of the list is kept, the end dropped. @@ -179,12 +179,12 @@ Scratch scripts in `/tmp/ws2/` (not committed), importing the worktree's modules `"Maxwell, Voices of the Void"`: WER (test normalizer), count of `maxwell` and `voices of the void` vs reference counts, wall-clock, punctuation count (a prompt without a final full stop may suppress punctuation — if so, `normalizeHint` appends `.` and Task 1 tests are updated). -- [ ] Short-window hallucination: 1–3 s low-noise and 2 s speech snippets with/without hint, turbo +- [x] Short-window hallucination: 1–3 s low-noise and 2 s speech snippets with/without hint, turbo + small model; output must not contain hint words that are absent from the audio. -- [ ] Server path (venv, `mobiuslabsgmbh/faster-whisper-large-v3-turbo` cached, int8 CPU): +- [x] Server path (venv, `mobiuslabsgmbh/faster-whisper-large-v3-turbo` cached, int8 CPU): en_alien none / initial_prompt / hotwords / both. Keep hotwords only if it measurably beats initial_prompt alone on hinted-word recall without WER loss. Implement the outcome in sidecar.py. -- [ ] Write D-WS2a (bottom of `docs/Decisions.md`) with numbers labelled CPU/onnxruntime-node, +- [x] Write D-WS2a (bottom of `docs/Decisions.md`) with numbers labelled CPU/onnxruntime-node, truncation rule, wrapper rationale, 4.3.2 check. Architecture.md data-flow note; WorkBreakdown tick/noticed. Commit.