From 6b338a2c58713336c0709215bc6fec0f2a850bee Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 00:13:42 +0100 Subject: [PATCH] Remove the finished workstream 2 plan (vocabulary hint) Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-2-vocabulary-hint.md | 216 ---------------------------- 1 file changed, 216 deletions(-) delete mode 100644 docs/plans/001-2-vocabulary-hint.md diff --git a/docs/plans/001-2-vocabulary-hint.md b/docs/plans/001-2-vocabulary-hint.md deleted file mode 100644 index be78c57..0000000 --- a/docs/plans/001-2-vocabulary-hint.md +++ /dev/null @@ -1,216 +0,0 @@ ---- -status: in-progress ---- - -# Vocabulary hint ("Names and terms") Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no nested -> subagents — coordination plan 001-0). Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** an optional "Names and terms" hint, remembered in localStorage, fed to Whisper as a -decoder prompt on every window of every browser pass (file + live) and to faster-whisper on the -server engine. - -**Architecture:** the UI sends the raw hint as `prompt?: string` (shared interface, 001-0). A pure -`src/lib/hint.ts` owns normalization, storage and the truncation rule; the worker turns the hint -into `<|startofprev|> + ids` once per transcribe message and passes it through the pipeline as a -private generate kwarg, which a one-time wrapper around `model.generate` (`src/worker/prompt.ts`) -turns into `decoder_input_ids` and strips from the output again. Transformers.js 4.2.0 **and** -4.3.2 accept `prompt_ids` in the config but never use it (commented out in -`modeling_whisper.js`), and `_decode_asr` would print prompt tokens as transcript text — hence the -wrapper. Server: `parseLanguageAndTask` validates `prompt` for both routes (D-032 pattern), the -sidecar applies the same truncation rule (Python twin, parity-tested) and passes token ids as -`initial_prompt` (+ `hotwords` only if measured to help). - -**Tech Stack:** Bun 1.3, React 19, Transformers.js 4.2.0 (patched), faster-whisper 1.2.1 venv at -`/home/user/venvs/faster-whisper` (measurements only). - -**Spec:** `docs/specs/001-transcription-quality-and-speed.md` (Goal 2; file-mode accuracy gate); -`docs/plans/001-0-coordination.md` (shared interfaces, global constraints). - -## Global Constraints - -- Field name `prompt?: string` on the transcribe message, raw user text; tokenized/truncated in the - worker. Live and server paths carry the same name. -- Decision entry `## D-WS2a — …`, cited as `D-WS2a` in code comments. No real `D-###`. -- Heavy jobs (real-model tests, full `bun test`, measurements, downloads) under - `flock /tmp/whisper-heavy.lock`. Ports 3120–3129. No push/publish. -- Plain-language UI, optional, never required (PRODUCT.md); colour tokens only (DESIGN.md). -- `max_new_tokens: 440` (D-021) keeps the single-pass path; with a prompt it is capped so prompt + - SOT + new tokens ≤ 448 decoder positions. -- Measurements are onnxruntime-node (CPU) numbers; say so. WebGPU effects → owner's browser check. - -## Decisions fixed by this plan - -- Storage key `whisper-vocabulary-hint`; `HINT_MAX_CHARS = 1000` (textarea `maxLength`, server 400 - above it — URL-safe since it travels as a query param). -- `normalizeHint`: drop control chars, one term per line or comma-separated → lines trimmed, empty - lines dropped, joined with `", "`, whitespace collapsed. Empty → `""` (= no prompt). -- `PROMPT_TOKEN_BUDGET = 120` (measured; was 223 = 448/2 − 1, see D-WS2a). -- **Truncation rule:** encode `" " + hint`; if > budget, keep the longest prefix of whole - (space-separated) words that fits, trailing `,;` stripped from the kept text; if not even the - first word fits, its first 223 tokens. The start of the list is kept, the end dropped. -- Prompt prefix computed once per transcribe message (not per window); per-window cost is the - extra decoder positions only. - -## Review Focus - -1. **Very long hint (> 223 tokens)** — truncated at a word boundary, never an error, first window - not garbled. Pinned by Task 1 (unit), Task 2 (cap of `max_new_tokens`), Task 7 (real model, - 1000-char hint). -2. **Short / near-silent live windows with a hint** — must not emit the hint text (Whisper's - "transcribe the prompt" failure). Task 7 real-model test on low-noise audio. -3. **Prompt tokens leaking into the transcript** (`_decode_asr` treats them as text) — Task 2 unit - test + Task 7 real-model check that output does not start with the hint. -4. **Non-ASCII hints on the server path** (Danish names in a query param) — Task 4 handler test - with `Århus, Ærø` round-trips to the sidecar request. -5. **`.en` models** (no language/task tokens) — Task 2 test with init tokens `[SOT]` only; the hint - applies to every catalog model, multilingual or not. - ---- - -### Task 1: `src/lib/hint.ts` — normalize, store, truncate - -**Files:** Create `src/lib/hint.ts`; Test `tests/hint.test.ts` - -**Interfaces — Produces:** -- `HINT_STORAGE_KEY`, `HINT_MAX_CHARS = 1000`, `PROMPT_TOKEN_BUDGET = 223` -- `normalizeHint(raw: string): string` -- `loadHint(storage?: StorageLike | null): string`, `saveHint(hint: string, storage?)` (StorageLike - from `src/lib/engine.ts`; never throw) -- `fitHintToBudget(hint: string, encode: (text: string) => number[], budget = PROMPT_TOKEN_BUDGET): number[]` - (normalizes internally; `[]` for empty) - -- [x] Step 1: failing tests — `normalizeHint(" Maxwell\n\n Voices of the Void \n")` === - `"Maxwell, Voices of the Void"`; control chars removed; `"a, b"` unchanged; whitespace-only → `""`. - `loadHint`/`saveHint` round-trip with a Map-backed storage, throwing storage → `""`/no throw. - `fitHintToBudget` with fake `encode = (t) => [...t].map(c => c.charCodeAt(0))`: short hint → - encode(" Maxwell, Voices of the Void"); budget 10 on `"alpha, beta gamma"` → ids of `" alpha"` - (trailing comma stripped, whole words only); single 50-char word with budget 10 → first 10 ids; - 5000-word hint with budget 223 → length ≤ 223 and decodes to a word prefix of the hint. -- [x] Step 2: `bun test tests/hint.test.ts` → FAIL (module missing). -- [x] Step 3: implement; binary search over word count (encode is monotone in prefix length). -- [x] Step 4: tests PASS. Commit. - -### Task 2: `src/worker/prompt.ts` — decoder prompt through the pipeline - -**Files:** Create `src/worker/prompt.ts`; Test `tests/prompt.test.ts` - -**Interfaces — Consumes:** `fitHintToBudget`. **Produces:** -- `PROMPT_PREFIX_KWARG = "whisper_web_prompt_prefix"`, `WHISPER_DECODER_POSITIONS = 448` -- `buildPromptPrefix(tokenizer: { encode(t: string, o: {add_special_tokens: false}): number[]; convert_tokens_to_ids(t: string): number }, hint: string): number[]` - → `[]` or `[<|startofprev|>, ...ids]` -- `installPromptSupport(model: PromptableModel): void` — idempotent; wraps `model.generate`: pops - the kwarg; empty → original call unchanged; else `decoder_input_ids = [...prefix, - ...model._retrieve_init_tokens(model._prepare_generation_config(null, rest))]`, `max_new_tokens = - min(rest.max_new_tokens ?? 440, 448 − decoder_input_ids.length)`, and returns the output Tensor - sliced `[:, prefix.length:]`. - -- [x] Step 1: failing tests with a fake model (records kwargs, returns a real `Tensor` int64 of - `[decoder_input_ids..., generated...]`): no kwarg → kwargs passed through untouched and no - `decoder_input_ids`; with prefix `[50362, 1, 2]` and init `[50258, 50259, 50360]` → - decoder_input_ids equal concatenation, private kwarg absent, output tokens start at 50258; - `.en`-style init `[50257]` works; 224-token prefix + 440 requested → max_new_tokens = 448 − 227; - installing twice wraps once. `buildPromptPrefix` with a fake tokenizer: `""` → `[]`; `"Maxwell"` - → `[prevId, ...encode(" Maxwell")]`. -- [x] Step 2: FAIL. Step 3: implement. Step 4: PASS. Commit. - -### Task 3: protocol + worker + engine + live plumbing - -**Files:** Modify `src/worker/protocol.ts`, `src/worker/inference.worker.ts`, `src/lib/engine.ts`; -Tests `tests/protocol.test.ts`, `tests/engine.test.ts` - -**Interfaces — Produces:** `TranscribeMessage.prompt?: string`; `EngineTranscribeOptions.prompt?: -string`; BrowserEngine forwards non-empty `prompt`; ServerEngine adds `prompt` query param when -non-empty. - -- [x] Step 1: failing tests — guard accepts string `prompt`, rejects `prompt: 5`; BrowserEngine - posts `prompt` (and omits it when `""`/undefined); ServerEngine URL carries - `prompt=Maxwell%2C+Voices…` (URLSearchParams encoding) and none when empty. -- [x] Step 2: FAIL. Step 3: implement; worker: `installPromptSupport(asr.model)` once after - pipeline creation (and lazily in transcribe), `options[PROMPT_PREFIX_KWARG] = - buildPromptPrefix(asr.tokenizer, message.prompt ?? "")` when non-empty; cite D-WS2a. Step 4: PASS; - typecheck. Commit. - -### Task 4: server validation, sidecar request, live WS, sidecar.py - -**Files:** Modify `src/server/transcribe.ts`, `src/server/live.ts`, `src/index.ts`, -`server-py/sidecar.py`, `tests/fixtures/fake_sidecar.py`; Tests `tests/transcribe.test.ts`, -`tests/server-live.test.ts`, `tests/hint-parity.test.ts` - -**Interfaces — Produces:** `parseLanguageAndTask(url)` → `{ language?, task, prompt? }` | error -string (`prompt` = `normalizeHint(raw)` or absent; raw > 1000 chars → `"prompt too long (max 1000 -characters)"`); `SidecarTranscribeRequest.prompt?: string`; sidecar JSON `"prompt": str | null`; -`LiveSessionData.prompt?: string`; Python `fit_prompt_ids(hint, encode, budget=223) -> list[int]` -(+ `normalize_hint`). - -- [x] Step 1: failing tests — parse: `?prompt=Maxwell%0AVoid` → `"Maxwell, Void"`; empty/whitespace - → no prompt; 1001 chars → error string; POST with `prompt=Århus, Ærø` reaches sidecar request as - that string (Review Focus 4); live session passes `prompt` to the sidecar; createSidecar - pass-through via fake sidecar echo (`prompt=` appended only when present); parity: Python - `fit_prompt_ids` == TS `fitHintToBudget` with the char-code fake encoder over a table of cases - (spawn `python3 -I -c` importing `server-py/sidecar.py`, which imports faster-whisper only in - `main()`). -- [x] Step 2: FAIL. Step 3: implement; sidecar encodes with - `cached_model.hf_tokenizer.encode(text, add_special_tokens=False).ids`, passes - `initial_prompt=ids` (hotwords decided in Task 6). Step 4: PASS. Commit. - -### Task 5: UI — `VocabularyHint` component + App wiring (impeccable) - -**Files:** Create `src/components/VocabularyHint.tsx`; Modify `src/App.tsx`, `src/index.css`, -`DESIGN.md` - -- [x] Step 1: render below the language/task options for **every** model: label "Names and terms - (optional)", 2-row textarea (`maxLength={HINT_MAX_CHARS}`), placeholder with an example, one line - of help (what it does, that it is remembered in this browser, long lists are cut at the end). - Disabled while busy/live. State initialized from `loadHint()`, saved on change. -- [x] Step 2: pass `prompt: hint` in the file run, `transcribeLiveSegment` options, and the server - live WS URL (`prompt` param). Typecheck + fast suite. -- [x] Step 3: impeccable polish/harden on the component; tokens only; DESIGN.md component line. -- [x] Step 4: build, serve `PORT=3120 NODE_ENV=production bun src/index.ts`, headless Chrome - screenshots light/dark × 390 px/1280 px; fix what looks wrong. Commit. - -### Task 6: measurements (under the lock) → D-WS2a - -Scratch scripts in `/tmp/ws2/` (not committed), importing the worktree's modules. - -- [x] Browser path (onnxruntime-node, catalog dtypes, `wasm` defaults): en_alien whole file, turbo - q4 and one small model (base.en or tiny.en, downloaded under the lock), without hint vs hint - `"Maxwell, Voices of the Void"`: WER (test normalizer), count of `maxwell` and `voices of the - void` vs reference counts, wall-clock, punctuation count (a prompt without a final full stop may - suppress punctuation — if so, `normalizeHint` appends `.` and Task 1 tests are updated). -- [x] Short-window hallucination: 1–3 s low-noise and 2 s speech snippets with/without hint, turbo - + small model; output must not contain hint words that are absent from the audio. -- [x] Server path (venv, `mobiuslabsgmbh/faster-whisper-large-v3-turbo` cached, int8 CPU): - en_alien none / initial_prompt / hotwords / both. Keep hotwords only if it measurably beats - initial_prompt alone on hinted-word recall without WER loss. Implement the outcome in sidecar.py. -- [x] Write D-WS2a (bottom of `docs/Decisions.md`) with numbers labelled CPU/onnxruntime-node, - truncation rule, wrapper rationale, 4.3.2 check. Architecture.md data-flow note; WorkBreakdown - tick/noticed. Commit. - -### Task 7: real-model tests (`tests/live-real-model.test.ts`) - -- [x] Whole-file assertion in its own small block after `fileWer` is computed: - `expect(fileWer).toBeLessThanOrEqual(MAX_WER_BY_CLIP[clip.base] ?? MAX_WER)` (file-mode gate, - shared with workstream 3 — keep it separate for an easy merge). -- [x] New `describe("vocabulary hint (real model)")` using the worker's `installPromptSupport` / - `buildPromptPrefix`: (a) first 120 s of en_alien with the hint → hinted words present at least - as often as without, WER ≤ 0.35, output does not begin with the hint; (b) en_aiproblem with a - 1000-char hint → no error, WER ≤ MAX_WER; (c) 2 s low-noise audio with the hint → output contains - neither "Maxwell" nor "Void". -- [x] Run the file under the lock; green. Commit. - -### Task 8: finish - -- [ ] `bun run typecheck`, fast suite, `bun run build` green; `git status` clean. -- [ ] `git rm docs/plans/001-2-vocabulary-hint.md`, commit last. - -## Resume notes (after the pause) - -- Tasks 1–5 done in commits 942799d…f7d0525 (UI screenshots taken at 3120 before the pause). -- Task 6: browser-path runs done (`/tmp/ws2/run2.jsonl`, `run3.jsonl`, out-*.txt); they showed a - near-full-budget prompt (≈220 tokens) hurting base.en badly (WER 0.339 → 0.476; a 300-term - synthetic list → 1.05), so the budget is being re-measured before D-WS2a is written. - faster-whisper none/initial/hotwords/both measured (`/tmp/ws2/fw.jsonl`); outcome not yet applied. -- Task 7: tests written (WIP commit 826bcd2); the three hint tests passed once under the lock; the - full file (whole-file gate) not yet run.