Merge feat/vocabulary-hint: names-and-terms decoder prompt (spec 001 workstream 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/Architecture.md
#	docs/Decisions.md
#	docs/WorkBreakdown.md
#	server-py/sidecar.py
#	src/App.tsx
#	src/server/live.ts
#	src/server/transcribe.ts
#	tests/fixtures/fake_sidecar.py
#	tests/transcribe.test.ts
This commit is contained in:
Lucas Winther committed 2026-10-30 01:35:28 +01:00
commit 2c87471fdc
26 files changed
+1177 -21

No files matched your search

+15 -2
View File
@@ -67,7 +67,7 @@ src/
index.html # HTML entry, imports frontend.tsx
frontend.tsx # React root
App.tsx # top-level UI composition
components/ # ModelPicker, TranscriptionOptions, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
components/ # ModelPicker, TranscriptionOptions, VocabularyHint, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
lib/
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
@@ -76,9 +76,11 @@ src/
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
engine.ts # engine abstraction: BrowserEngine | ServerEngine
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
worker/
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
protocol.ts # typed postMessage messages (shared with UI)
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
translateProtocol.ts # typed messages for the translate worker
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
@@ -150,8 +152,15 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
transfers the `Float32Array` to the worker. For multilingual models the
`transcribe` message also carries `language` (or auto-detect) and `task`
(transcribe | translate — translate only for models with `canTranslate`, D-WS4a).
For every model it may carry `prompt`: the optional "Names and terms" hint
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-WS2a).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
cut to a 120-token prompt budget at a whole word) and is prepended
to the decoder input of every 30 s window by a wrapper around the model's
`generate()`, which also strips it from the output again (D-WS2a).
Live passes (4.1b) carry the same `prompt`.
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
@@ -230,7 +239,11 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
JSON-lines HTTP response (D-011). File requests (`mode: "file"`) decode with
JSON-lines HTTP response (D-011). `language`, `task` and the vocabulary hint
(`prompt`) travel as query params (also on the live WebSocket) and are
validated by `parseLanguageAndTask` (D-032, D-WS2a); the sidecar cuts the
hint with the same rule as the browser and passes it to faster-whisper as
`initial_prompt` (D-WS2a). File requests (`mode: "file"`) decode with
faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live
passes send `mode: "live"` and stay sequential (D-WS5a). The shared catalog maps model ids to
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,