Merge feat/vocabulary-hint: names-and-terms decoder prompt (spec 001 workstream 2)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/Architecture.md # docs/Decisions.md # docs/WorkBreakdown.md # server-py/sidecar.py # src/App.tsx # src/server/live.ts # src/server/transcribe.ts # tests/fixtures/fake_sidecar.py # tests/transcribe.test.ts
This commit is contained in:
commit
2c87471fdc
26 files changed
+1177
-21
No files matched your search
+15
-2
@@ -67,7 +67,7 @@ src/
|
||||
index.html # HTML entry, imports frontend.tsx
|
||||
frontend.tsx # React root
|
||||
App.tsx # top-level UI composition
|
||||
components/ # ModelPicker, TranscriptionOptions, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
|
||||
components/ # ModelPicker, TranscriptionOptions, VocabularyHint, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
|
||||
lib/
|
||||
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
|
||||
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
|
||||
@@ -76,9 +76,11 @@ src/
|
||||
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
|
||||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||||
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
|
||||
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
|
||||
worker/
|
||||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
|
||||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
|
||||
translateProtocol.ts # typed messages for the translate worker
|
||||
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
|
||||
@@ -150,8 +152,15 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
transfers the `Float32Array` to the worker. For multilingual models the
|
||||
`transcribe` message also carries `language` (or auto-detect) and `task`
|
||||
(transcribe | translate — translate only for models with `canTranslate`, D-WS4a).
|
||||
For every model it may carry `prompt`: the optional "Names and terms" hint
|
||||
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-WS2a).
|
||||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
|
||||
cut to a 120-token prompt budget at a whole word) and is prepended
|
||||
to the decoder input of every 30 s window by a wrapper around the model's
|
||||
`generate()`, which also strips it from the output again (D-WS2a).
|
||||
Live passes (4.1b) carry the same `prompt`.
|
||||
|
||||
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
|
||||
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
|
||||
@@ -230,7 +239,11 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
|
||||
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
|
||||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||||
JSON-lines HTTP response (D-011). File requests (`mode: "file"`) decode with
|
||||
JSON-lines HTTP response (D-011). `language`, `task` and the vocabulary hint
|
||||
(`prompt`) travel as query params (also on the live WebSocket) and are
|
||||
validated by `parseLanguageAndTask` (D-032, D-WS2a); the sidecar cuts the
|
||||
hint with the same rule as the browser and passes it to faster-whisper as
|
||||
`initial_prompt` (D-WS2a). File requests (`mode: "file"`) decode with
|
||||
faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live
|
||||
passes send `mode: "live"` and stay sequential (D-WS5a). The shared catalog maps model ids to
|
||||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||||
|
||||
Reference in new issue
Block a user