lucasw89/whisper_web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge feat/translate: canTranslate + Opus-MT transcript translation (spec 001 workstream 4)

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/Decisions.md
This commit is contained in:
Lucas Winther
2026-10-30 01:15:05 +01:00
parent 316f64609b b88f406ac6
commit c94689e2e0
28 changed files with 1888 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Decisions.md
+71
View File
@@ -1360,3 +1360,74 @@ check is re-run.
**Not covered here:** a real browser's ort-web wasm build (the same
`ort-wasm-simd-threaded.*` files, but multithreaded under COOP/COEP) —
`tests/offline.md` §2's WASM step is the owner's browser check.
## D-WS4a — Whisper's "Translate to English" only on models trained for it (`canTranslate`)
**Status:** Accepted (2026-10-29) · spec 001 Goal 4a
`WhisperModel` gains `canTranslate: boolean` (`src/lib/models.ts`): **true** for multilingual
tiny / base / small / medium, **false** for every `.en` model (no task token) and for
`whisper-large-v3-turbo` — OpenAI's model card says turbo was fine-tuned on transcription data only
and is not trained for translation, so its "translate" output is unreliable. Any future turbo-based
fine-tune defaults to false until tested. `TranscriptionOptions` renders the Task control only where
true (language stays). **Stale task:** the `task` state is kept, but `taskForModel(modelId, task)`
maps "translate" to "transcribe" for a model that can't translate; App applies it on model switch
(picker and the server-engine auto-switch to turbo) and again when building every file / live run's
options, so a leftover "translate" can never reach the worker or the server. **Not changed:** the
server's `parseLanguageAndTask` (D-032) still accepts "translate" for any model — the browser never
sends it for turbo now, and a direct API caller asking faster-whisper to translate with turbo gets
what faster-whisper does; rejecting it server-side would be a behaviour change for API users with no
UI benefit. Speech in another language than English can still end up in English (or Danish) through
text translation of the finished transcript (D-WS4b), for every model.
## D-WS4b — Text translation of finished transcripts with Opus-MT, in its own worker
**Status:** Accepted (2026-10-29) · spec 001 Goal 4b
**What:** any finished transcript — browser file, browser live, server engine — can be translated
da→en or en→da from a secondary "Translate transcript…" disclosure under the transcript
(`TranslatePanel`). Whisper's own task keeps its label "Translate to English" (D-WS4a). The
translation renders as a second transcript card ("Translation (English)") with its own
`.txt` / `.srt` / `.txt+ts` exports named `<file>.<en|da>.<ext>`.
**Models:** `Xenova/opus-mt-da-en` and `Xenova/opus-mt-en-da` — Transformers.js ONNX exports of
Helsinki-NLP's Opus-MT (MarianMT, ~77 M params; upstream licence Apache-2.0, the Xenova repos
carry no separate licence). Loaded at **q8**: encoder 49.8 MB + merged decoder 57.1 MB +
tokenizer/config 5.7 MB = **~113 MB per direction**, one-time. Verified to load and translate under
the pinned Transformers.js 4.2.0 (onnxruntime-node; headless-Chrome WASM run below). WASM only, with
`graphOptimizationLevel: "basic"` (D-014 — Marian ties its embeddings like Whisper's 8-bit decoders).
**Where it runs — a dedicated worker** (`src/worker/translate.worker.ts`, served at
`/translate.worker.js`, precached), not a message type on the Whisper worker: onnxruntime fixes its
WASM thread pool at first init (D-023), so a Whisper worker initialised for WebGPU — the owner's
setup — would run MT single-threaded; a separate worker also keeps a Whisper cancel/restart
(D-018/D-025) from killing a translation and vice versa, and keeps workstreams 2/3's edits to
`inference.worker.ts` conflict-free. Nothing goes to the server for any engine (anti-goal: text stays
on the device). **Downloads:** only when the user clicks Translate; the hint states the size first
("Downloads a translation model once (~113 MB), then works offline") and switches to "Translation
model downloaded — works offline" once the Cache API holds it (D-005/D-020). The storage panel lists
and deletes the two models like Whisper weights.
**Chunking (Review Focus 5):** `buildTranslationUnits` merges Whisper chunks until one ends a sentence
(cap 8 chunks) and gives the unit the merged time span, so `.srt` keeps working with slightly longer
cues; untimed text is split per line. Each unit is cut by `splitForTranslation` into **one piece per
sentence** (`Intl.Segmenter`, abbreviation/initial fragments merged forward), an over-budget sentence
wrapped on commas, then words, under 200 real tokens (Opus-MT's input limit is 512; the tokenizer
otherwise *truncates silently*). One sentence per call is required, not just tidy: given two
sentences Opus-MT dropped the second one in the probe ("…I live in Copenhagen. How are you?" →
first sentence only). Order and every word are pinned by unit tests on all of `en_alien.txt` and by
the opt-in real-model test (`MT_INTEGRATION=1`), which also translates a 900-word unpunctuated run-on.
**Batching is off (batch size 1):** Transformers.js 4.2's batched Marian generation is broken —
every row of a padded batch comes back as row 0's translation (q8 and fp32, any optimisation level,
onnxruntime-node). Batch 8 would be ~4× faster; re-measure with `scripts/bench-translate.ts` after
the Transformers.js upgrade (workstream 1) before raising `DEFAULT_TRANSLATE_BATCH_SIZE`.
**Measured (onnxruntime-node, CPU, 16 cores shared with other workstreams; not browser numbers):**
`en_alien.txt` (7 470 chars, 279 lines → 293 sentence pieces) en→da **58 s** (~128 chars/s,
~0.2 s/sentence) in the bench (71 s / ~105 chars/s on a re-run with other workstreams' jobs
running), 76 s in the real-model test under load; `da_festuge.txt` (3 sentences)
da→en 2.8 s incl. warm-up; model load from cache 0.3–0.4 s. Extrapolated, a 90-minute meeting
(~80 000 chars) is ~10 min on this CPU — a background action, hence the sentence-count progress and
Cancel. **Headless Chrome (WASM, `--disable-gpu`, cross-origin isolated, 12 threads):** a real
end-to-end run — Whisper Tiny transcript of `da_festuge.m4a`, "Translate transcript…" → download
`Xenova/opus-mt-en-da` → "Translating… 0 of 1 sentence" → "Translation (Danish)" card with four
exports — took 4–8 s from click to result for that one-sentence transcript (download included);
no browser number for a long transcript was taken. Sample da→en: "At the Aarhus Festival Secretariat we have been running very strong in the
last few months."
**Errors** go through `classifyError` with the new stage `"translate"` (out-of-memory suggests closing
tabs instead of a smaller Whisper model; a failed download is a retryable network error). Cancel
terminates the translate worker only.
**Not done / follow-ups:** WebGPU for Opus-MT (unmeasurable here); more language pairs (the catalog
is two entries, `TRANSLATION_MODELS`); a translation of live text *while* recording.