Merge feat/translate: canTranslate + Opus-MT transcript translation (spec 001 workstream 4)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/Decisions.md
2026-10-30 01:15:05 +01:00
parent 316f64609b b88f406ac6
commit c94689e2e0
28 changed files with 1888 additions and 30 deletions
@@ -27,6 +27,9 @@ Logged-in users may optionally route inference to the server.
 │   ├─ Device selection: WebNN → WebGPU → WASM fallback            │
 │   └─ Model weights cached via browser Cache API                  │
 │                                                                  │
+│  Translate Web Worker (D-WS4b)                                   │
+│   └─ Transformers.js Opus-MT (MarianMT), WASM, on user request   │
+│                                                                  │
 │  Service Worker (PWA)                                            │
 │   ├─ Precached app shell (offline-first)                         │
 │   └─ Works with zero network once installed + model cached       │
@@ -38,6 +41,7 @@ Logged-in users may optionally route inference to the server.
 │   ├─ /            → src/index.html (bundled React app)           │
 │   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
 │   │                 since Bun doesn't bundle `new Worker` graphs) │
+│   ├─ /translate.worker.js → translation worker bundle (same)     │
 │   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
 │   ├─ /api/transcribe → server engine (auth required)             │
 │   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
@@ -71,9 +75,13 @@ src/
    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
+    translation.ts         # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
+    translate.worker.ts    # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
+    translateProtocol.ts   # typed messages for the translate worker
+    opusMt.ts              # Opus-MT loader shared by the worker, bench script and real-model test
  pwa/
    sw.ts                  # service worker (app-shell precache, /ort runtime cache)
    precache.ts            # placeholder-injection helpers shared by build + dev serve
@@ -141,7 +149,7 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
   `AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
   transfers the `Float32Array` to the worker. For multilingual models the
   `transcribe` message also carries `language` (or auto-detect) and `task`
-   (transcribe | translate).
+   (transcribe | translate — translate only for models with `canTranslate`, D-WS4a).
 5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
   back via `{type:"partial"}` messages; final result as `{type:"result"}`.

@@ -176,6 +184,24 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
   only when signed in with the server engine available (`EngineToggle`);
   anonymous/offline users always get the browser-engine path above.

+### 4.1c Transcript translation (any engine — D-WS4b)
+
+1. Any finished transcript (browser file, browser live, server engine) shows a
+   "Translate transcript…" disclosure (`TranslatePanel`). Whisper's own
+   "Translate to English" task stays in `TranscriptionOptions`, only for
+   models with `canTranslate` (D-WS4a).
+2. On **Translate**, the UI groups the transcript's chunks into sentence-ending
+   units (`buildTranslationUnits`, merged time spans) and posts their texts to
+   `translate.worker.js` — a separate worker with its own onnxruntime instance.
+3. The worker loads the Opus-MT model (q8, ~113 MB, downloaded on this first
+   request and cached by Transformers.js like Whisper weights), splits every
+   unit into sentence pieces under a token budget (`splitForTranslation`,
+   Opus-MT's 512-token input), translates them one by one, and answers one
+   string per unit. The UI reassembles `{ text, chunks }` with the units'
+   timestamps and renders it in a second `TranscriptView` with its own
+   `.txt` / `.srt` exports (`<file>.<lang>.srt`). The text never leaves the
+   device, whichever engine produced it.
+
 ### 4.2 Offline (PWA)
 - Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
  (hashed JS/CSS, `/`, worker bundle, manifest, icons, `/fonts/*` Inter subsets) on install into
@@ -1360,3 +1360,74 @@ check is re-run.
 **Not covered here:** a real browser's ort-web wasm build (the same
 `ort-wasm-simd-threaded.*` files, but multithreaded under COOP/COEP) —
 `tests/offline.md` §2's WASM step is the owner's browser check.
+
+## D-WS4a — Whisper's "Translate to English" only on models trained for it (`canTranslate`)
+**Status:** Accepted (2026-10-29) · spec 001 Goal 4a
+`WhisperModel` gains `canTranslate: boolean` (`src/lib/models.ts`): **true** for multilingual
+tiny / base / small / medium, **false** for every `.en` model (no task token) and for
+`whisper-large-v3-turbo` — OpenAI's model card says turbo was fine-tuned on transcription data only
+and is not trained for translation, so its "translate" output is unreliable. Any future turbo-based
+fine-tune defaults to false until tested. `TranscriptionOptions` renders the Task control only where
+true (language stays). **Stale task:** the `task` state is kept, but `taskForModel(modelId, task)`
+maps "translate" to "transcribe" for a model that can't translate; App applies it on model switch
+(picker and the server-engine auto-switch to turbo) and again when building every file / live run's
+options, so a leftover "translate" can never reach the worker or the server. **Not changed:** the
+server's `parseLanguageAndTask` (D-032) still accepts "translate" for any model — the browser never
+sends it for turbo now, and a direct API caller asking faster-whisper to translate with turbo gets
+what faster-whisper does; rejecting it server-side would be a behaviour change for API users with no
+UI benefit. Speech in another language than English can still end up in English (or Danish) through
+text translation of the finished transcript (D-WS4b), for every model.
+
+## D-WS4b — Text translation of finished transcripts with Opus-MT, in its own worker
+**Status:** Accepted (2026-10-29) · spec 001 Goal 4b
+**What:** any finished transcript — browser file, browser live, server engine — can be translated
+da→en or en→da from a secondary "Translate transcript…" disclosure under the transcript
+(`TranslatePanel`). Whisper's own task keeps its label "Translate to English" (D-WS4a). The
+translation renders as a second transcript card ("Translation (English)") with its own
+`.txt` / `.srt` / `.txt+ts` exports named `<file>.<en|da>.<ext>`.
+**Models:** `Xenova/opus-mt-da-en` and `Xenova/opus-mt-en-da` — Transformers.js ONNX exports of
+Helsinki-NLP's Opus-MT (MarianMT, ~77 M params; upstream licence Apache-2.0, the Xenova repos
+carry no separate licence). Loaded at **q8**: encoder 49.8 MB + merged decoder 57.1 MB +
+tokenizer/config 5.7 MB = **~113 MB per direction**, one-time. Verified to load and translate under
+the pinned Transformers.js 4.2.0 (onnxruntime-node; headless-Chrome WASM run below). WASM only, with
+`graphOptimizationLevel: "basic"` (D-014 — Marian ties its embeddings like Whisper's 8-bit decoders).
+**Where it runs — a dedicated worker** (`src/worker/translate.worker.ts`, served at
+`/translate.worker.js`, precached), not a message type on the Whisper worker: onnxruntime fixes its
+WASM thread pool at first init (D-023), so a Whisper worker initialised for WebGPU — the owner's
+setup — would run MT single-threaded; a separate worker also keeps a Whisper cancel/restart
+(D-018/D-025) from killing a translation and vice versa, and keeps workstreams 2/3's edits to
+`inference.worker.ts` conflict-free. Nothing goes to the server for any engine (anti-goal: text stays
+on the device). **Downloads:** only when the user clicks Translate; the hint states the size first
+("Downloads a translation model once (~113 MB), then works offline") and switches to "Translation
+model downloaded — works offline" once the Cache API holds it (D-005/D-020). The storage panel lists
+and deletes the two models like Whisper weights.
+**Chunking (Review Focus 5):** `buildTranslationUnits` merges Whisper chunks until one ends a sentence
+(cap 8 chunks) and gives the unit the merged time span, so `.srt` keeps working with slightly longer
+cues; untimed text is split per line. Each unit is cut by `splitForTranslation` into **one piece per
+sentence** (`Intl.Segmenter`, abbreviation/initial fragments merged forward), an over-budget sentence
+wrapped on commas, then words, under 200 real tokens (Opus-MT's input limit is 512; the tokenizer
+otherwise *truncates silently*). One sentence per call is required, not just tidy: given two
+sentences Opus-MT dropped the second one in the probe ("…I live in Copenhagen. How are you?" →
+first sentence only). Order and every word are pinned by unit tests on all of `en_alien.txt` and by
+the opt-in real-model test (`MT_INTEGRATION=1`), which also translates a 900-word unpunctuated run-on.
+**Batching is off (batch size 1):** Transformers.js 4.2's batched Marian generation is broken —
+every row of a padded batch comes back as row 0's translation (q8 and fp32, any optimisation level,
+onnxruntime-node). Batch 8 would be ~4× faster; re-measure with `scripts/bench-translate.ts` after
+the Transformers.js upgrade (workstream 1) before raising `DEFAULT_TRANSLATE_BATCH_SIZE`.
+**Measured (onnxruntime-node, CPU, 16 cores shared with other workstreams; not browser numbers):**
+`en_alien.txt` (7 470 chars, 279 lines → 293 sentence pieces) en→da **58 s** (~128 chars/s,
+~0.2 s/sentence) in the bench (71 s / ~105 chars/s on a re-run with other workstreams' jobs
+running), 76 s in the real-model test under load; `da_festuge.txt` (3 sentences)
+da→en 2.8 s incl. warm-up; model load from cache 0.3–0.4 s. Extrapolated, a 90-minute meeting
+(~80 000 chars) is ~10 min on this CPU — a background action, hence the sentence-count progress and
+Cancel. **Headless Chrome (WASM, `--disable-gpu`, cross-origin isolated, 12 threads):** a real
+end-to-end run — Whisper Tiny transcript of `da_festuge.m4a`, "Translate transcript…" → download
+`Xenova/opus-mt-en-da` → "Translating… 0 of 1 sentence" → "Translation (Danish)" card with four
+exports — took 4–8 s from click to result for that one-sentence transcript (download included);
+no browser number for a long transcript was taken. Sample da→en: "At the Aarhus Festival Secretariat we have been running very strong in the
+last few months."
+**Errors** go through `classifyError` with the new stage `"translate"` (out-of-memory suggests closing
+tabs instead of a smaller Whisper model; a failed download is a retryable network error). Cancel
+terminates the translate worker only.
+**Not done / follow-ups:** WebGPU for Opus-MT (unmeasurable here); more language pairs (the catalog
+is two entries, `TRANSLATION_MODELS`); a translation of live text *while* recording.
@@ -65,6 +65,8 @@ Notes (M1):
      surface active backend in UI
 - [x] Handle model switching (dispose old pipeline, load new)
 - [x] Language / task options for multilingual models (transcribe vs translate)
+- [x] Whisper translate only on models trained for it (`canTranslate`; hidden for
+      large-v3-turbo and `.en`, stale "translate" resets to transcribe — D-WS4a)
 - [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
      combinations in `models.ts`

@@ -347,6 +349,16 @@ cached.

 ---

+## Spec 001 — Translate (workstream 4)
+- [x] `canTranslate` per model; Task control hidden where false; stale
+      "translate" resets to transcribe (D-WS4a)
+- [x] Text translation of any finished transcript, da→en / en→da, Opus-MT
+      in its own worker, sentence-chunked, timestamps kept for exports (D-WS4b)
+- [~] Real-browser checks: offline translation (`tests/offline.md` §5),
+      WebGPU-machine speed (owner)
+
+---
+
 ## Deliberately out of scope (for now)
 Arbitrary Hub model ids, OAuth providers, multi-user transcript storage,
 whisper.cpp server backend, i18n of the UI.
@@ -378,6 +390,14 @@ Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29.
      — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the
      Backend select overflowed its panel on 390 px-wide phones.

+Found while building transcript translation (D-WS4b), 2026-10-29:
+- [ ] Transformers.js 4.2 batched Marian generation returns row 0's output for
+      every row of a padded batch, so translation runs one sentence per call
+      (~4× slower than batch 8 would be). Re-check after the 4.3 upgrade with
+      `bun scripts/bench-translate.ts` ("output identical" across batch sizes).
+- [ ] The server API still accepts `task: "translate"` with large-v3-turbo
+      (only the UI hides it, D-WS4a).
+
 ## Recurring
 - [ ] Maintain `AGENTS.md`:
  - **Add a rule only in response to a real failure.** When an agent repeatedly