whisper_web/docs/plans/001-4-translate.md

status

status
in-progress

001-4 — Translate: implementation plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — coordination plan). Steps use checkbox (- [ ]) syntax for tracking.

Goal: (4a) a per-model canTranslate flag that hides Whisper's "Translate to English" where it is unsupported and resets a stale task: "translate"; (4b) on-device Opus-MT text translation of any finished transcript (da→en, en→da), with timestamps kept for .srt / timestamped .txt exports.

Architecture: canTranslate lives in the catalog (src/lib/models.ts) with a pure taskForModel() guard used by App on model switch and on every run. Text MT runs in a dedicated worker (src/worker/translate.worker.ts) — its own onnxruntime instance with full WASM threads, independent of the Whisper worker's lifetime (cancel/D-018 terminate, D-025 restart) and of its accelerator (an ort runtime first initialised for WebGPU is pinned single-threaded, D-023). Pure planning code (src/lib/translation.ts) groups transcript chunks into sentence-ending units, splits units into sentence pieces under a token budget, and reassembles translated units with the merged timestamps. UI: a secondary <details> disclosure "Translate transcript…" under the transcript.

Tech Stack: Bun, React 19, Transformers.js 4.2.0 (translation pipeline, MarianMT), onnxruntime-web (WASM) in the browser, onnxruntime-node for measurement.

Spec: docs/specs/001-transcription-quality-and-speed.md (Goal 4, call "Text translation (4b) scope"); coordination: docs/plans/001-0-coordination.md.

Global Constraints

• AGENTS.md rules: Bun only; no cross-origin subresources (models come from huggingface.co via Transformers.js like Whisper; ort from /ort/); colours as tokens; errors via classifyError.
• D-020 / anti-goal: the MT model is downloaded only when the user clicks Translate; the panel states the one-time size first. Cached via Transformers.js Cache API (D-005); offline works once cached.
• Nothing leaves the device: translation runs in the browser for every engine (server results too).
• Copy: Whisper's option stays "Translate to English"; the new entry is "Translate transcript…".
• canTranslate: false for onnx-community/whisper-large-v3-turbo and every .en model; true for multilingual tiny/base/small/medium.
• Decision ids: D-WS4a (canTranslate), D-WS4b (text MT). Ports 3140–3149. Heavy jobs under flock /tmp/whisper-heavy.lock.

Review Focus

1. Transcript > 512 tokens (Opus-MT input limit): split on sentence boundaries, order preserved, nothing dropped → Task 3 test splitForTranslation keeps every word of en_alien in order with every piece under budget + real-model test (Task 4) on all of en_alien.txt.
2. Multi-sentence input to Opus-MT drops later sentences (observed in the probe: "…Copenhagen. How are you?" → only the first sentence) → each sentence is its own piece (Task 3 test).
3. Unpunctuated text (live output, or a 2-minute run-on): hard-wrap on commas, then words, so no piece exceeds the budget (Task 3 test).
4. Stale task: "translate" after switching to turbo / a .en model / the server auto-switch to turbo → reset to transcribe (Task 1 test on taskForModel, App wired at every switch + run).
5. Offline / model not cached → the Translate button still offers the download; a failed download surfaces a classified network error with Retry (Task 5, errors test).

---

Task 1: canTranslate + stale-task guard (4a)

Files: Modify src/lib/models.ts, src/components/TranscriptionOptions.tsx, src/App.tsx; Test tests/models.test.ts, new tests/transcriptionOptions.test.tsx.

Interfaces — Produces: WhisperModel.canTranslate: boolean; taskForModel(modelId: string, task: WhisperTask): WhisperTask; TranscriptionOptions prop canTranslate: boolean.

• Failing tests: canTranslate false for turbo and every .en, true for the other multilingual ids; taskForModel("onnx-community/whisper-large-v3-turbo", "translate") === "transcribe", unchanged for whisper-small; unknown id → "transcribe". renderToStaticMarkup of TranscriptionOptions with canTranslate={false} contains no "Translate to English" and no Task select; with true contains it.
• Implement; App: setTask(t => taskForModel(next, t)) in handleModelSelect and the server auto-switch effect; runs pass task: taskForModel(modelId, task).
• Typecheck, fast suite; D-WS4a entry; commit.

Task 2: error stage for translation

Files: src/lib/errors.ts, tests/errors.test.ts.

• Failing test: classifyError("Out of memory", { stage: "translate" }) → suggestion mentions a shorter transcript / closing tabs, not "smaller model"; network on stage: "translate" retryable.
• Add "translate" to ErrorStage; commit with Task 3.

Task 3: pure translation planning (src/lib/translation.ts)

Interfaces — Produces:

• TRANSLATION_MODELS: readonly TranslationModel[] — { id, source: "da"|"en", target, label, approxBytes } for Xenova/opus-mt-da-en, Xenova/opus-mt-en-da (q8, ~113 MB each).
• getTranslationModel(source, target), guessSourceLanguage(text, hint?): "da" | "en".
• buildTranslationUnits(result: { text; chunks }): TranslationUnit[] — { text: string; timestamp: [number, number | null] | null }; consecutive chunks merged until the text ends a sentence (cap 8 chunks); no chunks → one unit per non-empty line.
• splitForTranslation(text, { maxTokens = 200, countTokens = estimateTokens }): string[] — Intl.Segmenter sentences (abbreviation-only fragments merge forward), then comma, then word wrap.
• assembleTranslation(units, translated: string[]): { text; chunks }.
• Tests (tests/translation.test.ts): unit merging & timestamps, null end, no-chunk fallback, sentence splitting, Review Focus 1–3, assemble length mismatch throws, guessSourceLanguage.
• Red → green → commit.

Task 4: translate worker + protocol + measurement

Files: Create src/worker/translate.worker.ts, src/worker/translateProtocol.ts, tests/translateProtocol.test.ts, tests/translate-real-model.test.ts (skips when the Hub is unreachable and the model is not cached; run under the lock), scripts/bench-translate.ts.

• Protocol: UI→worker { type: "translate"; requestId; modelId; units: string[] }; worker→UI translate-load-progress (file/loaded/total/status), translate-progress { done; total } (pieces), translate-result { texts: string[] }, translate-error { stage: "load" | "translate"; message }; guards isTranslateRequest / isTranslateWorkerMessage.
• Worker: pipeline("translation", id, { dtype: "q8", device: "wasm", session_options: { graphOptimizationLevel: "basic" } }) (D-014 — Marian also ties embeddings), WASM threads as D-023, wasmPaths "./ort/"; one cached pipeline per model id; pieces via splitForTranslation with the real tokenizer count; batches of BATCH_SIZE (measured).
• Bundle + serve: src/index.ts dev route /translate.worker.js, scripts/build.ts emits and precaches it.
• Measure en_alien.txt → da and da_festuge → en on onnxruntime-node (batch 1 vs 4 vs 8); record.
• Commit.

Task 5: UI — TranslatePanel

Files: Create src/components/TranslatePanel.tsx; modify src/App.tsx, src/components/TranscriptView.tsx (title prop + exportSuffix), src/components/StorageManager.tsx (list MT models), src/index.css.

• <details className="translate-panel"><summary>Translate transcript…</summary> with direction select ("Danish → English", "English → Danish", default from guessSourceLanguage), size line when not cached ("Downloads a translation model (~113 MB, one time)"), Translate / Cancel buttons, progress ("Downloading translation model… 40%", "Translating… 12 of 80 sentences"), error box via classifyError(…, { stage: "translate" | "load" }) with Retry when retryable.
• Result rendered by TranscriptView with title "Translation (English)" and exports named <file>.en.txt / .en.srt.
• State resets whenever result changes; Cancel terminates the translate worker.
• impeccable polish pass; screenshots light/dark × desktop/mobile; real headless-Chrome run of a translation (WASM) if feasible.
• Commit.

Task 6: docs + wrap-up

• D-WS4b entry (models, licences, numbers); Architecture.md (translate worker), AGENTS.md rule wording (two workers), WorkBreakdown, README if needed, tests/offline.md step.
• typecheck, fast suite, build; git rm this plan; commit.