diff --git a/AGENTS.md b/AGENTS.md index de0e81b..0809f55 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,6 +17,8 @@ Test all: `bun test` — ~19 min: `tests/live-real-model.test.ts` downloads and weights whenever huggingface.co is reachable. Run it when touching the live pipeline or decoding. Build: `bun run build` (rewrites `dist/`) Opt-in real sidecar test (needs `pip install -r server-py/requirements.txt`): `FW_INTEGRATION=1 bun test tests/transcribe.test.ts` +Opt-in real translation test (~3 min, downloads 2 × 113 MB Opus-MT): `MT_INTEGRATION=1 bun test tests/translate-real-model.test.ts`; +translation speed: `bun scripts/bench-translate.ts` Typecheck: `bun run typecheck` (tsc 7, < 1 s). No linter configured. Browser-only behaviour (offline PWA, mic, WebGPU) has no automated coverage: follow `tests/offline.md`, @@ -35,7 +37,8 @@ step with the commands above, in the same commit. dotenv. Use `Bun.serve()` routes + HTML imports, `bun:sqlite`, built-in `WebSocket`, `Bun.file`, `Bun.$`. Bun API docs: `node_modules/bun-types/docs/`. - **Two engines, two libraries (D-013).** Browser inference = Transformers.js, inside - `src/worker/inference.worker.ts` only — never on the main thread. Server inference = faster-whisper + `src/worker/inference.worker.ts` only (transcript translation: `src/worker/translate.worker.ts`, + D-WS4b) — never on the main thread. Server inference = faster-whisper via `server-py/sidecar.py` — never Transformers.js on the server. - **`@huggingface/transformers` is patched and a devDependency on purpose.** It is pinned by `patches/@huggingface%2Ftransformers@4.2.0.patch` (D-027/D-028) and kept out of runtime deps so the diff --git a/DESIGN.md b/DESIGN.md index e6dd7c9..357d6d2 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -31,3 +31,9 @@ One BEM-ish class block per React component in `src/components/` (`.model-picker `.live-controls`, `.engine-toggle`, `.auth-panel`, `.update-toast`, …), with modifier classes like `-selected`, `-active`, `-disabled`. Interactive states use `:hover:enabled` and `:focus-visible`. `.visually-hidden` exists for screen-reader-only text. + +Secondary disclosures (`.storage-manager`, `.translate-panel`) share one look: a 12px-radius +`--panel` card with a semibold ``, muted 0.85rem hints, outlined 8px buttons. Inside the +translate panel the one action (`.translate-button`) is the only accent-outlined button and fills +with `--accent` / `--on-accent` on hover; a translation renders as a second `.transcript` card below +it rather than a card nested inside the panel. diff --git a/docs/Architecture.md b/docs/Architecture.md index ffa963b..6afa86e 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -27,6 +27,9 @@ Logged-in users may optionally route inference to the server. │ ├─ Device selection: WebNN → WebGPU → WASM fallback │ │ └─ Model weights cached via browser Cache API │ │ │ +│ Translate Web Worker (D-WS4b) │ +│ └─ Transformers.js Opus-MT (MarianMT), WASM, on user request │ +│ │ │ Service Worker (PWA) │ │ ├─ Precached app shell (offline-first) │ │ └─ Works with zero network once installed + model cached │ @@ -38,6 +41,7 @@ Logged-in users may optionally route inference to the server. │ ├─ / → src/index.html (bundled React app) │ │ ├─ /inference.worker.js → worker bundle (Bun.build at startup, │ │ │ since Bun doesn't bundle `new Worker` graphs) │ +│ ├─ /translate.worker.js → translation worker bundle (same) │ │ ├─ /api/auth/* → Passport.js (local strategy) + sessions │ │ ├─ /api/transcribe → server engine (auth required) │ │ └─ /manifest.webmanifest, /sw.js → PWA assets │ @@ -71,9 +75,13 @@ src/ models.ts # model catalog: ids, sizes, dtype defaults per device, languages modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator) engine.ts # engine abstraction: BrowserEngine | ServerEngine + translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b) worker/ inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection protocol.ts # typed postMessage messages (shared with UI) + translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b) + translateProtocol.ts # typed messages for the translate worker + opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test pwa/ sw.ts # service worker (app-shell precache, /ort runtime cache) precache.ts # placeholder-injection helpers shared by build + dev serve @@ -176,6 +184,24 @@ environment). See `WorkBreakdown.md` for the itemized checklist.) only when signed in with the server engine available (`EngineToggle`); anonymous/offline users always get the browser-engine path above. +### 4.1c Transcript translation (any engine — D-WS4b) + +1. Any finished transcript (browser file, browser live, server engine) shows a + "Translate transcript…" disclosure (`TranslatePanel`). Whisper's own + "Translate to English" task stays in `TranscriptionOptions`, only for + models with `canTranslate` (D-WS4a). +2. On **Translate**, the UI groups the transcript's chunks into sentence-ending + units (`buildTranslationUnits`, merged time spans) and posts their texts to + `translate.worker.js` — a separate worker with its own onnxruntime instance. +3. The worker loads the Opus-MT model (q8, ~113 MB, downloaded on this first + request and cached by Transformers.js like Whisper weights), splits every + unit into sentence pieces under a token budget (`splitForTranslation`, + Opus-MT's 512-token input), translates them one by one, and answers one + string per unit. The UI reassembles `{ text, chunks }` with the units' + timestamps and renders it in a second `TranscriptView` with its own + `.txt` / `.srt` exports (`..srt`). The text never leaves the + device, whichever engine produced it. + ### 4.2 Offline (PWA) - Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell (hashed JS/CSS, `/`, worker bundle, manifest, icons, `/fonts/*` Inter subsets) on install into diff --git a/docs/Decisions.md b/docs/Decisions.md index 342ad0b..44f1207 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1241,3 +1241,52 @@ sends it for turbo now, and a direct API caller asking faster-whisper to transla what faster-whisper does; rejecting it server-side would be a behaviour change for API users with no UI benefit. Speech in another language than English can still end up in English (or Danish) through text translation of the finished transcript (D-WS4b), for every model. + +## D-WS4b — Text translation of finished transcripts with Opus-MT, in its own worker +**Status:** Accepted (2026-10-29) · spec 001 Goal 4b +**What:** any finished transcript — browser file, browser live, server engine — can be translated +da→en or en→da from a secondary "Translate transcript…" disclosure under the transcript +(`TranslatePanel`). Whisper's own task keeps its label "Translate to English" (D-WS4a). The +translation renders as a second transcript card ("Translation (English)") with its own +`.txt` / `.srt` / `.txt+ts` exports named `..`. +**Models:** `Xenova/opus-mt-da-en` and `Xenova/opus-mt-en-da` — Transformers.js ONNX exports of +Helsinki-NLP's Opus-MT (MarianMT, ~77 M params; upstream licence Apache-2.0, the Xenova repos +carry no separate licence). Loaded at **q8**: encoder 49.8 MB + merged decoder 57.1 MB + +tokenizer/config 5.7 MB = **~113 MB per direction**, one-time. Verified to load and translate under +the pinned Transformers.js 4.2.0 (onnxruntime-node; headless-Chrome WASM run below). WASM only, with +`graphOptimizationLevel: "basic"` (D-014 — Marian ties its embeddings like Whisper's 8-bit decoders). +**Where it runs — a dedicated worker** (`src/worker/translate.worker.ts`, served at +`/translate.worker.js`, precached), not a message type on the Whisper worker: onnxruntime fixes its +WASM thread pool at first init (D-023), so a Whisper worker initialised for WebGPU — the owner's +setup — would run MT single-threaded; a separate worker also keeps a Whisper cancel/restart +(D-018/D-025) from killing a translation and vice versa, and keeps workstreams 2/3's edits to +`inference.worker.ts` conflict-free. Nothing goes to the server for any engine (anti-goal: text stays +on the device). **Downloads:** only when the user clicks Translate; the hint states the size first +("Downloads a translation model once (~113 MB), then works offline") and switches to "Translation +model downloaded — works offline" once the Cache API holds it (D-005/D-020). The storage panel lists +and deletes the two models like Whisper weights. +**Chunking (Review Focus 5):** `buildTranslationUnits` merges Whisper chunks until one ends a sentence +(cap 8 chunks) and gives the unit the merged time span, so `.srt` keeps working with slightly longer +cues; untimed text is split per line. Each unit is cut by `splitForTranslation` into **one piece per +sentence** (`Intl.Segmenter`, abbreviation/initial fragments merged forward), an over-budget sentence +wrapped on commas, then words, under 200 real tokens (Opus-MT's input limit is 512; the tokenizer +otherwise *truncates silently*). One sentence per call is required, not just tidy: given two +sentences Opus-MT dropped the second one in the probe ("…I live in Copenhagen. How are you?" → +first sentence only). Order and every word are pinned by unit tests on all of `en_alien.txt` and by +the opt-in real-model test (`MT_INTEGRATION=1`), which also translates a 900-word unpunctuated run-on. +**Batching is off (batch size 1):** Transformers.js 4.2's batched Marian generation is broken — +every row of a padded batch comes back as row 0's translation (q8 and fp32, any optimisation level, +onnxruntime-node). Batch 8 would be ~4× faster; re-measure with `scripts/bench-translate.ts` after +the Transformers.js upgrade (workstream 1) before raising `DEFAULT_TRANSLATE_BATCH_SIZE`. +**Measured (onnxruntime-node, CPU, 16 cores shared with other workstreams; not browser numbers):** +`en_alien.txt` (7 470 chars, 279 lines → 293 sentence pieces) en→da **58 s** (~128 chars/s, +~0.2 s/sentence) in the bench, 76 s in the real-model test under load; `da_festuge.txt` (3 sentences) +da→en 2.8 s incl. warm-up; model load from cache 0.3–0.4 s. Extrapolated, a 90-minute meeting +(~80 000 chars) is ~10 min on this CPU — a background action, hence the sentence-count progress and +Cancel. Sample da→en: "At the Aarhus Festival Secretariat we have been running very strong in the +last few months." +**Errors** go through `classifyError` with the new stage `"translate"` (out-of-memory suggests closing +tabs instead of a smaller Whisper model; a failed download is a retryable network error). Cancel +terminates the translate worker only. +**Not done / follow-ups:** WebGPU for Opus-MT (unmeasurable here); more language pairs (the catalog +is two entries, `TRANSLATION_MODELS`); a translation of live text *while* recording. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 2ff0310..6b61cad 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -343,6 +343,16 @@ cached. --- +## Spec 001 — Translate (workstream 4) +- [x] `canTranslate` per model; Task control hidden where false; stale + "translate" resets to transcribe (D-WS4a) +- [x] Text translation of any finished transcript, da→en / en→da, Opus-MT + in its own worker, sentence-chunked, timestamps kept for exports (D-WS4b) +- [~] Real-browser checks: offline translation (`tests/offline.md` §5), + WebGPU-machine speed (owner) + +--- + ## Deliberately out of scope (for now) Arbitrary Hub model ids, OAuth providers, multi-user transcript storage, whisper.cpp server backend, i18n of the UI. @@ -374,6 +384,14 @@ Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29. — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the Backend select overflowed its panel on 390 px-wide phones. +Found while building transcript translation (D-WS4b), 2026-10-29: +- [ ] Transformers.js 4.2 batched Marian generation returns row 0's output for + every row of a padded batch, so translation runs one sentence per call + (~4× slower than batch 8 would be). Re-check after the 4.3 upgrade with + `bun scripts/bench-translate.ts` ("output identical" across batch sizes). +- [ ] The server API still accepts `task: "translate"` with large-v3-turbo + (only the UI hides it, D-WS4a). + ## Recurring - [ ] Maintain `AGENTS.md`: - **Add a rule only in response to a real failure.** When an agent repeatedly diff --git a/tests/offline.md b/tests/offline.md index c1f0912..21b9d09 100644 --- a/tests/offline.md +++ b/tests/offline.md @@ -106,6 +106,24 @@ Result: Works. Result: Works +## 5. Transcript translation offline (D-WS4b) + +1. Online: transcribe anything, open **Translate transcript…**, pick a + direction, click **Translate**. The hint said "Downloads a translation + model once (~113 MB)"; the status shows "Downloading the translation + model… N%", then "Translating… n of m sentences", then a second card + "Translation (English|Danish)" with `.txt` / `.srt` / `.txt+ts` exports + named `..`. +2. Cache Storage → `transformers-cache` now holds `Xenova/opus-mt-…` files; + `whisper-shell-*` holds `/translate.worker.js`. The hint reads + "Translation model downloaded — works offline". +3. Go offline (DevTools → Network → Offline), reload, transcribe with a + cached Whisper model, translate again in the same direction: it must work + with no network request. The other (never downloaded) direction must fail + with "You appear to be offline…" and a Retry button, not hang. + +Result: _pending (needs a real browser)_ + ## Recording the result When this passes in a real browser, flip the `[~]` offline-verification item