From 2bef4cb80ad5bd4c75833ee59deb8d3a44ec3b05 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Thu, 29 Oct 2026 21:27:01 +0100 Subject: [PATCH] Draft spec 001: transcription quality and speed Co-Authored-By: Claude Opus 5.5 --- .../001-transcription-quality-and-speed.md | 156 ++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 docs/specs/001-transcription-quality-and-speed.md diff --git a/docs/specs/001-transcription-quality-and-speed.md b/docs/specs/001-transcription-quality-and-speed.md new file mode 100644 index 0000000..cf3a9dd --- /dev/null +++ b/docs/specs/001-transcription-quality-and-speed.md @@ -0,0 +1,156 @@ +--- +status: draft +--- + +# 001 — Transcription quality and speed + +## Intent and why + +(Owner answers, 2026-10-29: English meetings, browser engine, Large v3 Turbo, ~1 h 30 min files; +the backend in use was **believed to be WebGPU**; no real meeting recording is available for +measurement.) + +The owner transcribes **English meeting recordings (~1 h 30 min)** in the **browser engine** with +**Whisper Large v3 Turbo**, and finds the app "too slow and clunky compared to the real one" +(faster-whisper running natively): transcription is slow, and the WER is high — especially on +**names and domain jargon**. A research report (2026-10-29, summarised below where it matters) +concluded that no better in-browser engine exists than the current Transformers.js/onnxruntime-web +stack, that most of faster-whisper's speed comes from portable *pipeline* logic (VAD, segmenting, +quality fallback, batching) the browser path lacks, and that further wins come from Danish +fine-tunes, server-engine settings, translation fixes, a runtime upgrade, and (for English) Parakeet. + +Users are mostly non-technical (PRODUCT.md § Audience): improvements must work with defaults and +plain-language UI, while every technical control stays available. + +**Success:** measurable improvement, not a target number. For each workstream, before/after +numbers from the repo's own harness (wall-clock, WER against `tests/audio_files/*.txt`, generated +token counts) on CPU/WASM, recorded in the workstream's `D-###` entry. The owner explicitly asked +not to aim for a specific speed. + +## Goals + +Seven workstreams, run in parallel (separate agents, separate git worktrees) after this spec is +approved: + +1. **Transformers.js 4.2.0 → 4.3.2.** Rebase `patches/@huggingface%2Ftransformers@4.2.0.patch` + (sequential session loading, D-027/D-028). Re-test what the upgrade may obsolete — D-014 (WASM + `graphOptimizationLevel: "basic"`), D-021 (`max_new_tokens: 440` to avoid the seek loop), D-022 + (medium fp16), D-026 (Firefox WebGPU) — as far as the environment allows; record results as new + entries that supersede, never rewrite. +2. **Vocabulary hint (names and terms).** An optional free-text field, plain wording (e.g. "Names + and terms (optional)"), remembered in the browser (localStorage). Passed to Whisper as a decoder + prompt for every window of a file transcription and every live pass (Transformers.js + `prompt_ids`), and to the server engine as faster-whisper's `initial_prompt` (and `hotwords` if + it measurably helps). Works for every Whisper model in the catalog. +3. **Browser long-form pipeline** (report Task 4), a model-agnostic module in the worker, in + TypeScript, behind a small interface (so a future Rust→WASM implementation can replace it; it + must call the existing Transformers.js models, not bring its own inference): + - Silero VAD (ONNX, run on the same onnxruntime-web instance) to drop silence and cut on speech + boundaries, merging speech into chunks ≤ ~30 s (faster-whisper defaults as the starting point: + `min_silence_duration_ms=2000`, `speech_pad_ms=400`). + - Repetition guard first (`no_repeat_ngram_size` / `repetition_penalty`, as D-035 proposed); if + loops remain, per-segment quality check with temperature fallback (reject when gzip + compression ratio > 2.4 or avg log-prob < −1.0; retry at 0.2, 0.4 … 1.0). + - Language detected once per file on the first speech segment, then pinned. + - **Batching is out of this round** (owner runs WebGPU; it can't be measured here, and it would + otherwise ship untested). Keep the pipeline interface able to take it later. + - `TranscribeProgressTracker` and its tests updated for variable-length segments. + - Live mode (D-016/D-024) keeps its own segmenter unless VAD measurably helps there too. +4. **Translate.** + - (a) A per-model `canTranslate` flag in `src/lib/models.ts`: false for large-v3-turbo, every + `.en` model, and any turbo-based fine-tune not tested for translation. `TranscriptionOptions` + hides the translate option where it's false. + - (b) Text translation of the finished transcript with Opus-MT (MarianMT) through Transformers.js + — same runtime, no new dependency — enabling both da→en and en→da. **First thing to cut** if + the workstream grows; (a) ships regardless. +5. **Server engine speedups** (report Task 1): in `server-py/sidecar.py`, `vad_filter=True` and + `faster_whisper.BatchedInferencePipeline(...).transcribe(..., batch_size=8)`. Segment streaming + to `src/server/transcribe.ts` and the D-021 heartbeat must keep working. +6. **Danish fine-tuned Whisper models** (report Task 2): + - Candidates: `thorhojhus/whisper-small-danish` (MIT), `thorhojhus/whisper-large-v3-turbo-danish` + (MIT), `danish-foundation-models/edda-v0.2` (Apache-2.0 per the report — **confirm licence and + that it is plain Whisper architecture before using it**; drop it otherwise). + - Convert each to the onnx-community layout (encoder + merged KV-cached decoder; fp32/fp16/q8/q4) + and to CTranslate2 int8 for the server engine. + - **Publish the conversions to the owner's Hugging Face account** (owner's decision), crediting + the original authors and carrying their licences — **but show the owner first**: convert and + verify locally, then stop and present what would be published (repos, files, sizes, model + cards, measured WER). Account name and write token are requested at that point; nothing is + uploaded before the owner says go. Catalog entries land with the repo ids the owner confirms. + - Add catalog entries (`verified: false`), with `fasterWhisperModel` pointing at the published + CT2 conversion. Check that language tokens / generation config survived the export. +7. **Parakeet TDT 0.6B v3 as an English-only engine** (report Task 6): offered as its own model + choice, desktop-only (its size trips the mobile guard, D-027–D-029), WASM int8 as the baseline + path. Use Transformers.js if it supports Parakeet natively; otherwise `parakeet.js` (MIT, runs on + onnxruntime-web — no second ML runtime). Needs its own engine path: the browser path assumes the + Whisper pipeline and `WhisperTextStreamer`. Must chunk to ≤ ~35 s per encoder call. + +## Non-goals + +- A specific speed target. +- Replacing Transformers.js/onnxruntime-web with another runtime, or porting CTranslate2 to WASM. +- Starting a Rust/WASM build from this repo (the Rust-vs-TypeScript decision for the pipeline stays + with the owner; TypeScript behind an interface until then). +- Non-Whisper Danish models (`RyeAI/ekko-v1-tiny`, `syvai/hviske-v6`), Moonshine, LLM-decoder ASR. +- ffmpeg.wasm for undecodable video formats. + +## Anti-goals (failure even if it "works") + +- Any audio or transcript leaving the device on the browser engine. The vocabulary hint, VAD, + translation, and Parakeet all run locally. +- Making the app harder for a non-technical user: new required settings, ML jargon in primary UI, + or a "simple mode" that removes controls. +- Accuracy regressions hidden behind speed gains: `tests/live-accuracy.test.ts` and + `tests/live-real-model.test.ts` must stay green; a guard that changes correct wording counts. +- Breaking offline-first: anything new the browser needs (VAD model, Opus-MT, Parakeet) is a + user-initiated download cached like Whisper weights (D-005, D-020), never fetched implicitly. +- Setting a catalog `verified` flag to `true` without a real-browser run. + +## Constraints + +- Repo rules in `AGENTS.md`: Bun only; no cross-origin subresources (COEP); colours as tokens; + every behaviour change gets a new `D-###` entry; specs here, plans in `docs/plans/` (removed + before merge); TDD; branches `feat/` `fix/` `chore/`; `bun run typecheck` + fast suite + build + green before merge. +- Environment: headless Chrome is available, but **no GPU and no microphone**. WebGPU and + WebNN behaviour can't be measured here; CPU/WASM and the onnxruntime-node harness can. Because + the owner runs WebGPU, every change whose effect is WebGPU-specific is listed in the final + report as "needs the owner's browser check"; no WebGPU-only default changes ship on WASM + numbers alone. +- Workstream 1 changes the runtime all browser workstreams depend on: it goes first and fast; the + others branch from it if it lands quickly, otherwise proceed on 4.2.0 and rebase. +- Workstreams 2, 3 and 4 all edit `src/worker/inference.worker.ts` / `protocol.ts`: merge in a + fixed order and resolve conflicts at each merge. +- Publishing to Hugging Face is the only outward-facing step; it needs a write token for the + owner's account. +- New runtime dependencies need owner approval (AGENTS.md); `parakeet.js` is pre-approved by this + spec if Transformers.js lacks native Parakeet support. + +## Calls made by the spec author (not discussed with the owner — flag if wrong) + +- **Rollout of the new browser pipeline (workstream 3).** It becomes the default file-transcription + path only if, on every measurement clip, it is not worse on WER (tolerance +0.01 absolute) and is + faster. Otherwise it ships behind an "Advanced" option, off by default, and the old 30 s + chunk/stride path stays the default. The old path also remains the fallback when the VAD model + isn't cached (offline, never downloaded). Each guard/threshold is judged the same way, so mixed + results never ship as a silent default. +- **A file-mode accuracy gate.** `tests/live-real-model.test.ts` computes whole-file WER but only + asserts live WER; workstreams 2 and 3 add a whole-file WER assertion (same per-clip bars) so + file-mode regressions fail the suite. +- **Text translation (4b) scope.** Any finished transcript can be translated — browser-engine file + and live, and server-engine results — because it is text-only post-processing in the browser + worker (nothing new on the server). Where Whisper's own translate is available (`canTranslate`), + both are offered; Whisper's stays labelled "Translate to English", the new one "Translate + transcript…". +- **Conversion runs here**, in a Python virtualenv outside the repo (optimum, transformers, + ctranslate2); nothing Python is added to the app beyond the existing sidecar. +- **Measurement fallback** (no real meeting recording exists, per the owner). The builder makes a stand-in + (existing clips joined with inserted silence to resemble a meeting's pauses, plus a hint list of + the names in `en_alien`'s reference), clearly labelled as synthetic in every D-### entry. + +## Left to the builder + +Exact UI placement and copy (within PRODUCT.md's audience rules), protocol message shapes, module +boundaries, test fixtures, measurement scripts, VAD/guard thresholds after measurement, which +Opus-MT checkpoints, dtype defaults for new catalog entries, and how Parakeet appears in the model +picker.