Draft spec 001: transcription quality and speed
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
e195883b45
commit
2bef4cb80a
@@ -0,0 +1,156 @@
|
||||
---
|
||||
status: draft
|
||||
---
|
||||
|
||||
# 001 — Transcription quality and speed
|
||||
|
||||
## Intent and why
|
||||
|
||||
(Owner answers, 2026-10-29: English meetings, browser engine, Large v3 Turbo, ~1 h 30 min files;
|
||||
the backend in use was **believed to be WebGPU**; no real meeting recording is available for
|
||||
measurement.)
|
||||
|
||||
The owner transcribes **English meeting recordings (~1 h 30 min)** in the **browser engine** with
|
||||
**Whisper Large v3 Turbo**, and finds the app "too slow and clunky compared to the real one"
|
||||
(faster-whisper running natively): transcription is slow, and the WER is high — especially on
|
||||
**names and domain jargon**. A research report (2026-10-29, summarised below where it matters)
|
||||
concluded that no better in-browser engine exists than the current Transformers.js/onnxruntime-web
|
||||
stack, that most of faster-whisper's speed comes from portable *pipeline* logic (VAD, segmenting,
|
||||
quality fallback, batching) the browser path lacks, and that further wins come from Danish
|
||||
fine-tunes, server-engine settings, translation fixes, a runtime upgrade, and (for English) Parakeet.
|
||||
|
||||
Users are mostly non-technical (PRODUCT.md § Audience): improvements must work with defaults and
|
||||
plain-language UI, while every technical control stays available.
|
||||
|
||||
**Success:** measurable improvement, not a target number. For each workstream, before/after
|
||||
numbers from the repo's own harness (wall-clock, WER against `tests/audio_files/*.txt`, generated
|
||||
token counts) on CPU/WASM, recorded in the workstream's `D-###` entry. The owner explicitly asked
|
||||
not to aim for a specific speed.
|
||||
|
||||
## Goals
|
||||
|
||||
Seven workstreams, run in parallel (separate agents, separate git worktrees) after this spec is
|
||||
approved:
|
||||
|
||||
1. **Transformers.js 4.2.0 → 4.3.2.** Rebase `patches/@huggingface%[email protected]`
|
||||
(sequential session loading, D-027/D-028). Re-test what the upgrade may obsolete — D-014 (WASM
|
||||
`graphOptimizationLevel: "basic"`), D-021 (`max_new_tokens: 440` to avoid the seek loop), D-022
|
||||
(medium fp16), D-026 (Firefox WebGPU) — as far as the environment allows; record results as new
|
||||
entries that supersede, never rewrite.
|
||||
2. **Vocabulary hint (names and terms).** An optional free-text field, plain wording (e.g. "Names
|
||||
and terms (optional)"), remembered in the browser (localStorage). Passed to Whisper as a decoder
|
||||
prompt for every window of a file transcription and every live pass (Transformers.js
|
||||
`prompt_ids`), and to the server engine as faster-whisper's `initial_prompt` (and `hotwords` if
|
||||
it measurably helps). Works for every Whisper model in the catalog.
|
||||
3. **Browser long-form pipeline** (report Task 4), a model-agnostic module in the worker, in
|
||||
TypeScript, behind a small interface (so a future Rust→WASM implementation can replace it; it
|
||||
must call the existing Transformers.js models, not bring its own inference):
|
||||
- Silero VAD (ONNX, run on the same onnxruntime-web instance) to drop silence and cut on speech
|
||||
boundaries, merging speech into chunks ≤ ~30 s (faster-whisper defaults as the starting point:
|
||||
`min_silence_duration_ms=2000`, `speech_pad_ms=400`).
|
||||
- Repetition guard first (`no_repeat_ngram_size` / `repetition_penalty`, as D-035 proposed); if
|
||||
loops remain, per-segment quality check with temperature fallback (reject when gzip
|
||||
compression ratio > 2.4 or avg log-prob < −1.0; retry at 0.2, 0.4 … 1.0).
|
||||
- Language detected once per file on the first speech segment, then pinned.
|
||||
- **Batching is out of this round** (owner runs WebGPU; it can't be measured here, and it would
|
||||
otherwise ship untested). Keep the pipeline interface able to take it later.
|
||||
- `TranscribeProgressTracker` and its tests updated for variable-length segments.
|
||||
- Live mode (D-016/D-024) keeps its own segmenter unless VAD measurably helps there too.
|
||||
4. **Translate.**
|
||||
- (a) A per-model `canTranslate` flag in `src/lib/models.ts`: false for large-v3-turbo, every
|
||||
`.en` model, and any turbo-based fine-tune not tested for translation. `TranscriptionOptions`
|
||||
hides the translate option where it's false.
|
||||
- (b) Text translation of the finished transcript with Opus-MT (MarianMT) through Transformers.js
|
||||
— same runtime, no new dependency — enabling both da→en and en→da. **First thing to cut** if
|
||||
the workstream grows; (a) ships regardless.
|
||||
5. **Server engine speedups** (report Task 1): in `server-py/sidecar.py`, `vad_filter=True` and
|
||||
`faster_whisper.BatchedInferencePipeline(...).transcribe(..., batch_size=8)`. Segment streaming
|
||||
to `src/server/transcribe.ts` and the D-021 heartbeat must keep working.
|
||||
6. **Danish fine-tuned Whisper models** (report Task 2):
|
||||
- Candidates: `thorhojhus/whisper-small-danish` (MIT), `thorhojhus/whisper-large-v3-turbo-danish`
|
||||
(MIT), `danish-foundation-models/edda-v0.2` (Apache-2.0 per the report — **confirm licence and
|
||||
that it is plain Whisper architecture before using it**; drop it otherwise).
|
||||
- Convert each to the onnx-community layout (encoder + merged KV-cached decoder; fp32/fp16/q8/q4)
|
||||
and to CTranslate2 int8 for the server engine.
|
||||
- **Publish the conversions to the owner's Hugging Face account** (owner's decision), crediting
|
||||
the original authors and carrying their licences — **but show the owner first**: convert and
|
||||
verify locally, then stop and present what would be published (repos, files, sizes, model
|
||||
cards, measured WER). Account name and write token are requested at that point; nothing is
|
||||
uploaded before the owner says go. Catalog entries land with the repo ids the owner confirms.
|
||||
- Add catalog entries (`verified: false`), with `fasterWhisperModel` pointing at the published
|
||||
CT2 conversion. Check that language tokens / generation config survived the export.
|
||||
7. **Parakeet TDT 0.6B v3 as an English-only engine** (report Task 6): offered as its own model
|
||||
choice, desktop-only (its size trips the mobile guard, D-027–D-029), WASM int8 as the baseline
|
||||
path. Use Transformers.js if it supports Parakeet natively; otherwise `parakeet.js` (MIT, runs on
|
||||
onnxruntime-web — no second ML runtime). Needs its own engine path: the browser path assumes the
|
||||
Whisper pipeline and `WhisperTextStreamer`. Must chunk to ≤ ~35 s per encoder call.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- A specific speed target.
|
||||
- Replacing Transformers.js/onnxruntime-web with another runtime, or porting CTranslate2 to WASM.
|
||||
- Starting a Rust/WASM build from this repo (the Rust-vs-TypeScript decision for the pipeline stays
|
||||
with the owner; TypeScript behind an interface until then).
|
||||
- Non-Whisper Danish models (`RyeAI/ekko-v1-tiny`, `syvai/hviske-v6`), Moonshine, LLM-decoder ASR.
|
||||
- ffmpeg.wasm for undecodable video formats.
|
||||
|
||||
## Anti-goals (failure even if it "works")
|
||||
|
||||
- Any audio or transcript leaving the device on the browser engine. The vocabulary hint, VAD,
|
||||
translation, and Parakeet all run locally.
|
||||
- Making the app harder for a non-technical user: new required settings, ML jargon in primary UI,
|
||||
or a "simple mode" that removes controls.
|
||||
- Accuracy regressions hidden behind speed gains: `tests/live-accuracy.test.ts` and
|
||||
`tests/live-real-model.test.ts` must stay green; a guard that changes correct wording counts.
|
||||
- Breaking offline-first: anything new the browser needs (VAD model, Opus-MT, Parakeet) is a
|
||||
user-initiated download cached like Whisper weights (D-005, D-020), never fetched implicitly.
|
||||
- Setting a catalog `verified` flag to `true` without a real-browser run.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Repo rules in `AGENTS.md`: Bun only; no cross-origin subresources (COEP); colours as tokens;
|
||||
every behaviour change gets a new `D-###` entry; specs here, plans in `docs/plans/` (removed
|
||||
before merge); TDD; branches `feat/` `fix/` `chore/`; `bun run typecheck` + fast suite + build
|
||||
green before merge.
|
||||
- Environment: headless Chrome is available, but **no GPU and no microphone**. WebGPU and
|
||||
WebNN behaviour can't be measured here; CPU/WASM and the onnxruntime-node harness can. Because
|
||||
the owner runs WebGPU, every change whose effect is WebGPU-specific is listed in the final
|
||||
report as "needs the owner's browser check"; no WebGPU-only default changes ship on WASM
|
||||
numbers alone.
|
||||
- Workstream 1 changes the runtime all browser workstreams depend on: it goes first and fast; the
|
||||
others branch from it if it lands quickly, otherwise proceed on 4.2.0 and rebase.
|
||||
- Workstreams 2, 3 and 4 all edit `src/worker/inference.worker.ts` / `protocol.ts`: merge in a
|
||||
fixed order and resolve conflicts at each merge.
|
||||
- Publishing to Hugging Face is the only outward-facing step; it needs a write token for the
|
||||
owner's account.
|
||||
- New runtime dependencies need owner approval (AGENTS.md); `parakeet.js` is pre-approved by this
|
||||
spec if Transformers.js lacks native Parakeet support.
|
||||
|
||||
## Calls made by the spec author (not discussed with the owner — flag if wrong)
|
||||
|
||||
- **Rollout of the new browser pipeline (workstream 3).** It becomes the default file-transcription
|
||||
path only if, on every measurement clip, it is not worse on WER (tolerance +0.01 absolute) and is
|
||||
faster. Otherwise it ships behind an "Advanced" option, off by default, and the old 30 s
|
||||
chunk/stride path stays the default. The old path also remains the fallback when the VAD model
|
||||
isn't cached (offline, never downloaded). Each guard/threshold is judged the same way, so mixed
|
||||
results never ship as a silent default.
|
||||
- **A file-mode accuracy gate.** `tests/live-real-model.test.ts` computes whole-file WER but only
|
||||
asserts live WER; workstreams 2 and 3 add a whole-file WER assertion (same per-clip bars) so
|
||||
file-mode regressions fail the suite.
|
||||
- **Text translation (4b) scope.** Any finished transcript can be translated — browser-engine file
|
||||
and live, and server-engine results — because it is text-only post-processing in the browser
|
||||
worker (nothing new on the server). Where Whisper's own translate is available (`canTranslate`),
|
||||
both are offered; Whisper's stays labelled "Translate to English", the new one "Translate
|
||||
transcript…".
|
||||
- **Conversion runs here**, in a Python virtualenv outside the repo (optimum, transformers,
|
||||
ctranslate2); nothing Python is added to the app beyond the existing sidecar.
|
||||
- **Measurement fallback** (no real meeting recording exists, per the owner). The builder makes a stand-in
|
||||
(existing clips joined with inserted silence to resemble a meeting's pauses, plus a hint list of
|
||||
the names in `en_alien`'s reference), clearly labelled as synthetic in every D-### entry.
|
||||
|
||||
## Left to the builder
|
||||
|
||||
Exact UI placement and copy (within PRODUCT.md's audience rules), protocol message shapes, module
|
||||
boundaries, test fixtures, measurement scripts, VAD/guard thresholds after measurement, which
|
||||
Opus-MT checkpoints, dtype defaults for new catalog entries, and how Parakeet appears in the model
|
||||
picker.
|
||||
Reference in New Issue
Block a user