lucasw89/whisper_web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge feat/parakeet-engine: Parakeet TDT 0.6B v3 English engine (spec 001 workstream 7)

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	AGENTS.md
#	bun.lock
#	docs/Decisions.md
#	src/lib/models.ts
#	src/worker/inference.worker.ts
#	src/worker/protocol.ts
#	tests/models.test.ts
This commit is contained in:
Lucas Winther
2026-10-30 01:46:51 +01:00
parent 9c89ab047f b66ba7e577
commit 30969e86b7
29 changed files with 1741 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Decisions.md
+90
View File
@@ -1703,3 +1703,93 @@ path (`choosePipeline`). Live mode keeps its energy gate (D-016).
**Rejected:** bundling the model in `dist/` (2 MB on every install for an
off-by-default option); running it on the main thread (D-013).
**Consequence:** the first "Skip silence" run online downloads 2.2 MB.
## D-WS7a — Parakeet TDT 0.6B v3 as an English-only, desktop-only browser model via parakeet.js on the shared onnxruntime-web
**Status:** Accepted (2026-10-29, spec 001 Goal 7)
**Runtime decision (evidence).** `@huggingface/transformers` 4.2.0 (ours) and
4.3.2 (npm `latest`) were inspected from their published tarballs: both ship
only `ParakeetForCTC` (`models/parakeet/modeling_parakeet.js`, identical in
both), routed by the ASR pipeline as `model_type: 'parakeet_ctc'` through the
wav2vec2 CTC path (argmax, no timestamps, no chunking). Neither contains a
TDT/transducer decoder. Parakeet TDT therefore runs on **parakeet.js 1.4.4**
(MIT; pre-approved by the spec for exactly this case), a devDependency
bundled into the worker only (like Transformers.js, D-031).
**How it runs on one runtime.** parakeet.js pins its own onnxruntime-web
1.24.1 and its `fromUrls`/`fromHub` → `initOrt` path imports it, sets a
jsdelivr `wasmPaths` (blocked by COEP, D-023; not offline) and fetches
weights outside our cache. We never call those: `src/worker/parakeetEngine.ts`
constructs `ParakeetModel` directly from sessions created on
`onnxruntime-web/webgpu` — the module Transformers.js imports — with our own
vocab tokenizer and parakeet.js's `JsPreprocessor` (`parakeet.js/mel`). A Bun
build plugin (`scripts/ortAlias.ts`, used by the build and the dev server)
resolves parakeet.js's bare `onnxruntime-web` import to `src/worker/ortShim.ts`,
so the bundle carries one ort (`tests/workerBundle.test.ts` asserts it, and
that without the plugin the 1.24.1 copy *would* be bundled). We rely on
`onnxruntime-web` being hoisted from Transformers.js rather than pinning it
ourselves, so the two can never drift apart (workstream 1's upgrade moves both).
**Model.** `ysdede/parakeet-tdt-0.6b-v3-onnx` (NVIDIA's CC-BY-4.0 weights,
NeMo→ONNX), URLs pinned to revision `e5356bf6…` — the one measured and
whose exact file sizes the catalog shows (int8 encoder 652.2 MB + int8
decoder_joint 18.2 MB + vocab 0.1 MB = "~670 MB"; WebGPU fp16 encoder
1.24 GB). Attribution in README. The model knows 25 European languages; the
app offers it **English-only** (spec), so there is no language/task UI.
**Engine path.** Catalog entries gain `family?: "whisper" | "parakeet"`
(absent ⇒ whisper) and optional `exactBytes`; `fasterWhisperModel` becomes
optional (browser-only entries have none, so the server engine 400s them and
the picker's server list excludes them). The worker dispatches on the family.
Weights go through `src/worker/modelFiles.ts` into the `transformers-cache`
Cache Storage cache under their Hub URLs, posting the same `load-progress`
messages — badges, storage panel, delete, ModelLoadProgress unchanged
(D-005, D-033); a file is `put` only once complete. Download stays
user-initiated (D-020). Audio is cut into ≤ 30 s windows (spec: ≤ ~35 s per
encoder call) at the quietest 100 ms in the last 6 s before each limit, so no
overlap merging is needed; we don't use parakeet.js's `transcribeLongAudio`
(20–180 s windows, 90 s default). Word timestamps become sentence-sized
`TranscriptChunk`s (break after `.?!`, on ≥ 1 s pauses, or at 12 s). File
passes post one `partial` + absolute `transcribe-progress` per window; live
passes return `result` like Whisper, so live mode works unchanged.
**Devices.** WASM int8 is the baseline and the only path "auto" picks. An
explicit WebGPU choice uses the fp16 encoder on WebGPU + int8 decoder on WASM
(parakeet.js's "hybrid" split), only with `shader-f16`, falling back to WASM
through the D-025 restart. Not auto: WebGPU int8 is reported broken, the fp16
encoder is a separate 1.24 GB download, and nothing WebGPU can be measured
here. WebNN has no path (runs WASM). WASM sessions use graph optimization
"all" (D-014's "basic" cap is about Whisper's 8-bit tied embeddings; measured
fine here).
**Mobile.** Load peak ≈ 2 × 652 MB = 1.3 GB (D-027 arithmetic) — above every
budget D-029 treats as safe (deviceMemory 8 ⇒ 800 MB). Instead of a confirm,
`modelsForDevice` hides it on phones/tablets (incl. iPadOS) and
`confirmRiskyMobileLoad` refuses it.
**Measurements (CPU only; this machine, 16 cores; synthetic nothing).**
`PARAKEET_REAL=1 bun test tests/parakeet-real-model.test.ts` (onnxruntime-web
WASM under Bun, 8 threads, the worker's own engine code):
| clip | audio | file WER | file wall | live WER | live wall |
|---|---|---|---|---|---|
| en_aiproblem | 14.4 s | 0.030 | 0.9 s | 0.030 | 2.0 s |
| en_alien | 888.4 s | 0.096 | 36.4 s | 0.115 | 282 s |
`scripts/bench-parakeet.ts` (whole-file, same normalization, 12 threads,
model load excluded, one run each under the shared lock):
| engine (runtime) | en_aiproblem WER / wall | en_alien WER / wall |
|---|---|---|
| Parakeet int8 (onnxruntime-node CPU) | 0.030 / 0.3 s | 0.100 / 15.3 s |
| Parakeet int8 (onnxruntime-web WASM) | 0.030 / 0.9 s | 0.096 / 34.4 s |
| Large v3 Turbo q4 (onnxruntime-node, worker options) | 0.000 / 2.9 s | 0.255 / 183.7 s |
(D-039 recorded Turbo en_alien whole-file 0.261 / 420 s on a busier run.) On
the long, noisy clip Parakeet more than halves Turbo's WER and is ~12× faster
on the same runtime; on the short clean clip Turbo is perfect and Parakeet
misses one word ("agentic" → "agency"). No WASM number for Turbo: the
Transformers.js node build always uses onnxruntime-node.
**Consequence.** For English, Parakeet is the accurate *and* fast choice on
desktop; Whisper stays the default and the only multilingual/translate
option. Live mode is supported (faster than real time on WASM here).
**Needs the owner's browser check:** real-Chrome WASM load + transcription
speed/memory (headless Chrome run here, see WorkBreakdown); explicit WebGPU
(fp16 encoder) on the owner's GPU — success, speed vs WASM, and whether the
fallback restart behaves; Firefox/Safari WASM.
**Revisit:** if Transformers.js gains a TDT decoder, move to it and drop
parakeet.js + the alias plugin; if WebGPU measures well on real hardware,
consider letting "auto" pick it.