Merge feat/parakeet-engine: Parakeet TDT 0.6B v3 English engine (spec 001 workstream 7)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # AGENTS.md # bun.lock # docs/Decisions.md # src/lib/models.ts # src/worker/inference.worker.ts # src/worker/protocol.ts # tests/models.test.ts
2026-10-30 01:46:51 +01:00
parent 9c89ab047f b66ba7e577
commit 30969e86b7
29 changed files with 1741 additions and 54 deletions
@@ -72,13 +72,17 @@ src/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    live.ts                # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
    mic.ts                 # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
-    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
+    models.ts              # model catalog: ids, family (whisper|parakeet), sizes, dtype defaults per device, languages
+    parakeet.ts            # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
    translation.ts         # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
    hint.ts                # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
  worker/
-    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection
+    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
+    parakeetEngine.ts      # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-WS7a)
+    modelFiles.ts          # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
+    ortShim.ts             # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
    protocol.ts            # typed postMessage messages (shared with UI)
    prompt.ts              # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
    translate.worker.ts    # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
@@ -279,11 +283,40 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
 - Anonymous users are never blocked from browser inference — auth only unlocks the
  server engine.

+### 4.1c Parakeet (English-only, desktop-only browser model — D-WS7a)
+Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). For
+`family: "parakeet"` the worker skips the Transformers.js pipeline:
+1. **Load:** `modelFiles.ts` fetches `vocab.txt`, the decoder_joint and the
+   encoder from the Hub (revision-pinned URLs, `lib/parakeet.ts`) into the same
+   `transformers-cache` Cache Storage cache Transformers.js uses, posting the same
+   `load-progress` messages — so badges, the storage panel, delete and
+   ModelLoadProgress work unchanged. A file is cached only once complete.
+   Devices: `parakeetDeviceAttempts` — WASM int8 unless WebGPU is chosen
+   explicitly (fp16 encoder on WebGPU + int8 decoder on WASM, needs
+   `shader-f16`); the D-025 restart rule applies.
+2. **Runtime:** `parakeetEngine.ts` builds parakeet.js's `ParakeetModel` from our
+   own sessions on `onnxruntime-web/webgpu` — the module Transformers.js imports,
+   so there is one runtime and one `/ort/` wasm. parakeet.js's `initOrt` (own ort
+   copy, CDN wasm path) is never called; `scripts/ortAlias.ts` points its import
+   at `worker/ortShim.ts` so the bundle carries one ort (`tests/workerBundle.test.ts`).
+3. **Transcribe:** audio is planned into ≤ 30 s windows cut at the quietest 100 ms
+   near each limit (`subarray` views, no copies); each window's word timestamps
+   become sentence-sized `TranscriptChunk`s with absolute times. File passes post
+   one `partial` + `transcribe-progress` per window; live passes resolve via
+   `result` like Whisper's, so `LiveTranscriber`, TranscriptView and export are
+   untouched. `language`/`task` are ignored (English only).
+4. **UI:** the picker lists it under English-only with a one-line note, hides it on
+   phones/tablets (`modelsForDevice`), and the server-engine list never includes
+   it (no faster-whisper name). With it selected, a note replaces the language/task
+   controls.
+
 ## 5. Invariants

 - **Privacy:** browser-engine audio and transcripts never touch the network.
 - **Anonymous-first:** every feature except the server engine works logged out.
 - **Offline-first:** app shell + previously-downloaded models must work offline.
- **One browser inference codebase:** Transformers.js in the browser; the server
-  engine runs faster-whisper (D-013). The model catalog stays shared across both.
+- **One browser inference runtime:** onnxruntime-web, driven by Transformers.js for
+  Whisper and by parakeet.js for Parakeet (D-WS7a) — the same ort instance, never a
+  second runtime; the server engine runs faster-whisper (D-013). The model catalog
+  stays shared across both engines (browser-only entries have no faster-whisper name).
 - **Worker isolation:** inference never runs on the main thread.
@@ -1703,3 +1703,93 @@ path (`choosePipeline`). Live mode keeps its energy gate (D-016).
 **Rejected:** bundling the model in `dist/` (2 MB on every install for an
 off-by-default option); running it on the main thread (D-013).
 **Consequence:** the first "Skip silence" run online downloads 2.2 MB.
+
+## D-WS7a — Parakeet TDT 0.6B v3 as an English-only, desktop-only browser model via parakeet.js on the shared onnxruntime-web
+**Status:** Accepted (2026-10-29, spec 001 Goal 7)
+**Runtime decision (evidence).** `@huggingface/transformers` 4.2.0 (ours) and
+4.3.2 (npm `latest`) were inspected from their published tarballs: both ship
+only `ParakeetForCTC` (`models/parakeet/modeling_parakeet.js`, identical in
+both), routed by the ASR pipeline as `model_type: 'parakeet_ctc'` through the
+wav2vec2 CTC path (argmax, no timestamps, no chunking). Neither contains a
+TDT/transducer decoder. Parakeet TDT therefore runs on **parakeet.js 1.4.4**
+(MIT; pre-approved by the spec for exactly this case), a devDependency
+bundled into the worker only (like Transformers.js, D-031).
+**How it runs on one runtime.** parakeet.js pins its own onnxruntime-web
+1.24.1 and its `fromUrls`/`fromHub` → `initOrt` path imports it, sets a
+jsdelivr `wasmPaths` (blocked by COEP, D-023; not offline) and fetches
+weights outside our cache. We never call those: `src/worker/parakeetEngine.ts`
+constructs `ParakeetModel` directly from sessions created on
+`onnxruntime-web/webgpu` — the module Transformers.js imports — with our own
+vocab tokenizer and parakeet.js's `JsPreprocessor` (`parakeet.js/mel`). A Bun
+build plugin (`scripts/ortAlias.ts`, used by the build and the dev server)
+resolves parakeet.js's bare `onnxruntime-web` import to `src/worker/ortShim.ts`,
+so the bundle carries one ort (`tests/workerBundle.test.ts` asserts it, and
+that without the plugin the 1.24.1 copy *would* be bundled). We rely on
+`onnxruntime-web` being hoisted from Transformers.js rather than pinning it
+ourselves, so the two can never drift apart (workstream 1's upgrade moves both).
+**Model.** `ysdede/parakeet-tdt-0.6b-v3-onnx` (NVIDIA's CC-BY-4.0 weights,
+NeMo→ONNX), URLs pinned to revision `e5356bf6…` — the one measured and
+whose exact file sizes the catalog shows (int8 encoder 652.2 MB + int8
+decoder_joint 18.2 MB + vocab 0.1 MB = "~670 MB"; WebGPU fp16 encoder
+1.24 GB). Attribution in README. The model knows 25 European languages; the
+app offers it **English-only** (spec), so there is no language/task UI.
+**Engine path.** Catalog entries gain `family?: "whisper" | "parakeet"`
+(absent ⇒ whisper) and optional `exactBytes`; `fasterWhisperModel` becomes
+optional (browser-only entries have none, so the server engine 400s them and
+the picker's server list excludes them). The worker dispatches on the family.
+Weights go through `src/worker/modelFiles.ts` into the `transformers-cache`
+Cache Storage cache under their Hub URLs, posting the same `load-progress`
+messages — badges, storage panel, delete, ModelLoadProgress unchanged
+(D-005, D-033); a file is `put` only once complete. Download stays
+user-initiated (D-020). Audio is cut into ≤ 30 s windows (spec: ≤ ~35 s per
+encoder call) at the quietest 100 ms in the last 6 s before each limit, so no
+overlap merging is needed; we don't use parakeet.js's `transcribeLongAudio`
+(20–180 s windows, 90 s default). Word timestamps become sentence-sized
+`TranscriptChunk`s (break after `.?!`, on ≥ 1 s pauses, or at 12 s). File
+passes post one `partial` + absolute `transcribe-progress` per window; live
+passes return `result` like Whisper, so live mode works unchanged.
+**Devices.** WASM int8 is the baseline and the only path "auto" picks. An
+explicit WebGPU choice uses the fp16 encoder on WebGPU + int8 decoder on WASM
+(parakeet.js's "hybrid" split), only with `shader-f16`, falling back to WASM
+through the D-025 restart. Not auto: WebGPU int8 is reported broken, the fp16
+encoder is a separate 1.24 GB download, and nothing WebGPU can be measured
+here. WebNN has no path (runs WASM). WASM sessions use graph optimization
+"all" (D-014's "basic" cap is about Whisper's 8-bit tied embeddings; measured
+fine here).
+**Mobile.** Load peak ≈ 2 × 652 MB = 1.3 GB (D-027 arithmetic) — above every
+budget D-029 treats as safe (deviceMemory 8 ⇒ 800 MB). Instead of a confirm,
+`modelsForDevice` hides it on phones/tablets (incl. iPadOS) and
+`confirmRiskyMobileLoad` refuses it.
+**Measurements (CPU only; this machine, 16 cores; synthetic nothing).**
+`PARAKEET_REAL=1 bun test tests/parakeet-real-model.test.ts` (onnxruntime-web
+WASM under Bun, 8 threads, the worker's own engine code):
+
+| clip | audio | file WER | file wall | live WER | live wall |
+|---|---|---|---|---|---|
+| en_aiproblem | 14.4 s | 0.030 | 0.9 s | 0.030 | 2.0 s |
+| en_alien | 888.4 s | 0.096 | 36.4 s | 0.115 | 282 s |
+
+`scripts/bench-parakeet.ts` (whole-file, same normalization, 12 threads,
+model load excluded, one run each under the shared lock):
+
+| engine (runtime) | en_aiproblem WER / wall | en_alien WER / wall |
+|---|---|---|
+| Parakeet int8 (onnxruntime-node CPU) | 0.030 / 0.3 s | 0.100 / 15.3 s |
+| Parakeet int8 (onnxruntime-web WASM) | 0.030 / 0.9 s | 0.096 / 34.4 s |
+| Large v3 Turbo q4 (onnxruntime-node, worker options) | 0.000 / 2.9 s | 0.255 / 183.7 s |
+
+(D-039 recorded Turbo en_alien whole-file 0.261 / 420 s on a busier run.) On
+the long, noisy clip Parakeet more than halves Turbo's WER and is ~12× faster
+on the same runtime; on the short clean clip Turbo is perfect and Parakeet
+misses one word ("agentic" → "agency"). No WASM number for Turbo: the
+Transformers.js node build always uses onnxruntime-node.
+**Consequence.** For English, Parakeet is the accurate *and* fast choice on
+desktop; Whisper stays the default and the only multilingual/translate
+option. Live mode is supported (faster than real time on WASM here).
+**Needs the owner's browser check:** real-Chrome WASM load + transcription
+speed/memory (headless Chrome run here, see WorkBreakdown); explicit WebGPU
+(fp16 encoder) on the owner's GPU — success, speed vs WASM, and whether the
+fallback restart behaves; Firefox/Safari WASM.
+**Revisit:** if Transformers.js gains a TDT decoder, move to it and drop
+parakeet.js + the alias plugin; if WebGPU measures well on real hardware,
+consider letting "auto" pick it.
@@ -372,6 +372,22 @@ cached.

 ---

+## Spec 001 Goal 7 — Parakeet TDT 0.6B v3 (English, desktop, D-WS7a)
+- [x] Runtime decision: Transformers.js 4.2.0/4.3.2 have CTC-only Parakeet → parakeet.js on the
+      shared onnxruntime-web (one runtime in the bundle, `tests/workerBundle.test.ts`).
+- [x] Catalog `family` + exact sizes; worker engine path (≤ 30 s windows, word → chunk
+      timestamps, progress/partials); Cache-API downloader (`modelFiles.ts`).
+- [x] File and live mode; WASM int8 baseline; explicit WebGPU = fp16 encoder (opt-in only).
+- [x] Hidden on phones/tablets; English-only note instead of language/task controls.
+- [x] Real-model WER/wall-clock vs Large v3 Turbo (CPU), opt-in `PARAKEET_REAL=1` test.
+- [x] Headless Chrome (no GPU, fresh profile, prod build, COOP/COEP on): picking Parakeet and
+      dropping `en_aiproblem.m4a` downloaded the weights, cached them ("available offline"),
+      ran on WASM (12 threads) and showed the transcript, 2 timestamp segments and the
+      .txt/.srt/.txt+ts/Copy exports — 15 s end to end including the ~670 MB download. Picker
+      screenshots: desktop light/dark show the row + English-only note; a phone UA hides it.
+- [ ] Owner's browser check: real Chrome WASM load/transcribe (speed, memory), explicit
+      WebGPU on a real GPU (works? faster than WASM? fallback restart), Firefox/Safari WASM.
+
 ## Deliberately out of scope (for now)
 Arbitrary Hub model ids, OAuth providers, multi-user transcript storage,
 whisper.cpp server backend, i18n of the UI.
@@ -433,6 +449,11 @@ Found while building the long-form pipeline (spec 001 Goal 3, D-WS3a), 2026-10-2
      `en_alien`'s reference; `tests/meetingFixture.ts` returns audio + reference
      only. Add the list when the vocabulary hint (workstream 2) lands.

+- [ ] Parakeet v3 also transcribes 24 other European languages, Danish included, but the app
+      offers it English-only (spec 001) — unmeasured on `da_festuge`. Candidate for a later spec.
+- [ ] parakeet.js still logs `[Parakeet.js]` lines to the worker console on some paths; harmless,
+      not silenced (no logger hook in the library).
+
 ## Recurring
 - [ ] Maintain `AGENTS.md`:
  - **Add a rule only in response to a real failure.** When an agent repeatedly