whisper_web/docs/plans/001-7-parakeet.md

---
status: in-progress
---

# 001-7 — Parakeet TDT 0.6B v3 engine path: implementation plan

> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents —
> coordination plan). Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** NVIDIA Parakeet TDT 0.6B v3 as an English-only, desktop-only browser-engine model choice
(file + live), WASM int8 baseline, opt-in WebGPU, measured against Whisper Large v3 Turbo.

**Architecture:** The catalog gains `family?: "whisper" | "parakeet"`. The worker dispatches on it:
Whisper keeps the Transformers.js pipeline; Parakeet runs `parakeet.js`'s `ParakeetModel` (TDT greedy
decoder + JS mel preprocessor) on **the same onnxruntime-web instance Transformers.js imports**
(`onnxruntime-web/webgpu`), with weights fetched by our own Cache-API downloader into the
`transformers-cache` cache (so D-005 badges / storage panel / delete work unchanged). Audio is cut into
≤ 30 s windows at the quietest point near each boundary; word timestamps become sentence chunks in the
app's `TranscriptChunk` shape, so TranscriptView, export, progress, and live work unchanged.

**Tech Stack:** Bun, TypeScript, React 19, onnxruntime-web 1.26.0-dev (via Transformers.js 4.2.0),
`parakeet.js` 1.4.4 (MIT, devDependency, bundled into the worker only).

**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 7; coordination
`docs/plans/001-0-coordination.md`.

## Runtime decision (Step 1 — done before planning; evidence for D-WS7a)

- `@huggingface/transformers` 4.2.0 **and** 4.3.2 (npm `latest`) ship only
  `models/parakeet/modeling_parakeet.js` → `ParakeetForCTC`; the ASR pipeline routes
  `model_type: 'parakeet_ctc'` through the wav2vec2 CTC path (argmax, no timestamps, no chunking).
  No TDT/transducer decoder exists in either (grep `tdt|transducer|rnnt`: no hits). The model dirs are
  identical between the two versions. ⇒ **no native Parakeet TDT support → `parakeet.js`.**
- `parakeet.js` 1.4.4 pins `onnxruntime-web` 1.24.1 and dynamically imports bare `onnxruntime-web` in
  `backend.js` (`initOrt`, CDN `wasmPaths` default). We never call `fromUrls`/`fromHub`/`initOrt`; we
  construct `ParakeetModel` directly with our `ort`, our tokenizer, and `JsPreprocessor`. A Bun build
  plugin resolves parakeet.js's bare `onnxruntime-web` import to `onnxruntime-web/webgpu` so the bundle
  holds one ort (no second runtime, no CDN path).
- Model repo: `ysdede/parakeet-tdt-0.6b-v3-onnx` (CC-BY-4.0, NeMo conversion), pinned revision
  `e5356bf6e24bdb7a269536b0e9e7869bc034b893`. Files: `encoder-model.int8.onnx` 652,183,999 B,
  `decoder_joint-model.int8.onnx` 18,202,004 B, `encoder-model.fp16.onnx` 1,238,960,452 B,
  `decoder_joint-model.fp16.onnx` 36,266,140 B, `vocab.txt` 102,132 B.
- Spike (ort-node CPU, 12 threads, naive 30 s cuts): en_aiproblem WER 0.030 in 0.34 s; en_alien
  WER 0.116 in 18.1 s. onnxruntime-web WASM in Bun: en_alien WER 0.104 in 45.2 s. Turbo: 0.261 /
  420 s (D-039).

## Global Constraints

- AGENTS.md rules: Bun only; no cross-origin subresources (COEP require-corp) — weights from
  huggingface.co via `fetch` in the worker as today, ort `.wasm` from `/ort/`; colours as tokens;
  `classifyError` for errors; TDD; docs duties in the same commit.
- Decision ids: `D-WS7a` (+ `D-WS7b` …) placeholders, cited in code comments.
- Heavy commands (`bun test` real-model, benches, downloads) under `flock /tmp/whisper-heavy.lock`.
- Ports 3170–3179. No push/publish. Plan `git rm`'d last.
- `family?` absent ⇒ `"whisper"`; existing catalog entries unedited.
- D-020: selecting never downloads; D-005: cached via Cache API, shown with real size.
- Mobile (D-027–D-029): Parakeet hidden in the picker and blocked from loading on `isLikelyMobile`.
- English only: no language/task controls; plain copy that translation isn't available.
- WebGPU: never auto-selected for Parakeet; explicit WebGPU = fp16 encoder on WebGPU + int8 decoder
  on WASM, only with `shader-f16`, falling back to WASM (D-025 restart). Owner's browser check.
- Measurements are CPU numbers (ort-node and ort-web WASM in Bun); say so.

## Review Focus

1. **90-minute file:** windows are `subarray` views (no copies of the whole file per window); only
   the word list grows. Test: `planChunks` on 90 min of samples returns ≥ 180 windows each ≤ 30 s,
   contiguous, total = length (Task 2).
2. **Silence-only audio:** empty transcript, progress reaches total, no throw (Task 2,
   `transcribeWindows` with a recognizer returning no words).
3. **Speech crossing a window boundary:** cut lands in the quietest 100 ms frame of the last 6 s —
   test with a silent gap planted at 27 s (Task 2).
4. **Parakeet left selected, then phone UA:** never offered, never loaded (Task 4 `modelsForDevice`;
   Task 6 App guard).
5. **Cancel mid-download / mid-transcribe:** worker termination (D-018) — the downloader only `put`s a
   complete response, so a half download never shows as "available offline" (Task 3 test).

---

### Task 1: Catalog `family`, exact sizes, Parakeet entry

**Files:** Modify `src/lib/models.ts`, `tests/models.test.ts`.

**Interfaces — Produces:**
- `type ModelFamily = "whisper" | "parakeet"`; `WhisperModel.family?: ModelFamily`;
  `modelFamily(model: WhisperModel | undefined): ModelFamily` (absent ⇒ "whisper").
- `WhisperModel.fasterWhisperModel?: string` (absent for browser-only families).
- `WhisperModel.exactBytes?: { encoder: Partial<Record<WhisperDtype, number>>; decoder: Partial<Record<WhisperDtype, number>>; aux: number }`
  — when present, `estimateDownloadBytes` / `estimatePeakLoadBytes` use it.
- `PARAKEET_MODEL_ID = "ysdede/parakeet-tdt-0.6b-v3-onnx"`, `PARAKEET_REVISION`.
- Entry: displayName "Parakeet v3 (English)", `multilingual: false`, `family: "parakeet"`,
  defaults wasm `"q8"`, webgpu `{ encoder_model: "fp16", decoder_model_merged: "q8" }`, webnn `"q8"`.

- [ ] Tests: `modelFamily` default; Parakeet entry family/multilingual/no fasterWhisperModel;
  wasm download bytes = 652,183,999 + 18,202,004 + 102,132; webgpu = fp16 encoder + int8 decoder +
  aux; peak = 2 × encoder bytes; existing fasterWhisperModel test restricted to Whisper family.
- [ ] Fail → implement → pass → typecheck → commit.

### Task 2: Pure Parakeet logic (`src/lib/parakeet.ts`)

**Files:** Create `src/lib/parakeet.ts`, `tests/parakeet.test.ts`.

**Produces:**
- `planChunks(length: number, energyAt: (start: number, end: number) => number, opts?: { sampleRate?: number; maxSeconds?: number; searchSeconds?: number; frameSeconds?: number }): Array<[number, number]>`
  — defaults 16 000 / 30 / 6 / 0.1; plus `planAudioChunks(audio: Float32Array): Array<[number, number]>`
  using mean-square energy.
- `wordsToChunks(words: ParakeetWord[], opts?: { gapSeconds?: number; maxSegmentSeconds?: number }): TranscriptChunk[]`
  — break after `.?!` (optionally followed by quotes/brackets), on gap ≥ 1.0 s, or once a segment
  reaches 12 s; chunk text `" " + words.join(" ")` (Whisper convention: leading space).
- `ParakeetWord = { text: string; start_time: number; end_time: number }`.
- `parseVocab(text: string): string[]`; `class ParakeetVocabTokenizer` (`id2token`, `blankId`,
  `blankToken`, `sanitizedTokens`, `decode(ids)`) — the shape `ParakeetModel` reads.
- `parakeetFileName(part: "encoder" | "decoder", dtype: WhisperDtype): string` (q8 → `.int8.onnx`,
  fp16 → `.fp16.onnx`, else throws) and `parakeetFileUrl(file: string): string`
  (`https://huggingface.co/<id>/resolve/<rev>/<file>`).
- `transcribeWindows(audio: Float32Array, recognize: (window: Float32Array, offsetSeconds: number) => Promise<ParakeetWord[]>, callbacks?: { onWindow?: (chunks: TranscriptChunk[], processedSeconds: number, totalSeconds: number) => void }): Promise<{ text: string; chunks: TranscriptChunk[] }>`.

- [ ] Tests: window bounds (≤ 30 s, contiguous, cover all; short audio = one window; 90 min);
  quiet-gap cut at 27 s; `wordsToChunks` sentence/gap/max splits and timestamps; vocab parse +
  decode (`▁` → space, blank skipped, no space before punctuation); file names/URL; 
  `transcribeWindows` offsets/progress/partials and silence-only.
- [ ] Fail → implement → pass → commit.

### Task 3: Cache-API model file downloader (`src/worker/modelFiles.ts`)

**Produces:** `fetchModelFile(url: string, file: string, deps: { caches: CacheStorageLike | undefined; fetch: typeof fetch; onProgress: (message: LoadProgressMessage) => void }): Promise<Uint8Array>`
— cache name `TRANSFORMERS_CACHE_NAME`; hit ⇒ `initiate`/`done`, no fetch; miss ⇒ streamed
`progress` (loaded/total/progress), `put` only after the full body, then `done`; non-OK ⇒ throws
`Could not download <file> (HTTP <status>)`.

- [ ] Tests with fake cache/fetch: hit, miss+store+progress, HTTP error doesn't store, no Cache
  Storage ⇒ still fetches.
- [ ] Fail → implement → pass → commit.

### Task 4: Device policy + mobile filter

**Files:** `src/worker/protocol.ts` (+ `tests/protocol.test.ts`), `src/lib/mobileMemoryGuard.ts`
(+ `tests/mobileMemoryGuard.test.ts`).

**Produces:** `parakeetDeviceAttempts(preference: DevicePreference, capabilities: DeviceCapabilities): InferenceDevice[]`
— `"webgpu"` with `webgpu && webgpuF16` ⇒ `["webgpu","wasm"]`; everything else ⇒ `["wasm"]`
(auto never picks WebGPU, WebNN unsupported). `isModelAllowedOnDevice(model, hint): boolean` /
`modelsForDevice(models, hint)` — false for `family === "parakeet"` when `isLikelyMobile`.

- [ ] Tests → fail → implement → pass → commit.

### Task 5: Worker engine path

**Files:** Create `src/worker/parakeetEngine.ts`; modify `src/worker/inference.worker.ts`,
`scripts/build.ts`, `src/index.ts` (shared `ortAliasPlugin` in `scripts/ortAlias.ts`).

**Produces:** `createParakeetEngine(ort: OrtLike, assets: { encoder: Uint8Array; decoder: Uint8Array; vocab: string }, opts: { encoderOptions: object; decoderOptions: object }): Promise<ParakeetEngine>`;
`ParakeetEngine.transcribe(audio, callbacks) → { text, chunks }`, `.dispose()`.
Worker: `load` dispatches on `modelFamily(getModel(id))`; Parakeet load = fetch files (Task 3) →
sessions on the attempt device (Task 4, D-025 restart reused) → `ready`. `transcribe` dispatches;
Parakeet ignores language/task, posts `transcribe-progress`/`partial` per window when streaming,
`result` always (live works through the same message).

- [ ] Real-model test `tests/parakeet-real-model.test.ts` (opt-in `PARAKEET_REAL=1`, reads files from
  `node_modules/@huggingface/transformers/.cache/<id>/`, runs `createParakeetEngine` on
  onnxruntime-web WASM): whole-file and live WER per clip ≤ the turbo bars (0.25; en_alien 0.35),
  chunks span the clip. Run under the lock.
- [ ] Build check: `dist/inference.worker.js` contains the 1.26.0-dev ort version string and **not**
  `1.24.1` (no second runtime bundled); result recorded in D-WS7a.
- [ ] Commit.

### Task 6: UI

**Files:** `src/App.tsx`, `src/components/ModelPicker.tsx`, `src/index.css` if needed.
(impeccable for copy/layout.)

- Picker: `modelsForDevice` filters Parakeet out on phones; Parakeet row gets a one-line note
  "English only · much faster on long recordings · no translation" (final copy via impeccable).
- App: if the selected model isn't allowed on this device, fall back to `DEFAULT_MODEL_ID`; guard
  load paths; the WASM "slower than real time" live hint only for Whisper family.
- [ ] Typecheck, build, headless Chrome screenshots (desktop + phone UA, light/dark) on port 3170.
- [ ] Commit.

### Task 7: Measurement + docs

- [ ] `scripts/bench-parakeet.ts`: per clip WER + wall-clock for Parakeet (ort-node, ort-web WASM)
  and Turbo (ort-node, worker options) — run under the lock.
- [ ] D-WS7a (runtime + engine path + numbers), Architecture.md (worker families), WorkBreakdown,
  README/AGENTS command for the opt-in test, CC-BY attribution note.
- [ ] Fast suite, typecheck, build; commit; `git rm` this plan; commit.