whisper_web/docs/plans/001-7-parakeet.md

status

status
in-progress

001-7 — Parakeet TDT 0.6B v3 engine path: implementation plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — coordination plan). Steps use checkbox (- [ ]) syntax for tracking.

Goal: NVIDIA Parakeet TDT 0.6B v3 as an English-only, desktop-only browser-engine model choice (file + live), WASM int8 baseline, opt-in WebGPU, measured against Whisper Large v3 Turbo.

Architecture: The catalog gains family?: "whisper" | "parakeet". The worker dispatches on it: Whisper keeps the Transformers.js pipeline; Parakeet runs parakeet.js's ParakeetModel (TDT greedy decoder + JS mel preprocessor) on the same onnxruntime-web instance Transformers.js imports (onnxruntime-web/webgpu), with weights fetched by our own Cache-API downloader into the transformers-cache cache (so D-005 badges / storage panel / delete work unchanged). Audio is cut into ≤ 30 s windows at the quietest point near each boundary; word timestamps become sentence chunks in the app's TranscriptChunk shape, so TranscriptView, export, progress, and live work unchanged.

Tech Stack: Bun, TypeScript, React 19, onnxruntime-web 1.26.0-dev (via Transformers.js 4.2.0), parakeet.js 1.4.4 (MIT, devDependency, bundled into the worker only).

Spec: docs/specs/001-transcription-quality-and-speed.md Goal 7; coordination docs/plans/001-0-coordination.md.

Runtime decision (Step 1 — done before planning; evidence for D-WS7a)

• @huggingface/transformers 4.2.0 and 4.3.2 (npm latest) ship only models/parakeet/modeling_parakeet.js → ParakeetForCTC; the ASR pipeline routes model_type: 'parakeet_ctc' through the wav2vec2 CTC path (argmax, no timestamps, no chunking). No TDT/transducer decoder exists in either (grep tdt|transducer|rnnt: no hits). The model dirs are identical between the two versions. ⇒ no native Parakeet TDT support → parakeet.js.
• parakeet.js 1.4.4 pins onnxruntime-web 1.24.1 and dynamically imports bare onnxruntime-web in backend.js (initOrt, CDN wasmPaths default). We never call fromUrls/fromHub/initOrt; we construct ParakeetModel directly with our ort, our tokenizer, and JsPreprocessor. A Bun build plugin resolves parakeet.js's bare onnxruntime-web import to onnxruntime-web/webgpu so the bundle holds one ort (no second runtime, no CDN path).
• Model repo: ysdede/parakeet-tdt-0.6b-v3-onnx (CC-BY-4.0, NeMo conversion), pinned revision e5356bf6e24bdb7a269536b0e9e7869bc034b893. Files: encoder-model.int8.onnx 652,183,999 B, decoder_joint-model.int8.onnx 18,202,004 B, encoder-model.fp16.onnx 1,238,960,452 B, decoder_joint-model.fp16.onnx 36,266,140 B, vocab.txt 102,132 B.
• Spike (ort-node CPU, 12 threads, naive 30 s cuts): en_aiproblem WER 0.030 in 0.34 s; en_alien WER 0.116 in 18.1 s. onnxruntime-web WASM in Bun: en_alien WER 0.104 in 45.2 s. Turbo: 0.261 / 420 s (D-039).

Global Constraints

• AGENTS.md rules: Bun only; no cross-origin subresources (COEP require-corp) — weights from huggingface.co via fetch in the worker as today, ort .wasm from /ort/; colours as tokens; classifyError for errors; TDD; docs duties in the same commit.
• Decision ids: D-WS7a (+ D-WS7b …) placeholders, cited in code comments.
• Heavy commands (bun test real-model, benches, downloads) under flock /tmp/whisper-heavy.lock.
• Ports 3170–3179. No push/publish. Plan git rm'd last.
• family? absent ⇒ "whisper"; existing catalog entries unedited.
• D-020: selecting never downloads; D-005: cached via Cache API, shown with real size.
• Mobile (D-027–D-029): Parakeet hidden in the picker and blocked from loading on isLikelyMobile.
• English only: no language/task controls; plain copy that translation isn't available.
• WebGPU: never auto-selected for Parakeet; explicit WebGPU = fp16 encoder on WebGPU + int8 decoder on WASM, only with shader-f16, falling back to WASM (D-025 restart). Owner's browser check.
• Measurements are CPU numbers (ort-node and ort-web WASM in Bun); say so.

Review Focus

1. 90-minute file: windows are subarray views (no copies of the whole file per window); only the word list grows. Test: planChunks on 90 min of samples returns ≥ 180 windows each ≤ 30 s, contiguous, total = length (Task 2).
2. Silence-only audio: empty transcript, progress reaches total, no throw (Task 2, transcribeWindows with a recognizer returning no words).
3. Speech crossing a window boundary: cut lands in the quietest 100 ms frame of the last 6 s — test with a silent gap planted at 27 s (Task 2).
4. Parakeet left selected, then phone UA: never offered, never loaded (Task 4 modelsForDevice; Task 6 App guard).
5. Cancel mid-download / mid-transcribe: worker termination (D-018) — the downloader only puts a complete response, so a half download never shows as "available offline" (Task 3 test).

---

Task 1: Catalog family, exact sizes, Parakeet entry

Files: Modify src/lib/models.ts, tests/models.test.ts.

Interfaces — Produces:

• type ModelFamily = "whisper" | "parakeet"; WhisperModel.family?: ModelFamily; modelFamily(model: WhisperModel | undefined): ModelFamily (absent ⇒ "whisper").

• WhisperModel.fasterWhisperModel?: string (absent for browser-only families).

• WhisperModel.exactBytes?: { encoder: Partial<Record<WhisperDtype, number>>; decoder: Partial<Record<WhisperDtype, number>>; aux: number } — when present, estimateDownloadBytes / estimatePeakLoadBytes use it.

• PARAKEET_MODEL_ID = "ysdede/parakeet-tdt-0.6b-v3-onnx", PARAKEET_REVISION.

• Entry: displayName "Parakeet v3 (English)", multilingual: false, family: "parakeet", defaults wasm "q8", webgpu { encoder_model: "fp16", decoder_model_merged: "q8" }, webnn "q8".

• Tests: modelFamily default; Parakeet entry family/multilingual/no fasterWhisperModel; wasm download bytes = 652,183,999 + 18,202,004 + 102,132; webgpu = fp16 encoder + int8 decoder + aux; peak = 2 × encoder bytes; existing fasterWhisperModel test restricted to Whisper family.

• Fail → implement → pass → typecheck → commit.

Task 2: Pure Parakeet logic (src/lib/parakeet.ts)

Files: Create src/lib/parakeet.ts, tests/parakeet.test.ts.

Produces:

• planChunks(length: number, energyAt: (start: number, end: number) => number, opts?: { sampleRate?: number; maxSeconds?: number; searchSeconds?: number; frameSeconds?: number }): Array<[number, number]> — defaults 16 000 / 30 / 6 / 0.1; plus planAudioChunks(audio: Float32Array): Array<[number, number]> using mean-square energy.

• wordsToChunks(words: ParakeetWord[], opts?: { gapSeconds?: number; maxSegmentSeconds?: number }): TranscriptChunk[] — break after .?! (optionally followed by quotes/brackets), on gap ≥ 1.0 s, or once a segment reaches 12 s; chunk text " " + words.join(" ") (Whisper convention: leading space).

• ParakeetWord = { text: string; start_time: number; end_time: number }.

• parseVocab(text: string): string[]; class ParakeetVocabTokenizer (id2token, blankId, blankToken, sanitizedTokens, decode(ids)) — the shape ParakeetModel reads.

• parakeetFileName(part: "encoder" | "decoder", dtype: WhisperDtype): string (q8 → .int8.onnx, fp16 → .fp16.onnx, else throws) and parakeetFileUrl(file: string): string (https://huggingface.co/<id>/resolve/<rev>/<file>).

• transcribeWindows(audio: Float32Array, recognize: (window: Float32Array, offsetSeconds: number) => Promise<ParakeetWord[]>, callbacks?: { onWindow?: (chunks: TranscriptChunk[], processedSeconds: number, totalSeconds: number) => void }): Promise<{ text: string; chunks: TranscriptChunk[] }>.

• Tests: window bounds (≤ 30 s, contiguous, cover all; short audio = one window; 90 min); quiet-gap cut at 27 s; wordsToChunks sentence/gap/max splits and timestamps; vocab parse + decode (▁ → space, blank skipped, no space before punctuation); file names/URL; transcribeWindows offsets/progress/partials and silence-only.

• Fail → implement → pass → commit.

Task 3: Cache-API model file downloader (src/worker/modelFiles.ts)

Produces: fetchModelFile(url: string, file: string, deps: { caches: CacheStorageLike | undefined; fetch: typeof fetch; onProgress: (message: LoadProgressMessage) => void }): Promise<Uint8Array> — cache name TRANSFORMERS_CACHE_NAME; hit ⇒ initiate/done, no fetch; miss ⇒ streamed progress (loaded/total/progress), put only after the full body, then done; non-OK ⇒ throws Could not download <file> (HTTP <status>).

• Tests with fake cache/fetch: hit, miss+store+progress, HTTP error doesn't store, no Cache Storage ⇒ still fetches.
• Fail → implement → pass → commit.

Task 4: Device policy + mobile filter

Files: src/worker/protocol.ts (+ tests/protocol.test.ts), src/lib/mobileMemoryGuard.ts (+ tests/mobileMemoryGuard.test.ts).

Produces: parakeetDeviceAttempts(preference: DevicePreference, capabilities: DeviceCapabilities): InferenceDevice[] — "webgpu" with webgpu && webgpuF16 ⇒ ["webgpu","wasm"]; everything else ⇒ ["wasm"] (auto never picks WebGPU, WebNN unsupported). isModelAllowedOnDevice(model, hint): boolean / modelsForDevice(models, hint) — false for family === "parakeet" when isLikelyMobile.

• Tests → fail → implement → pass → commit.

Task 5: Worker engine path

Files: Create src/worker/parakeetEngine.ts; modify src/worker/inference.worker.ts, scripts/build.ts, src/index.ts (shared ortAliasPlugin in scripts/ortAlias.ts).

Produces: createParakeetEngine(ort: OrtLike, assets: { encoder: Uint8Array; decoder: Uint8Array; vocab: string }, opts: { encoderOptions: object; decoderOptions: object }): Promise<ParakeetEngine>; ParakeetEngine.transcribe(audio, callbacks) → { text, chunks }, .dispose(). Worker: load dispatches on modelFamily(getModel(id)); Parakeet load = fetch files (Task 3) → sessions on the attempt device (Task 4, D-025 restart reused) → ready. transcribe dispatches; Parakeet ignores language/task, posts transcribe-progress/partial per window when streaming, result always (live works through the same message).

• Real-model test tests/parakeet-real-model.test.ts (opt-in PARAKEET_REAL=1, reads files from node_modules/@huggingface/transformers/.cache/<id>/, runs createParakeetEngine on onnxruntime-web WASM): whole-file and live WER per clip ≤ the turbo bars (0.25; en_alien 0.35), chunks span the clip. Run under the lock.
• Build check: dist/inference.worker.js contains the 1.26.0-dev ort version string and not 1.24.1 (no second runtime bundled); result recorded in D-WS7a.
• Commit.

Task 6: UI

Files: src/App.tsx, src/components/ModelPicker.tsx, src/index.css if needed. (impeccable for copy/layout.)

• Picker: modelsForDevice filters Parakeet out on phones; Parakeet row gets a one-line note "English only · much faster on long recordings · no translation" (final copy via impeccable).
• App: if the selected model isn't allowed on this device, fall back to DEFAULT_MODEL_ID; guard load paths; the WASM "slower than real time" live hint only for Whisper family.
• Typecheck, build, headless Chrome screenshots (desktop + phone UA, light/dark) on port 3170.
• Commit.

Task 7: Measurement + docs

• scripts/bench-parakeet.ts: per clip WER + wall-clock for Parakeet (ort-node, ort-web WASM) and Turbo (ort-node, worker options) — run under the lock.
• D-WS7a (runtime + engine path + numbers), Architecture.md (worker families), WorkBreakdown, README/AGENTS command for the opt-in test, CC-BY attribution note.
• Fast suite, typecheck, build; commit; git rm this plan; commit.