diff --git a/docs/plans/001-7-parakeet.md b/docs/plans/001-7-parakeet.md deleted file mode 100644 index e648695..0000000 --- a/docs/plans/001-7-parakeet.md +++ /dev/null @@ -1,187 +0,0 @@ ---- -status: in-progress ---- - -# 001-7 — Parakeet TDT 0.6B v3 engine path: implementation plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — -> coordination plan). Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** NVIDIA Parakeet TDT 0.6B v3 as an English-only, desktop-only browser-engine model choice -(file + live), WASM int8 baseline, opt-in WebGPU, measured against Whisper Large v3 Turbo. - -**Architecture:** The catalog gains `family?: "whisper" | "parakeet"`. The worker dispatches on it: -Whisper keeps the Transformers.js pipeline; Parakeet runs `parakeet.js`'s `ParakeetModel` (TDT greedy -decoder + JS mel preprocessor) on **the same onnxruntime-web instance Transformers.js imports** -(`onnxruntime-web/webgpu`), with weights fetched by our own Cache-API downloader into the -`transformers-cache` cache (so D-005 badges / storage panel / delete work unchanged). Audio is cut into -≤ 30 s windows at the quietest point near each boundary; word timestamps become sentence chunks in the -app's `TranscriptChunk` shape, so TranscriptView, export, progress, and live work unchanged. - -**Tech Stack:** Bun, TypeScript, React 19, onnxruntime-web 1.26.0-dev (via Transformers.js 4.2.0), -`parakeet.js` 1.4.4 (MIT, devDependency, bundled into the worker only). - -**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 7; coordination -`docs/plans/001-0-coordination.md`. - -## Runtime decision (Step 1 — done before planning; evidence for D-WS7a) - -- `@huggingface/transformers` 4.2.0 **and** 4.3.2 (npm `latest`) ship only - `models/parakeet/modeling_parakeet.js` → `ParakeetForCTC`; the ASR pipeline routes - `model_type: 'parakeet_ctc'` through the wav2vec2 CTC path (argmax, no timestamps, no chunking). - No TDT/transducer decoder exists in either (grep `tdt|transducer|rnnt`: no hits). The model dirs are - identical between the two versions. ⇒ **no native Parakeet TDT support → `parakeet.js`.** -- `parakeet.js` 1.4.4 pins `onnxruntime-web` 1.24.1 and dynamically imports bare `onnxruntime-web` in - `backend.js` (`initOrt`, CDN `wasmPaths` default). We never call `fromUrls`/`fromHub`/`initOrt`; we - construct `ParakeetModel` directly with our `ort`, our tokenizer, and `JsPreprocessor`. A Bun build - plugin resolves parakeet.js's bare `onnxruntime-web` import to `onnxruntime-web/webgpu` so the bundle - holds one ort (no second runtime, no CDN path). -- Model repo: `ysdede/parakeet-tdt-0.6b-v3-onnx` (CC-BY-4.0, NeMo conversion), pinned revision - `e5356bf6e24bdb7a269536b0e9e7869bc034b893`. Files: `encoder-model.int8.onnx` 652,183,999 B, - `decoder_joint-model.int8.onnx` 18,202,004 B, `encoder-model.fp16.onnx` 1,238,960,452 B, - `decoder_joint-model.fp16.onnx` 36,266,140 B, `vocab.txt` 102,132 B. -- Spike (ort-node CPU, 12 threads, naive 30 s cuts): en_aiproblem WER 0.030 in 0.34 s; en_alien - WER 0.116 in 18.1 s. onnxruntime-web WASM in Bun: en_alien WER 0.104 in 45.2 s. Turbo: 0.261 / - 420 s (D-039). - -## Global Constraints - -- AGENTS.md rules: Bun only; no cross-origin subresources (COEP require-corp) — weights from - huggingface.co via `fetch` in the worker as today, ort `.wasm` from `/ort/`; colours as tokens; - `classifyError` for errors; TDD; docs duties in the same commit. -- Decision ids: `D-WS7a` (+ `D-WS7b` …) placeholders, cited in code comments. -- Heavy commands (`bun test` real-model, benches, downloads) under `flock /tmp/whisper-heavy.lock`. -- Ports 3170–3179. No push/publish. Plan `git rm`'d last. -- `family?` absent ⇒ `"whisper"`; existing catalog entries unedited. -- D-020: selecting never downloads; D-005: cached via Cache API, shown with real size. -- Mobile (D-027–D-029): Parakeet hidden in the picker and blocked from loading on `isLikelyMobile`. -- English only: no language/task controls; plain copy that translation isn't available. -- WebGPU: never auto-selected for Parakeet; explicit WebGPU = fp16 encoder on WebGPU + int8 decoder - on WASM, only with `shader-f16`, falling back to WASM (D-025 restart). Owner's browser check. -- Measurements are CPU numbers (ort-node and ort-web WASM in Bun); say so. - -## Review Focus - -1. **90-minute file:** windows are `subarray` views (no copies of the whole file per window); only - the word list grows. Test: `planChunks` on 90 min of samples returns ≥ 180 windows each ≤ 30 s, - contiguous, total = length (Task 2). -2. **Silence-only audio:** empty transcript, progress reaches total, no throw (Task 2, - `transcribeWindows` with a recognizer returning no words). -3. **Speech crossing a window boundary:** cut lands in the quietest 100 ms frame of the last 6 s — - test with a silent gap planted at 27 s (Task 2). -4. **Parakeet left selected, then phone UA:** never offered, never loaded (Task 4 `modelsForDevice`; - Task 6 App guard). -5. **Cancel mid-download / mid-transcribe:** worker termination (D-018) — the downloader only `put`s a - complete response, so a half download never shows as "available offline" (Task 3 test). - ---- - -### Task 1: Catalog `family`, exact sizes, Parakeet entry - -**Files:** Modify `src/lib/models.ts`, `tests/models.test.ts`. - -**Interfaces — Produces:** -- `type ModelFamily = "whisper" | "parakeet"`; `WhisperModel.family?: ModelFamily`; - `modelFamily(model: WhisperModel | undefined): ModelFamily` (absent ⇒ "whisper"). -- `WhisperModel.fasterWhisperModel?: string` (absent for browser-only families). -- `WhisperModel.exactBytes?: { encoder: Partial>; decoder: Partial>; aux: number }` - — when present, `estimateDownloadBytes` / `estimatePeakLoadBytes` use it. -- `PARAKEET_MODEL_ID = "ysdede/parakeet-tdt-0.6b-v3-onnx"`, `PARAKEET_REVISION`. -- Entry: displayName "Parakeet v3 (English)", `multilingual: false`, `family: "parakeet"`, - defaults wasm `"q8"`, webgpu `{ encoder_model: "fp16", decoder_model_merged: "q8" }`, webnn `"q8"`. - -- [x] Tests: `modelFamily` default; Parakeet entry family/multilingual/no fasterWhisperModel; - wasm download bytes = 652,183,999 + 18,202,004 + 102,132; webgpu = fp16 encoder + int8 decoder + - aux; peak = 2 × encoder bytes; existing fasterWhisperModel test restricted to Whisper family. -- [x] Fail → implement → pass → typecheck → commit. - -### Task 2: Pure Parakeet logic (`src/lib/parakeet.ts`) - -**Files:** Create `src/lib/parakeet.ts`, `tests/parakeet.test.ts`. - -**Produces:** -- `planChunks(length: number, energyAt: (start: number, end: number) => number, opts?: { sampleRate?: number; maxSeconds?: number; searchSeconds?: number; frameSeconds?: number }): Array<[number, number]>` - — defaults 16 000 / 30 / 6 / 0.1; plus `planAudioChunks(audio: Float32Array): Array<[number, number]>` - using mean-square energy. -- `wordsToChunks(words: ParakeetWord[], opts?: { gapSeconds?: number; maxSegmentSeconds?: number }): TranscriptChunk[]` - — break after `.?!` (optionally followed by quotes/brackets), on gap ≥ 1.0 s, or once a segment - reaches 12 s; chunk text `" " + words.join(" ")` (Whisper convention: leading space). -- `ParakeetWord = { text: string; start_time: number; end_time: number }`. -- `parseVocab(text: string): string[]`; `class ParakeetVocabTokenizer` (`id2token`, `blankId`, - `blankToken`, `sanitizedTokens`, `decode(ids)`) — the shape `ParakeetModel` reads. -- `parakeetFileName(part: "encoder" | "decoder", dtype: WhisperDtype): string` (q8 → `.int8.onnx`, - fp16 → `.fp16.onnx`, else throws) and `parakeetFileUrl(file: string): string` - (`https://huggingface.co//resolve//`). -- `transcribeWindows(audio: Float32Array, recognize: (window: Float32Array, offsetSeconds: number) => Promise, callbacks?: { onWindow?: (chunks: TranscriptChunk[], processedSeconds: number, totalSeconds: number) => void }): Promise<{ text: string; chunks: TranscriptChunk[] }>`. - -- [x] Tests: window bounds (≤ 30 s, contiguous, cover all; short audio = one window; 90 min); - quiet-gap cut at 27 s; `wordsToChunks` sentence/gap/max splits and timestamps; vocab parse + - decode (`▁` → space, blank skipped, no space before punctuation); file names/URL; - `transcribeWindows` offsets/progress/partials and silence-only. -- [x] Fail → implement → pass → commit. - -### Task 3: Cache-API model file downloader (`src/worker/modelFiles.ts`) - -**Produces:** `fetchModelFile(url: string, file: string, deps: { caches: CacheStorageLike | undefined; fetch: typeof fetch; onProgress: (message: LoadProgressMessage) => void }): Promise` -— cache name `TRANSFORMERS_CACHE_NAME`; hit ⇒ `initiate`/`done`, no fetch; miss ⇒ streamed -`progress` (loaded/total/progress), `put` only after the full body, then `done`; non-OK ⇒ throws -`Could not download (HTTP )`. - -- [x] Tests with fake cache/fetch: hit, miss+store+progress, HTTP error doesn't store, no Cache - Storage ⇒ still fetches. -- [x] Fail → implement → pass → commit. - -### Task 4: Device policy + mobile filter - -**Files:** `src/worker/protocol.ts` (+ `tests/protocol.test.ts`), `src/lib/mobileMemoryGuard.ts` -(+ `tests/mobileMemoryGuard.test.ts`). - -**Produces:** `parakeetDeviceAttempts(preference: DevicePreference, capabilities: DeviceCapabilities): InferenceDevice[]` -— `"webgpu"` with `webgpu && webgpuF16` ⇒ `["webgpu","wasm"]`; everything else ⇒ `["wasm"]` -(auto never picks WebGPU, WebNN unsupported). `isModelAllowedOnDevice(model, hint): boolean` / -`modelsForDevice(models, hint)` — false for `family === "parakeet"` when `isLikelyMobile`. - -- [x] Tests → fail → implement → pass → commit. - -### Task 5: Worker engine path - -**Files:** Create `src/worker/parakeetEngine.ts`; modify `src/worker/inference.worker.ts`, -`scripts/build.ts`, `src/index.ts` (shared `ortAliasPlugin` in `scripts/ortAlias.ts`). - -**Produces:** `createParakeetEngine(ort: OrtLike, assets: { encoder: Uint8Array; decoder: Uint8Array; vocab: string }, opts: { encoderOptions: object; decoderOptions: object }): Promise`; -`ParakeetEngine.transcribe(audio, callbacks) → { text, chunks }`, `.dispose()`. -Worker: `load` dispatches on `modelFamily(getModel(id))`; Parakeet load = fetch files (Task 3) → -sessions on the attempt device (Task 4, D-025 restart reused) → `ready`. `transcribe` dispatches; -Parakeet ignores language/task, posts `transcribe-progress`/`partial` per window when streaming, -`result` always (live works through the same message). - -- [x] Real-model test `tests/parakeet-real-model.test.ts` (opt-in `PARAKEET_REAL=1`, reads files from - `node_modules/@huggingface/transformers/.cache//`, runs `createParakeetEngine` on - onnxruntime-web WASM): whole-file and live WER per clip ≤ the turbo bars (0.25; en_alien 0.35), - chunks span the clip. Run under the lock. -- [x] Build check: `dist/inference.worker.js` contains the 1.26.0-dev ort version string and **not** - `1.24.1` (no second runtime bundled); result recorded in D-WS7a. -- [x] Commit. - -### Task 6: UI - -**Files:** `src/App.tsx`, `src/components/ModelPicker.tsx`, `src/index.css` if needed. -(impeccable for copy/layout.) - -- Picker: `modelsForDevice` filters Parakeet out on phones; Parakeet row gets a one-line note - "English only · much faster on long recordings · no translation" (final copy via impeccable). -- App: if the selected model isn't allowed on this device, fall back to `DEFAULT_MODEL_ID`; guard - load paths; the WASM "slower than real time" live hint only for Whisper family. -- [x] Typecheck, build. (Screenshots taken in /tmp before the pause; re-verify after resume.) -- [ ] Headless Chrome screenshots (desktop + phone UA, light/dark) on port 3170, re-taken on the - final build; headless-Chrome end-to-end file run (progress, transcript, .txt/.srt export). -- [x] Commit. - -### Task 7: Measurement + docs - -- [x] `scripts/bench-parakeet.ts`: per clip WER + wall-clock for Parakeet (ort-node, ort-web WASM) - and Turbo (ort-node, worker options) — run under the lock. -- [x] D-WS7a (runtime + engine path + numbers), Architecture.md (worker families), WorkBreakdown, - README/AGENTS command for the opt-in test, CC-BY attribution note. -- [x] Docs committed (9cd92b4). -- [ ] Fast suite, typecheck, build; commit; `git rm` this plan; commit.