Merge feat/parakeet-engine: Parakeet TDT 0.6B v3 English engine (spec 001 workstream 7)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	AGENTS.md
#	bun.lock
#	docs/Decisions.md
#	src/lib/models.ts
#	src/worker/inference.worker.ts
#	src/worker/protocol.ts
#	tests/models.test.ts
This commit is contained in:
Lucas Winther committed 2026-10-30 01:46:51 +01:00
commit 30969e86b7
29 files changed
+1741 -54

No files matched your search

+37 -4
View File
@@ -72,13 +72,17 @@ src/
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
mic.ts # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
models.ts # model catalog: ids, sizes, dtype defaults per device, languages
models.ts # model catalog: ids, family (whisper|parakeet), sizes, dtype defaults per device, languages
parakeet.ts # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
engine.ts # engine abstraction: BrowserEngine | ServerEngine
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
worker/
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
parakeetEngine.ts # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-WS7a)
modelFiles.ts # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
ortShim.ts # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
protocol.ts # typed postMessage messages (shared with UI)
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
@@ -279,11 +283,40 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
- Anonymous users are never blocked from browser inference — auth only unlocks the
server engine.
### 4.1c Parakeet (English-only, desktop-only browser model — D-WS7a)
Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). For
`family: "parakeet"` the worker skips the Transformers.js pipeline:
1. **Load:** `modelFiles.ts` fetches `vocab.txt`, the decoder_joint and the
encoder from the Hub (revision-pinned URLs, `lib/parakeet.ts`) into the same
`transformers-cache` Cache Storage cache Transformers.js uses, posting the same
`load-progress` messages — so badges, the storage panel, delete and
ModelLoadProgress work unchanged. A file is cached only once complete.
Devices: `parakeetDeviceAttempts` — WASM int8 unless WebGPU is chosen
explicitly (fp16 encoder on WebGPU + int8 decoder on WASM, needs
`shader-f16`); the D-025 restart rule applies.
2. **Runtime:** `parakeetEngine.ts` builds parakeet.js's `ParakeetModel` from our
own sessions on `onnxruntime-web/webgpu` — the module Transformers.js imports,
so there is one runtime and one `/ort/` wasm. parakeet.js's `initOrt` (own ort
copy, CDN wasm path) is never called; `scripts/ortAlias.ts` points its import
at `worker/ortShim.ts` so the bundle carries one ort (`tests/workerBundle.test.ts`).
3. **Transcribe:** audio is planned into ≤ 30 s windows cut at the quietest 100 ms
near each limit (`subarray` views, no copies); each window's word timestamps
become sentence-sized `TranscriptChunk`s with absolute times. File passes post
one `partial` + `transcribe-progress` per window; live passes resolve via
`result` like Whisper's, so `LiveTranscriber`, TranscriptView and export are
untouched. `language`/`task` are ignored (English only).
4. **UI:** the picker lists it under English-only with a one-line note, hides it on
phones/tablets (`modelsForDevice`), and the server-engine list never includes
it (no faster-whisper name). With it selected, a note replaces the language/task
controls.
## 5. Invariants
- **Privacy:** browser-engine audio and transcripts never touch the network.
- **Anonymous-first:** every feature except the server engine works logged out.
- **Offline-first:** app shell + previously-downloaded models must work offline.
- **One browser inference codebase:** Transformers.js in the browser; the server
engine runs faster-whisper (D-013). The model catalog stays shared across both.
- **One browser inference runtime:** onnxruntime-web, driven by Transformers.js for
Whisper and by parakeet.js for Parakeet (D-WS7a) — the same ort instance, never a
second runtime; the server engine runs faster-whisper (D-013). The model catalog
stays shared across both engines (browser-only entries have no faster-whisper name).
- **Worker isolation:** inference never runs on the main thread.