Merge feat/parakeet-engine: Parakeet TDT 0.6B v3 English engine (spec 001 workstream 7)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # AGENTS.md # bun.lock # docs/Decisions.md # src/lib/models.ts # src/worker/inference.worker.ts # src/worker/protocol.ts # tests/models.test.ts
This commit is contained in:
commit
30969e86b7
29 files changed
+1741
-54
No files matched your search
+37
-4
@@ -72,13 +72,17 @@ src/
|
||||
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
|
||||
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
|
||||
mic.ts # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
|
||||
models.ts # model catalog: ids, sizes, dtype defaults per device, languages
|
||||
models.ts # model catalog: ids, family (whisper|parakeet), sizes, dtype defaults per device, languages
|
||||
parakeet.ts # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
|
||||
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
|
||||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||||
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
|
||||
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
|
||||
worker/
|
||||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
|
||||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
|
||||
parakeetEngine.ts # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-WS7a)
|
||||
modelFiles.ts # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
|
||||
ortShim.ts # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
|
||||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
|
||||
@@ -279,11 +283,40 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
- Anonymous users are never blocked from browser inference — auth only unlocks the
|
||||
server engine.
|
||||
|
||||
### 4.1c Parakeet (English-only, desktop-only browser model — D-WS7a)
|
||||
Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). For
|
||||
`family: "parakeet"` the worker skips the Transformers.js pipeline:
|
||||
1. **Load:** `modelFiles.ts` fetches `vocab.txt`, the decoder_joint and the
|
||||
encoder from the Hub (revision-pinned URLs, `lib/parakeet.ts`) into the same
|
||||
`transformers-cache` Cache Storage cache Transformers.js uses, posting the same
|
||||
`load-progress` messages — so badges, the storage panel, delete and
|
||||
ModelLoadProgress work unchanged. A file is cached only once complete.
|
||||
Devices: `parakeetDeviceAttempts` — WASM int8 unless WebGPU is chosen
|
||||
explicitly (fp16 encoder on WebGPU + int8 decoder on WASM, needs
|
||||
`shader-f16`); the D-025 restart rule applies.
|
||||
2. **Runtime:** `parakeetEngine.ts` builds parakeet.js's `ParakeetModel` from our
|
||||
own sessions on `onnxruntime-web/webgpu` — the module Transformers.js imports,
|
||||
so there is one runtime and one `/ort/` wasm. parakeet.js's `initOrt` (own ort
|
||||
copy, CDN wasm path) is never called; `scripts/ortAlias.ts` points its import
|
||||
at `worker/ortShim.ts` so the bundle carries one ort (`tests/workerBundle.test.ts`).
|
||||
3. **Transcribe:** audio is planned into ≤ 30 s windows cut at the quietest 100 ms
|
||||
near each limit (`subarray` views, no copies); each window's word timestamps
|
||||
become sentence-sized `TranscriptChunk`s with absolute times. File passes post
|
||||
one `partial` + `transcribe-progress` per window; live passes resolve via
|
||||
`result` like Whisper's, so `LiveTranscriber`, TranscriptView and export are
|
||||
untouched. `language`/`task` are ignored (English only).
|
||||
4. **UI:** the picker lists it under English-only with a one-line note, hides it on
|
||||
phones/tablets (`modelsForDevice`), and the server-engine list never includes
|
||||
it (no faster-whisper name). With it selected, a note replaces the language/task
|
||||
controls.
|
||||
|
||||
## 5. Invariants
|
||||
|
||||
- **Privacy:** browser-engine audio and transcripts never touch the network.
|
||||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||||
- **One browser inference codebase:** Transformers.js in the browser; the server
|
||||
engine runs faster-whisper (D-013). The model catalog stays shared across both.
|
||||
- **One browser inference runtime:** onnxruntime-web, driven by Transformers.js for
|
||||
Whisper and by parakeet.js for Parakeet (D-WS7a) — the same ort instance, never a
|
||||
second runtime; the server engine runs faster-whisper (D-013). The model catalog
|
||||
stays shared across both engines (browser-only entries have no faster-whisper name).
|
||||
- **Worker isolation:** inference never runs on the main thread.
|
||||
Reference in new issue
Block a user