Renumber spec 001 decision placeholders to D-040–D-051; record the Danish publication
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed
1 parent
30969e86b7
commit
0632f7843b
71 files changed
+244
-236
No files matched your search
+20
-20
@@ -27,7 +27,7 @@ Logged-in users may optionally route inference to the server.
|
||||
│ ├─ Device selection: WebNN → WebGPU → WASM fallback │
|
||||
│ └─ Model weights cached via browser Cache API │
|
||||
│ │
|
||||
│ Translate Web Worker (D-WS4b) │
|
||||
│ Translate Web Worker (D-045) │
|
||||
│ └─ Transformers.js Opus-MT (MarianMT), WASM, on user request │
|
||||
│ │
|
||||
│ Service Worker (PWA) │
|
||||
@@ -76,19 +76,19 @@ src/
|
||||
parakeet.ts # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
|
||||
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
|
||||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||||
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
|
||||
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
|
||||
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-045)
|
||||
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-048)
|
||||
worker/
|
||||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
|
||||
parakeetEngine.ts # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-WS7a)
|
||||
parakeetEngine.ts # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-051)
|
||||
modelFiles.ts # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
|
||||
ortShim.ts # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
|
||||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
|
||||
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-048)
|
||||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-045)
|
||||
translateProtocol.ts # typed messages for the translate worker
|
||||
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
|
||||
longform/ # long-form file pipeline (D-WS3a): pure vad.ts / quality.ts / pipeline.ts
|
||||
longform/ # long-form file pipeline (D-049): pure vad.ts / quality.ts / pipeline.ts
|
||||
# behind transcribeLongForm(audio, deps, options); transformers.ts implements
|
||||
# the deps (Silero VAD + Whisper decodes) on the same Transformers.js runtime
|
||||
pwa/
|
||||
@@ -158,17 +158,17 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
`AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
|
||||
transfers the `Float32Array` to the worker. For multilingual models the
|
||||
`transcribe` message also carries `language` (or auto-detect) and `task`
|
||||
(transcribe | translate — translate only for models with `canTranslate`, D-WS4a).
|
||||
(transcribe | translate — translate only for models with `canTranslate`, D-044).
|
||||
For every model it may carry `prompt`: the optional "Names and terms" hint
|
||||
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-WS2a).
|
||||
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-048).
|
||||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
|
||||
cut to a 120-token prompt budget at a whole word) and is prepended
|
||||
to the decoder input of every 30 s window by a wrapper around the model's
|
||||
`generate()`, which also strips it from the output again (D-WS2a).
|
||||
`generate()`, which also strips it from the output again (D-048).
|
||||
Live passes (4.1b) carry the same `prompt`.
|
||||
6. **Long-form branch (D-WS3a/D-WS3b, "Advanced → Skip silence", off by default).**
|
||||
6. **Long-form branch (D-049/D-050, "Advanced → Skip silence", off by default).**
|
||||
With the option on, every load posted for a file run or the Download button carries
|
||||
`vad: true`; once the Whisper pipeline is ready the worker loads Silero VAD
|
||||
(`onnx-community/silero-vad`, 2.2 MB, Cache API like the weights; offline it is
|
||||
@@ -183,7 +183,7 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
chunked path whenever the VAD model isn't loaded, and live passes never take
|
||||
this branch (live keeps its RMS segmenter, D-016/D-024). A finished transcript
|
||||
with no words shows "No speech was found in this recording." The names-and-terms
|
||||
hint (`prompt`) is applied to every chunk's decode the same way (D-WS2a).
|
||||
hint (`prompt`) is applied to every chunk's decode the same way (D-048).
|
||||
|
||||
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
|
||||
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
|
||||
@@ -216,12 +216,12 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
only when signed in with the server engine available (`EngineToggle`);
|
||||
anonymous/offline users always get the browser-engine path above.
|
||||
|
||||
### 4.1c Transcript translation (any engine — D-WS4b)
|
||||
### 4.1c Transcript translation (any engine — D-045)
|
||||
|
||||
1. Any finished transcript (browser file, browser live, server engine) shows a
|
||||
"Translate transcript…" disclosure (`TranslatePanel`). Whisper's own
|
||||
"Translate to English" task stays in `TranscriptionOptions`, only for
|
||||
models with `canTranslate` (D-WS4a).
|
||||
models with `canTranslate` (D-044).
|
||||
2. On **Translate**, the UI groups the transcript's chunks into sentence-ending
|
||||
units (`buildTranslationUnits`, merged time spans) and posts their texts to
|
||||
`translate.worker.js` — a separate worker with its own onnxruntime instance.
|
||||
@@ -248,7 +248,7 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
|
||||
independent of the SW, which never touches that cache (D-005). `/api/*` is
|
||||
network-only. The Silero VAD model of the long-form branch lives there too
|
||||
(D-WS3b); offline, the worker checks that cache before loading it and never
|
||||
(D-050); offline, the worker checks that cache before loading it and never
|
||||
fetches, falling back to the chunked path if it is missing.
|
||||
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
|
||||
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
|
||||
@@ -266,11 +266,11 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||||
JSON-lines HTTP response (D-011). `language`, `task` and the vocabulary hint
|
||||
(`prompt`) travel as query params (also on the live WebSocket) and are
|
||||
validated by `parseLanguageAndTask` (D-032, D-WS2a); the sidecar cuts the
|
||||
validated by `parseLanguageAndTask` (D-032, D-048); the sidecar cuts the
|
||||
hint with the same rule as the browser and passes it to faster-whisper as
|
||||
`initial_prompt` (D-WS2a). File requests (`mode: "file"`) decode with
|
||||
`initial_prompt` (D-048). File requests (`mode: "file"`) decode with
|
||||
faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live
|
||||
passes send `mode: "live"` and stay sequential (D-WS5a). The shared catalog maps model ids to
|
||||
passes send `mode: "live"` and stay sequential (D-047). The shared catalog maps model ids to
|
||||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||||
the server engine reports unavailable; the browser/anonymous flow is unaffected.
|
||||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||||
@@ -283,7 +283,7 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
- Anonymous users are never blocked from browser inference — auth only unlocks the
|
||||
server engine.
|
||||
|
||||
### 4.1c Parakeet (English-only, desktop-only browser model — D-WS7a)
|
||||
### 4.1c Parakeet (English-only, desktop-only browser model — D-051)
|
||||
Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). For
|
||||
`family: "parakeet"` the worker skips the Transformers.js pipeline:
|
||||
1. **Load:** `modelFiles.ts` fetches `vocab.txt`, the decoder_joint and the
|
||||
@@ -316,7 +316,7 @@ Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). Fo
|
||||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||||
- **One browser inference runtime:** onnxruntime-web, driven by Transformers.js for
|
||||
Whisper and by parakeet.js for Parakeet (D-WS7a) — the same ort instance, never a
|
||||
Whisper and by parakeet.js for Parakeet (D-051) — the same ort instance, never a
|
||||
second runtime; the server engine runs faster-whisper (D-013). The model catalog
|
||||
stays shared across both engines (browser-only entries have no faster-whisper name).
|
||||
- **Worker isolation:** inference never runs on the main thread.
|
||||
Reference in new issue
Block a user