Merge feat/longform-pipeline: VAD long-form pipeline, opt-in (spec 001 workstream 3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/Architecture.md
#	docs/Decisions.md
#	src/App.tsx
#	src/index.css
#	src/lib/engine.ts
#	src/worker/inference.worker.ts
#	src/worker/protocol.ts
#	tests/engine.test.ts
#	tests/live-real-model.test.ts
#	tests/offline.md
#	tests/protocol.test.ts
This commit is contained in:
Lucas Winther committed 2026-10-30 01:39:17 +01:00
commit 9c89ab047f
31 files changed
+2264 -16

No files matched your search

+22 -1
View File
@@ -84,6 +84,9 @@ src/
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
translateProtocol.ts # typed messages for the translate worker
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
longform/ # long-form file pipeline (D-WS3a): pure vad.ts / quality.ts / pipeline.ts
# behind transcribeLongForm(audio, deps, options); transformers.ts implements
# the deps (Silero VAD + Whisper decodes) on the same Transformers.js runtime
pwa/
sw.ts # service worker (app-shell precache, /ort runtime cache)
precache.ts # placeholder-injection helpers shared by build + dev serve
@@ -161,6 +164,22 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
to the decoder input of every 30 s window by a wrapper around the model's
`generate()`, which also strips it from the output again (D-WS2a).
Live passes (4.1b) carry the same `prompt`.
6. **Long-form branch (D-WS3a/D-WS3b, "Advanced → Skip silence", off by default).**
With the option on, every load posted for a file run or the Download button carries
`vad: true`; once the Whisper pipeline is ready the worker loads Silero VAD
(`onnx-community/silero-vad`, 2.2 MB, Cache API like the weights; offline it is
used only if already cached). A file pass with `longForm: true` then runs
`transcribeLongForm` (`worker/longform/`): VAD speech probabilities over the
whole buffer → speech spans (faster-whisper `vad.py` port) → chunks ≤ 30 s cut on
speech boundaries → one Whisper decode per chunk (language detected once on the
first chunk and pinned for multilingual models) → timestamps mapped back to the
file's timeline; progress follows the variable chunks
(`TranscribeProgressTracker.forWindows`). Same `partial` /
`transcribe-progress` / `result` messages as step 5. `choosePipeline` keeps the
chunked path whenever the VAD model isn't loaded, and live passes never take
this branch (live keeps its RMS segmenter, D-016/D-024). A finished transcript
with no words shows "No speech was found in this recording." The names-and-terms
hint (`prompt`) is applied to every chunk's decode the same way (D-WS2a).
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
@@ -224,7 +243,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
large to precache eagerly.
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
independent of the SW, which never touches that cache (D-005). `/api/*` is
network-only.
network-only. The Silero VAD model of the long-form branch lives there too
(D-WS3b); offline, the worker checks that cache before loading it and never
fetches, falling back to the chunked path if it is missing.
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
- Result: with the app installed and at least one model previously loaded, the full