Merge feat/longform-pipeline: VAD long-form pipeline, opt-in (spec 001 workstream 3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/Architecture.md # docs/Decisions.md # src/App.tsx # src/index.css # src/lib/engine.ts # src/worker/inference.worker.ts # src/worker/protocol.ts # tests/engine.test.ts # tests/live-real-model.test.ts # tests/offline.md # tests/protocol.test.ts
This commit is contained in:
commit
9c89ab047f
31 files changed
+2264
-16
No files matched your search
+22
-1
@@ -84,6 +84,9 @@ src/
|
||||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
|
||||
translateProtocol.ts # typed messages for the translate worker
|
||||
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
|
||||
longform/ # long-form file pipeline (D-WS3a): pure vad.ts / quality.ts / pipeline.ts
|
||||
# behind transcribeLongForm(audio, deps, options); transformers.ts implements
|
||||
# the deps (Silero VAD + Whisper decodes) on the same Transformers.js runtime
|
||||
pwa/
|
||||
sw.ts # service worker (app-shell precache, /ort runtime cache)
|
||||
precache.ts # placeholder-injection helpers shared by build + dev serve
|
||||
@@ -161,6 +164,22 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
to the decoder input of every 30 s window by a wrapper around the model's
|
||||
`generate()`, which also strips it from the output again (D-WS2a).
|
||||
Live passes (4.1b) carry the same `prompt`.
|
||||
6. **Long-form branch (D-WS3a/D-WS3b, "Advanced → Skip silence", off by default).**
|
||||
With the option on, every load posted for a file run or the Download button carries
|
||||
`vad: true`; once the Whisper pipeline is ready the worker loads Silero VAD
|
||||
(`onnx-community/silero-vad`, 2.2 MB, Cache API like the weights; offline it is
|
||||
used only if already cached). A file pass with `longForm: true` then runs
|
||||
`transcribeLongForm` (`worker/longform/`): VAD speech probabilities over the
|
||||
whole buffer → speech spans (faster-whisper `vad.py` port) → chunks ≤ 30 s cut on
|
||||
speech boundaries → one Whisper decode per chunk (language detected once on the
|
||||
first chunk and pinned for multilingual models) → timestamps mapped back to the
|
||||
file's timeline; progress follows the variable chunks
|
||||
(`TranscribeProgressTracker.forWindows`). Same `partial` /
|
||||
`transcribe-progress` / `result` messages as step 5. `choosePipeline` keeps the
|
||||
chunked path whenever the VAD model isn't loaded, and live passes never take
|
||||
this branch (live keeps its RMS segmenter, D-016/D-024). A finished transcript
|
||||
with no words shows "No speech was found in this recording." The names-and-terms
|
||||
hint (`prompt`) is applied to every chunk's decode the same way (D-WS2a).
|
||||
|
||||
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
|
||||
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
|
||||
@@ -224,7 +243,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
large to precache eagerly.
|
||||
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
|
||||
independent of the SW, which never touches that cache (D-005). `/api/*` is
|
||||
network-only.
|
||||
network-only. The Silero VAD model of the long-form branch lives there too
|
||||
(D-WS3b); offline, the worker checks that cache before loading it and never
|
||||
fetches, falling back to the chunked path if it is missing.
|
||||
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
|
||||
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
|
||||
- Result: with the app installed and at least one model previously loaded, the full
|
||||
|
||||
Reference in new issue
Block a user