Merge feat/longform-pipeline: VAD long-form pipeline, opt-in (spec 001 workstream 3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	docs/Architecture.md
#	docs/Decisions.md
#	src/App.tsx
#	src/index.css
#	src/lib/engine.ts
#	src/worker/inference.worker.ts
#	src/worker/protocol.ts
#	tests/engine.test.ts
#	tests/live-real-model.test.ts
#	tests/offline.md
#	tests/protocol.test.ts
This commit is contained in:
Lucas Winther committed 2026-10-30 01:39:17 +01:00
commit 9c89ab047f
31 files changed
+2264 -16

No files matched your search

+84
View File
@@ -1619,3 +1619,87 @@ itself and accepts only text, so the sidecar passes it the budget-cut ids
decoded back to text (re-encoding gives the same ids, checked with the real
tokenizer incl. "Århus, Ærø"); server-live passes keep the ids. Pinned by the
stubbed-sidecar test "the vocabulary hint reaches the batched pipeline as text".
## D-WS3a — Long-form file pipeline (Silero VAD chunks, language pinned once) ships as an Advanced option, off by default
**Status:** Accepted (2026-10-30) — spec 001 Goal 3
Browser file transcription cuts audio into fixed 30 s windows with 5 s
overlap (Transformers.js's chunked pipeline), decoding silence and
overlaps, and with "Auto-detect" Transformers.js 4.2 silently decodes as
English.
**Decision:** `src/worker/longform/` adds `transcribeLongForm(audio, deps,
options)`: Silero VAD speech probabilities over the whole buffer → speech
spans (line-for-line port of faster-whisper 1.2.1 `get_speech_timestamps`,
parity-checked against the Python on 20 000 random windows) → chunks
≤ 30 s cut on speech boundaries (`collect_chunks`) → one Whisper decode per
chunk → timestamps mapped back to the file's timeline. For a multilingual
model without a language, it detects once on the first speech chunk (one
decoder step on that chunk's encoder output, which the decode then reuses)
and pins it for every chunk. Defaults: VAD `threshold 0.5`,
`min_silence 2000 ms`, `speech_pad 400 ms`, `max_speech 30 s`; repetition
guard **off** (`no_repeat_ngram_size 0`, `repetition_penalty 1`);
temperature fallback (compression ratio > 2.4 / avg log-prob < −1.0 →
0.2…1.0) implemented but **off**. Inference is injected (`LongFormDeps`),
so the 90-minute case is unit-tested: VAD gets the caller's buffer, decodes
get ≤ 30 s copies. Audio with no speech returns an empty transcript without
a decode; the UI shows "No speech was found in this recording."
**Measurements** (onnxruntime-node CPU, Large v3 Turbo q4 = the browser's
WASM weights; machine shared with six other workstreams, old/new run
back-to-back per clip; WER vs `tests/audio_files/*.txt`):
| clip | old s | long-form s | old WER | long-form WER | old / lf tokens |
|---|---|---|---|---|---|
| en_aiproblem (14 s) | 3.6 | 3.3 | 0.000 | 0.000 | 47 / 47 |
| da_festuge, `da` (21 s) | 3.4 · 6.3 | 3.5 · 6.2 | 0.141 | 0.141 | 122 / 121 |
| da_festuge, auto | 6.7 | 4.7 | 0.938 (English) | 0.141 (`da` detected) | 88 / 121 |
| en_alien (15 min) | 210.2 | 115.0 | 0.255 | 0.140 | 3933 / 2258 |
| meeting (**synthetic**, 2 min 45 s) | 41.9 | 37.5 | 0.0317 | 0.0422 | 776 / 594 |
(da_festuge with `da` was measured twice, in two load conditions. The
meeting is `tests/meetingFixture.ts`: en_aiproblem and three en_alien
excerpts joined by 3–8 s pauses of −60 dBFS noise — **synthetic**. Its
long-form errors are mostly spelling of casual speech: "gonna" for
"going to", "killin", "livestreamed".)
**Rollout (spec rule: default only if ≤ old WER + 0.01 and faster on every
clip):** not met. The synthetic meeting is +0.0106 WER (just over the
tolerance) and da_festuge with an explicit language is a tie within
noise (3.5 vs 3.4 s, then 6.2 vs 6.3 s). So the old chunked path stays
the default and long-form ships as **Advanced → "Skip silence in uploaded
files"**, off by default, remembered in localStorage (`whisper-long-form`).
It is the only path whose "Auto-detect" actually detects, and roughly
halves time and WER on the long, pause-heavy en_alien.
The repetition guard and fallback variants were not measured (heavy runs
kept to one baseline + one long-form run per clip); neither is on, so no
unmeasured setting ships as a default.
**Consequence:** live mode is untouched (its RMS segmenter, D-016/D-024).
`tests/live-real-model.test.ts` now asserts whole-file WER of the chunked
path and of long-form (explicit and auto language) against the per-clip
bars. Whisper's `generate()` drops kwargs not in `forward_params`, so
`createWhisperDeps` adds `encoder_outputs` to that list on its pipeline
instance; a Transformers.js upgrade must keep the "one encoder pass per
chunk" assertion green. WebGPU timings are unmeasured.
**Integration note (2026-10-30, merge with D-WS2a):** the long-form adapter
(`worker/longform/transformers.ts`) applies the names-and-terms hint to every
chunk's decode with the same decoder-prompt prefix as the chunked path
(tokenized once per pass); before the merge the `prompt` option reached the
adapter but was ignored. Pinned by a unit test in
`tests/longform-transformers.test.ts`.
## D-WS3b — Silero VAD model: onnx-community/silero-vad from the Hub, cached like Whisper weights, never fetched offline
**Status:** Accepted (2026-10-30) — spec 001 Goal 3
**Decision:** the long-form pipeline's VAD is Silero VAD v5 as exported by
`onnx-community/silero-vad` (`onnx/model.onnx`, fp32, 2 243 022 bytes, MIT;
commit `e71cae9` on 2026-10-30). It loads through Transformers.js's
`AutoModel` as a bare ONNX graph (`model_type: "custom"`) on the same
onnxruntime as Whisper (WASM device — a 2 MB model gains nothing from a
GPU), fetched from the Hub like the weights (CORS, so COEP allows it) and
stored in the same Cache API store (D-005). Windows of 512 samples with 64
samples of context and the `[2, 1, 128]` state, as in Silero's own
wrapper. It is loaded only with "Skip silence" on, after the Whisper
pipeline is ready, on the same user-initiated load (D-020). Offline
(`navigator.onLine === false`) it loads only if already cached and never
fetches; missing or failed, the worker logs it and files take the chunked
path (`choosePipeline`). Live mode keeps its energy gate (D-016).
**Rejected:** bundling the model in `dist/` (2 MB on every install for an
off-by-default option); running it on the main thread (D-013).
**Consequence:** the first "Skip silence" run online downloads 2.2 MB.