diff --git a/docs/Decisions.md b/docs/Decisions.md index 5dc8f08..4ae42f8 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1224,3 +1224,80 @@ whole-file output is equally bad, which is exactly what the absolute bar exists to catch. **Consequence:** full `bun test` ≈ 19 min on CPU (en_alien ≈ 20 min of it on a cold run); the fast suite excludes this file. + +## D-WS3a — Long-form file pipeline (Silero VAD chunks, language pinned once) ships as an Advanced option, off by default +**Status:** Accepted (2026-10-30) — spec 001 Goal 3 +Browser file transcription cuts audio into fixed 30 s windows with 5 s +overlap (Transformers.js's chunked pipeline), decoding silence and +overlaps, and with "Auto-detect" Transformers.js 4.2 silently decodes as +English. +**Decision:** `src/worker/longform/` adds `transcribeLongForm(audio, deps, +options)`: Silero VAD speech probabilities over the whole buffer → speech +spans (line-for-line port of faster-whisper 1.2.1 `get_speech_timestamps`, +parity-checked against the Python on 20 000 random windows) → chunks +≤ 30 s cut on speech boundaries (`collect_chunks`) → one Whisper decode per +chunk → timestamps mapped back to the file's timeline. For a multilingual +model without a language, it detects once on the first speech chunk (one +decoder step on that chunk's encoder output, which the decode then reuses) +and pins it for every chunk. Defaults: VAD `threshold 0.5`, +`min_silence 2000 ms`, `speech_pad 400 ms`, `max_speech 30 s`; repetition +guard **off** (`no_repeat_ngram_size 0`, `repetition_penalty 1`); +temperature fallback (compression ratio > 2.4 / avg log-prob < −1.0 → +0.2…1.0) implemented but **off**. Inference is injected (`LongFormDeps`), +so the 90-minute case is unit-tested: VAD gets the caller's buffer, decodes +get ≤ 30 s copies. Audio with no speech returns an empty transcript without +a decode; the UI shows "No speech was found in this recording." +**Measurements** (onnxruntime-node CPU, Large v3 Turbo q4 = the browser's +WASM weights; machine shared with six other workstreams, old/new run +back-to-back per clip; WER vs `tests/audio_files/*.txt`): + +| clip | old s | long-form s | old WER | long-form WER | old / lf tokens | +|---|---|---|---|---|---| +| en_aiproblem (14 s) | 3.6 | 3.3 | 0.000 | 0.000 | 47 / 47 | +| da_festuge, `da` (21 s) | 3.4 · 6.3 | 3.5 · 6.2 | 0.141 | 0.141 | 122 / 121 | +| da_festuge, auto | 6.7 | 4.7 | 0.938 (English) | 0.141 (`da` detected) | 88 / 121 | +| en_alien (15 min) | 210.2 | 115.0 | 0.255 | 0.140 | 3933 / 2258 | +| meeting (**synthetic**, 2 min 45 s) | 41.9 | 37.5 | 0.0317 | 0.0422 | 776 / 594 | + +(da_festuge with `da` was measured twice, in two load conditions. The +meeting is `tests/meetingFixture.ts`: en_aiproblem and three en_alien +excerpts joined by 3–8 s pauses of −60 dBFS noise — **synthetic**. Its +long-form errors are mostly spelling of casual speech: "gonna" for +"going to", "killin", "livestreamed".) +**Rollout (spec rule: default only if ≤ old WER + 0.01 and faster on every +clip):** not met. The synthetic meeting is +0.0106 WER (just over the +tolerance) and da_festuge with an explicit language is a tie within +noise (3.5 vs 3.4 s, then 6.2 vs 6.3 s). So the old chunked path stays +the default and long-form ships as **Advanced → "Skip silence in uploaded +files"**, off by default, remembered in localStorage (`whisper-long-form`). +It is the only path whose "Auto-detect" actually detects, and roughly +halves time and WER on the long, pause-heavy en_alien. +The repetition guard and fallback variants were not measured (heavy runs +kept to one baseline + one long-form run per clip); neither is on, so no +unmeasured setting ships as a default. +**Consequence:** live mode is untouched (its RMS segmenter, D-016/D-024). +`tests/live-real-model.test.ts` now asserts whole-file WER of the chunked +path and of long-form (explicit and auto language) against the per-clip +bars. Whisper's `generate()` drops kwargs not in `forward_params`, so +`createWhisperDeps` adds `encoder_outputs` to that list on its pipeline +instance; a Transformers.js upgrade must keep the "one encoder pass per +chunk" assertion green. WebGPU timings are unmeasured. + +## D-WS3b — Silero VAD model: onnx-community/silero-vad from the Hub, cached like Whisper weights, never fetched offline +**Status:** Accepted (2026-10-30) — spec 001 Goal 3 +**Decision:** the long-form pipeline's VAD is Silero VAD v5 as exported by +`onnx-community/silero-vad` (`onnx/model.onnx`, fp32, 2 243 022 bytes, MIT; +commit `e71cae9` on 2026-10-30). It loads through Transformers.js's +`AutoModel` as a bare ONNX graph (`model_type: "custom"`) on the same +onnxruntime as Whisper (WASM device — a 2 MB model gains nothing from a +GPU), fetched from the Hub like the weights (CORS, so COEP allows it) and +stored in the same Cache API store (D-005). Windows of 512 samples with 64 +samples of context and the `[2, 1, 128]` state, as in Silero's own +wrapper. It is loaded only with "Skip silence" on, after the Whisper +pipeline is ready, on the same user-initiated load (D-020). Offline +(`navigator.onLine === false`) it loads only if already cached and never +fetches; missing or failed, the worker logs it and files take the chunked +path (`choosePipeline`). Live mode keeps its energy gate (D-016). +**Rejected:** bundling the model in `dist/` (2 MB on every install for an +off-by-default option); running it on the main thread (D-013). +**Consequence:** the first "Skip silence" run online downloads 2.2 MB. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 336b0b4..176bbe7 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -372,6 +372,16 @@ Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29. — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the Backend select overflowed its panel on 390 px-wide phones. +Found while building the long-form pipeline (spec 001 Goal 3, D-WS3a), 2026-10-29: +- [ ] "Auto-detect" on the default (chunked) file path does not detect: with no + `language`, Transformers.js 4.2 logs "No language specified - defaulting to + English" and decodes Danish audio as an English translation (da_festuge WER + 0.94 vs 0.14 with `da`). Long-form ("Skip silence") detects once and pins + the language; the chunked path is unchanged. +- [ ] Spec 001's measurement fallback mentions a hint list of the names in + `en_alien`'s reference; `tests/meetingFixture.ts` returns audio + reference + only. Add the list when the vocabulary hint (workstream 2) lands. + ## Recurring - [ ] Maintain `AGENTS.md`: - **Add a rule only in response to a real failure.** When an agent repeatedly diff --git a/docs/plans/001-3-longform-pipeline.md b/docs/plans/001-3-longform-pipeline.md index 094e896..505b13d 100644 --- a/docs/plans/001-3-longform-pipeline.md +++ b/docs/plans/001-3-longform-pipeline.md @@ -204,7 +204,7 @@ normalize), `scripts/measure-longform.ts`. - [x] `scripts/measure-longform.ts [--clips …] [--config name]` prints per clip: wall-clock, WER, generated tokens, decode calls — for `old` (worker's current options) and long-form configs. Clips: en_alien, da_festuge with `da`, da_festuge auto, en_aiproblem, meeting (synthetic). -- [ ] Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants +- [x] (one baseline + one long-form run per clip; guard/fallback variants not run, both stay off — D-WS3a) Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants (`no_repeat_ngram_size 3/5`, `repetition_penalty 1.1`); fallback on if loops remain; VAD `min_silence 2000` vs `160`. Choose defaults by the rollout rule; record the table in D-WS3a. - [x] Commit "Add the synthetic meeting fixture and long-form measurement script". @@ -239,8 +239,8 @@ empty finished transcript. ### Task 9: Docs and finish -- [ ] `docs/Decisions.md`: append `D-WS3a` (pipeline, measurement table labelled synthetic where +- [x] `docs/Decisions.md`: append `D-WS3a` (pipeline, measurement table labelled synthetic where so, rollout decision) and `D-WS3b` (VAD model, loading, offline, live kept on the energy gate). -- [ ] `docs/Architecture.md` §3 layout + §4.1 flow (long-form branch) + §4.2 offline note; +- [x] `docs/Architecture.md` §3 layout + §4.1 flow (long-form branch) + §4.2 offline note; `docs/WorkBreakdown.md` ticks + "Noticed, not fixed"; `tests/offline.md` manual VAD step. - [ ] typecheck, fast suite, build; real-model test under the lock; `git rm` this plan; commit.