Improvements and tests for live transcription.

This commit is contained in:
Lucas Winther committed 2026-07-14 14:14:01 +02:00
1 parent e4be47f707
commit 18acead43c
14 files changed
+1138 -45

No files matched your search

+24 -9
View File
@@ -174,19 +174,34 @@ up-to-30 s windows), so "live" is approximated by a **segment loop**:
- **Capture (main thread):** `getUserMedia` → `AudioContext` at the device's
native rate → an `AudioWorklet` chunker posts ~100 ms Float32 frames to the
main thread, which resamples them to 16 kHz with the existing pure
`resampleLinear` (D-012's documented fallback; adequate for speech).
The worklet module is loaded from a **Blob URL** built from an inline code
string — no extra served file, no build/service-worker changes, and it keeps
working offline. Where `audioWorklet`/Blob modules are unavailable
(older Safari), fall back to a `ScriptProcessorNode`. The context is *not*
forced to 16 kHz because Firefox cannot connect a `MediaStreamAudioSourceNode`
across sample rates.
main thread, which resamples them to 16 kHz with the pure, stateful
`StreamingResampler` (windowed-sinc low-pass; `resampleLinear` has no
anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above
8 kHz into the speech band and measurably hurt accuracy — see
`tests/resampler.test.ts`). The worklet module is loaded from a **Blob URL**
built from an inline code string — no extra served file, no
build/service-worker changes, and it keeps working offline. Where
`audioWorklet`/Blob modules are unavailable (older Safari), fall back to a
`ScriptProcessorNode`. The context is *not* forced to 16 kHz because Firefox
cannot connect a `MediaStreamAudioSourceNode` across sample rates.
- **Segmentation (pure, unit-tested):** a simple RMS energy gate
(`src/lib/live.ts`) splits speech into segments: a segment starts at speech
onset (with a short pre-roll), commits after a run of trailing silence or at
a max length (~25 s, under Whisper's 30 s window), and pure-silence audio is
a max length (under Whisper's 30 s window), and pure-silence audio is
dropped entirely — Whisper hallucinates on silence, so never transcribe it.
Two accuracy refinements (2026-07-14): the gate has **hysteresis** (once a
segment is open, frames stay "speech" down to half the onset threshold, so
quiet phonemes don't fragment utterances), and a max-length commit is a
**soft cut** — it splits at the last sub-threshold frame in the trailing
window instead of mid-word at the hard cap, seeding the next segment with
the remainder (a mid-word cut corrupts the boundary word in both segments).
Dropped (near-silent) segments also clear any tentative text they produced,
so a hallucinated tentative pass can neither linger on screen nor be adopted
as the next segment's provisional text. Production tuning lives in
`LIVE_TUNING` (`src/lib/live.ts`), shared by `src/App.tsx` and the accuracy
harness (`tests/live-accuracy.test.ts` — tone-word oracle engine + WER, so
segmentation/scheduling/resampling accuracy is regression-tested under
`bun test` without a model).
No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
- **Inference (existing worker, serialized):** the current segment is
re-transcribed through the **existing** inference worker/pipeline on a