Improvements and tests for live transcription.
This commit is contained in:
1 parent
e4be47f707
commit
18acead43c
14 files changed
+1138
-45
No files matched your search
+24
-9
@@ -174,19 +174,34 @@ up-to-30 s windows), so "live" is approximated by a **segment loop**:
|
||||
|
||||
- **Capture (main thread):** `getUserMedia` → `AudioContext` at the device's
|
||||
native rate → an `AudioWorklet` chunker posts ~100 ms Float32 frames to the
|
||||
main thread, which resamples them to 16 kHz with the existing pure
|
||||
`resampleLinear` (D-012's documented fallback; adequate for speech).
|
||||
The worklet module is loaded from a **Blob URL** built from an inline code
|
||||
string — no extra served file, no build/service-worker changes, and it keeps
|
||||
working offline. Where `audioWorklet`/Blob modules are unavailable
|
||||
(older Safari), fall back to a `ScriptProcessorNode`. The context is *not*
|
||||
forced to 16 kHz because Firefox cannot connect a `MediaStreamAudioSourceNode`
|
||||
across sample rates.
|
||||
main thread, which resamples them to 16 kHz with the pure, stateful
|
||||
`StreamingResampler` (windowed-sinc low-pass; `resampleLinear` has no
|
||||
anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above
|
||||
8 kHz into the speech band and measurably hurt accuracy — see
|
||||
`tests/resampler.test.ts`). The worklet module is loaded from a **Blob URL**
|
||||
built from an inline code string — no extra served file, no
|
||||
build/service-worker changes, and it keeps working offline. Where
|
||||
`audioWorklet`/Blob modules are unavailable (older Safari), fall back to a
|
||||
`ScriptProcessorNode`. The context is *not* forced to 16 kHz because Firefox
|
||||
cannot connect a `MediaStreamAudioSourceNode` across sample rates.
|
||||
- **Segmentation (pure, unit-tested):** a simple RMS energy gate
|
||||
(`src/lib/live.ts`) splits speech into segments: a segment starts at speech
|
||||
onset (with a short pre-roll), commits after a run of trailing silence or at
|
||||
a max length (~25 s, under Whisper's 30 s window), and pure-silence audio is
|
||||
a max length (under Whisper's 30 s window), and pure-silence audio is
|
||||
dropped entirely — Whisper hallucinates on silence, so never transcribe it.
|
||||
Two accuracy refinements (2026-07-14): the gate has **hysteresis** (once a
|
||||
segment is open, frames stay "speech" down to half the onset threshold, so
|
||||
quiet phonemes don't fragment utterances), and a max-length commit is a
|
||||
**soft cut** — it splits at the last sub-threshold frame in the trailing
|
||||
window instead of mid-word at the hard cap, seeding the next segment with
|
||||
the remainder (a mid-word cut corrupts the boundary word in both segments).
|
||||
Dropped (near-silent) segments also clear any tentative text they produced,
|
||||
so a hallucinated tentative pass can neither linger on screen nor be adopted
|
||||
as the next segment's provisional text. Production tuning lives in
|
||||
`LIVE_TUNING` (`src/lib/live.ts`), shared by `src/App.tsx` and the accuracy
|
||||
harness (`tests/live-accuracy.test.ts` — tone-word oracle engine + WER, so
|
||||
segmentation/scheduling/resampling accuracy is regression-tested under
|
||||
`bun test` without a model).
|
||||
No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
|
||||
- **Inference (existing worker, serialized):** the current segment is
|
||||
re-transcribed through the **existing** inference worker/pipeline on a
|
||||
|
||||
Reference in new issue
Block a user