whisper_web/tests/live.md

Manual live-transcription verification (M7, D-016)

The segment loop (src/lib/live.ts) and the protocol changes are covered by unit tests, and transcript accuracy of the pipeline (segmentation, pass scheduling, mic-path resampling) is regression-tested as WER by tests/live-accuracy.test.ts + tests/resampler.test.ts. End-to-end quality with real Whisper weights and real speech is regression-tested too: tests/live-real-model.test.ts runs every recording in tests/audio_files/ through the full mic path with whisper-large-v3-turbo (q4) and holds live WER against the reference transcript (≤ 0.25; measured ≈ 0.11 for da_festuge, ≈ 0.06 for en_aiproblem) and against whole-file mode (it skips itself when the model is neither cached nor downloadable). To cover a new recording, drop <lang>_<name>.<m4a|mp3|wav|…> plus its reference transcript <lang>_<name>.txt into tests/audio_files/ — the test discovers it, decodes it with ffmpeg-static, pins <lang> as the Whisper language, and fails loudly on unpaired or unprefixed files. Microphone capture, the AudioWorklet path, and WebGPU/WASM execution in an actual browser can only be exercised manually. This is that procedure.

Known model behavior (not a pipeline bug): with language auto, Whisper misdetects the Festuge clip as English and returns a rough English translation — in whole-file mode as well as live. Danish (and likely other non-English) users should pick their language explicitly; the worker forwards it and accuracy is then as tested above. Re-run it after any change to src/lib/live.ts, src/lib/mic.ts, src/components/LiveControls.tsx, or the live wiring in src/App.tsx.

Prerequisites

• A machine with a working microphone.
• Chrome/Edge (WebGPU) and ideally also Firefox and Safari — the three differ exactly where mic capture is fragile (context sample rates, AudioWorklet Blob-URL modules; Safari exercises the ScriptProcessorNode fallback if the worklet path fails there).
• getUserMedia requires a secure context; http://localhost qualifies:

  bun dev            # or: bun run build && bun run start
  

1. Permission flow

1. Open the app, pick whisper-tiny.en, click Start live transcription.
2. The browser prompts for the microphone. Deny it: the button must return to idle and the error "Microphone access was denied…" must appear; the app stays usable (file flow unaffected).
3. Click again and allow: button becomes Stop with a pulsing red dot; the browser's mic indicator turns on. If the model wasn't downloaded yet, the download progress panel appears and live text simply starts once it is ready (segment passes queue in the worker).

2. Live transcription behavior

1. Speak a few sentences with natural pauses. Expect:
   • While speaking: dimmed italic tentative text appears within a couple of seconds (tiny.en; slower models lag proportionally) and refreshes.
   • After a ~1 s pause: the sentence commits — the text turns from italic to regular ("committed") and no longer changes.
   • Silence alone (say nothing for 30 s) must produce no text — the energy gate drops silent segments, so no hallucinated "thank you."-style output.
2. DevTools → Network: zero requests while speaking (weights may download at the start; the audio itself never leaves the device).
3. During a live session, ModelPicker / options / dropzone are disabled.
4. Model choice sanity: repeat briefly with a multilingual model + a fixed language and confirm the language/task options are honored.

3. Stop & result

1. Click Stop mid-sentence. The button shows "Finishing…" while the last segment gets its final pass — if several segments are still queued (slow model), the transcript heading counts them down ("Finishing up… transcribing N remaining segments", D-024) — then the normal Transcript panel appears: full text, Copy button, and a Timestamps section whose segment times are absolute (a sentence spoken ~10 s in must not be timestamped 0:00).
2. The browser's mic indicator must turn off immediately on Stop.
3. Start a new live session afterwards, then also transcribe a file — both must still work (session cleanup, request correlation).

4. Quality/latency knobs (only if needed)

Production tuning lives in LIVE_TUNING (src/lib/live.ts) and is shared by src/App.tsx and the accuracy harness (tests/live-accuracy.test.ts), so a knob change is automatically re-validated for WER under bun test. If commits are too eager/lazy in real rooms, tune LIVE_TUNING.segmenter (commitSilenceSeconds 1.2, maxSegmentSeconds 29) or the LiveSegmenterOptions defaults (speechRms gate 0.015, sustainRmsRatio 0.5, softCutWindowSeconds 3, minSpeechSeconds 0.25) and the throttle (tentativeIntervalMs 2000, minTentativeSeconds 1). Record any tuned values in D-016.

5. Offline live transcription

With the PWA installed and a model cached (see tests/offline.md), go offline (airplane mode) and run a live session — it must work identically: capture, worklet (Blob URL, no network), inference, stop/transcript.

Recording the result

When this passes in a real browser, flip the [~] manual-verification item in WorkBreakdown.md M7 to [x] and note browser + version here.

• Verified on: browser/version, date