whisper_web/tests/live.md

Manual live-transcription verification (M7, D-016)

The segment loop (src/lib/live.ts) and the protocol changes are covered by unit tests, and transcript accuracy of the pipeline (segmentation, pass scheduling, mic-path resampling) is regression-tested as WER by tests/live-accuracy.test.ts + tests/resampler.test.ts. End-to-end quality with real Whisper weights and real speech is regression-tested too: tests/live-real-model.test.ts runs every recording in tests/audio_files/ through the full mic path with whisper-large-v3-turbo (q4) and holds live WER against the reference transcript (≤ 0.25; measured ≈ 0.11 for da_festuge, ≈ 0.06 for en_aiproblem; the noisy 15-min en_alien stream has its own measured bar of 0.35 in MAX_WER_BY_CLIP, D-039) and against whole-file mode (it skips itself when the model is neither cached nor downloadable). Each clip's time budget scales with its length, so en_alien alone takes ~20 min on CPU. To cover a new recording, drop <lang>_<name>.<m4a|mp3|wav|…> plus its reference transcript <lang>_<name>.txt into tests/audio_files/ — the test discovers it, decodes it with ffmpeg-static, pins <lang> as the Whisper language, and fails loudly on unpaired or unprefixed files. Microphone capture, the AudioWorklet path, and WebGPU/WASM execution in an actual browser can only be exercised manually. This is that procedure.

Known model behavior (not a pipeline bug): with language auto, Whisper misdetects the Festuge clip as English and returns a rough English translation — in whole-file mode as well as live. Danish (and likely other non-English) users should pick their language explicitly; the worker forwards it and accuracy is then as tested above. Re-run it after any change to src/lib/live.ts, src/lib/mic.ts, src/components/LiveControls.tsx, or the live wiring in src/App.tsx.

Prerequisites

• A machine with a working microphone.
• Chrome/Edge (WebGPU) and ideally also Firefox and Safari — the three differ exactly where mic capture is fragile (context sample rates, AudioWorklet Blob-URL modules; Safari exercises the ScriptProcessorNode fallback if the worklet path fails there).
• getUserMedia requires a secure context; http://localhost qualifies:

  bun dev            # or: bun run build && bun run start
  

1. Permission flow

1. Open the app, pick whisper-tiny.en, click Start live transcription.
2. The browser prompts for the microphone. Deny it: the button must return to idle and the error "Microphone access was denied…" must appear; the app stays usable (file flow unaffected).
3. Click again and allow: the browser's mic indicator turns on immediately. If the model was already loaded, the button becomes Stop with a pulsing red dot right away. If not (first use, or a model/device switch), the button instead shows "Loading model…" (disabled, no live dot, no "Listening…" hint) while the download/progress panel appears, and only flips to Stop/pulsing dot once the pipeline actually reports ready (D-034) — it must never claim "Listening" before that.

2. Live transcription behavior

1. Speak a few sentences with natural pauses. Expect:
   • While speaking: dimmed italic tentative text appears within a couple of seconds (tiny.en; slower models lag proportionally) and refreshes.
   • After a ~1 s pause: the sentence commits — the text turns from italic to regular ("committed") and no longer changes.
   • Silence alone (say nothing for 30 s) must produce no text — the energy gate drops silent segments, so no hallucinated "thank you."-style output.
2. DevTools → Network: zero requests while speaking (weights may download at the start; the audio itself never leaves the device).
3. During a live session, ModelPicker / options / dropzone are disabled.
4. Model choice sanity: repeat briefly with a multilingual model + a fixed language and confirm the language/task options are honored.

3. Stop & result

1. Click Stop mid-sentence. The button shows "Finishing…" while the last segment gets its final pass — if several segments are still queued (slow model), the transcript heading counts them down ("Finishing up… transcribing N remaining segments", D-024) — then the normal Transcript panel appears: full text, Copy button, and a Timestamps section whose segment times are absolute (a sentence spoken ~10 s in must not be timestamped 0:00).
2. The browser's mic indicator must turn off immediately on Stop.
3. Start a new live session afterwards, then also transcribe a file — both must still work (session cleanup, request correlation).

4. Quality/latency knobs (only if needed)

Production tuning lives in LIVE_TUNING (src/lib/live.ts) and is shared by src/App.tsx and the accuracy harness (tests/live-accuracy.test.ts), so a knob change is automatically re-validated for WER under bun test. If commits are too eager/lazy in real rooms, tune LIVE_TUNING.segmenter (commitSilenceSeconds 1.2, maxSegmentSeconds 29) or the LiveSegmenterOptions defaults (speechRms gate 0.015, sustainRmsRatio 0.5, softCutWindowSeconds 3, minSpeechSeconds 0.25) and the throttle (tentativeIntervalMs 2000, minTentativeSeconds 1). Record any tuned values in D-016.

5. Offline live transcription

With the PWA installed and a model cached (see tests/offline.md), go offline (airplane mode) and run a live session — it must work identically: capture, worklet (Blob URL, no network), inference, stop/transcript.

Recording the result

When this passes in a real browser, flip the [~] manual-verification item in WorkBreakdown.md M7 to [x] and note browser + version here.

• Verified on: browser/version, date