Lucas Winther
500cf3b36e
startLive()'s browser-engine branch flipped liveStatus to "listening" (the red live-dot, "Listening..." hint) the instant the mic was granted, even if the Whisper pipeline was still loading — the user saw an active-looking session while nothing was being transcribed yet. transcribeLiveSegment() already correctly awaits the model-ready gate before posting a segment, so this was purely a misleading-status bug. Adds a "loading-model" LiveStatus between "starting" and "listening"; startLive() now waits on the same browserModelReady() gate the transcribe path uses before claiming "listening", with race-safe cleanup if the load fails while waiting. Server-engine live is unaffected. Adds Decisions.md D-034; updates tests/live.md's manual verification steps.
107 lines
5.5 KiB
Markdown
107 lines
5.5 KiB
Markdown
# Manual live-transcription verification (M7, D-016)
|
|
|
|
The segment loop (`src/lib/live.ts`) and the protocol changes are covered by
|
|
unit tests, and transcript *accuracy* of the pipeline (segmentation, pass
|
|
scheduling, mic-path resampling) is regression-tested as WER by
|
|
`tests/live-accuracy.test.ts` + `tests/resampler.test.ts`. End-to-end quality
|
|
with real Whisper weights and real speech is regression-tested too:
|
|
`tests/live-real-model.test.ts` runs **every** recording in
|
|
`tests/audio_files/` through the full mic path with `whisper-large-v3-turbo`
|
|
(q4) and holds live WER against the reference transcript (≤ 0.25; measured
|
|
≈ 0.11 for `da_festuge`, ≈ 0.06 for `en_aiproblem`) and against whole-file
|
|
mode (it skips itself when the model is neither cached nor downloadable).
|
|
To cover a new recording, drop `<lang>_<name>.<m4a|mp3|wav|…>` plus its
|
|
reference transcript `<lang>_<name>.txt` into `tests/audio_files/` — the test
|
|
discovers it, decodes it with ffmpeg-static, pins `<lang>` as the Whisper
|
|
language, and fails loudly on unpaired or unprefixed files. Microphone capture,
|
|
the AudioWorklet path, and WebGPU/WASM execution in an actual browser can
|
|
only be exercised manually. This is that procedure.
|
|
|
|
Known model behavior (not a pipeline bug): with language **auto**, Whisper
|
|
misdetects the Festuge clip as English and returns a rough English
|
|
translation — in whole-file mode as well as live. Danish (and likely other
|
|
non-English) users should pick their language explicitly; the worker forwards
|
|
it and accuracy is then as tested above.
|
|
Re-run it after any change to `src/lib/live.ts`, `src/lib/mic.ts`,
|
|
`src/components/LiveControls.tsx`, or the live wiring in `src/App.tsx`.
|
|
|
|
## Prerequisites
|
|
|
|
- A machine with a working microphone.
|
|
- Chrome/Edge (WebGPU) and ideally also Firefox and Safari — the three differ
|
|
exactly where mic capture is fragile (context sample rates, AudioWorklet
|
|
Blob-URL modules; Safari exercises the ScriptProcessorNode fallback if the
|
|
worklet path fails there).
|
|
- `getUserMedia` requires a secure context; `http://localhost` qualifies:
|
|
```sh
|
|
bun dev # or: bun run build && bun run start
|
|
```
|
|
|
|
## 1. Permission flow
|
|
|
|
1. Open the app, pick `whisper-tiny.en`, click **Start live transcription**.
|
|
2. The browser prompts for the microphone. **Deny** it: the button must return
|
|
to idle and the error "Microphone access was denied…" must appear; the app
|
|
stays usable (file flow unaffected).
|
|
3. Click again and **allow**: the browser's mic indicator turns on immediately.
|
|
If the model was already loaded, the button becomes **Stop** with a pulsing
|
|
red dot right away. If not (first use, or a model/device switch), the
|
|
button instead shows **"Loading model…"** (disabled, no live dot, no
|
|
"Listening…" hint) while the download/progress panel appears, and only
|
|
flips to **Stop**/pulsing dot once the pipeline actually reports ready
|
|
(D-034) — it must never claim "Listening" before that.
|
|
|
|
## 2. Live transcription behavior
|
|
|
|
1. Speak a few sentences with natural pauses. Expect:
|
|
- While speaking: dimmed italic *tentative* text appears within a couple of
|
|
seconds (tiny.en; slower models lag proportionally) and refreshes.
|
|
- After a ~1 s pause: the sentence commits — the text turns from italic to
|
|
regular ("committed") and no longer changes.
|
|
- Silence alone (say nothing for 30 s) must produce **no** text — the
|
|
energy gate drops silent segments, so no hallucinated "thank you."-style
|
|
output.
|
|
2. DevTools → Network: **zero** requests while speaking (weights may download
|
|
at the start; the audio itself never leaves the device).
|
|
3. During a live session, ModelPicker / options / dropzone are disabled.
|
|
4. Model choice sanity: repeat briefly with a multilingual model + a fixed
|
|
language and confirm the language/task options are honored.
|
|
|
|
## 3. Stop & result
|
|
|
|
1. Click **Stop** mid-sentence. The button shows "Finishing…" while the last
|
|
segment gets its final pass — if several segments are still queued (slow
|
|
model), the transcript heading counts them down ("Finishing up…
|
|
transcribing N remaining segments", D-024) — then the normal
|
|
**Transcript** panel appears:
|
|
full text, Copy button, and a Timestamps section whose segment times are
|
|
**absolute** (a sentence spoken ~10 s in must not be timestamped 0:00).
|
|
2. The browser's mic indicator must turn off immediately on Stop.
|
|
3. Start a new live session afterwards, then also transcribe a file — both
|
|
must still work (session cleanup, request correlation).
|
|
|
|
## 4. Quality/latency knobs (only if needed)
|
|
|
|
Production tuning lives in `LIVE_TUNING` (`src/lib/live.ts`) and is shared by
|
|
`src/App.tsx` and the accuracy harness (`tests/live-accuracy.test.ts`), so a
|
|
knob change is automatically re-validated for WER under `bun test`. If commits
|
|
are too eager/lazy in real rooms, tune `LIVE_TUNING.segmenter`
|
|
(`commitSilenceSeconds` 1.2, `maxSegmentSeconds` 29) or the
|
|
`LiveSegmenterOptions` defaults (`speechRms` gate 0.015, `sustainRmsRatio`
|
|
0.5, `softCutWindowSeconds` 3, `minSpeechSeconds` 0.25) and the throttle
|
|
(`tentativeIntervalMs` 2000, `minTentativeSeconds` 1). Record any tuned
|
|
values in D-016.
|
|
|
|
## 5. Offline live transcription
|
|
|
|
With the PWA installed and a model cached (see `tests/offline.md`), go
|
|
offline (airplane mode) and run a live session — it must work identically:
|
|
capture, worklet (Blob URL, no network), inference, stop/transcript.
|
|
|
|
## Recording the result
|
|
|
|
When this passes in a real browser, flip the `[~]` manual-verification item
|
|
in `WorkBreakdown.md` M7 to `[x]` and note browser + version here.
|
|
|
|
- [ ] Verified on: _browser/version, date_
|