Enhanced transcription usability.

This commit is contained in:
Lucas Winther committed 2026-07-17 15:18:15 +02:00
1 parent 770cfc9ff7
commit 74858f63fc
8 files changed
+244 -21

No files matched your search

+7 -3
View File
@@ -147,9 +147,13 @@ are the remaining M5 work; see `WorkBreakdown.md`.)
3. The current segment is re-transcribed through the normal inference worker
on a rolling basis (throttled, single-flight, latest-snapshot-wins) as
*tentative* text; a committed segment gets one final pass and freezes.
`requestId` on the protocol messages correlates responses.
4. Stopping flushes the segmenter, drains pending final passes, and yields a
normal transcript (text + absolute-timestamped chunks) in `TranscriptView`.
When inference runs slower than speech, queued committed segments are
batched into merged final passes (one Whisper window each, D-024) so the
backlog stays bounded. `requestId` on the protocol messages correlates
responses.
4. Stopping flushes the segmenter, drains pending final passes (merged, with
a "N remaining segments" countdown in the UI), and yields a normal
transcript (text + absolute-timestamped chunks) in `TranscriptView`.
Everything runs on-device; works offline like the file flow. Server-engine
live input is a recorded future want (WorkBreakdown M7, D-016).
+41
View File
@@ -396,3 +396,44 @@ validation failed. Buffer with '' label is invalid". The worker now sets
fixes its thread pool at first-session init, so the first attempt in a
worker decides; a wasm fallback after a failed accelerator attempt may run
single-threaded until the next worker respawn — correct over fast.
## D-024 — Live mode batches queued final passes into merged passes
**Status:** Accepted (2026-07-17)
Reported symptom: stopping a live session got stuck on "Finishing up…" while
the machine sat at full CPU — indistinguishable from a hang, and the saturated
cores (D-023 uses most of them) made the whole browser feel locked up.
**Cause — a scheduling backlog, not a deadlock.** Whisper pads every inference
pass to a full 30 s encoder window, so a pass costs nearly the same whether it
carries 1 s or 29 s of audio (measured on the browser runtime, ort-web WASM EP
medium q4: ~4 s per pass at 12 threads, ~9–12 s at 4 threads, for 1–9 s
segments alike). Segments therefore commit faster than their final passes
complete whenever the model is slower than speech (medium/large on WASM), the
committed-segment queue grows all session, and `stop()` used to drain it **one
pass per segment** — a 60 s session on 4 threads measured a 52 s drain; longer
sessions scale it into minutes. All the while the UI showed only a static
"Finishing up…".
**Fix (two parts):**
1. `LiveTranscriber.pump()` batches consecutive queued segments into **one
merged final pass**, capped at the segmenter's `maxSegmentSeconds` so a
merged pass stays a single Whisper window. Segment audio is concatenated
(segments end on trailing silence or a soft cut, so splices land in natural
gaps); the result's chunk timestamps are mapped piecewise back to absolute
session time via each segment's own start offset. The backlog — and the
stop drain — is now bounded by ~one pass per 29 s of speech instead of one
pass per segment (the same 60 s/4-thread scenario drains in ~2 passes).
2. `App.tsx` surfaces `LiveUpdate.pendingFinalPasses` while stopping:
"Finishing up… transcribing N remaining segments" counts down instead of
freezing, on both the browser and the server-live (D-019) paths.
**Alternatives rejected:** skipping final passes for segments that already had
tentative text (during a backlog new segments get *no* tentative pass, so
skipping loses their words entirely); a "stop now" escape hatch (same data
loss, silently); merging across the 30 s window (impossible — Whisper's
context is one window).
**Consequence:** a merged pass attributes its whole text to the batch's first
segment (per-segment text attribution is meaningless after joining anyway) and
inter-segment silence is not re-inserted, so chunk timestamps inside a merged
pass are exact per segment region but a chunk spanning a splice maps to its
start's region. Covered by `tests/live.test.ts` (batching + cap) and the WER
gates (`tests/live-accuracy.test.ts`, `tests/live-real-model.test.ts` — the
real-model test's fast feed makes real inference lag, so merging is exercised
there with real weights).