Enhanced transcription usability.
This commit is contained in:
1 parent
770cfc9ff7
commit
74858f63fc
8 files changed
+244
-21
No files matched your search
@@ -147,9 +147,13 @@ are the remaining M5 work; see `WorkBreakdown.md`.)
|
||||
3. The current segment is re-transcribed through the normal inference worker
|
||||
on a rolling basis (throttled, single-flight, latest-snapshot-wins) as
|
||||
*tentative* text; a committed segment gets one final pass and freezes.
|
||||
`requestId` on the protocol messages correlates responses.
|
||||
4. Stopping flushes the segmenter, drains pending final passes, and yields a
|
||||
normal transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
||||
When inference runs slower than speech, queued committed segments are
|
||||
batched into merged final passes (one Whisper window each, D-024) so the
|
||||
backlog stays bounded. `requestId` on the protocol messages correlates
|
||||
responses.
|
||||
4. Stopping flushes the segmenter, drains pending final passes (merged, with
|
||||
a "N remaining segments" countdown in the UI), and yields a normal
|
||||
transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
||||
Everything runs on-device; works offline like the file flow. Server-engine
|
||||
live input is a recorded future want (WorkBreakdown M7, D-016).
|
||||
|
||||
|
||||
@@ -396,3 +396,44 @@ validation failed. Buffer with '' label is invalid". The worker now sets
|
||||
fixes its thread pool at first-session init, so the first attempt in a
|
||||
worker decides; a wasm fallback after a failed accelerator attempt may run
|
||||
single-threaded until the next worker respawn — correct over fast.
|
||||
|
||||
## D-024 — Live mode batches queued final passes into merged passes
|
||||
**Status:** Accepted (2026-07-17)
|
||||
Reported symptom: stopping a live session got stuck on "Finishing up…" while
|
||||
the machine sat at full CPU — indistinguishable from a hang, and the saturated
|
||||
cores (D-023 uses most of them) made the whole browser feel locked up.
|
||||
**Cause — a scheduling backlog, not a deadlock.** Whisper pads every inference
|
||||
pass to a full 30 s encoder window, so a pass costs nearly the same whether it
|
||||
carries 1 s or 29 s of audio (measured on the browser runtime, ort-web WASM EP
|
||||
medium q4: ~4 s per pass at 12 threads, ~9–12 s at 4 threads, for 1–9 s
|
||||
segments alike). Segments therefore commit faster than their final passes
|
||||
complete whenever the model is slower than speech (medium/large on WASM), the
|
||||
committed-segment queue grows all session, and `stop()` used to drain it **one
|
||||
pass per segment** — a 60 s session on 4 threads measured a 52 s drain; longer
|
||||
sessions scale it into minutes. All the while the UI showed only a static
|
||||
"Finishing up…".
|
||||
**Fix (two parts):**
|
||||
1. `LiveTranscriber.pump()` batches consecutive queued segments into **one
|
||||
merged final pass**, capped at the segmenter's `maxSegmentSeconds` so a
|
||||
merged pass stays a single Whisper window. Segment audio is concatenated
|
||||
(segments end on trailing silence or a soft cut, so splices land in natural
|
||||
gaps); the result's chunk timestamps are mapped piecewise back to absolute
|
||||
session time via each segment's own start offset. The backlog — and the
|
||||
stop drain — is now bounded by ~one pass per 29 s of speech instead of one
|
||||
pass per segment (the same 60 s/4-thread scenario drains in ~2 passes).
|
||||
2. `App.tsx` surfaces `LiveUpdate.pendingFinalPasses` while stopping:
|
||||
"Finishing up… transcribing N remaining segments" counts down instead of
|
||||
freezing, on both the browser and the server-live (D-019) paths.
|
||||
**Alternatives rejected:** skipping final passes for segments that already had
|
||||
tentative text (during a backlog new segments get *no* tentative pass, so
|
||||
skipping loses their words entirely); a "stop now" escape hatch (same data
|
||||
loss, silently); merging across the 30 s window (impossible — Whisper's
|
||||
context is one window).
|
||||
**Consequence:** a merged pass attributes its whole text to the batch's first
|
||||
segment (per-segment text attribution is meaningless after joining anyway) and
|
||||
inter-segment silence is not re-inserted, so chunk timestamps inside a merged
|
||||
pass are exact per segment region but a chunk spanning a splice maps to its
|
||||
start's region. Covered by `tests/live.test.ts` (batching + cap) and the WER
|
||||
gates (`tests/live-accuracy.test.ts`, `tests/live-real-model.test.ts` — the
|
||||
real-model test's fast feed makes real inference lag, so merging is exercised
|
||||
there with real weights).
|
||||
Reference in new issue
Block a user