Remove the finished server batched-VAD plan; spec 001 and D-WS5a are the record
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
07753a496b
commit
7fb8dde423
@@ -1,102 +0,0 @@
|
||||
# Server engine: batched + VAD faster-whisper Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan
|
||||
> task-by-task (inline, no subagents — coordination plan). Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** file transcriptions on the server engine run through faster-whisper's
|
||||
`BatchedInferencePipeline` with Silero VAD, without breaking segment streaming, cancel, the D-021
|
||||
heartbeat, or the one-loaded-model cache.
|
||||
|
||||
**Architecture:** the sidecar request gains an optional `mode: "file" | "live"` (absent ⇒ `"file"`).
|
||||
`"file"` → `BatchedInferencePipeline(model).transcribe(audio, vad_filter=True, batch_size=8)`;
|
||||
`"live"` → the existing sequential `WhisperModel.transcribe` (short ≤ 29 s passes gain nothing from
|
||||
batching; decided from measurements in Task 1). The pipeline wrapper is cached with the model.
|
||||
Bun side: `SidecarTranscribeRequest.mode`, `createSidecar` forwards it, `live.ts` sends `"live"`.
|
||||
|
||||
**Tech Stack:** Python 3 + faster-whisper 1.2.1 (venv `/home/user/venvs/faster-whisper`), Bun test.
|
||||
|
||||
**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 5; `docs/plans/001-0-coordination.md`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Only `server-py/sidecar.py` is this workstream's core; Bun edits limited to the protocol field.
|
||||
- Heavy jobs (pip, benchmarks, `FW_INTEGRATION=1`) under `flock /tmp/whisper-heavy.lock`.
|
||||
- Decision entry `## D-WS5a — …` at the bottom of `docs/Decisions.md`; cite `D-WS5a` in code.
|
||||
- `requirements.txt` stays `faster-whisper>=1.1,<2` (BatchedInferencePipeline exists since 1.1.0).
|
||||
- No new env vars unless measurement demands one (then README env table).
|
||||
- Measurements: CPU int8, this machine (16 cores, shared), labelled as such.
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. **Cancel during a batch** — batched segments arrive in bursts (one forward per 8 VAD chunks); a
|
||||
cancel must still end the run with `done {cancelled: true}` and no further segments. Test: stub
|
||||
sidecar "slow" model, abort after first segment (Task 2).
|
||||
2. **Long silent stretch before the first segment** (VAD + feature extraction over the whole file
|
||||
happen eagerly in `transcribe()`, then a full batch decodes) — heartbeat lines must keep flowing.
|
||||
Test: injectable heartbeat interval, fake sidecar that waits, assert blank lines (Task 3).
|
||||
3. **Silence-only audio** — VAD finds no speech: must answer `done` with zero segments, not error.
|
||||
Test: stub "silence" model yields nothing → `done` (Task 2); real run on tone in FW_INTEGRATION.
|
||||
4. **Model switch** — still one model resident; the batched wrapper must be rebuilt for the new
|
||||
model, not reused. Test: stub reports load count and which model the pipeline wraps (Task 2).
|
||||
5. **Old Bun ↔ new sidecar** — request without `mode` must behave as `"file"`. Test (Task 2).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Measure (outside the repo)
|
||||
|
||||
**Files:** `/home/user/ws5-bench/bench.py` (not committed).
|
||||
|
||||
- [x] Venv `/home/user/venvs/faster-whisper` with requirements + jiwer (under lock); confirm API.
|
||||
- [x] large-v3-turbo, both clips: `base` (current sidecar), `vad`, `batched` (vad+bs8),
|
||||
`batched_b1` (beam 1). Record wall, WER (repo normalize), first-segment time, max gap.
|
||||
Done 2026-10-29 (`/home/user/ws5-bench/turbo-*.jsonl`); VAD-parameter follow-up in
|
||||
`diag2.out` / `round3-*.jsonl` (threshold 0.2 + pad 800 ms recovers most deletions).
|
||||
- [x] Live-style passes (clip sliced into 6 s pieces): `live_seq` vs `live_batched` vs `live_vad`.
|
||||
- [x] medium.en on en_alien: `base` vs `batched` if time allows.
|
||||
- [x] Decide: beam size, live mode routing.
|
||||
|
||||
### Task 2: Sidecar `mode` routing, tested against a stubbed faster-whisper
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/fixtures/stub_fw/faster_whisper/__init__.py`, `tests/fixtures/stub_fw/numpy/__init__.py`
|
||||
- Modify: `server-py/sidecar.py`, `tests/transcribe.test.ts`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: sidecar stdin `transcribe` message accepts `"mode": "file" | "live"` (default `"file"`).
|
||||
- Stub: `WhisperModel(name, **kw)` counts loads; `.transcribe(audio, **kw)` yields segments whose
|
||||
text reports `seq` + kwargs; `BatchedInferencePipeline(model).transcribe(audio, **kw)` yields
|
||||
`batched model=<name> loads=<n>` + kwargs. Model `"slow"` yields 20 segments 0.1 s apart;
|
||||
`"silence"` yields none. Stub numpy: `float32`, `fromfile(path, dtype)` → bytes.
|
||||
- Test runs `createSidecar(["python3", "-S", REAL_SIDECAR])` with `PYTHONPATH=stub_fw` (createSidecar
|
||||
inherits env; set `process.env.PYTHONPATH` around the test or pass `env` — add optional
|
||||
`env` arg only if needed).
|
||||
|
||||
- [x] Failing tests: default/`file` → batched with `vad_filter=True batch_size=8`; `live` →
|
||||
sequential, no `vad_filter`; model switch reloads and pipeline wraps the new model, same model
|
||||
doesn't reload; `slow` + abort → `cancelled: true`, < 20 segments; `silence` → done, 0 segments.
|
||||
- [x] Implement in `sidecar.py`; update docstring protocol; cite D-WS5a.
|
||||
- [x] Green; commit.
|
||||
|
||||
### Task 3: Bun side — `mode` field, live sends `"live"`, heartbeat test
|
||||
|
||||
**Files:** `src/server/transcribe.ts`, `src/server/live.ts`, `tests/fixtures/fake_sidecar.py`,
|
||||
`tests/transcribe.test.ts`, `tests/server-live.test.ts`
|
||||
|
||||
- [x] Failing tests: fake sidecar "ok" echoes `mode=`; `createSidecar` default sends `file`, explicit
|
||||
`live` forwarded; server-live test asserts the request has `mode: "live"`; HTTP handler passes
|
||||
no mode (⇒ file) — assert `requests[0].mode` undefined or `"file"`.
|
||||
- [x] Heartbeat: `TranscribeDeps.heartbeatIntervalMs?` (default 5000); test with 20 ms and a fake
|
||||
sidecar that waits ~120 ms before the first segment; the response text contains ≥ 2 blank
|
||||
lines before the first segment line.
|
||||
- [x] Implement; typecheck; commit.
|
||||
|
||||
### Task 4: Real-model verification + docs
|
||||
|
||||
- [x] `FW_INTEGRATION=1 SIDECAR_PYTHON=/home/user/venvs/faster-whisper/bin/python bun test
|
||||
tests/transcribe.test.ts` under the lock; extend the integration block with a real speech
|
||||
run (first 60 s of `en_alien`, mode file, asserts ≥ 1 segment and a key word) and the tone
|
||||
(silence-ish) run finishing `done`.
|
||||
- [x] `docs/Decisions.md` D-WS5a with the before/after table; `tests/server-engine.md` updated;
|
||||
`docs/Architecture.md` sidecar protocol line if it describes the message; WorkBreakdown tick.
|
||||
- [x] typecheck, fast suite, build; commit.
|
||||
- [x] `git rm` this plan; commit.
|
||||
Reference in New Issue
Block a user