diff --git a/docs/Architecture.md b/docs/Architecture.md index 6afa86e..aa98f13 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -230,7 +230,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.) (`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no browser/WASM build, so it is server-only — the browser stays on Transformers.js). `src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed - JSON-lines HTTP response (D-011). The shared catalog maps model ids to + JSON-lines HTTP response (D-011). File requests (`mode: "file"`) decode with + faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live + passes send `mode: "live"` and stay sequential (D-WS5a). The shared catalog maps model ids to `Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent, the server engine reports unavailable; the browser/anonymous flow is unaffected. 4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the diff --git a/docs/Decisions.md b/docs/Decisions.md index a413986..01acac5 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1483,3 +1483,48 @@ use beam 5. **Pending the owner:** the Hugging Face account name, then upload the six prepared folders (model cards credit the original authors and carry MIT, or Apache-2.0 + NOTICE), then set `DANISH_MODELS_HF_OWNER`. The integrator sets `canTranslate: false` on these entries at merge. + +## D-WS5a — Server engine: batched faster-whisper + tuned Silero VAD for files; live passes stay sequential +**Status:** Accepted (2026-10-30, spec 001 Goal 5) +`server-py/sidecar.py` decoded every request with the sequential +`WhisperModel.transcribe`, no VAD. +**Decision:** the sidecar request gains `mode: "file" | "live"` (absent ⇒ +`"file"`, so an older Bun still gets the new path). `"file"` (`POST +/api/transcribe`) runs `BatchedInferencePipeline(model).transcribe(audio, +vad_filter=True, vad_parameters={threshold: 0.2, speech_pad_ms: 800}, +batch_size=8)`; the pipeline is a weightless wrapper cached and rebuilt with +the one resident model. `"live"` (server-live passes, `src/server/live.ts`) +keeps the sequential decoder. Beam size stays at the default 5. +**Measurements** (CPU int8, AMD Ryzen 9 9950X, 16 cores shared with other +jobs — load average 7–24 during runs, so wall-clock is ±20 %; WER via jiwer +on the `tests/live-real-model.test.ts` normalisation vs `tests/audio_files/*.txt`): + +| model | audio | config | wall-clock | WER | +|---|---|---|---|---| +| large-v3-turbo | en_alien (888 s) | sequential (before) | 97 s / 130 s (two runs) | 0.097 / 0.067 | +| large-v3-turbo | en_alien | sequential + VAD | 79 s | 0.136 | +| large-v3-turbo | en_alien | batched, default VAD (0.5 / 400 ms) | 47 s / 59 s | 0.117 | +| large-v3-turbo | en_alien | batched, beam 1 | 44 s | 0.114 | +| large-v3-turbo | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **51 s / 58 s** | **0.086** | +| large-v3-turbo | en_alien | batched, VAD 0.2 / 800 ms, word-level timestamps | 56 s | 0.144 | +| large-v3-turbo | da_festuge (21.5 s, da) | sequential (before) | 2.9 s | 0.094 | +| large-v3-turbo | da_festuge | batched, default VAD | 2.9 s | 0.141 | +| large-v3-turbo | da_festuge | **batched, VAD 0.2 / 800 ms (after)** | **2.7 s** | **0.109** | +| medium.en | en_alien | sequential (before) | 114 s | 0.126 | +| medium.en | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **67 s** | **0.111** | + +The sequential decoder is not deterministic run to run on en_alien (temperature +fallback; 0.067–0.097 over three runs), while the batched result repeated +exactly. Default-VAD errors were deletions (121 vs 64 sequential): Silero at +0.5 dropped quiet speech under music; 0.2 / 800 ms recovered most of them. +da_festuge's gap is one word of 64. Live-style 6 s passes over the first +180 s of en_alien: sequential 2.45 s/pass, batched 2.12, sequential + VAD +1.88 — within the load noise, each losing a few words, and real live +passes are already speech-gated client-side (D-016), so live stays as it was. +**Consequences:** file transcriptions are ≈ 1.7–2.5× faster on CPU at +equal-or-better WER on en_alien (one word worse on the 21 s Danish clip). Segments are coarser — one per VAD chunk (≤ 30 s) instead of one per +sentence — because word-level timestamps (`without_timestamps=False`) cost +WER here (0.144); SRT/VTT cues from the server engine are correspondingly +long. Segments arrive in bursts of 8 chunks (≈ 4 min of speech) and the first +burst can take ~15–25 s on a long file; the D-021 heartbeat covers it, and +cancel lands at the next burst boundary. No new env vars. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 1b59cbc..67fb229 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -153,6 +153,9 @@ Notes (M4): (CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared model catalog to faster-whisper model ids in `src/lib/models.ts` +- [x] Batched inference + tuned Silero VAD for server-engine file + transcriptions, live passes sequential (spec 001 Goal 5, D-WS5a): + ≈ 2× faster on CPU int8 at equal-or-better WER; segments ≤ 30 s chunks - [x] Degrade gracefully when Python/the sidecar is absent (server engine unavailable; browser engine untouched — anonymous path deploys with Bun alone) - [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011), diff --git a/server-py/sidecar.py b/server-py/sidecar.py index 95f8d3b..125d337 100644 --- a/server-py/sidecar.py +++ b/server-py/sidecar.py @@ -9,7 +9,8 @@ stdio — one JSON object per line, no framing beyond that: stdin -> {"type": "transcribe", "id": "1", "model": "tiny.en", "audio_path": "/tmp/...", "language": "en" | null, - "task": "transcribe" | "translate"} + "task": "transcribe" | "translate", + "mode": "file" | "live"} # optional, default "file" stdout <- {"type": "segment", "id": "1", "start": 0.0, "end": 4.2, "text": " Hello world."} # 0..n of these stdout <- {"type": "done", "id": "1", "cancelled": false} @@ -24,6 +25,18 @@ container decoding happens here. Requests are processed strictly one at a time; a reader thread keeps consuming stdin meanwhile so `cancel` can interrupt the segment loop of the request in flight. +Decoding strategy by mode (Decisions.md D-WS5a): + "file" faster-whisper's BatchedInferencePipeline with its Silero VAD + (vad_filter=True, VAD_PARAMETERS): silence is dropped, speech is cut + into <= 30 s chunks and BATCH_SIZE chunks decode per forward pass. + Without word-level timestamps each chunk is one segment (<= 30 s). + Segments come out in bursts (one burst per batch), so a cancel + takes effect at the next burst boundary; the Bun side's heartbeat + (D-021) covers the longer quiet stretch before the first burst. + "live" the plain sequential WhisperModel.transcribe: server-live passes + are short (<= 29 s, often a few seconds), so batching has nothing to + batch and VAD only adds latency and can clip a short utterance. + The last-used model stays loaded between requests (WorkBreakdown M5 warm-up: never reload weights per request) and is replaced — not accumulated — when a request names a different model, bounding memory to one model. @@ -47,6 +60,13 @@ import sys import threading SAMPLE_RATE = 16000 +BATCH_SIZE = 8 # D-WS5a: VAD chunks decoded per batched forward pass +# D-WS5a: Silero VAD tuned for file mode. faster-whisper's batched default +# (threshold 0.5, 400 ms padding) dropped quiet and overlapped speech — +# deletions nearly doubled on en_alien; 0.2 / 800 ms brought WER back to the +# sequential decoder's level. min_silence_duration_ms stays at VadOptions' +# 2000 ms default (a dict replaces the pipeline's 160 ms preset). +VAD_PARAMETERS = {"threshold": 0.2, "speech_pad_ms": 800} def emit(message): @@ -101,6 +121,7 @@ def main(): cached_name = None cached_model = None + cached_pipeline = None # BatchedInferencePipeline wrapping cached_model emit({"type": "ready", "version": getattr(faster_whisper, "__version__", "?")}) @@ -115,26 +136,42 @@ def main(): audio_path = request["audio_path"] language = request.get("language") or None task = request.get("task") or "transcribe" + mode = request.get("mode") or "file" if model_name != cached_name: - cached_model = None # release before loading the replacement + # Release both before loading the replacement: one model resident. + cached_model = cached_pipeline = None + cached_name = None cached_model = faster_whisper.WhisperModel( model_name, device=device, compute_type=compute_type, download_root=download_root, ) + # A thin wrapper (no weights of its own) — shares cached_model. + cached_pipeline = faster_whisper.BatchedInferencePipeline(cached_model) cached_name = model_name audio = np.fromfile(audio_path, dtype=np.float32) - segments, _info = cached_model.transcribe( - audio, language=language, task=task - ) + if mode == "live": + segments, _info = cached_model.transcribe( + audio, language=language, task=task + ) + else: + segments, _info = cached_pipeline.transcribe( + audio, + language=language, + task=task, + vad_filter=True, + vad_parameters=dict(VAD_PARAMETERS), # fresh: faster-whisper may mutate it + batch_size=BATCH_SIZE, + ) was_cancelled = False # `segments` is a generator: inference happens during iteration, - # so each segment can stream out (and cancel can cut in) while the - # next one is still being decoded. + # so segments stream out (and cancel can cut in) while later ones + # are still being decoded — per segment in live mode, per batch of + # BATCH_SIZE speech chunks in file mode (D-WS5a). for segment in segments: if request_id in cancelled: was_cancelled = True diff --git a/src/server/live.ts b/src/server/live.ts index 25bf36f..61bfe5b 100644 --- a/src/server/live.ts +++ b/src/server/live.ts @@ -60,6 +60,8 @@ export function createLiveWebSocket(deps: LiveWebSocketDeps) { audioPath, language: ws.data.language, task: ws.data.task, + // Short passes: sequential decode, no batching/VAD (D-WS5a). + mode: "live", }, (segment) => { text.push(segment.text); diff --git a/src/server/transcribe.ts b/src/server/transcribe.ts index eb95835..fba9192 100644 --- a/src/server/transcribe.ts +++ b/src/server/transcribe.ts @@ -47,8 +47,15 @@ export interface SidecarTranscribeRequest { audioPath: string; language?: string; task?: WhisperTask; + /** + * "file" (default): whole uploads — batched inference over VAD speech + * chunks. "live": short server-live passes — sequential, no VAD (D-WS5a). + */ + mode?: SidecarMode; } +export type SidecarMode = "file" | "live"; + export interface SidecarStatus { available: boolean; reason?: string; @@ -265,6 +272,7 @@ export function createSidecar(command?: readonly string[]): Sidecar { audio_path: request.audioPath, language: request.language ?? null, task: request.task ?? "transcribe", + mode: request.mode ?? "file", }); } catch (error) { pending.delete(id); @@ -308,6 +316,8 @@ export interface TranscribeDeps { maxUploadBytes?: number; /** Directory for uploaded-PCM temp files. Default: the OS temp dir. */ tempDir?: string; + /** Blank-line heartbeat period on the ndjson stream (D-021). Default 5 s. */ + heartbeatIntervalMs?: number; } function json(body: unknown, status: number): Response { @@ -401,6 +411,7 @@ async function writeBodyToFile( export function createTranscribeHandlers(deps: TranscribeDeps): TranscribeHandlers { const maxUploadBytes = deps.maxUploadBytes ?? DEFAULT_MAX_UPLOAD_BYTES; const tempDir = deps.tempDir ?? tmpdir(); + const heartbeatIntervalMs = deps.heartbeatIntervalMs ?? HEARTBEAT_INTERVAL_MS; return { async status(request) { @@ -452,7 +463,7 @@ export function createTranscribeHandlers(deps: TranscribeDeps): TranscribeHandle } catch { clearInterval(heartbeat); } - }, HEARTBEAT_INTERVAL_MS); + }, heartbeatIntervalMs); const chunks: TranscriptChunk[] = []; try { await deps.sidecar.transcribe( diff --git a/tests/fixtures/fake_sidecar.py b/tests/fixtures/fake_sidecar.py index ac4f3e6..b69b8e2 100644 --- a/tests/fixtures/fake_sidecar.py +++ b/tests/fixtures/fake_sidecar.py @@ -5,7 +5,7 @@ Used by tests/transcribe.test.ts to exercise the real Bun-side supervisor requested model name: "ok" two segments (the second reports the audio byte count, - language and task for pass-through assertions), then done + language, task and mode for pass-through assertions), then done "boom" an error line "die" exit(1) mid-request (crash simulation) "wait-for-cancel" one segment, then block until the next stdin line @@ -53,6 +53,7 @@ for line in sys.stdin: emit({"type": "segment", "id": request_id, "start": 0.0, "end": 1.5, "text": " Hello"}) emit({"type": "segment", "id": request_id, "start": 1.5, "end": 3.0, - "text": " world. [model=%s bytes=%d lang=%s task=%s]" - % (model, size, message.get("language"), message.get("task"))}) + "text": " world. [model=%s bytes=%d lang=%s task=%s mode=%s]" + % (model, size, message.get("language"), message.get("task"), + message.get("mode"))}) emit({"type": "done", "id": request_id, "cancelled": False}) diff --git a/tests/fixtures/stub_fw/faster_whisper/__init__.py b/tests/fixtures/stub_fw/faster_whisper/__init__.py new file mode 100644 index 0000000..e46df41 --- /dev/null +++ b/tests/fixtures/stub_fw/faster_whisper/__init__.py @@ -0,0 +1,61 @@ +"""faster_whisper stand-in that reports how server-py/sidecar.py called it. + +Lets tests/transcribe.test.ts check the real sidecar's request routing +(D-WS5a: file → BatchedInferencePipeline + VAD, live → sequential) without +CTranslate2 or model weights. Every segment's text describes the call: + + " seq model= loads= " WhisperModel.transcribe + " batched model= loads= " BatchedInferencePipeline + +Model names with special behaviour: + "slow" 20 segments, 0.1 s apart (cancel lands between them) + "silence" no segments at all (VAD found no speech) +""" + +import time + +__version__ = "stub" + +_loads = {"count": 0} + + +class _Segment: + def __init__(self, start, end, text): + self.start = start + self.end = end + self.text = text + + +def _describe(kind, model, kwargs): + options = " ".join( + "%s=%s" % (key, kwargs[key]) for key in sorted(kwargs) if kwargs[key] is not None + ) + return " %s model=%s loads=%d %s" % (kind, model.name, _loads["count"], options) + + +def _segments(kind, model, kwargs): + if model.name == "silence": + return + if model.name == "slow": + for index in range(20): + yield _Segment(float(index), float(index + 1), " slow %d" % index) + time.sleep(0.1) + return + yield _Segment(0.0, 1.0, _describe(kind, model, kwargs)) + + +class WhisperModel: + def __init__(self, name, device=None, compute_type=None, download_root=None): + _loads["count"] += 1 + self.name = name + + def transcribe(self, audio, **kwargs): + return _segments("seq", self, kwargs), None + + +class BatchedInferencePipeline: + def __init__(self, model): + self.model = model + + def transcribe(self, audio, **kwargs): + return _segments("batched", self.model, kwargs), None diff --git a/tests/fixtures/stub_fw/numpy/__init__.py b/tests/fixtures/stub_fw/numpy/__init__.py new file mode 100644 index 0000000..9cb2297 --- /dev/null +++ b/tests/fixtures/stub_fw/numpy/__init__.py @@ -0,0 +1,13 @@ +"""Minimal numpy stand-in for running server-py/sidecar.py without numpy. + +Put this directory on PYTHONPATH (with `python3 -S` so the real site-packages +stay out). The sidecar only calls `np.fromfile(path, dtype=np.float32)`; the +stub returns the raw bytes so the faster_whisper stub can report their size. +""" + +float32 = "float32" + + +def fromfile(path, dtype=None): + with open(path, "rb") as handle: + return handle.read() diff --git a/tests/server-engine.md b/tests/server-engine.md index 1b238f0..6ac7ba9 100644 --- a/tests/server-engine.md +++ b/tests/server-engine.md @@ -21,7 +21,19 @@ FW_INTEGRATION=1 bun test tests/transcribe.test.ts ``` Runs the real sidecar end to end (spawn → ready handshake → `tiny.en` load → -transcribe → done). First run downloads ~75 MB of weights from the Hub. +transcribe → done), then the first 60 s of `tests/audio_files/en_alien.opus` +(needs `ffmpeg`) in both decoding modes (D-WS5a): `file` (faster-whisper's +`BatchedInferencePipeline` + Silero VAD, what `POST /api/transcribe` uses) and +`live` (sequential, what server-live passes use). First run downloads ~75 MB +of weights from the Hub. Point `SIDECAR_PYTHON` at a virtualenv's python if +faster-whisper isn't installed for `python3` itself. + +Without faster-whisper at all, `bun test tests/transcribe.test.ts` still runs +the real `server-py/sidecar.py` against a stub (`tests/fixtures/stub_fw`) to +check the mode routing, the one-model cache and cancel between segments. +With `FW_INTEGRATION=1` it also pushes the whole 15-min `en_alien` clip +(`tiny.en`) through `POST /api/transcribe` — heartbeat lines before the first +burst, segments in bursts — and cancels a batched run after its first burst. - [ ] The `real faster-whisper sidecar` describe block passes. @@ -39,7 +51,14 @@ transcribe → done). First run downloads ~75 MB of weights from the Hub. where `audio.f32` is raw float32 mono 16 kHz PCM). 5. Transcribe the same clip through the server engine. -- [ ] Segments stream in as JSON lines (not one blob at the end). +- [ ] Segments stream in as JSON lines (not one blob at the end). With the + batched pipeline (D-WS5a) they arrive in bursts — one burst per 8 + speech chunks (≈ up to 4 min of audio) — with blank heartbeat lines + every 5 s in between; on a clip under ~30 s of speech, everything + arrives as a single segment at the end. Each segment covers one VAD + speech chunk (up to 30 s), not one sentence — expected (D-WS5a). +- [ ] Cancelling a long server transcription (abort / page refresh) stops + the sidecar within one burst; the next request then starts normally. - [ ] Both transcripts agree in content (minor punctuation/casing drift between Transformers.js and faster-whisper is expected). - [ ] Signed out, the toggle is gone and `POST /api/transcribe` answers 401. diff --git a/tests/server-live.test.ts b/tests/server-live.test.ts index 3c41acc..da024e7 100644 --- a/tests/server-live.test.ts +++ b/tests/server-live.test.ts @@ -111,7 +111,7 @@ function sessionData(overrides: Partial = {}): LiveSessionData } describe("createLiveWebSocket", () => { - test("a committed segment writes the exact PCM bytes and passes model/language/task through", async () => { + test("a committed segment writes the exact PCM bytes and passes model/language/task/mode through", async () => { const tempDir = await makeTempDir(); const sidecar = fakeSidecar(); const handler = createLiveWebSocket({ sidecar }); @@ -123,7 +123,8 @@ describe("createLiveWebSocket", () => { await waitFor(() => sidecar.requests.length > 0); expect(sidecar.requests[0]).toEqual( - expect.objectContaining({ model: "base", language: "da", task: "translate" }), + // Live passes are short: sequential decoding, no batching/VAD (D-WS5a). + expect.objectContaining({ model: "base", language: "da", task: "translate", mode: "live" }), ); // The bytes written to the temp file are exactly the committed segment's // samples — not, e.g., a whole shared buffer (see the byteOffset/byteLength fix). diff --git a/tests/transcribe.test.ts b/tests/transcribe.test.ts index 2b54b66..b8033b1 100644 --- a/tests/transcribe.test.ts +++ b/tests/transcribe.test.ts @@ -52,6 +52,8 @@ function fakeSidecar( segments?: SidecarSegment[]; error?: string; waitForAbort?: boolean; + /** Quiet time before the first segment (model load, VAD, first batch). */ + delayMs?: number; } = {}, ): FakeSidecar { const segments = behavior.segments ?? [ @@ -72,6 +74,7 @@ function fakeSidecar( new Uint8Array(await Bun.file(request.audioPath).arrayBuffer()), ); if (behavior.error) throw new Error(behavior.error); + if (behavior.delayMs) await Bun.sleep(behavior.delayMs); onSegment(segments[0]!); if (behavior.waitForAbort) { await new Promise((resolve) => { @@ -323,6 +326,25 @@ describe("POST /api/transcribe", () => { expect(await readdir(tempDir)).toEqual([]); }); + test("heartbeat blank lines keep a quiet stream alive before the first segment (D-021)", async () => { + // The batched pipeline (D-WS5a) runs VAD over the whole file and decodes a + // full batch before its first segment, so the quiet stretch is longer. + const handlers = createTranscribeHandlers({ + sidecar: fakeSidecar({ delayMs: 200 }), + getSessionUser: sessionFromCookie, + tempDir: await makeTempDir(), + heartbeatIntervalMs: 20, + }); + const response = await handlers.post( + postRequest("?model=onnx-community/whisper-base", pcmBody([0.1]), { cookie: "sid=ok" }), + ); + const text = await response.text(); + const beforeFirstSegment = text.slice(0, text.indexOf("{")); + expect(beforeFirstSegment.length).toBeGreaterThanOrEqual(2); + expect(beforeFirstSegment.trim()).toBe(""); + expect(text).toContain('"type":"result"'); + }); + test("sidecar errors travel in-stream after the 200 header", async () => { const tempDir = await makeTempDir(); const handlers = createTranscribeHandlers({ @@ -385,8 +407,12 @@ describe("createSidecar (fake python sidecar)", () => { expect(outcome).toEqual({ cancelled: false }); expect(segments).toHaveLength(2); expect(segments[0]).toEqual({ start: 0, end: 1.5, text: " Hello" }); - // The fixture echoes what it received (audio bytes, language, task). - expect(segments[1]!.text).toContain("model=ok bytes=12 lang=da task=translate"); + // The fixture echoes what it received (audio bytes, language, task, mode). + expect(segments[1]!.text).toContain("model=ok bytes=12 lang=da task=translate mode=file"); + + const live: SidecarSegment[] = []; + await sidecar.transcribe({ model: "ok", audioPath, mode: "live" }, (s) => live.push(s)); + expect(live[1]!.text).toContain("mode=live"); }); test("sidecar 'error' lines reject that request only", async () => { @@ -455,6 +481,90 @@ describe("createSidecar (degradation, D-013)", () => { }); }); +// --------------------------------------------------------------------------- +// The real server-py/sidecar.py against a stubbed faster_whisper/numpy +// (tests/fixtures/stub_fw): request routing, model cache, cancel (D-WS5a). +// `-S` keeps real site-packages out, so this runs the same everywhere. +// --------------------------------------------------------------------------- + +describe("server-py/sidecar.py (stubbed faster-whisper, D-WS5a)", () => { + const STUB_DIR = join(import.meta.dir, "fixtures/stub_fw"); + const stubbedSidecar = () => + createSidecar(["env", `PYTHONPATH=${STUB_DIR}`, "python3", "-S", REAL_SIDECAR]); + + async function run( + sidecar: Sidecar, + request: Omit, + signal?: AbortSignal, + onSegment: (segment: SidecarSegment) => void = () => {}, + ): Promise<{ texts: string[]; cancelled: boolean }> { + const audioPath = join(await makeTempDir(), "audio.f32"); + await Bun.write(audioPath, pcmBody([0, 0.5, -0.5, 0])); + const texts: string[] = []; + const outcome = await sidecar.transcribe( + { ...request, audioPath }, + (segment) => { + texts.push(segment.text); + onSegment(segment); + }, + signal, + ); + return { texts, cancelled: outcome.cancelled }; + } + + test("file mode (and no mode) runs the batched pipeline with VAD, batch_size=8", async () => { + const sidecar = stubbedSidecar(); + const explicit = await run(sidecar, { model: "m1", mode: "file", language: "da" }); + expect(explicit.texts).toHaveLength(1); + expect(explicit.texts[0]).toContain("batched model=m1"); + expect(explicit.texts[0]).toContain("vad_filter=True"); + expect(explicit.texts[0]).toContain("batch_size=8"); + // Tuned Silero VAD: the default (0.5 / 400 ms) dropped quiet speech (D-WS5a). + expect(explicit.texts[0]).toContain("vad_parameters={'threshold': 0.2, 'speech_pad_ms': 800}"); + expect(explicit.texts[0]).toContain("language=da"); + expect(explicit.texts[0]).toContain("task=transcribe"); + + const implicit = await run(sidecar, { model: "m1" }); + expect(implicit.texts[0]).toContain("batched model=m1"); + }); + + test("live mode stays sequential without VAD (short passes, D-WS5a)", async () => { + const sidecar = stubbedSidecar(); + const { texts } = await run(sidecar, { model: "m1", mode: "live", task: "translate" }); + expect(texts[0]).toContain("seq model=m1"); + expect(texts[0]).not.toContain("vad_filter"); + expect(texts[0]).not.toContain("batch_size"); + expect(texts[0]).toContain("task=translate"); + }); + + test("one model stays loaded across modes; a switch reloads and rewraps", async () => { + const sidecar = stubbedSidecar(); + expect((await run(sidecar, { model: "m1" })).texts[0]).toContain("loads=1"); + expect((await run(sidecar, { model: "m1", mode: "live" })).texts[0]).toContain("loads=1"); + const switched = (await run(sidecar, { model: "m2" })).texts[0]; + expect(switched).toContain("batched model=m2"); + expect(switched).toContain("loads=2"); + }); + + test("cancel lands between batched segments", async () => { + const sidecar = stubbedSidecar(); + const controller = new AbortController(); + const { texts, cancelled } = await run(sidecar, { model: "slow" }, controller.signal, () => + controller.abort(), + ); + expect(cancelled).toBe(true); + expect(texts.length).toBeGreaterThanOrEqual(1); + expect(texts.length).toBeLessThan(20); + }); + + test("audio with no speech finishes with zero segments, not an error", async () => { + const sidecar = stubbedSidecar(); + const { texts, cancelled } = await run(sidecar, { model: "silence" }); + expect(texts).toEqual([]); + expect(cancelled).toBe(false); + }); +}); + // --------------------------------------------------------------------------- // Opt-in: real faster-whisper end to end (needs python3 + requirements.txt // installed and network/cached weights). See tests/server-engine.md. @@ -486,4 +596,119 @@ describe.skipIf(!process.env.FW_INTEGRATION)("real faster-whisper sidecar", () = }, { timeout: 300_000 }, ); + + test( + "file mode (batched + VAD) and live mode both transcribe real speech (D-WS5a)", + async () => { + const sidecar = createSidecar(); + // First 60 s of en_alien as raw float32 16 kHz mono — what the browser uploads. + const clip = join(import.meta.dir, "audio_files/en_alien.opus"); + const pcm = + await Bun.$`ffmpeg -v error -i ${clip} -t 60 -f f32le -ac 1 -ar 16000 -`.arrayBuffer(); + const audioPath = join(await makeTempDir(), "alien.f32"); + await Bun.write(audioPath, pcm); + + for (const mode of ["file", "live"] as const) { + const segments: SidecarSegment[] = []; + const outcome = await sidecar.transcribe( + { model: "tiny.en", audioPath, language: "en", mode }, + (segment) => segments.push(segment), + ); + expect(outcome.cancelled).toBe(false); + expect(segments.length).toBeGreaterThan(0); + const text = segments.map((segment) => segment.text).join("").toLowerCase(); + expect(text).toContain("voices of the void"); + // Timestamps stay in source-audio time after VAD drops silence. + expect(segments.at(-1)!.end).toBeLessThanOrEqual(61); + } + }, + { timeout: 300_000 }, + ); + + // The whole 15-min en_alien clip (≈ 17 VAD chunks ⇒ 3 batches) as uploaded PCM. + async function wholeAlienPcm(): Promise { + const clip = join(import.meta.dir, "audio_files/en_alien.opus"); + const pcm = await Bun.$`ffmpeg -v error -i ${clip} -f f32le -ac 1 -ar 16000 -`.arrayBuffer(); + const audioPath = join(await makeTempDir(), "alien-full.f32"); + await Bun.write(audioPath, pcm); + return audioPath; + } + + test( + "a long file streams in bursts behind heartbeats over HTTP (D-021, D-WS5a)", + async () => { + const handlers = createTranscribeHandlers({ + sidecar: createSidecar(), + getSessionUser: sessionFromCookie, + tempDir: await makeTempDir(), + heartbeatIntervalMs: 100, + }); + const body = await Bun.file(await wholeAlienPcm()).arrayBuffer(); + const response = await handlers.post( + postRequest("?model=onnx-community/whisper-tiny.en&language=en", body, { + cookie: "sid=ok", + }), + ); + expect(response.status).toBe(200); + + // Timestamp every line as it arrives. + const started = performance.now(); + const lines: { at: number; line: string }[] = []; + const decoder = new TextDecoder(); + let buffered = ""; + for await (const chunk of response.body!) { + buffered += decoder.decode(chunk, { stream: true }); + let newline: number; + while ((newline = buffered.indexOf("\n")) >= 0) { + lines.push({ at: performance.now() - started, line: buffered.slice(0, newline) }); + buffered = buffered.slice(newline + 1); + } + } + const segmentLines = lines.filter(({ line }) => line.includes('"type":"segment"')); + const firstSegment = lines.findIndex(({ line }) => line.includes('"type":"segment"')); + // VAD over 15 min of audio + the first batch: heartbeats fill the gap. + expect(lines.slice(0, firstSegment).filter(({ line }) => line === "").length).toBeGreaterThan(0); + expect(segmentLines.length).toBeGreaterThan(3); + // Streamed, not one blob at the end: later bursts arrive measurably later. + expect(segmentLines.at(-1)!.at - segmentLines[0]!.at).toBeGreaterThan(100); + expect(lines.at(-1)!.line).toContain('"type":"result"'); + console.log( + "[D-WS5a] segment arrival (ms):", + segmentLines.map(({ at }) => Math.round(at)).join(" "), + ); + }, + { timeout: 600_000 }, + ); + + test( + "cancelling a batched file run stops it early and the next request still works", + async () => { + const sidecar = createSidecar(); + const audioPath = await wholeAlienPcm(); + const controller = new AbortController(); + const cancelledRun: SidecarSegment[] = []; + const outcome = await sidecar.transcribe( + { model: "tiny.en", audioPath, language: "en" }, + (segment) => { + cancelledRun.push(segment); + controller.abort(); + }, + controller.signal, + ); + expect(outcome.cancelled).toBe(true); + // At most the first burst (8 chunks) got out; the full clip is ≈ 17 chunks. + expect(cancelledRun.length).toBeGreaterThan(0); + expect(cancelledRun.length).toBeLessThanOrEqual(8); + expect(cancelledRun.at(-1)!.end).toBeLessThan(800); + + const again: SidecarSegment[] = []; + const next = await sidecar.transcribe( + { model: "tiny.en", audioPath, language: "en" }, + (segment) => again.push(segment), + ); + expect(next.cancelled).toBe(false); + expect(again.length).toBeGreaterThan(cancelledRun.length); + }, + { timeout: 600_000 }, + ); });