Merge feat/server-batched-vad: faster-whisper batched + VAD (spec 001 workstream 5)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/Decisions.md
This commit is contained in:
Lucas Winther
@@ -230,7 +230,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
|
||||
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
|
||||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||||
JSON-lines HTTP response (D-011). The shared catalog maps model ids to
|
||||
JSON-lines HTTP response (D-011). File requests (`mode: "file"`) decode with
|
||||
faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live
|
||||
passes send `mode: "live"` and stay sequential (D-WS5a). The shared catalog maps model ids to
|
||||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||||
the server engine reports unavailable; the browser/anonymous flow is unaffected.
|
||||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||||
|
||||
@@ -1483,3 +1483,48 @@ use beam 5.
|
||||
**Pending the owner:** the Hugging Face account name, then upload the six prepared folders (model
|
||||
cards credit the original authors and carry MIT, or Apache-2.0 + NOTICE), then set
|
||||
`DANISH_MODELS_HF_OWNER`. The integrator sets `canTranslate: false` on these entries at merge.
|
||||
|
||||
## D-WS5a — Server engine: batched faster-whisper + tuned Silero VAD for files; live passes stay sequential
|
||||
**Status:** Accepted (2026-10-30, spec 001 Goal 5)
|
||||
`server-py/sidecar.py` decoded every request with the sequential
|
||||
`WhisperModel.transcribe`, no VAD.
|
||||
**Decision:** the sidecar request gains `mode: "file" | "live"` (absent ⇒
|
||||
`"file"`, so an older Bun still gets the new path). `"file"` (`POST
|
||||
/api/transcribe`) runs `BatchedInferencePipeline(model).transcribe(audio,
|
||||
vad_filter=True, vad_parameters={threshold: 0.2, speech_pad_ms: 800},
|
||||
batch_size=8)`; the pipeline is a weightless wrapper cached and rebuilt with
|
||||
the one resident model. `"live"` (server-live passes, `src/server/live.ts`)
|
||||
keeps the sequential decoder. Beam size stays at the default 5.
|
||||
**Measurements** (CPU int8, AMD Ryzen 9 9950X, 16 cores shared with other
|
||||
jobs — load average 7–24 during runs, so wall-clock is ±20 %; WER via jiwer
|
||||
on the `tests/live-real-model.test.ts` normalisation vs `tests/audio_files/*.txt`):
|
||||
|
||||
| model | audio | config | wall-clock | WER |
|
||||
|---|---|---|---|---|
|
||||
| large-v3-turbo | en_alien (888 s) | sequential (before) | 97 s / 130 s (two runs) | 0.097 / 0.067 |
|
||||
| large-v3-turbo | en_alien | sequential + VAD | 79 s | 0.136 |
|
||||
| large-v3-turbo | en_alien | batched, default VAD (0.5 / 400 ms) | 47 s / 59 s | 0.117 |
|
||||
| large-v3-turbo | en_alien | batched, beam 1 | 44 s | 0.114 |
|
||||
| large-v3-turbo | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **51 s / 58 s** | **0.086** |
|
||||
| large-v3-turbo | en_alien | batched, VAD 0.2 / 800 ms, word-level timestamps | 56 s | 0.144 |
|
||||
| large-v3-turbo | da_festuge (21.5 s, da) | sequential (before) | 2.9 s | 0.094 |
|
||||
| large-v3-turbo | da_festuge | batched, default VAD | 2.9 s | 0.141 |
|
||||
| large-v3-turbo | da_festuge | **batched, VAD 0.2 / 800 ms (after)** | **2.7 s** | **0.109** |
|
||||
| medium.en | en_alien | sequential (before) | 114 s | 0.126 |
|
||||
| medium.en | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **67 s** | **0.111** |
|
||||
|
||||
The sequential decoder is not deterministic run to run on en_alien (temperature
|
||||
fallback; 0.067–0.097 over three runs), while the batched result repeated
|
||||
exactly. Default-VAD errors were deletions (121 vs 64 sequential): Silero at
|
||||
0.5 dropped quiet speech under music; 0.2 / 800 ms recovered most of them.
|
||||
da_festuge's gap is one word of 64. Live-style 6 s passes over the first
|
||||
180 s of en_alien: sequential 2.45 s/pass, batched 2.12, sequential + VAD
|
||||
1.88 — within the load noise, each losing a few words, and real live
|
||||
passes are already speech-gated client-side (D-016), so live stays as it was.
|
||||
**Consequences:** file transcriptions are ≈ 1.7–2.5× faster on CPU at
|
||||
equal-or-better WER on en_alien (one word worse on the 21 s Danish clip). Segments are coarser — one per VAD chunk (≤ 30 s) instead of one per
|
||||
sentence — because word-level timestamps (`without_timestamps=False`) cost
|
||||
WER here (0.144); SRT/VTT cues from the server engine are correspondingly
|
||||
long. Segments arrive in bursts of 8 chunks (≈ 4 min of speech) and the first
|
||||
burst can take ~15–25 s on a long file; the D-021 heartbeat covers it, and
|
||||
cancel lands at the next burst boundary. No new env vars.
|
||||
|
||||
@@ -153,6 +153,9 @@ Notes (M4):
|
||||
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
|
||||
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
|
||||
model catalog to faster-whisper model ids in `src/lib/models.ts`
|
||||
- [x] Batched inference + tuned Silero VAD for server-engine file
|
||||
transcriptions, live passes sequential (spec 001 Goal 5, D-WS5a):
|
||||
≈ 2× faster on CPU int8 at equal-or-better WER; segments ≤ 30 s chunks
|
||||
- [x] Degrade gracefully when Python/the sidecar is absent (server engine
|
||||
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
|
||||
- [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
|
||||
|
||||
+44
-7
@@ -9,7 +9,8 @@ stdio — one JSON object per line, no framing beyond that:
|
||||
|
||||
stdin -> {"type": "transcribe", "id": "1", "model": "tiny.en",
|
||||
"audio_path": "/tmp/...", "language": "en" | null,
|
||||
"task": "transcribe" | "translate"}
|
||||
"task": "transcribe" | "translate",
|
||||
"mode": "file" | "live"} # optional, default "file"
|
||||
stdout <- {"type": "segment", "id": "1", "start": 0.0, "end": 4.2,
|
||||
"text": " Hello world."} # 0..n of these
|
||||
stdout <- {"type": "done", "id": "1", "cancelled": false}
|
||||
@@ -24,6 +25,18 @@ container decoding happens here. Requests are processed strictly one at a
|
||||
time; a reader thread keeps consuming stdin meanwhile so `cancel` can
|
||||
interrupt the segment loop of the request in flight.
|
||||
|
||||
Decoding strategy by mode (Decisions.md D-WS5a):
|
||||
"file" faster-whisper's BatchedInferencePipeline with its Silero VAD
|
||||
(vad_filter=True, VAD_PARAMETERS): silence is dropped, speech is cut
|
||||
into <= 30 s chunks and BATCH_SIZE chunks decode per forward pass.
|
||||
Without word-level timestamps each chunk is one segment (<= 30 s).
|
||||
Segments come out in bursts (one burst per batch), so a cancel
|
||||
takes effect at the next burst boundary; the Bun side's heartbeat
|
||||
(D-021) covers the longer quiet stretch before the first burst.
|
||||
"live" the plain sequential WhisperModel.transcribe: server-live passes
|
||||
are short (<= 29 s, often a few seconds), so batching has nothing to
|
||||
batch and VAD only adds latency and can clip a short utterance.
|
||||
|
||||
The last-used model stays loaded between requests (WorkBreakdown M5 warm-up:
|
||||
never reload weights per request) and is replaced — not accumulated — when a
|
||||
request names a different model, bounding memory to one model.
|
||||
@@ -47,6 +60,13 @@ import sys
|
||||
import threading
|
||||
|
||||
SAMPLE_RATE = 16000
|
||||
BATCH_SIZE = 8 # D-WS5a: VAD chunks decoded per batched forward pass
|
||||
# D-WS5a: Silero VAD tuned for file mode. faster-whisper's batched default
|
||||
# (threshold 0.5, 400 ms padding) dropped quiet and overlapped speech —
|
||||
# deletions nearly doubled on en_alien; 0.2 / 800 ms brought WER back to the
|
||||
# sequential decoder's level. min_silence_duration_ms stays at VadOptions'
|
||||
# 2000 ms default (a dict replaces the pipeline's 160 ms preset).
|
||||
VAD_PARAMETERS = {"threshold": 0.2, "speech_pad_ms": 800}
|
||||
|
||||
|
||||
def emit(message):
|
||||
@@ -101,6 +121,7 @@ def main():
|
||||
|
||||
cached_name = None
|
||||
cached_model = None
|
||||
cached_pipeline = None # BatchedInferencePipeline wrapping cached_model
|
||||
|
||||
emit({"type": "ready", "version": getattr(faster_whisper, "__version__", "?")})
|
||||
|
||||
@@ -115,26 +136,42 @@ def main():
|
||||
audio_path = request["audio_path"]
|
||||
language = request.get("language") or None
|
||||
task = request.get("task") or "transcribe"
|
||||
mode = request.get("mode") or "file"
|
||||
|
||||
if model_name != cached_name:
|
||||
cached_model = None # release before loading the replacement
|
||||
# Release both before loading the replacement: one model resident.
|
||||
cached_model = cached_pipeline = None
|
||||
cached_name = None
|
||||
cached_model = faster_whisper.WhisperModel(
|
||||
model_name,
|
||||
device=device,
|
||||
compute_type=compute_type,
|
||||
download_root=download_root,
|
||||
)
|
||||
# A thin wrapper (no weights of its own) — shares cached_model.
|
||||
cached_pipeline = faster_whisper.BatchedInferencePipeline(cached_model)
|
||||
cached_name = model_name
|
||||
|
||||
audio = np.fromfile(audio_path, dtype=np.float32)
|
||||
segments, _info = cached_model.transcribe(
|
||||
audio, language=language, task=task
|
||||
)
|
||||
if mode == "live":
|
||||
segments, _info = cached_model.transcribe(
|
||||
audio, language=language, task=task
|
||||
)
|
||||
else:
|
||||
segments, _info = cached_pipeline.transcribe(
|
||||
audio,
|
||||
language=language,
|
||||
task=task,
|
||||
vad_filter=True,
|
||||
vad_parameters=dict(VAD_PARAMETERS), # fresh: faster-whisper may mutate it
|
||||
batch_size=BATCH_SIZE,
|
||||
)
|
||||
|
||||
was_cancelled = False
|
||||
# `segments` is a generator: inference happens during iteration,
|
||||
# so each segment can stream out (and cancel can cut in) while the
|
||||
# next one is still being decoded.
|
||||
# so segments stream out (and cancel can cut in) while later ones
|
||||
# are still being decoded — per segment in live mode, per batch of
|
||||
# BATCH_SIZE speech chunks in file mode (D-WS5a).
|
||||
for segment in segments:
|
||||
if request_id in cancelled:
|
||||
was_cancelled = True
|
||||
|
||||
@@ -60,6 +60,8 @@ export function createLiveWebSocket(deps: LiveWebSocketDeps) {
|
||||
audioPath,
|
||||
language: ws.data.language,
|
||||
task: ws.data.task,
|
||||
// Short passes: sequential decode, no batching/VAD (D-WS5a).
|
||||
mode: "live",
|
||||
},
|
||||
(segment) => {
|
||||
text.push(segment.text);
|
||||
|
||||
@@ -47,8 +47,15 @@ export interface SidecarTranscribeRequest {
|
||||
audioPath: string;
|
||||
language?: string;
|
||||
task?: WhisperTask;
|
||||
/**
|
||||
* "file" (default): whole uploads — batched inference over VAD speech
|
||||
* chunks. "live": short server-live passes — sequential, no VAD (D-WS5a).
|
||||
*/
|
||||
mode?: SidecarMode;
|
||||
}
|
||||
|
||||
export type SidecarMode = "file" | "live";
|
||||
|
||||
export interface SidecarStatus {
|
||||
available: boolean;
|
||||
reason?: string;
|
||||
@@ -265,6 +272,7 @@ export function createSidecar(command?: readonly string[]): Sidecar {
|
||||
audio_path: request.audioPath,
|
||||
language: request.language ?? null,
|
||||
task: request.task ?? "transcribe",
|
||||
mode: request.mode ?? "file",
|
||||
});
|
||||
} catch (error) {
|
||||
pending.delete(id);
|
||||
@@ -308,6 +316,8 @@ export interface TranscribeDeps {
|
||||
maxUploadBytes?: number;
|
||||
/** Directory for uploaded-PCM temp files. Default: the OS temp dir. */
|
||||
tempDir?: string;
|
||||
/** Blank-line heartbeat period on the ndjson stream (D-021). Default 5 s. */
|
||||
heartbeatIntervalMs?: number;
|
||||
}
|
||||
|
||||
function json(body: unknown, status: number): Response {
|
||||
@@ -401,6 +411,7 @@ async function writeBodyToFile(
|
||||
export function createTranscribeHandlers(deps: TranscribeDeps): TranscribeHandlers {
|
||||
const maxUploadBytes = deps.maxUploadBytes ?? DEFAULT_MAX_UPLOAD_BYTES;
|
||||
const tempDir = deps.tempDir ?? tmpdir();
|
||||
const heartbeatIntervalMs = deps.heartbeatIntervalMs ?? HEARTBEAT_INTERVAL_MS;
|
||||
|
||||
return {
|
||||
async status(request) {
|
||||
@@ -452,7 +463,7 @@ export function createTranscribeHandlers(deps: TranscribeDeps): TranscribeHandle
|
||||
} catch {
|
||||
clearInterval(heartbeat);
|
||||
}
|
||||
}, HEARTBEAT_INTERVAL_MS);
|
||||
}, heartbeatIntervalMs);
|
||||
const chunks: TranscriptChunk[] = [];
|
||||
try {
|
||||
await deps.sidecar.transcribe(
|
||||
|
||||
Vendored
+4
-3
@@ -5,7 +5,7 @@ Used by tests/transcribe.test.ts to exercise the real Bun-side supervisor
|
||||
requested model name:
|
||||
|
||||
"ok" two segments (the second reports the audio byte count,
|
||||
language and task for pass-through assertions), then done
|
||||
language, task and mode for pass-through assertions), then done
|
||||
"boom" an error line
|
||||
"die" exit(1) mid-request (crash simulation)
|
||||
"wait-for-cancel" one segment, then block until the next stdin line
|
||||
@@ -53,6 +53,7 @@ for line in sys.stdin:
|
||||
emit({"type": "segment", "id": request_id, "start": 0.0, "end": 1.5,
|
||||
"text": " Hello"})
|
||||
emit({"type": "segment", "id": request_id, "start": 1.5, "end": 3.0,
|
||||
"text": " world. [model=%s bytes=%d lang=%s task=%s]"
|
||||
% (model, size, message.get("language"), message.get("task"))})
|
||||
"text": " world. [model=%s bytes=%d lang=%s task=%s mode=%s]"
|
||||
% (model, size, message.get("language"), message.get("task"),
|
||||
message.get("mode"))})
|
||||
emit({"type": "done", "id": request_id, "cancelled": False})
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""faster_whisper stand-in that reports how server-py/sidecar.py called it.
|
||||
|
||||
Lets tests/transcribe.test.ts check the real sidecar's request routing
|
||||
(D-WS5a: file → BatchedInferencePipeline + VAD, live → sequential) without
|
||||
CTranslate2 or model weights. Every segment's text describes the call:
|
||||
|
||||
" seq model=<name> loads=<n> <sorted kwargs>" WhisperModel.transcribe
|
||||
" batched model=<name> loads=<n> <sorted kwargs>" BatchedInferencePipeline
|
||||
|
||||
Model names with special behaviour:
|
||||
"slow" 20 segments, 0.1 s apart (cancel lands between them)
|
||||
"silence" no segments at all (VAD found no speech)
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
__version__ = "stub"
|
||||
|
||||
_loads = {"count": 0}
|
||||
|
||||
|
||||
class _Segment:
|
||||
def __init__(self, start, end, text):
|
||||
self.start = start
|
||||
self.end = end
|
||||
self.text = text
|
||||
|
||||
|
||||
def _describe(kind, model, kwargs):
|
||||
options = " ".join(
|
||||
"%s=%s" % (key, kwargs[key]) for key in sorted(kwargs) if kwargs[key] is not None
|
||||
)
|
||||
return " %s model=%s loads=%d %s" % (kind, model.name, _loads["count"], options)
|
||||
|
||||
|
||||
def _segments(kind, model, kwargs):
|
||||
if model.name == "silence":
|
||||
return
|
||||
if model.name == "slow":
|
||||
for index in range(20):
|
||||
yield _Segment(float(index), float(index + 1), " slow %d" % index)
|
||||
time.sleep(0.1)
|
||||
return
|
||||
yield _Segment(0.0, 1.0, _describe(kind, model, kwargs))
|
||||
|
||||
|
||||
class WhisperModel:
|
||||
def __init__(self, name, device=None, compute_type=None, download_root=None):
|
||||
_loads["count"] += 1
|
||||
self.name = name
|
||||
|
||||
def transcribe(self, audio, **kwargs):
|
||||
return _segments("seq", self, kwargs), None
|
||||
|
||||
|
||||
class BatchedInferencePipeline:
|
||||
def __init__(self, model):
|
||||
self.model = model
|
||||
|
||||
def transcribe(self, audio, **kwargs):
|
||||
return _segments("batched", self.model, kwargs), None
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
"""Minimal numpy stand-in for running server-py/sidecar.py without numpy.
|
||||
|
||||
Put this directory on PYTHONPATH (with `python3 -S` so the real site-packages
|
||||
stay out). The sidecar only calls `np.fromfile(path, dtype=np.float32)`; the
|
||||
stub returns the raw bytes so the faster_whisper stub can report their size.
|
||||
"""
|
||||
|
||||
float32 = "float32"
|
||||
|
||||
|
||||
def fromfile(path, dtype=None):
|
||||
with open(path, "rb") as handle:
|
||||
return handle.read()
|
||||
+21
-2
@@ -21,7 +21,19 @@ FW_INTEGRATION=1 bun test tests/transcribe.test.ts
|
||||
```
|
||||
|
||||
Runs the real sidecar end to end (spawn → ready handshake → `tiny.en` load →
|
||||
transcribe → done). First run downloads ~75 MB of weights from the Hub.
|
||||
transcribe → done), then the first 60 s of `tests/audio_files/en_alien.opus`
|
||||
(needs `ffmpeg`) in both decoding modes (D-WS5a): `file` (faster-whisper's
|
||||
`BatchedInferencePipeline` + Silero VAD, what `POST /api/transcribe` uses) and
|
||||
`live` (sequential, what server-live passes use). First run downloads ~75 MB
|
||||
of weights from the Hub. Point `SIDECAR_PYTHON` at a virtualenv's python if
|
||||
faster-whisper isn't installed for `python3` itself.
|
||||
|
||||
Without faster-whisper at all, `bun test tests/transcribe.test.ts` still runs
|
||||
the real `server-py/sidecar.py` against a stub (`tests/fixtures/stub_fw`) to
|
||||
check the mode routing, the one-model cache and cancel between segments.
|
||||
With `FW_INTEGRATION=1` it also pushes the whole 15-min `en_alien` clip
|
||||
(`tiny.en`) through `POST /api/transcribe` — heartbeat lines before the first
|
||||
burst, segments in bursts — and cancels a batched run after its first burst.
|
||||
|
||||
- [ ] The `real faster-whisper sidecar` describe block passes.
|
||||
|
||||
@@ -39,7 +51,14 @@ transcribe → done). First run downloads ~75 MB of weights from the Hub.
|
||||
where `audio.f32` is raw float32 mono 16 kHz PCM).
|
||||
5. Transcribe the same clip through the server engine.
|
||||
|
||||
- [ ] Segments stream in as JSON lines (not one blob at the end).
|
||||
- [ ] Segments stream in as JSON lines (not one blob at the end). With the
|
||||
batched pipeline (D-WS5a) they arrive in bursts — one burst per 8
|
||||
speech chunks (≈ up to 4 min of audio) — with blank heartbeat lines
|
||||
every 5 s in between; on a clip under ~30 s of speech, everything
|
||||
arrives as a single segment at the end. Each segment covers one VAD
|
||||
speech chunk (up to 30 s), not one sentence — expected (D-WS5a).
|
||||
- [ ] Cancelling a long server transcription (abort / page refresh) stops
|
||||
the sidecar within one burst; the next request then starts normally.
|
||||
- [ ] Both transcripts agree in content (minor punctuation/casing drift
|
||||
between Transformers.js and faster-whisper is expected).
|
||||
- [ ] Signed out, the toggle is gone and `POST /api/transcribe` answers 401.
|
||||
|
||||
@@ -111,7 +111,7 @@ function sessionData(overrides: Partial<LiveSessionData> = {}): LiveSessionData
|
||||
}
|
||||
|
||||
describe("createLiveWebSocket", () => {
|
||||
test("a committed segment writes the exact PCM bytes and passes model/language/task through", async () => {
|
||||
test("a committed segment writes the exact PCM bytes and passes model/language/task/mode through", async () => {
|
||||
const tempDir = await makeTempDir();
|
||||
const sidecar = fakeSidecar();
|
||||
const handler = createLiveWebSocket({ sidecar });
|
||||
@@ -123,7 +123,8 @@ describe("createLiveWebSocket", () => {
|
||||
|
||||
await waitFor(() => sidecar.requests.length > 0);
|
||||
expect(sidecar.requests[0]).toEqual(
|
||||
expect.objectContaining({ model: "base", language: "da", task: "translate" }),
|
||||
// Live passes are short: sequential decoding, no batching/VAD (D-WS5a).
|
||||
expect.objectContaining({ model: "base", language: "da", task: "translate", mode: "live" }),
|
||||
);
|
||||
// The bytes written to the temp file are exactly the committed segment's
|
||||
// samples — not, e.g., a whole shared buffer (see the byteOffset/byteLength fix).
|
||||
|
||||
+227
-2
@@ -52,6 +52,8 @@ function fakeSidecar(
|
||||
segments?: SidecarSegment[];
|
||||
error?: string;
|
||||
waitForAbort?: boolean;
|
||||
/** Quiet time before the first segment (model load, VAD, first batch). */
|
||||
delayMs?: number;
|
||||
} = {},
|
||||
): FakeSidecar {
|
||||
const segments = behavior.segments ?? [
|
||||
@@ -72,6 +74,7 @@ function fakeSidecar(
|
||||
new Uint8Array(await Bun.file(request.audioPath).arrayBuffer()),
|
||||
);
|
||||
if (behavior.error) throw new Error(behavior.error);
|
||||
if (behavior.delayMs) await Bun.sleep(behavior.delayMs);
|
||||
onSegment(segments[0]!);
|
||||
if (behavior.waitForAbort) {
|
||||
await new Promise<void>((resolve) => {
|
||||
@@ -323,6 +326,25 @@ describe("POST /api/transcribe", () => {
|
||||
expect(await readdir(tempDir)).toEqual([]);
|
||||
});
|
||||
|
||||
test("heartbeat blank lines keep a quiet stream alive before the first segment (D-021)", async () => {
|
||||
// The batched pipeline (D-WS5a) runs VAD over the whole file and decodes a
|
||||
// full batch before its first segment, so the quiet stretch is longer.
|
||||
const handlers = createTranscribeHandlers({
|
||||
sidecar: fakeSidecar({ delayMs: 200 }),
|
||||
getSessionUser: sessionFromCookie,
|
||||
tempDir: await makeTempDir(),
|
||||
heartbeatIntervalMs: 20,
|
||||
});
|
||||
const response = await handlers.post(
|
||||
postRequest("?model=onnx-community/whisper-base", pcmBody([0.1]), { cookie: "sid=ok" }),
|
||||
);
|
||||
const text = await response.text();
|
||||
const beforeFirstSegment = text.slice(0, text.indexOf("{"));
|
||||
expect(beforeFirstSegment.length).toBeGreaterThanOrEqual(2);
|
||||
expect(beforeFirstSegment.trim()).toBe("");
|
||||
expect(text).toContain('"type":"result"');
|
||||
});
|
||||
|
||||
test("sidecar errors travel in-stream after the 200 header", async () => {
|
||||
const tempDir = await makeTempDir();
|
||||
const handlers = createTranscribeHandlers({
|
||||
@@ -385,8 +407,12 @@ describe("createSidecar (fake python sidecar)", () => {
|
||||
expect(outcome).toEqual({ cancelled: false });
|
||||
expect(segments).toHaveLength(2);
|
||||
expect(segments[0]).toEqual({ start: 0, end: 1.5, text: " Hello" });
|
||||
// The fixture echoes what it received (audio bytes, language, task).
|
||||
expect(segments[1]!.text).toContain("model=ok bytes=12 lang=da task=translate");
|
||||
// The fixture echoes what it received (audio bytes, language, task, mode).
|
||||
expect(segments[1]!.text).toContain("model=ok bytes=12 lang=da task=translate mode=file");
|
||||
|
||||
const live: SidecarSegment[] = [];
|
||||
await sidecar.transcribe({ model: "ok", audioPath, mode: "live" }, (s) => live.push(s));
|
||||
expect(live[1]!.text).toContain("mode=live");
|
||||
});
|
||||
|
||||
test("sidecar 'error' lines reject that request only", async () => {
|
||||
@@ -455,6 +481,90 @@ describe("createSidecar (degradation, D-013)", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// The real server-py/sidecar.py against a stubbed faster_whisper/numpy
|
||||
// (tests/fixtures/stub_fw): request routing, model cache, cancel (D-WS5a).
|
||||
// `-S` keeps real site-packages out, so this runs the same everywhere.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe("server-py/sidecar.py (stubbed faster-whisper, D-WS5a)", () => {
|
||||
const STUB_DIR = join(import.meta.dir, "fixtures/stub_fw");
|
||||
const stubbedSidecar = () =>
|
||||
createSidecar(["env", `PYTHONPATH=${STUB_DIR}`, "python3", "-S", REAL_SIDECAR]);
|
||||
|
||||
async function run(
|
||||
sidecar: Sidecar,
|
||||
request: Omit<SidecarTranscribeRequest, "audioPath">,
|
||||
signal?: AbortSignal,
|
||||
onSegment: (segment: SidecarSegment) => void = () => {},
|
||||
): Promise<{ texts: string[]; cancelled: boolean }> {
|
||||
const audioPath = join(await makeTempDir(), "audio.f32");
|
||||
await Bun.write(audioPath, pcmBody([0, 0.5, -0.5, 0]));
|
||||
const texts: string[] = [];
|
||||
const outcome = await sidecar.transcribe(
|
||||
{ ...request, audioPath },
|
||||
(segment) => {
|
||||
texts.push(segment.text);
|
||||
onSegment(segment);
|
||||
},
|
||||
signal,
|
||||
);
|
||||
return { texts, cancelled: outcome.cancelled };
|
||||
}
|
||||
|
||||
test("file mode (and no mode) runs the batched pipeline with VAD, batch_size=8", async () => {
|
||||
const sidecar = stubbedSidecar();
|
||||
const explicit = await run(sidecar, { model: "m1", mode: "file", language: "da" });
|
||||
expect(explicit.texts).toHaveLength(1);
|
||||
expect(explicit.texts[0]).toContain("batched model=m1");
|
||||
expect(explicit.texts[0]).toContain("vad_filter=True");
|
||||
expect(explicit.texts[0]).toContain("batch_size=8");
|
||||
// Tuned Silero VAD: the default (0.5 / 400 ms) dropped quiet speech (D-WS5a).
|
||||
expect(explicit.texts[0]).toContain("vad_parameters={'threshold': 0.2, 'speech_pad_ms': 800}");
|
||||
expect(explicit.texts[0]).toContain("language=da");
|
||||
expect(explicit.texts[0]).toContain("task=transcribe");
|
||||
|
||||
const implicit = await run(sidecar, { model: "m1" });
|
||||
expect(implicit.texts[0]).toContain("batched model=m1");
|
||||
});
|
||||
|
||||
test("live mode stays sequential without VAD (short passes, D-WS5a)", async () => {
|
||||
const sidecar = stubbedSidecar();
|
||||
const { texts } = await run(sidecar, { model: "m1", mode: "live", task: "translate" });
|
||||
expect(texts[0]).toContain("seq model=m1");
|
||||
expect(texts[0]).not.toContain("vad_filter");
|
||||
expect(texts[0]).not.toContain("batch_size");
|
||||
expect(texts[0]).toContain("task=translate");
|
||||
});
|
||||
|
||||
test("one model stays loaded across modes; a switch reloads and rewraps", async () => {
|
||||
const sidecar = stubbedSidecar();
|
||||
expect((await run(sidecar, { model: "m1" })).texts[0]).toContain("loads=1");
|
||||
expect((await run(sidecar, { model: "m1", mode: "live" })).texts[0]).toContain("loads=1");
|
||||
const switched = (await run(sidecar, { model: "m2" })).texts[0];
|
||||
expect(switched).toContain("batched model=m2");
|
||||
expect(switched).toContain("loads=2");
|
||||
});
|
||||
|
||||
test("cancel lands between batched segments", async () => {
|
||||
const sidecar = stubbedSidecar();
|
||||
const controller = new AbortController();
|
||||
const { texts, cancelled } = await run(sidecar, { model: "slow" }, controller.signal, () =>
|
||||
controller.abort(),
|
||||
);
|
||||
expect(cancelled).toBe(true);
|
||||
expect(texts.length).toBeGreaterThanOrEqual(1);
|
||||
expect(texts.length).toBeLessThan(20);
|
||||
});
|
||||
|
||||
test("audio with no speech finishes with zero segments, not an error", async () => {
|
||||
const sidecar = stubbedSidecar();
|
||||
const { texts, cancelled } = await run(sidecar, { model: "silence" });
|
||||
expect(texts).toEqual([]);
|
||||
expect(cancelled).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Opt-in: real faster-whisper end to end (needs python3 + requirements.txt
|
||||
// installed and network/cached weights). See tests/server-engine.md.
|
||||
@@ -486,4 +596,119 @@ describe.skipIf(!process.env.FW_INTEGRATION)("real faster-whisper sidecar", () =
|
||||
},
|
||||
{ timeout: 300_000 },
|
||||
);
|
||||
|
||||
test(
|
||||
"file mode (batched + VAD) and live mode both transcribe real speech (D-WS5a)",
|
||||
async () => {
|
||||
const sidecar = createSidecar();
|
||||
// First 60 s of en_alien as raw float32 16 kHz mono — what the browser uploads.
|
||||
const clip = join(import.meta.dir, "audio_files/en_alien.opus");
|
||||
const pcm =
|
||||
await Bun.$`ffmpeg -v error -i ${clip} -t 60 -f f32le -ac 1 -ar 16000 -`.arrayBuffer();
|
||||
const audioPath = join(await makeTempDir(), "alien.f32");
|
||||
await Bun.write(audioPath, pcm);
|
||||
|
||||
for (const mode of ["file", "live"] as const) {
|
||||
const segments: SidecarSegment[] = [];
|
||||
const outcome = await sidecar.transcribe(
|
||||
{ model: "tiny.en", audioPath, language: "en", mode },
|
||||
(segment) => segments.push(segment),
|
||||
);
|
||||
expect(outcome.cancelled).toBe(false);
|
||||
expect(segments.length).toBeGreaterThan(0);
|
||||
const text = segments.map((segment) => segment.text).join("").toLowerCase();
|
||||
expect(text).toContain("voices of the void");
|
||||
// Timestamps stay in source-audio time after VAD drops silence.
|
||||
expect(segments.at(-1)!.end).toBeLessThanOrEqual(61);
|
||||
}
|
||||
},
|
||||
{ timeout: 300_000 },
|
||||
);
|
||||
|
||||
// The whole 15-min en_alien clip (≈ 17 VAD chunks ⇒ 3 batches) as uploaded PCM.
|
||||
async function wholeAlienPcm(): Promise<string> {
|
||||
const clip = join(import.meta.dir, "audio_files/en_alien.opus");
|
||||
const pcm = await Bun.$`ffmpeg -v error -i ${clip} -f f32le -ac 1 -ar 16000 -`.arrayBuffer();
|
||||
const audioPath = join(await makeTempDir(), "alien-full.f32");
|
||||
await Bun.write(audioPath, pcm);
|
||||
return audioPath;
|
||||
}
|
||||
|
||||
test(
|
||||
"a long file streams in bursts behind heartbeats over HTTP (D-021, D-WS5a)",
|
||||
async () => {
|
||||
const handlers = createTranscribeHandlers({
|
||||
sidecar: createSidecar(),
|
||||
getSessionUser: sessionFromCookie,
|
||||
tempDir: await makeTempDir(),
|
||||
heartbeatIntervalMs: 100,
|
||||
});
|
||||
const body = await Bun.file(await wholeAlienPcm()).arrayBuffer();
|
||||
const response = await handlers.post(
|
||||
postRequest("?model=onnx-community/whisper-tiny.en&language=en", body, {
|
||||
cookie: "sid=ok",
|
||||
}),
|
||||
);
|
||||
expect(response.status).toBe(200);
|
||||
|
||||
// Timestamp every line as it arrives.
|
||||
const started = performance.now();
|
||||
const lines: { at: number; line: string }[] = [];
|
||||
const decoder = new TextDecoder();
|
||||
let buffered = "";
|
||||
for await (const chunk of response.body!) {
|
||||
buffered += decoder.decode(chunk, { stream: true });
|
||||
let newline: number;
|
||||
while ((newline = buffered.indexOf("\n")) >= 0) {
|
||||
lines.push({ at: performance.now() - started, line: buffered.slice(0, newline) });
|
||||
buffered = buffered.slice(newline + 1);
|
||||
}
|
||||
}
|
||||
const segmentLines = lines.filter(({ line }) => line.includes('"type":"segment"'));
|
||||
const firstSegment = lines.findIndex(({ line }) => line.includes('"type":"segment"'));
|
||||
// VAD over 15 min of audio + the first batch: heartbeats fill the gap.
|
||||
expect(lines.slice(0, firstSegment).filter(({ line }) => line === "").length).toBeGreaterThan(0);
|
||||
expect(segmentLines.length).toBeGreaterThan(3);
|
||||
// Streamed, not one blob at the end: later bursts arrive measurably later.
|
||||
expect(segmentLines.at(-1)!.at - segmentLines[0]!.at).toBeGreaterThan(100);
|
||||
expect(lines.at(-1)!.line).toContain('"type":"result"');
|
||||
console.log(
|
||||
"[D-WS5a] segment arrival (ms):",
|
||||
segmentLines.map(({ at }) => Math.round(at)).join(" "),
|
||||
);
|
||||
},
|
||||
{ timeout: 600_000 },
|
||||
);
|
||||
|
||||
test(
|
||||
"cancelling a batched file run stops it early and the next request still works",
|
||||
async () => {
|
||||
const sidecar = createSidecar();
|
||||
const audioPath = await wholeAlienPcm();
|
||||
const controller = new AbortController();
|
||||
const cancelledRun: SidecarSegment[] = [];
|
||||
const outcome = await sidecar.transcribe(
|
||||
{ model: "tiny.en", audioPath, language: "en" },
|
||||
(segment) => {
|
||||
cancelledRun.push(segment);
|
||||
controller.abort();
|
||||
},
|
||||
controller.signal,
|
||||
);
|
||||
expect(outcome.cancelled).toBe(true);
|
||||
// At most the first burst (8 chunks) got out; the full clip is ≈ 17 chunks.
|
||||
expect(cancelledRun.length).toBeGreaterThan(0);
|
||||
expect(cancelledRun.length).toBeLessThanOrEqual(8);
|
||||
expect(cancelledRun.at(-1)!.end).toBeLessThan(800);
|
||||
|
||||
const again: SidecarSegment[] = [];
|
||||
const next = await sidecar.transcribe(
|
||||
{ model: "tiny.en", audioPath, language: "en" },
|
||||
(segment) => again.push(segment),
|
||||
);
|
||||
expect(next.cancelled).toBe(false);
|
||||
expect(again.length).toBeGreaterThan(cancelledRun.length);
|
||||
},
|
||||
{ timeout: 600_000 },
|
||||
);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user