whisper_web/docs/plans/001-3-longform-pipeline.md

status

status
in-progress

001-3 — Browser long-form pipeline Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — coordination plan 001-0). Steps use checkbox (- [ ]) syntax for tracking.

Goal: a model-agnostic long-form file-transcription pipeline in the inference worker (Silero VAD → speech chunks ≤ 30 s → repetition guard → optional temperature fallback → language pinned once), measured against the current 30 s/5 s chunked path and rolled out per the spec's rule.

Architecture: src/worker/longform/ holds pure TypeScript (vad.ts, quality.ts, pipeline.ts) behind transcribeLongForm(audio, deps, options); all inference is injected through LongFormDeps, implemented over the existing Transformers.js runtime in transformers.ts (Silero VAD as a custom AutoModel on the same onnxruntime, Whisper through the already-loaded ASR pipeline). The worker chooses long-form only when asked and the VAD model loaded; otherwise the old path runs unchanged.

Tech stack: Bun 1.3, TypeScript, @huggingface/transformers 4.2.0 (patched), onnxruntime-node in tests / onnxruntime-web in the browser, ffmpeg-static (tests only).

Spec: docs/specs/001-transcription-quality-and-speed.md Goal 3, calls "Rollout of the new browser pipeline", "A file-mode accuracy gate", "Measurement fallback". Coordination: docs/plans/001-0-coordination.md.

Global Constraints

• AGENTS.md rules in full: Bun only; no cross-origin subresources (Hub fetch is CORS like Whisper weights); colours as tokens; TDD; docs duties in the same commit.
• Decision ids are placeholders: D-WS3a (pipeline + measurements + rollout), D-WS3b (VAD model source, loading, offline). Cite them in code comments.
• Heavy jobs (real-model tests, full bun test, measurements, downloads) run under flock /tmp/whisper-heavy.lock. Ports 3130–3139. No push, no publish.
• Shared interface (fixed): transcribeLongForm(audio: Float32Array, deps: LongFormDeps, options: LongFormOptions): Promise<LongFormResult>; LongFormOptions has language?: string, task?: WhisperTask, prompt?: string (passed to every decode; no-op when undefined).
• tests/meetingFixture.ts exports buildMeetingFixture(): Promise<{ audio: Float32Array; reference: string }> — 16 kHz mono, clips joined by 2–8 s pauses, en_alien excerpts included; labelled synthetic everywhere it is reported.
• VAD defaults to start from: threshold 0.5, min_silence_duration_ms 2000, speech_pad_ms 400, max_speech_duration_s 30. Fallback thresholds: compression ratio > 2.4 or avg log-prob < −1.0 → retry at temperatures 0.2, 0.4, 0.6, 0.8, 1.0.
• Rollout: long-form is the default only if on every measurement clip it is not worse on WER (≤ old + 0.01 absolute) and faster; otherwise an "Advanced" option, off by default. Each guard/threshold is judged the same way. Old path stays the fallback when VAD isn't cached.
• Batching out of scope; live mode keeps its own segmenter (D-016/D-024).
• Measurements are onnxruntime-node CPU numbers (machine shared with six other workstreams — run old/new interleaved, say so). WebGPU effects → final report as "needs owner's browser check".

Review Focus

1. 90-minute file — no full-length copy of the audio per stage, no per-chunk state growing beyond the transcript itself. Pinned in Task 4 (90-minute input: VAD sees the caller's buffer, decode sees ≤ 30 s chunks).
2. Offline / VAD never downloaded — file transcription runs the old path with no network request. Pinned in Task 5 (localOnly load with an empty cache rejects without calling fetch) and Task 7 (choosePipeline falls back to chunked).
3. Silence-only or music-only audio — empty transcript, clear message, no decode, no hang. Pinned in Task 4 (no speech → empty result, decode never called), Task 5 (real VAD finds no speech in silence/tones/noise) and Task 7 (emptyTranscriptNotice).
4. Auto language on a multilingual model — Transformers.js 4.2 silently forces English when no language is given; long-form detects once on the first speech chunk and pins it. Pinned in Task 4 (detectLanguage called once, result passed to every decode) and measured on da_festuge without a language.
5. Correct repeated wording vs. the repetition guard — a guard that blocks legitimate repeats counts as a regression; judged by the WER rule in Task 6, not assumed.

---

Task 1: Pure VAD segmentation (vad.ts)

Files: Create src/worker/longform/vad.ts; Test tests/longform-vad.test.ts

Interfaces — Produces:

• VAD_WINDOW_SAMPLES = 512, SAMPLE_RATE = 16_000

• interface VadOptions { threshold: number; negThreshold?: number; minSpeechDurationMs: number; maxSpeechDurationS: number; minSilenceDurationMs: number; speechPadMs: number }, DEFAULT_VAD_OPTIONS (0.5, —, 0, 30, 2000, 400)

• interface SampleSpan { start: number; end: number } (samples, end exclusive)

• getSpeechTimestamps(probs: ArrayLike<number>, audioLength: number, options?: Partial<VadOptions>): SampleSpan[] — line-for-line port of faster-whisper 1.2.1 vad.py:get_speech_timestamps after the model call.

• interface SpeechChunk { spans: SampleSpan[]; samples: number } and collectChunks(spans: SampleSpan[], maxSamples: number): SpeechChunk[] — port of collect_chunks (metadata only, never copies audio).

• chunkAudio(audio: Float32Array, chunk: SpeechChunk): Float32Array — one ≤ 30 s copy.

• chunkTimeToOriginal(chunk: SpeechChunk, seconds: number): number — chunk-relative → absolute seconds (piecewise over spans; a time exactly on a splice maps to the earlier span's end).

• Tests: probs all < 0.35 → []; one burst (windows 10–40 at 0.9) → one span padded by 6400 samples each side, clamped to [0, len]; two bursts 1 s apart (< 2 s min silence) → one span; two bursts 3 s apart → two spans, pad split when gap < 2·pad; continuous speech of 70 s with maxSpeechDurationS 30 → spans ≤ 30 s; collectChunks packs 12 s+12 s+12 s into [24 s, 12 s]; chunkTimeToOriginal maps 0 → first span start and a time past span 1's length into span 2; chunkAudio length equals chunk.samples.

• Run bun test tests/longform-vad.test.ts → FAIL (module missing); implement; → PASS.

• Commit "Add pure VAD speech-timestamp and chunking functions for the long-form pipeline".

Task 2: Progress tracker for variable windows

Files: Modify src/worker/protocol.ts (TranscribeProgressTracker); Test tests/protocol.test.ts

Interfaces — Produces: TranscribeProgressTracker.forWindows(totalSeconds: number, windows: ProgressWindow[]): TranscribeProgressTracker with interface ProgressWindow { toAbsolute(seconds: number): number; endSeconds: number; nextStartSeconds: number }. timestamp() maps through the current window's toAbsolute; finalizeWindow() advances to that window's nextStartSeconds (skipped silence counts as processed; last window → total). Existing fixed-stride constructor and its tests unchanged.

• Tests: windows [0–20 s], [35–60 s] (with a gap): timestamp(5) → 5; finalizeWindow() → 35; timestamp(10) in window 2 → 45; regressions swallowed; final finalizeWindow() → total; timestamps beyond a window clamp to its endSeconds.
• FAIL → implement → PASS → commit "Let the transcribe progress tracker follow variable-length windows".

Task 3: Quality checks (quality.ts)

Files: Create src/worker/longform/quality.ts; Test tests/longform-quality.test.ts

Interfaces — Produces: compressionRatio(text: string): Promise<number> (UTF-8 bytes / zlib CompressionStream("deflate") bytes, = faster-whisper get_compression_ratio); DEFAULT_FALLBACK = { temperatures: [0, 0.2, 0.4, 0.6, 0.8, 1.0], compressionRatioThreshold: 2.4, logProbThreshold: -1.0 }; needsFallback(result: { compressionRatio: number; avgLogProb: number | null }, thresholds): boolean (null log-prob never triggers).

• Tests: a 40× repeated phrase → ratio > 2.4 and needsFallback true; a normal English sentence → ratio < 2.4; avgLogProb −1.5 → true; −0.3 and null → false.
• FAIL → implement → PASS → commit "Add compression-ratio and log-prob quality checks for decode fallback".

Task 4: transcribeLongForm (pure orchestration)

Files: Create src/worker/longform/types.ts, pipeline.ts, index.ts; Test tests/longform-pipeline.test.ts

Interfaces — Produces (types.ts):

interface DecodeRequest { language?: string; task?: WhisperTask; prompt?: string; temperature: number;
  noRepeatNgramSize: number; repetitionPenalty: number;
  onPartial?(text: string): void; onTimestamp?(chunkSeconds: number): void }
interface DecodeResult { text: string; chunks: TranscriptChunk[] /* chunk-relative */;
  generatedTokens: number; avgLogProb: number | null }
interface LongFormDeps {
  speechProbabilities(audio: Float32Array): Promise<Float32Array>; // one per 512-sample window
  decode(audio: Float32Array, request: DecodeRequest): Promise<DecodeResult>;
  detectLanguage?(audio: Float32Array): Promise<string>; // absent ⇒ English-only model
}
interface LongFormOptions { language?: string; task?: WhisperTask; prompt?: string;
  vad?: Partial<VadOptions>; guard?: { noRepeatNgramSize: number; repetitionPenalty: number };
  fallback?: Partial<typeof DEFAULT_FALLBACK> | false;
  onPartial?(text: string): void; onProgress?(processedSeconds: number, totalSeconds: number): void }
interface LongFormResult { text: string; chunks: TranscriptChunk[] /* absolute */; language?: string;
  noSpeech: boolean; speechSeconds: number; decodeCalls: number; generatedTokens: number }

DEFAULT_GUARD and the default fallback are set from Task 6's measurements (start: guard { noRepeatNgramSize: 0, repetitionPenalty: 1 }, fallback off).

• Tests (fake deps; probs built from a speech mask):
  • no speech → empty result, decode never called (noSpeech: true, text "", progress reaches total).
  • two speech regions 10 s apart → two decodes, chunk timestamps mapped to absolute seconds.
  • language "auto"/undefined + detectLanguage → called once with the first chunk, its result passed to every decode; explicit language → detect never called; no detectLanguage dep → no language sent.
  • task and prompt reach every DecodeRequest; guard values reach every request.
  • fallback: first decode returns a 40× repeated text → retried at 0.2 and the retry's text kept; every temperature failing → the lowest-compression attempt kept; fallback: false → one decode.
  • onProgress is monotone and ends at total; onPartial forwards decode partials.
  • 90-minute input: VAD sees the caller's buffer, decode sees ≤ 30 s chunks — 5 400 s Float32Array, speech mask alternating 20 s speech / 5 s silence; assert speechProbabilities got the identical object, every decode input ≤ 480 000 samples, decode calls ≈ speech/≤30 s, finishes < 5 s.
• FAIL → implement (pipeline.ts: VAD → spans → collectChunks(…, 30 s) → per chunk: chunkAudio, decode with fallback loop, map chunks, progress via forWindows) → PASS.
• Commit "Add the long-form transcription pipeline behind transcribeLongForm".

Task 5: Transformers.js adapters (transformers.ts)

Files: Create src/worker/longform/transformers.ts; Test tests/longform-transformers.test.ts

Interfaces — Produces:

• VAD_MODEL_ID = "onnx-community/silero-vad" (MIT, onnx/model.onnx fp32, 2.24 MB) — D-WS3b.
• loadSileroVad(options: { localOnly: boolean; device?: "wasm" | "cpu" }): Promise<SileroVad>; SileroVad.speechProbabilities(audio) runs 512-sample windows with 64 samples of context and the [2,1,128] state (official v5 wrapper), reusing one input buffer.
• createWhisperDeps(asr: AutomaticSpeechRecognitionPipeline, options: { multilingual: boolean }): Pick<LongFormDeps, "decode" | "detectLanguage"> — decode calls asr(chunk, {return_timestamps: true, max_new_tokens: 440, language, task, no_repeat_ngram_size, repetition_penalty, do_sample: temperature > 0, temperature, streamer, logits_processor}) with a fresh logits list whose last processor records log-probs and token count; detectLanguage runs one decoder step from <|startoftranscript|> with all non-language logits masked.
• Tests: localOnly load with an empty cache rejects without calling fetch (temp env.cacheDir, globalThis.fetch spy). Real-VAD tests (skip if neither cached nor reachable, download under the lock): 10 s of zeros, a 440 Hz tone and white noise → getSpeechTimestamps returns []; en_aiproblem → spans covering ≥ 80 % of its speech.
• FAIL → implement → PASS → commit "Run Silero VAD and Whisper decodes for the long-form pipeline on Transformers.js".

Task 6: Meeting fixture, measurement, and tuning

Files: Create tests/meetingFixture.ts, tests/clips.ts (ffmpeg decode to 16 kHz + WER normalize), scripts/measure-longform.ts.

• buildMeetingFixture(): 2 s lead-in, en_aiproblem, then three en_alien excerpts cut at reference-line boundaries (times read once from a timestamped turbo transcript, hard-coded with their line ranges), pauses [3, 8, 2, 5] s filled with deterministic −60 dBFS noise. Test (fast suite, skips without ffmpeg): length = sum of parts, reference non-empty, deterministic.
• scripts/measure-longform.ts [--clips …] [--config name] prints per clip: wall-clock, WER, generated tokens, decode calls — for old (worker's current options) and long-form configs. Clips: en_alien, da_festuge with da, da_festuge auto, en_aiproblem, meeting (synthetic).
• Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants (no_repeat_ngram_size 3/5, repetition_penalty 1.1); fallback on if loops remain; VAD min_silence 2000 vs 160. Choose defaults by the rollout rule; record the table in D-WS3a.
• Commit "Add the synthetic meeting fixture and long-form measurement script".

Task 7: Worker, protocol, engine and UI wiring

Files: Modify src/worker/protocol.ts, src/worker/inference.worker.ts, src/lib/engine.ts, src/App.tsx, src/components/TranscriptionOptions.tsx (or a small AdvancedOptions), src/components/TranscriptView.tsx; Tests tests/protocol.test.ts, tests/engine.test.ts.

Interfaces — Produces: LoadMessage.vad?: boolean (load Silero alongside Whisper; offline ⇒ cache only); TranscribeMessage.longForm?: boolean; pure choosePipeline({ longForm, vadReady }): "longform" | "chunked"; EngineTranscribeOptions.longForm?: boolean; emptyTranscriptNotice(result): string | null → "No speech was found in this recording." for an empty finished transcript.

• Tests: guards accept/reject vad/longForm types; choosePipeline → chunked when VAD not ready or not requested; BrowserEngine forwards longForm; emptyTranscriptNotice.
• Worker: after the Whisper pipeline is ready, if vad load Silero (localOnly: navigator.onLine === false), failures logged and leave vadReady false; transcribe dispatches per choosePipeline; long-form posts the same partial / transcribe-progress / result.
• UI per Task 6's rollout outcome: default-on, or an "Advanced" disclosure with a plain-language checkbox (off by default, remembered in localStorage) that also sets vad on loads.
• typecheck, fast suite, build → commit "Run browser file transcription through the long-form pipeline when the VAD model is available".

Task 8: File-mode accuracy gate

Files: Modify tests/live-real-model.test.ts (one new describe block at the end).

• Per clip: whole-file WER of the default file path (long-form with real deps if default, else old options) ≤ MAX_WER_BY_CLIP[clip] ?? MAX_WER; long-form additionally ≤ old + 0.01 if it is the default.
• Run under the lock → PASS → commit "Assert whole-file WER per clip in the real-model test".

Task 9: Docs and finish

• docs/Decisions.md: append D-WS3a (pipeline, measurement table labelled synthetic where so, rollout decision) and D-WS3b (VAD model, loading, offline, live kept on the energy gate).
• docs/Architecture.md §3 layout + §4.1 flow (long-form branch) + §4.2 offline note; docs/WorkBreakdown.md ticks + "Noticed, not fixed"; tests/offline.md manual VAD step.
• typecheck, fast suite, build; real-model test under the lock; git rm this plan; commit.