diff --git a/docs/plans/001-3-longform-pipeline.md b/docs/plans/001-3-longform-pipeline.md new file mode 100644 index 0000000..108f87d --- /dev/null +++ b/docs/plans/001-3-longform-pipeline.md @@ -0,0 +1,246 @@ +--- +status: in-progress +--- + +# 001-3 — Browser long-form pipeline Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — +> coordination plan `001-0`). Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** a model-agnostic long-form file-transcription pipeline in the inference worker +(Silero VAD → speech chunks ≤ 30 s → repetition guard → optional temperature fallback → language +pinned once), measured against the current 30 s/5 s chunked path and rolled out per the spec's rule. + +**Architecture:** `src/worker/longform/` holds pure TypeScript (`vad.ts`, `quality.ts`, +`pipeline.ts`) behind `transcribeLongForm(audio, deps, options)`; all inference is injected through +`LongFormDeps`, implemented over the existing Transformers.js runtime in `transformers.ts` (Silero +VAD as a `custom` AutoModel on the same onnxruntime, Whisper through the already-loaded ASR +pipeline). The worker chooses long-form only when asked and the VAD model loaded; otherwise the old +path runs unchanged. + +**Tech stack:** Bun 1.3, TypeScript, `@huggingface/transformers` 4.2.0 (patched), onnxruntime-node in +tests / onnxruntime-web in the browser, ffmpeg-static (tests only). + +**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 3, calls "Rollout of the new +browser pipeline", "A file-mode accuracy gate", "Measurement fallback". Coordination: +`docs/plans/001-0-coordination.md`. + +## Global Constraints + +- AGENTS.md rules in full: Bun only; no cross-origin subresources (Hub fetch is CORS like Whisper + weights); colours as tokens; TDD; docs duties in the same commit. +- Decision ids are placeholders: `D-WS3a` (pipeline + measurements + rollout), `D-WS3b` (VAD model + source, loading, offline). Cite them in code comments. +- Heavy jobs (real-model tests, full `bun test`, measurements, downloads) run under + `flock /tmp/whisper-heavy.lock`. Ports 3130–3139. No push, no publish. +- Shared interface (fixed): `transcribeLongForm(audio: Float32Array, deps: LongFormDeps, options: + LongFormOptions): Promise`; `LongFormOptions` has `language?: string`, + `task?: WhisperTask`, `prompt?: string` (passed to every decode; no-op when undefined). +- `tests/meetingFixture.ts` exports `buildMeetingFixture(): Promise<{ audio: Float32Array; + reference: string }>` — 16 kHz mono, clips joined by 2–8 s pauses, `en_alien` excerpts included; + labelled **synthetic** everywhere it is reported. +- VAD defaults to start from: `threshold 0.5`, `min_silence_duration_ms 2000`, `speech_pad_ms 400`, + `max_speech_duration_s 30`. Fallback thresholds: compression ratio > 2.4 or avg log-prob < −1.0 → + retry at temperatures 0.2, 0.4, 0.6, 0.8, 1.0. +- Rollout: long-form is the default only if on **every** measurement clip it is not worse on WER + (≤ old + 0.01 absolute) **and** faster; otherwise an "Advanced" option, off by default. Each + guard/threshold is judged the same way. Old path stays the fallback when VAD isn't cached. +- Batching out of scope; live mode keeps its own segmenter (D-016/D-024). +- Measurements are onnxruntime-node CPU numbers (machine shared with six other workstreams — run + old/new interleaved, say so). WebGPU effects → final report as "needs owner's browser check". + +## Review Focus + +1. **90-minute file** — no full-length copy of the audio per stage, no per-chunk state growing + beyond the transcript itself. Pinned in Task 4 (`90-minute input: VAD sees the caller's buffer, + decode sees ≤ 30 s chunks`). +2. **Offline / VAD never downloaded** — file transcription runs the old path with no network + request. Pinned in Task 5 (`localOnly load with an empty cache rejects without calling fetch`) + and Task 7 (`choosePipeline` falls back to `chunked`). +3. **Silence-only or music-only audio** — empty transcript, clear message, no decode, no hang. + Pinned in Task 4 (`no speech → empty result, decode never called`), Task 5 (real VAD finds no + speech in silence/tones/noise) and Task 7 (`emptyTranscriptNotice`). +4. **Auto language on a multilingual model** — Transformers.js 4.2 silently forces English when no + language is given; long-form detects once on the first speech chunk and pins it. Pinned in + Task 4 (`detectLanguage called once, result passed to every decode`) and measured on da_festuge + without a language. +5. **Correct repeated wording vs. the repetition guard** — a guard that blocks legitimate repeats + counts as a regression; judged by the WER rule in Task 6, not assumed. + +--- + +### Task 1: Pure VAD segmentation (`vad.ts`) + +**Files:** Create `src/worker/longform/vad.ts`; Test `tests/longform-vad.test.ts` + +**Interfaces — Produces:** +- `VAD_WINDOW_SAMPLES = 512`, `SAMPLE_RATE = 16_000` +- `interface VadOptions { threshold: number; negThreshold?: number; minSpeechDurationMs: number; + maxSpeechDurationS: number; minSilenceDurationMs: number; speechPadMs: number }`, + `DEFAULT_VAD_OPTIONS` (0.5, —, 0, 30, 2000, 400) +- `interface SampleSpan { start: number; end: number }` (samples, end exclusive) +- `getSpeechTimestamps(probs: ArrayLike, audioLength: number, options?: Partial): SampleSpan[]` + — line-for-line port of faster-whisper 1.2.1 `vad.py:get_speech_timestamps` after the model call. +- `interface SpeechChunk { spans: SampleSpan[]; samples: number }` and + `collectChunks(spans: SampleSpan[], maxSamples: number): SpeechChunk[]` — port of `collect_chunks` + (metadata only, never copies audio). +- `chunkAudio(audio: Float32Array, chunk: SpeechChunk): Float32Array` — one ≤ 30 s copy. +- `chunkTimeToOriginal(chunk: SpeechChunk, seconds: number): number` — chunk-relative → absolute + seconds (piecewise over spans; a time exactly on a splice maps to the earlier span's end). + +- [ ] Tests: probs all < 0.35 → `[]`; one burst (windows 10–40 at 0.9) → one span padded by 6400 + samples each side, clamped to [0, len]; two bursts 1 s apart (< 2 s min silence) → one span; two + bursts 3 s apart → two spans, pad split when gap < 2·pad; continuous speech of 70 s with + `maxSpeechDurationS 30` → spans ≤ 30 s; `collectChunks` packs 12 s+12 s+12 s into [24 s, 12 s]; + `chunkTimeToOriginal` maps 0 → first span start and a time past span 1's length into span 2; + `chunkAudio` length equals `chunk.samples`. +- [ ] Run `bun test tests/longform-vad.test.ts` → FAIL (module missing); implement; → PASS. +- [ ] Commit "Add pure VAD speech-timestamp and chunking functions for the long-form pipeline". + +### Task 2: Progress tracker for variable windows + +**Files:** Modify `src/worker/protocol.ts` (`TranscribeProgressTracker`); Test `tests/protocol.test.ts` + +**Interfaces — Produces:** `TranscribeProgressTracker.forWindows(totalSeconds: number, windows: +ProgressWindow[]): TranscribeProgressTracker` with `interface ProgressWindow { toAbsolute(seconds: +number): number; endSeconds: number; nextStartSeconds: number }`. `timestamp()` maps through the +current window's `toAbsolute`; `finalizeWindow()` advances to that window's `nextStartSeconds` +(skipped silence counts as processed; last window → total). Existing fixed-stride constructor and +its tests unchanged. + +- [ ] Tests: windows `[0–20 s]`, `[35–60 s]` (with a gap): `timestamp(5)` → 5; `finalizeWindow()` → + 35; `timestamp(10)` in window 2 → 45; regressions swallowed; final `finalizeWindow()` → total; + timestamps beyond a window clamp to its `endSeconds`. +- [ ] FAIL → implement → PASS → commit "Let the transcribe progress tracker follow variable-length windows". + +### Task 3: Quality checks (`quality.ts`) + +**Files:** Create `src/worker/longform/quality.ts`; Test `tests/longform-quality.test.ts` + +**Interfaces — Produces:** `compressionRatio(text: string): Promise` (UTF-8 bytes / zlib +`CompressionStream("deflate")` bytes, = faster-whisper `get_compression_ratio`); +`DEFAULT_FALLBACK = { temperatures: [0, 0.2, 0.4, 0.6, 0.8, 1.0], compressionRatioThreshold: 2.4, +logProbThreshold: -1.0 }`; `needsFallback(result: { compressionRatio: number; avgLogProb: number | +null }, thresholds): boolean` (null log-prob never triggers). + +- [ ] Tests: a 40× repeated phrase → ratio > 2.4 and `needsFallback` true; a normal English + sentence → ratio < 2.4; avgLogProb −1.5 → true; −0.3 and null → false. +- [ ] FAIL → implement → PASS → commit "Add compression-ratio and log-prob quality checks for decode fallback". + +### Task 4: `transcribeLongForm` (pure orchestration) + +**Files:** Create `src/worker/longform/types.ts`, `pipeline.ts`, `index.ts`; Test `tests/longform-pipeline.test.ts` + +**Interfaces — Produces (types.ts):** +```ts +interface DecodeRequest { language?: string; task?: WhisperTask; prompt?: string; temperature: number; + noRepeatNgramSize: number; repetitionPenalty: number; + onPartial?(text: string): void; onTimestamp?(chunkSeconds: number): void } +interface DecodeResult { text: string; chunks: TranscriptChunk[] /* chunk-relative */; + generatedTokens: number; avgLogProb: number | null } +interface LongFormDeps { + speechProbabilities(audio: Float32Array): Promise; // one per 512-sample window + decode(audio: Float32Array, request: DecodeRequest): Promise; + detectLanguage?(audio: Float32Array): Promise; // absent ⇒ English-only model +} +interface LongFormOptions { language?: string; task?: WhisperTask; prompt?: string; + vad?: Partial; guard?: { noRepeatNgramSize: number; repetitionPenalty: number }; + fallback?: Partial | false; + onPartial?(text: string): void; onProgress?(processedSeconds: number, totalSeconds: number): void } +interface LongFormResult { text: string; chunks: TranscriptChunk[] /* absolute */; language?: string; + noSpeech: boolean; speechSeconds: number; decodeCalls: number; generatedTokens: number } +``` +`DEFAULT_GUARD` and the default `fallback` are set from Task 6's measurements (start: guard +`{ noRepeatNgramSize: 0, repetitionPenalty: 1 }`, fallback off). + +- [ ] Tests (fake deps; probs built from a speech mask): + - `no speech → empty result, decode never called` (`noSpeech: true`, text "", progress reaches total). + - two speech regions 10 s apart → two decodes, chunk timestamps mapped to absolute seconds. + - `language "auto"/undefined + detectLanguage → called once with the first chunk, its result + passed to every decode`; explicit `language` → detect never called; no `detectLanguage` dep → + no language sent. + - `task` and `prompt` reach every `DecodeRequest`; guard values reach every request. + - fallback: first decode returns a 40× repeated text → retried at 0.2 and the retry's text kept; + every temperature failing → the lowest-compression attempt kept; `fallback: false` → one decode. + - `onProgress` is monotone and ends at total; `onPartial` forwards decode partials. + - `90-minute input: VAD sees the caller's buffer, decode sees ≤ 30 s chunks` — 5 400 s + Float32Array, speech mask alternating 20 s speech / 5 s silence; assert + `speechProbabilities` got the identical object, every decode input ≤ 480 000 samples, decode + calls ≈ speech/≤30 s, finishes < 5 s. +- [ ] FAIL → implement (`pipeline.ts`: VAD → spans → `collectChunks(…, 30 s)` → per chunk: + `chunkAudio`, decode with fallback loop, map chunks, progress via `forWindows`) → PASS. +- [ ] Commit "Add the long-form transcription pipeline behind transcribeLongForm". + +### Task 5: Transformers.js adapters (`transformers.ts`) + +**Files:** Create `src/worker/longform/transformers.ts`; Test `tests/longform-transformers.test.ts` + +**Interfaces — Produces:** +- `VAD_MODEL_ID = "onnx-community/silero-vad"` (MIT, `onnx/model.onnx` fp32, 2.24 MB) — D-WS3b. +- `loadSileroVad(options: { localOnly: boolean; device?: "wasm" | "cpu" }): Promise`; + `SileroVad.speechProbabilities(audio)` runs 512-sample windows with 64 samples of context and the + `[2,1,128]` state (official v5 wrapper), reusing one input buffer. +- `createWhisperDeps(asr: AutomaticSpeechRecognitionPipeline, options: { multilingual: boolean }): + Pick` — `decode` calls `asr(chunk, {return_timestamps: + true, max_new_tokens: 440, language, task, no_repeat_ngram_size, repetition_penalty, + do_sample: temperature > 0, temperature, streamer, logits_processor})` with a fresh logits list + whose last processor records log-probs and token count; `detectLanguage` runs one decoder step + from `<|startoftranscript|>` with all non-language logits masked. +- [ ] Tests: `localOnly load with an empty cache rejects without calling fetch` (temp `env.cacheDir`, + `globalThis.fetch` spy). Real-VAD tests (skip if neither cached nor reachable, download under the + lock): 10 s of zeros, a 440 Hz tone and white noise → `getSpeechTimestamps` returns `[]`; + `en_aiproblem` → spans covering ≥ 80 % of its speech. +- [ ] FAIL → implement → PASS → commit "Run Silero VAD and Whisper decodes for the long-form pipeline on Transformers.js". + +### Task 6: Meeting fixture, measurement, and tuning + +**Files:** Create `tests/meetingFixture.ts`, `tests/clips.ts` (ffmpeg decode to 16 kHz + WER +normalize), `scripts/measure-longform.ts`. + +- [ ] `buildMeetingFixture()`: 2 s lead-in, en_aiproblem, then three en_alien excerpts cut at + reference-line boundaries (times read once from a timestamped turbo transcript, hard-coded with + their line ranges), pauses `[3, 8, 2, 5]` s filled with deterministic −60 dBFS noise. Test + (fast suite, skips without ffmpeg): length = sum of parts, reference non-empty, deterministic. +- [ ] `scripts/measure-longform.ts [--clips …] [--config name]` prints per clip: wall-clock, WER, + generated tokens, decode calls — for `old` (worker's current options) and long-form configs. + Clips: en_alien, da_festuge with `da`, da_festuge auto, en_aiproblem, meeting (synthetic). +- [ ] Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants + (`no_repeat_ngram_size 3/5`, `repetition_penalty 1.1`); fallback on if loops remain; VAD + `min_silence 2000` vs `160`. Choose defaults by the rollout rule; record the table in D-WS3a. +- [ ] Commit "Add the synthetic meeting fixture and long-form measurement script". + +### Task 7: Worker, protocol, engine and UI wiring + +**Files:** Modify `src/worker/protocol.ts`, `src/worker/inference.worker.ts`, `src/lib/engine.ts`, +`src/App.tsx`, `src/components/TranscriptionOptions.tsx` (or a small `AdvancedOptions`), +`src/components/TranscriptView.tsx`; Tests `tests/protocol.test.ts`, `tests/engine.test.ts`. + +**Interfaces — Produces:** `LoadMessage.vad?: boolean` (load Silero alongside Whisper; offline ⇒ +cache only); `TranscribeMessage.longForm?: boolean`; pure `choosePipeline({ longForm, vadReady }): +"longform" | "chunked"`; `EngineTranscribeOptions.longForm?: boolean`; +`emptyTranscriptNotice(result): string | null` → "No speech was found in this recording." for an +empty finished transcript. +- [ ] Tests: guards accept/reject `vad`/`longForm` types; `choosePipeline` → chunked when VAD not + ready or not requested; BrowserEngine forwards `longForm`; `emptyTranscriptNotice`. +- [ ] Worker: after the Whisper pipeline is ready, if `vad` load Silero (`localOnly: + navigator.onLine === false`), failures logged and leave `vadReady` false; transcribe dispatches + per `choosePipeline`; long-form posts the same `partial` / `transcribe-progress` / `result`. +- [ ] UI per Task 6's rollout outcome: default-on, or an "Advanced" disclosure with a plain-language + checkbox (off by default, remembered in localStorage) that also sets `vad` on loads. +- [ ] typecheck, fast suite, build → commit "Run browser file transcription through the long-form pipeline when the VAD model is available". + +### Task 8: File-mode accuracy gate + +**Files:** Modify `tests/live-real-model.test.ts` (one new `describe` block at the end). +- [ ] Per clip: whole-file WER of the default file path (long-form with real deps if default, else + old options) ≤ `MAX_WER_BY_CLIP[clip] ?? MAX_WER`; long-form additionally ≤ old + 0.01 if it is + the default. +- [ ] Run under the lock → PASS → commit "Assert whole-file WER per clip in the real-model test". + +### Task 9: Docs and finish + +- [ ] `docs/Decisions.md`: append `D-WS3a` (pipeline, measurement table labelled synthetic where + so, rollout decision) and `D-WS3b` (VAD model, loading, offline, live kept on the energy gate). +- [ ] `docs/Architecture.md` §3 layout + §4.1 flow (long-form branch) + §4.2 offline note; + `docs/WorkBreakdown.md` ticks + "Noticed, not fixed"; `tests/offline.md` manual VAD step. +- [ ] typecheck, fast suite, build; real-model test under the lock; `git rm` this plan; commit.