From cb80ab5f9d5788cc0cdc66f9c0bcaa011ff37fea Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 00:43:51 +0100 Subject: [PATCH] Remove plan 001-3; spec 001 and D-WS3a/D-WS3b are the record Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-3-longform-pipeline.md | 246 -------------------------- 1 file changed, 246 deletions(-) delete mode 100644 docs/plans/001-3-longform-pipeline.md diff --git a/docs/plans/001-3-longform-pipeline.md b/docs/plans/001-3-longform-pipeline.md deleted file mode 100644 index 09a7a77..0000000 --- a/docs/plans/001-3-longform-pipeline.md +++ /dev/null @@ -1,246 +0,0 @@ ---- -status: in-progress ---- - -# 001-3 — Browser long-form pipeline Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents — -> coordination plan `001-0`). Steps use checkbox (`- [x]`) syntax for tracking. - -**Goal:** a model-agnostic long-form file-transcription pipeline in the inference worker -(Silero VAD → speech chunks ≤ 30 s → repetition guard → optional temperature fallback → language -pinned once), measured against the current 30 s/5 s chunked path and rolled out per the spec's rule. - -**Architecture:** `src/worker/longform/` holds pure TypeScript (`vad.ts`, `quality.ts`, -`pipeline.ts`) behind `transcribeLongForm(audio, deps, options)`; all inference is injected through -`LongFormDeps`, implemented over the existing Transformers.js runtime in `transformers.ts` (Silero -VAD as a `custom` AutoModel on the same onnxruntime, Whisper through the already-loaded ASR -pipeline). The worker chooses long-form only when asked and the VAD model loaded; otherwise the old -path runs unchanged. - -**Tech stack:** Bun 1.3, TypeScript, `@huggingface/transformers` 4.2.0 (patched), onnxruntime-node in -tests / onnxruntime-web in the browser, ffmpeg-static (tests only). - -**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 3, calls "Rollout of the new -browser pipeline", "A file-mode accuracy gate", "Measurement fallback". Coordination: -`docs/plans/001-0-coordination.md`. - -## Global Constraints - -- AGENTS.md rules in full: Bun only; no cross-origin subresources (Hub fetch is CORS like Whisper - weights); colours as tokens; TDD; docs duties in the same commit. -- Decision ids are placeholders: `D-WS3a` (pipeline + measurements + rollout), `D-WS3b` (VAD model - source, loading, offline). Cite them in code comments. -- Heavy jobs (real-model tests, full `bun test`, measurements, downloads) run under - `flock /tmp/whisper-heavy.lock`. Ports 3130–3139. No push, no publish. -- Shared interface (fixed): `transcribeLongForm(audio: Float32Array, deps: LongFormDeps, options: - LongFormOptions): Promise`; `LongFormOptions` has `language?: string`, - `task?: WhisperTask`, `prompt?: string` (passed to every decode; no-op when undefined). -- `tests/meetingFixture.ts` exports `buildMeetingFixture(): Promise<{ audio: Float32Array; - reference: string }>` — 16 kHz mono, clips joined by 2–8 s pauses, `en_alien` excerpts included; - labelled **synthetic** everywhere it is reported. -- VAD defaults to start from: `threshold 0.5`, `min_silence_duration_ms 2000`, `speech_pad_ms 400`, - `max_speech_duration_s 30`. Fallback thresholds: compression ratio > 2.4 or avg log-prob < −1.0 → - retry at temperatures 0.2, 0.4, 0.6, 0.8, 1.0. -- Rollout: long-form is the default only if on **every** measurement clip it is not worse on WER - (≤ old + 0.01 absolute) **and** faster; otherwise an "Advanced" option, off by default. Each - guard/threshold is judged the same way. Old path stays the fallback when VAD isn't cached. -- Batching out of scope; live mode keeps its own segmenter (D-016/D-024). -- Measurements are onnxruntime-node CPU numbers (machine shared with six other workstreams — run - old/new interleaved, say so). WebGPU effects → final report as "needs owner's browser check". - -## Review Focus - -1. **90-minute file** — no full-length copy of the audio per stage, no per-chunk state growing - beyond the transcript itself. Pinned in Task 4 (`90-minute input: VAD sees the caller's buffer, - decode sees ≤ 30 s chunks`). -2. **Offline / VAD never downloaded** — file transcription runs the old path with no network - request. Pinned in Task 5 (`localOnly load with an empty cache rejects without calling fetch`) - and Task 7 (`choosePipeline` falls back to `chunked`). -3. **Silence-only or music-only audio** — empty transcript, clear message, no decode, no hang. - Pinned in Task 4 (`no speech → empty result, decode never called`), Task 5 (real VAD finds no - speech in silence/tones/noise) and Task 7 (`emptyTranscriptNotice`). -4. **Auto language on a multilingual model** — Transformers.js 4.2 silently forces English when no - language is given; long-form detects once on the first speech chunk and pins it. Pinned in - Task 4 (`detectLanguage called once, result passed to every decode`) and measured on da_festuge - without a language. -5. **Correct repeated wording vs. the repetition guard** — a guard that blocks legitimate repeats - counts as a regression; judged by the WER rule in Task 6, not assumed. - ---- - -### Task 1: Pure VAD segmentation (`vad.ts`) - -**Files:** Create `src/worker/longform/vad.ts`; Test `tests/longform-vad.test.ts` - -**Interfaces — Produces:** -- `VAD_WINDOW_SAMPLES = 512`, `SAMPLE_RATE = 16_000` -- `interface VadOptions { threshold: number; negThreshold?: number; minSpeechDurationMs: number; - maxSpeechDurationS: number; minSilenceDurationMs: number; speechPadMs: number }`, - `DEFAULT_VAD_OPTIONS` (0.5, —, 0, 30, 2000, 400) -- `interface SampleSpan { start: number; end: number }` (samples, end exclusive) -- `getSpeechTimestamps(probs: ArrayLike, audioLength: number, options?: Partial): SampleSpan[]` - — line-for-line port of faster-whisper 1.2.1 `vad.py:get_speech_timestamps` after the model call. -- `interface SpeechChunk { spans: SampleSpan[]; samples: number }` and - `collectChunks(spans: SampleSpan[], maxSamples: number): SpeechChunk[]` — port of `collect_chunks` - (metadata only, never copies audio). -- `chunkAudio(audio: Float32Array, chunk: SpeechChunk): Float32Array` — one ≤ 30 s copy. -- `chunkTimeToOriginal(chunk: SpeechChunk, seconds: number): number` — chunk-relative → absolute - seconds (piecewise over spans; a time exactly on a splice maps to the earlier span's end). - -- [x] Tests: probs all < 0.35 → `[]`; one burst (windows 10–40 at 0.9) → one span padded by 6400 - samples each side, clamped to [0, len]; two bursts 1 s apart (< 2 s min silence) → one span; two - bursts 3 s apart → two spans, pad split when gap < 2·pad; continuous speech of 70 s with - `maxSpeechDurationS 30` → spans ≤ 30 s; `collectChunks` packs 12 s+12 s+12 s into [24 s, 12 s]; - `chunkTimeToOriginal` maps 0 → first span start and a time past span 1's length into span 2; - `chunkAudio` length equals `chunk.samples`. -- [x] Run `bun test tests/longform-vad.test.ts` → FAIL (module missing); implement; → PASS. -- [x] Commit "Add pure VAD speech-timestamp and chunking functions for the long-form pipeline". - -### Task 2: Progress tracker for variable windows - -**Files:** Modify `src/worker/protocol.ts` (`TranscribeProgressTracker`); Test `tests/protocol.test.ts` - -**Interfaces — Produces:** `TranscribeProgressTracker.forWindows(totalSeconds: number, windows: -ProgressWindow[]): TranscribeProgressTracker` with `interface ProgressWindow { toAbsolute(seconds: -number): number; endSeconds: number; nextStartSeconds: number }`. `timestamp()` maps through the -current window's `toAbsolute`; `finalizeWindow()` advances to that window's `nextStartSeconds` -(skipped silence counts as processed; last window → total). Existing fixed-stride constructor and -its tests unchanged. - -- [x] Tests: windows `[0–20 s]`, `[35–60 s]` (with a gap): `timestamp(5)` → 5; `finalizeWindow()` → - 35; `timestamp(10)` in window 2 → 45; regressions swallowed; final `finalizeWindow()` → total; - timestamps beyond a window clamp to its `endSeconds`. -- [x] FAIL → implement → PASS → commit "Let the transcribe progress tracker follow variable-length windows". - -### Task 3: Quality checks (`quality.ts`) - -**Files:** Create `src/worker/longform/quality.ts`; Test `tests/longform-quality.test.ts` - -**Interfaces — Produces:** `compressionRatio(text: string): Promise` (UTF-8 bytes / zlib -`CompressionStream("deflate")` bytes, = faster-whisper `get_compression_ratio`); -`DEFAULT_FALLBACK = { temperatures: [0, 0.2, 0.4, 0.6, 0.8, 1.0], compressionRatioThreshold: 2.4, -logProbThreshold: -1.0 }`; `needsFallback(result: { compressionRatio: number; avgLogProb: number | -null }, thresholds): boolean` (null log-prob never triggers). - -- [x] Tests: a 40× repeated phrase → ratio > 2.4 and `needsFallback` true; a normal English - sentence → ratio < 2.4; avgLogProb −1.5 → true; −0.3 and null → false. -- [x] FAIL → implement → PASS → commit "Add compression-ratio and log-prob quality checks for decode fallback". - -### Task 4: `transcribeLongForm` (pure orchestration) - -**Files:** Create `src/worker/longform/types.ts`, `pipeline.ts`, `index.ts`; Test `tests/longform-pipeline.test.ts` - -**Interfaces — Produces (types.ts):** -```ts -interface DecodeRequest { language?: string; task?: WhisperTask; prompt?: string; temperature: number; - noRepeatNgramSize: number; repetitionPenalty: number; - onPartial?(text: string): void; onTimestamp?(chunkSeconds: number): void } -interface DecodeResult { text: string; chunks: TranscriptChunk[] /* chunk-relative */; - generatedTokens: number; avgLogProb: number | null } -interface LongFormDeps { - speechProbabilities(audio: Float32Array): Promise; // one per 512-sample window - decode(audio: Float32Array, request: DecodeRequest): Promise; - detectLanguage?(audio: Float32Array): Promise; // absent ⇒ English-only model -} -interface LongFormOptions { language?: string; task?: WhisperTask; prompt?: string; - vad?: Partial; guard?: { noRepeatNgramSize: number; repetitionPenalty: number }; - fallback?: Partial | false; - onPartial?(text: string): void; onProgress?(processedSeconds: number, totalSeconds: number): void } -interface LongFormResult { text: string; chunks: TranscriptChunk[] /* absolute */; language?: string; - noSpeech: boolean; speechSeconds: number; decodeCalls: number; generatedTokens: number } -``` -`DEFAULT_GUARD` and the default `fallback` are set from Task 6's measurements (start: guard -`{ noRepeatNgramSize: 0, repetitionPenalty: 1 }`, fallback off). - -- [x] Tests (fake deps; probs built from a speech mask): - - `no speech → empty result, decode never called` (`noSpeech: true`, text "", progress reaches total). - - two speech regions 10 s apart → two decodes, chunk timestamps mapped to absolute seconds. - - `language "auto"/undefined + detectLanguage → called once with the first chunk, its result - passed to every decode`; explicit `language` → detect never called; no `detectLanguage` dep → - no language sent. - - `task` and `prompt` reach every `DecodeRequest`; guard values reach every request. - - fallback: first decode returns a 40× repeated text → retried at 0.2 and the retry's text kept; - every temperature failing → the lowest-compression attempt kept; `fallback: false` → one decode. - - `onProgress` is monotone and ends at total; `onPartial` forwards decode partials. - - `90-minute input: VAD sees the caller's buffer, decode sees ≤ 30 s chunks` — 5 400 s - Float32Array, speech mask alternating 20 s speech / 5 s silence; assert - `speechProbabilities` got the identical object, every decode input ≤ 480 000 samples, decode - calls ≈ speech/≤30 s, finishes < 5 s. -- [x] FAIL → implement (`pipeline.ts`: VAD → spans → `collectChunks(…, 30 s)` → per chunk: - `chunkAudio`, decode with fallback loop, map chunks, progress via `forWindows`) → PASS. -- [x] Commit "Add the long-form transcription pipeline behind transcribeLongForm". - -### Task 5: Transformers.js adapters (`transformers.ts`) - -**Files:** Create `src/worker/longform/transformers.ts`; Test `tests/longform-transformers.test.ts` - -**Interfaces — Produces:** -- `VAD_MODEL_ID = "onnx-community/silero-vad"` (MIT, `onnx/model.onnx` fp32, 2.24 MB) — D-WS3b. -- `loadSileroVad(options: { localOnly: boolean; device?: "wasm" | "cpu" }): Promise`; - `SileroVad.speechProbabilities(audio)` runs 512-sample windows with 64 samples of context and the - `[2,1,128]` state (official v5 wrapper), reusing one input buffer. -- `createWhisperDeps(asr: AutomaticSpeechRecognitionPipeline, options: { multilingual: boolean }): - Pick` — `decode` calls `asr(chunk, {return_timestamps: - true, max_new_tokens: 440, language, task, no_repeat_ngram_size, repetition_penalty, - do_sample: temperature > 0, temperature, streamer, logits_processor})` with a fresh logits list - whose last processor records log-probs and token count; `detectLanguage` runs one decoder step - from `<|startoftranscript|>` with all non-language logits masked. -- [x] Tests: `localOnly load with an empty cache rejects without calling fetch` (temp `env.cacheDir`, - `globalThis.fetch` spy). Real-VAD tests (skip if neither cached nor reachable, download under the - lock): 10 s of zeros, a 440 Hz tone and white noise → `getSpeechTimestamps` returns `[]`; - `en_aiproblem` → spans covering ≥ 80 % of its speech. -- [x] FAIL → implement → PASS → commit "Run Silero VAD and Whisper decodes for the long-form pipeline on Transformers.js". - -### Task 6: Meeting fixture, measurement, and tuning - -**Files:** Create `tests/meetingFixture.ts`, `tests/clips.ts` (ffmpeg decode to 16 kHz + WER -normalize), `scripts/measure-longform.ts`. - -- [x] `buildMeetingFixture()`: 2 s lead-in, en_aiproblem, then three en_alien excerpts cut at - reference-line boundaries (times read once from a timestamped turbo transcript, hard-coded with - their line ranges), pauses `[3, 8, 2, 5]` s filled with deterministic −60 dBFS noise. Test - (fast suite, skips without ffmpeg): length = sum of parts, reference non-empty, deterministic. -- [x] `scripts/measure-longform.ts [--clips …] [--config name]` prints per clip: wall-clock, WER, - generated tokens, decode calls — for `old` (worker's current options) and long-form configs. - Clips: en_alien, da_festuge with `da`, da_festuge auto, en_aiproblem, meeting (synthetic). -- [x] (one baseline + one long-form run per clip; guard/fallback variants not run, both stay off — D-WS3a) Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants - (`no_repeat_ngram_size 3/5`, `repetition_penalty 1.1`); fallback on if loops remain; VAD - `min_silence 2000` vs `160`. Choose defaults by the rollout rule; record the table in D-WS3a. -- [x] Commit "Add the synthetic meeting fixture and long-form measurement script". - -### Task 7: Worker, protocol, engine and UI wiring - -**Files:** Modify `src/worker/protocol.ts`, `src/worker/inference.worker.ts`, `src/lib/engine.ts`, -`src/App.tsx`, `src/components/TranscriptionOptions.tsx` (or a small `AdvancedOptions`), -`src/components/TranscriptView.tsx`; Tests `tests/protocol.test.ts`, `tests/engine.test.ts`. - -**Interfaces — Produces:** `LoadMessage.vad?: boolean` (load Silero alongside Whisper; offline ⇒ -cache only); `TranscribeMessage.longForm?: boolean`; pure `choosePipeline({ longForm, vadReady }): -"longform" | "chunked"`; `EngineTranscribeOptions.longForm?: boolean`; -`emptyTranscriptNotice(result): string | null` → "No speech was found in this recording." for an -empty finished transcript. -- [x] Tests: guards accept/reject `vad`/`longForm` types; `choosePipeline` → chunked when VAD not - ready or not requested; BrowserEngine forwards `longForm`; `emptyTranscriptNotice`. -- [x] Worker: after the Whisper pipeline is ready, if `vad` load Silero (`localOnly: - navigator.onLine === false`), failures logged and leave `vadReady` false; transcribe dispatches - per `choosePipeline`; long-form posts the same `partial` / `transcribe-progress` / `result`. -- [x] UI per Task 6's rollout outcome: default-on, or an "Advanced" disclosure with a plain-language - checkbox (off by default, remembered in localStorage) that also sets `vad` on loads. -- [x] typecheck, fast suite, build → commit "Run browser file transcription through the long-form pipeline when the VAD model is available". - -### Task 8: File-mode accuracy gate - -**Files:** Modify `tests/live-real-model.test.ts` (one new `describe` block at the end). -- [x] Per clip: whole-file WER of the default file path (long-form with real deps if default, else - old options) ≤ `MAX_WER_BY_CLIP[clip] ?? MAX_WER`; long-form additionally ≤ old + 0.01 if it is - the default. -- [x] Run under the lock → PASS → commit "Assert whole-file WER per clip in the real-model test". - -### Task 9: Docs and finish - -- [x] `docs/Decisions.md`: append `D-WS3a` (pipeline, measurement table labelled synthetic where - so, rollout decision) and `D-WS3b` (VAD model, loading, offline, live kept on the energy gate). -- [x] `docs/Architecture.md` §3 layout + §4.1 flow (long-form branch) + §4.2 offline note; - `docs/WorkBreakdown.md` ticks + "Noticed, not fixed"; `tests/offline.md` manual VAD step. -- [x] typecheck, fast suite, build; real-model test under the lock; `git rm` this plan; commit.