247 lines
16 KiB
Markdown
247 lines
16 KiB
Markdown
---
|
||
status: in-progress
|
||
---
|
||
|
||
# 001-3 — Browser long-form pipeline Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no subagents —
|
||
> coordination plan `001-0`). Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** a model-agnostic long-form file-transcription pipeline in the inference worker
|
||
(Silero VAD → speech chunks ≤ 30 s → repetition guard → optional temperature fallback → language
|
||
pinned once), measured against the current 30 s/5 s chunked path and rolled out per the spec's rule.
|
||
|
||
**Architecture:** `src/worker/longform/` holds pure TypeScript (`vad.ts`, `quality.ts`,
|
||
`pipeline.ts`) behind `transcribeLongForm(audio, deps, options)`; all inference is injected through
|
||
`LongFormDeps`, implemented over the existing Transformers.js runtime in `transformers.ts` (Silero
|
||
VAD as a `custom` AutoModel on the same onnxruntime, Whisper through the already-loaded ASR
|
||
pipeline). The worker chooses long-form only when asked and the VAD model loaded; otherwise the old
|
||
path runs unchanged.
|
||
|
||
**Tech stack:** Bun 1.3, TypeScript, `@huggingface/transformers` 4.2.0 (patched), onnxruntime-node in
|
||
tests / onnxruntime-web in the browser, ffmpeg-static (tests only).
|
||
|
||
**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 3, calls "Rollout of the new
|
||
browser pipeline", "A file-mode accuracy gate", "Measurement fallback". Coordination:
|
||
`docs/plans/001-0-coordination.md`.
|
||
|
||
## Global Constraints
|
||
|
||
- AGENTS.md rules in full: Bun only; no cross-origin subresources (Hub fetch is CORS like Whisper
|
||
weights); colours as tokens; TDD; docs duties in the same commit.
|
||
- Decision ids are placeholders: `D-WS3a` (pipeline + measurements + rollout), `D-WS3b` (VAD model
|
||
source, loading, offline). Cite them in code comments.
|
||
- Heavy jobs (real-model tests, full `bun test`, measurements, downloads) run under
|
||
`flock /tmp/whisper-heavy.lock`. Ports 3130–3139. No push, no publish.
|
||
- Shared interface (fixed): `transcribeLongForm(audio: Float32Array, deps: LongFormDeps, options:
|
||
LongFormOptions): Promise<LongFormResult>`; `LongFormOptions` has `language?: string`,
|
||
`task?: WhisperTask`, `prompt?: string` (passed to every decode; no-op when undefined).
|
||
- `tests/meetingFixture.ts` exports `buildMeetingFixture(): Promise<{ audio: Float32Array;
|
||
reference: string }>` — 16 kHz mono, clips joined by 2–8 s pauses, `en_alien` excerpts included;
|
||
labelled **synthetic** everywhere it is reported.
|
||
- VAD defaults to start from: `threshold 0.5`, `min_silence_duration_ms 2000`, `speech_pad_ms 400`,
|
||
`max_speech_duration_s 30`. Fallback thresholds: compression ratio > 2.4 or avg log-prob < −1.0 →
|
||
retry at temperatures 0.2, 0.4, 0.6, 0.8, 1.0.
|
||
- Rollout: long-form is the default only if on **every** measurement clip it is not worse on WER
|
||
(≤ old + 0.01 absolute) **and** faster; otherwise an "Advanced" option, off by default. Each
|
||
guard/threshold is judged the same way. Old path stays the fallback when VAD isn't cached.
|
||
- Batching out of scope; live mode keeps its own segmenter (D-016/D-024).
|
||
- Measurements are onnxruntime-node CPU numbers (machine shared with six other workstreams — run
|
||
old/new interleaved, say so). WebGPU effects → final report as "needs owner's browser check".
|
||
|
||
## Review Focus
|
||
|
||
1. **90-minute file** — no full-length copy of the audio per stage, no per-chunk state growing
|
||
beyond the transcript itself. Pinned in Task 4 (`90-minute input: VAD sees the caller's buffer,
|
||
decode sees ≤ 30 s chunks`).
|
||
2. **Offline / VAD never downloaded** — file transcription runs the old path with no network
|
||
request. Pinned in Task 5 (`localOnly load with an empty cache rejects without calling fetch`)
|
||
and Task 7 (`choosePipeline` falls back to `chunked`).
|
||
3. **Silence-only or music-only audio** — empty transcript, clear message, no decode, no hang.
|
||
Pinned in Task 4 (`no speech → empty result, decode never called`), Task 5 (real VAD finds no
|
||
speech in silence/tones/noise) and Task 7 (`emptyTranscriptNotice`).
|
||
4. **Auto language on a multilingual model** — Transformers.js 4.2 silently forces English when no
|
||
language is given; long-form detects once on the first speech chunk and pins it. Pinned in
|
||
Task 4 (`detectLanguage called once, result passed to every decode`) and measured on da_festuge
|
||
without a language.
|
||
5. **Correct repeated wording vs. the repetition guard** — a guard that blocks legitimate repeats
|
||
counts as a regression; judged by the WER rule in Task 6, not assumed.
|
||
|
||
---
|
||
|
||
### Task 1: Pure VAD segmentation (`vad.ts`)
|
||
|
||
**Files:** Create `src/worker/longform/vad.ts`; Test `tests/longform-vad.test.ts`
|
||
|
||
**Interfaces — Produces:**
|
||
- `VAD_WINDOW_SAMPLES = 512`, `SAMPLE_RATE = 16_000`
|
||
- `interface VadOptions { threshold: number; negThreshold?: number; minSpeechDurationMs: number;
|
||
maxSpeechDurationS: number; minSilenceDurationMs: number; speechPadMs: number }`,
|
||
`DEFAULT_VAD_OPTIONS` (0.5, —, 0, 30, 2000, 400)
|
||
- `interface SampleSpan { start: number; end: number }` (samples, end exclusive)
|
||
- `getSpeechTimestamps(probs: ArrayLike<number>, audioLength: number, options?: Partial<VadOptions>): SampleSpan[]`
|
||
— line-for-line port of faster-whisper 1.2.1 `vad.py:get_speech_timestamps` after the model call.
|
||
- `interface SpeechChunk { spans: SampleSpan[]; samples: number }` and
|
||
`collectChunks(spans: SampleSpan[], maxSamples: number): SpeechChunk[]` — port of `collect_chunks`
|
||
(metadata only, never copies audio).
|
||
- `chunkAudio(audio: Float32Array, chunk: SpeechChunk): Float32Array` — one ≤ 30 s copy.
|
||
- `chunkTimeToOriginal(chunk: SpeechChunk, seconds: number): number` — chunk-relative → absolute
|
||
seconds (piecewise over spans; a time exactly on a splice maps to the earlier span's end).
|
||
|
||
- [ ] Tests: probs all < 0.35 → `[]`; one burst (windows 10–40 at 0.9) → one span padded by 6400
|
||
samples each side, clamped to [0, len]; two bursts 1 s apart (< 2 s min silence) → one span; two
|
||
bursts 3 s apart → two spans, pad split when gap < 2·pad; continuous speech of 70 s with
|
||
`maxSpeechDurationS 30` → spans ≤ 30 s; `collectChunks` packs 12 s+12 s+12 s into [24 s, 12 s];
|
||
`chunkTimeToOriginal` maps 0 → first span start and a time past span 1's length into span 2;
|
||
`chunkAudio` length equals `chunk.samples`.
|
||
- [ ] Run `bun test tests/longform-vad.test.ts` → FAIL (module missing); implement; → PASS.
|
||
- [ ] Commit "Add pure VAD speech-timestamp and chunking functions for the long-form pipeline".
|
||
|
||
### Task 2: Progress tracker for variable windows
|
||
|
||
**Files:** Modify `src/worker/protocol.ts` (`TranscribeProgressTracker`); Test `tests/protocol.test.ts`
|
||
|
||
**Interfaces — Produces:** `TranscribeProgressTracker.forWindows(totalSeconds: number, windows:
|
||
ProgressWindow[]): TranscribeProgressTracker` with `interface ProgressWindow { toAbsolute(seconds:
|
||
number): number; endSeconds: number; nextStartSeconds: number }`. `timestamp()` maps through the
|
||
current window's `toAbsolute`; `finalizeWindow()` advances to that window's `nextStartSeconds`
|
||
(skipped silence counts as processed; last window → total). Existing fixed-stride constructor and
|
||
its tests unchanged.
|
||
|
||
- [ ] Tests: windows `[0–20 s]`, `[35–60 s]` (with a gap): `timestamp(5)` → 5; `finalizeWindow()` →
|
||
35; `timestamp(10)` in window 2 → 45; regressions swallowed; final `finalizeWindow()` → total;
|
||
timestamps beyond a window clamp to its `endSeconds`.
|
||
- [ ] FAIL → implement → PASS → commit "Let the transcribe progress tracker follow variable-length windows".
|
||
|
||
### Task 3: Quality checks (`quality.ts`)
|
||
|
||
**Files:** Create `src/worker/longform/quality.ts`; Test `tests/longform-quality.test.ts`
|
||
|
||
**Interfaces — Produces:** `compressionRatio(text: string): Promise<number>` (UTF-8 bytes / zlib
|
||
`CompressionStream("deflate")` bytes, = faster-whisper `get_compression_ratio`);
|
||
`DEFAULT_FALLBACK = { temperatures: [0, 0.2, 0.4, 0.6, 0.8, 1.0], compressionRatioThreshold: 2.4,
|
||
logProbThreshold: -1.0 }`; `needsFallback(result: { compressionRatio: number; avgLogProb: number |
|
||
null }, thresholds): boolean` (null log-prob never triggers).
|
||
|
||
- [ ] Tests: a 40× repeated phrase → ratio > 2.4 and `needsFallback` true; a normal English
|
||
sentence → ratio < 2.4; avgLogProb −1.5 → true; −0.3 and null → false.
|
||
- [ ] FAIL → implement → PASS → commit "Add compression-ratio and log-prob quality checks for decode fallback".
|
||
|
||
### Task 4: `transcribeLongForm` (pure orchestration)
|
||
|
||
**Files:** Create `src/worker/longform/types.ts`, `pipeline.ts`, `index.ts`; Test `tests/longform-pipeline.test.ts`
|
||
|
||
**Interfaces — Produces (types.ts):**
|
||
```ts
|
||
interface DecodeRequest { language?: string; task?: WhisperTask; prompt?: string; temperature: number;
|
||
noRepeatNgramSize: number; repetitionPenalty: number;
|
||
onPartial?(text: string): void; onTimestamp?(chunkSeconds: number): void }
|
||
interface DecodeResult { text: string; chunks: TranscriptChunk[] /* chunk-relative */;
|
||
generatedTokens: number; avgLogProb: number | null }
|
||
interface LongFormDeps {
|
||
speechProbabilities(audio: Float32Array): Promise<Float32Array>; // one per 512-sample window
|
||
decode(audio: Float32Array, request: DecodeRequest): Promise<DecodeResult>;
|
||
detectLanguage?(audio: Float32Array): Promise<string>; // absent ⇒ English-only model
|
||
}
|
||
interface LongFormOptions { language?: string; task?: WhisperTask; prompt?: string;
|
||
vad?: Partial<VadOptions>; guard?: { noRepeatNgramSize: number; repetitionPenalty: number };
|
||
fallback?: Partial<typeof DEFAULT_FALLBACK> | false;
|
||
onPartial?(text: string): void; onProgress?(processedSeconds: number, totalSeconds: number): void }
|
||
interface LongFormResult { text: string; chunks: TranscriptChunk[] /* absolute */; language?: string;
|
||
noSpeech: boolean; speechSeconds: number; decodeCalls: number; generatedTokens: number }
|
||
```
|
||
`DEFAULT_GUARD` and the default `fallback` are set from Task 6's measurements (start: guard
|
||
`{ noRepeatNgramSize: 0, repetitionPenalty: 1 }`, fallback off).
|
||
|
||
- [ ] Tests (fake deps; probs built from a speech mask):
|
||
- `no speech → empty result, decode never called` (`noSpeech: true`, text "", progress reaches total).
|
||
- two speech regions 10 s apart → two decodes, chunk timestamps mapped to absolute seconds.
|
||
- `language "auto"/undefined + detectLanguage → called once with the first chunk, its result
|
||
passed to every decode`; explicit `language` → detect never called; no `detectLanguage` dep →
|
||
no language sent.
|
||
- `task` and `prompt` reach every `DecodeRequest`; guard values reach every request.
|
||
- fallback: first decode returns a 40× repeated text → retried at 0.2 and the retry's text kept;
|
||
every temperature failing → the lowest-compression attempt kept; `fallback: false` → one decode.
|
||
- `onProgress` is monotone and ends at total; `onPartial` forwards decode partials.
|
||
- `90-minute input: VAD sees the caller's buffer, decode sees ≤ 30 s chunks` — 5 400 s
|
||
Float32Array, speech mask alternating 20 s speech / 5 s silence; assert
|
||
`speechProbabilities` got the identical object, every decode input ≤ 480 000 samples, decode
|
||
calls ≈ speech/≤30 s, finishes < 5 s.
|
||
- [ ] FAIL → implement (`pipeline.ts`: VAD → spans → `collectChunks(…, 30 s)` → per chunk:
|
||
`chunkAudio`, decode with fallback loop, map chunks, progress via `forWindows`) → PASS.
|
||
- [ ] Commit "Add the long-form transcription pipeline behind transcribeLongForm".
|
||
|
||
### Task 5: Transformers.js adapters (`transformers.ts`)
|
||
|
||
**Files:** Create `src/worker/longform/transformers.ts`; Test `tests/longform-transformers.test.ts`
|
||
|
||
**Interfaces — Produces:**
|
||
- `VAD_MODEL_ID = "onnx-community/silero-vad"` (MIT, `onnx/model.onnx` fp32, 2.24 MB) — D-WS3b.
|
||
- `loadSileroVad(options: { localOnly: boolean; device?: "wasm" | "cpu" }): Promise<SileroVad>`;
|
||
`SileroVad.speechProbabilities(audio)` runs 512-sample windows with 64 samples of context and the
|
||
`[2,1,128]` state (official v5 wrapper), reusing one input buffer.
|
||
- `createWhisperDeps(asr: AutomaticSpeechRecognitionPipeline, options: { multilingual: boolean }):
|
||
Pick<LongFormDeps, "decode" | "detectLanguage">` — `decode` calls `asr(chunk, {return_timestamps:
|
||
true, max_new_tokens: 440, language, task, no_repeat_ngram_size, repetition_penalty,
|
||
do_sample: temperature > 0, temperature, streamer, logits_processor})` with a fresh logits list
|
||
whose last processor records log-probs and token count; `detectLanguage` runs one decoder step
|
||
from `<|startoftranscript|>` with all non-language logits masked.
|
||
- [ ] Tests: `localOnly load with an empty cache rejects without calling fetch` (temp `env.cacheDir`,
|
||
`globalThis.fetch` spy). Real-VAD tests (skip if neither cached nor reachable, download under the
|
||
lock): 10 s of zeros, a 440 Hz tone and white noise → `getSpeechTimestamps` returns `[]`;
|
||
`en_aiproblem` → spans covering ≥ 80 % of its speech.
|
||
- [ ] FAIL → implement → PASS → commit "Run Silero VAD and Whisper decodes for the long-form pipeline on Transformers.js".
|
||
|
||
### Task 6: Meeting fixture, measurement, and tuning
|
||
|
||
**Files:** Create `tests/meetingFixture.ts`, `tests/clips.ts` (ffmpeg decode to 16 kHz + WER
|
||
normalize), `scripts/measure-longform.ts`.
|
||
|
||
- [ ] `buildMeetingFixture()`: 2 s lead-in, en_aiproblem, then three en_alien excerpts cut at
|
||
reference-line boundaries (times read once from a timestamped turbo transcript, hard-coded with
|
||
their line ranges), pauses `[3, 8, 2, 5]` s filled with deterministic −60 dBFS noise. Test
|
||
(fast suite, skips without ffmpeg): length = sum of parts, reference non-empty, deterministic.
|
||
- [ ] `scripts/measure-longform.ts [--clips …] [--config name]` prints per clip: wall-clock, WER,
|
||
generated tokens, decode calls — for `old` (worker's current options) and long-form configs.
|
||
Clips: en_alien, da_festuge with `da`, da_festuge auto, en_aiproblem, meeting (synthetic).
|
||
- [ ] Run under the lock, old/new interleaved: baseline; long-form guard off; guard variants
|
||
(`no_repeat_ngram_size 3/5`, `repetition_penalty 1.1`); fallback on if loops remain; VAD
|
||
`min_silence 2000` vs `160`. Choose defaults by the rollout rule; record the table in D-WS3a.
|
||
- [ ] Commit "Add the synthetic meeting fixture and long-form measurement script".
|
||
|
||
### Task 7: Worker, protocol, engine and UI wiring
|
||
|
||
**Files:** Modify `src/worker/protocol.ts`, `src/worker/inference.worker.ts`, `src/lib/engine.ts`,
|
||
`src/App.tsx`, `src/components/TranscriptionOptions.tsx` (or a small `AdvancedOptions`),
|
||
`src/components/TranscriptView.tsx`; Tests `tests/protocol.test.ts`, `tests/engine.test.ts`.
|
||
|
||
**Interfaces — Produces:** `LoadMessage.vad?: boolean` (load Silero alongside Whisper; offline ⇒
|
||
cache only); `TranscribeMessage.longForm?: boolean`; pure `choosePipeline({ longForm, vadReady }):
|
||
"longform" | "chunked"`; `EngineTranscribeOptions.longForm?: boolean`;
|
||
`emptyTranscriptNotice(result): string | null` → "No speech was found in this recording." for an
|
||
empty finished transcript.
|
||
- [ ] Tests: guards accept/reject `vad`/`longForm` types; `choosePipeline` → chunked when VAD not
|
||
ready or not requested; BrowserEngine forwards `longForm`; `emptyTranscriptNotice`.
|
||
- [ ] Worker: after the Whisper pipeline is ready, if `vad` load Silero (`localOnly:
|
||
navigator.onLine === false`), failures logged and leave `vadReady` false; transcribe dispatches
|
||
per `choosePipeline`; long-form posts the same `partial` / `transcribe-progress` / `result`.
|
||
- [ ] UI per Task 6's rollout outcome: default-on, or an "Advanced" disclosure with a plain-language
|
||
checkbox (off by default, remembered in localStorage) that also sets `vad` on loads.
|
||
- [ ] typecheck, fast suite, build → commit "Run browser file transcription through the long-form pipeline when the VAD model is available".
|
||
|
||
### Task 8: File-mode accuracy gate
|
||
|
||
**Files:** Modify `tests/live-real-model.test.ts` (one new `describe` block at the end).
|
||
- [ ] Per clip: whole-file WER of the default file path (long-form with real deps if default, else
|
||
old options) ≤ `MAX_WER_BY_CLIP[clip] ?? MAX_WER`; long-form additionally ≤ old + 0.01 if it is
|
||
the default.
|
||
- [ ] Run under the lock → PASS → commit "Assert whole-file WER per clip in the real-model test".
|
||
|
||
### Task 9: Docs and finish
|
||
|
||
- [ ] `docs/Decisions.md`: append `D-WS3a` (pipeline, measurement table labelled synthetic where
|
||
so, rollout decision) and `D-WS3b` (VAD model, loading, offline, live kept on the energy gate).
|
||
- [ ] `docs/Architecture.md` §3 layout + §4.1 flow (long-form branch) + §4.2 offline note;
|
||
`docs/WorkBreakdown.md` ticks + "Noticed, not fixed"; `tests/offline.md` manual VAD step.
|
||
- [ ] typecheck, fast suite, build; real-model test under the lock; `git rm` this plan; commit.
|