diff --git a/docs/Decisions.md b/docs/Decisions.md index 199b036..76e2ffa 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1033,3 +1033,96 @@ the worker/mic stack, which the project's live tests (`tests/live.test.ts`, segment/scheduling logic in `src/lib/live.ts` instead. Manual verification via `tests/live.md` remains the intended coverage for this UI path, and its permission-flow step was updated accordingly (see Consequence above). + +## D-035 — `whisper-tiny` (multilingual) "slower than whisper-tiny.en" is real, but only on out-of-distribution audio, and it is a decode-length effect, not an asset/config gap +**Status:** Investigated, no code fix warranted (2026-07-30) +Reported symptom: `onnx-community/whisper-tiny` (multilingual) feels "a good +bit slower" than `onnx-community/whisper-tiny.en` at inference. Three +candidate mechanisms were checked, in order, using the +`Symbol.for("onnxruntime")` ort-web-under-Bun harness (real onnxruntime-web +wasm EP, q8 dtype — the catalog's WASM default for both models, D-006) driven +through the exact pipeline call `src/worker/inference.worker.ts` makes +(`chunk_length_s: 30, stride_length_s: 5, return_timestamps: true, +max_new_tokens: 440`, D-021's single-pass path), single-threaded to avoid +cross-process CPU contention (a real hazard in this sandbox: two unrelated +sibling-agent `bun test` runs at 500%+ CPU each inflated early timings 3–5× +until killed — matches the pre-existing "ort-web under Bun harness" tooling +note about contention). +1. **Hub asset parity — ruled out.** `curl`'d the Hub tree API directly (not + just `WebFetch`'s HTML summary) for both repos' `onnx/` directories: both + ship an equivalent `decoder_model_merged_quantized.onnx` (merged, + KV-cache-enabled) at nearly identical size — multilingual 30,719,241 B vs + `.en` 30,718,858 B (383 B apart, consistent with one extra vocab row: + 51865 vs 51864 tokens). No fallback to a non-merged/non-cached decoder for + either model. +2. **Catalog/dtype config — ruled out** (per the task brief, already checked + before this investigation): both catalog entries resolve to the same + `wasm`/`webgpu`/`webnn` dtypes; the `verified` flag difference is + metadata-only. +3. **Actual measured timing — this is where the real, reproducible effect + is, and it's content-dependent:** + - On `tests/audio_files/en_aiproblem.m4a` (14.4 s, clean English speech, + both models transcribe it correctly): no reliable difference. + Interleaved back-to-back runs (both orders tried, to rule out + first-vs-second bias) gave multilingual/`.en` warm-average ratios of + 0.938 and 0.891 — i.e. multilingual was *slightly faster*, well within + run-to-run noise (individual pass times ranged 1150–1460 ms across + both models). The "extra 1–2 prompt tokens" mechanism the task brief + had already flagged as probably-negligible is confirmed negligible here. + - On `tests/audio_files/da_festuge.m4a` (21.5 s, Danish speech, no + `language` option passed — the real default when a user leaves the UI's + language picker on "auto", exactly as `src/App.tsx` sends it): **both + models mistranscribe it as English and fall into a greedy-decoding + repetition loop** (a known Whisper failure mode on audio that doesn't + match the assumed language, not specific to this app), but the two + checkpoints' loops differ hugely in how many tokens they run before + landing on `eos_token_id`: + - `whisper-tiny.en`: 67 generated tokens (`" The first one is the first + one. ... "` repeating), pass time 2.6–5.7 s. + - `whisper-tiny` (multilingual): 434 generated tokens — essentially the + entire `max_new_tokens: 440` budget — (`" ... the main clar, the main + clar, ..."` repeating), pass time 9.5–14 s. + - Ratio: ~6.5× more tokens, ~2–5× wall-clock (sub-linear vs. token count + because KV-cache growth cost isn't the whole per-token cost; still a + large, user-visible gap). A second same-order and a reversed-order + run reproduced the same order-of-magnitude split. +**Mechanism:** this is not a compute-cost-per-token difference (ruled out by +the clean-English measurement) and not an asset/config gap (ruled out above). +It's that the multilingual model's forced decoder prompt carries an extra +task token (`<|transcribe|>`, from its `forced_decoder_ids`) that `.en`'s +prompt doesn't have, and — on audio where the model is already +out-of-distribution enough to hallucinate — that different prompt context +measurably changes *how long the hallucination loop runs before EOS*, not +just the compute cost of the prompt tokens themselves. Because this app's +`max_new_tokens: 440` (D-021) forces the classic single-pass path with no +`no_repeat_ngram_size`/`repetition_penalty` set in either model's +`generation_config.json` or in `inference.worker.ts`'s pipeline options, +there is nothing to cut a non-terminating decode short before the hard +token cap — so whichever checkpoint happens to run closer to that cap on a +given input is dramatically slower for that one pass. Confirmed generation +configs for both models differ only in the multilingual-specific fields +(`lang_to_id`, `task_to_id`, an extra forced token at position 2) — neither +sets a repetition guard. +**Fix:** none applied. This is inherent behavior of the two upstream +checkpoints reacting differently to the same repetition-loop failure mode, +not a bug in this app's catalog, config, or asset selection — there is +nothing in `src/lib/models.ts` to change (both already resolve to identical, +merged, cached-KV decoders at the same dtype on every device tier). Inventing +a per-model workaround (e.g. steering multilingual `.en`-adjacent use cases +onto the `.en` model) was rejected: it would silently break real multilingual +transcription for users who need it, to paper over a worst-case latency tail +on inputs the model was always going to mistranscribe anyway. +**Scope:** tiny/tiny.en only measured; the same forced-decoder-ids asymmetry +exists for every multilingual/`.en` pair in the catalog (D-006), so the same +content-dependent latency tail plausibly applies to base/small/medium too, +untested here. +**Revisit:** if this becomes a recurring complaint (not just a one-off +report), the actionable mitigation is adding `no_repeat_ngram_size` (e.g. 3) +and/or a `repetition_penalty` to the pipeline options in +`inference.worker.ts` — that targets the actual failure mode (unbounded +repetition loops on out-of-distribution audio) for both `.en` and +multilingual models, rather than papering over one symptom of it. That +change needs its own accuracy verification against +`tests/live-accuracy.test.ts` and `tests/live-real-model.test.ts` (a +repetition guard can change correct-transcript wording too) before landing, +which is out of scope for this investigation.