Document D-035: whisper-tiny multilingual vs .en speed gap is a content-dependent hallucination-loop effect, not a config bug

Measured with the ort-web-under-Bun harness: no reliable speed difference on
clean, language-matched audio. On out-of-distribution audio (wrong/undetected
language), the multilingual model's extra forced task token measurably
changes how long its repetition-loop hallucination runs before EOS (434 vs 67
tokens on the da_festuge fixture), which is where the reported slowness comes
from. Both models resolve to identical, merged, KV-cache-enabled decoders at
the same dtype — no catalog/config fix applies here. Documents the mechanism
and flags a future no_repeat_ngram_size/repetition_penalty mitigation as the
actionable follow-up if this recurs.

docs/Decisions.md

@@ -1033,3 +1033,96 @@ the worker/mic stack, which the project's live tests (`tests/live.test.ts`,
 segment/scheduling logic in `src/lib/live.ts` instead. Manual verification
 via `tests/live.md` remains the intended coverage for this UI path, and its
 permission-flow step was updated accordingly (see Consequence above).
+
+## D-035 — `whisper-tiny` (multilingual) "slower than whisper-tiny.en" is real, but only on out-of-distribution audio, and it is a decode-length effect, not an asset/config gap
+**Status:** Investigated, no code fix warranted (2026-07-30)
+Reported symptom: `onnx-community/whisper-tiny` (multilingual) feels "a good
+bit slower" than `onnx-community/whisper-tiny.en` at inference. Three
+candidate mechanisms were checked, in order, using the
+`Symbol.for("onnxruntime")` ort-web-under-Bun harness (real onnxruntime-web
+wasm EP, q8 dtype — the catalog's WASM default for both models, D-006) driven
+through the exact pipeline call `src/worker/inference.worker.ts` makes
+(`chunk_length_s: 30, stride_length_s: 5, return_timestamps: true,
+max_new_tokens: 440`, D-021's single-pass path), single-threaded to avoid
+cross-process CPU contention (a real hazard in this sandbox: two unrelated
+sibling-agent `bun test` runs at 500%+ CPU each inflated early timings 3–5×
+until killed — matches the pre-existing "ort-web under Bun harness" tooling
+note about contention).
+1. **Hub asset parity — ruled out.** `curl`'d the Hub tree API directly (not
+   just `WebFetch`'s HTML summary) for both repos' `onnx/` directories: both
+   ship an equivalent `decoder_model_merged_quantized.onnx` (merged,
+   KV-cache-enabled) at nearly identical size — multilingual 30,719,241 B vs
+   `.en` 30,718,858 B (383 B apart, consistent with one extra vocab row:
+   51865 vs 51864 tokens). No fallback to a non-merged/non-cached decoder for
+   either model.
+2. **Catalog/dtype config — ruled out** (per the task brief, already checked
+   before this investigation): both catalog entries resolve to the same
+   `wasm`/`webgpu`/`webnn` dtypes; the `verified` flag difference is
+   metadata-only.
+3. **Actual measured timing — this is where the real, reproducible effect
+   is, and it's content-dependent:**
+   - On `tests/audio_files/en_aiproblem.m4a` (14.4 s, clean English speech,
+     both models transcribe it correctly): no reliable difference.
+     Interleaved back-to-back runs (both orders tried, to rule out
+     first-vs-second bias) gave multilingual/`.en` warm-average ratios of
+     0.938 and 0.891 — i.e. multilingual was *slightly faster*, well within
+     run-to-run noise (individual pass times ranged 1150–1460 ms across
+     both models). The "extra 1–2 prompt tokens" mechanism the task brief
+     had already flagged as probably-negligible is confirmed negligible here.
+   - On `tests/audio_files/da_festuge.m4a` (21.5 s, Danish speech, no
+     `language` option passed — the real default when a user leaves the UI's
+     language picker on "auto", exactly as `src/App.tsx` sends it): **both
+     models mistranscribe it as English and fall into a greedy-decoding
+     repetition loop** (a known Whisper failure mode on audio that doesn't
+     match the assumed language, not specific to this app), but the two
+     checkpoints' loops differ hugely in how many tokens they run before
+     landing on `eos_token_id`:
+     - `whisper-tiny.en`: 67 generated tokens (`" The first one is the first
+       one. ... "` repeating), pass time 2.6–5.7 s.
+     - `whisper-tiny` (multilingual): 434 generated tokens — essentially the
+       entire `max_new_tokens: 440` budget — (`" ... the main clar, the main
+       clar, ..."` repeating), pass time 9.5–14 s.
+     - Ratio: ~6.5× more tokens, ~2–5× wall-clock (sub-linear vs. token count
+       because KV-cache growth cost isn't the whole per-token cost; still a
+       large, user-visible gap). A second same-order and a reversed-order
+       run reproduced the same order-of-magnitude split.
+**Mechanism:** this is not a compute-cost-per-token difference (ruled out by
+the clean-English measurement) and not an asset/config gap (ruled out above).
+It's that the multilingual model's forced decoder prompt carries an extra
+task token (`<|transcribe|>`, from its `forced_decoder_ids`) that `.en`'s
+prompt doesn't have, and — on audio where the model is already
+out-of-distribution enough to hallucinate — that different prompt context
+measurably changes *how long the hallucination loop runs before EOS*, not
+just the compute cost of the prompt tokens themselves. Because this app's
+`max_new_tokens: 440` (D-021) forces the classic single-pass path with no
+`no_repeat_ngram_size`/`repetition_penalty` set in either model's
+`generation_config.json` or in `inference.worker.ts`'s pipeline options,
+there is nothing to cut a non-terminating decode short before the hard
+token cap — so whichever checkpoint happens to run closer to that cap on a
+given input is dramatically slower for that one pass. Confirmed generation
+configs for both models differ only in the multilingual-specific fields
+(`lang_to_id`, `task_to_id`, an extra forced token at position 2) — neither
+sets a repetition guard.
+**Fix:** none applied. This is inherent behavior of the two upstream
+checkpoints reacting differently to the same repetition-loop failure mode,
+not a bug in this app's catalog, config, or asset selection — there is
+nothing in `src/lib/models.ts` to change (both already resolve to identical,
+merged, cached-KV decoders at the same dtype on every device tier). Inventing
+a per-model workaround (e.g. steering multilingual `.en`-adjacent use cases
+onto the `.en` model) was rejected: it would silently break real multilingual
+transcription for users who need it, to paper over a worst-case latency tail
+on inputs the model was always going to mistranscribe anyway.
+**Scope:** tiny/tiny.en only measured; the same forced-decoder-ids asymmetry
+exists for every multilingual/`.en` pair in the catalog (D-006), so the same
+content-dependent latency tail plausibly applies to base/small/medium too,
+untested here.
+**Revisit:** if this becomes a recurring complaint (not just a one-off
+report), the actionable mitigation is adding `no_repeat_ngram_size` (e.g. 3)
+and/or a `repetition_penalty` to the pipeline options in
+`inference.worker.ts` — that targets the actual failure mode (unbounded
+repetition loops on out-of-distribution audio) for both `.en` and
+multilingual models, rather than papering over one symptom of it. That
+change needs its own accuracy verification against
+`tests/live-accuracy.test.ts` and `tests/live-real-model.test.ts` (a
+repetition guard can change correct-transcript wording too) before landing,
+which is out of scope for this investigation.