Detect the language once for auto-detect file passes on the chunked path (D-053)

Transformers.js has no Whisper language detection and silently decoded
Danish as English whenever "Skip silence" was off or unavailable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas Winther
2026-10-30 14:35:09 +01:00
co-authored by Claude Opus 5.5
parent 3583d60847
commit dd9dfc628b
6 changed files with 112 additions and 6 deletions
+3
View File
@@ -163,6 +163,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-048).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
With auto-detect on a multilingual model, it first detects the language on
the first 30 s (`worker/language.ts`, D-053) and pins it — Transformers.js
itself would default to English.
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
cut to a 120-token prompt budget at a whole word) and is prepended
to the decoder input of every 30 s window by a wrapper around the model's
+25
View File
@@ -1821,3 +1821,28 @@ chunked path. Live mode is unaffected.
**Caveat:** the WER numbers behind this use reference transcripts generated
with Whisper (owner, 2026-10-30), so they favour Whisper-like output; re-check
once correct transcripts are available.
## D-053 — "Auto" language on the chunked file path: detect once on the first 30 s, then pin it
**Status:** Accepted (2026-10-30)
Transformers.js 4.3.2 still has no Whisper language detection
(`modeling_whisper.js`: `// TODO: Implement language detection` → "No
language specified - defaulting to English"). With "Auto-detect" selected,
a file transcribed on the chunked path — the fallback whenever "Skip
silence" (D-052) is off or its VAD model is unavailable — was decoded as
English: Danish `da_festuge` came out as a rough English translation (WER
0.94 vs 0.14 with `da`, found in D-049's measurements).
**Decision:** `src/worker/language.ts` `detectFileLanguage` reuses the
long-form pipeline's one-step detector (D-049: encoder pass + one decoder
step restricted to language tokens) on the first 30 s of the file, and the
worker pins the result for the whole chunked pass. Only file passes on
multilingual models without an explicit language; `.en` models and an
explicit language are unchanged.
**Verified:** `tests/live-real-model.test.ts` "auto language on the chunked
file path": da_festuge → `da` (WER within its bar), en_aiproblem and
en_alien → `en` (Large v3 Turbo, onnxruntime-node).
**Cost:** one extra encoder pass per file (the chunked pipeline can't reuse
it). **Not covered:** live passes with "auto" still decode as English —
detecting on every short live pass would add an encoder run per pass; it
needs a once-per-session design (WorkBreakdown).
**Caveat:** leading silence or music in the first 30 s can mislead
detection; long-form detects on the first *speech* chunk instead.
+6 -1
View File
@@ -441,7 +441,7 @@ Found in spec 001 workstream 6 (D-046), open:
this repo has.
Found while building the long-form pipeline (spec 001 Goal 3, D-049), 2026-10-29:
- [ ] "Auto-detect" on the default (chunked) file path does not detect: with no
- [x] "Auto-detect" on the default (chunked) file path does not detect: with no
`language`, Transformers.js 4.2 logs "No language specified - defaulting to
English" and decodes Danish audio as an English translation (da_festuge WER
0.94 vs 0.14 with `da`). Long-form ("Skip silence") detects once and pins
@@ -449,6 +449,11 @@ Found while building the long-form pipeline (spec 001 Goal 3, D-049), 2026-10-29
- [ ] Spec 001's measurement fallback mentions a hint list of the names in
`en_alien`'s reference; `tests/meetingFixture.ts` returns audio + reference
only. Add the list when the vocabulary hint (workstream 2) lands.
— Fixed for file passes: detect once on the first 30 s and pin (D-053).
- [ ] "Auto-detect" on **live** passes (browser engine) still decodes as English:
each live pass would need its own detection (an extra encoder run per
pass). Needs a once-per-session design (detect on the first committed
segment, then pin) — D-053.
- [ ] Parakeet v3 also transcribes 24 other European languages, Danish included, but the app
offers it English-only (spec 001) — unmeasured on `da_festuge`. Candidate for a later spec.