Detect the language once for auto-detect file passes on the chunked path (D-053)
Transformers.js has no Whisper language detection and silently decoded Danish as English whenever "Skip silence" was off or unavailable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
3583d60847
commit
dd9dfc628b
@@ -163,6 +163,9 @@ environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-048).
|
||||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
With auto-detect on a multilingual model, it first detects the language on
|
||||
the first 30 s (`worker/language.ts`, D-053) and pins it — Transformers.js
|
||||
itself would default to English.
|
||||
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
|
||||
cut to a 120-token prompt budget at a whole word) and is prepended
|
||||
to the decoder input of every 30 s window by a wrapper around the model's
|
||||
|
||||
@@ -1821,3 +1821,28 @@ chunked path. Live mode is unaffected.
|
||||
**Caveat:** the WER numbers behind this use reference transcripts generated
|
||||
with Whisper (owner, 2026-10-30), so they favour Whisper-like output; re-check
|
||||
once correct transcripts are available.
|
||||
|
||||
## D-053 — "Auto" language on the chunked file path: detect once on the first 30 s, then pin it
|
||||
**Status:** Accepted (2026-10-30)
|
||||
Transformers.js 4.3.2 still has no Whisper language detection
|
||||
(`modeling_whisper.js`: `// TODO: Implement language detection` → "No
|
||||
language specified - defaulting to English"). With "Auto-detect" selected,
|
||||
a file transcribed on the chunked path — the fallback whenever "Skip
|
||||
silence" (D-052) is off or its VAD model is unavailable — was decoded as
|
||||
English: Danish `da_festuge` came out as a rough English translation (WER
|
||||
0.94 vs 0.14 with `da`, found in D-049's measurements).
|
||||
**Decision:** `src/worker/language.ts` `detectFileLanguage` reuses the
|
||||
long-form pipeline's one-step detector (D-049: encoder pass + one decoder
|
||||
step restricted to language tokens) on the first 30 s of the file, and the
|
||||
worker pins the result for the whole chunked pass. Only file passes on
|
||||
multilingual models without an explicit language; `.en` models and an
|
||||
explicit language are unchanged.
|
||||
**Verified:** `tests/live-real-model.test.ts` "auto language on the chunked
|
||||
file path": da_festuge → `da` (WER within its bar), en_aiproblem and
|
||||
en_alien → `en` (Large v3 Turbo, onnxruntime-node).
|
||||
**Cost:** one extra encoder pass per file (the chunked pipeline can't reuse
|
||||
it). **Not covered:** live passes with "auto" still decode as English —
|
||||
detecting on every short live pass would add an encoder run per pass; it
|
||||
needs a once-per-session design (WorkBreakdown).
|
||||
**Caveat:** leading silence or music in the first 30 s can mislead
|
||||
detection; long-form detects on the first *speech* chunk instead.
|
||||
|
||||
@@ -441,7 +441,7 @@ Found in spec 001 workstream 6 (D-046), open:
|
||||
this repo has.
|
||||
|
||||
Found while building the long-form pipeline (spec 001 Goal 3, D-049), 2026-10-29:
|
||||
- [ ] "Auto-detect" on the default (chunked) file path does not detect: with no
|
||||
- [x] "Auto-detect" on the default (chunked) file path does not detect: with no
|
||||
`language`, Transformers.js 4.2 logs "No language specified - defaulting to
|
||||
English" and decodes Danish audio as an English translation (da_festuge WER
|
||||
0.94 vs 0.14 with `da`). Long-form ("Skip silence") detects once and pins
|
||||
@@ -449,6 +449,11 @@ Found while building the long-form pipeline (spec 001 Goal 3, D-049), 2026-10-29
|
||||
- [ ] Spec 001's measurement fallback mentions a hint list of the names in
|
||||
`en_alien`'s reference; `tests/meetingFixture.ts` returns audio + reference
|
||||
only. Add the list when the vocabulary hint (workstream 2) lands.
|
||||
— Fixed for file passes: detect once on the first 30 s and pin (D-053).
|
||||
- [ ] "Auto-detect" on **live** passes (browser engine) still decodes as English:
|
||||
each live pass would need its own detection (an extra encoder run per
|
||||
pass). Needs a once-per-session design (detect on the first committed
|
||||
segment, then pin) — D-053.
|
||||
|
||||
- [ ] Parakeet v3 also transcribes 24 other European languages, Danish included, but the app
|
||||
offers it English-only (spec 001) — unmeasured on `da_festuge`. Candidate for a later spec.
|
||||
|
||||
Reference in New Issue
Block a user