Merge feat/danish-models: Danish fine-tune catalog entries (spec 001 workstream 6)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # docs/Decisions.md # docs/WorkBreakdown.md
2026-10-30 01:27:45 +01:00
parent c94689e2e0 498e361b8a
commit 439d099279
4 changed files with 202 additions and 1 deletions
@@ -1431,3 +1431,55 @@ tabs instead of a smaller Whisper model; a failed download is a retryable networ
 terminates the translate worker only.
 **Not done / follow-ups:** WebGPU for Opus-MT (unmeasurable here); more language pairs (the catalog
 is two entries, `TRANSLATION_MODELS`); a translation of live text *while* recording.
+
+## D-WS6a — Danish fine-tuned Whisper models: our own ONNX + CTranslate2 conversions, unpublished until the owner says go
+**Status:** Proposed (2026-10-29, spec 001 Goal 6)
+**Candidates checked (licence + architecture, from each repo's model card, config and notices):**
+- `thorhojhus/whisper-small-danish` — MIT; plain `WhisperForConditionalGeneration`, whisper-small
+  shapes and tokenizer unchanged (80 mel bins, 12/12 layers, vocab 51865). **Kept.**
+- `thorhojhus/whisper-large-v3-turbo-danish` — MIT; plain large-v3-turbo (128 mel, 32/4 layers,
+  vocab 51866). **Kept.**
+- `danish-foundation-models/edda-v0.2` — **Apache-2.0 confirmed** (licensor: the Alexandra
+  Institute; base model MIT); plain large-v3-turbo architecture, so Transformers.js loads it like the
+  stock turbo. Its `generation_config.json` pins `language: "da"`, `task: "transcribe"` and
+  `num_beams: 5`. **Kept.**
+**Catalog:** three entries with `verified: false` everywhere, ids built from one exported constant
+`DANISH_MODELS_HF_OWNER` (placeholder `"PENDING-OWNER"`): `<owner>/<name>-ONNX` for the browser
+engine and `<owner>/<name>-ct2-int8` as `fasterWhisperModel` (a Hub repo id — faster-whisper accepts
+those as well as its aliases). Publishing under the owner's account is a one-line change.
+**Conversions (outside the repo, in `/home/user/hf-publish/<repo>/`; nothing uploaded):** optimum 2.1
+`main_export` (fp32, KV-cached merged decoder) → onnxruntime fp16 / dynamic q8 / MatMulNBits q4 with
+the stock onnx-community settings; `ct2-transformers-converter --quantization int8`. File names, sizes
+and `transformers.js_config.use_external_data_format` match the stock `onnx-community/whisper-small`
+and `-large-v3-turbo` repos to within 0.2 MB per file, with one exception: our q8 turbo decoder
+(173 MB) quantizes the token embedding, while the stock one (440 MB) keeps it in fp32. Two export
+traps, now fixed in the tooling and described in each model card: optimum needs `accelerate`
+installed, or the tied `embed_tokens`/`proj_out` weight is stored twice (+159 MB on small); and
+onnxruntime's fp16 converter makes the merged decoder's If branches cyclic unless their outputs get
+branch-local names first. `generation_config.json` (`<|da|>` = 50285, `task_to_id`) and the tokenizer
+are copied unchanged and asserted equal to the source.
+**Measured** (WER on `da_festuge` with `language: "da"`; ONNX loaded from the local folder through
+Transformers.js 4.2.0 on onnxruntime-node, CT2 through faster-whisper 1.2.1 int8 beam 5; CPU only, one
+21.5 s clip, so the results are a conversion sanity check, not a benchmark):
+
+| model | q8 | q4 | fp16 enc + q4 dec | fp16 | CT2 int8 |
+|---|---|---|---|---|---|
+| whisper-small-danish | 0.109 | 0.094 | 0.094 | 0.094 | 0.078 |
+| stock whisper-small | 0.297 | 0.281 | — | — | 0.234 |
+| whisper-large-v3-turbo-danish | 0.141 | 0.141 | 0.094 | 0.094 | 0.094 |
+| edda-v0.2 | 0.109 | 0.063 | 0.094 | 0.094 | 0.078 |
+| stock whisper-large-v3-turbo | 0.125 | 0.141 | — | — | 0.094 |
+
+The small fine-tune cuts WER by about a factor of three. Neither turbo fine-tune separates from the
+stock turbo on this easy clip, so a longer, harder Danish test is needed to judge them. **D-022
+check:** the turbo fp16 encoders do not overflow on onnxruntime-node; whole-model fp16 gives the
+same transcript as fp16 + q4. WebGPU/WebNN behaviour still needs a real browser.
+**Dtype defaults:** WASM q4 for all three, including small (q4 matched or beat q8 and was 1.4–1.6×
+faster on CPU); WebGPU fp16 encoder + q4 decoder; WebNN fp16. All stay `verified: false`.
+**edda's `num_beams: 5`:** Transformers.js 4.2 has no beam search (its `BeamSearchSampler` returns
+`num_beams` candidates but generation keeps only the top-1), so browser decoding is greedy. It does
+not crash; the only cost is a full-vocabulary top-k per step. The server engine (faster-whisper) does
+use beam 5.
+**Pending the owner:** the Hugging Face account name, then upload the six prepared folders (model
+cards credit the original authors and carry MIT, or Apache-2.0 + NOTICE), then set
+`DANISH_MODELS_HF_OWNER`. The integrator sets `canTranslate: false` on these entries at merge.
@@ -69,6 +69,12 @@ Notes (M1):
      large-v3-turbo and `.en`, stale "translate" resets to transcribe — D-WS4a)
 - [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
      combinations in `models.ts`
+- [~] Danish fine-tunes (spec 001 Goal 6, D-WS6a): `whisper-small-danish`,
+      `whisper-large-v3-turbo-danish`, `edda-v0.2`. We converted them to ONNX
+      (fp32/fp16/q8/q4) and CT2 int8 and verified them on CPU. The catalog
+      entries hang off `DANISH_MODELS_HF_OWNER`. Still pending: the owner's go
+      for the Hugging Face upload (folders prepared, not uploaded), then the
+      owner constant, then browser checks of WebGPU/WebNN fp16.

 Notes (M2):
 - Quantized-variant verification is pending manual in-browser testing (no real
@@ -398,6 +404,15 @@ Found while building transcript translation (D-WS4b), 2026-10-29:
 - [ ] The server API still accepts `task: "translate"` with large-v3-turbo
      (only the UI hides it, D-WS4a).

+Found in spec 001 workstream 6 (D-WS6a), open:
+- [ ] Transformers.js 4.2 has no beam search: `num_beams > 1` in a model's
+      `generation_config.json` (edda-v0.2 ships 5) silently decodes greedily
+      and does a full-vocabulary top-k per step (D-WS6a). Not fixed: it would
+      mean overriding `num_beams: 1` in the worker, or a patch.
+- [ ] On `da_festuge`, neither Danish turbo fine-tune beats the stock turbo
+      (D-WS6a). Judging them needs a longer, harder Danish reference clip than
+      this repo has.
+
 ## Recurring
 - [ ] Maintain `AGENTS.md`:
  - **Add a rule only in response to a real failure.** When an agent repeatedly
@@ -173,6 +173,38 @@ const ACCEL_Q4: DtypeDefault = {
  note: "q4: the medium fp16 conversions fail to load / overflow (D-022).",
 };

+// ---------------------------------------------------------------------------
+// Danish fine-tunes (D-WS6a)
+// ---------------------------------------------------------------------------
+
+/**
+ * Hugging Face account that hosts our conversions of the Danish Whisper
+ * fine-tunes (D-WS6a). Both the ONNX repos (browser engine) and the
+ * CTranslate2 int8 repos (server engine) hang off this one value, so
+ * publishing under the owner's account is a one-line change here. Until the
+ * owner confirms the account and the repos exist, it stays a placeholder.
+ */
+export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER";
+
+/**
+ * WASM default for the Danish conversions (D-WS6a): measured on our own files
+ * (da_festuge, onnxruntime-node CPU), q4 matched or beat q8 on WER and was
+ * faster for all three — small included, unlike the stock small (q8).
+ */
+const DANISH_WASM_Q4: DtypeDefault = {
+  dtype: "q4",
+  verified: false,
+  note: "q4: best measured WER and fastest of q8/q4 on our Danish conversions (D-WS6a).",
+};
+
+/** Catalog id + matching faster-whisper repo for one published conversion. */
+function danishRepos(name: string): Pick<WhisperModel, "id" | "fasterWhisperModel"> {
+  return {
+    id: `${DANISH_MODELS_HF_OWNER}/${name}-ONNX`,
+    fasterWhisperModel: `${DANISH_MODELS_HF_OWNER}/${name}-ct2-int8`,
+  };
+}
+
 // ---------------------------------------------------------------------------
 // The catalog (D-006: curated, not exhaustive)
 // ---------------------------------------------------------------------------
@@ -279,6 +311,43 @@ export const MODELS: readonly WhisperModel[] = [
    decoderParamsM: 174,
    defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 },
  },
+  // Danish fine-tunes (D-WS6a): unchanged Whisper architectures, so the
+  // parameter counts are their base models'. Conversions by us, weights by
+  // the original authors (credited in each published repo's model card).
+  {
+    // thorhojhus/whisper-small-danish (MIT), fine-tune of whisper-small.
+    ...danishRepos("whisper-small-danish"),
+    displayName: "Whisper Small (Danish)",
+    multilingual: true,
+    // Translation untested on these fine-tunes (spec 001, D-WS4a).
+    canTranslate: false,
+    encoderParamsM: 88,
+    decoderParamsM: 156,
+    defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 },
+  },
+  {
+    // thorhojhus/whisper-large-v3-turbo-danish (MIT).
+    ...danishRepos("whisper-large-v3-turbo-danish"),
+    displayName: "Whisper Large v3 Turbo (Danish)",
+    multilingual: true,
+    // Translation untested on these fine-tunes (spec 001, D-WS4a).
+    canTranslate: false,
+    encoderParamsM: 635,
+    decoderParamsM: 174,
+    defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 },
+  },
+  {
+    // danish-foundation-models/edda-v0.2 (Apache-2.0, Alexandra Institute),
+    // fine-tune of whisper-large-v3-turbo.
+    ...danishRepos("edda-v0.2"),
+    displayName: "Edda v0.2 (Danish)",
+    multilingual: true,
+    // Translation untested on these fine-tunes (spec 001, D-WS4a).
+    canTranslate: false,
+    encoderParamsM: 635,
+    decoderParamsM: 174,
+    defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 },
+  },
 ];

 export function getModel(id: string): WhisperModel | undefined {
@@ -1,5 +1,6 @@
 import { describe, test, expect } from "bun:test";
 import {
+  DANISH_MODELS_HF_OWNER,
  DEFAULT_MODEL_ID,
  MODELS,
  WHISPER_LANGUAGES,
@@ -33,6 +34,9 @@ describe("MODELS catalog", () => {
      "onnx-community/whisper-medium.en_timestamped",
      "onnx-community/whisper-medium-ONNX",
      "onnx-community/whisper-large-v3-turbo",
+      `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX`,
+      `${DANISH_MODELS_HF_OWNER}/whisper-large-v3-turbo-danish-ONNX`,
+      `${DANISH_MODELS_HF_OWNER}/edda-v0.2-ONNX`,
    ]);
    expect(new Set(ids).size).toBe(ids.length);
  });
@@ -71,7 +75,9 @@ describe("MODELS catalog", () => {
  });

  test("every model maps to a faster-whisper name for the server engine (D-013)", () => {
-    for (const model of MODELS) {
+    // The Danish conversions (D-WS6a) have no faster-whisper alias; their
+    // server-engine name is a Hub repo id — covered by the Danish suite below.
+    for (const model of MODELS.filter((m) => !m.id.startsWith(`${DANISH_MODELS_HF_OWNER}/`))) {
      // The onnx-community ids and faster-whisper's aliases share the same
      // size suffix ("tiny.en", …, "large-v3-turbo") once repo-name
      // decorations ("_timestamped", "-ONNX") are stripped; keeping them
@@ -87,6 +93,65 @@ describe("MODELS catalog", () => {
  });
 });

+describe("Danish fine-tune conversions (D-WS6a)", () => {
+  const owner = `${DANISH_MODELS_HF_OWNER}/`;
+  const danish = MODELS.filter((model) => model.id.startsWith(owner));
+
+  test("are published under one owner constant, still a placeholder until the owner confirms", () => {
+    expect(DANISH_MODELS_HF_OWNER).toBe("PENDING-OWNER");
+    expect(danish.map((model) => model.id)).toEqual([
+      `${owner}whisper-small-danish-ONNX`,
+      `${owner}whisper-large-v3-turbo-danish-ONNX`,
+      `${owner}edda-v0.2-ONNX`,
+    ]);
+  });
+
+  test("point the server engine at the matching CTranslate2 int8 conversion under the same owner", () => {
+    for (const model of danish) {
+      const ct2 = model.id.replace(/-ONNX$/, "-ct2-int8");
+      expect(model.fasterWhisperModel).toBe(ct2);
+      expect(getFasterWhisperModel(model.id)).toBe(ct2);
+    }
+  });
+
+  test("are multilingual (language/task are sent) and unverified on every device", () => {
+    for (const model of danish) {
+      expect(model.multilingual).toBe(true);
+      for (const device of ["wasm", "webgpu", "webnn"] as const) {
+        expect(model.defaults[device].verified).toBe(false);
+      }
+    }
+  });
+
+  test("share their base model's parameter counts (unchanged architecture)", () => {
+    const small = getModel("onnx-community/whisper-small")!;
+    const turbo = getModel("onnx-community/whisper-large-v3-turbo")!;
+    const [smallDa, turboDa, edda] = danish;
+    for (const [model, base] of [
+      [smallDa!, small],
+      [turboDa!, turbo],
+      [edda!, turbo],
+    ] as const) {
+      expect(model.encoderParamsM).toBe(base.encoderParamsM);
+      expect(model.decoderParamsM).toBe(base.decoderParamsM);
+    }
+  });
+
+  test("default to the dtypes measured on the converted files (D-WS6a)", () => {
+    // da_festuge on onnxruntime-node: q4 matched or beat q8 and was faster for
+    // all three, small included; fp16 encoders did not overflow (D-022 check).
+    for (const model of danish) {
+      expect(resolveDefaultDtype(model.id, "wasm")).toBe("q4");
+      expect(resolveDefaultDtype(model.id, "webgpu")).toEqual({
+        encoder_model: "fp16",
+        decoder_model_merged: "q4",
+      });
+      expect(resolveDefaultDtype(model.id, "webnn")).toBe("fp16");
+      expect(model.defaults.wasm.note).toContain("D-WS6a");
+    }
+  });
+});
+
 describe("resolveDefaultDtype", () => {
  test("WASM defaults: q8 for small models, q4 for large ones", () => {
    expect(resolveDefaultDtype("onnx-community/whisper-tiny.en", "wasm")).toBe("q8");