From f8ce3120fe027d8ff0493a67d3926e07d6303cfa Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Thu, 29 Oct 2026 21:35:40 +0100 Subject: [PATCH 1/5] Plan workstream 6: Danish fine-tuned Whisper conversions and catalog entries Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-6-danish-models.md | 115 ++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100644 docs/plans/001-6-danish-models.md diff --git a/docs/plans/001-6-danish-models.md b/docs/plans/001-6-danish-models.md new file mode 100644 index 0000000..c573f4a --- /dev/null +++ b/docs/plans/001-6-danish-models.md @@ -0,0 +1,115 @@ +# 001-6 Danish fine-tuned Whisper models Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (inline, no subagents) to +> implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** convert the Danish Whisper fine-tunes to the onnx-community (Transformers.js) and CTranslate2 +int8 layouts, verify them locally, prepare (not upload) one publication folder per would-be Hugging Face +repo, and add `verified: false` catalog entries whose ids hang off one owner constant. + +**Architecture:** all conversion work happens outside the repo (`/home/user/venvs/convert`, sources in +`/home/user/hf-src/`, outputs in `/home/user/hf-publish//`, helper scripts in +`/home/user/hf-tools/`). The repo only gains catalog data in `src/lib/models.ts`, its tests, and docs. + +**Tech Stack:** Python 3.13 venv (optimum[onnxruntime], transformers, onnx, onnxruntime, ctranslate2), +faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-node) for WER. + +**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 6; `docs/plans/001-0-coordination.md`. + +## Global Constraints + +- `flock /tmp/whisper-heavy.lock` around every download, `pip install`, conversion, WER/benchmark run. +- Venvs only in `/home/user/venvs/`. Nothing leaves the machine: no HF upload, no repo creation, no push. +- Decision placeholder `D-WS6a` (not a real `D-###`); cite it in code comments. +- Do **not** add `canTranslate` (integrator sets it to `false` for the new entries at merge). +- New entries `verified: false` on every device; ids `` `${DANISH_MODELS_HF_OWNER}/` ``. +- "Show the owner first": the final report lists repos, files, sizes, cards and WER; uploading waits for go. +- Measurements are CPU (onnxruntime-node / CTranslate2) numbers on one 21.5 s clip; say so. + +## Review Focus + +1. **Owner constant unchanged at merge** — ids must still resolve: catalog tests assert ids are built from + `DANISH_MODELS_HF_OWNER`, so changing that one line moves every id + `fasterWhisperModel` together. +2. **Language tokens lost in export** — `generation_config.json` must keep `lang_to_id["<|da|>"]` and + `task_to_id`; checked per output folder in Task 3 (script asserts). +3. **fp16 encoder overflow on large models (D-022)** — turbo fine-tunes' fp16 encoders are WER-checked on + onnxruntime-node; if garbage, the catalog's WebGPU/WebNN defaults avoid fp16 encoder (Task 5). +4. **edda's `num_beams: 5` generation config** — Transformers.js 4.2 must not crash or silently + misbehave; WER run in Task 4 uses the shipped `generation_config.json` unmodified. +5. **Server engine with a Hub-id model name** — `fasterWhisperModel` is a repo id, not an alias; existing + test that derives aliases from ids must exempt the Danish entries explicitly, not be loosened globally. + +--- + +### Task 1: Licence + architecture check (done during planning; record it) + +- [x] `thorhojhus/whisper-small-danish`: MIT (card + THIRD_PARTY_NOTICES), `WhisperForConditionalGeneration`, + unchanged whisper-small architecture/tokenizer (80 mel, 12/12 layers, vocab 51865). +- [x] `thorhojhus/whisper-large-v3-turbo-danish`: MIT, plain large-v3-turbo (128 mel, 32/4 layers, vocab 51866). +- [x] `danish-foundation-models/edda-v0.2`: Apache-2.0 (licensor: Alexandra Institute), plain + large-v3-turbo; generation config pins `language: da`, `num_beams: 5`. Kept. + +### Task 2: Conversion venv + export/quantize tooling + +**Files:** Create `/home/user/hf-tools/export_onnx.py`, `/home/user/hf-tools/quantize.py`, +`/home/user/hf-tools/check_layout.py` (outside the repo). + +- [ ] `python3 -m venv /home/user/venvs/convert`; `flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common`. +- [ ] Download each source repo snapshot under the lock to `/home/user/hf-src/` (`huggingface_hub.snapshot_download`, read-only). +- [ ] `export_onnx.py `: optimum `main_export(task="automatic-speech-recognition-with-past", dtype fp32)` + → keep only `encoder_model.onnx(+_data)` and `decoder_model_merged.onnx(+_data)` under `/onnx/`; + copy config/tokenizer/generation/preprocessor/normalizer/added_tokens/special_tokens/vocab/merges. +- [ ] `quantize.py `: for `encoder_model`, `decoder_model_merged` write `_fp16` (onnxruntime + `float16.convert_float_to_float16`, keep_io_types), `_quantized` (q8: `quantize_dynamic`, QUInt8, + op_types from stock `onnx-community/whisper-small/quantize_config.json`), `_q4` + (`MatMul4BitsQuantizer`, block 32, symmetric). Encoder > 2 GB → external data + `transformers.js_config.use_external_data_format` in config.json, as the stock turbo repo. +- [ ] `check_layout.py `: asserts every file the stock repo has for those dtypes + exists, `<|da|>` id in `generation_config.json` equals the source, and onnxruntime can create a session + for each `.onnx` (ORT_ENABLE_BASIC, D-014). + +### Task 3: Convert all three (ONNX + CT2) + +- [ ] For each kept model: `flock … export_onnx.py`, `flock … quantize.py`, `check_layout.py` → pass. +- [ ] `flock … ct2-transformers-converter --model --output_dir /home/user/hf-publish/-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json` (+ generation config / normalizer if present). + +### Task 4: Verify (WER) + +**Files:** Create `/home/user/hf-tools/wer.ts` (Bun; imports the worktree's `tests/dsp.ts` +`wordErrorRate`; same normalisation and worker options as `tests/live-real-model.test.ts`: +`chunk_length_s 30, stride 5, return_timestamps, max_new_tokens 440, language "da"`). +Create `/home/user/hf-tools/ct2_wer.py`. + +- [ ] `wer.ts --local --dtype q8|q4` with `env.localModelPath`, `env.allowRemoteModels = false`; + run under the lock for every conversion at q8 and q4, plus fp16-encoder+q4-decoder for the WebGPU default. +- [ ] Same for stock `onnx-community/whisper-small` (q8, q4) and `onnx-community/whisper-large-v3-turbo` (q4, q8). +- [ ] `ct2_wer.py ` with faster-whisper (language da, beam 5, CPU int8) vs stock `small` / `large-v3-turbo`. +- [ ] Record wall-clock + WER table (goes into D-WS6a and the model cards). + +### Task 5: Catalog entries (TDD) + +**Files:** Modify `src/lib/models.ts`, `tests/models.test.ts`. + +**Interfaces — Produces:** `export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"`; entries with +ids `` `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` `` etc. (exact names per Task 6), +`fasterWhisperModel` = `` `${DANISH_MODELS_HF_OWNER}/-ct2-int8` ``, `multilingual: true`. + +- [ ] Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and + `fasterWhisperModel` start with `${DANISH_MODELS_HF_OWNER}/`; none `verified`; `getFasterWhisperModel` + returns the CT2 repo; dtype defaults equal the measured choice; the existing alias test skips only ids + under the owner constant. +- [ ] Run → fail. Implement entries. Run → pass; typecheck; commit. + +### Task 6: Publication folders + model cards + +- [ ] Proposed names: `whisper-small-danish-ONNX`, `whisper-small-danish-ct2-int8`, + `whisper-large-v3-turbo-danish-ONNX`, `whisper-large-v3-turbo-danish-ct2-int8`, `edda-v0.2-ONNX`, + `edda-v0.2-ct2-int8` under ``. +- [ ] Each folder gets README.md (front matter: licence carried, `base_model`, `library_name`; credit to the + original authors, conversion commands, measured WER, intended use with whisper_web), LICENSE/ + THIRD_PARTY_NOTICES from the source, and a `SIZES.txt` (not uploaded — listed for the owner). + +### Task 7: Docs + finish + +- [ ] `docs/Decisions.md`: append `## D-WS6a — …` (licences, layout, measurements, dtype choice, pending owner go). +- [ ] `docs/WorkBreakdown.md`: add/tick the item; "Noticed, not fixed" for anything left. +- [ ] `bun run typecheck`, fast suite, `bun run build` green; `git rm` this plan as the last commit. From 2a4f5b16a57234308e9ab292fddf7e1e4d3ee073 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Thu, 29 Oct 2026 22:00:33 +0100 Subject: [PATCH 2/5] Add Danish fine-tune catalog entries under a single pending Hugging Face owner constant Co-Authored-By: Claude Opus 5.5 --- docs/Decisions.md | 16 +++++++++++++ src/lib/models.ts | 52 +++++++++++++++++++++++++++++++++++++++++++ tests/models.test.ts | 53 +++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 120 insertions(+), 1 deletion(-) diff --git a/docs/Decisions.md b/docs/Decisions.md index 5dc8f08..ce27729 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1224,3 +1224,19 @@ whole-file output is equally bad, which is exactly what the absolute bar exists to catch. **Consequence:** full `bun test` ≈ 19 min on CPU (en_alien ≈ 20 min of it on a cold run); the fast suite excludes this file. + +## D-WS6a — Danish fine-tuned Whisper models: our own ONNX + CTranslate2 conversions, unpublished until the owner says go +**Status:** Proposed (2026-10-29, spec 001 Goal 6) +**Candidates checked (licence + architecture, from each repo's model card, config and notices):** +- `thorhojhus/whisper-small-danish` — MIT; plain `WhisperForConditionalGeneration`, whisper-small + shapes and tokenizer unchanged (80 mel bins, 12/12 layers, vocab 51865). **Kept.** +- `thorhojhus/whisper-large-v3-turbo-danish` — MIT; plain large-v3-turbo (128 mel, 32/4 layers, + vocab 51866). **Kept.** +- `danish-foundation-models/edda-v0.2` — **Apache-2.0 confirmed** (licensor: the Alexandra + Institute; base model MIT); plain large-v3-turbo architecture, so Transformers.js loads it like the + stock turbo. Its `generation_config.json` pins `language: "da"`, `task: "transcribe"` and + `num_beams: 5`. **Kept.** +**Catalog:** three entries with `verified: false` everywhere, ids built from one exported constant +`DANISH_MODELS_HF_OWNER` (placeholder `"PENDING-OWNER"`): `/-ONNX` for the browser +engine and `/-ct2-int8` as `fasterWhisperModel` (a Hub repo id — faster-whisper accepts +those as well as its aliases). Publishing under the owner's account is a one-line change. diff --git a/src/lib/models.ts b/src/lib/models.ts index ad82c0c..06e900b 100644 --- a/src/lib/models.ts +++ b/src/lib/models.ts @@ -164,6 +164,27 @@ const ACCEL_Q4: DtypeDefault = { note: "q4: the medium fp16 conversions fail to load / overflow (D-022).", }; +// --------------------------------------------------------------------------- +// Danish fine-tunes (D-WS6a) +// --------------------------------------------------------------------------- + +/** + * Hugging Face account that hosts our conversions of the Danish Whisper + * fine-tunes (D-WS6a). Both the ONNX repos (browser engine) and the + * CTranslate2 int8 repos (server engine) hang off this one value, so + * publishing under the owner's account is a one-line change here. Until the + * owner confirms the account and the repos exist, it stays a placeholder. + */ +export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"; + +/** Catalog id + matching faster-whisper repo for one published conversion. */ +function danishRepos(name: string): Pick { + return { + id: `${DANISH_MODELS_HF_OWNER}/${name}-ONNX`, + fasterWhisperModel: `${DANISH_MODELS_HF_OWNER}/${name}-ct2-int8`, + }; +} + // --------------------------------------------------------------------------- // The catalog (D-006: curated, not exhaustive) // --------------------------------------------------------------------------- @@ -259,6 +280,37 @@ export const MODELS: readonly WhisperModel[] = [ decoderParamsM: 174, defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, }, + // Danish fine-tunes (D-WS6a): unchanged Whisper architectures, so the + // parameter counts are their base models'. Conversions by us, weights by + // the original authors (credited in each published repo's model card). + { + // thorhojhus/whisper-small-danish (MIT), fine-tune of whisper-small. + ...danishRepos("whisper-small-danish"), + displayName: "Whisper Small (Danish)", + multilingual: true, + encoderParamsM: 88, + decoderParamsM: 156, + defaults: { wasm: WASM_Q8, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + }, + { + // thorhojhus/whisper-large-v3-turbo-danish (MIT). + ...danishRepos("whisper-large-v3-turbo-danish"), + displayName: "Whisper Large v3 Turbo (Danish)", + multilingual: true, + encoderParamsM: 635, + decoderParamsM: 174, + defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + }, + { + // danish-foundation-models/edda-v0.2 (Apache-2.0, Alexandra Institute), + // fine-tune of whisper-large-v3-turbo. + ...danishRepos("edda-v0.2"), + displayName: "Edda v0.2 (Danish)", + multilingual: true, + encoderParamsM: 635, + decoderParamsM: 174, + defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + }, ]; export function getModel(id: string): WhisperModel | undefined { diff --git a/tests/models.test.ts b/tests/models.test.ts index 1ef20c3..79d89ea 100644 --- a/tests/models.test.ts +++ b/tests/models.test.ts @@ -1,5 +1,6 @@ import { describe, test, expect } from "bun:test"; import { + DANISH_MODELS_HF_OWNER, DEFAULT_MODEL_ID, MODELS, WHISPER_LANGUAGES, @@ -32,6 +33,9 @@ describe("MODELS catalog", () => { "onnx-community/whisper-medium.en_timestamped", "onnx-community/whisper-medium-ONNX", "onnx-community/whisper-large-v3-turbo", + `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX`, + `${DANISH_MODELS_HF_OWNER}/whisper-large-v3-turbo-danish-ONNX`, + `${DANISH_MODELS_HF_OWNER}/edda-v0.2-ONNX`, ]); expect(new Set(ids).size).toBe(ids.length); }); @@ -70,7 +74,9 @@ describe("MODELS catalog", () => { }); test("every model maps to a faster-whisper name for the server engine (D-013)", () => { - for (const model of MODELS) { + // The Danish conversions (D-WS6a) have no faster-whisper alias; their + // server-engine name is a Hub repo id — covered by the Danish suite below. + for (const model of MODELS.filter((m) => !m.id.startsWith(`${DANISH_MODELS_HF_OWNER}/`))) { // The onnx-community ids and faster-whisper's aliases share the same // size suffix ("tiny.en", …, "large-v3-turbo") once repo-name // decorations ("_timestamped", "-ONNX") are stripped; keeping them @@ -86,6 +92,51 @@ describe("MODELS catalog", () => { }); }); +describe("Danish fine-tune conversions (D-WS6a)", () => { + const owner = `${DANISH_MODELS_HF_OWNER}/`; + const danish = MODELS.filter((model) => model.id.startsWith(owner)); + + test("are published under one owner constant, still a placeholder until the owner confirms", () => { + expect(DANISH_MODELS_HF_OWNER).toBe("PENDING-OWNER"); + expect(danish.map((model) => model.id)).toEqual([ + `${owner}whisper-small-danish-ONNX`, + `${owner}whisper-large-v3-turbo-danish-ONNX`, + `${owner}edda-v0.2-ONNX`, + ]); + }); + + test("point the server engine at the matching CTranslate2 int8 conversion under the same owner", () => { + for (const model of danish) { + const ct2 = model.id.replace(/-ONNX$/, "-ct2-int8"); + expect(model.fasterWhisperModel).toBe(ct2); + expect(getFasterWhisperModel(model.id)).toBe(ct2); + } + }); + + test("are multilingual (language/task are sent) and unverified on every device", () => { + for (const model of danish) { + expect(model.multilingual).toBe(true); + for (const device of ["wasm", "webgpu", "webnn"] as const) { + expect(model.defaults[device].verified).toBe(false); + } + } + }); + + test("share their base model's parameter counts (unchanged architecture)", () => { + const small = getModel("onnx-community/whisper-small")!; + const turbo = getModel("onnx-community/whisper-large-v3-turbo")!; + const [smallDa, turboDa, edda] = danish; + for (const [model, base] of [ + [smallDa!, small], + [turboDa!, turbo], + [edda!, turbo], + ] as const) { + expect(model.encoderParamsM).toBe(base.encoderParamsM); + expect(model.decoderParamsM).toBe(base.decoderParamsM); + } + }); +}); + describe("resolveDefaultDtype", () => { test("WASM defaults: q8 for small models, q4 for large ones", () => { expect(resolveDefaultDtype("onnx-community/whisper-tiny.en", "wasm")).toBe("q8"); From 951aad8f55c094552849ee9f75351ce44762fe10 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 01:12:28 +0100 Subject: [PATCH 3/5] Default the Danish conversions to q4 on WASM, as measured on the converted files Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-6-danish-models.md | 32 +++++++++++++++---------------- src/lib/models.ts | 17 +++++++++++++--- tests/models.test.ts | 14 ++++++++++++++ 3 files changed, 44 insertions(+), 19 deletions(-) diff --git a/docs/plans/001-6-danish-models.md b/docs/plans/001-6-danish-models.md index c573f4a..6ff4e13 100644 --- a/docs/plans/001-6-danish-models.md +++ b/docs/plans/001-6-danish-models.md @@ -1,7 +1,7 @@ # 001-6 Danish fine-tuned Whisper models Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (inline, no subagents) to -> implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking. **Goal:** convert the Danish Whisper fine-tunes to the onnx-community (Transformers.js) and CTranslate2 int8 layouts, verify them locally, prepare (not upload) one publication folder per would-be Hugging Face @@ -54,23 +54,23 @@ faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-nod **Files:** Create `/home/user/hf-tools/export_onnx.py`, `/home/user/hf-tools/quantize.py`, `/home/user/hf-tools/check_layout.py` (outside the repo). -- [ ] `python3 -m venv /home/user/venvs/convert`; `flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common`. -- [ ] Download each source repo snapshot under the lock to `/home/user/hf-src/` (`huggingface_hub.snapshot_download`, read-only). -- [ ] `export_onnx.py `: optimum `main_export(task="automatic-speech-recognition-with-past", dtype fp32)` +- [x] `python3 -m venv /home/user/venvs/convert`; `flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common`. +- [x] Download each source repo snapshot under the lock to `/home/user/hf-src/` (`huggingface_hub.snapshot_download`, read-only). +- [x] `export_onnx.py `: optimum `main_export(task="automatic-speech-recognition-with-past", dtype fp32)` → keep only `encoder_model.onnx(+_data)` and `decoder_model_merged.onnx(+_data)` under `/onnx/`; copy config/tokenizer/generation/preprocessor/normalizer/added_tokens/special_tokens/vocab/merges. -- [ ] `quantize.py `: for `encoder_model`, `decoder_model_merged` write `_fp16` (onnxruntime +- [x] `quantize.py `: for `encoder_model`, `decoder_model_merged` write `_fp16` (onnxruntime `float16.convert_float_to_float16`, keep_io_types), `_quantized` (q8: `quantize_dynamic`, QUInt8, op_types from stock `onnx-community/whisper-small/quantize_config.json`), `_q4` (`MatMul4BitsQuantizer`, block 32, symmetric). Encoder > 2 GB → external data + `transformers.js_config.use_external_data_format` in config.json, as the stock turbo repo. -- [ ] `check_layout.py `: asserts every file the stock repo has for those dtypes +- [x] `check_layout.py `: asserts every file the stock repo has for those dtypes exists, `<|da|>` id in `generation_config.json` equals the source, and onnxruntime can create a session for each `.onnx` (ORT_ENABLE_BASIC, D-014). ### Task 3: Convert all three (ONNX + CT2) -- [ ] For each kept model: `flock … export_onnx.py`, `flock … quantize.py`, `check_layout.py` → pass. -- [ ] `flock … ct2-transformers-converter --model --output_dir /home/user/hf-publish/-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json` (+ generation config / normalizer if present). +- [x] For each kept model: `flock … export_onnx.py`, `flock … quantize.py`, `check_layout.py` → pass. +- [x] `flock … ct2-transformers-converter --model --output_dir /home/user/hf-publish/-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json` (+ generation config / normalizer if present). ### Task 4: Verify (WER) @@ -79,11 +79,11 @@ faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-nod `chunk_length_s 30, stride 5, return_timestamps, max_new_tokens 440, language "da"`). Create `/home/user/hf-tools/ct2_wer.py`. -- [ ] `wer.ts --local --dtype q8|q4` with `env.localModelPath`, `env.allowRemoteModels = false`; +- [x] `wer.ts --local --dtype q8|q4` with `env.localModelPath`, `env.allowRemoteModels = false`; run under the lock for every conversion at q8 and q4, plus fp16-encoder+q4-decoder for the WebGPU default. -- [ ] Same for stock `onnx-community/whisper-small` (q8, q4) and `onnx-community/whisper-large-v3-turbo` (q4, q8). -- [ ] `ct2_wer.py ` with faster-whisper (language da, beam 5, CPU int8) vs stock `small` / `large-v3-turbo`. -- [ ] Record wall-clock + WER table (goes into D-WS6a and the model cards). +- [x] Same for stock `onnx-community/whisper-small` (q8, q4) and `onnx-community/whisper-large-v3-turbo` (q4, q8). +- [x] `ct2_wer.py ` with faster-whisper (language da, beam 5, CPU int8) vs stock `small` / `large-v3-turbo`. +- [x] Record wall-clock + WER table (goes into D-WS6a and the model cards). ### Task 5: Catalog entries (TDD) @@ -93,18 +93,18 @@ Create `/home/user/hf-tools/ct2_wer.py`. ids `` `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` `` etc. (exact names per Task 6), `fasterWhisperModel` = `` `${DANISH_MODELS_HF_OWNER}/-ct2-int8` ``, `multilingual: true`. -- [ ] Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and +- [x] Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and `fasterWhisperModel` start with `${DANISH_MODELS_HF_OWNER}/`; none `verified`; `getFasterWhisperModel` returns the CT2 repo; dtype defaults equal the measured choice; the existing alias test skips only ids under the owner constant. -- [ ] Run → fail. Implement entries. Run → pass; typecheck; commit. +- [x] Run → fail. Implement entries. Run → pass; typecheck; commit. ### Task 6: Publication folders + model cards -- [ ] Proposed names: `whisper-small-danish-ONNX`, `whisper-small-danish-ct2-int8`, +- [x] Proposed names: `whisper-small-danish-ONNX`, `whisper-small-danish-ct2-int8`, `whisper-large-v3-turbo-danish-ONNX`, `whisper-large-v3-turbo-danish-ct2-int8`, `edda-v0.2-ONNX`, `edda-v0.2-ct2-int8` under ``. -- [ ] Each folder gets README.md (front matter: licence carried, `base_model`, `library_name`; credit to the +- [x] Each folder gets README.md (front matter: licence carried, `base_model`, `library_name`; credit to the original authors, conversion commands, measured WER, intended use with whisper_web), LICENSE/ THIRD_PARTY_NOTICES from the source, and a `SIZES.txt` (not uploaded — listed for the owner). diff --git a/src/lib/models.ts b/src/lib/models.ts index 06e900b..b942b2d 100644 --- a/src/lib/models.ts +++ b/src/lib/models.ts @@ -177,6 +177,17 @@ const ACCEL_Q4: DtypeDefault = { */ export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"; +/** + * WASM default for the Danish conversions (D-WS6a): measured on our own files + * (da_festuge, onnxruntime-node CPU), q4 matched or beat q8 on WER and was + * faster for all three — small included, unlike the stock small (q8). + */ +const DANISH_WASM_Q4: DtypeDefault = { + dtype: "q4", + verified: false, + note: "q4: best measured WER and fastest of q8/q4 on our Danish conversions (D-WS6a).", +}; + /** Catalog id + matching faster-whisper repo for one published conversion. */ function danishRepos(name: string): Pick { return { @@ -290,7 +301,7 @@ export const MODELS: readonly WhisperModel[] = [ multilingual: true, encoderParamsM: 88, decoderParamsM: 156, - defaults: { wasm: WASM_Q8, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, }, { // thorhojhus/whisper-large-v3-turbo-danish (MIT). @@ -299,7 +310,7 @@ export const MODELS: readonly WhisperModel[] = [ multilingual: true, encoderParamsM: 635, decoderParamsM: 174, - defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, }, { // danish-foundation-models/edda-v0.2 (Apache-2.0, Alexandra Institute), @@ -309,7 +320,7 @@ export const MODELS: readonly WhisperModel[] = [ multilingual: true, encoderParamsM: 635, decoderParamsM: 174, - defaults: { wasm: WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, + defaults: { wasm: DANISH_WASM_Q4, webgpu: WEBGPU_FP16_Q4, webnn: WEBNN_FP16 }, }, ]; diff --git a/tests/models.test.ts b/tests/models.test.ts index 79d89ea..48e40c1 100644 --- a/tests/models.test.ts +++ b/tests/models.test.ts @@ -135,6 +135,20 @@ describe("Danish fine-tune conversions (D-WS6a)", () => { expect(model.decoderParamsM).toBe(base.decoderParamsM); } }); + + test("default to the dtypes measured on the converted files (D-WS6a)", () => { + // da_festuge on onnxruntime-node: q4 matched or beat q8 and was faster for + // all three, small included; fp16 encoders did not overflow (D-022 check). + for (const model of danish) { + expect(resolveDefaultDtype(model.id, "wasm")).toBe("q4"); + expect(resolveDefaultDtype(model.id, "webgpu")).toEqual({ + encoder_model: "fp16", + decoder_model_merged: "q4", + }); + expect(resolveDefaultDtype(model.id, "webnn")).toBe("fp16"); + expect(model.defaults.wasm.note).toContain("D-WS6a"); + } + }); }); describe("resolveDefaultDtype", () => { From 19228297f168224aaaf000b573daf00f096675ac Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 01:13:27 +0100 Subject: [PATCH 4/5] Record the Danish conversions' measurements, dtype choice and pending upload in D-WS6a Co-Authored-By: Claude Opus 5.5 --- docs/Decisions.md | 36 +++++++++++++++++++++++++++++++ docs/WorkBreakdown.md | 15 +++++++++++++ docs/plans/001-6-danish-models.md | 6 +++--- 3 files changed, 54 insertions(+), 3 deletions(-) diff --git a/docs/Decisions.md b/docs/Decisions.md index ce27729..2f465b5 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -1240,3 +1240,39 @@ on a cold run); the fast suite excludes this file. `DANISH_MODELS_HF_OWNER` (placeholder `"PENDING-OWNER"`): `/-ONNX` for the browser engine and `/-ct2-int8` as `fasterWhisperModel` (a Hub repo id — faster-whisper accepts those as well as its aliases). Publishing under the owner's account is a one-line change. +**Conversions (outside the repo, in `/home/user/hf-publish//`; nothing uploaded):** optimum 2.1 +`main_export` (fp32, KV-cached merged decoder) → onnxruntime fp16 / dynamic q8 / MatMulNBits q4 with +the stock onnx-community settings; `ct2-transformers-converter --quantization int8`. File names, sizes +and `transformers.js_config.use_external_data_format` match the stock `onnx-community/whisper-small` +and `-large-v3-turbo` repos to within 0.2 MB per file, with one exception: our q8 turbo decoder +(173 MB) quantizes the token embedding, while the stock one (440 MB) keeps it in fp32. Two export +traps, now fixed in the tooling and described in each model card: optimum needs `accelerate` +installed, or the tied `embed_tokens`/`proj_out` weight is stored twice (+159 MB on small); and +onnxruntime's fp16 converter makes the merged decoder's If branches cyclic unless their outputs get +branch-local names first. `generation_config.json` (`<|da|>` = 50285, `task_to_id`) and the tokenizer +are copied unchanged and asserted equal to the source. +**Measured** (WER on `da_festuge` with `language: "da"`; ONNX loaded from the local folder through +Transformers.js 4.2.0 on onnxruntime-node, CT2 through faster-whisper 1.2.1 int8 beam 5; CPU only, one +21.5 s clip, so the results are a conversion sanity check, not a benchmark): + +| model | q8 | q4 | fp16 enc + q4 dec | fp16 | CT2 int8 | +|---|---|---|---|---|---| +| whisper-small-danish | 0.109 | 0.094 | 0.094 | 0.094 | 0.078 | +| stock whisper-small | 0.297 | 0.281 | — | — | 0.234 | +| whisper-large-v3-turbo-danish | 0.141 | 0.141 | 0.094 | 0.094 | 0.094 | +| edda-v0.2 | 0.109 | 0.063 | 0.094 | 0.094 | 0.078 | +| stock whisper-large-v3-turbo | 0.125 | 0.141 | — | — | 0.094 | + +The small fine-tune cuts WER by about a factor of three. Neither turbo fine-tune separates from the +stock turbo on this easy clip, so a longer, harder Danish test is needed to judge them. **D-022 +check:** the turbo fp16 encoders do not overflow on onnxruntime-node; whole-model fp16 gives the +same transcript as fp16 + q4. WebGPU/WebNN behaviour still needs a real browser. +**Dtype defaults:** WASM q4 for all three, including small (q4 matched or beat q8 and was 1.4–1.6× +faster on CPU); WebGPU fp16 encoder + q4 decoder; WebNN fp16. All stay `verified: false`. +**edda's `num_beams: 5`:** Transformers.js 4.2 has no beam search (its `BeamSearchSampler` returns +`num_beams` candidates but generation keeps only the top-1), so browser decoding is greedy. It does +not crash; the only cost is a full-vocabulary top-k per step. The server engine (faster-whisper) does +use beam 5. +**Pending the owner:** the Hugging Face account name, then upload the six prepared folders (model +cards credit the original authors and carry MIT, or Apache-2.0 + NOTICE), then set +`DANISH_MODELS_HF_OWNER`. The integrator sets `canTranslate: false` on these entries at merge. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 336b0b4..b6bfdc4 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -67,6 +67,12 @@ Notes (M1): - [x] Language / task options for multilingual models (transcribe vs translate) - [~] Verify quantized variants (q4/q8/fp16) on both backends; record working combinations in `models.ts` +- [~] Danish fine-tunes (spec 001 Goal 6, D-WS6a): `whisper-small-danish`, + `whisper-large-v3-turbo-danish`, `edda-v0.2`. We converted them to ONNX + (fp32/fp16/q8/q4) and CT2 int8 and verified them on CPU. The catalog + entries hang off `DANISH_MODELS_HF_OWNER`. Still pending: the owner's go + for the Hugging Face upload (folders prepared, not uploaded), then the + owner constant, then browser checks of WebGPU/WebNN fp16. Notes (M2): - Quantized-variant verification is pending manual in-browser testing (no real @@ -372,6 +378,15 @@ Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29. — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the Backend select overflowed its panel on 390 px-wide phones. +Found in spec 001 workstream 6 (D-WS6a), open: +- [ ] Transformers.js 4.2 has no beam search: `num_beams > 1` in a model's + `generation_config.json` (edda-v0.2 ships 5) silently decodes greedily + and does a full-vocabulary top-k per step (D-WS6a). Not fixed: it would + mean overriding `num_beams: 1` in the worker, or a patch. +- [ ] On `da_festuge`, neither Danish turbo fine-tune beats the stock turbo + (D-WS6a). Judging them needs a longer, harder Danish reference clip than + this repo has. + ## Recurring - [ ] Maintain `AGENTS.md`: - **Add a rule only in response to a real failure.** When an agent repeatedly diff --git a/docs/plans/001-6-danish-models.md b/docs/plans/001-6-danish-models.md index 6ff4e13..9844a69 100644 --- a/docs/plans/001-6-danish-models.md +++ b/docs/plans/001-6-danish-models.md @@ -110,6 +110,6 @@ ids `` `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` `` etc. (exact name ### Task 7: Docs + finish -- [ ] `docs/Decisions.md`: append `## D-WS6a — …` (licences, layout, measurements, dtype choice, pending owner go). -- [ ] `docs/WorkBreakdown.md`: add/tick the item; "Noticed, not fixed" for anything left. -- [ ] `bun run typecheck`, fast suite, `bun run build` green; `git rm` this plan as the last commit. +- [x] `docs/Decisions.md`: append `## D-WS6a — …` (licences, layout, measurements, dtype choice, pending owner go). +- [x] `docs/WorkBreakdown.md`: add/tick the item; "Noticed, not fixed" for anything left. +- [x] `bun run typecheck`, fast suite, `bun run build` green; `git rm` this plan as the last commit. From 498e361b8a86fe1b0ea9a5eed6a5da9472a03399 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 01:13:27 +0100 Subject: [PATCH 5/5] Remove the workstream 6 plan now that the Danish models work is done Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-6-danish-models.md | 115 ------------------------------ 1 file changed, 115 deletions(-) delete mode 100644 docs/plans/001-6-danish-models.md diff --git a/docs/plans/001-6-danish-models.md b/docs/plans/001-6-danish-models.md deleted file mode 100644 index 9844a69..0000000 --- a/docs/plans/001-6-danish-models.md +++ /dev/null @@ -1,115 +0,0 @@ -# 001-6 Danish fine-tuned Whisper models Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (inline, no subagents) to -> implement this plan task-by-task. Steps use checkbox (`- [x]`) syntax for tracking. - -**Goal:** convert the Danish Whisper fine-tunes to the onnx-community (Transformers.js) and CTranslate2 -int8 layouts, verify them locally, prepare (not upload) one publication folder per would-be Hugging Face -repo, and add `verified: false` catalog entries whose ids hang off one owner constant. - -**Architecture:** all conversion work happens outside the repo (`/home/user/venvs/convert`, sources in -`/home/user/hf-src/`, outputs in `/home/user/hf-publish//`, helper scripts in -`/home/user/hf-tools/`). The repo only gains catalog data in `src/lib/models.ts`, its tests, and docs. - -**Tech Stack:** Python 3.13 venv (optimum[onnxruntime], transformers, onnx, onnxruntime, ctranslate2), -faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-node) for WER. - -**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 6; `docs/plans/001-0-coordination.md`. - -## Global Constraints - -- `flock /tmp/whisper-heavy.lock` around every download, `pip install`, conversion, WER/benchmark run. -- Venvs only in `/home/user/venvs/`. Nothing leaves the machine: no HF upload, no repo creation, no push. -- Decision placeholder `D-WS6a` (not a real `D-###`); cite it in code comments. -- Do **not** add `canTranslate` (integrator sets it to `false` for the new entries at merge). -- New entries `verified: false` on every device; ids `` `${DANISH_MODELS_HF_OWNER}/` ``. -- "Show the owner first": the final report lists repos, files, sizes, cards and WER; uploading waits for go. -- Measurements are CPU (onnxruntime-node / CTranslate2) numbers on one 21.5 s clip; say so. - -## Review Focus - -1. **Owner constant unchanged at merge** — ids must still resolve: catalog tests assert ids are built from - `DANISH_MODELS_HF_OWNER`, so changing that one line moves every id + `fasterWhisperModel` together. -2. **Language tokens lost in export** — `generation_config.json` must keep `lang_to_id["<|da|>"]` and - `task_to_id`; checked per output folder in Task 3 (script asserts). -3. **fp16 encoder overflow on large models (D-022)** — turbo fine-tunes' fp16 encoders are WER-checked on - onnxruntime-node; if garbage, the catalog's WebGPU/WebNN defaults avoid fp16 encoder (Task 5). -4. **edda's `num_beams: 5` generation config** — Transformers.js 4.2 must not crash or silently - misbehave; WER run in Task 4 uses the shipped `generation_config.json` unmodified. -5. **Server engine with a Hub-id model name** — `fasterWhisperModel` is a repo id, not an alias; existing - test that derives aliases from ids must exempt the Danish entries explicitly, not be loosened globally. - ---- - -### Task 1: Licence + architecture check (done during planning; record it) - -- [x] `thorhojhus/whisper-small-danish`: MIT (card + THIRD_PARTY_NOTICES), `WhisperForConditionalGeneration`, - unchanged whisper-small architecture/tokenizer (80 mel, 12/12 layers, vocab 51865). -- [x] `thorhojhus/whisper-large-v3-turbo-danish`: MIT, plain large-v3-turbo (128 mel, 32/4 layers, vocab 51866). -- [x] `danish-foundation-models/edda-v0.2`: Apache-2.0 (licensor: Alexandra Institute), plain - large-v3-turbo; generation config pins `language: da`, `num_beams: 5`. Kept. - -### Task 2: Conversion venv + export/quantize tooling - -**Files:** Create `/home/user/hf-tools/export_onnx.py`, `/home/user/hf-tools/quantize.py`, -`/home/user/hf-tools/check_layout.py` (outside the repo). - -- [x] `python3 -m venv /home/user/venvs/convert`; `flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common`. -- [x] Download each source repo snapshot under the lock to `/home/user/hf-src/` (`huggingface_hub.snapshot_download`, read-only). -- [x] `export_onnx.py `: optimum `main_export(task="automatic-speech-recognition-with-past", dtype fp32)` - → keep only `encoder_model.onnx(+_data)` and `decoder_model_merged.onnx(+_data)` under `/onnx/`; - copy config/tokenizer/generation/preprocessor/normalizer/added_tokens/special_tokens/vocab/merges. -- [x] `quantize.py `: for `encoder_model`, `decoder_model_merged` write `_fp16` (onnxruntime - `float16.convert_float_to_float16`, keep_io_types), `_quantized` (q8: `quantize_dynamic`, QUInt8, - op_types from stock `onnx-community/whisper-small/quantize_config.json`), `_q4` - (`MatMul4BitsQuantizer`, block 32, symmetric). Encoder > 2 GB → external data + `transformers.js_config.use_external_data_format` in config.json, as the stock turbo repo. -- [x] `check_layout.py `: asserts every file the stock repo has for those dtypes - exists, `<|da|>` id in `generation_config.json` equals the source, and onnxruntime can create a session - for each `.onnx` (ORT_ENABLE_BASIC, D-014). - -### Task 3: Convert all three (ONNX + CT2) - -- [x] For each kept model: `flock … export_onnx.py`, `flock … quantize.py`, `check_layout.py` → pass. -- [x] `flock … ct2-transformers-converter --model --output_dir /home/user/hf-publish/-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json` (+ generation config / normalizer if present). - -### Task 4: Verify (WER) - -**Files:** Create `/home/user/hf-tools/wer.ts` (Bun; imports the worktree's `tests/dsp.ts` -`wordErrorRate`; same normalisation and worker options as `tests/live-real-model.test.ts`: -`chunk_length_s 30, stride 5, return_timestamps, max_new_tokens 440, language "da"`). -Create `/home/user/hf-tools/ct2_wer.py`. - -- [x] `wer.ts --local --dtype q8|q4` with `env.localModelPath`, `env.allowRemoteModels = false`; - run under the lock for every conversion at q8 and q4, plus fp16-encoder+q4-decoder for the WebGPU default. -- [x] Same for stock `onnx-community/whisper-small` (q8, q4) and `onnx-community/whisper-large-v3-turbo` (q4, q8). -- [x] `ct2_wer.py ` with faster-whisper (language da, beam 5, CPU int8) vs stock `small` / `large-v3-turbo`. -- [x] Record wall-clock + WER table (goes into D-WS6a and the model cards). - -### Task 5: Catalog entries (TDD) - -**Files:** Modify `src/lib/models.ts`, `tests/models.test.ts`. - -**Interfaces — Produces:** `export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"`; entries with -ids `` `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` `` etc. (exact names per Task 6), -`fasterWhisperModel` = `` `${DANISH_MODELS_HF_OWNER}/-ct2-int8` ``, `multilingual: true`. - -- [x] Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and - `fasterWhisperModel` start with `${DANISH_MODELS_HF_OWNER}/`; none `verified`; `getFasterWhisperModel` - returns the CT2 repo; dtype defaults equal the measured choice; the existing alias test skips only ids - under the owner constant. -- [x] Run → fail. Implement entries. Run → pass; typecheck; commit. - -### Task 6: Publication folders + model cards - -- [x] Proposed names: `whisper-small-danish-ONNX`, `whisper-small-danish-ct2-int8`, - `whisper-large-v3-turbo-danish-ONNX`, `whisper-large-v3-turbo-danish-ct2-int8`, `edda-v0.2-ONNX`, - `edda-v0.2-ct2-int8` under ``. -- [x] Each folder gets README.md (front matter: licence carried, `base_model`, `library_name`; credit to the - original authors, conversion commands, measured WER, intended use with whisper_web), LICENSE/ - THIRD_PARTY_NOTICES from the source, and a `SIZES.txt` (not uploaded — listed for the owner). - -### Task 7: Docs + finish - -- [x] `docs/Decisions.md`: append `## D-WS6a — …` (licences, layout, measurements, dtype choice, pending owner go). -- [x] `docs/WorkBreakdown.md`: add/tick the item; "Noticed, not fixed" for anything left. -- [x] `bun run typecheck`, fast suite, `bun run build` green; `git rm` this plan as the last commit.