whisper_web/docs/plans/001-6-danish-models.md

001-6 Danish fine-tuned Whisper models Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans (inline, no subagents) to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: convert the Danish Whisper fine-tunes to the onnx-community (Transformers.js) and CTranslate2 int8 layouts, verify them locally, prepare (not upload) one publication folder per would-be Hugging Face repo, and add verified: false catalog entries whose ids hang off one owner constant.

Architecture: all conversion work happens outside the repo (/home/user/venvs/convert, sources in /home/user/hf-src/<repo>, outputs in /home/user/hf-publish/<repo-name>/, helper scripts in /home/user/hf-tools/). The repo only gains catalog data in src/lib/models.ts, its tests, and docs.

Tech Stack: Python 3.13 venv (optimum[onnxruntime], transformers, onnx, onnxruntime, ctranslate2), faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-node) for WER.

Spec: docs/specs/001-transcription-quality-and-speed.md Goal 6; docs/plans/001-0-coordination.md.

Global Constraints

• flock /tmp/whisper-heavy.lock around every download, pip install, conversion, WER/benchmark run.
• Venvs only in /home/user/venvs/<name>. Nothing leaves the machine: no HF upload, no repo creation, no push.
• Decision placeholder D-WS6a (not a real D-###); cite it in code comments.
• Do not add canTranslate (integrator sets it to false for the new entries at merge).
• New entries verified: false on every device; ids `${DANISH_MODELS_HF_OWNER}/<repo>`.
• "Show the owner first": the final report lists repos, files, sizes, cards and WER; uploading waits for go.
• Measurements are CPU (onnxruntime-node / CTranslate2) numbers on one 21.5 s clip; say so.

Review Focus

1. Owner constant unchanged at merge — ids must still resolve: catalog tests assert ids are built from DANISH_MODELS_HF_OWNER, so changing that one line moves every id + fasterWhisperModel together.
2. Language tokens lost in export — generation_config.json must keep lang_to_id["<|da|>"] and task_to_id; checked per output folder in Task 3 (script asserts).
3. fp16 encoder overflow on large models (D-022) — turbo fine-tunes' fp16 encoders are WER-checked on onnxruntime-node; if garbage, the catalog's WebGPU/WebNN defaults avoid fp16 encoder (Task 5).
4. edda's num_beams: 5 generation config — Transformers.js 4.2 must not crash or silently misbehave; WER run in Task 4 uses the shipped generation_config.json unmodified.
5. Server engine with a Hub-id model name — fasterWhisperModel is a repo id, not an alias; existing test that derives aliases from ids must exempt the Danish entries explicitly, not be loosened globally.

---

Task 1: Licence + architecture check (done during planning; record it)

• thorhojhus/whisper-small-danish: MIT (card + THIRD_PARTY_NOTICES), WhisperForConditionalGeneration, unchanged whisper-small architecture/tokenizer (80 mel, 12/12 layers, vocab 51865).
• thorhojhus/whisper-large-v3-turbo-danish: MIT, plain large-v3-turbo (128 mel, 32/4 layers, vocab 51866).
• danish-foundation-models/edda-v0.2: Apache-2.0 (licensor: Alexandra Institute), plain large-v3-turbo; generation config pins language: da, num_beams: 5. Kept.

Task 2: Conversion venv + export/quantize tooling

Files: Create /home/user/hf-tools/export_onnx.py, /home/user/hf-tools/quantize.py, /home/user/hf-tools/check_layout.py (outside the repo).

• python3 -m venv /home/user/venvs/convert; flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common.
• Download each source repo snapshot under the lock to /home/user/hf-src/<repo> (huggingface_hub.snapshot_download, read-only).
• export_onnx.py <src> <out>: optimum main_export(task="automatic-speech-recognition-with-past", dtype fp32) → keep only encoder_model.onnx(+_data) and decoder_model_merged.onnx(+_data) under <out>/onnx/; copy config/tokenizer/generation/preprocessor/normalizer/added_tokens/special_tokens/vocab/merges.
• quantize.py <out>: for encoder_model, decoder_model_merged write _fp16 (onnxruntime float16.convert_float_to_float16, keep_io_types), _quantized (q8: quantize_dynamic, QUInt8, op_types from stock onnx-community/whisper-small/quantize_config.json), _q4 (MatMul4BitsQuantizer, block 32, symmetric). Encoder > 2 GB → external data + transformers.js_config.use_external_data_format in config.json, as the stock turbo repo.
• check_layout.py <out> <stock-file-list>: asserts every file the stock repo has for those dtypes exists, <|da|> id in generation_config.json equals the source, and onnxruntime can create a session for each .onnx (ORT_ENABLE_BASIC, D-014).

Task 3: Convert all three (ONNX + CT2)

• For each kept model: flock … export_onnx.py, flock … quantize.py, check_layout.py → pass.
• flock … ct2-transformers-converter --model <src> --output_dir /home/user/hf-publish/<name>-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json (+ generation config / normalizer if present).

Task 4: Verify (WER)

Files: Create /home/user/hf-tools/wer.ts (Bun; imports the worktree's tests/dsp.ts wordErrorRate; same normalisation and worker options as tests/live-real-model.test.ts: chunk_length_s 30, stride 5, return_timestamps, max_new_tokens 440, language "da"). Create /home/user/hf-tools/ct2_wer.py.

• wer.ts --local <dir> --dtype q8|q4 with env.localModelPath, env.allowRemoteModels = false; run under the lock for every conversion at q8 and q4, plus fp16-encoder+q4-decoder for the WebGPU default.
• Same for stock onnx-community/whisper-small (q8, q4) and onnx-community/whisper-large-v3-turbo (q4, q8).
• ct2_wer.py <ct2 dir> with faster-whisper (language da, beam 5, CPU int8) vs stock small / large-v3-turbo.
• Record wall-clock + WER table (goes into D-WS6a and the model cards).

Task 5: Catalog entries (TDD)

Files: Modify src/lib/models.ts, tests/models.test.ts.

Interfaces — Produces: export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"; entries with ids `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` etc. (exact names per Task 6), fasterWhisperModel = `${DANISH_MODELS_HF_OWNER}/<name>-ct2-int8`, multilingual: true.

• Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and fasterWhisperModel start with ${DANISH_MODELS_HF_OWNER}/; none verified; getFasterWhisperModel returns the CT2 repo; dtype defaults equal the measured choice; the existing alias test skips only ids under the owner constant.
• Run → fail. Implement entries. Run → pass; typecheck; commit.

Task 6: Publication folders + model cards

• Proposed names: whisper-small-danish-ONNX, whisper-small-danish-ct2-int8, whisper-large-v3-turbo-danish-ONNX, whisper-large-v3-turbo-danish-ct2-int8, edda-v0.2-ONNX, edda-v0.2-ct2-int8 under <HF_OWNER>.
• Each folder gets README.md (front matter: licence carried, base_model, library_name; credit to the original authors, conversion commands, measured WER, intended use with whisper_web), LICENSE/ THIRD_PARTY_NOTICES from the source, and a SIZES.txt (not uploaded — listed for the owner).

Task 7: Docs + finish

• docs/Decisions.md: append ## D-WS6a — … (licences, layout, measurements, dtype choice, pending owner go).
• docs/WorkBreakdown.md: add/tick the item; "Noticed, not fixed" for anything left.
• bun run typecheck, fast suite, bun run build green; git rm this plan as the last commit.