diff --git a/docs/plans/001-6-danish-models.md b/docs/plans/001-6-danish-models.md new file mode 100644 index 0000000..c573f4a --- /dev/null +++ b/docs/plans/001-6-danish-models.md @@ -0,0 +1,115 @@ +# 001-6 Danish fine-tuned Whisper models Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (inline, no subagents) to +> implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** convert the Danish Whisper fine-tunes to the onnx-community (Transformers.js) and CTranslate2 +int8 layouts, verify them locally, prepare (not upload) one publication folder per would-be Hugging Face +repo, and add `verified: false` catalog entries whose ids hang off one owner constant. + +**Architecture:** all conversion work happens outside the repo (`/home/user/venvs/convert`, sources in +`/home/user/hf-src/`, outputs in `/home/user/hf-publish//`, helper scripts in +`/home/user/hf-tools/`). The repo only gains catalog data in `src/lib/models.ts`, its tests, and docs. + +**Tech Stack:** Python 3.13 venv (optimum[onnxruntime], transformers, onnx, onnxruntime, ctranslate2), +faster-whisper venv for CT2 checks, Bun + Transformers.js 4.2.0 (onnxruntime-node) for WER. + +**Spec:** `docs/specs/001-transcription-quality-and-speed.md` Goal 6; `docs/plans/001-0-coordination.md`. + +## Global Constraints + +- `flock /tmp/whisper-heavy.lock` around every download, `pip install`, conversion, WER/benchmark run. +- Venvs only in `/home/user/venvs/`. Nothing leaves the machine: no HF upload, no repo creation, no push. +- Decision placeholder `D-WS6a` (not a real `D-###`); cite it in code comments. +- Do **not** add `canTranslate` (integrator sets it to `false` for the new entries at merge). +- New entries `verified: false` on every device; ids `` `${DANISH_MODELS_HF_OWNER}/` ``. +- "Show the owner first": the final report lists repos, files, sizes, cards and WER; uploading waits for go. +- Measurements are CPU (onnxruntime-node / CTranslate2) numbers on one 21.5 s clip; say so. + +## Review Focus + +1. **Owner constant unchanged at merge** — ids must still resolve: catalog tests assert ids are built from + `DANISH_MODELS_HF_OWNER`, so changing that one line moves every id + `fasterWhisperModel` together. +2. **Language tokens lost in export** — `generation_config.json` must keep `lang_to_id["<|da|>"]` and + `task_to_id`; checked per output folder in Task 3 (script asserts). +3. **fp16 encoder overflow on large models (D-022)** — turbo fine-tunes' fp16 encoders are WER-checked on + onnxruntime-node; if garbage, the catalog's WebGPU/WebNN defaults avoid fp16 encoder (Task 5). +4. **edda's `num_beams: 5` generation config** — Transformers.js 4.2 must not crash or silently + misbehave; WER run in Task 4 uses the shipped `generation_config.json` unmodified. +5. **Server engine with a Hub-id model name** — `fasterWhisperModel` is a repo id, not an alias; existing + test that derives aliases from ids must exempt the Danish entries explicitly, not be loosened globally. + +--- + +### Task 1: Licence + architecture check (done during planning; record it) + +- [x] `thorhojhus/whisper-small-danish`: MIT (card + THIRD_PARTY_NOTICES), `WhisperForConditionalGeneration`, + unchanged whisper-small architecture/tokenizer (80 mel, 12/12 layers, vocab 51865). +- [x] `thorhojhus/whisper-large-v3-turbo-danish`: MIT, plain large-v3-turbo (128 mel, 32/4 layers, vocab 51866). +- [x] `danish-foundation-models/edda-v0.2`: Apache-2.0 (licensor: Alexandra Institute), plain + large-v3-turbo; generation config pins `language: da`, `num_beams: 5`. Kept. + +### Task 2: Conversion venv + export/quantize tooling + +**Files:** Create `/home/user/hf-tools/export_onnx.py`, `/home/user/hf-tools/quantize.py`, +`/home/user/hf-tools/check_layout.py` (outside the repo). + +- [ ] `python3 -m venv /home/user/venvs/convert`; `flock … pip install "optimum[onnxruntime]" transformers onnx onnxruntime ctranslate2 onnxconverter-common`. +- [ ] Download each source repo snapshot under the lock to `/home/user/hf-src/` (`huggingface_hub.snapshot_download`, read-only). +- [ ] `export_onnx.py `: optimum `main_export(task="automatic-speech-recognition-with-past", dtype fp32)` + → keep only `encoder_model.onnx(+_data)` and `decoder_model_merged.onnx(+_data)` under `/onnx/`; + copy config/tokenizer/generation/preprocessor/normalizer/added_tokens/special_tokens/vocab/merges. +- [ ] `quantize.py `: for `encoder_model`, `decoder_model_merged` write `_fp16` (onnxruntime + `float16.convert_float_to_float16`, keep_io_types), `_quantized` (q8: `quantize_dynamic`, QUInt8, + op_types from stock `onnx-community/whisper-small/quantize_config.json`), `_q4` + (`MatMul4BitsQuantizer`, block 32, symmetric). Encoder > 2 GB → external data + `transformers.js_config.use_external_data_format` in config.json, as the stock turbo repo. +- [ ] `check_layout.py `: asserts every file the stock repo has for those dtypes + exists, `<|da|>` id in `generation_config.json` equals the source, and onnxruntime can create a session + for each `.onnx` (ORT_ENABLE_BASIC, D-014). + +### Task 3: Convert all three (ONNX + CT2) + +- [ ] For each kept model: `flock … export_onnx.py`, `flock … quantize.py`, `check_layout.py` → pass. +- [ ] `flock … ct2-transformers-converter --model --output_dir /home/user/hf-publish/-ct2-int8 --quantization int8 --copy_files tokenizer.json preprocessor_config.json` (+ generation config / normalizer if present). + +### Task 4: Verify (WER) + +**Files:** Create `/home/user/hf-tools/wer.ts` (Bun; imports the worktree's `tests/dsp.ts` +`wordErrorRate`; same normalisation and worker options as `tests/live-real-model.test.ts`: +`chunk_length_s 30, stride 5, return_timestamps, max_new_tokens 440, language "da"`). +Create `/home/user/hf-tools/ct2_wer.py`. + +- [ ] `wer.ts --local --dtype q8|q4` with `env.localModelPath`, `env.allowRemoteModels = false`; + run under the lock for every conversion at q8 and q4, plus fp16-encoder+q4-decoder for the WebGPU default. +- [ ] Same for stock `onnx-community/whisper-small` (q8, q4) and `onnx-community/whisper-large-v3-turbo` (q4, q8). +- [ ] `ct2_wer.py ` with faster-whisper (language da, beam 5, CPU int8) vs stock `small` / `large-v3-turbo`. +- [ ] Record wall-clock + WER table (goes into D-WS6a and the model cards). + +### Task 5: Catalog entries (TDD) + +**Files:** Modify `src/lib/models.ts`, `tests/models.test.ts`. + +**Interfaces — Produces:** `export const DANISH_MODELS_HF_OWNER = "PENDING-OWNER"`; entries with +ids `` `${DANISH_MODELS_HF_OWNER}/whisper-small-danish-ONNX` `` etc. (exact names per Task 6), +`fasterWhisperModel` = `` `${DANISH_MODELS_HF_OWNER}/-ct2-int8` ``, `multilingual: true`. + +- [ ] Failing tests: lineup includes the Danish ids (built from the constant); every Danish entry's id and + `fasterWhisperModel` start with `${DANISH_MODELS_HF_OWNER}/`; none `verified`; `getFasterWhisperModel` + returns the CT2 repo; dtype defaults equal the measured choice; the existing alias test skips only ids + under the owner constant. +- [ ] Run → fail. Implement entries. Run → pass; typecheck; commit. + +### Task 6: Publication folders + model cards + +- [ ] Proposed names: `whisper-small-danish-ONNX`, `whisper-small-danish-ct2-int8`, + `whisper-large-v3-turbo-danish-ONNX`, `whisper-large-v3-turbo-danish-ct2-int8`, `edda-v0.2-ONNX`, + `edda-v0.2-ct2-int8` under ``. +- [ ] Each folder gets README.md (front matter: licence carried, `base_model`, `library_name`; credit to the + original authors, conversion commands, measured WER, intended use with whisper_web), LICENSE/ + THIRD_PARTY_NOTICES from the source, and a `SIZES.txt` (not uploaded — listed for the owner). + +### Task 7: Docs + finish + +- [ ] `docs/Decisions.md`: append `## D-WS6a — …` (licences, layout, measurements, dtype choice, pending owner go). +- [ ] `docs/WorkBreakdown.md`: add/tick the item; "Noticed, not fixed" for anything left. +- [ ] `bun run typecheck`, fast suite, `bun run build` green; `git rm` this plan as the last commit.