whisper_web/docs/plans/001-1-transformers-upgrade.md

status

status
in-progress

001-1 — Transformers.js 4.2.0 → 4.3.2 Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no nested subagents — coordination plan rule). Steps use checkbox (- [ ]) syntax for tracking.

Goal: run the browser engine on @huggingface/transformers 4.3.2 with the D-027/D-028 sequential-session patch carried over, and re-test the decisions the upgrade may obsolete.

Architecture: dependency bump + bun patch regeneration; a small runtime-guard test file pins the installed version, the patch, and the upstream seek-loop condition D-021 depends on, so a future bump that silently drops the patch or changes the condition fails the fast suite. Re-tests run as scratch measurement scripts outside the repo (/tmp/ws1/); their numbers go into D-WS1* entries.

Tech Stack: Bun 1.3, @huggingface/transformers 4.3.2 (onnxruntime-web 1.31.0-dev.20260914, onnxruntime-node 1.30.0), bun:test.

Spec: docs/specs/001-transcription-quality-and-speed.md (Goal 1); coordination rules in docs/plans/001-0-coordination.md.

Global Constraints

• Bun only; no new runtime dependency; transformers stays a devDependency (D-031).
• New decisions are ## D-WS1<letter> — … appended at the bottom of docs/Decisions.md; Accepted entries are superseded, never rewritten.
• Heavy commands (real-model runs, benchmarks, model downloads, full bun test) under flock /tmp/whisper-heavy.lock. Ports 3110–3119. No push, no publishing.
• Measurements are CPU numbers (onnxruntime-node, or onnxruntime-web wasm EP under Bun); say which.
• No WebGPU-only default change ships on WASM numbers; WebGPU items → "needs owner's browser check".

Review Focus

1. Patch silently lost on reinstall (lockfile/patchedDependencies key mismatch) → the app loads fine but mobile OOM returns. Pinned by Task 1's patch-applied test.
2. onnxruntime-web file names changed (new version may ship different .mjs/.wasm names) → /ort/ serving and dist/ort copy must still contain what ort-web requests. Task 1: build and check dist/ort/ contains ort-wasm-simd-threaded.jsep.{mjs,wasm} (the WebGPU bundle's files).
3. Upstream changes the seek-loop condition → D-021's max_new_tokens could become wrong or unneeded without notice. Pinned by Task 2's source-tripwire test.
4. onnxruntime-node 1.30 drifts real-model WER → tests/live-real-model.test.ts under the lock (Task 5).
5. q8 decoders on WASM at default optimization level (D-014's failure) — Task 3 measures every level before any change to the worker's session options.

---

Task 1: Upgrade to 4.3.2 and carry the patch

Files:

• Create: tests/transformers-runtime.test.ts
• Modify: package.json, bun.lock, AGENTS.md (patch path), docs/WorkBreakdown.md
• Create: patches/@huggingface%2Ftransformers@4.3.2.patch; Delete: the 4.2.0 patch

Interfaces:

• Produces: tests/transformers-runtime.test.ts (Task 2 adds a case to it).

• Step 1: Write the failing test describe("@huggingface/transformers runtime"):

  • "installed version is the pinned 4.3.2" — node_modules/@huggingface/transformers/package.json .version === "4.3.2".
  • "patchedDependencies points at an existing patch for the installed version" — root package.json patchedDependencies["@huggingface/transformers@4.3.2"] is a path that exists.
  • "web bundle's constructSessions carries the sequential-load patch (D-027/D-028)" — the constructSessions function text sliced from dist/transformers.web.js contains options?.sequentialSessionLoad and both Promise.all( and for (const name of Object.keys(names)).

• Step 2: Run bun test tests/transformers-runtime.test.ts → FAIL on version (4.2.0) and key.

• Step 3: bun add -d @huggingface/transformers@4.3.2 (remove the 4.2.0 patchedDependencies entry first if bun refuses), bun patch @huggingface/transformers, re-apply the 4.2.0 hunk to constructSessions in dist/transformers.web.js (upstream is unchanged: still a bare Promise.all), bun patch --commit node_modules/@huggingface/transformers. git rm the 4.2.0 patch. Re-link the shared .cache symlink if the reinstall removed it.

• Step 4: Run the test → PASS; bun install --frozen-lockfile clean; typecheck; fast suite; bun run build; ls dist/ort shows the files ort-web 1.31 requests.

• Step 5: docs: AGENTS.md patch filename; D-WS1a (upgrade, patch carried, why upstream doesn't obsolete it, ort versions, Parakeet = CTC only); WorkBreakdown tick. Commit "Upgrade @huggingface/transformers to 4.3.2 and carry the sequential-session patch".

Task 2: D-021 re-test (seek loop)

Files: Modify tests/transformers-runtime.test.ts, docs/Decisions.md, src/worker/inference.worker.ts (comment only: "4.2" → "4.2–4.3", cite D-WS1b).

• Step 1: Test "Whisper generate still takes the seek loop when max_new_tokens is unset (D-021)" — src/models/whisper/modeling_whisper.js contains if (generation_config.return_timestamps && !kwargs.max_new_tokens). Run → PASS expected (tripwire; it encodes the revisit condition, verified against the 4.3.2 source).
• Step 2: Measure (/tmp/ws1/d021.ts, onnxruntime-node, under the lock): tiny.en q8 on en_alien.opus (ffmpeg-static → 16 kHz f32), pipeline options as the worker (chunk_length_s 30, stride_length_s 5, return_timestamps true), with and without max_new_tokens: 440; record wall-clock, generate-pass count (wrap model.generate), window count, WER vs en_alien.txt (tests/dsp.ts wordErrorRate).
• Step 3: D-WS1b with the numbers; keep max_new_tokens: 440 unless the seek path is now both faster and not worse on WER. Commit.

Task 3: D-014 re-test (WASM graphOptimizationLevel)

Files: docs/Decisions.md; src/worker/inference.worker.ts only if the outcome is "drop".

• Step 1: Measure (/tmp/ws1/d014.ts): globalThis[Symbol.for("onnxruntime")] = await import("onnxruntime-web") before importing transformers, wasm EP, 1 thread; for onnx-community/whisper-tiny decoder_model_merged_quantized.onnx create a session at disabled/basic/extended/all; record success/error. If all load: transcribe en_aiproblem with tiny.en q8 at basic vs default (all), 3 warm runs each, wall-clock + text identity.
• Step 2: If default fails → keep, D-WS1c "still needed on ort-web 1.31.0-dev". If it loads and is not slower → remove the override (worker comment cites D-WS1c), run a q8 transcribe again through the harness with the worker's new options. Commit.

Task 4: WebGPU-affecting changes (D-022, D-025/D-026) — inspection only

• Step 1: Read 4.3.0/4.3.1/4.3.2 release notes and diff 4.2.0↔4.3.2 src/ for WebGPU/fp16/Firefox/Safari changes; check ort-web 1.31 dist for Gecko/fp16 handling.
• Step 2: D-WS1d listing each item as "needs owner's browser check" with what to try (explicit WebGPU on Firefox for D-026; medium fp16 for D-022; Safari 26 WebGPU). No code change. Commit.

Task 5: Verify and finish

• typecheck, fast suite, build green; flock /tmp/whisper-heavy.lock bun test tests/live-real-model.test.ts green; record WER per clip in D-WS1a if changed from the bars.
• git status clean; last commit git rm docs/plans/001-1-transformers-upgrade.md.