123 lines
7.3 KiB
Markdown
123 lines
7.3 KiB
Markdown
---
|
||
status: in-progress
|
||
---
|
||
|
||
# 001-1 — Transformers.js 4.2.0 → 4.3.2 Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (inline, no nested
|
||
> subagents — coordination plan rule). Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** run the browser engine on `@huggingface/transformers` 4.3.2 with the D-027/D-028
|
||
sequential-session patch carried over, and re-test the decisions the upgrade may obsolete.
|
||
|
||
**Architecture:** dependency bump + `bun patch` regeneration; a small runtime-guard test file pins
|
||
the installed version, the patch, and the upstream seek-loop condition D-021 depends on, so a future
|
||
bump that silently drops the patch or changes the condition fails the fast suite. Re-tests run as
|
||
scratch measurement scripts outside the repo (`/tmp/ws1/`); their numbers go into `D-WS1*` entries.
|
||
|
||
**Tech Stack:** Bun 1.3, `@huggingface/transformers` 4.3.2 (onnxruntime-web 1.31.0-dev.20260914,
|
||
onnxruntime-node 1.30.0), bun:test.
|
||
|
||
**Spec:** `docs/specs/001-transcription-quality-and-speed.md` (Goal 1); coordination rules in
|
||
`docs/plans/001-0-coordination.md`.
|
||
|
||
## Global Constraints
|
||
|
||
- Bun only; no new runtime dependency; transformers stays a devDependency (D-031).
|
||
- New decisions are `## D-WS1<letter> — …` appended at the bottom of `docs/Decisions.md`; Accepted
|
||
entries are superseded, never rewritten.
|
||
- Heavy commands (real-model runs, benchmarks, model downloads, full `bun test`) under
|
||
`flock /tmp/whisper-heavy.lock`. Ports 3110–3119. No push, no publishing.
|
||
- Measurements are CPU numbers (onnxruntime-node, or onnxruntime-web wasm EP under Bun); say which.
|
||
- No WebGPU-only default change ships on WASM numbers; WebGPU items → "needs owner's browser check".
|
||
|
||
## Review Focus
|
||
|
||
1. **Patch silently lost on reinstall** (lockfile/`patchedDependencies` key mismatch) → the app
|
||
loads fine but mobile OOM returns. Pinned by Task 1's patch-applied test.
|
||
2. **onnxruntime-web file names changed** (new version may ship different `.mjs/.wasm` names) →
|
||
`/ort/` serving and `dist/ort` copy must still contain what ort-web requests. Task 1: build and
|
||
check `dist/ort/` contains `ort-wasm-simd-threaded.jsep.{mjs,wasm}` (the WebGPU bundle's files).
|
||
3. **Upstream changes the seek-loop condition** → D-021's `max_new_tokens` could become wrong or
|
||
unneeded without notice. Pinned by Task 2's source-tripwire test.
|
||
4. **onnxruntime-node 1.30 drifts real-model WER** → `tests/live-real-model.test.ts` under the lock
|
||
(Task 5).
|
||
5. **q8 decoders on WASM at default optimization level** (D-014's failure) — Task 3 measures every
|
||
level before any change to the worker's session options.
|
||
|
||
---
|
||
|
||
### Task 1: Upgrade to 4.3.2 and carry the patch
|
||
|
||
**Files:**
|
||
- Create: `tests/transformers-runtime.test.ts`
|
||
- Modify: `package.json`, `bun.lock`, `AGENTS.md` (patch path), `docs/WorkBreakdown.md`
|
||
- Create: `patches/@huggingface%2Ftransformers@4.3.2.patch`; Delete: the 4.2.0 patch
|
||
|
||
**Interfaces:**
|
||
- Produces: `tests/transformers-runtime.test.ts` (Task 2 adds a case to it).
|
||
|
||
- [x] **Step 1: Write the failing test** `describe("@huggingface/transformers runtime")`:
|
||
- `"installed version is the pinned 4.3.2"` — `node_modules/@huggingface/transformers/package.json`
|
||
`.version === "4.3.2"`.
|
||
- `"patchedDependencies points at an existing patch for the installed version"` — root
|
||
`package.json` `patchedDependencies["@huggingface/transformers@4.3.2"]` is a path that exists.
|
||
- `"web bundle's constructSessions carries the sequential-load patch (D-027/D-028)"` — the
|
||
`constructSessions` function text sliced from `dist/transformers.web.js` contains
|
||
`options?.sequentialSessionLoad` and both `Promise.all(` and `for (const name of Object.keys(names))`.
|
||
- [x] **Step 2: Run** `bun test tests/transformers-runtime.test.ts` → FAIL on version (4.2.0) and key.
|
||
- [x] **Step 3:** `bun add -d @huggingface/transformers@4.3.2` (remove the 4.2.0
|
||
`patchedDependencies` entry first if bun refuses), `bun patch @huggingface/transformers`, re-apply
|
||
the 4.2.0 hunk to `constructSessions` in `dist/transformers.web.js` (upstream is unchanged:
|
||
still a bare `Promise.all`), `bun patch --commit node_modules/@huggingface/transformers`.
|
||
`git rm` the 4.2.0 patch. Re-link the shared `.cache` symlink if the reinstall removed it.
|
||
- [x] **Step 4: Run** the test → PASS; `bun install --frozen-lockfile` clean; typecheck; fast
|
||
suite; `bun run build`; `ls dist/ort` shows the files ort-web 1.31 requests.
|
||
- [x] **Step 5:** docs: AGENTS.md patch filename; `D-WS1a` (upgrade, patch carried, why upstream
|
||
doesn't obsolete it, ort versions, Parakeet = CTC only); WorkBreakdown tick. Commit
|
||
"Upgrade @huggingface/transformers to 4.3.2 and carry the sequential-session patch".
|
||
|
||
### Task 2: D-021 re-test (seek loop)
|
||
|
||
**Files:** Modify `tests/transformers-runtime.test.ts`, `docs/Decisions.md`,
|
||
`src/worker/inference.worker.ts` (comment only: "4.2" → "4.2–4.3", cite D-WS1b).
|
||
|
||
- [x] **Step 1: Test** `"Whisper generate still takes the seek loop when max_new_tokens is unset (D-021)"`
|
||
— `src/models/whisper/modeling_whisper.js` contains
|
||
`if (generation_config.return_timestamps && !kwargs.max_new_tokens)`. Run → PASS expected
|
||
(tripwire; it encodes the revisit condition, verified against the 4.3.2 source).
|
||
- [x] **Step 2: Measure** (`/tmp/ws1/d021.ts`, onnxruntime-node, under the lock): tiny.en q8 on
|
||
`en_alien.opus` (ffmpeg-static → 16 kHz f32), pipeline options as the worker
|
||
(`chunk_length_s 30, stride_length_s 5, return_timestamps true`), with and without
|
||
`max_new_tokens: 440`; record wall-clock, generate-pass count (wrap `model.generate`), window
|
||
count, WER vs `en_alien.txt` (`tests/dsp.ts` `wordErrorRate`).
|
||
- [x] **Step 3:** `D-WS1b` with the numbers; keep `max_new_tokens: 440` unless the seek path is now
|
||
both faster and not worse on WER. Commit.
|
||
|
||
### Task 3: D-014 re-test (WASM graphOptimizationLevel)
|
||
|
||
**Files:** `docs/Decisions.md`; `src/worker/inference.worker.ts` only if the outcome is "drop".
|
||
|
||
- [ ] **Step 1: Measure** (`/tmp/ws1/d014.ts`): `globalThis[Symbol.for("onnxruntime")] =
|
||
await import("onnxruntime-web")` before importing transformers, wasm EP, 1 thread; for
|
||
`onnx-community/whisper-tiny` `decoder_model_merged_quantized.onnx` create a session at
|
||
`disabled/basic/extended/all`; record success/error. If all load: transcribe `en_aiproblem` with
|
||
tiny.en q8 at `basic` vs default (`all`), 3 warm runs each, wall-clock + text identity.
|
||
- [ ] **Step 2:** If default fails → keep, `D-WS1c` "still needed on ort-web 1.31.0-dev". If it
|
||
loads and is not slower → remove the override (worker comment cites D-WS1c), run a q8 transcribe
|
||
again through the harness with the worker's new options. Commit.
|
||
|
||
### Task 4: WebGPU-affecting changes (D-022, D-025/D-026) — inspection only
|
||
|
||
- [x] **Step 1:** Read 4.3.0/4.3.1/4.3.2 release notes and diff 4.2.0↔4.3.2 `src/` for
|
||
WebGPU/fp16/Firefox/Safari changes; check ort-web 1.31 dist for Gecko/fp16 handling.
|
||
- [x] **Step 2:** `D-WS1d` listing each item as "needs owner's browser check" with what to try
|
||
(explicit WebGPU on Firefox for D-026; medium fp16 for D-022; Safari 26 WebGPU). No code change.
|
||
Commit.
|
||
|
||
### Task 5: Verify and finish
|
||
|
||
- [ ] typecheck, fast suite, build green; `flock /tmp/whisper-heavy.lock bun test
|
||
tests/live-real-model.test.ts` green; record WER per clip in D-WS1a if changed from the bars.
|
||
- [ ] `git status` clean; last commit `git rm docs/plans/001-1-transformers-upgrade.md`.
|