whisper_web/docs/Decisions.md

# Decisions

Lightweight ADR log. One entry per decision that shapes the codebase. Statuses:
**Accepted** (act on it), **Proposed** (default choice, revisit before building on it),
**Superseded** (kept for history, points to replacement).

Add new entries at the bottom with the next `D-###` id. Never rewrite an Accepted
entry to mean something else — supersede it.

---

## D-001 — Bun as the only runtime and toolchain
**Status:** Accepted (mandated by CLAUDE.md)
Bun replaces Node, vite/webpack (via `bun build` + HTML imports), jest/vitest
(`bun test`), express (`Bun.serve`), better-sqlite3 (`bun:sqlite`), and dotenv.
**Consequence:** no vite plugin ecosystem; PWA assets (service worker, manifest) are
wired by hand instead of via a plugin.

## D-002 — Transformers.js (`@huggingface/transformers`) for all Whisper inference
**Status:** Accepted for browser inference; the server side is **superseded by D-013**
(faster-whisper).
Same library in the browser (WebGPU/WASM) and on the server (ONNX Runtime under Bun).
**Why:** one model catalog, one pipeline API, no native whisper.cpp build step.
**Trade-off:** server CPU inference is slower than whisper.cpp; acceptable — the server
engine is a convenience for logged-in users, not the primary path.

## D-003 — Inference runs in a dedicated Web Worker
**Status:** Accepted
Model load + transcription block for seconds-to-minutes; running them on the main
thread would freeze the UI. Communication via a small typed postMessage protocol
(`src/worker/protocol.ts`) shared between UI and worker.

## D-004 — WebGPU with automatic WASM fallback
**Status:** Accepted (amended by D-017 — "auto" now probes WebNN first)
Detect via `navigator.gpu?.requestAdapter()` inside the worker; pass
`device: "webgpu" | "wasm"` to the pipeline. The active backend is surfaced in the UI
but users don't have to configure anything. Encoder-heavy Whisper benefits ~10-40× from
WebGPU where available (Chromium); WASM keeps Firefox/Safari working.

## D-005 — Model weights cached via the browser Cache API (Transformers.js default)
**Status:** Accepted
Transformers.js already persists Hub downloads in the Cache API (`useBrowserCache`).
We do **not** re-cache weights in the service worker — the SW precaches only the app
shell. **Consequence:** offline = installed PWA + at least one previously loaded model;
the UI must show per-model "downloaded / not downloaded" state (query
`caches` for hub URLs) so users can prepare for offline use.

## D-006 — Quantized model catalog, curated not exhaustive
**Status:** Proposed
Expose a curated list (`src/lib/models.ts`): `whisper-tiny(.en)`, `base(.en)`,
`small(.en)`, `medium(.en)`, `large-v3-turbo`, each with sensible dtype defaults
(e.g. q4/q8 on WASM, fp16/q4 on WebGPU) and a size estimate shown before download.
**Why not free-form model ids:** predictable UX, tested combinations, honest
size/VRAM warnings. Revisit if users ask for arbitrary Hub ids.

## D-007 — PWA: hand-rolled service worker, app-shell precache
**Status:** Accepted
No Workbox/vite-plugin-pwa (conflicts with D-001 tooling). `bun build` emits hashed
assets to `dist/`; a small build step writes the precache manifest into `sw.ts`.
Strategy: precache app shell, cache-first for static assets, network-only for `/api/*`.

## D-008 — Passport.js via a connect-compat adapter over Bun.serve
**Status:** Accepted (Passport mandated by CLAUDE.md; adapter is the how)
Passport expects connect/express-style `(req, res, next)` and `req.login()`. We are
not allowed to use express (D-001), so `src/server/auth.ts` provides a thin adapter:
wrap Bun's `Request` in a minimal req/res shim, invoke the passport authenticate
chain, translate the outcome back to a `Response`. Local (username/password)
strategy first; other strategies plug into the same adapter later.
**Alternative rejected:** dropping Passport for hand-rolled auth — CLAUDE.md names
Passport explicitly.

## D-009 — Sessions + users in `bun:sqlite`
**Status:** Accepted
Single-file SQLite DB (path from env, default `./data/app.db`). Tables: `users`
(id, username, password_hash — Argon2 via `Bun.password`), `sessions`
(id, user_id, expires_at). Cookie: HttpOnly, SameSite=Lax, Secure in production.
No Redis/Postgres until there's a scaling reason (would be `Bun.redis` / `Bun.sql`).

## D-010 — Engine abstraction: `BrowserEngine` and `ServerEngine` behind one interface
**Status:** Accepted
`src/lib/engine.ts` defines `Engine { load(model), transcribe(audio), onProgress }`.
The UI never branches on engine type beyond the toggle. Keeps the server engine
(milestone M5) from leaking into UI code built in M1–M4.

## D-011 — Server engine transport: streaming HTTP response first, WebSocket if needed
**Status:** Accepted (implemented in M5, 2026-07-10)
`POST /api/transcribe` returns a streamed (chunked JSON-lines) response with progress
and partial transcripts. Simpler than WebSockets (no connection lifecycle, works with
plain fetch + ReadableStream). Switch to `Bun.serve` websockets only if we need
bidirectional control (e.g. cancel mid-run) that abort signals can't cover.
**As implemented:** `application/x-ndjson` of `segment`* then `result` | `error`
lines (wire types + guards shared client/server via `src/lib/engine.ts`); the
request body is raw float32 mono 16 kHz PCM — the decoded form the browser
already has — so the server never decodes containers. Cancel-mid-run needed no
WebSocket: `fetch`'s AbortSignal reaches `request.signal`, which forwards a
`cancel` to the sidecar (it checks between segments).

## D-012 — Audio decoding on the main thread via Web Audio API
**Status:** Accepted
`decodeAudioData` (handles most audio and audio-in-video containers) → mono downmix →
resample to 16 kHz → transfer Float32Array to worker (zero-copy). No ffmpeg.wasm
unless we hit real container-support gaps; note in `src/lib/audio.ts` if we do.

## D-013 — Server engine runs faster-whisper (Python sidecar)
**Status:** Accepted (user requirement, 2026-07-10; supersedes the server half of D-002)
Server-side inference (M5) uses [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
(CTranslate2) instead of Transformers.js under Bun. Browser inference is untouched —
faster-whisper **cannot** run in a browser (checked 2026-07): it is Python over native
C++ CTranslate2, which has no WASM build (OpenNMT/CTranslate2#528 open since 2021, no
progress) and relies on native SIMD/threading kernels. So D-002 stays Accepted for the
browser, where Transformers.js remains the engine.
**Shape:** Bun stays the only web runtime. `src/server/transcribe.ts` talks to a small
Python sidecar (`server-py/`, faster-whisper behind a minimal stdio or localhost-HTTP
interface) that `Bun.spawn` starts and supervises. The `POST /api/transcribe` contract
(D-011: streamed JSON-lines) is unchanged — faster-whisper's segment generator maps
naturally onto it.
**Consequence:** the "one inference codebase" invariant narrows to "one *browser*
inference codebase + one shared model catalog"; `src/lib/models.ts` needs a mapping
from catalog ids to faster-whisper/CTranslate2 model names (`Systran/faster-whisper-*`);
deployment gains a Python ≥3.9 dependency (server engine only — the anonymous/browser
path still deploys with Bun alone, and the app must degrade gracefully when the
sidecar is absent).
**Why:** CTranslate2 CPU inference is several times faster and more memory-efficient
than ONNX Runtime under Bun; the server engine becomes genuinely useful rather than a
slow convenience.

## D-014 — WASM sessions run at graphOptimizationLevel "basic"
**Status:** **Superseded by D-WS1c** (bug workaround, 2026-07-10; the ort-web
build in Transformers.js 4.3.2 no longer has the bug, so the cap is gone)
On the WASM backend, the inference worker passes
`session_options: { graphOptimizationLevel: "basic" }` to the Transformers.js
pipeline. The onnxruntime-web build bundled with Transformers.js 4.2
(1.26.0-dev.20260416) has a broken level-2 QDQ graph transform
(`qdq_actions.cc` `TransposeDQWeightsForMatMulNBits`) that fails session
creation for the 8-bit Whisper decoders (`_quantized`/`_int8`/`_uint8`), whose
tied `embed_tokens` weight is both gathered (embedding) and matmul'd (lm_head):
"Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale".
Reproduced outside the browser against `decoder_model_merged_quantized.onnx`
of `onnx-community/whisper-tiny`; levels "all"/"extended" fail, "basic"/
"disabled" load fine, and onnxruntime-node 1.24.3 loads the same file at every
level (regression in the ort-web dev build, not the model).
**Scope:** WASM only — WebGPU keeps the default level. The q4 conversions use
native `MatMulNBits` ops (no QDQ), so skipping level-2 costs the larger WASM
models nothing.
**Revisit:** when Transformers.js bumps its onnxruntime-web dependency past
1.26.0-dev.20260416, retest q8-on-WASM at the default level and drop this.

## D-015 — Runtime feature flags via server env vars; first flag gates auth
**Status:** Accepted (2026-07-10)
Feature flags are read from the **server's** process environment at startup
(`src/server/flags.ts`, `envFlag()`: unset/empty → the flag's default;
`false`/`0`/`no`/`off` any case → off; anything else → on). Runtime-only by
design: one Docker image serves every configuration — flags are flipped in
docker-compose.yml / `docker run -e`, never at build time, so the client
bundle stays identical across deployments.
**First flag:** `AUTH_ENABLED` (default **on**) gates the M4 login/register
feature. Off, the server 404s every `/api/auth/*` route. The client is never
told separately: `fetchAuthStatus` (the `GET /api/auth/me` probe `useAuth`
already makes on mount) reads 404 as "feature off" and `<AuthForms />` renders
nothing. The Sign in button therefore only appears after the probe confirms
the endpoints exist — a disabled deployment never flashes auth UI, and an
offline/unreachable server also hides it (signing in would be impossible
anyway), preserving the anonymous-first invariant (no error surfaced).
**Consequence:** any future flag that must reach the browser needs a
server-derived signal (an endpoint the client can probe, like /me here) —
`BUN_PUBLIC_*` build-time inlining is not used for feature flags.

## D-016 — Live microphone transcription via segment re-transcription (browser engine)
**Status:** Accepted (2026-07-10; promotes "live transcription" out of the
WorkBreakdown out-of-scope list → milestone M7)
Live (microphone) transcription runs on the **browser engine only** — same
privacy story as file uploads: audio never touches the network, works for
anonymous users, works offline. Whisper is not a streaming model (it consumes
up-to-30 s windows), so "live" is approximated by a **segment loop**:

- **Capture (main thread):** `getUserMedia` → `AudioContext` at the device's
  native rate → an `AudioWorklet` chunker posts ~100 ms Float32 frames to the
  main thread, which resamples them to 16 kHz with the pure, stateful
  `StreamingResampler` (windowed-sinc low-pass; `resampleLinear` has no
  anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above
  8 kHz into the speech band and measurably hurt accuracy — see
  `tests/resampler.test.ts`). The worklet module is loaded from a **Blob URL**
  built from an inline code string — no extra served file, no
  build/service-worker changes, and it keeps working offline. Where
  `audioWorklet`/Blob modules are unavailable (older Safari), fall back to a
  `ScriptProcessorNode`. The context is *not* forced to 16 kHz because Firefox
  cannot connect a `MediaStreamAudioSourceNode` across sample rates.
- **Segmentation (pure, unit-tested):** a simple RMS energy gate
  (`src/lib/live.ts`) splits speech into segments: a segment starts at speech
  onset (with a short pre-roll), commits after a run of trailing silence or at
  a max length (under Whisper's 30 s window), and pure-silence audio is
  dropped entirely — Whisper hallucinates on silence, so never transcribe it.
  Two accuracy refinements (2026-07-14): the gate has **hysteresis** (once a
  segment is open, frames stay "speech" down to half the onset threshold, so
  quiet phonemes don't fragment utterances), and a max-length commit is a
  **soft cut** — it splits at the last sub-threshold frame in the trailing
  window instead of mid-word at the hard cap, seeding the next segment with
  the remainder (a mid-word cut corrupts the boundary word in both segments).
  Dropped (near-silent) segments also clear any tentative text they produced,
  so a hallucinated tentative pass can neither linger on screen nor be adopted
  as the next segment's provisional text. Production tuning lives in
  `LIVE_TUNING` (`src/lib/live.ts`), shared by `src/App.tsx` and the accuracy
  harness (`tests/live-accuracy.test.ts` — tone-word oracle engine + WER, so
  segmentation/scheduling/resampling accuracy is regression-tested under
  `bun test` without a model).
  No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
- **Inference (existing worker, serialized):** the current segment is
  re-transcribed through the **existing** inference worker/pipeline on a
  rolling basis (tentative text, replaced each pass); a committed segment gets
  one final pass and its text freezes. At most one inference is in flight;
  final passes take priority over tentative refreshes, which are throttled and
  always use the latest audio snapshot (intermediate snapshots are skipped, so
  the loop degrades gracefully when inference is slower than real time).
- **Protocol:** UI↔worker messages gain an optional `requestId` echoed on
  `partial`/`result`/`error`, so the live controller can correlate responses
  and coexist with the file flow. The file flow keeps working without ids.

**Alternatives rejected:** streaming into the worker with a custom
chunk-stitching decoder (large fork of Transformers.js internals for marginal
latency); MediaRecorder + re-decoding compressed blobs (adds codec latency and
`decodeAudioData` round-trips for no benefit over raw PCM taps).
**Consequence:** live quality/latency is bounded by model speed on the user's
device — the UI must keep tentative text visibly distinct from committed text.
Server-engine live input (WebSocket streaming to faster-whisper) is a separate
future decision; D-011's "WebSocket if needed" note is the hook.

## D-017 — "Auto" device order: WebNN → WebGPU → WASM
**Status:** Accepted (2026-07-14; amends D-004, which only knew WebGPU → WASM)
With WebNN support added (commit "WebNN support initial"), the `"auto"` device
preference resolves in priority order **WebNN → WebGPU → WASM**: the worker
probes `navigator.ml` (WebNN) first, then `navigator.gpu.requestAdapter()`
(WebGPU), and always keeps WASM as the last resort. The probe/ordering policy
is pure and unit-tested (`deviceAttempts` in `src/worker/protocol.ts`,
`tests/protocol.test.ts`); the worker only supplies the capability probe.
Pipeline-creation failures walk the same chain — an auto-detected WebNN that
fails still tries WebGPU before WASM. An **explicit** `"webnn"` or `"webgpu"`
preference deliberately falls back straight to WASM (no detour through the
other accelerator — the user asked for a specific backend, so honor it or take
the universal fallback). The selected backend stays user-visible: the worker's
`{status:"ready", device}` message drives a "Running on …" badge next to the
Backend select (`ModelPicker`) and the engine line in the footer (D-004's
"surfaced in the UI" requirement).

## D-018 — Cancel a file transcription by terminating the worker
**Status:** Accepted (2026-07-14)
Users can cancel an in-flight file run (decode, model download kicked off by
the drop, or the transcription pass itself) via a Cancel button next to the
file name. Transformers.js exposes no abort API for a running `asr()` call,
and on the WASM backend the pass blocks the worker's event loop — a "cancel"
postMessage would sit unread until the pass finished anyway. So cancellation
**terminates the inference worker** outright; it is the only reliable
preemption and also aborts an in-flight weight download. A `fileRun` counter
in `App.tsx` marks the decode that was running when the user cancelled as
stale (`decodeAudioData` itself cannot be aborted), so it never posts audio to
the fresh worker. The next run respawns the worker and reloads the pipeline —
cheap, because the weights stay in the browser Cache API (D-005); the UI
resets `loadStarted`/`loadProgress` so the reload starts from a clean slate.
Live mode is untouched: it has its own stop/drain semantics (D-016), and a
live session never coexists with a file run (mutually exclusive in the UI).
**Alternatives rejected:** a cancel message handled between chunks (dead on
WASM, where the event loop is blocked; and still seconds of latency
elsewhere); `AbortSignal` plumbed into Transformers.js (no upstream support).
**Consequence:** cancel throws away the loaded pipeline even when only the
transcription was unwanted — an accepted cost since reloading from cache takes
a few seconds.
**Amendment (2026-07-14, EngineToggle landing):** a server-engine run cancels
by aborting the `/api/transcribe` fetch instead (`AbortController`; the server
cancels the sidecar pass between segments, D-011/D-013) — the inference worker
and its loaded pipeline are left untouched in that path.

## D-019 — Server-based live microphone transcription
**Status:** Accepted
**Why:** M7 future want. Enables logged-in users on weak devices to transcribe live microphone audio using the server engine.
**Shape:** The client opens a WebSocket connection to `/api/transcribe/live`. The client streams 16kHz Float32 PCM chunks (from `MicCapture`) over the WebSocket. The server runs `LiveTranscriber` and `LiveSegmenter` (from `src/lib/live.ts`, which is pure and runs perfectly in Node/Bun). For each segment/pass, the server's `LiveTranscriber` writes the audio to a temporary file and invokes the Python sidecar. The server sends `LiveUpdate` objects back to the client over the WebSocket.
**Consequence:** The client does zero ML and minimal buffering, acting only as a mic-streamer and UI-renderer for live passes on the server.

## D-020 — Model downloads are user-initiated, never a side effect of selection
**Status:** Accepted (2026-07-14)
Selecting a model in `ModelPicker` (or changing the backend preference) no longer
starts the weight download. Weights load exactly when they are needed or asked for:
- **Lazy:** a file drop or live start on the **browser** engine posts the
  (idempotent) load, as both paths already did — the download overlaps decoding /
  mic startup.
- **Explicit:** a **Download** button on the selected, not-yet-cached model in
  `ModelPicker` starts the load immediately — this keeps D-005's "prepare for
  offline use" story alive now that selection alone caches nothing.
- **Server engine:** server runs never touch the browser pipeline, so with the
  toggle on "server" nothing downloads. (Previously the eager selection load
  downloaded browser weights even when every transcription would run server-side.)
**Consequence:** model/backend switches just reset the load UI state
(`loadStarted`, progress); the next load starts from clean progress. The explicit
download is a full pipeline load, so it also warms up inference and surfaces the
"Running on …" backend badge.
**Amendment (2026-07-14):** while the engine toggle is on "server" (signed in,
available), `ModelPicker` states explicitly that nothing is downloaded to the
device; the Download button stays (with a matching tooltip) for on-device /
offline preparation.

## D-021 — Long-file hardening: no idle timeout on the transcribe stream, heartbeats, single-pass Whisper generate
**Status:** Accepted (2026-07-15)
Long uploads (tested: 15-min Opus, `tests/audio_files/en_alien.opus`) hung or died
on both engines. Three independent causes, all length-dependent:
1. **Bun's default 10 s `idleTimeout` reset `/api/transcribe` mid-stream.** Segment
   gaps legitimately exceed 10 s (sidecar spawn + model load/download before the
   first segment; slow decode windows — 30 s+ gaps observed with large-v3-turbo on
   CPU). The stream then ended without a `result` line, so the client reported
   "connection was lost" after minutes of apparent hang. Fix: `src/index.ts` calls
   `server.timeout(request, 0)` for `POST /api/transcribe` (and 90 s for
   `GET /api/transcribe/status`, whose first probe waits up to 60 s for the sidecar
   import), and `src/server/transcribe.ts` enqueues a blank ndjson line every 5 s as
   a heartbeat — reverse proxies get liveness, dead connections surface as enqueue
   failures, and `ServerEngine` already skips blank lines.
2. **Bun's default 128 MB `maxRequestBodySize` rejected uploads > ~35 min** (raw
   f32 PCM is 3.84 MB/min) before the route's own 512 MB limit ever applied. Fix:
   `maxRequestBodySize` is set to the shared `MAX_UPLOAD_BYTES` export.
3. **Browser engine: Transformers.js 4.2's Whisper `generate()` takes a seek-loop
   path when `return_timestamps` is set without `max_new_tokens`** — 1–3 full decode
   passes per window, redundant under the pipeline's own 30 s/5 s chunking. Measured
   on the 15-min clip (tiny.en, onnxruntime-node): 80 generate passes for 44 windows,
   1.4× slower, WER 44.7 % vs 31.5 % (duplicated boundary text), and the streamer's
   `on_finalize` fired per pass instead of per window, so `TranscribeProgressTracker`
   saturated the progress bar at ~60 % of the run and the UI looked frozen for the
   remainder (on WASM: many minutes). A window whose last boundary timestamp is 0 can
   even loop forever. Fix: the worker passes `max_new_tokens: 440`, which selects the
   classic single-pass-per-window path (440 ≈ 448 decoder positions minus prompt).
**Revisit:** when Transformers.js changes the seek-loop condition (currently
`return_timestamps && !max_new_tokens`, `src/models/whisper/modeling_whisper.js`),
retest whether the explicit `max_new_tokens` is still needed.

## D-022 — Medium models never default to fp16 in the browser
**Status:** Accepted (2026-07-15)
Reported symptom: transcribing `en_aiproblem` in-browser with Whisper Medium
printed just "the". The medium catalog entries defaulted to fp16 on WebNN
(whole model) and WebGPU (encoder) — and D-017's "auto" order now tries WebNN
first, so browsers exposing `navigator.ml` silently moved medium onto fp16.
Two problems with those fp16 conversions
(`onnx-community/whisper-medium(.en_timestamped|-ONNX)`):
1. The fp16 ONNX files crash onnxruntime's graph optimizer at session
   creation (`SimplifiedLayerNormFusion` fails on their
   `InsertedPrecisionFreeCast_*` nodes) — reproduced on both onnxruntime-node
   and onnxruntime-web 1.26.0-dev (wasm EP). EPs whose optimizer pipeline
   skips that fusion can load the model and hit problem 2 instead.
2. Whisper medium/large encoder activations are the classic fp16 overflow
   case — degenerate output with no error raised.
Meanwhile q4 medium was verified correct end-to-end on the real browser
runtime (onnxruntime-web wasm EP driven under Bun via the
`Symbol.for("onnxruntime")` override, `graphOptimizationLevel: "basic"` as in
D-014, `max_new_tokens: 440` as in D-021; both the `.en` and multilingual
conversions, auto language detect included). **Fix:** medium's `webgpu` and
`webnn` defaults are q4 (`ACCEL_Q4` in `src/lib/models.ts`). A backend that
cannot execute q4 (`MatMulNBits`) throws at pipeline creation and falls down
the D-017 device chain — strictly better than silently emitting garbage.
**Scope:** medium only — that's where load-failure evidence exists.
tiny/base/small keep fp16 on WebNN and small keeps fp16+q4 on WebGPU;
large-v3-turbo keeps fp16+q4 on WebGPU (the widely-used Transformers.js demo
configuration). If a one-word/garbage transcript ever shows up on those, suspect
their fp16 encoders and extend this decision.
**Revisit:** if the onnx-community medium conversions are re-exported with
working fp16 files, or once WebNN/WebGPU runs can be exercised in CI.

## D-023 — Cross-origin isolation (COOP/COEP) so WASM inference is multithreaded
**Status:** Accepted (2026-07-15)
Reported symptom: after D-022 moved medium to q4, a 14 s clip transcribed
correctly in-browser but "took ages" while native whisper needs seconds.
Cause: without `crossOriginIsolated` there is no SharedArrayBuffer, and
onnxruntime-web silently runs its (SIMD-threaded build!) WASM backend on
**one** thread. Measured on the exact browser runtime (ort-web wasm EP,
medium q4, the 14.4 s en_aiproblem clip): 52.1 s at 1 thread, 15.3 s at 4
(ort's default cap once isolation exists), 9.5 s at 12 — a 5.5× speedup.
**Fix:** `src/index.ts` sends `Cross-Origin-Opener-Policy: same-origin` +
`Cross-Origin-Embedder-Policy: require-corp` on every response in both serve
modes (`withIsolationHeaders`). Bun cannot attach headers to an HTMLBundle
route, so in development the shell moved to an internal `/__shell` route and
`fetch()` re-serves `/` (and SPA fallbacks) with the headers stamped on. The
inference worker raises `env.backends.onnx.wasm.numThreads` to
`min(hardwareConcurrency − 1, 12)` when `crossOriginIsolated` is true (ort
would otherwise cap at 4; without isolation the default is left alone since
ort clamps to 1 anyway).
**Consequences:** COEP `require-corp` blocks cross-origin subresources that
are neither CORS- nor CORP-approved. Everything the app loads today survives:
Hub model downloads use `fetch()` in CORS mode, all other assets are
same-origin, and the service worker preserves the headers it cached, so the
installed/offline app stays isolated too. Any future cross-origin `<img>`,
font, or iframe must ship CORS/CORP headers or be proxied same-origin.
**Verified:** headers present on `/`, worker, sw.js, `/ort/*` in both dev and
production serve modes; dev HMR asset routes unaffected by the `/__shell`
indirection. Real-browser thread scaling still deserves a spot check (the
5.5× number is ort-web under Bun, same runtime but not a browser tab).
**Amendment (2026-07-15, same day):** threads are per-device now. With
SharedArrayBuffer newly available, onnxruntime-web's *own* default thread
count becomes ~4 — and the JSEP accelerator EPs (WebGPU, WebNN) are not
thread-safe: real-browser WebGPU runs failed at OrtRun with "WebGPU
validation failed. Buffer with '' label is invalid". The worker now sets
`numThreads` right before each device attempt (`configureWasmThreads`):
`min(cores − 1, 12)` for the pure `wasm` device, **explicitly 1** for
`webgpu`/`webnn` (relying on the default is what broke). onnxruntime-web
fixes its thread pool at first-session init, so the first attempt in a
worker decides; a wasm fallback after a failed accelerator attempt may run
single-threaded until the next worker respawn — correct over fast.

## D-024 — Live mode batches queued final passes into merged passes
**Status:** Accepted (2026-07-17)
Reported symptom: stopping a live session got stuck on "Finishing up…" while
the machine sat at full CPU — indistinguishable from a hang, and the saturated
cores (D-023 uses most of them) made the whole browser feel locked up.
**Cause — a scheduling backlog, not a deadlock.** Whisper pads every inference
pass to a full 30 s encoder window, so a pass costs nearly the same whether it
carries 1 s or 29 s of audio (measured on the browser runtime, ort-web WASM EP
medium q4: ~4 s per pass at 12 threads, ~9–12 s at 4 threads, for 1–9 s
segments alike). Segments therefore commit faster than their final passes
complete whenever the model is slower than speech (medium/large on WASM), the
committed-segment queue grows all session, and `stop()` used to drain it **one
pass per segment** — a 60 s session on 4 threads measured a 52 s drain; longer
sessions scale it into minutes. All the while the UI showed only a static
"Finishing up…".
**Fix (two parts):**
1. `LiveTranscriber.pump()` batches consecutive queued segments into **one
   merged final pass**, capped at the segmenter's `maxSegmentSeconds` so a
   merged pass stays a single Whisper window. Segment audio is concatenated
   (segments end on trailing silence or a soft cut, so splices land in natural
   gaps); the result's chunk timestamps are mapped piecewise back to absolute
   session time via each segment's own start offset. The backlog — and the
   stop drain — is now bounded by ~one pass per 29 s of speech instead of one
   pass per segment (the same 60 s/4-thread scenario drains in ~2 passes).
2. `App.tsx` surfaces `LiveUpdate.pendingFinalPasses` while stopping:
   "Finishing up… transcribing N remaining segments" counts down instead of
   freezing, on both the browser and the server-live (D-019) paths.
**Alternatives rejected:** skipping final passes for segments that already had
tentative text (during a backlog new segments get *no* tentative pass, so
skipping loses their words entirely); a "stop now" escape hatch (same data
loss, silently); merging across the 30 s window (impossible — Whisper's
context is one window).
**Consequence:** a merged pass attributes its whole text to the batch's first
segment (per-segment text attribution is meaningless after joining anyway) and
inter-segment silence is not re-inserted, so chunk timestamps inside a merged
pass are exact per segment region but a chunk spanning a splice maps to its
start's region. Covered by `tests/live.test.ts` (batching + cap) and the WER
gates (`tests/live-accuracy.test.ts`, `tests/live-real-model.test.ts` — the
real-model test's fast feed makes real inference lag, so merging is exercised
there with real weights).

## D-025 — A degraded-thread WASM fallback restarts the worker; WebGPU is skipped when fp16 needs an absent shader-f16
**Status:** Accepted (2026-07-17)
Reported symptom: Firefox performance far below Chrome, live transcription
worst. Chrome resolves D-017's auto chain to an accelerator; Firefox has no
WebNN and its WebGPU (where exposed at all) fails onnxruntime-web's
Chromium-built JSEP EP — notably it lacks the `shader-f16` feature, which the
catalog's fp16-encoder WebGPU defaults (small, large-v3-turbo) need. So
Firefox lands on WASM — but *how* it lands there was the real bug: the failed
WebGPU attempt initializes ort's WASM runtime at the accelerator-safe
`numThreads: 1` (D-023 amendment), the thread pool is fixed at first init,
and the fallback session then runs **single-threaded for the worker's whole
life** (measured 5.5×: 52 s vs 9.5 s on a 14 s clip). Live never respawns the
worker, and every live pass pads to a full 30 s window (D-024), so passes
took ~40–60 s: tentative text froze and the stop drain ballooned — D-023
amendment's "correct over fast" edge case is Firefox's *default* path.
**Fix (three parts):**
1. **Restart instead of a degraded fallback.** When `createPipeline` reaches
   the `wasm` attempt after any accelerator attempt ran and multithreading is
   actually available (`desiredWasmThreads() > 1`), the worker posts
   `{type:"load-restart", modelId, preference, device:"wasm"}` and aborts the
   load (sentinel error, never surfaced). The UI terminates the worker,
   respawns it, re-issues the load with the explicit device, and remembers
   `modelId::preference → device` for the session so later loads skip the
   doomed attempt (session-only: a browser update may fix the backend; an
   explicit user backend choice is a different key, so it retries once).
   To make the respawn safe, transcribe posts now wait on a **load gate** in
   `App.tsx` that resolves on `ready` (rejects on load error/cancel/crash):
   no pass is ever queued inside a worker a restart is about to terminate —
   its transferred audio buffer would die with it. The worker serialized
   passes on the load anyway, so nothing starts later than before.
2. **shader-f16 probe.** The worker's WebGPU probe also records
   `adapter.features.has("shader-f16")`; when the dtype that a WebGPU attempt
   would use requires fp16 (`dtypeRequiresF16`, pure + unit-tested) and the
   feature is absent, WebGPU is skipped outright — Firefox then loads
   straight into multithreaded WASM with no wasted attempt.
3. **Live-on-WASM hint.** With live active on `wasm` and a heavy model
   (encoder ≥ 300 M params: medium, large-v3-turbo), the UI says the model
   runs slower than real time and suggests a smaller one — WASM lag there is
   inherent (D-024's fixed window cost), not a bug, so tell the user instead
   of looking frozen.
**Alternatives rejected:** re-posting in-flight passes after respawn (their
buffers are transferred — gone with the worker); a scout worker pre-flighting
the accelerator (doubles accelerated-path load time to save the failure
path); ort-web's proxy-worker mode to escape the pinned pool (undocumented
interplay with JSEP).
**Consequence:** the accelerator-failure path costs one worker respawn + a
pipeline reload from cache (a few seconds, once per model+preference per
session). A genuine both-attempts-fail load still reports its error
normally. `deviceAttempts` is unchanged; the new policy pieces
(`dtypeRequiresF16`, the `load-restart` message guard) are unit-tested in
`tests/protocol.test.ts`. Real-Firefox verification (badge shows WASM with
full threads, live keeps up on small models) still needs a manual run — same
no-browser dev-env caveat as M2/M3.
**Amendment (2026-07-17, same day — field report: tiny fixed, medium still
slow in Firefox):** three observability/robustness additions to tell "slow is
the CPU-vs-GPU physics" apart from "something is still degraded":
1. The probe **skips software (fallback) WebGPU adapters**
   (`isFallbackAdapter`, both spec locations) — SwiftShader/llvmpipe/WARP
   would *succeed* at session creation and then run slower than multithreaded
   WASM, and the fallback chain only reacts to failures, not slowness.
   onnxruntime-web itself uses whatever adapter it is handed (verified in
   1.26.0-dev: no fallback-adapter or Firefox guard).
2. The `ready` message now carries **`threads`** — read back from
   `env.backends.onnx.wasm.numThreads` *after* session creation, so ort's own
   clamp-to-1 (missing isolation) is visible. The footer shows it:
   "WASM (on-device, 11 threads)".
3. The worker **logs the device chain** (`[whisper] …` console lines): each
   skipped backend (software adapter, missing shader-f16), each failed
   attempt with its error, and the final device+threads — previously fallback
   errors were swallowed silently, making field reports unactionable.
Expected healthy Firefox numbers for medium q4 on WASM (from the D-023/D-024
measurements): ~4 s per 30 s window at 12 threads, ~9–12 s at 4 — i.e. a
14 s clip in ~10 s. Chromium's GPU path is a further ~3–5× faster; that
remaining gap is inherent until Firefox's WebGPU works with ort-web's JSEP
EP. If the badge shows 1 thread or the console shows a surviving software
adapter, it is a bug in this decision's scope instead.

## D-026 — "Auto" never picks WebGPU on Firefox
**Status:** Accepted (2026-07-17)
Field-verified on Firefox (D-025's observability build): explicit
Backend = WASM runs medium with a full thread pool and healthy speed, while
explicit Backend = WebGPU (`device=webgpu wasmThreads=1`, correct per D-023)
**completes but runs far slower** than the same machine's multithreaded WASM.
That is the worst case for D-017's failure-driven fallback chain: ort-web's
WebGPU EP (Chromium-developed; no Gecko handling in 1.26.0-dev, verified)
creates the session *successfully* on Firefox, so nothing ever falls back —
"auto" happily selects the slowest available backend.
**Fix:** the worker's capability probe skips the WebGPU probe entirely when
the preference is `"auto"` and the UA is Firefox (`/Firefox\//` on
`navigator.userAgent`), with a `[whisper]` console line saying why. Firefox
auto therefore resolves straight to multithreaded WASM (no accelerator
attempt → no D-025 restart either). An **explicit** WebGPU selection is still
honored — that is the deliberate retest hatch: when a future
Transformers.js/ort-web bump gains real Gecko WebGPU support, verify via the
explicit path, then delete the UA check.
**Why UA sniffing:** there is no feature to detect — the sessions create and
run; only throughput distinguishes them, and a benchmark probe would cost
more than it saves. The check is one line, logged, scoped to `"auto"`, and
targets the one engine with field evidence.
**Revisit:** on any onnxruntime-web bump — retest explicit WebGPU on Firefox
(file `tests/live.md` note) and drop the check when it reaches parity.

## D-027 — Mobile OOM crash on Medium+ loads: sequential session loading (patched) + a warn/confirm gate
**Status:** Accepted (2026-07-28)
Reported symptom: downloading Whisper Medium (English) on a phone crashed the
whole browser — on Android, badly enough to briefly kick the OS back to the
home screen (consistent with the low-memory killer reaping background
processes, including the launcher, once the tab's memory spike triggered
system-wide pressure).
**Cause.** Whisper Medium's smallest available dtype on any device is q4
(D-022) — still ~460 MB combined encoder+decoder weights. Two independent,
compounding sources of transient memory at load time:
1. `@huggingface/transformers` buffers each ONNX weight file fully into one
   JS-heap `Uint8Array` (`readResponse`/`loadResourceFile`,
   `src/utils/hub.js`/`hub/utils.js`) before it is cached or handed to
   onnxruntime — unavoidable either way (with or without a
   `progress_callback`, and regardless of whether a URL string or a buffer
   is passed to `InferenceSession.create`): onnxruntime-web's WASM backend
   has no streaming ONNX parser, so the whole file must exist as one
   contiguous buffer to construct a session.
2. `constructSessions` (`src/models/session.js`) loaded the encoder and
   decoder sessions **concurrently** via `Promise.all`, so both full-size
   JS buffers were resident at once — needlessly roughly doubling the
   already-large per-file peak (on top of onnxruntime-web's own
   WASM-heap `_malloc` + `HEAPU8.set` copy inside `copyFromExternalBuffer`,
   made once per file at session-creation time, which is itself already
   serialized across sessions in web environments).
Net: peak transient memory for Medium approached ~2-3x the ~460 MB combined
weight size — comfortably inside WASM32's 4 GiB address space, but well
past the much tighter per-tab memory budgets mobile Safari/Chrome enforce,
which respond to going over budget with an outright, uncatchable tab kill
(no JS `error` event fires — `worker.onerror` in `App.tsx` never sees it).
There was no mobile detection anywhere in the app: Medium downloaded/loaded
identically on a phone and a workstation.
**Fix (two parts):**
1. **Sequential session loading (real fix, no UX cost).** `constructSessions`
   is patched (`bun patch`, `patches/@huggingface%[email protected]`,
   tracked via `patchedDependencies` in `package.json` so it survives
   `bun install`) to load sub-model sessions one at a time instead of via
   `Promise.all`. This lets the previous file's JS buffer become garbage
   before the next file is fetched, roughly halving the load-time peak for
   any multi-file model (Whisper's encoder+decoder) — a strict improvement
   with no downside, so it applies unconditionally, not just on mobile.
   Investigated and rejected as *not* actionable: skipping the
   `progress_callback`, `env.customCache`, and passing a URL string instead
   of a buffer to onnxruntime-web — all three still end up materializing the
   whole file, per the analysis above. Fixing that fully would require
   forking onnxruntime-web itself (out of scope).
2. **Warn/confirm gate on mobile (`src/lib/mobileMemoryGuard.ts`,
   `estimatePeakLoadBytes` in `src/lib/models.ts`).** Even halved, Medium's
   peak (~555 MB estimated: `2 × max(encoderBytes, decoderBytes)`, reflecting
   the now-sequential loading) can still exceed a constrained phone's tab
   budget. `App.tsx`'s `confirmRiskyMobileLoad` gates all three ways a
   browser-engine load can start — the explicit Download button, and the
   lazy auto-load on file-drop / live-start (D-020) — behind
   `shouldWarnBeforeLoad`: mobile UA detection (Android/iPhone/iPod, plus
   iPadOS's desktop-masquerading Safari UA via touch-point count, same
   UA-sniffing rationale as D-026 — no memory-pressure API exists to probe
   instead) combined with `navigator.deviceMemory` when Chromium reports it,
   falling back to a flat ~400 MB threshold where it doesn't (all of
   iOS/Safari). A native `window.confirm()` states the estimated memory need
   and the crash risk in plain language; declining aborts the load with no
   state changes, accepting remembers the modelId+device combination for the
   rest of the session so the same load doesn't ask twice. Desktop/laptop
   browsers are never gated — no crash evidence exists there.
**Alternatives rejected:** hard-blocking Medium+ on mobile with no override
(some phones have enough RAM to be fine; the confirm lets the user judge
their own device); forking onnxruntime-web for a streaming session-creation
path (large undertaking for a runtime we don't own upstream).
**Consequence:** every model load is now faster-to-safe (sequential
sessions cost a little wall-clock time — files download one after another
instead of in parallel — in exchange for materially lower peak memory).
Mobile users choosing Medium/Large now see one extra confirmation step;
Tiny/Base/Small are unaffected (their estimated peaks stay under the
warning thresholds).
**Revisit:** if field reports show the sequential-load patch alone resolves
mobile crashes in practice, consider loosening the confirm thresholds; if
onnxruntime-web ever gains a streaming/incremental session-creation API,
drop the patch and re-measure before touching the thresholds again.
**Amendment (2026-07-29, D-028):** "a little wall-clock time" undersold the
cost — see D-028. The sequential patch no longer applies unconditionally;
`options.sequentialSessionLoad` (set by `App.tsx`, true only when
`shouldWarnBeforeLoad` is true for the load) now gates it.

## D-028 — Sequential session loading gated to actual mobile-crash risk, not unconditional
**Status:** Accepted (2026-07-29)
Reported symptom: `encoder_model_q4.onnx` (Whisper Medium) downloaded far
slower after D-027 landed than before it — reproduced on desktop too, ruling
out phone CPU/network as the cause. Measured in Chrome DevTools: ~1.6 MB/s
sustained on a connection that otherwise performs fine, downloading from
Hugging Face's Xet-backed CDN (`us.aws.cdn.hf.co/xet-bridge-us/...`).
**Cause.** D-027's `constructSessions` patch made the encoder and decoder
downloads always sequential — one HTTP request in flight at a time — to keep
both weight buffers from being resident together. On a long-RTT path (this
CDN is US-hosted), a single HTTP/2 stream typically can't fill the available
bandwidth; two concurrent requests (the pre-D-027 `Promise.all`) filled more
of it. Serializing the downloads dropped every load back to one flow's
throughput ceiling — not "a little" wall-clock cost as D-027 claimed, but
multiple times slower on exactly this kind of connection, and paid by every
user regardless of device.
**Fix.** The sequential/parallel choice is now conditional, not baked into
the patch: `constructSessions` branches on a new `options.sequentialSessionLoad`
flag (`patches/@huggingface%[email protected]`) — parallel
`Promise.all` (the original, fast behavior) when falsy/absent, the D-027
for-loop when true. `App.tsx` computes this per load via `isMobileLoadRisk`
(extracted from `confirmRiskyMobileLoad`'s existing hint/`estimatePeakLoadBytes`
logic — the same `shouldWarnBeforeLoad` check the crash-warning dialog uses)
and threads it through a new `sequentialSessionLoad` field on the worker's
`LoadMessage` (`protocol.ts`) into the `pipeline()` call
(`inference.worker.ts`). Crucially this is evaluated independently of the
confirm dialog's answer — even a load the user clicked "Continue anyway" on
still loads sequentially, since the memory risk that motivated D-027 doesn't
go away just because the user accepted it.
**Consequence:** desktop/laptop loads (and any mobile load under the
size/`deviceMemory` risk threshold) get back the full concurrent-download
speed that predates D-027; only loads that would already trigger the mobile
crash warning pay the sequential-loading wall-clock cost, which is exactly
where D-027's OOM fix is actually needed.
**Revisit:** if Hugging Face's Xet CDN behavior changes, or if a future model
in the catalog is large enough to risk mobile OOM at a size where the
confirm-dialog threshold doesn't yet trigger, re-check that the two
thresholds (`shouldWarnBeforeLoad`'s and this gate's) are meant to stay
identical.

## D-029 — Mobile crash warning was silently disabled on every phone but the lowest-end
**Status:** Accepted (2026-07-29)
Reported symptom: the D-027 mobile crash `window.confirm()` never appeared
for Whisper Medium (or any model), even on devices it's explicitly meant to
protect.
**Cause.** `shouldWarnBeforeLoad`'s `deviceMemory`-scaled branch used
`SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.5`. Checked against the actual
catalog (`estimatePeakLoadBytes`) for every `navigator.deviceMemory` value
Chromium can report:

| deviceMemory | "safe" budget (old 0.5) | Medium (556 MB) | Large v3 Turbo (762 MB) |
|---|---|---|---|
| 1 | 500 MB | warns | warns |
| 2 | 1 GB | **silent** | **silent** |
| 4 | 2 GB | **silent** | **silent** |
| 8 | 4 GB | **silent** | **silent** |

`deviceMemory` of 2 or more is ordinary today, not just flagship-tier — so
in practice the warning only ever fired on the oldest/cheapest Android
phones and on iOS (which lacks the API entirely and always used the flat
400 MB fallback, D-027). The exact mid/low-range phones the original crash
report was about — the whole reason D-027 exists — cleared the threshold and
got no warning at all.
**Fix:** `SAFE_TAB_FRACTION_OF_DEVICE_MEMORY` lowered to 0.1
(`src/lib/mobileMemoryGuard.ts`). Re-checking the same table: deviceMemory
2 and 4 now warn for Medium+ (200 MB / 400 MB budgets, both under 556 MB);
deviceMemory 6-8 (600-800 MB budgets) still don't, treating genuine
high-end devices as having real headroom. `tests/mobileMemoryGuard.test.ts`
gained a regression case pinning deviceMemory 2/4 to `true` for Medium's
peak — the exact gap this decision closes.
**Alternatives rejected:** dropping the `deviceMemory` scaling entirely in
favor of always using the flat 400 MB fallback (simpler, but throws away the
one signal that lets genuinely high-RAM devices skip an unnecessary
confirm); a hard absolute cap on top of the fraction (more robust long-term,
but two knobs to calibrate against no real data beats one — revisit if a
future model's peak lands awkwardly relative to the 0.1 fraction).
**Revisit:** this is still a guess, same as D-027's original 0.5 was — there
is no real crash-telemetry feedback loop. If field reports show 0.1 is still
too generous (or too aggressive) for some device class, adjust the constant
rather than the shape of the heuristic.

## D-030 — Error taxonomy: pattern-matched classification, not structured error types
**Status:** Accepted (2026-07-29)
Every failure path (decode, model load, transcription, a crashed worker, a
live session) used to end at `setError(rawMessage)`, rendered as one plain
string — no distinction between "this file won't decode" (try another file),
"the tab ran out of memory" (try a smaller model), a lost WebGPU device
(switch backend), or a dropped network connection mid-download (retry). The
WorkBreakdown M6 item asked for exactly those four categories.
**Shape:** `src/lib/errors.ts` exports a pure `classifyError(raw, context)`:
`context.stage` (`"decode" | "load" | "transcribe" | "worker-crash"`) plus
`context.offline` (`navigator.onLine`) disambiguate cases the message text
alone can't, then substring pattern-matching on the message sorts the rest
into `unsupported-file` / `out-of-memory` / `device-lost` / `network` /
`generic`, each with a user-facing `message`, an optional actionable
`suggestion`, and `retryable`. **Why pattern-matching, not typed errors from
each layer:** the failures cross Web Audio, Transformers.js, onnxruntime-web
(WASM and the WebGPU/WebNN JSEP EPs), and `fetch` — none of which expose a
shared error hierarchy the app controls, so there is nothing to `instanceof`
against; the only thing every layer reliably produces is an `Error` (or
DOMException) with a message string. A worker crash (`Worker.onerror`) with
an **empty** message is itself classified as out-of-memory: D-027 found that
the browser's silent OOM tab-kill fires no JS error event with details at
all, so an empty message *is* the OOM signal in that one context — the
classifier only applies that inference when `context.stage ===
"worker-crash"`, not to an empty message anywhere else.
**UI wiring (`src/App.tsx`):** the `error` state changed from a `string` to
`{ classified: ClassifiedError; retryKind?: "load" | "file" }`. `retryKind`
is resolved to an actual retry action **at render time** (`onClick={() =>
startLoad(modelId, devicePreference)}` / replaying `lastFileRef.current`
through `handleFile`), reading current component state directly — not
computed inside the worker's `message`/`error` event listeners, which are
registered once per worker lifetime (`ensureWorker`) and would otherwise
close over whichever render's `modelId`/`devicePreference` happened to be
current at registration, going stale on every later model/device switch.
**Consequence:** only `network` and `device-lost` are `retryable` (an
unsupported file or an OOM won't succeed by retrying the identical action);
the error box shows a Retry button only for those, alongside the suggestion
text. Live-mode errors (mic permission, WS disconnects) still classify
through the same function for a consistent error box, but never carry
`retryKind` — the existing Start/Stop controls are already the retry path
there.
**Alternatives rejected:** wrapping each subsystem's calls in typed error
classes at the boundary (real fix, but Transformers.js/onnxruntime-web
throw plain `Error`s from deep internals — retrofitting types there means
forking dependencies, out of scope); classifying in the worker instead of
the UI (the worker already forwards a string message over postMessage —
classifying there would just move the same pattern-matching one hop over
for no benefit, and the UI is where `context.offline`/retry actions live).
**Revisit:** if a future onnxruntime-web/Transformers.js version starts
surfacing richer structured errors (e.g. a `GPUDeviceLostInfo` object
instead of a stringified message), prefer inspecting that directly over
adding more substring patterns.

## D-031 — Docker image hardening: exclude local venvs, move Transformers.js out of the runtime image
**Status:** Accepted (2026-07-29)
WorkBreakdown's M6 "production build + `bun start` deployment check" item —
a `docker build .` from this checkout, not a fresh clone, exposed two real
bugs the documented image sizes (WorkBreakdown M5 notes: "244 MB" /
"405 MB") had already gone stale against.
**Bug 1 — local Python venvs leak into every image.** `server-py/venv` and
`server-py/.venv` (created locally to test the sidecar against a real
faster-whisper install) are invisible to `git status` — `python -m venv`
drops its own nested `.gitignore` (`*`) inside the venv directory — but
Docker does not read `.gitignore`, and `.dockerignore` had no matching
pattern. `COPY server-py ./server-py` therefore copied both venvs (858 MB
combined) into the image; measured effect on the Bun-only image: 999 MB
instead of the intended size, with a completely unused,
machine-specific Python interpreter baked in at `/app/server-py/venv` and
`/app/server-py/.venv`. **Fix:** `.dockerignore` gained `**/venv/` and
`**/.venv/`. Verified: rebuilding after the fix, `find /app/server-py`
inside the image no longer lists either directory.
**Bug 2 — `@huggingface/transformers` (and its `onnxruntime-node` +
`onnxruntime-web` dependencies, 341 MB combined) had no reason to be in the
*production runtime* image.** Grepped: the only file in `src/` that imports
`@huggingface/transformers` as a value (not a type) is
`src/worker/inference.worker.ts`, and that file is bundled once, at build
time, into the static `dist/inference.worker.js` the production server
just serves as a file (confirmed: `src/index.ts`'s `Bun.build`-based
on-the-fly bundling path — needed because Bun doesn't follow `new
Worker(...)` graphs, per M1's Architecture notes — only runs in
**development** mode; production never calls it). No server-side module
(`src/server/*`, `src/index.ts`, `src/lib/models.ts`'s only import from the
package is `import type`, erased at transpile time) touches the package at
runtime. Yet it was a `dependencies` entry, so the Dockerfile's runtime
stage (`bun install --frozen-lockfile --production`) installed it and its
full dependency tree anyway — dead weight on every deploy. **Fix:**
`@huggingface/transformers` moved to `devDependencies` in `package.json`.
The **build** stage's `bun install --frozen-lockfile` (no `--production`)
still installs devDependencies, so `scripts/build.ts`'s `Bun.build` call
(which needs the package on disk to bundle the worker) is unaffected; the
patch in `patches/@huggingface%[email protected]` still applies
there the same way. Verified: production `node_modules` in the built image
dropped from 409 MB to 7.9 MB (react/react-dom/passport only); the running
container still served `/inference.worker.js` correctly (534,714 bytes,
byte-identical to the pre-change build) and `/api/transcribe/status`
correctly reported the expected `faster-whisper`/`python3` unavailability
message — nothing at runtime needed the package.
**Net effect, measured against a real `docker build .`/`--target
with-server-engine` in this checkout:**
| Image | Before both fixes | After |
| --- | --- | --- |
| Bun-only (`runtime`) | 2.18 GB (venv leak) | ~421 MB |
| with-server-engine | not separately measured (inherits the leak) | ~7.1 GB — see below |
**A third, non-bug finding while re-measuring:** the with-server-engine
image is legitimately several GB, not the "405 MB" previously written down
— its last `RUN` layer pre-downloads both server-selectable models
(`large-v3-turbo`, `medium.en`, matching the `EngineToggle`'s hardcoded
allow-list in `App.tsx`) into `/opt/faster-whisper/models` at *build* time,
which alone measured 3.15 GB; the venv/faster-whisper/ctranslate2 install
layer is a separate 451 MB. This is intentional (comment already in the
Dockerfile: baking the weights into the image, outside `/app/data`, means
no first-request cold-download and no dependence on the data volume for
those two models) — not something to shrink — but the number that made it
into WorkBreakdown's M5 notes and README's env-var table was never updated
to match, and the README's Persistence section wrongly implied these
weights live in the `/app/data` volume (`FASTER_WHISPER_DOWNLOAD_ROOT`'s
actual default in that image is `/opt/faster-whisper/models`, not
`/app/data/faster-whisper-models`). **Fix:** README's env-var table,
Persistence section, and a new "Image sizes" subsection now state the
verified numbers and the reason for each; WorkBreakdown's M5 notes updated
to match.
**Consequence:** none of this is CI-visible — GitLab CI (`.gitlab-ci.yml`)
builds from a clean checkout that never has `server-py/venv` on disk, so
Bug 1 only bit a local `docker build .` run from a working directory that
had one (as this checkout did); the `.dockerignore` fix is still correct
defense-in-depth regardless. Bug 2 was real everywhere, CI included — every
pushed Bun-only image has been ~340 MB heavier than necessary since
`@huggingface/transformers` became a production dependency.
**Revisit:** if the with-server-engine catalog ever grows beyond the two
baked-in models, either accept runtime downloads for the extra ones (point
`FASTER_WHISPER_DOWNLOAD_ROOT` at `/app/data` for those, per the README's
override note) or extend the Dockerfile's pre-download `RUN` line —
re-measure and re-document the image size either way.

## D-032 — Server-live WebSocket brought in line with the HTTP engine's DI/validation/test conventions
**Status:** Accepted (2026-07-29)
D-019's server-engine live transcription (`GET /api/transcribe/live` →
`src/server/live.ts`) had landed and was wired end-to-end in `App.tsx`, but
three things about it lagged the standard the rest of M5 set with
`src/server/transcribe.ts`:
1. **Not injectable.** `live.ts` imported the real `sidecar` singleton
   directly and called it from inside the `LiveTranscriber`'s `transcribe`
   closure, unlike `createTranscribeHandlers(deps)`, whose whole point (per
   its own comment) is that `Sidecar` is "injectable — tests use a fake".
   There was no way to unit-test the WebSocket bridge without a real Python
   sidecar.
2. **Unvalidated query params.** `POST /api/transcribe`'s `parseParams`
   validates `language` against `LANGUAGE_RE` and `task` against
   `"transcribe" | "translate"` before it ever reaches the sidecar protocol.
   The WS upgrade route in `src/index.ts` skipped both checks —
   `url.searchParams.get("task") || undefined` forwarded an arbitrary string
   straight into `LiveSessionData.task` and on to the Python sidecar's
   JSON-lines protocol. Also skipped: the upfront `sidecar.status()` check the HTTP
   route makes before doing any work (D-013's "degrade gracefully" —
   opening a socket that can only report unavailability after the first
   segment reaches the sidecar is a worse failure mode than a prompt 503).
3. **Zero test coverage.** No `tests/server-live.*` file existed; every
   other stateful server module in the app (`transcribe.ts`, `auth.ts`,
   `db.ts`) has one.
**Fix:**
1. `src/server/transcribe.ts` exports `parseLanguageAndTask(url)` (extracted
   from `parseParams`, same validation, same 400-text-on-string-return
   contract); `src/index.ts`'s `GET /api/transcribe/live` case now calls it
   and 400s on an invalid value, then calls `sidecar.status()` and 503s with
   the reason before upgrading — matching `POST /api/transcribe` exactly.
2. `src/server/live.ts` exports `createLiveWebSocket(deps: { sidecar:
   Sidecar })`; `liveWebSocket` (the instance `src/index.ts` wires into
   `Bun.serve`) is just `createLiveWebSocket({ sidecar })` against the real
   singleton. `LiveSessionData.user`/`task` are now typed `SessionUser`/
   `WhisperTask` instead of `any`/`string | undefined` (task always has a
   value now that the WS route defaults it like the HTTP route does), which
   let the pre-existing `options.websocket = liveWebSocket as any` cast in
   `src/index.ts` drop to a plain assignment — it was papering over the
   untyped `user`/`task` fields, not a real incompatibility with `Bun.serve`'s
   generic `websocket` option.
3. `tests/server-live.test.ts` exercises `createLiveWebSocket` against a fake
   `Sidecar` (same shape as `tests/transcribe.test.ts`'s): a committed
   segment writes exactly its own PCM bytes to the temp file and cleans up
   after, model/language/task pass through, a sidecar error surfaces via the
   `update` message's `error` field instead of crashing the session, a
   `{"type":"stop"}` frame drains and closes with a `result` message, and a
   malformed text frame is ignored. `tests/transcribe.test.ts` gained direct
   cases for `parseLanguageAndTask`.
**Incidental correctness fix found while rewriting the `transcribe` closure:**
the original code wrote `new Uint8Array(audio.buffer as ArrayBuffer)` to the
temp file — the whole underlying `ArrayBuffer`, ignoring `audio.byteOffset`/
`audio.byteLength`. Every real caller today happens to pass a fresh
`Float32Array` from `lib/live.ts`'s `concat()` helper (always offset 0, full
length), so this never manifested, but `LiveTranscribeFn`'s type signature
(`(audio: Float32Array) => ...`) makes no such promise to callers. Restored
the original's `audio.byteOffset, audio.byteLength` slice (which the
pre-refactor code already had — it would have been an actual regression to
drop it during the DI rewrite).
**Consequence:** the server-live path now has the same testability and
input-validation posture as the HTTP path; no behavior change for a valid
client, since `App.tsx` already only ever sends catalog model ids and
`"transcribe" | "translate"` task values.
**Alternatives rejected:** leaving `live.ts` untested and testing only
through a real spawned sidecar (matches `createSidecar`'s own integration
tests, but that suite already covers the sidecar process itself — the gap
was specifically the WebSocket/`LiveTranscriber` glue, which needs no Python
process to exercise).

## D-033 — "Downloading model…" said that even when the weights came straight from cache
**Status:** Accepted (2026-07-30)
Reported symptom: with a model's weights already fully cached in the browser
(Cache API, D-005), starting a load still showed `ModelLoadProgress`'s
heading, "Downloading model `<id>`", and its "this happens once" hint — as
if bytes were coming over the network, when nothing was.
**Cause.** `node_modules/@huggingface/transformers/src/utils/hub.js`'s
`loadResourceFile` computes a local `cacheHit` boolean but its
`progress_callback` dispatch always fires `{ status: 'download', ... }`
unconditionally — `cacheHit` only reaches the callback on a Firefox-specific
branch that skips progress entirely. So `inference.worker.ts`'s
`forwardProgress()` has no reliable "this came from cache" signal to forward;
it just relays `initiate`/`download`/`progress`/`done` verbatim, and
`ModelLoadProgress`'s heading was hardcoded to the download copy regardless.
**Fix.** Rather than patch the vendored library for a cosmetic flag (the
existing `bun patch` at `patches/@huggingface%2Ftransformers@4.2.0.patch` is
reserved for the two structural D-027/D-028 session-loading changes — adding
an unrelated third patch just for this is disproportionate and fragile
against future upstream bumps), the app answers the question itself, before
the load ever starts, from information it already tracks: `App.tsx` already
queries `lib/modelCache.ts`'s `getDownloadedModels` into a `downloaded`
`Set<string>` for `ModelPicker`'s "available offline" badges. A new
`loadedFromCache` state is set from `downloaded.has(nextModelId)` at the top
of `startLoad()` — the single function every load path (file-drop lazy load,
live-start lazy load, the explicit Download button, and the error box's
Retry) already funnels through — and reset to `false` everywhere `loadStarted`
/`loadProgress` are reset (model switch, device-preference switch, cancel,
the server-engine forced model switch), so a stale flag never survives into
an unrelated load. It is threaded as a new required prop into
`ModelLoadProgress`, whose heading/hint now read "Loading model from cache…"
/ "This model is already downloaded — just loading it into memory." when
true, unchanged otherwise. No worker protocol change: the decision is made
entirely from state the UI has before it ever posts the `load` message.
**Partial-cache handling.** `matchCachedModels` (the pure matcher backing
`getDownloadedModels`) flags a model "downloaded" on any single cached
`.onnx` file, not all of them — a genuinely partial download (e.g. encoder
cached, decoder not) would misreport as fully cached here. This is the same
imprecision `ModelPicker`'s existing "available offline" badge already
lives with; per this fix's scope, no new finer-grained per-file cache
probing was added — `modelCache.ts` stays the single source of truth for
"is this downloaded," reused as-is rather than duplicated at a different
granularity.
**Consequence:** the loading UI now matches reality for the common
already-cached case (reopening the app, switching back to a previously used
model); first-time downloads are unaffected. `ModelPicker`'s own "Download"
button never needed a matching change — it is only rendered for
`!isDownloaded` models, so it can never show "Downloading…" for an
already-cached one.
**Revisit:** if `modelCache.ts` ever gains a per-file/fully-cached check
(e.g. for the storage-management panel's exact byte accounting), reuse it
here too so a genuinely partial cache falls back to the "Downloading" copy
instead of "Loading from cache."

## D-034 — Browser-engine live status no longer claims "Listening" before the model is ready
**Status:** Accepted (2026-07-30)
**Symptom:** On the browser engine (not `useServer`), clicking "Start live
transcription" showed the full "recording" UI — the pulsing red `live-dot`
and the "Listening — like everything else, your audio stays on this device."
hint (`src/components/LiveControls.tsx`) — as soon as microphone permission
was granted, even when the model still had to load (first use of a model, or
a model/device switch mid-session that reset the pipeline). The user saw what
looked like an active live session while nothing was actually being
transcribed yet.
**Cause:** in `startLive()`'s browser-engine branch (`src/App.tsx`), the
`await startMicCapture(...)` call was immediately followed by
`setLiveStatus("listening")` with no check on whether the pipeline had
actually reported `ready`. The model load itself was already correctly
kicked off just above (idempotent `startLoad`/`load` postMessage, D-020) and
`transcribeLiveSegment()` already correctly `await`s `browserModelReady()`
(the promise on `loadGate.current`, resolved by the worker's `{status:
"ready"}` message or rejected on a load-stage error/cancel, D-025) before
posting any segment — so no segments were ever lost or misrouted to a wrong
pipeline. This was a pure UI/status bug, not a data-correctness one. The
server-engine (`useServer`) branch never loads a browser pipeline, so it was
never affected and is untouched by this fix.
**Fix:**
1. `LiveStatus` (`src/components/LiveControls.tsx`) gains a `"loading-model"`
   member between `"starting"` and `"listening"`, labeled "Loading model…".
   It disables the Start/Stop button exactly like `"starting"`/`"stopping"`
   do, and — because only `status === "listening"` sets the component's
   `listening` boolean — it renders neither the `live-dot` nor the
   "Listening — …" hint (falls into the same "not listening yet" branch as
   `"idle"`/`"starting"`/`"stopping"`).
2. In `startLive()`, once `startMicCapture` resolves and `liveRef.current` is
   set: if `loadProgress.ready` is already `true` (pipeline loaded earlier
   this session, no model/device switch pending), status goes straight to
   `"listening"` as before. Otherwise status becomes `"loading-model"` and the
   code `await`s `browserModelReady()`:
   - On resolve, it re-checks `liveRef.current === session` (the user could
     have hit Stop, or started a fresh session, while the wait was pending)
     before flipping to `"listening"` — never resurrects a torn-down session.
   - On reject (load failed), it stops the mic and clears `liveRef.current`
     (same re-check) and returns status to `"idle"`, **without** calling
     `setError()` itself — the worker's own `"error"` message handler already
     calls `setError()` for `stage === "load"` failures (`src/App.tsx`
     ~line 380), so a second call here would have shown two error messages
     for one failure.
3. `stopLive()` needed no change: its existing guard
   (`if (!session || liveStatus !== "listening") return;`) already makes it a
   no-op during `"loading-model"`, exactly mirroring how it already no-ops
   during `"starting"` — consistent with the Stop button being disabled for
   both statuses per (1).
**Consequence:** live-dot/hint/button now truthfully reflect whether audio is
actually reaching a ready pipeline. The only added latency is the same wait
`transcribeLiveSegment` was already imposing on the first segment pass — nothing
downstream changed. `tests/live.md`'s permission-flow step was updated to
describe the new "Loading model…" interim state instead of the old
immediate-"Stop" behavior.
**Alternatives rejected:** gating on `loadGate.current === null` instead of
`loadProgress.ready` — `loadGate.current` is only nulled out on an explicit
model/device-switch reset or load cancel, not after a successful `ready`, so
it stays non-null (holding an already-resolved promise) for the rest of the
session; checking it would have made every subsequent live start take a
detour through `"loading-model"` for a promise that resolves on the same
tick, an unnecessary (if harmless) status flash. `loadProgress.ready` is the
same signal `showModelProgress`/`modelReady` already use elsewhere in
`App.tsx` and tracks exactly what the fix needs: has *this* model/device
selection reported ready yet.
**Testing:** no unit test added — `LiveControls` has no existing
component-level test file to extend, and the state machine this fix changes
(`startLive()`'s wait/race handling around a real `Worker`, `MediaStream`, and
DOM) has no natural seam for a `bun test` unit test without mocking most of
the worker/mic stack, which the project's live tests (`tests/live.test.ts`,
`tests/live-accuracy.test.ts`) deliberately avoid — they test the pure
segment/scheduling logic in `src/lib/live.ts` instead. Manual verification
via `tests/live.md` remains the intended coverage for this UI path, and its
permission-flow step was updated accordingly (see Consequence above).

## D-035 — `whisper-tiny` (multilingual) "slower than whisper-tiny.en" is real, but only on out-of-distribution audio, and it is a decode-length effect, not an asset/config gap
**Status:** Investigated, no code fix warranted (2026-07-30)
Reported symptom: `onnx-community/whisper-tiny` (multilingual) feels "a good
bit slower" than `onnx-community/whisper-tiny.en` at inference. Three
candidate mechanisms were checked, in order, using the
`Symbol.for("onnxruntime")` ort-web-under-Bun harness (real onnxruntime-web
wasm EP, q8 dtype — the catalog's WASM default for both models, D-006) driven
through the exact pipeline call `src/worker/inference.worker.ts` makes
(`chunk_length_s: 30, stride_length_s: 5, return_timestamps: true,
max_new_tokens: 440`, D-021's single-pass path), single-threaded to avoid
cross-process CPU contention (a real hazard in this sandbox: two unrelated
sibling-agent `bun test` runs at 500%+ CPU each inflated early timings 3–5×
until killed — matches the pre-existing "ort-web under Bun harness" tooling
note about contention).
1. **Hub asset parity — ruled out.** `curl`'d the Hub tree API directly (not
   just `WebFetch`'s HTML summary) for both repos' `onnx/` directories: both
   ship an equivalent `decoder_model_merged_quantized.onnx` (merged,
   KV-cache-enabled) at nearly identical size — multilingual 30,719,241 B vs
   `.en` 30,718,858 B (383 B apart, consistent with one extra vocab row:
   51865 vs 51864 tokens). No fallback to a non-merged/non-cached decoder for
   either model.
2. **Catalog/dtype config — ruled out** (per the task brief, already checked
   before this investigation): both catalog entries resolve to the same
   `wasm`/`webgpu`/`webnn` dtypes; the `verified` flag difference is
   metadata-only.
3. **Actual measured timing — this is where the real, reproducible effect
   is, and it's content-dependent:**
   - On `tests/audio_files/en_aiproblem.m4a` (14.4 s, clean English speech,
     both models transcribe it correctly): no reliable difference.
     Interleaved back-to-back runs (both orders tried, to rule out
     first-vs-second bias) gave multilingual/`.en` warm-average ratios of
     0.938 and 0.891 — i.e. multilingual was *slightly faster*, well within
     run-to-run noise (individual pass times ranged 1150–1460 ms across
     both models). The "extra 1–2 prompt tokens" mechanism the task brief
     had already flagged as probably-negligible is confirmed negligible here.
   - On `tests/audio_files/da_festuge.m4a` (21.5 s, Danish speech, no
     `language` option passed — the real default when a user leaves the UI's
     language picker on "auto", exactly as `src/App.tsx` sends it): **both
     models mistranscribe it as English and fall into a greedy-decoding
     repetition loop** (a known Whisper failure mode on audio that doesn't
     match the assumed language, not specific to this app), but the two
     checkpoints' loops differ hugely in how many tokens they run before
     landing on `eos_token_id`:
     - `whisper-tiny.en`: 67 generated tokens (`" The first one is the first
       one. ... "` repeating), pass time 2.6–5.7 s.
     - `whisper-tiny` (multilingual): 434 generated tokens — essentially the
       entire `max_new_tokens: 440` budget — (`" ... the main clar, the main
       clar, ..."` repeating), pass time 9.5–14 s.
     - Ratio: ~6.5× more tokens, ~2–5× wall-clock (sub-linear vs. token count
       because KV-cache growth cost isn't the whole per-token cost; still a
       large, user-visible gap). A second same-order and a reversed-order
       run reproduced the same order-of-magnitude split.
**Mechanism:** this is not a compute-cost-per-token difference (ruled out by
the clean-English measurement) and not an asset/config gap (ruled out above).
It's that the multilingual model's forced decoder prompt carries an extra
task token (`<|transcribe|>`, from its `forced_decoder_ids`) that `.en`'s
prompt doesn't have, and — on audio where the model is already
out-of-distribution enough to hallucinate — that different prompt context
measurably changes *how long the hallucination loop runs before EOS*, not
just the compute cost of the prompt tokens themselves. Because this app's
`max_new_tokens: 440` (D-021) forces the classic single-pass path with no
`no_repeat_ngram_size`/`repetition_penalty` set in either model's
`generation_config.json` or in `inference.worker.ts`'s pipeline options,
there is nothing to cut a non-terminating decode short before the hard
token cap — so whichever checkpoint happens to run closer to that cap on a
given input is dramatically slower for that one pass. Confirmed generation
configs for both models differ only in the multilingual-specific fields
(`lang_to_id`, `task_to_id`, an extra forced token at position 2) — neither
sets a repetition guard.
**Fix:** none applied. This is inherent behavior of the two upstream
checkpoints reacting differently to the same repetition-loop failure mode,
not a bug in this app's catalog, config, or asset selection — there is
nothing in `src/lib/models.ts` to change (both already resolve to identical,
merged, cached-KV decoders at the same dtype on every device tier). Inventing
a per-model workaround (e.g. steering multilingual `.en`-adjacent use cases
onto the `.en` model) was rejected: it would silently break real multilingual
transcription for users who need it, to paper over a worst-case latency tail
on inputs the model was always going to mistranscribe anyway.
**Scope:** tiny/tiny.en only measured; the same forced-decoder-ids asymmetry
exists for every multilingual/`.en` pair in the catalog (D-006), so the same
content-dependent latency tail plausibly applies to base/small/medium too,
untested here.
**Revisit:** if this becomes a recurring complaint (not just a one-off
report), the actionable mitigation is adding `no_repeat_ngram_size` (e.g. 3)
and/or a `repetition_penalty` to the pipeline options in
`inference.worker.ts` — that targets the actual failure mode (unbounded
repetition loops on out-of-distribution audio) for both `.en` and
multilingual models, rather than papering over one symptom of it. That
change needs its own accuracy verification against
`tests/live-accuracy.test.ts` and `tests/live-real-model.test.ts` (a
repetition guard can change correct-transcript wording too) before landing,
which is out of scope for this investigation.

## D-036 — Root `AGENTS.md` is the single source of truth for agents
**Status:** Accepted (2026-10-29)
`docs/CLAUDE.md` called itself the single source of truth, but Claude Code
only auto-loads it when working inside `docs/`, other agents (Codex, Gemini,
Cursor) never read it, and most of it was the generic `bun init` boilerplate
(sample `Bun.serve`/HTML snippets) rather than repo-specific rules.
**Decision:** a root `AGENTS.md` (read natively by Codex/Gemini, imported by
the root `CLAUDE.md` via `@AGENTS.md`) holds the rules: verified commands, the
conventions agents would otherwise break (Bun only, D-013 engine split, the
transformers patch, COEP, D-020, D-030, D-032), boundaries, and docs duties.
`CLAUDE.md`, `GEMINI.md`, and `docs/CLAUDE.md` are pointers only. The product
spec from `docs/CLAUDE.md` §1 moved to `PRODUCT.md`; `DESIGN.md` records the
shipped visual system (both read by the impeccable skill). Specs from the
superpowers workflow live permanently in `docs/specs/`; plans live in
`docs/plans/` only while work is in progress and are `git rm`'d before
merging. Agent tool permissions are defined in `docs/PERMISSIONS.md` and
rendered to `.claude/settings.json`, `.codex/`, `.gemini/settings.json`.
**Kept as-is:** `docs/Architecture.md`, `docs/Decisions.md` (same filenames —
renaming would break every existing reference) and `docs/WorkBreakdown.md`
as the one backlog (its own rule: no separate backlog), rather than adding a
`docs/TODO.md`.
**Trade-off:** `AGENTS.md` is kept short (<150 lines) on purpose — rationale
goes here, not there. Changes to it redirect every agent session, so review
them like CI config.

## D-037 — Light theme via `prefers-color-scheme`; Inter self-hosted outside Bun's CSS bundler
**Status:** Accepted (2026-10-29)
The UI was dark-only by accident: `:root` declared `color-scheme: light
dark` but every colour was a hard-coded dark value, so light-mode OS users
got native light form controls on a dark page. The font stack also named
`Inter` first without ever loading it — everyone saw the system font.
**Theme:** every colour is now a token on `:root` (dark values unchanged,
so the dark theme renders identically); `@media (prefers-color-scheme:
light)` overrides the tokens with a mirrored light palette (same blue
accent family, darkened for contrast). No in-app toggle — follow the OS.
All text/background pairs measured ≥ 4.5:1 in both themes. The PWA
manifest/icons stay dark (a manifest has one `theme_color`); the
`<meta name="theme-color">` tags follow the scheme.
**Font:** maintainer chose to self-host Inter (over dropping it from the
stack). Source: `@fontsource-variable/inter` (OFL-1.1), a devDependency
like Transformers.js (D-031) because only the build reads it. COEP
`require-corp` (D-023) rules out a font CDN anyway. Two Bun 1.3.14
behaviours shaped the wiring (`scripts/fonts.ts`):
1. Bun's HTML-entrypoint CSS bundler **inlines every resolvable `url()`
   asset as base64** — `loader: { ".woff2": "file" }` is ignored. Bundled
   normally, the 7 subsets turned the render-blocking stylesheet from
   14 KB into 305 KB and defeated `unicode-range` lazy loading. So the
   production build marks `*.woff2` `external`, copies the subsets to
   `dist/fonts/`, and precaches them (and hashes their bytes into the SW
   cache version). The dev server has no `external` option, so
   `ensureFonts()` copies them into gitignored `src/fonts/` where the dev
   bundler can resolve (and inline) them.
2. Bun's minifier rewrites `format('woff2-variations')` to the unquoted
   `format(woff2-variations)`, which is not a valid keyword — Chrome then
   drops the whole `src` and registers **no** faces. Use `format("woff2")`
   (variable woff2 needs nothing more).
Verified in headless Chrome: 7 faces registered, only the latin subset
fetched for English UI text.

## D-038 — Typecheck with TypeScript 7; CI gates image builds on typecheck + fast tests
**Status:** Accepted (2026-10-29)
`tsconfig.json` was already strict, but `typescript` was not installed and
Bun strips types without checking them, so nothing ever type-checked.
**Decision:** `typescript` (7.x, the native compiler — a full check of all
57 project files takes < 1 s) is a devDependency; `bun run typecheck`
runs `tsc --noEmit`. The code base was already clean. `.gitlab-ci.yml`
gains a `check` stage before `build`: `bun install --frozen-lockfile`,
`bun run typecheck`, and the fast test suite (`bun test
--path-ignore-patterns='tests/live-real-model.test.ts'`, ~6 s). The
real-model accuracy test stays local-only: it downloads ~600 MB of
weights and runs for many minutes on CPU.
**Consequence:** a type error or failing unit test now blocks image
pushes.

## D-039 — Real-model accuracy test: time budget scales with clip length; per-clip WER bars only with a measurement
**Status:** Accepted (2026-10-29)
`tests/live-real-model.test.ts` had failed since `en_alien` (the 15-min
D-021 long-file fixture) was added on 2026-07-15: the test auto-enrolls
every clip in `tests/audio_files/`, but its 900 s timeout and absolute
`MAX_WER = 0.25` were calibrated on 14–21 s clean clips.
**Investigation (instrumented run, no timeout):** no hang — 124 transcribe
passes, 1189 s total (whole-file pass 420 s for 888 s of audio; live
passes re-transcribe overlapping segments, 3452 s of audio in total).
With the time available it then failed on accuracy instead: live WER
0.251, and the whole-file baseline itself 0.261 — the clip (gaming
livestream: music, game voices, crosstalk) is simply harder than the
bar, independent of the live pipeline. The relative check (live ≤ file +
0.1) passed.
**Decision:** each clip's timeout is `900 s (download headroom) + 3 s ×
clip seconds` (duration from an ffmpeg header probe; ≈ 2× the measured
1.34 s/s). `MAX_WER_BY_CLIP` gives `en_alien` an absolute bar of 0.35 with
the measurement recorded beside it. **Rejected:** deriving the absolute
bar from the whole-file WER — a broken model would then pass whenever its
whole-file output is equally bad, which is exactly what the absolute bar
exists to catch.
**Consequence:** full `bun test` ≈ 19 min on CPU (en_alien ≈ 20 min of it
on a cold run); the fast suite excludes this file.

## D-WS1a — Transformers.js 4.2.0 → 4.3.2, sequential-session patch carried over
**Status:** Accepted (2026-10-29, spec 001 Goal 1)
`@huggingface/transformers` is now pinned to exactly `4.3.2` (latest; 4.3.0
added Safari 26+ WebGPU). It brings onnxruntime-web
`1.31.0-dev.20260914-8d85527a0` (was `1.26.0-dev.20260416`) and
onnxruntime-node `1.30.0` (was `1.24.3`).
**Patch.** Upstream `constructSessions` (`dist/transformers.web.js`) is
still the bare `Promise.all` over sub-model sessions, so D-027/D-028's
`options.sequentialSessionLoad` branch is still needed. Regenerated with
`bun patch` as `patches/@huggingface%2Ftransformers@4.3.2.patch` (the same
hunk, byte-identical apart from line offsets); the 4.2.0 patch is deleted.
The version is pinned exactly (no caret) because the patch is keyed by
version: a caret range would let a fresh lockfile resolve a version the
patch does not apply to.
**Guard.** `tests/transformers-runtime.test.ts` (fast suite) fails if the
installed version, the `patchedDependencies` key, or the patched
`constructSessions` body drift apart — a lost patch would otherwise only
show up as mobile tab crashes.
**Unchanged by the bump:** onnxruntime-web's WebGPU bundle still loads the
`ort-wasm-simd-threaded.asyncify.{mjs,wasm}` pair (same as 1.26), and
`scripts/build.ts` / `/ort/` already copy every `ort-wasm-*` file.
**Parakeet (for spec 001 Goal 7):** 4.3.2 supports only the CTC variant —
`ParakeetForCTC` (`model_type: "parakeet_ctc"`, routed through the ASR
pipeline's wav2vec2/CTC path) plus `ParakeetFeatureExtractor`. There is no
TDT/RNN-T (transducer) decoder or joint-network loop anywhere in the
package, so **Parakeet TDT 0.6B v3 is not supported natively**; per the
spec, that path uses `parakeet.js`.

## D-WS1d — WebGPU-side effects of the 4.3.2 upgrade: inspected, not measured (needs the owner's browser check)
**Status:** Open — inspection only (2026-10-29)
This machine has no GPU and no browser WebGPU, so D-022, D-025 and D-026
cannot be re-measured here. What the upgrade changes on that side, from the
4.3.0–4.3.2 release notes and a 4.2.0↔4.3.2 source diff:
- **onnxruntime-web 1.26.0-dev.20260416 → 1.31.0-dev.20260914.** Five minor
  versions of WebGPU EP work land at once. The WebGPU bundle still uses the
  same `ort-wasm-simd-threaded.asyncify` binary pair (now 26.9 MB, was
  23.6 MB — a one-time re-download, evicted correctly because the SW cache
  version hashes the worker bundle, which embeds the ort version). The JS
  layer still has no Firefox/Gecko or Safari special-casing, and still
  checks `shader-f16` / `isFallbackAdapter` as before.
- **Safari 26+ WebGPU (Transformers.js 4.3.0, PR 1700).** The upstream change
  only alters which binary Transformers.js picks for its jsDelivr default
  (`IS_SAFARI` → `IS_SAFARI_BELOW_26`). This app overrides `wasmPaths` with
  `./ort/`, so it never used that default; whether WebGPU is attempted on
  Safari is decided by the worker's own probe (`navigator.gpu`, D-017/D-025),
  unchanged.
- **Whisper / ASR pipeline / streamers:** no functional change (comment
  edits, `processor.batch_decode` → `tokenizer.batch_decode` on a
  non-Whisper path, one `transpose` → `permute` rename). "Fix Whisper
  progress callbacks" and "fix duplicate model file downloads when
  `progress_callback` is active" (4.3.0) are in the file-loading path:
  download progress may look different, and a duplicate fetch is avoided.
- **Session/dtype selection (`session.js`, `dtypes.js`):** unchanged apart
  from no longer exporting two constants.
**Needs the owner's browser check** (decisions stay as they are until then):
1. **Large v3 Turbo on WebGPU (the owner's setup):** load and transcribe
   `tests/audio_files/en_aiproblem.m4a`; compare wall-clock with the 4.2.0
   build and confirm the text is sane (fp16 encoder + q4 decoder, D-022
   scope note).
2. **D-026:** Firefox with Backend = WebGPU explicitly selected — still far
   slower than WASM? If it is now at parity, drop the Firefox UA check.
3. **D-022:** medium stays q4 on WebGPU/WebNN. A retest of the fp16
   conversions is only worth doing if the owner wants medium faster on GPU.
4. **D-025:** Firefox "auto" lands on multithreaded WASM (footer shows
   "WASM (on-device, N threads)" with N > 1).
5. **Safari 26 (macOS/iOS):** what the footer reports for "auto"; if WebGPU,
   check speed and output.

## D-WS1b — D-021's `max_new_tokens: 440` is still needed on Transformers.js 4.3.2
**Status:** Accepted (2026-10-29) — re-test of D-021 part 3, which stands unchanged
**Source:** 4.3.2's `WhisperForConditionalGeneration.generate()` still takes
the seek loop on exactly D-021's condition,
`generation_config.return_timestamps && !kwargs.max_new_tokens`
(`src/models/whisper/modeling_whisper.js`); `_generate_with_seek` is
unchanged. `tests/transformers-runtime.test.ts` now pins that line, so a
future bump that changes it fails the fast suite and prompts a re-measure.
**Measured** (tiny.en q8 — the catalog WASM default — on onnxruntime-node,
CPU, 16-core shared machine, same harness for both versions; pipeline options
as the worker: 30 s chunks, 5 s stride, `return_timestamps`; 888 s
`en_alien.opus`, 44 windows):

| Transformers.js | `max_new_tokens` | generate passes | wall-clock | WER |
|---|---|---|---|---|
| 4.2.0 | 440 | 44 | 54.7 s | 0.315 |
| 4.2.0 | unset (seek loop) | 83 | 66.6 s | 0.447 |
| 4.3.2 | 440 | 44 | 55.5 s | 0.315 |
| 4.3.2 | unset (seek loop) | 84 | 119.7 s | 0.449 |

Pass counts and WER are deterministic and reproduce D-021 (80 passes,
0.447 vs 0.315); wall-clock is noisy here (other workstreams' light jobs
share the CPU), so read it only as "the seek loop is slower", not as a
version-to-version speed comparison.
**Decision:** keep `max_new_tokens: 440` in `src/worker/inference.worker.ts`
and in `tests/live-real-model.test.ts`.

## D-WS1c — WASM sessions back at the default graph optimization level (D-014's "basic" cap dropped)
**Status:** Accepted (2026-10-30) — supersedes D-014
**Re-test** of D-014's revisit condition after the Transformers.js 4.3.2 bump
(D-WS1a), which brings onnxruntime-web `1.31.0-dev.20260914-8d85527a0`.
Measured under Bun with onnxruntime-web's own wasm EP (not onnxruntime-node),
1 thread, AMD Ryzen 9 9950X shared with other workstreams.
1. **Session creation**, `onnx-community/whisper-tiny`
   `decoder_model_merged_quantized.onnx` (D-014's repro file), every level:

   | onnxruntime-web | disabled | basic | extended | all |
   |---|---|---|---|---|
   | 1.26.0-dev.20260416 (control) | OK | OK | FAIL | FAIL |
   | 1.31.0-dev.20260914 | OK | OK | OK | OK |

   The control still fails with D-014's exact error (`qdq_actions.cc:137
   TransposeDQWeightsForMatMulNBits Missing required scale:
   model.decoder.embed_tokens.weight_merged_0_scale`); 1.31 loads at all four.
2. **Transcription**, Transformers.js 4.3.2 pipeline on that ort-web build,
   `whisper-tiny.en` q8 (catalog WASM default), `en_aiproblem` (14.4 s),
   worker options (30 s chunks, 5 s stride, timestamps, `max_new_tokens:
   440`). Both pipelines in one process, 1 warm-up each, then 8 interleaved
   runs (A/B order alternating) so machine-load drift hits both equally:

   | level | warm runs (s) | median | WER | text |
   |---|---|---|---|---|
   | `basic` (D-014) | 1.49 1.44 1.36 1.39 1.41 1.47 1.48 1.60 | 1.47 s | 0.000 | identical |
   | default (`all`) | 1.38 1.47 1.34 1.39 1.37 1.42 1.43 1.43 | 1.42 s | 0.000 | identical |

   Same transcript byte-for-byte at both levels and across every run; the
   default level is ~3 % faster here (within noise; certainly not slower).
**Decision:** `src/worker/inference.worker.ts` no longer passes
`session_options` — WASM and WebGPU both use onnxruntime's default level.
`tests/transformers-runtime.test.ts` pins the onnxruntime-web version this
was measured on and that the worker sets no `graphOptimizationLevel`; a
different ort-web build fails the fast suite until this session-creation
check is re-run.
**Not covered here:** a real browser's ort-web wasm build (the same
`ort-wasm-simd-threaded.*` files, but multithreaded under COOP/COEP) —
`tests/offline.md` §2's WASM step is the owner's browser check.

## D-WS4a — Whisper's "Translate to English" only on models trained for it (`canTranslate`)
**Status:** Accepted (2026-10-29) · spec 001 Goal 4a
`WhisperModel` gains `canTranslate: boolean` (`src/lib/models.ts`): **true** for multilingual
tiny / base / small / medium, **false** for every `.en` model (no task token) and for
`whisper-large-v3-turbo` — OpenAI's model card says turbo was fine-tuned on transcription data only
and is not trained for translation, so its "translate" output is unreliable. Any future turbo-based
fine-tune defaults to false until tested. `TranscriptionOptions` renders the Task control only where
true (language stays). **Stale task:** the `task` state is kept, but `taskForModel(modelId, task)`
maps "translate" to "transcribe" for a model that can't translate; App applies it on model switch
(picker and the server-engine auto-switch to turbo) and again when building every file / live run's
options, so a leftover "translate" can never reach the worker or the server. **Not changed:** the
server's `parseLanguageAndTask` (D-032) still accepts "translate" for any model — the browser never
sends it for turbo now, and a direct API caller asking faster-whisper to translate with turbo gets
what faster-whisper does; rejecting it server-side would be a behaviour change for API users with no
UI benefit. Speech in another language than English can still end up in English (or Danish) through
text translation of the finished transcript (D-WS4b), for every model.

## D-WS4b — Text translation of finished transcripts with Opus-MT, in its own worker
**Status:** Accepted (2026-10-29) · spec 001 Goal 4b
**What:** any finished transcript — browser file, browser live, server engine — can be translated
da→en or en→da from a secondary "Translate transcript…" disclosure under the transcript
(`TranslatePanel`). Whisper's own task keeps its label "Translate to English" (D-WS4a). The
translation renders as a second transcript card ("Translation (English)") with its own
`.txt` / `.srt` / `.txt+ts` exports named `<file>.<en|da>.<ext>`.
**Models:** `Xenova/opus-mt-da-en` and `Xenova/opus-mt-en-da` — Transformers.js ONNX exports of
Helsinki-NLP's Opus-MT (MarianMT, ~77 M params; upstream licence Apache-2.0, the Xenova repos
carry no separate licence). Loaded at **q8**: encoder 49.8 MB + merged decoder 57.1 MB +
tokenizer/config 5.7 MB = **~113 MB per direction**, one-time. Verified to load and translate under
the pinned Transformers.js 4.2.0 (onnxruntime-node; headless-Chrome WASM run below). WASM only, with
`graphOptimizationLevel: "basic"` (D-014 — Marian ties its embeddings like Whisper's 8-bit decoders).
**Where it runs — a dedicated worker** (`src/worker/translate.worker.ts`, served at
`/translate.worker.js`, precached), not a message type on the Whisper worker: onnxruntime fixes its
WASM thread pool at first init (D-023), so a Whisper worker initialised for WebGPU — the owner's
setup — would run MT single-threaded; a separate worker also keeps a Whisper cancel/restart
(D-018/D-025) from killing a translation and vice versa, and keeps workstreams 2/3's edits to
`inference.worker.ts` conflict-free. Nothing goes to the server for any engine (anti-goal: text stays
on the device). **Downloads:** only when the user clicks Translate; the hint states the size first
("Downloads a translation model once (~113 MB), then works offline") and switches to "Translation
model downloaded — works offline" once the Cache API holds it (D-005/D-020). The storage panel lists
and deletes the two models like Whisper weights.
**Chunking (Review Focus 5):** `buildTranslationUnits` merges Whisper chunks until one ends a sentence
(cap 8 chunks) and gives the unit the merged time span, so `.srt` keeps working with slightly longer
cues; untimed text is split per line. Each unit is cut by `splitForTranslation` into **one piece per
sentence** (`Intl.Segmenter`, abbreviation/initial fragments merged forward), an over-budget sentence
wrapped on commas, then words, under 200 real tokens (Opus-MT's input limit is 512; the tokenizer
otherwise *truncates silently*). One sentence per call is required, not just tidy: given two
sentences Opus-MT dropped the second one in the probe ("…I live in Copenhagen. How are you?" →
first sentence only). Order and every word are pinned by unit tests on all of `en_alien.txt` and by
the opt-in real-model test (`MT_INTEGRATION=1`), which also translates a 900-word unpunctuated run-on.
**Batching is off (batch size 1):** Transformers.js 4.2's batched Marian generation is broken —
every row of a padded batch comes back as row 0's translation (q8 and fp32, any optimisation level,
onnxruntime-node). Batch 8 would be ~4× faster; re-measure with `scripts/bench-translate.ts` after
the Transformers.js upgrade (workstream 1) before raising `DEFAULT_TRANSLATE_BATCH_SIZE`.
**Measured (onnxruntime-node, CPU, 16 cores shared with other workstreams; not browser numbers):**
`en_alien.txt` (7 470 chars, 279 lines → 293 sentence pieces) en→da **58 s** (~128 chars/s,
~0.2 s/sentence) in the bench (71 s / ~105 chars/s on a re-run with other workstreams' jobs
running), 76 s in the real-model test under load; `da_festuge.txt` (3 sentences)
da→en 2.8 s incl. warm-up; model load from cache 0.3–0.4 s. Extrapolated, a 90-minute meeting
(~80 000 chars) is ~10 min on this CPU — a background action, hence the sentence-count progress and
Cancel. **Headless Chrome (WASM, `--disable-gpu`, cross-origin isolated, 12 threads):** a real
end-to-end run — Whisper Tiny transcript of `da_festuge.m4a`, "Translate transcript…" → download
`Xenova/opus-mt-en-da` → "Translating… 0 of 1 sentence" → "Translation (Danish)" card with four
exports — took 4–8 s from click to result for that one-sentence transcript (download included);
no browser number for a long transcript was taken. Sample da→en: "At the Aarhus Festival Secretariat we have been running very strong in the
last few months."
**Errors** go through `classifyError` with the new stage `"translate"` (out-of-memory suggests closing
tabs instead of a smaller Whisper model; a failed download is a retryable network error). Cancel
terminates the translate worker only.
**Not done / follow-ups:** WebGPU for Opus-MT (unmeasurable here); more language pairs (the catalog
is two entries, `TRANSLATION_MODELS`); a translation of live text *while* recording.

## D-WS6a — Danish fine-tuned Whisper models: our own ONNX + CTranslate2 conversions, unpublished until the owner says go
**Status:** Proposed (2026-10-29, spec 001 Goal 6)
**Candidates checked (licence + architecture, from each repo's model card, config and notices):**
- `thorhojhus/whisper-small-danish` — MIT; plain `WhisperForConditionalGeneration`, whisper-small
  shapes and tokenizer unchanged (80 mel bins, 12/12 layers, vocab 51865). **Kept.**
- `thorhojhus/whisper-large-v3-turbo-danish` — MIT; plain large-v3-turbo (128 mel, 32/4 layers,
  vocab 51866). **Kept.**
- `danish-foundation-models/edda-v0.2` — **Apache-2.0 confirmed** (licensor: the Alexandra
  Institute; base model MIT); plain large-v3-turbo architecture, so Transformers.js loads it like the
  stock turbo. Its `generation_config.json` pins `language: "da"`, `task: "transcribe"` and
  `num_beams: 5`. **Kept.**
**Catalog:** three entries with `verified: false` everywhere, ids built from one exported constant
`DANISH_MODELS_HF_OWNER` (placeholder `"PENDING-OWNER"`): `<owner>/<name>-ONNX` for the browser
engine and `<owner>/<name>-ct2-int8` as `fasterWhisperModel` (a Hub repo id — faster-whisper accepts
those as well as its aliases). Publishing under the owner's account is a one-line change.
**Conversions (outside the repo, in `/home/user/hf-publish/<repo>/`; nothing uploaded):** optimum 2.1
`main_export` (fp32, KV-cached merged decoder) → onnxruntime fp16 / dynamic q8 / MatMulNBits q4 with
the stock onnx-community settings; `ct2-transformers-converter --quantization int8`. File names, sizes
and `transformers.js_config.use_external_data_format` match the stock `onnx-community/whisper-small`
and `-large-v3-turbo` repos to within 0.2 MB per file, with one exception: our q8 turbo decoder
(173 MB) quantizes the token embedding, while the stock one (440 MB) keeps it in fp32. Two export
traps, now fixed in the tooling and described in each model card: optimum needs `accelerate`
installed, or the tied `embed_tokens`/`proj_out` weight is stored twice (+159 MB on small); and
onnxruntime's fp16 converter makes the merged decoder's If branches cyclic unless their outputs get
branch-local names first. `generation_config.json` (`<|da|>` = 50285, `task_to_id`) and the tokenizer
are copied unchanged and asserted equal to the source.
**Measured** (WER on `da_festuge` with `language: "da"`; ONNX loaded from the local folder through
Transformers.js 4.2.0 on onnxruntime-node, CT2 through faster-whisper 1.2.1 int8 beam 5; CPU only, one
21.5 s clip, so the results are a conversion sanity check, not a benchmark):

| model | q8 | q4 | fp16 enc + q4 dec | fp16 | CT2 int8 |
|---|---|---|---|---|---|
| whisper-small-danish | 0.109 | 0.094 | 0.094 | 0.094 | 0.078 |
| stock whisper-small | 0.297 | 0.281 | — | — | 0.234 |
| whisper-large-v3-turbo-danish | 0.141 | 0.141 | 0.094 | 0.094 | 0.094 |
| edda-v0.2 | 0.109 | 0.063 | 0.094 | 0.094 | 0.078 |
| stock whisper-large-v3-turbo | 0.125 | 0.141 | — | — | 0.094 |

The small fine-tune cuts WER by about a factor of three. Neither turbo fine-tune separates from the
stock turbo on this easy clip, so a longer, harder Danish test is needed to judge them. **D-022
check:** the turbo fp16 encoders do not overflow on onnxruntime-node; whole-model fp16 gives the
same transcript as fp16 + q4. WebGPU/WebNN behaviour still needs a real browser.
**Dtype defaults:** WASM q4 for all three, including small (q4 matched or beat q8 and was 1.4–1.6×
faster on CPU); WebGPU fp16 encoder + q4 decoder; WebNN fp16. All stay `verified: false`.
**edda's `num_beams: 5`:** Transformers.js 4.2 has no beam search (its `BeamSearchSampler` returns
`num_beams` candidates but generation keeps only the top-1), so browser decoding is greedy. It does
not crash; the only cost is a full-vocabulary top-k per step. The server engine (faster-whisper) does
use beam 5.
**Pending the owner:** the Hugging Face account name, then upload the six prepared folders (model
cards credit the original authors and carry MIT, or Apache-2.0 + NOTICE), then set
`DANISH_MODELS_HF_OWNER`. The integrator sets `canTranslate: false` on these entries at merge.

## D-WS5a — Server engine: batched faster-whisper + tuned Silero VAD for files; live passes stay sequential
**Status:** Accepted (2026-10-30, spec 001 Goal 5)
`server-py/sidecar.py` decoded every request with the sequential
`WhisperModel.transcribe`, no VAD.
**Decision:** the sidecar request gains `mode: "file" | "live"` (absent ⇒
`"file"`, so an older Bun still gets the new path). `"file"` (`POST
/api/transcribe`) runs `BatchedInferencePipeline(model).transcribe(audio,
vad_filter=True, vad_parameters={threshold: 0.2, speech_pad_ms: 800},
batch_size=8)`; the pipeline is a weightless wrapper cached and rebuilt with
the one resident model. `"live"` (server-live passes, `src/server/live.ts`)
keeps the sequential decoder. Beam size stays at the default 5.
**Measurements** (CPU int8, AMD Ryzen 9 9950X, 16 cores shared with other
jobs — load average 7–24 during runs, so wall-clock is ±20 %; WER via jiwer
on the `tests/live-real-model.test.ts` normalisation vs `tests/audio_files/*.txt`):

| model | audio | config | wall-clock | WER |
|---|---|---|---|---|
| large-v3-turbo | en_alien (888 s) | sequential (before) | 97 s / 130 s (two runs) | 0.097 / 0.067 |
| large-v3-turbo | en_alien | sequential + VAD | 79 s | 0.136 |
| large-v3-turbo | en_alien | batched, default VAD (0.5 / 400 ms) | 47 s / 59 s | 0.117 |
| large-v3-turbo | en_alien | batched, beam 1 | 44 s | 0.114 |
| large-v3-turbo | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **51 s / 58 s** | **0.086** |
| large-v3-turbo | en_alien | batched, VAD 0.2 / 800 ms, word-level timestamps | 56 s | 0.144 |
| large-v3-turbo | da_festuge (21.5 s, da) | sequential (before) | 2.9 s | 0.094 |
| large-v3-turbo | da_festuge | batched, default VAD | 2.9 s | 0.141 |
| large-v3-turbo | da_festuge | **batched, VAD 0.2 / 800 ms (after)** | **2.7 s** | **0.109** |
| medium.en | en_alien | sequential (before) | 114 s | 0.126 |
| medium.en | en_alien | **batched, VAD 0.2 / 800 ms (after)** | **67 s** | **0.111** |

The sequential decoder is not deterministic run to run on en_alien (temperature
fallback; 0.067–0.097 over three runs), while the batched result repeated
exactly. Default-VAD errors were deletions (121 vs 64 sequential): Silero at
0.5 dropped quiet speech under music; 0.2 / 800 ms recovered most of them.
da_festuge's gap is one word of 64. Live-style 6 s passes over the first
180 s of en_alien: sequential 2.45 s/pass, batched 2.12, sequential + VAD
1.88 — within the load noise, each losing a few words, and real live
passes are already speech-gated client-side (D-016), so live stays as it was.
**Consequences:** file transcriptions are ≈ 1.7–2.5× faster on CPU at
equal-or-better WER on en_alien (one word worse on the 21 s Danish clip). Segments are coarser — one per VAD chunk (≤ 30 s) instead of one per
sentence — because word-level timestamps (`without_timestamps=False`) cost
WER here (0.144); SRT/VTT cues from the server engine are correspondingly
long. Segments arrive in bursts of 8 chunks (≈ 4 min of speech) and the first
burst can take ~15–25 s on a long file; the D-021 heartbeat covers it, and
cancel lands at the next burst boundary. No new env vars.

## D-WS2a — "Names and terms" vocabulary hint as a Whisper decoder prompt, cut to 120 tokens
**Status:** Accepted (2026-10-29) — spec 001, Goal 2. Placeholder id; the
integrator renumbers it.
**Context:** users want names and jargon spelled their way. Whisper takes a
"previous text" prompt (`<|startofprev|> … <|startoftranscript|>`).
Transformers.js 4.2.0 — and 4.3.2, checked in workstream 1's worktree —
declare `prompt_ids` in the Whisper generation config but never use it (the
parameter is commented out in `WhisperForConditionalGeneration.generate`),
and the pipeline's `_decode_asr` prints every non-special decoder token, so
a prompt passed as `decoder_input_ids` would appear in the transcript.
**Decision:**
- UI: an optional "Names and terms" textarea for every model
  (`VocabularyHint`), raw text kept in localStorage (`whisper-vocabulary-hint`),
  ≤ 1000 characters, sent as `prompt` on the worker's transcribe message, on
  the server engine's query string, and on the live WebSocket URL.
- `src/lib/hint.ts` normalizes (one term per line or comma-separated →
  `"a, b, c"`, control characters dropped) and owns the **truncation rule**:
  prompt text is `" " + hint` ending in a full stop (unless it ends in
  `.?!`); over budget → the longest prefix of whole space-separated words
  that fits (dangling `,`/`;` removed, full stop re-added) — the start of the
  list is kept, the end dropped; if not even the first word fits, its first
  `budget` tokens. Budget **120 tokens** (Whisper allows 223). The Python twin
  `fit_prompt_ids` in `server-py/sidecar.py` is parity-tested.
- Browser: `src/worker/prompt.ts` builds `<|startofprev|> + ids` once per pass
  and passes it through the ASR pipeline as a private generate kwarg; a
  one-time wrapper around `model.generate` prepends it to the init tokens of
  every 30 s window, caps `max_new_tokens` so prompt + init + new ≤ 448
  decoder positions (D-021's 440 otherwise), and slices it off the output.
  Live passes use the same path.
- Server: `parseLanguageAndTask` validates `prompt` (400 above 1000 chars) for
  the HTTP and WebSocket routes; the sidecar passes the cut token ids as
  faster-whisper's `initial_prompt`. **No `hotwords`.**
**Measurements** (en_alien, 888 s gaming livestream, whole file; WER vs
`tests/audio_files/en_alien.txt`, single runs; hinted terms counted as
maxwell / voices of the void / slenderman / runescape, reference 9/1/1/1;
hint "Maxwell, Voices of the Void, Slenderman, RuneScape, UFO" = 21 tokens):
- Browser path, **onnxruntime-node CPU**, catalog WASM dtypes (turbo q4,
  base.en q8), the worker's prompt code:

  | model | hint | prompt tokens | WER | terms |
  |---|---|---|---|---|
  | large-v3-turbo | none | 0 | 0.255 | 9/1/0/1 |
  | large-v3-turbo | short list | 21 | **0.120** | 9/1/1/1 |
  | large-v3-turbo | 54-name realistic list | 120 | 0.253 | 9/1/1/1 |
  | large-v3-turbo | 54-name realistic list | 221 | 0.205 | 10/1/1/1 |
  | large-v3-turbo | 300 synthetic "TermN" | 120 | 0.266 | 9/1/1/1 |
  | base.en | none | 0 | 0.339 | 7/1/0/1 |
  | base.en | short list, no final full stop | 21 | 0.566 | 4/1/1/0 |
  | base.en | short list | 21 | 0.319 | 7/1/1/1 |
  | base.en | realistic list | 32 / 64 / 96 / 120 | 0.286 / 0.306 / 0.305 / 0.284 | 9/1/1/1 at 32 and 120 |
  | base.en | realistic list | 160 / 218 | 0.386 / **0.476** | 8/1/1/1, 6/1/1/0 |
  | base.en | 300 synthetic "TermN" | 96 / 120 / 223 | 0.530 / 0.474 / **1.050** | |

  Wall-clock per whole-file pass (turbo 141 s without a hint, 129–206 s
  with) varied with machine load more than with the hint; the extra cost is
  only the prompt's decoder positions per window.
- Without a final full stop the prompt made base.en drop punctuation and loop
  (punctuation 239 → 183) — hence the full stop.
- Short windows (live): 10 cases × 2 models (silence 1/3 s, noise 1–3 s,
  2 s speech snippets from en_aiproblem and da_festuge), with and without the
  short hint: **0 / 20** outputs contained a hint word the no-hint output
  lacked. One 2 s Danish snippet on turbo (language pinned to `da`) came out
  as English words with the English hint ("I have been at an order June
  August." vs "at den orde tøde i august.").
- Server path, faster-whisper 1.2.1 large-v3-turbo int8 CPU, short hint:
  none 0.118 (9/1/0/1), `initial_prompt` 0.085, `hotwords` 0.077, both 0.112
  — all with 9/1/1/1. Hotwords did not improve hinted-word recall over
  `initial_prompt` and combining them lost WER, so the plan's bar ("only if it
  measurably helps") is not met.
**Why 120 tokens:** with prompts near Whisper's 223-token cap the small
model degraded badly (base.en 0.339 → 0.476 realistic, 1.05 synthetic —
repetition and dropped text), while at ≤ 120 tokens a realistic list stayed
at or below the no-hint WER on both models. A junk list still hurts base.en
at 96–120 tokens (0.47–0.53); the UI says "put the most important names
first" once the text passes ~300 characters (≈ 120 tokens of names), and
short lists measured best everywhere.
**Rejected:** Transformers.js `prompt_ids` (unused in 4.2.0/4.3.2);
patching `_decode_asr` (a wrapper avoids touching the transformers patch);
the full 223-token budget (above); per-model budgets (one rule, one parity
test; revisit if a larger list is wanted on turbo).
**Consequence:** every window pays up to 121 extra decoder positions with a
long hint. WebGPU behaviour of the prompt path is unmeasured here (CPU
only) — needs the owner's browser check.

**Integration note (2026-10-30, merge with D-WS5a):** faster-whisper's
`BatchedInferencePipeline` (server file mode) re-encodes `initial_prompt`
itself and accepts only text, so the sidecar passes it the budget-cut ids
decoded back to text (re-encoding gives the same ids, checked with the real
tokenizer incl. "Århus, Ærø"); server-live passes keep the ids. Pinned by the
stubbed-sidecar test "the vocabulary hint reaches the batched pipeline as text".

## D-WS3a — Long-form file pipeline (Silero VAD chunks, language pinned once) ships as an Advanced option, off by default
**Status:** Accepted (2026-10-30) — spec 001 Goal 3
Browser file transcription cuts audio into fixed 30 s windows with 5 s
overlap (Transformers.js's chunked pipeline), decoding silence and
overlaps, and with "Auto-detect" Transformers.js 4.2 silently decodes as
English.
**Decision:** `src/worker/longform/` adds `transcribeLongForm(audio, deps,
options)`: Silero VAD speech probabilities over the whole buffer → speech
spans (line-for-line port of faster-whisper 1.2.1 `get_speech_timestamps`,
parity-checked against the Python on 20 000 random windows) → chunks
≤ 30 s cut on speech boundaries (`collect_chunks`) → one Whisper decode per
chunk → timestamps mapped back to the file's timeline. For a multilingual
model without a language, it detects once on the first speech chunk (one
decoder step on that chunk's encoder output, which the decode then reuses)
and pins it for every chunk. Defaults: VAD `threshold 0.5`,
`min_silence 2000 ms`, `speech_pad 400 ms`, `max_speech 30 s`; repetition
guard **off** (`no_repeat_ngram_size 0`, `repetition_penalty 1`);
temperature fallback (compression ratio > 2.4 / avg log-prob < −1.0 →
0.2…1.0) implemented but **off**. Inference is injected (`LongFormDeps`),
so the 90-minute case is unit-tested: VAD gets the caller's buffer, decodes
get ≤ 30 s copies. Audio with no speech returns an empty transcript without
a decode; the UI shows "No speech was found in this recording."
**Measurements** (onnxruntime-node CPU, Large v3 Turbo q4 = the browser's
WASM weights; machine shared with six other workstreams, old/new run
back-to-back per clip; WER vs `tests/audio_files/*.txt`):

| clip | old s | long-form s | old WER | long-form WER | old / lf tokens |
|---|---|---|---|---|---|
| en_aiproblem (14 s) | 3.6 | 3.3 | 0.000 | 0.000 | 47 / 47 |
| da_festuge, `da` (21 s) | 3.4 · 6.3 | 3.5 · 6.2 | 0.141 | 0.141 | 122 / 121 |
| da_festuge, auto | 6.7 | 4.7 | 0.938 (English) | 0.141 (`da` detected) | 88 / 121 |
| en_alien (15 min) | 210.2 | 115.0 | 0.255 | 0.140 | 3933 / 2258 |
| meeting (**synthetic**, 2 min 45 s) | 41.9 | 37.5 | 0.0317 | 0.0422 | 776 / 594 |

(da_festuge with `da` was measured twice, in two load conditions. The
meeting is `tests/meetingFixture.ts`: en_aiproblem and three en_alien
excerpts joined by 3–8 s pauses of −60 dBFS noise — **synthetic**. Its
long-form errors are mostly spelling of casual speech: "gonna" for
"going to", "killin", "livestreamed".)
**Rollout (spec rule: default only if ≤ old WER + 0.01 and faster on every
clip):** not met. The synthetic meeting is +0.0106 WER (just over the
tolerance) and da_festuge with an explicit language is a tie within
noise (3.5 vs 3.4 s, then 6.2 vs 6.3 s). So the old chunked path stays
the default and long-form ships as **Advanced → "Skip silence in uploaded
files"**, off by default, remembered in localStorage (`whisper-long-form`).
It is the only path whose "Auto-detect" actually detects, and roughly
halves time and WER on the long, pause-heavy en_alien.
The repetition guard and fallback variants were not measured (heavy runs
kept to one baseline + one long-form run per clip); neither is on, so no
unmeasured setting ships as a default.
**Consequence:** live mode is untouched (its RMS segmenter, D-016/D-024).
`tests/live-real-model.test.ts` now asserts whole-file WER of the chunked
path and of long-form (explicit and auto language) against the per-clip
bars. Whisper's `generate()` drops kwargs not in `forward_params`, so
`createWhisperDeps` adds `encoder_outputs` to that list on its pipeline
instance; a Transformers.js upgrade must keep the "one encoder pass per
chunk" assertion green. WebGPU timings are unmeasured.

**Integration note (2026-10-30, merge with D-WS2a):** the long-form adapter
(`worker/longform/transformers.ts`) applies the names-and-terms hint to every
chunk's decode with the same decoder-prompt prefix as the chunked path
(tokenized once per pass); before the merge the `prompt` option reached the
adapter but was ignored. Pinned by a unit test in
`tests/longform-transformers.test.ts`.

## D-WS3b — Silero VAD model: onnx-community/silero-vad from the Hub, cached like Whisper weights, never fetched offline
**Status:** Accepted (2026-10-30) — spec 001 Goal 3
**Decision:** the long-form pipeline's VAD is Silero VAD v5 as exported by
`onnx-community/silero-vad` (`onnx/model.onnx`, fp32, 2 243 022 bytes, MIT;
commit `e71cae9` on 2026-10-30). It loads through Transformers.js's
`AutoModel` as a bare ONNX graph (`model_type: "custom"`) on the same
onnxruntime as Whisper (WASM device — a 2 MB model gains nothing from a
GPU), fetched from the Hub like the weights (CORS, so COEP allows it) and
stored in the same Cache API store (D-005). Windows of 512 samples with 64
samples of context and the `[2, 1, 128]` state, as in Silero's own
wrapper. It is loaded only with "Skip silence" on, after the Whisper
pipeline is ready, on the same user-initiated load (D-020). Offline
(`navigator.onLine === false`) it loads only if already cached and never
fetches; missing or failed, the worker logs it and files take the chunked
path (`choosePipeline`). Live mode keeps its energy gate (D-016).
**Rejected:** bundling the model in `dist/` (2 MB on every install for an
off-by-default option); running it on the main thread (D-013).
**Consequence:** the first "Skip silence" run online downloads 2.2 MB.

## D-WS7a — Parakeet TDT 0.6B v3 as an English-only, desktop-only browser model via parakeet.js on the shared onnxruntime-web
**Status:** Accepted (2026-10-29, spec 001 Goal 7)
**Runtime decision (evidence).** `@huggingface/transformers` 4.2.0 (ours) and
4.3.2 (npm `latest`) were inspected from their published tarballs: both ship
only `ParakeetForCTC` (`models/parakeet/modeling_parakeet.js`, identical in
both), routed by the ASR pipeline as `model_type: 'parakeet_ctc'` through the
wav2vec2 CTC path (argmax, no timestamps, no chunking). Neither contains a
TDT/transducer decoder. Parakeet TDT therefore runs on **parakeet.js 1.4.4**
(MIT; pre-approved by the spec for exactly this case), a devDependency
bundled into the worker only (like Transformers.js, D-031).
**How it runs on one runtime.** parakeet.js pins its own onnxruntime-web
1.24.1 and its `fromUrls`/`fromHub` → `initOrt` path imports it, sets a
jsdelivr `wasmPaths` (blocked by COEP, D-023; not offline) and fetches
weights outside our cache. We never call those: `src/worker/parakeetEngine.ts`
constructs `ParakeetModel` directly from sessions created on
`onnxruntime-web/webgpu` — the module Transformers.js imports — with our own
vocab tokenizer and parakeet.js's `JsPreprocessor` (`parakeet.js/mel`). A Bun
build plugin (`scripts/ortAlias.ts`, used by the build and the dev server)
resolves parakeet.js's bare `onnxruntime-web` import to `src/worker/ortShim.ts`,
so the bundle carries one ort (`tests/workerBundle.test.ts` asserts it, and
that without the plugin the 1.24.1 copy *would* be bundled). We rely on
`onnxruntime-web` being hoisted from Transformers.js rather than pinning it
ourselves, so the two can never drift apart (workstream 1's upgrade moves both).
**Model.** `ysdede/parakeet-tdt-0.6b-v3-onnx` (NVIDIA's CC-BY-4.0 weights,
NeMo→ONNX), URLs pinned to revision `e5356bf6…` — the one measured and
whose exact file sizes the catalog shows (int8 encoder 652.2 MB + int8
decoder_joint 18.2 MB + vocab 0.1 MB = "~670 MB"; WebGPU fp16 encoder
1.24 GB). Attribution in README. The model knows 25 European languages; the
app offers it **English-only** (spec), so there is no language/task UI.
**Engine path.** Catalog entries gain `family?: "whisper" | "parakeet"`
(absent ⇒ whisper) and optional `exactBytes`; `fasterWhisperModel` becomes
optional (browser-only entries have none, so the server engine 400s them and
the picker's server list excludes them). The worker dispatches on the family.
Weights go through `src/worker/modelFiles.ts` into the `transformers-cache`
Cache Storage cache under their Hub URLs, posting the same `load-progress`
messages — badges, storage panel, delete, ModelLoadProgress unchanged
(D-005, D-033); a file is `put` only once complete. Download stays
user-initiated (D-020). Audio is cut into ≤ 30 s windows (spec: ≤ ~35 s per
encoder call) at the quietest 100 ms in the last 6 s before each limit, so no
overlap merging is needed; we don't use parakeet.js's `transcribeLongAudio`
(20–180 s windows, 90 s default). Word timestamps become sentence-sized
`TranscriptChunk`s (break after `.?!`, on ≥ 1 s pauses, or at 12 s). File
passes post one `partial` + absolute `transcribe-progress` per window; live
passes return `result` like Whisper, so live mode works unchanged.
**Devices.** WASM int8 is the baseline and the only path "auto" picks. An
explicit WebGPU choice uses the fp16 encoder on WebGPU + int8 decoder on WASM
(parakeet.js's "hybrid" split), only with `shader-f16`, falling back to WASM
through the D-025 restart. Not auto: WebGPU int8 is reported broken, the fp16
encoder is a separate 1.24 GB download, and nothing WebGPU can be measured
here. WebNN has no path (runs WASM). WASM sessions use graph optimization
"all" (D-014's "basic" cap is about Whisper's 8-bit tied embeddings; measured
fine here).
**Mobile.** Load peak ≈ 2 × 652 MB = 1.3 GB (D-027 arithmetic) — above every
budget D-029 treats as safe (deviceMemory 8 ⇒ 800 MB). Instead of a confirm,
`modelsForDevice` hides it on phones/tablets (incl. iPadOS) and
`confirmRiskyMobileLoad` refuses it.
**Measurements (CPU only; this machine, 16 cores; synthetic nothing).**
`PARAKEET_REAL=1 bun test tests/parakeet-real-model.test.ts` (onnxruntime-web
WASM under Bun, 8 threads, the worker's own engine code):

| clip | audio | file WER | file wall | live WER | live wall |
|---|---|---|---|---|---|
| en_aiproblem | 14.4 s | 0.030 | 0.9 s | 0.030 | 2.0 s |
| en_alien | 888.4 s | 0.096 | 36.4 s | 0.115 | 282 s |

`scripts/bench-parakeet.ts` (whole-file, same normalization, 12 threads,
model load excluded, one run each under the shared lock):

| engine (runtime) | en_aiproblem WER / wall | en_alien WER / wall |
|---|---|---|
| Parakeet int8 (onnxruntime-node CPU) | 0.030 / 0.3 s | 0.100 / 15.3 s |
| Parakeet int8 (onnxruntime-web WASM) | 0.030 / 0.9 s | 0.096 / 34.4 s |
| Large v3 Turbo q4 (onnxruntime-node, worker options) | 0.000 / 2.9 s | 0.255 / 183.7 s |

(D-039 recorded Turbo en_alien whole-file 0.261 / 420 s on a busier run.) On
the long, noisy clip Parakeet more than halves Turbo's WER and is ~12× faster
on the same runtime; on the short clean clip Turbo is perfect and Parakeet
misses one word ("agentic" → "agency"). No WASM number for Turbo: the
Transformers.js node build always uses onnxruntime-node.
**Consequence.** For English, Parakeet is the accurate *and* fast choice on
desktop; Whisper stays the default and the only multilingual/translate
option. Live mode is supported (faster than real time on WASM here).
**Needs the owner's browser check:** real-Chrome WASM load + transcription
speed/memory (headless Chrome run here, see WorkBreakdown); explicit WebGPU
(fp16 encoder) on the owner's GPU — success, speed vs WASM, and whether the
fallback restart behaves; Firefox/Safari WASM.
**Revisit:** if Transformers.js gains a TDT decoder, move to it and drop
parakeet.js + the alias plugin; if WebGPU measures well on real hardware,
consider letting "auto" pick it.