whisper_web/docs/Decisions.md

Decisions

Lightweight ADR log. One entry per decision that shapes the codebase. Statuses: Accepted (act on it), Proposed (default choice, revisit before building on it), Superseded (kept for history, points to replacement).

Add new entries at the bottom with the next D-### id. Never rewrite an Accepted entry to mean something else — supersede it.

---

D-001 — Bun as the only runtime and toolchain

Status: Accepted (mandated by CLAUDE.md) Bun replaces Node, vite/webpack (via bun build + HTML imports), jest/vitest (bun test), express (Bun.serve), better-sqlite3 (bun:sqlite), and dotenv. Consequence: no vite plugin ecosystem; PWA assets (service worker, manifest) are wired by hand instead of via a plugin.

D-002 — Transformers.js (@huggingface/transformers) for all Whisper inference

Status: Accepted for browser inference; the server side is superseded by D-013 (faster-whisper). Same library in the browser (WebGPU/WASM) and on the server (ONNX Runtime under Bun). Why: one model catalog, one pipeline API, no native whisper.cpp build step. Trade-off: server CPU inference is slower than whisper.cpp; acceptable — the server engine is a convenience for logged-in users, not the primary path.

D-003 — Inference runs in a dedicated Web Worker

Status: Accepted Model load + transcription block for seconds-to-minutes; running them on the main thread would freeze the UI. Communication via a small typed postMessage protocol (src/worker/protocol.ts) shared between UI and worker.

D-004 — WebGPU with automatic WASM fallback

Status: Accepted (amended by D-017 — "auto" now probes WebNN first) Detect via navigator.gpu?.requestAdapter() inside the worker; pass device: "webgpu" | "wasm" to the pipeline. The active backend is surfaced in the UI but users don't have to configure anything. Encoder-heavy Whisper benefits ~10-40× from WebGPU where available (Chromium); WASM keeps Firefox/Safari working.

D-005 — Model weights cached via the browser Cache API (Transformers.js default)

Status: Accepted Transformers.js already persists Hub downloads in the Cache API (useBrowserCache). We do not re-cache weights in the service worker — the SW precaches only the app shell. Consequence: offline = installed PWA + at least one previously loaded model; the UI must show per-model "downloaded / not downloaded" state (query caches for hub URLs) so users can prepare for offline use.

D-006 — Quantized model catalog, curated not exhaustive

Status: Proposed Expose a curated list (src/lib/models.ts): whisper-tiny(.en), base(.en), small(.en), medium(.en), large-v3-turbo, each with sensible dtype defaults (e.g. q4/q8 on WASM, fp16/q4 on WebGPU) and a size estimate shown before download. Why not free-form model ids: predictable UX, tested combinations, honest size/VRAM warnings. Revisit if users ask for arbitrary Hub ids.

D-007 — PWA: hand-rolled service worker, app-shell precache

Status: Accepted No Workbox/vite-plugin-pwa (conflicts with D-001 tooling). bun build emits hashed assets to dist/; a small build step writes the precache manifest into sw.ts. Strategy: precache app shell, cache-first for static assets, network-only for /api/*.

D-008 — Passport.js via a connect-compat adapter over Bun.serve

Status: Accepted (Passport mandated by CLAUDE.md; adapter is the how) Passport expects connect/express-style (req, res, next) and req.login(). We are not allowed to use express (D-001), so src/server/auth.ts provides a thin adapter: wrap Bun's Request in a minimal req/res shim, invoke the passport authenticate chain, translate the outcome back to a Response. Local (username/password) strategy first; other strategies plug into the same adapter later. Alternative rejected: dropping Passport for hand-rolled auth — CLAUDE.md names Passport explicitly.

D-009 — Sessions + users in bun:sqlite

Status: Accepted Single-file SQLite DB (path from env, default ./data/app.db). Tables: users (id, username, password_hash — Argon2 via Bun.password), sessions (id, user_id, expires_at). Cookie: HttpOnly, SameSite=Lax, Secure in production. No Redis/Postgres until there's a scaling reason (would be Bun.redis / Bun.sql).

D-010 — Engine abstraction: BrowserEngine and ServerEngine behind one interface

Status: Accepted src/lib/engine.ts defines Engine { load(model), transcribe(audio), onProgress }. The UI never branches on engine type beyond the toggle. Keeps the server engine (milestone M5) from leaking into UI code built in M1–M4.

D-011 — Server engine transport: streaming HTTP response first, WebSocket if needed

Status: Accepted (implemented in M5, 2026-07-10) POST /api/transcribe returns a streamed (chunked JSON-lines) response with progress and partial transcripts. Simpler than WebSockets (no connection lifecycle, works with plain fetch + ReadableStream). Switch to Bun.serve websockets only if we need bidirectional control (e.g. cancel mid-run) that abort signals can't cover. As implemented: application/x-ndjson of segment* then result | error lines (wire types + guards shared client/server via src/lib/engine.ts); the request body is raw float32 mono 16 kHz PCM — the decoded form the browser already has — so the server never decodes containers. Cancel-mid-run needed no WebSocket: fetch's AbortSignal reaches request.signal, which forwards a cancel to the sidecar (it checks between segments).

D-012 — Audio decoding on the main thread via Web Audio API

Status: Accepted decodeAudioData (handles most audio and audio-in-video containers) → mono downmix → resample to 16 kHz → transfer Float32Array to worker (zero-copy). No ffmpeg.wasm unless we hit real container-support gaps; note in src/lib/audio.ts if we do.

D-013 — Server engine runs faster-whisper (Python sidecar)

Status: Accepted (user requirement, 2026-07-10; supersedes the server half of D-002) Server-side inference (M5) uses faster-whisper (CTranslate2) instead of Transformers.js under Bun. Browser inference is untouched — faster-whisper cannot run in a browser (checked 2026-07): it is Python over native C++ CTranslate2, which has no WASM build (OpenNMT/CTranslate2#528 open since 2021, no progress) and relies on native SIMD/threading kernels. So D-002 stays Accepted for the browser, where Transformers.js remains the engine. Shape: Bun stays the only web runtime. src/server/transcribe.ts talks to a small Python sidecar (server-py/, faster-whisper behind a minimal stdio or localhost-HTTP interface) that Bun.spawn starts and supervises. The POST /api/transcribe contract (D-011: streamed JSON-lines) is unchanged — faster-whisper's segment generator maps naturally onto it. Consequence: the "one inference codebase" invariant narrows to "one browser inference codebase + one shared model catalog"; src/lib/models.ts needs a mapping from catalog ids to faster-whisper/CTranslate2 model names (Systran/faster-whisper-*); deployment gains a Python ≥3.9 dependency (server engine only — the anonymous/browser path still deploys with Bun alone, and the app must degrade gracefully when the sidecar is absent). Why: CTranslate2 CPU inference is several times faster and more memory-efficient than ONNX Runtime under Bun; the server engine becomes genuinely useful rather than a slow convenience.

D-014 — WASM sessions run at graphOptimizationLevel "basic"

Status: Accepted (bug workaround, 2026-07-10) On the WASM backend, the inference worker passes session_options: { graphOptimizationLevel: "basic" } to the Transformers.js pipeline. The onnxruntime-web build bundled with Transformers.js 4.2 (1.26.0-dev.20260416) has a broken level-2 QDQ graph transform (qdq_actions.cc TransposeDQWeightsForMatMulNBits) that fails session creation for the 8-bit Whisper decoders (_quantized/_int8/_uint8), whose tied embed_tokens weight is both gathered (embedding) and matmul'd (lm_head): "Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale". Reproduced outside the browser against decoder_model_merged_quantized.onnx of onnx-community/whisper-tiny; levels "all"/"extended" fail, "basic"/ "disabled" load fine, and onnxruntime-node 1.24.3 loads the same file at every level (regression in the ort-web dev build, not the model). Scope: WASM only — WebGPU keeps the default level. The q4 conversions use native MatMulNBits ops (no QDQ), so skipping level-2 costs the larger WASM models nothing. Revisit: when Transformers.js bumps its onnxruntime-web dependency past 1.26.0-dev.20260416, retest q8-on-WASM at the default level and drop this.

D-015 — Runtime feature flags via server env vars; first flag gates auth

Status: Accepted (2026-07-10) Feature flags are read from the server's process environment at startup (src/server/flags.ts, envFlag(): unset/empty → the flag's default; false/0/no/off any case → off; anything else → on). Runtime-only by design: one Docker image serves every configuration — flags are flipped in docker-compose.yml / docker run -e, never at build time, so the client bundle stays identical across deployments. First flag: AUTH_ENABLED (default on) gates the M4 login/register feature. Off, the server 404s every /api/auth/* route. The client is never told separately: fetchAuthStatus (the GET /api/auth/me probe useAuth already makes on mount) reads 404 as "feature off" and <AuthForms /> renders nothing. The Sign in button therefore only appears after the probe confirms the endpoints exist — a disabled deployment never flashes auth UI, and an offline/unreachable server also hides it (signing in would be impossible anyway), preserving the anonymous-first invariant (no error surfaced). Consequence: any future flag that must reach the browser needs a server-derived signal (an endpoint the client can probe, like /me here) — BUN_PUBLIC_* build-time inlining is not used for feature flags.

D-016 — Live microphone transcription via segment re-transcription (browser engine)

Status: Accepted (2026-07-10; promotes "live transcription" out of the WorkBreakdown out-of-scope list → milestone M7) Live (microphone) transcription runs on the browser engine only — same privacy story as file uploads: audio never touches the network, works for anonymous users, works offline. Whisper is not a streaming model (it consumes up-to-30 s windows), so "live" is approximated by a segment loop:

• Capture (main thread): getUserMedia → AudioContext at the device's native rate → an AudioWorklet chunker posts ~100 ms Float32 frames to the main thread, which resamples them to 16 kHz with the pure, stateful StreamingResampler (windowed-sinc low-pass; resampleLinear has no anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above 8 kHz into the speech band and measurably hurt accuracy — see tests/resampler.test.ts). The worklet module is loaded from a Blob URL built from an inline code string — no extra served file, no build/service-worker changes, and it keeps working offline. Where audioWorklet/Blob modules are unavailable (older Safari), fall back to a ScriptProcessorNode. The context is not forced to 16 kHz because Firefox cannot connect a MediaStreamAudioSourceNode across sample rates.
• Segmentation (pure, unit-tested): a simple RMS energy gate (src/lib/live.ts) splits speech into segments: a segment starts at speech onset (with a short pre-roll), commits after a run of trailing silence or at a max length (under Whisper's 30 s window), and pure-silence audio is dropped entirely — Whisper hallucinates on silence, so never transcribe it. Two accuracy refinements (2026-07-14): the gate has hysteresis (once a segment is open, frames stay "speech" down to half the onset threshold, so quiet phonemes don't fragment utterances), and a max-length commit is a soft cut — it splits at the last sub-threshold frame in the trailing window instead of mid-word at the hard cap, seeding the next segment with the remainder (a mid-word cut corrupts the boundary word in both segments). Dropped (near-silent) segments also clear any tentative text they produced, so a hallucinated tentative pass can neither linger on screen nor be adopted as the next segment's provisional text. Production tuning lives in LIVE_TUNING (src/lib/live.ts), shared by src/App.tsx and the accuracy harness (tests/live-accuracy.test.ts — tone-word oracle engine + WER, so segmentation/scheduling/resampling accuracy is regression-tested under bun test without a model). No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
• Inference (existing worker, serialized): the current segment is re-transcribed through the existing inference worker/pipeline on a rolling basis (tentative text, replaced each pass); a committed segment gets one final pass and its text freezes. At most one inference is in flight; final passes take priority over tentative refreshes, which are throttled and always use the latest audio snapshot (intermediate snapshots are skipped, so the loop degrades gracefully when inference is slower than real time).
• Protocol: UI↔worker messages gain an optional requestId echoed on partial/result/error, so the live controller can correlate responses and coexist with the file flow. The file flow keeps working without ids.

Alternatives rejected: streaming into the worker with a custom chunk-stitching decoder (large fork of Transformers.js internals for marginal latency); MediaRecorder + re-decoding compressed blobs (adds codec latency and decodeAudioData round-trips for no benefit over raw PCM taps). Consequence: live quality/latency is bounded by model speed on the user's device — the UI must keep tentative text visibly distinct from committed text. Server-engine live input (WebSocket streaming to faster-whisper) is a separate future decision; D-011's "WebSocket if needed" note is the hook.

D-017 — "Auto" device order: WebNN → WebGPU → WASM

Status: Accepted (2026-07-14; amends D-004, which only knew WebGPU → WASM) With WebNN support added (commit "WebNN support initial"), the "auto" device preference resolves in priority order WebNN → WebGPU → WASM: the worker probes navigator.ml (WebNN) first, then navigator.gpu.requestAdapter() (WebGPU), and always keeps WASM as the last resort. The probe/ordering policy is pure and unit-tested (deviceAttempts in src/worker/protocol.ts, tests/protocol.test.ts); the worker only supplies the capability probe. Pipeline-creation failures walk the same chain — an auto-detected WebNN that fails still tries WebGPU before WASM. An explicit "webnn" or "webgpu" preference deliberately falls back straight to WASM (no detour through the other accelerator — the user asked for a specific backend, so honor it or take the universal fallback). The selected backend stays user-visible: the worker's {status:"ready", device} message drives a "Running on …" badge next to the Backend select (ModelPicker) and the engine line in the footer (D-004's "surfaced in the UI" requirement).

D-018 — Cancel a file transcription by terminating the worker

Status: Accepted (2026-07-14) Users can cancel an in-flight file run (decode, model download kicked off by the drop, or the transcription pass itself) via a Cancel button next to the file name. Transformers.js exposes no abort API for a running asr() call, and on the WASM backend the pass blocks the worker's event loop — a "cancel" postMessage would sit unread until the pass finished anyway. So cancellation terminates the inference worker outright; it is the only reliable preemption and also aborts an in-flight weight download. A fileRun counter in App.tsx marks the decode that was running when the user cancelled as stale (decodeAudioData itself cannot be aborted), so it never posts audio to the fresh worker. The next run respawns the worker and reloads the pipeline — cheap, because the weights stay in the browser Cache API (D-005); the UI resets loadStarted/loadProgress so the reload starts from a clean slate. Live mode is untouched: it has its own stop/drain semantics (D-016), and a live session never coexists with a file run (mutually exclusive in the UI). Alternatives rejected: a cancel message handled between chunks (dead on WASM, where the event loop is blocked; and still seconds of latency elsewhere); AbortSignal plumbed into Transformers.js (no upstream support). Consequence: cancel throws away the loaded pipeline even when only the transcription was unwanted — an accepted cost since reloading from cache takes a few seconds. Amendment (2026-07-14, EngineToggle landing): a server-engine run cancels by aborting the /api/transcribe fetch instead (AbortController; the server cancels the sidecar pass between segments, D-011/D-013) — the inference worker and its loaded pipeline are left untouched in that path.

D-019 — Server-based live microphone transcription

Status: Accepted Why: M7 future want. Enables logged-in users on weak devices to transcribe live microphone audio using the server engine. Shape: The client opens a WebSocket connection to /api/transcribe/live. The client streams 16kHz Float32 PCM chunks (from MicCapture) over the WebSocket. The server runs LiveTranscriber and LiveSegmenter (from src/lib/live.ts, which is pure and runs perfectly in Node/Bun). For each segment/pass, the server's LiveTranscriber writes the audio to a temporary file and invokes the Python sidecar. The server sends LiveUpdate objects back to the client over the WebSocket. Consequence: The client does zero ML and minimal buffering, acting only as a mic-streamer and UI-renderer for live passes on the server.

D-020 — Model downloads are user-initiated, never a side effect of selection

Status: Accepted (2026-07-14) Selecting a model in ModelPicker (or changing the backend preference) no longer starts the weight download. Weights load exactly when they are needed or asked for:

• Lazy: a file drop or live start on the browser engine posts the (idempotent) load, as both paths already did — the download overlaps decoding / mic startup.
• Explicit: a Download button on the selected, not-yet-cached model in ModelPicker starts the load immediately — this keeps D-005's "prepare for offline use" story alive now that selection alone caches nothing.
• Server engine: server runs never touch the browser pipeline, so with the toggle on "server" nothing downloads. (Previously the eager selection load downloaded browser weights even when every transcription would run server-side.) Consequence: model/backend switches just reset the load UI state (loadStarted, progress); the next load starts from clean progress. The explicit download is a full pipeline load, so it also warms up inference and surfaces the "Running on …" backend badge. Amendment (2026-07-14): while the engine toggle is on "server" (signed in, available), ModelPicker states explicitly that nothing is downloaded to the device; the Download button stays (with a matching tooltip) for on-device / offline preparation.

D-021 — Long-file hardening: no idle timeout on the transcribe stream, heartbeats, single-pass Whisper generate

Status: Accepted (2026-07-15) Long uploads (tested: 15-min Opus, tests/audio_files/en_alien.opus) hung or died on both engines. Three independent causes, all length-dependent:

1. Bun's default 10 s idleTimeout reset /api/transcribe mid-stream. Segment gaps legitimately exceed 10 s (sidecar spawn + model load/download before the first segment; slow decode windows — 30 s+ gaps observed with large-v3-turbo on CPU). The stream then ended without a result line, so the client reported "connection was lost" after minutes of apparent hang. Fix: src/index.ts calls server.timeout(request, 0) for POST /api/transcribe (and 90 s for GET /api/transcribe/status, whose first probe waits up to 60 s for the sidecar import), and src/server/transcribe.ts enqueues a blank ndjson line every 5 s as a heartbeat — reverse proxies get liveness, dead connections surface as enqueue failures, and ServerEngine already skips blank lines.
2. Bun's default 128 MB maxRequestBodySize rejected uploads > ~35 min (raw f32 PCM is 3.84 MB/min) before the route's own 512 MB limit ever applied. Fix: maxRequestBodySize is set to the shared MAX_UPLOAD_BYTES export.
3. Browser engine: Transformers.js 4.2's Whisper generate() takes a seek-loop path when return_timestamps is set without max_new_tokens — 1–3 full decode passes per window, redundant under the pipeline's own 30 s/5 s chunking. Measured on the 15-min clip (tiny.en, onnxruntime-node): 80 generate passes for 44 windows, 1.4× slower, WER 44.7 % vs 31.5 % (duplicated boundary text), and the streamer's on_finalize fired per pass instead of per window, so TranscribeProgressTracker saturated the progress bar at ~60 % of the run and the UI looked frozen for the remainder (on WASM: many minutes). A window whose last boundary timestamp is 0 can even loop forever. Fix: the worker passes max_new_tokens: 440, which selects the classic single-pass-per-window path (440 ≈ 448 decoder positions minus prompt). Revisit: when Transformers.js changes the seek-loop condition (currently return_timestamps && !max_new_tokens, src/models/whisper/modeling_whisper.js), retest whether the explicit max_new_tokens is still needed.

D-022 — Medium models never default to fp16 in the browser

Status: Accepted (2026-07-15) Reported symptom: transcribing en_aiproblem in-browser with Whisper Medium printed just "the". The medium catalog entries defaulted to fp16 on WebNN (whole model) and WebGPU (encoder) — and D-017's "auto" order now tries WebNN first, so browsers exposing navigator.ml silently moved medium onto fp16. Two problems with those fp16 conversions (onnx-community/whisper-medium(.en_timestamped|-ONNX)):

1. The fp16 ONNX files crash onnxruntime's graph optimizer at session creation (SimplifiedLayerNormFusion fails on their InsertedPrecisionFreeCast_* nodes) — reproduced on both onnxruntime-node and onnxruntime-web 1.26.0-dev (wasm EP). EPs whose optimizer pipeline skips that fusion can load the model and hit problem 2 instead.
2. Whisper medium/large encoder activations are the classic fp16 overflow case — degenerate output with no error raised. Meanwhile q4 medium was verified correct end-to-end on the real browser runtime (onnxruntime-web wasm EP driven under Bun via the Symbol.for("onnxruntime") override, graphOptimizationLevel: "basic" as in D-014, max_new_tokens: 440 as in D-021; both the .en and multilingual conversions, auto language detect included). Fix: medium's webgpu and webnn defaults are q4 (ACCEL_Q4 in src/lib/models.ts). A backend that cannot execute q4 (MatMulNBits) throws at pipeline creation and falls down the D-017 device chain — strictly better than silently emitting garbage. Scope: medium only — that's where load-failure evidence exists. tiny/base/small keep fp16 on WebNN and small keeps fp16+q4 on WebGPU; large-v3-turbo keeps fp16+q4 on WebGPU (the widely-used Transformers.js demo configuration). If a one-word/garbage transcript ever shows up on those, suspect their fp16 encoders and extend this decision. Revisit: if the onnx-community medium conversions are re-exported with working fp16 files, or once WebNN/WebGPU runs can be exercised in CI.

D-023 — Cross-origin isolation (COOP/COEP) so WASM inference is multithreaded

Status: Accepted (2026-07-15) Reported symptom: after D-022 moved medium to q4, a 14 s clip transcribed correctly in-browser but "took ages" while native whisper needs seconds. Cause: without crossOriginIsolated there is no SharedArrayBuffer, and onnxruntime-web silently runs its (SIMD-threaded build!) WASM backend on one thread. Measured on the exact browser runtime (ort-web wasm EP, medium q4, the 14.4 s en_aiproblem clip): 52.1 s at 1 thread, 15.3 s at 4 (ort's default cap once isolation exists), 9.5 s at 12 — a 5.5× speedup. Fix: src/index.ts sends Cross-Origin-Opener-Policy: same-origin + Cross-Origin-Embedder-Policy: require-corp on every response in both serve modes (withIsolationHeaders). Bun cannot attach headers to an HTMLBundle route, so in development the shell moved to an internal /__shell route and fetch() re-serves / (and SPA fallbacks) with the headers stamped on. The inference worker raises env.backends.onnx.wasm.numThreads to min(hardwareConcurrency − 1, 12) when crossOriginIsolated is true (ort would otherwise cap at 4; without isolation the default is left alone since ort clamps to 1 anyway). Consequences: COEP require-corp blocks cross-origin subresources that are neither CORS- nor CORP-approved. Everything the app loads today survives: Hub model downloads use fetch() in CORS mode, all other assets are same-origin, and the service worker preserves the headers it cached, so the installed/offline app stays isolated too. Any future cross-origin <img>, font, or iframe must ship CORS/CORP headers or be proxied same-origin. Verified: headers present on /, worker, sw.js, /ort/* in both dev and production serve modes; dev HMR asset routes unaffected by the /__shell indirection. Real-browser thread scaling still deserves a spot check (the 5.5× number is ort-web under Bun, same runtime but not a browser tab). Amendment (2026-07-15, same day): threads are per-device now. With SharedArrayBuffer newly available, onnxruntime-web's own default thread count becomes ~4 — and the JSEP accelerator EPs (WebGPU, WebNN) are not thread-safe: real-browser WebGPU runs failed at OrtRun with "WebGPU validation failed. Buffer with '' label is invalid". The worker now sets numThreads right before each device attempt (configureWasmThreads): min(cores − 1, 12) for the pure wasm device, explicitly 1 for webgpu/webnn (relying on the default is what broke). onnxruntime-web fixes its thread pool at first-session init, so the first attempt in a worker decides; a wasm fallback after a failed accelerator attempt may run single-threaded until the next worker respawn — correct over fast.

D-024 — Live mode batches queued final passes into merged passes

Status: Accepted (2026-07-17) Reported symptom: stopping a live session got stuck on "Finishing up…" while the machine sat at full CPU — indistinguishable from a hang, and the saturated cores (D-023 uses most of them) made the whole browser feel locked up. Cause — a scheduling backlog, not a deadlock. Whisper pads every inference pass to a full 30 s encoder window, so a pass costs nearly the same whether it carries 1 s or 29 s of audio (measured on the browser runtime, ort-web WASM EP medium q4: ~4 s per pass at 12 threads, ~9–12 s at 4 threads, for 1–9 s segments alike). Segments therefore commit faster than their final passes complete whenever the model is slower than speech (medium/large on WASM), the committed-segment queue grows all session, and stop() used to drain it one pass per segment — a 60 s session on 4 threads measured a 52 s drain; longer sessions scale it into minutes. All the while the UI showed only a static "Finishing up…". Fix (two parts):

1. LiveTranscriber.pump() batches consecutive queued segments into one merged final pass, capped at the segmenter's maxSegmentSeconds so a merged pass stays a single Whisper window. Segment audio is concatenated (segments end on trailing silence or a soft cut, so splices land in natural gaps); the result's chunk timestamps are mapped piecewise back to absolute session time via each segment's own start offset. The backlog — and the stop drain — is now bounded by ~one pass per 29 s of speech instead of one pass per segment (the same 60 s/4-thread scenario drains in ~2 passes).
2. App.tsx surfaces LiveUpdate.pendingFinalPasses while stopping: "Finishing up… transcribing N remaining segments" counts down instead of freezing, on both the browser and the server-live (D-019) paths. Alternatives rejected: skipping final passes for segments that already had tentative text (during a backlog new segments get no tentative pass, so skipping loses their words entirely); a "stop now" escape hatch (same data loss, silently); merging across the 30 s window (impossible — Whisper's context is one window). Consequence: a merged pass attributes its whole text to the batch's first segment (per-segment text attribution is meaningless after joining anyway) and inter-segment silence is not re-inserted, so chunk timestamps inside a merged pass are exact per segment region but a chunk spanning a splice maps to its start's region. Covered by tests/live.test.ts (batching + cap) and the WER gates (tests/live-accuracy.test.ts, tests/live-real-model.test.ts — the real-model test's fast feed makes real inference lag, so merging is exercised there with real weights).

D-025 — A degraded-thread WASM fallback restarts the worker; WebGPU is skipped when fp16 needs an absent shader-f16

Status: Accepted (2026-07-17) Reported symptom: Firefox performance far below Chrome, live transcription worst. Chrome resolves D-017's auto chain to an accelerator; Firefox has no WebNN and its WebGPU (where exposed at all) fails onnxruntime-web's Chromium-built JSEP EP — notably it lacks the shader-f16 feature, which the catalog's fp16-encoder WebGPU defaults (small, large-v3-turbo) need. So Firefox lands on WASM — but how it lands there was the real bug: the failed WebGPU attempt initializes ort's WASM runtime at the accelerator-safe numThreads: 1 (D-023 amendment), the thread pool is fixed at first init, and the fallback session then runs single-threaded for the worker's whole life (measured 5.5×: 52 s vs 9.5 s on a 14 s clip). Live never respawns the worker, and every live pass pads to a full 30 s window (D-024), so passes took ~40–60 s: tentative text froze and the stop drain ballooned — D-023 amendment's "correct over fast" edge case is Firefox's default path. Fix (three parts):

1. Restart instead of a degraded fallback. When createPipeline reaches the wasm attempt after any accelerator attempt ran and multithreading is actually available (desiredWasmThreads() > 1), the worker posts {type:"load-restart", modelId, preference, device:"wasm"} and aborts the load (sentinel error, never surfaced). The UI terminates the worker, respawns it, re-issues the load with the explicit device, and remembers modelId::preference → device for the session so later loads skip the doomed attempt (session-only: a browser update may fix the backend; an explicit user backend choice is a different key, so it retries once). To make the respawn safe, transcribe posts now wait on a load gate in App.tsx that resolves on ready (rejects on load error/cancel/crash): no pass is ever queued inside a worker a restart is about to terminate — its transferred audio buffer would die with it. The worker serialized passes on the load anyway, so nothing starts later than before.
2. shader-f16 probe. The worker's WebGPU probe also records adapter.features.has("shader-f16"); when the dtype that a WebGPU attempt would use requires fp16 (dtypeRequiresF16, pure + unit-tested) and the feature is absent, WebGPU is skipped outright — Firefox then loads straight into multithreaded WASM with no wasted attempt.
3. Live-on-WASM hint. With live active on wasm and a heavy model (encoder ≥ 300 M params: medium, large-v3-turbo), the UI says the model runs slower than real time and suggests a smaller one — WASM lag there is inherent (D-024's fixed window cost), not a bug, so tell the user instead of looking frozen. Alternatives rejected: re-posting in-flight passes after respawn (their buffers are transferred — gone with the worker); a scout worker pre-flighting the accelerator (doubles accelerated-path load time to save the failure path); ort-web's proxy-worker mode to escape the pinned pool (undocumented interplay with JSEP). Consequence: the accelerator-failure path costs one worker respawn + a pipeline reload from cache (a few seconds, once per model+preference per session). A genuine both-attempts-fail load still reports its error normally. deviceAttempts is unchanged; the new policy pieces (dtypeRequiresF16, the load-restart message guard) are unit-tested in tests/protocol.test.ts. Real-Firefox verification (badge shows WASM with full threads, live keeps up on small models) still needs a manual run — same no-browser dev-env caveat as M2/M3. Amendment (2026-07-17, same day — field report: tiny fixed, medium still slow in Firefox): three observability/robustness additions to tell "slow is the CPU-vs-GPU physics" apart from "something is still degraded":
4. The probe skips software (fallback) WebGPU adapters (isFallbackAdapter, both spec locations) — SwiftShader/llvmpipe/WARP would succeed at session creation and then run slower than multithreaded WASM, and the fallback chain only reacts to failures, not slowness. onnxruntime-web itself uses whatever adapter it is handed (verified in 1.26.0-dev: no fallback-adapter or Firefox guard).
5. The ready message now carries threads — read back from env.backends.onnx.wasm.numThreads after session creation, so ort's own clamp-to-1 (missing isolation) is visible. The footer shows it: "WASM (on-device, 11 threads)".
6. The worker logs the device chain ([whisper] … console lines): each skipped backend (software adapter, missing shader-f16), each failed attempt with its error, and the final device+threads — previously fallback errors were swallowed silently, making field reports unactionable. Expected healthy Firefox numbers for medium q4 on WASM (from the D-023/D-024 measurements): ~4 s per 30 s window at 12 threads, ~9–12 s at 4 — i.e. a 14 s clip in ~10 s. Chromium's GPU path is a further ~3–5× faster; that remaining gap is inherent until Firefox's WebGPU works with ort-web's JSEP EP. If the badge shows 1 thread or the console shows a surviving software adapter, it is a bug in this decision's scope instead.

D-026 — "Auto" never picks WebGPU on Firefox

Status: Accepted (2026-07-17) Field-verified on Firefox (D-025's observability build): explicit Backend = WASM runs medium with a full thread pool and healthy speed, while explicit Backend = WebGPU (device=webgpu wasmThreads=1, correct per D-023) completes but runs far slower than the same machine's multithreaded WASM. That is the worst case for D-017's failure-driven fallback chain: ort-web's WebGPU EP (Chromium-developed; no Gecko handling in 1.26.0-dev, verified) creates the session successfully on Firefox, so nothing ever falls back — "auto" happily selects the slowest available backend. Fix: the worker's capability probe skips the WebGPU probe entirely when the preference is "auto" and the UA is Firefox (/Firefox\// on navigator.userAgent), with a [whisper] console line saying why. Firefox auto therefore resolves straight to multithreaded WASM (no accelerator attempt → no D-025 restart either). An explicit WebGPU selection is still honored — that is the deliberate retest hatch: when a future Transformers.js/ort-web bump gains real Gecko WebGPU support, verify via the explicit path, then delete the UA check. Why UA sniffing: there is no feature to detect — the sessions create and run; only throughput distinguishes them, and a benchmark probe would cost more than it saves. The check is one line, logged, scoped to "auto", and targets the one engine with field evidence. Revisit: on any onnxruntime-web bump — retest explicit WebGPU on Firefox (file tests/live.md note) and drop the check when it reaches parity.

D-027 — Mobile OOM crash on Medium+ loads: sequential session loading (patched) + a warn/confirm gate

Status: Accepted (2026-07-28) Reported symptom: downloading Whisper Medium (English) on a phone crashed the whole browser — on Android, badly enough to briefly kick the OS back to the home screen (consistent with the low-memory killer reaping background processes, including the launcher, once the tab's memory spike triggered system-wide pressure). Cause. Whisper Medium's smallest available dtype on any device is q4 (D-022) — still ~460 MB combined encoder+decoder weights. Two independent, compounding sources of transient memory at load time:

1. @huggingface/transformers buffers each ONNX weight file fully into one JS-heap Uint8Array (readResponse/loadResourceFile, src/utils/hub.js/hub/utils.js) before it is cached or handed to onnxruntime — unavoidable either way (with or without a progress_callback, and regardless of whether a URL string or a buffer is passed to InferenceSession.create): onnxruntime-web's WASM backend has no streaming ONNX parser, so the whole file must exist as one contiguous buffer to construct a session.
2. constructSessions (src/models/session.js) loaded the encoder and decoder sessions concurrently via Promise.all, so both full-size JS buffers were resident at once — needlessly roughly doubling the already-large per-file peak (on top of onnxruntime-web's own WASM-heap _malloc + HEAPU8.set copy inside copyFromExternalBuffer, made once per file at session-creation time, which is itself already serialized across sessions in web environments). Net: peak transient memory for Medium approached ~2-3x the ~460 MB combined weight size — comfortably inside WASM32's 4 GiB address space, but well past the much tighter per-tab memory budgets mobile Safari/Chrome enforce, which respond to going over budget with an outright, uncatchable tab kill (no JS error event fires — worker.onerror in App.tsx never sees it). There was no mobile detection anywhere in the app: Medium downloaded/loaded identically on a phone and a workstation. Fix (two parts):
3. Sequential session loading (real fix, no UX cost). constructSessions is patched (bun patch, patches/@huggingface%[email protected], tracked via patchedDependencies in package.json so it survives bun install) to load sub-model sessions one at a time instead of via Promise.all. This lets the previous file's JS buffer become garbage before the next file is fetched, roughly halving the load-time peak for any multi-file model (Whisper's encoder+decoder) — a strict improvement with no downside, so it applies unconditionally, not just on mobile. Investigated and rejected as not actionable: skipping the progress_callback, env.customCache, and passing a URL string instead of a buffer to onnxruntime-web — all three still end up materializing the whole file, per the analysis above. Fixing that fully would require forking onnxruntime-web itself (out of scope).
4. Warn/confirm gate on mobile (src/lib/mobileMemoryGuard.ts, estimatePeakLoadBytes in src/lib/models.ts). Even halved, Medium's peak (~555 MB estimated: 2 × max(encoderBytes, decoderBytes), reflecting the now-sequential loading) can still exceed a constrained phone's tab budget. App.tsx's confirmRiskyMobileLoad gates all three ways a browser-engine load can start — the explicit Download button, and the lazy auto-load on file-drop / live-start (D-020) — behind shouldWarnBeforeLoad: mobile UA detection (Android/iPhone/iPod, plus iPadOS's desktop-masquerading Safari UA via touch-point count, same UA-sniffing rationale as D-026 — no memory-pressure API exists to probe instead) combined with navigator.deviceMemory when Chromium reports it, falling back to a flat ~400 MB threshold where it doesn't (all of iOS/Safari). A native window.confirm() states the estimated memory need and the crash risk in plain language; declining aborts the load with no state changes, accepting remembers the modelId+device combination for the rest of the session so the same load doesn't ask twice. Desktop/laptop browsers are never gated — no crash evidence exists there. Alternatives rejected: hard-blocking Medium+ on mobile with no override (some phones have enough RAM to be fine; the confirm lets the user judge their own device); forking onnxruntime-web for a streaming session-creation path (large undertaking for a runtime we don't own upstream). Consequence: every model load is now faster-to-safe (sequential sessions cost a little wall-clock time — files download one after another instead of in parallel — in exchange for materially lower peak memory). Mobile users choosing Medium/Large now see one extra confirmation step; Tiny/Base/Small are unaffected (their estimated peaks stay under the warning thresholds). Revisit: if field reports show the sequential-load patch alone resolves mobile crashes in practice, consider loosening the confirm thresholds; if onnxruntime-web ever gains a streaming/incremental session-creation API, drop the patch and re-measure before touching the thresholds again. Amendment (2026-07-29, D-028): "a little wall-clock time" undersold the cost — see D-028. The sequential patch no longer applies unconditionally; options.sequentialSessionLoad (set by App.tsx, true only when shouldWarnBeforeLoad is true for the load) now gates it.

D-028 — Sequential session loading gated to actual mobile-crash risk, not unconditional

Status: Accepted (2026-07-29) Reported symptom: encoder_model_q4.onnx (Whisper Medium) downloaded far slower after D-027 landed than before it — reproduced on desktop too, ruling out phone CPU/network as the cause. Measured in Chrome DevTools: ~1.6 MB/s sustained on a connection that otherwise performs fine, downloading from Hugging Face's Xet-backed CDN (us.aws.cdn.hf.co/xet-bridge-us/...). Cause. D-027's constructSessions patch made the encoder and decoder downloads always sequential — one HTTP request in flight at a time — to keep both weight buffers from being resident together. On a long-RTT path (this CDN is US-hosted), a single HTTP/2 stream typically can't fill the available bandwidth; two concurrent requests (the pre-D-027 Promise.all) filled more of it. Serializing the downloads dropped every load back to one flow's throughput ceiling — not "a little" wall-clock cost as D-027 claimed, but multiple times slower on exactly this kind of connection, and paid by every user regardless of device. Fix. The sequential/parallel choice is now conditional, not baked into the patch: constructSessions branches on a new options.sequentialSessionLoad flag (patches/@huggingface%[email protected]) — parallel Promise.all (the original, fast behavior) when falsy/absent, the D-027 for-loop when true. App.tsx computes this per load via isMobileLoadRisk (extracted from confirmRiskyMobileLoad's existing hint/estimatePeakLoadBytes logic — the same shouldWarnBeforeLoad check the crash-warning dialog uses) and threads it through a new sequentialSessionLoad field on the worker's LoadMessage (protocol.ts) into the pipeline() call (inference.worker.ts). Crucially this is evaluated independently of the confirm dialog's answer — even a load the user clicked "Continue anyway" on still loads sequentially, since the memory risk that motivated D-027 doesn't go away just because the user accepted it. Consequence: desktop/laptop loads (and any mobile load under the size/deviceMemory risk threshold) get back the full concurrent-download speed that predates D-027; only loads that would already trigger the mobile crash warning pay the sequential-loading wall-clock cost, which is exactly where D-027's OOM fix is actually needed. Revisit: if Hugging Face's Xet CDN behavior changes, or if a future model in the catalog is large enough to risk mobile OOM at a size where the confirm-dialog threshold doesn't yet trigger, re-check that the two thresholds (shouldWarnBeforeLoad's and this gate's) are meant to stay identical.

D-029 — Mobile crash warning was silently disabled on every phone but the lowest-end

Status: Accepted (2026-07-29) Reported symptom: the D-027 mobile crash window.confirm() never appeared for Whisper Medium (or any model), even on devices it's explicitly meant to protect. Cause. shouldWarnBeforeLoad's deviceMemory-scaled branch used SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.5. Checked against the actual catalog (estimatePeakLoadBytes) for every navigator.deviceMemory value Chromium can report:

deviceMemory	"safe" budget (old 0.5)	Medium (556 MB)	Large v3 Turbo (762 MB)
1	500 MB	warns	warns
2	1 GB	silent	silent
4	2 GB	silent	silent
8	4 GB	silent	silent

deviceMemory of 2 or more is ordinary today, not just flagship-tier — so in practice the warning only ever fired on the oldest/cheapest Android phones and on iOS (which lacks the API entirely and always used the flat 400 MB fallback, D-027). The exact mid/low-range phones the original crash report was about — the whole reason D-027 exists — cleared the threshold and got no warning at all. Fix: SAFE_TAB_FRACTION_OF_DEVICE_MEMORY lowered to 0.1 (src/lib/mobileMemoryGuard.ts). Re-checking the same table: deviceMemory 2 and 4 now warn for Medium+ (200 MB / 400 MB budgets, both under 556 MB); deviceMemory 6-8 (600-800 MB budgets) still don't, treating genuine high-end devices as having real headroom. tests/mobileMemoryGuard.test.ts gained a regression case pinning deviceMemory 2/4 to true for Medium's peak — the exact gap this decision closes. Alternatives rejected: dropping the deviceMemory scaling entirely in favor of always using the flat 400 MB fallback (simpler, but throws away the one signal that lets genuinely high-RAM devices skip an unnecessary confirm); a hard absolute cap on top of the fraction (more robust long-term, but two knobs to calibrate against no real data beats one — revisit if a future model's peak lands awkwardly relative to the 0.1 fraction). Revisit: this is still a guess, same as D-027's original 0.5 was — there is no real crash-telemetry feedback loop. If field reports show 0.1 is still too generous (or too aggressive) for some device class, adjust the constant rather than the shape of the heuristic.

D-030 — Error taxonomy: pattern-matched classification, not structured error types

Status: Accepted (2026-07-29) Every failure path (decode, model load, transcription, a crashed worker, a live session) used to end at setError(rawMessage), rendered as one plain string — no distinction between "this file won't decode" (try another file), "the tab ran out of memory" (try a smaller model), a lost WebGPU device (switch backend), or a dropped network connection mid-download (retry). The WorkBreakdown M6 item asked for exactly those four categories. Shape: src/lib/errors.ts exports a pure classifyError(raw, context): context.stage ("decode" | "load" | "transcribe" | "worker-crash") plus context.offline (navigator.onLine) disambiguate cases the message text alone can't, then substring pattern-matching on the message sorts the rest into unsupported-file / out-of-memory / device-lost / network / generic, each with a user-facing message, an optional actionable suggestion, and retryable. Why pattern-matching, not typed errors from each layer: the failures cross Web Audio, Transformers.js, onnxruntime-web (WASM and the WebGPU/WebNN JSEP EPs), and fetch — none of which expose a shared error hierarchy the app controls, so there is nothing to instanceof against; the only thing every layer reliably produces is an Error (or DOMException) with a message string. A worker crash (Worker.onerror) with an empty message is itself classified as out-of-memory: D-027 found that the browser's silent OOM tab-kill fires no JS error event with details at all, so an empty message is the OOM signal in that one context — the classifier only applies that inference when context.stage === "worker-crash", not to an empty message anywhere else. UI wiring (src/App.tsx): the error state changed from a string to { classified: ClassifiedError; retryKind?: "load" | "file" }. retryKind is resolved to an actual retry action at render time (onClick={() => startLoad(modelId, devicePreference)} / replaying lastFileRef.current through handleFile), reading current component state directly — not computed inside the worker's message/error event listeners, which are registered once per worker lifetime (ensureWorker) and would otherwise close over whichever render's modelId/devicePreference happened to be current at registration, going stale on every later model/device switch. Consequence: only network and device-lost are retryable (an unsupported file or an OOM won't succeed by retrying the identical action); the error box shows a Retry button only for those, alongside the suggestion text. Live-mode errors (mic permission, WS disconnects) still classify through the same function for a consistent error box, but never carry retryKind — the existing Start/Stop controls are already the retry path there. Alternatives rejected: wrapping each subsystem's calls in typed error classes at the boundary (real fix, but Transformers.js/onnxruntime-web throw plain Errors from deep internals — retrofitting types there means forking dependencies, out of scope); classifying in the worker instead of the UI (the worker already forwards a string message over postMessage — classifying there would just move the same pattern-matching one hop over for no benefit, and the UI is where context.offline/retry actions live). Revisit: if a future onnxruntime-web/Transformers.js version starts surfacing richer structured errors (e.g. a GPUDeviceLostInfo object instead of a stringified message), prefer inspecting that directly over adding more substring patterns.

D-031 — Docker image hardening: exclude local venvs, move Transformers.js out of the runtime image

Status: Accepted (2026-07-29) WorkBreakdown's M6 "production build + bun start deployment check" item — a docker build . from this checkout, not a fresh clone, exposed two real bugs the documented image sizes (WorkBreakdown M5 notes: "244 MB" / "405 MB") had already gone stale against. Bug 1 — local Python venvs leak into every image. server-py/venv and server-py/.venv (created locally to test the sidecar against a real faster-whisper install) are invisible to git status — python -m venv drops its own nested .gitignore (*) inside the venv directory — but Docker does not read .gitignore, and .dockerignore had no matching pattern. COPY server-py ./server-py therefore copied both venvs (858 MB combined) into the image; measured effect on the Bun-only image: 999 MB instead of the intended size, with a completely unused, machine-specific Python interpreter baked in at /app/server-py/venv and /app/server-py/.venv. Fix: .dockerignore gained **/venv/ and **/.venv/. Verified: rebuilding after the fix, find /app/server-py inside the image no longer lists either directory. Bug 2 — @huggingface/transformers (and its onnxruntime-node + onnxruntime-web dependencies, 341 MB combined) had no reason to be in the production runtime image. Grepped: the only file in src/ that imports @huggingface/transformers as a value (not a type) is src/worker/inference.worker.ts, and that file is bundled once, at build time, into the static dist/inference.worker.js the production server just serves as a file (confirmed: src/index.ts's Bun.build-based on-the-fly bundling path — needed because Bun doesn't follow new Worker(...) graphs, per M1's Architecture notes — only runs in development mode; production never calls it). No server-side module (src/server/*, src/index.ts, src/lib/models.ts's only import from the package is import type, erased at transpile time) touches the package at runtime. Yet it was a dependencies entry, so the Dockerfile's runtime stage (bun install --frozen-lockfile --production) installed it and its full dependency tree anyway — dead weight on every deploy. Fix: @huggingface/transformers moved to devDependencies in package.json. The build stage's bun install --frozen-lockfile (no --production) still installs devDependencies, so scripts/build.ts's Bun.build call (which needs the package on disk to bundle the worker) is unaffected; the patch in patches/@huggingface%[email protected] still applies there the same way. Verified: production node_modules in the built image dropped from 409 MB to 7.9 MB (react/react-dom/passport only); the running container still served /inference.worker.js correctly (534,714 bytes, byte-identical to the pre-change build) and /api/transcribe/status correctly reported the expected faster-whisper/python3 unavailability message — nothing at runtime needed the package. Net effect, measured against a real docker build ./--target with-server-engine in this checkout:

Image	Before both fixes	After
Bun-only (runtime)	2.18 GB (venv leak)	~421 MB
with-server-engine	not separately measured (inherits the leak)	~7.1 GB — see below
A third, non-bug finding while re-measuring: the with-server-engine
image is legitimately several GB, not the "405 MB" previously written down
— its last RUN layer pre-downloads both server-selectable models
(large-v3-turbo, medium.en, matching the EngineToggle's hardcoded
allow-list in App.tsx) into /opt/faster-whisper/models at build time,
which alone measured 3.15 GB; the venv/faster-whisper/ctranslate2 install
layer is a separate 451 MB. This is intentional (comment already in the
Dockerfile: baking the weights into the image, outside /app/data, means
no first-request cold-download and no dependence on the data volume for
those two models) — not something to shrink — but the number that made it
into WorkBreakdown's M5 notes and README's env-var table was never updated
to match, and the README's Persistence section wrongly implied these
weights live in the /app/data volume (FASTER_WHISPER_DOWNLOAD_ROOT's
actual default in that image is /opt/faster-whisper/models, not
/app/data/faster-whisper-models). Fix: README's env-var table,
Persistence section, and a new "Image sizes" subsection now state the
verified numbers and the reason for each; WorkBreakdown's M5 notes updated
to match.
Consequence: none of this is CI-visible — GitLab CI (.gitlab-ci.yml)
builds from a clean checkout that never has server-py/venv on disk, so
Bug 1 only bit a local docker build . run from a working directory that
had one (as this checkout did); the .dockerignore fix is still correct
defense-in-depth regardless. Bug 2 was real everywhere, CI included — every
pushed Bun-only image has been ~340 MB heavier than necessary since
@huggingface/transformers became a production dependency.
Revisit: if the with-server-engine catalog ever grows beyond the two
baked-in models, either accept runtime downloads for the extra ones (point
FASTER_WHISPER_DOWNLOAD_ROOT at /app/data for those, per the README's
override note) or extend the Dockerfile's pre-download RUN line —
re-measure and re-document the image size either way.

D-032 — Server-live WebSocket brought in line with the HTTP engine's DI/validation/test conventions

Status: Accepted (2026-07-29) D-019's server-engine live transcription (GET /api/transcribe/live → src/server/live.ts) had landed and was wired end-to-end in App.tsx, but three things about it lagged the standard the rest of M5 set with src/server/transcribe.ts:

1. Not injectable. live.ts imported the real sidecar singleton directly and called it from inside the LiveTranscriber's transcribe closure, unlike createTranscribeHandlers(deps), whose whole point (per its own comment) is that Sidecar is "injectable — tests use a fake". There was no way to unit-test the WebSocket bridge without a real Python sidecar.
2. Unvalidated query params. POST /api/transcribe's parseParams validates language against LANGUAGE_RE and task against "transcribe" | "translate" before it ever reaches the sidecar protocol. The WS upgrade route in src/index.ts skipped both checks — url.searchParams.get("task") || undefined forwarded an arbitrary string straight into LiveSessionData.task and on to the Python sidecar's JSON-lines protocol. Also skipped: the upfront sidecar.status() check the HTTP route makes before doing any work (D-013's "degrade gracefully" — opening a socket that can only report unavailability after the first segment reaches the sidecar is a worse failure mode than a prompt 503).
3. Zero test coverage. No tests/server-live.* file existed; every other stateful server module in the app (transcribe.ts, auth.ts, db.ts) has one. Fix:
4. src/server/transcribe.ts exports parseLanguageAndTask(url) (extracted from parseParams, same validation, same 400-text-on-string-return contract); src/index.ts's GET /api/transcribe/live case now calls it and 400s on an invalid value, then calls sidecar.status() and 503s with the reason before upgrading — matching POST /api/transcribe exactly.
5. src/server/live.ts exports createLiveWebSocket(deps: { sidecar: Sidecar }); liveWebSocket (the instance src/index.ts wires into Bun.serve) is just createLiveWebSocket({ sidecar }) against the real singleton. LiveSessionData.user/task are now typed SessionUser/ WhisperTask instead of any/string | undefined (task always has a value now that the WS route defaults it like the HTTP route does), which let the pre-existing options.websocket = liveWebSocket as any cast in src/index.ts drop to a plain assignment — it was papering over the untyped user/task fields, not a real incompatibility with Bun.serve's generic websocket option.
6. tests/server-live.test.ts exercises createLiveWebSocket against a fake Sidecar (same shape as tests/transcribe.test.ts's): a committed segment writes exactly its own PCM bytes to the temp file and cleans up after, model/language/task pass through, a sidecar error surfaces via the update message's error field instead of crashing the session, a {"type":"stop"} frame drains and closes with a result message, and a malformed text frame is ignored. tests/transcribe.test.ts gained direct cases for parseLanguageAndTask. Incidental correctness fix found while rewriting the transcribe closure: the original code wrote new Uint8Array(audio.buffer as ArrayBuffer) to the temp file — the whole underlying ArrayBuffer, ignoring audio.byteOffset/ audio.byteLength. Every real caller today happens to pass a fresh Float32Array from lib/live.ts's concat() helper (always offset 0, full length), so this never manifested, but LiveTranscribeFn's type signature ((audio: Float32Array) => ...) makes no such promise to callers. Restored the original's audio.byteOffset, audio.byteLength slice (which the pre-refactor code already had — it would have been an actual regression to drop it during the DI rewrite). Consequence: the server-live path now has the same testability and input-validation posture as the HTTP path; no behavior change for a valid client, since App.tsx already only ever sends catalog model ids and "transcribe" | "translate" task values. Alternatives rejected: leaving live.ts untested and testing only through a real spawned sidecar (matches createSidecar's own integration tests, but that suite already covers the sidecar process itself — the gap was specifically the WebSocket/LiveTranscriber glue, which needs no Python process to exercise).

D-033 — "Downloading model…" said that even when the weights came straight from cache

Status: Accepted (2026-07-30) Reported symptom: with a model's weights already fully cached in the browser (Cache API, D-005), starting a load still showed ModelLoadProgress's heading, "Downloading model <id>", and its "this happens once" hint — as if bytes were coming over the network, when nothing was. Cause. node_modules/@huggingface/transformers/src/utils/hub.js's loadResourceFile computes a local cacheHit boolean but its progress_callback dispatch always fires { status: 'download', ... } unconditionally — cacheHit only reaches the callback on a Firefox-specific branch that skips progress entirely. So inference.worker.ts's forwardProgress() has no reliable "this came from cache" signal to forward; it just relays initiate/download/progress/done verbatim, and ModelLoadProgress's heading was hardcoded to the download copy regardless. Fix. Rather than patch the vendored library for a cosmetic flag (the existing bun patch at patches/@huggingface%[email protected] is reserved for the two structural D-027/D-028 session-loading changes — adding an unrelated third patch just for this is disproportionate and fragile against future upstream bumps), the app answers the question itself, before the load ever starts, from information it already tracks: App.tsx already queries lib/modelCache.ts's getDownloadedModels into a downloaded Set<string> for ModelPicker's "available offline" badges. A new loadedFromCache state is set from downloaded.has(nextModelId) at the top of startLoad() — the single function every load path (file-drop lazy load, live-start lazy load, the explicit Download button, and the error box's Retry) already funnels through — and reset to false everywhere loadStarted /loadProgress are reset (model switch, device-preference switch, cancel, the server-engine forced model switch), so a stale flag never survives into an unrelated load. It is threaded as a new required prop into ModelLoadProgress, whose heading/hint now read "Loading model from cache…" / "This model is already downloaded — just loading it into memory." when true, unchanged otherwise. No worker protocol change: the decision is made entirely from state the UI has before it ever posts the load message. Partial-cache handling. matchCachedModels (the pure matcher backing getDownloadedModels) flags a model "downloaded" on any single cached .onnx file, not all of them — a genuinely partial download (e.g. encoder cached, decoder not) would misreport as fully cached here. This is the same imprecision ModelPicker's existing "available offline" badge already lives with; per this fix's scope, no new finer-grained per-file cache probing was added — modelCache.ts stays the single source of truth for "is this downloaded," reused as-is rather than duplicated at a different granularity. Consequence: the loading UI now matches reality for the common already-cached case (reopening the app, switching back to a previously used model); first-time downloads are unaffected. ModelPicker's own "Download" button never needed a matching change — it is only rendered for !isDownloaded models, so it can never show "Downloading…" for an already-cached one. Revisit: if modelCache.ts ever gains a per-file/fully-cached check (e.g. for the storage-management panel's exact byte accounting), reuse it here too so a genuinely partial cache falls back to the "Downloading" copy instead of "Loading from cache."

D-034 — Browser-engine live status no longer claims "Listening" before the model is ready

Status: Accepted (2026-07-30) Symptom: On the browser engine (not useServer), clicking "Start live transcription" showed the full "recording" UI — the pulsing red live-dot and the "Listening — like everything else, your audio stays on this device." hint (src/components/LiveControls.tsx) — as soon as microphone permission was granted, even when the model still had to load (first use of a model, or a model/device switch mid-session that reset the pipeline). The user saw what looked like an active live session while nothing was actually being transcribed yet. Cause: in startLive()'s browser-engine branch (src/App.tsx), the await startMicCapture(...) call was immediately followed by setLiveStatus("listening") with no check on whether the pipeline had actually reported ready. The model load itself was already correctly kicked off just above (idempotent startLoad/load postMessage, D-020) and transcribeLiveSegment() already correctly awaits browserModelReady() (the promise on loadGate.current, resolved by the worker's {status: "ready"} message or rejected on a load-stage error/cancel, D-025) before posting any segment — so no segments were ever lost or misrouted to a wrong pipeline. This was a pure UI/status bug, not a data-correctness one. The server-engine (useServer) branch never loads a browser pipeline, so it was never affected and is untouched by this fix. Fix:

1. LiveStatus (src/components/LiveControls.tsx) gains a "loading-model" member between "starting" and "listening", labeled "Loading model…". It disables the Start/Stop button exactly like "starting"/"stopping" do, and — because only status === "listening" sets the component's listening boolean — it renders neither the live-dot nor the "Listening — …" hint (falls into the same "not listening yet" branch as "idle"/"starting"/"stopping").
2. In startLive(), once startMicCapture resolves and liveRef.current is set: if loadProgress.ready is already true (pipeline loaded earlier this session, no model/device switch pending), status goes straight to "listening" as before. Otherwise status becomes "loading-model" and the code awaits browserModelReady():
   • On resolve, it re-checks liveRef.current === session (the user could have hit Stop, or started a fresh session, while the wait was pending) before flipping to "listening" — never resurrects a torn-down session.
   • On reject (load failed), it stops the mic and clears liveRef.current (same re-check) and returns status to "idle", without calling setError() itself — the worker's own "error" message handler already calls setError() for stage === "load" failures (src/App.tsx ~line 380), so a second call here would have shown two error messages for one failure.
3. stopLive() needed no change: its existing guard (if (!session || liveStatus !== "listening") return;) already makes it a no-op during "loading-model", exactly mirroring how it already no-ops during "starting" — consistent with the Stop button being disabled for both statuses per (1). Consequence: live-dot/hint/button now truthfully reflect whether audio is actually reaching a ready pipeline. The only added latency is the same wait transcribeLiveSegment was already imposing on the first segment pass — nothing downstream changed. tests/live.md's permission-flow step was updated to describe the new "Loading model…" interim state instead of the old immediate-"Stop" behavior. Alternatives rejected: gating on loadGate.current === null instead of loadProgress.ready — loadGate.current is only nulled out on an explicit model/device-switch reset or load cancel, not after a successful ready, so it stays non-null (holding an already-resolved promise) for the rest of the session; checking it would have made every subsequent live start take a detour through "loading-model" for a promise that resolves on the same tick, an unnecessary (if harmless) status flash. loadProgress.ready is the same signal showModelProgress/modelReady already use elsewhere in App.tsx and tracks exactly what the fix needs: has this model/device selection reported ready yet. Testing: no unit test added — LiveControls has no existing component-level test file to extend, and the state machine this fix changes (startLive()'s wait/race handling around a real Worker, MediaStream, and DOM) has no natural seam for a bun test unit test without mocking most of the worker/mic stack, which the project's live tests (tests/live.test.ts, tests/live-accuracy.test.ts) deliberately avoid — they test the pure segment/scheduling logic in src/lib/live.ts instead. Manual verification via tests/live.md remains the intended coverage for this UI path, and its permission-flow step was updated accordingly (see Consequence above).

D-035 — whisper-tiny (multilingual) "slower than whisper-tiny.en" is real, but only on out-of-distribution audio, and it is a decode-length effect, not an asset/config gap

Status: Investigated, no code fix warranted (2026-07-30) Reported symptom: onnx-community/whisper-tiny (multilingual) feels "a good bit slower" than onnx-community/whisper-tiny.en at inference. Three candidate mechanisms were checked, in order, using the Symbol.for("onnxruntime") ort-web-under-Bun harness (real onnxruntime-web wasm EP, q8 dtype — the catalog's WASM default for both models, D-006) driven through the exact pipeline call src/worker/inference.worker.ts makes (chunk_length_s: 30, stride_length_s: 5, return_timestamps: true, max_new_tokens: 440, D-021's single-pass path), single-threaded to avoid cross-process CPU contention (a real hazard in this sandbox: two unrelated sibling-agent bun test runs at 500%+ CPU each inflated early timings 3–5× until killed — matches the pre-existing "ort-web under Bun harness" tooling note about contention).

1. Hub asset parity — ruled out. curl'd the Hub tree API directly (not just WebFetch's HTML summary) for both repos' onnx/ directories: both ship an equivalent decoder_model_merged_quantized.onnx (merged, KV-cache-enabled) at nearly identical size — multilingual 30,719,241 B vs .en 30,718,858 B (383 B apart, consistent with one extra vocab row: 51865 vs 51864 tokens). No fallback to a non-merged/non-cached decoder for either model.
2. Catalog/dtype config — ruled out (per the task brief, already checked before this investigation): both catalog entries resolve to the same wasm/webgpu/webnn dtypes; the verified flag difference is metadata-only.
3. Actual measured timing — this is where the real, reproducible effect is, and it's content-dependent:
   • On tests/audio_files/en_aiproblem.m4a (14.4 s, clean English speech, both models transcribe it correctly): no reliable difference. Interleaved back-to-back runs (both orders tried, to rule out first-vs-second bias) gave multilingual/.en warm-average ratios of 0.938 and 0.891 — i.e. multilingual was slightly faster, well within run-to-run noise (individual pass times ranged 1150–1460 ms across both models). The "extra 1–2 prompt tokens" mechanism the task brief had already flagged as probably-negligible is confirmed negligible here.
   • On tests/audio_files/da_festuge.m4a (21.5 s, Danish speech, no language option passed — the real default when a user leaves the UI's language picker on "auto", exactly as src/App.tsx sends it): both models mistranscribe it as English and fall into a greedy-decoding repetition loop (a known Whisper failure mode on audio that doesn't match the assumed language, not specific to this app), but the two checkpoints' loops differ hugely in how many tokens they run before landing on eos_token_id:
     • whisper-tiny.en: 67 generated tokens (" The first one is the first one. ... " repeating), pass time 2.6–5.7 s.
     • whisper-tiny (multilingual): 434 generated tokens — essentially the entire max_new_tokens: 440 budget — (" ... the main clar, the main clar, ..." repeating), pass time 9.5–14 s.
     • Ratio: ~6.5× more tokens, ~2–5× wall-clock (sub-linear vs. token count because KV-cache growth cost isn't the whole per-token cost; still a large, user-visible gap). A second same-order and a reversed-order run reproduced the same order-of-magnitude split. Mechanism: this is not a compute-cost-per-token difference (ruled out by the clean-English measurement) and not an asset/config gap (ruled out above). It's that the multilingual model's forced decoder prompt carries an extra task token (<|transcribe|>, from its forced_decoder_ids) that .en's prompt doesn't have, and — on audio where the model is already out-of-distribution enough to hallucinate — that different prompt context measurably changes how long the hallucination loop runs before EOS, not just the compute cost of the prompt tokens themselves. Because this app's max_new_tokens: 440 (D-021) forces the classic single-pass path with no no_repeat_ngram_size/repetition_penalty set in either model's generation_config.json or in inference.worker.ts's pipeline options, there is nothing to cut a non-terminating decode short before the hard token cap — so whichever checkpoint happens to run closer to that cap on a given input is dramatically slower for that one pass. Confirmed generation configs for both models differ only in the multilingual-specific fields (lang_to_id, task_to_id, an extra forced token at position 2) — neither sets a repetition guard. Fix: none applied. This is inherent behavior of the two upstream checkpoints reacting differently to the same repetition-loop failure mode, not a bug in this app's catalog, config, or asset selection — there is nothing in src/lib/models.ts to change (both already resolve to identical, merged, cached-KV decoders at the same dtype on every device tier). Inventing a per-model workaround (e.g. steering multilingual .en-adjacent use cases onto the .en model) was rejected: it would silently break real multilingual transcription for users who need it, to paper over a worst-case latency tail on inputs the model was always going to mistranscribe anyway. Scope: tiny/tiny.en only measured; the same forced-decoder-ids asymmetry exists for every multilingual/.en pair in the catalog (D-006), so the same content-dependent latency tail plausibly applies to base/small/medium too, untested here. Revisit: if this becomes a recurring complaint (not just a one-off report), the actionable mitigation is adding no_repeat_ngram_size (e.g. 3) and/or a repetition_penalty to the pipeline options in inference.worker.ts — that targets the actual failure mode (unbounded repetition loops on out-of-distribution audio) for both .en and multilingual models, rather than papering over one symptom of it. That change needs its own accuracy verification against tests/live-accuracy.test.ts and tests/live-real-model.test.ts (a repetition guard can change correct-transcript wording too) before landing, which is out of scope for this investigation.

D-036 — Root AGENTS.md is the single source of truth for agents

Status: Accepted (2026-10-29) docs/CLAUDE.md called itself the single source of truth, but Claude Code only auto-loads it when working inside docs/, other agents (Codex, Gemini, Cursor) never read it, and most of it was the generic bun init boilerplate (sample Bun.serve/HTML snippets) rather than repo-specific rules. Decision: a root AGENTS.md (read natively by Codex/Gemini, imported by the root CLAUDE.md via @AGENTS.md) holds the rules: verified commands, the conventions agents would otherwise break (Bun only, D-013 engine split, the transformers patch, COEP, D-020, D-030, D-032), boundaries, and docs duties. CLAUDE.md, GEMINI.md, and docs/CLAUDE.md are pointers only. The product spec from docs/CLAUDE.md §1 moved to PRODUCT.md; DESIGN.md records the shipped visual system (both read by the impeccable skill). Specs from the superpowers workflow live permanently in docs/specs/; plans live in docs/plans/ only while work is in progress and are git rm'd before merging. Agent tool permissions are defined in docs/PERMISSIONS.md and rendered to .claude/settings.json, .codex/, .gemini/settings.json. Kept as-is: docs/Architecture.md, docs/Decisions.md (same filenames — renaming would break every existing reference) and docs/WorkBreakdown.md as the one backlog (its own rule: no separate backlog), rather than adding a docs/TODO.md. Trade-off: AGENTS.md is kept short (<150 lines) on purpose — rationale goes here, not there. Changes to it redirect every agent session, so review them like CI config.

D-037 — Light theme via prefers-color-scheme; Inter self-hosted outside Bun's CSS bundler

Status: Accepted (2026-10-29) The UI was dark-only by accident: :root declared color-scheme: light dark but every colour was a hard-coded dark value, so light-mode OS users got native light form controls on a dark page. The font stack also named Inter first without ever loading it — everyone saw the system font. Theme: every colour is now a token on :root (dark values unchanged, so the dark theme renders identically); @media (prefers-color-scheme: light) overrides the tokens with a mirrored light palette (same blue accent family, darkened for contrast). No in-app toggle — follow the OS. All text/background pairs measured ≥ 4.5:1 in both themes. The PWA manifest/icons stay dark (a manifest has one theme_color); the <meta name="theme-color"> tags follow the scheme. Font: maintainer chose to self-host Inter (over dropping it from the stack). Source: @fontsource-variable/inter (OFL-1.1), a devDependency like Transformers.js (D-031) because only the build reads it. COEP require-corp (D-023) rules out a font CDN anyway. Two Bun 1.3.14 behaviours shaped the wiring (scripts/fonts.ts):

1. Bun's HTML-entrypoint CSS bundler inlines every resolvable url() asset as base64 — loader: { ".woff2": "file" } is ignored. Bundled normally, the 7 subsets turned the render-blocking stylesheet from 14 KB into 305 KB and defeated unicode-range lazy loading. So the production build marks *.woff2 external, copies the subsets to dist/fonts/, and precaches them (and hashes their bytes into the SW cache version). The dev server has no external option, so ensureFonts() copies them into gitignored src/fonts/ where the dev bundler can resolve (and inline) them.
2. Bun's minifier rewrites format('woff2-variations') to the unquoted format(woff2-variations), which is not a valid keyword — Chrome then drops the whole src and registers no faces. Use format("woff2") (variable woff2 needs nothing more). Verified in headless Chrome: 7 faces registered, only the latin subset fetched for English UI text.

D-038 — Typecheck with TypeScript 7; CI gates image builds on typecheck + fast tests

Status: Accepted (2026-10-29) tsconfig.json was already strict, but typescript was not installed and Bun strips types without checking them, so nothing ever type-checked. Decision: typescript (7.x, the native compiler — a full check of all 57 project files takes < 1 s) is a devDependency; bun run typecheck runs tsc --noEmit. The code base was already clean. .gitlab-ci.yml gains a check stage before build: bun install --frozen-lockfile, bun run typecheck, and the fast test suite (bun test --path-ignore-patterns='tests/live-real-model.test.ts', ~6 s). The real-model accuracy test stays local-only: it downloads ~600 MB of weights and runs for many minutes on CPU. Consequence: a type error or failing unit test now blocks image pushes.

D-039 — Real-model accuracy test: time budget scales with clip length; per-clip WER bars only with a measurement

Status: Accepted (2026-10-29) tests/live-real-model.test.ts had failed since en_alien (the 15-min D-021 long-file fixture) was added on 2026-07-15: the test auto-enrolls every clip in tests/audio_files/, but its 900 s timeout and absolute MAX_WER = 0.25 were calibrated on 14–21 s clean clips. Investigation (instrumented run, no timeout): no hang — 124 transcribe passes, 1189 s total (whole-file pass 420 s for 888 s of audio; live passes re-transcribe overlapping segments, 3452 s of audio in total). With the time available it then failed on accuracy instead: live WER 0.251, and the whole-file baseline itself 0.261 — the clip (gaming livestream: music, game voices, crosstalk) is simply harder than the bar, independent of the live pipeline. The relative check (live ≤ file + 0.1) passed. Decision: each clip's timeout is 900 s (download headroom) + 3 s × clip seconds (duration from an ffmpeg header probe; ≈ 2× the measured 1.34 s/s). MAX_WER_BY_CLIP gives en_alien an absolute bar of 0.35 with the measurement recorded beside it. Rejected: deriving the absolute bar from the whole-file WER — a broken model would then pass whenever its whole-file output is equally bad, which is exactly what the absolute bar exists to catch. Consequence: full bun test ≈ 19 min on CPU (en_alien ≈ 20 min of it on a cold run); the fast suite excludes this file.

D-WS6a — Danish fine-tuned Whisper models: our own ONNX + CTranslate2 conversions, unpublished until the owner says go

Status: Proposed (2026-10-29, spec 001 Goal 6) Candidates checked (licence + architecture, from each repo's model card, config and notices):

• thorhojhus/whisper-small-danish — MIT; plain WhisperForConditionalGeneration, whisper-small shapes and tokenizer unchanged (80 mel bins, 12/12 layers, vocab 51865). Kept.
• thorhojhus/whisper-large-v3-turbo-danish — MIT; plain large-v3-turbo (128 mel, 32/4 layers, vocab 51866). Kept.
• danish-foundation-models/edda-v0.2 — Apache-2.0 confirmed (licensor: the Alexandra Institute; base model MIT); plain large-v3-turbo architecture, so Transformers.js loads it like the stock turbo. Its generation_config.json pins language: "da", task: "transcribe" and num_beams: 5. Kept. Catalog: three entries with verified: false everywhere, ids built from one exported constant DANISH_MODELS_HF_OWNER (placeholder "PENDING-OWNER"): <owner>/<name>-ONNX for the browser engine and <owner>/<name>-ct2-int8 as fasterWhisperModel (a Hub repo id — faster-whisper accepts those as well as its aliases). Publishing under the owner's account is a one-line change.