839 lines
55 KiB
Markdown
839 lines
55 KiB
Markdown
# Decisions
|
||
|
||
Lightweight ADR log. One entry per decision that shapes the codebase. Statuses:
|
||
**Accepted** (act on it), **Proposed** (default choice, revisit before building on it),
|
||
**Superseded** (kept for history, points to replacement).
|
||
|
||
Add new entries at the bottom with the next `D-###` id. Never rewrite an Accepted
|
||
entry to mean something else — supersede it.
|
||
|
||
---
|
||
|
||
## D-001 — Bun as the only runtime and toolchain
|
||
**Status:** Accepted (mandated by CLAUDE.md)
|
||
Bun replaces Node, vite/webpack (via `bun build` + HTML imports), jest/vitest
|
||
(`bun test`), express (`Bun.serve`), better-sqlite3 (`bun:sqlite`), and dotenv.
|
||
**Consequence:** no vite plugin ecosystem; PWA assets (service worker, manifest) are
|
||
wired by hand instead of via a plugin.
|
||
|
||
## D-002 — Transformers.js (`@huggingface/transformers`) for all Whisper inference
|
||
**Status:** Accepted for browser inference; the server side is **superseded by D-013**
|
||
(faster-whisper).
|
||
Same library in the browser (WebGPU/WASM) and on the server (ONNX Runtime under Bun).
|
||
**Why:** one model catalog, one pipeline API, no native whisper.cpp build step.
|
||
**Trade-off:** server CPU inference is slower than whisper.cpp; acceptable — the server
|
||
engine is a convenience for logged-in users, not the primary path.
|
||
|
||
## D-003 — Inference runs in a dedicated Web Worker
|
||
**Status:** Accepted
|
||
Model load + transcription block for seconds-to-minutes; running them on the main
|
||
thread would freeze the UI. Communication via a small typed postMessage protocol
|
||
(`src/worker/protocol.ts`) shared between UI and worker.
|
||
|
||
## D-004 — WebGPU with automatic WASM fallback
|
||
**Status:** Accepted (amended by D-017 — "auto" now probes WebNN first)
|
||
Detect via `navigator.gpu?.requestAdapter()` inside the worker; pass
|
||
`device: "webgpu" | "wasm"` to the pipeline. The active backend is surfaced in the UI
|
||
but users don't have to configure anything. Encoder-heavy Whisper benefits ~10-40× from
|
||
WebGPU where available (Chromium); WASM keeps Firefox/Safari working.
|
||
|
||
## D-005 — Model weights cached via the browser Cache API (Transformers.js default)
|
||
**Status:** Accepted
|
||
Transformers.js already persists Hub downloads in the Cache API (`useBrowserCache`).
|
||
We do **not** re-cache weights in the service worker — the SW precaches only the app
|
||
shell. **Consequence:** offline = installed PWA + at least one previously loaded model;
|
||
the UI must show per-model "downloaded / not downloaded" state (query
|
||
`caches` for hub URLs) so users can prepare for offline use.
|
||
|
||
## D-006 — Quantized model catalog, curated not exhaustive
|
||
**Status:** Proposed
|
||
Expose a curated list (`src/lib/models.ts`): `whisper-tiny(.en)`, `base(.en)`,
|
||
`small(.en)`, `medium(.en)`, `large-v3-turbo`, each with sensible dtype defaults
|
||
(e.g. q4/q8 on WASM, fp16/q4 on WebGPU) and a size estimate shown before download.
|
||
**Why not free-form model ids:** predictable UX, tested combinations, honest
|
||
size/VRAM warnings. Revisit if users ask for arbitrary Hub ids.
|
||
|
||
## D-007 — PWA: hand-rolled service worker, app-shell precache
|
||
**Status:** Accepted
|
||
No Workbox/vite-plugin-pwa (conflicts with D-001 tooling). `bun build` emits hashed
|
||
assets to `dist/`; a small build step writes the precache manifest into `sw.ts`.
|
||
Strategy: precache app shell, cache-first for static assets, network-only for `/api/*`.
|
||
|
||
## D-008 — Passport.js via a connect-compat adapter over Bun.serve
|
||
**Status:** Accepted (Passport mandated by CLAUDE.md; adapter is the how)
|
||
Passport expects connect/express-style `(req, res, next)` and `req.login()`. We are
|
||
not allowed to use express (D-001), so `src/server/auth.ts` provides a thin adapter:
|
||
wrap Bun's `Request` in a minimal req/res shim, invoke the passport authenticate
|
||
chain, translate the outcome back to a `Response`. Local (username/password)
|
||
strategy first; other strategies plug into the same adapter later.
|
||
**Alternative rejected:** dropping Passport for hand-rolled auth — CLAUDE.md names
|
||
Passport explicitly.
|
||
|
||
## D-009 — Sessions + users in `bun:sqlite`
|
||
**Status:** Accepted
|
||
Single-file SQLite DB (path from env, default `./data/app.db`). Tables: `users`
|
||
(id, username, password_hash — Argon2 via `Bun.password`), `sessions`
|
||
(id, user_id, expires_at). Cookie: HttpOnly, SameSite=Lax, Secure in production.
|
||
No Redis/Postgres until there's a scaling reason (would be `Bun.redis` / `Bun.sql`).
|
||
|
||
## D-010 — Engine abstraction: `BrowserEngine` and `ServerEngine` behind one interface
|
||
**Status:** Accepted
|
||
`src/lib/engine.ts` defines `Engine { load(model), transcribe(audio), onProgress }`.
|
||
The UI never branches on engine type beyond the toggle. Keeps the server engine
|
||
(milestone M5) from leaking into UI code built in M1–M4.
|
||
|
||
## D-011 — Server engine transport: streaming HTTP response first, WebSocket if needed
|
||
**Status:** Accepted (implemented in M5, 2026-07-10)
|
||
`POST /api/transcribe` returns a streamed (chunked JSON-lines) response with progress
|
||
and partial transcripts. Simpler than WebSockets (no connection lifecycle, works with
|
||
plain fetch + ReadableStream). Switch to `Bun.serve` websockets only if we need
|
||
bidirectional control (e.g. cancel mid-run) that abort signals can't cover.
|
||
**As implemented:** `application/x-ndjson` of `segment`* then `result` | `error`
|
||
lines (wire types + guards shared client/server via `src/lib/engine.ts`); the
|
||
request body is raw float32 mono 16 kHz PCM — the decoded form the browser
|
||
already has — so the server never decodes containers. Cancel-mid-run needed no
|
||
WebSocket: `fetch`'s AbortSignal reaches `request.signal`, which forwards a
|
||
`cancel` to the sidecar (it checks between segments).
|
||
|
||
## D-012 — Audio decoding on the main thread via Web Audio API
|
||
**Status:** Accepted
|
||
`decodeAudioData` (handles most audio and audio-in-video containers) → mono downmix →
|
||
resample to 16 kHz → transfer Float32Array to worker (zero-copy). No ffmpeg.wasm
|
||
unless we hit real container-support gaps; note in `src/lib/audio.ts` if we do.
|
||
|
||
## D-013 — Server engine runs faster-whisper (Python sidecar)
|
||
**Status:** Accepted (user requirement, 2026-07-10; supersedes the server half of D-002)
|
||
Server-side inference (M5) uses [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
|
||
(CTranslate2) instead of Transformers.js under Bun. Browser inference is untouched —
|
||
faster-whisper **cannot** run in a browser (checked 2026-07): it is Python over native
|
||
C++ CTranslate2, which has no WASM build (OpenNMT/CTranslate2#528 open since 2021, no
|
||
progress) and relies on native SIMD/threading kernels. So D-002 stays Accepted for the
|
||
browser, where Transformers.js remains the engine.
|
||
**Shape:** Bun stays the only web runtime. `src/server/transcribe.ts` talks to a small
|
||
Python sidecar (`server-py/`, faster-whisper behind a minimal stdio or localhost-HTTP
|
||
interface) that `Bun.spawn` starts and supervises. The `POST /api/transcribe` contract
|
||
(D-011: streamed JSON-lines) is unchanged — faster-whisper's segment generator maps
|
||
naturally onto it.
|
||
**Consequence:** the "one inference codebase" invariant narrows to "one *browser*
|
||
inference codebase + one shared model catalog"; `src/lib/models.ts` needs a mapping
|
||
from catalog ids to faster-whisper/CTranslate2 model names (`Systran/faster-whisper-*`);
|
||
deployment gains a Python ≥3.9 dependency (server engine only — the anonymous/browser
|
||
path still deploys with Bun alone, and the app must degrade gracefully when the
|
||
sidecar is absent).
|
||
**Why:** CTranslate2 CPU inference is several times faster and more memory-efficient
|
||
than ONNX Runtime under Bun; the server engine becomes genuinely useful rather than a
|
||
slow convenience.
|
||
|
||
## D-014 — WASM sessions run at graphOptimizationLevel "basic"
|
||
**Status:** Accepted (bug workaround, 2026-07-10)
|
||
On the WASM backend, the inference worker passes
|
||
`session_options: { graphOptimizationLevel: "basic" }` to the Transformers.js
|
||
pipeline. The onnxruntime-web build bundled with Transformers.js 4.2
|
||
(1.26.0-dev.20260416) has a broken level-2 QDQ graph transform
|
||
(`qdq_actions.cc` `TransposeDQWeightsForMatMulNBits`) that fails session
|
||
creation for the 8-bit Whisper decoders (`_quantized`/`_int8`/`_uint8`), whose
|
||
tied `embed_tokens` weight is both gathered (embedding) and matmul'd (lm_head):
|
||
"Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale".
|
||
Reproduced outside the browser against `decoder_model_merged_quantized.onnx`
|
||
of `onnx-community/whisper-tiny`; levels "all"/"extended" fail, "basic"/
|
||
"disabled" load fine, and onnxruntime-node 1.24.3 loads the same file at every
|
||
level (regression in the ort-web dev build, not the model).
|
||
**Scope:** WASM only — WebGPU keeps the default level. The q4 conversions use
|
||
native `MatMulNBits` ops (no QDQ), so skipping level-2 costs the larger WASM
|
||
models nothing.
|
||
**Revisit:** when Transformers.js bumps its onnxruntime-web dependency past
|
||
1.26.0-dev.20260416, retest q8-on-WASM at the default level and drop this.
|
||
|
||
## D-015 — Runtime feature flags via server env vars; first flag gates auth
|
||
**Status:** Accepted (2026-07-10)
|
||
Feature flags are read from the **server's** process environment at startup
|
||
(`src/server/flags.ts`, `envFlag()`: unset/empty → the flag's default;
|
||
`false`/`0`/`no`/`off` any case → off; anything else → on). Runtime-only by
|
||
design: one Docker image serves every configuration — flags are flipped in
|
||
docker-compose.yml / `docker run -e`, never at build time, so the client
|
||
bundle stays identical across deployments.
|
||
**First flag:** `AUTH_ENABLED` (default **on**) gates the M4 login/register
|
||
feature. Off, the server 404s every `/api/auth/*` route. The client is never
|
||
told separately: `fetchAuthStatus` (the `GET /api/auth/me` probe `useAuth`
|
||
already makes on mount) reads 404 as "feature off" and `<AuthForms />` renders
|
||
nothing. The Sign in button therefore only appears after the probe confirms
|
||
the endpoints exist — a disabled deployment never flashes auth UI, and an
|
||
offline/unreachable server also hides it (signing in would be impossible
|
||
anyway), preserving the anonymous-first invariant (no error surfaced).
|
||
**Consequence:** any future flag that must reach the browser needs a
|
||
server-derived signal (an endpoint the client can probe, like /me here) —
|
||
`BUN_PUBLIC_*` build-time inlining is not used for feature flags.
|
||
|
||
## D-016 — Live microphone transcription via segment re-transcription (browser engine)
|
||
**Status:** Accepted (2026-07-10; promotes "live transcription" out of the
|
||
WorkBreakdown out-of-scope list → milestone M7)
|
||
Live (microphone) transcription runs on the **browser engine only** — same
|
||
privacy story as file uploads: audio never touches the network, works for
|
||
anonymous users, works offline. Whisper is not a streaming model (it consumes
|
||
up-to-30 s windows), so "live" is approximated by a **segment loop**:
|
||
|
||
- **Capture (main thread):** `getUserMedia` → `AudioContext` at the device's
|
||
native rate → an `AudioWorklet` chunker posts ~100 ms Float32 frames to the
|
||
main thread, which resamples them to 16 kHz with the pure, stateful
|
||
`StreamingResampler` (windowed-sinc low-pass; `resampleLinear` has no
|
||
anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above
|
||
8 kHz into the speech band and measurably hurt accuracy — see
|
||
`tests/resampler.test.ts`). The worklet module is loaded from a **Blob URL**
|
||
built from an inline code string — no extra served file, no
|
||
build/service-worker changes, and it keeps working offline. Where
|
||
`audioWorklet`/Blob modules are unavailable (older Safari), fall back to a
|
||
`ScriptProcessorNode`. The context is *not* forced to 16 kHz because Firefox
|
||
cannot connect a `MediaStreamAudioSourceNode` across sample rates.
|
||
- **Segmentation (pure, unit-tested):** a simple RMS energy gate
|
||
(`src/lib/live.ts`) splits speech into segments: a segment starts at speech
|
||
onset (with a short pre-roll), commits after a run of trailing silence or at
|
||
a max length (under Whisper's 30 s window), and pure-silence audio is
|
||
dropped entirely — Whisper hallucinates on silence, so never transcribe it.
|
||
Two accuracy refinements (2026-07-14): the gate has **hysteresis** (once a
|
||
segment is open, frames stay "speech" down to half the onset threshold, so
|
||
quiet phonemes don't fragment utterances), and a max-length commit is a
|
||
**soft cut** — it splits at the last sub-threshold frame in the trailing
|
||
window instead of mid-word at the hard cap, seeding the next segment with
|
||
the remainder (a mid-word cut corrupts the boundary word in both segments).
|
||
Dropped (near-silent) segments also clear any tentative text they produced,
|
||
so a hallucinated tentative pass can neither linger on screen nor be adopted
|
||
as the next segment's provisional text. Production tuning lives in
|
||
`LIVE_TUNING` (`src/lib/live.ts`), shared by `src/App.tsx` and the accuracy
|
||
harness (`tests/live-accuracy.test.ts` — tone-word oracle engine + WER, so
|
||
segmentation/scheduling/resampling accuracy is regression-tested under
|
||
`bun test` without a model).
|
||
No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
|
||
- **Inference (existing worker, serialized):** the current segment is
|
||
re-transcribed through the **existing** inference worker/pipeline on a
|
||
rolling basis (tentative text, replaced each pass); a committed segment gets
|
||
one final pass and its text freezes. At most one inference is in flight;
|
||
final passes take priority over tentative refreshes, which are throttled and
|
||
always use the latest audio snapshot (intermediate snapshots are skipped, so
|
||
the loop degrades gracefully when inference is slower than real time).
|
||
- **Protocol:** UI↔worker messages gain an optional `requestId` echoed on
|
||
`partial`/`result`/`error`, so the live controller can correlate responses
|
||
and coexist with the file flow. The file flow keeps working without ids.
|
||
|
||
**Alternatives rejected:** streaming into the worker with a custom
|
||
chunk-stitching decoder (large fork of Transformers.js internals for marginal
|
||
latency); MediaRecorder + re-decoding compressed blobs (adds codec latency and
|
||
`decodeAudioData` round-trips for no benefit over raw PCM taps).
|
||
**Consequence:** live quality/latency is bounded by model speed on the user's
|
||
device — the UI must keep tentative text visibly distinct from committed text.
|
||
Server-engine live input (WebSocket streaming to faster-whisper) is a separate
|
||
future decision; D-011's "WebSocket if needed" note is the hook.
|
||
|
||
## D-017 — "Auto" device order: WebNN → WebGPU → WASM
|
||
**Status:** Accepted (2026-07-14; amends D-004, which only knew WebGPU → WASM)
|
||
With WebNN support added (commit "WebNN support initial"), the `"auto"` device
|
||
preference resolves in priority order **WebNN → WebGPU → WASM**: the worker
|
||
probes `navigator.ml` (WebNN) first, then `navigator.gpu.requestAdapter()`
|
||
(WebGPU), and always keeps WASM as the last resort. The probe/ordering policy
|
||
is pure and unit-tested (`deviceAttempts` in `src/worker/protocol.ts`,
|
||
`tests/protocol.test.ts`); the worker only supplies the capability probe.
|
||
Pipeline-creation failures walk the same chain — an auto-detected WebNN that
|
||
fails still tries WebGPU before WASM. An **explicit** `"webnn"` or `"webgpu"`
|
||
preference deliberately falls back straight to WASM (no detour through the
|
||
other accelerator — the user asked for a specific backend, so honor it or take
|
||
the universal fallback). The selected backend stays user-visible: the worker's
|
||
`{status:"ready", device}` message drives a "Running on …" badge next to the
|
||
Backend select (`ModelPicker`) and the engine line in the footer (D-004's
|
||
"surfaced in the UI" requirement).
|
||
|
||
## D-018 — Cancel a file transcription by terminating the worker
|
||
**Status:** Accepted (2026-07-14)
|
||
Users can cancel an in-flight file run (decode, model download kicked off by
|
||
the drop, or the transcription pass itself) via a Cancel button next to the
|
||
file name. Transformers.js exposes no abort API for a running `asr()` call,
|
||
and on the WASM backend the pass blocks the worker's event loop — a "cancel"
|
||
postMessage would sit unread until the pass finished anyway. So cancellation
|
||
**terminates the inference worker** outright; it is the only reliable
|
||
preemption and also aborts an in-flight weight download. A `fileRun` counter
|
||
in `App.tsx` marks the decode that was running when the user cancelled as
|
||
stale (`decodeAudioData` itself cannot be aborted), so it never posts audio to
|
||
the fresh worker. The next run respawns the worker and reloads the pipeline —
|
||
cheap, because the weights stay in the browser Cache API (D-005); the UI
|
||
resets `loadStarted`/`loadProgress` so the reload starts from a clean slate.
|
||
Live mode is untouched: it has its own stop/drain semantics (D-016), and a
|
||
live session never coexists with a file run (mutually exclusive in the UI).
|
||
**Alternatives rejected:** a cancel message handled between chunks (dead on
|
||
WASM, where the event loop is blocked; and still seconds of latency
|
||
elsewhere); `AbortSignal` plumbed into Transformers.js (no upstream support).
|
||
**Consequence:** cancel throws away the loaded pipeline even when only the
|
||
transcription was unwanted — an accepted cost since reloading from cache takes
|
||
a few seconds.
|
||
**Amendment (2026-07-14, EngineToggle landing):** a server-engine run cancels
|
||
by aborting the `/api/transcribe` fetch instead (`AbortController`; the server
|
||
cancels the sidecar pass between segments, D-011/D-013) — the inference worker
|
||
and its loaded pipeline are left untouched in that path.
|
||
|
||
## D-019 — Server-based live microphone transcription
|
||
**Status:** Accepted
|
||
**Why:** M7 future want. Enables logged-in users on weak devices to transcribe live microphone audio using the server engine.
|
||
**Shape:** The client opens a WebSocket connection to `/api/transcribe/live`. The client streams 16kHz Float32 PCM chunks (from `MicCapture`) over the WebSocket. The server runs `LiveTranscriber` and `LiveSegmenter` (from `src/lib/live.ts`, which is pure and runs perfectly in Node/Bun). For each segment/pass, the server's `LiveTranscriber` writes the audio to a temporary file and invokes the Python sidecar. The server sends `LiveUpdate` objects back to the client over the WebSocket.
|
||
**Consequence:** The client does zero ML and minimal buffering, acting only as a mic-streamer and UI-renderer for live passes on the server.
|
||
|
||
## D-020 — Model downloads are user-initiated, never a side effect of selection
|
||
**Status:** Accepted (2026-07-14)
|
||
Selecting a model in `ModelPicker` (or changing the backend preference) no longer
|
||
starts the weight download. Weights load exactly when they are needed or asked for:
|
||
- **Lazy:** a file drop or live start on the **browser** engine posts the
|
||
(idempotent) load, as both paths already did — the download overlaps decoding /
|
||
mic startup.
|
||
- **Explicit:** a **Download** button on the selected, not-yet-cached model in
|
||
`ModelPicker` starts the load immediately — this keeps D-005's "prepare for
|
||
offline use" story alive now that selection alone caches nothing.
|
||
- **Server engine:** server runs never touch the browser pipeline, so with the
|
||
toggle on "server" nothing downloads. (Previously the eager selection load
|
||
downloaded browser weights even when every transcription would run server-side.)
|
||
**Consequence:** model/backend switches just reset the load UI state
|
||
(`loadStarted`, progress); the next load starts from clean progress. The explicit
|
||
download is a full pipeline load, so it also warms up inference and surfaces the
|
||
"Running on …" backend badge.
|
||
**Amendment (2026-07-14):** while the engine toggle is on "server" (signed in,
|
||
available), `ModelPicker` states explicitly that nothing is downloaded to the
|
||
device; the Download button stays (with a matching tooltip) for on-device /
|
||
offline preparation.
|
||
|
||
## D-021 — Long-file hardening: no idle timeout on the transcribe stream, heartbeats, single-pass Whisper generate
|
||
**Status:** Accepted (2026-07-15)
|
||
Long uploads (tested: 15-min Opus, `tests/audio_files/en_alien.opus`) hung or died
|
||
on both engines. Three independent causes, all length-dependent:
|
||
1. **Bun's default 10 s `idleTimeout` reset `/api/transcribe` mid-stream.** Segment
|
||
gaps legitimately exceed 10 s (sidecar spawn + model load/download before the
|
||
first segment; slow decode windows — 30 s+ gaps observed with large-v3-turbo on
|
||
CPU). The stream then ended without a `result` line, so the client reported
|
||
"connection was lost" after minutes of apparent hang. Fix: `src/index.ts` calls
|
||
`server.timeout(request, 0)` for `POST /api/transcribe` (and 90 s for
|
||
`GET /api/transcribe/status`, whose first probe waits up to 60 s for the sidecar
|
||
import), and `src/server/transcribe.ts` enqueues a blank ndjson line every 5 s as
|
||
a heartbeat — reverse proxies get liveness, dead connections surface as enqueue
|
||
failures, and `ServerEngine` already skips blank lines.
|
||
2. **Bun's default 128 MB `maxRequestBodySize` rejected uploads > ~35 min** (raw
|
||
f32 PCM is 3.84 MB/min) before the route's own 512 MB limit ever applied. Fix:
|
||
`maxRequestBodySize` is set to the shared `MAX_UPLOAD_BYTES` export.
|
||
3. **Browser engine: Transformers.js 4.2's Whisper `generate()` takes a seek-loop
|
||
path when `return_timestamps` is set without `max_new_tokens`** — 1–3 full decode
|
||
passes per window, redundant under the pipeline's own 30 s/5 s chunking. Measured
|
||
on the 15-min clip (tiny.en, onnxruntime-node): 80 generate passes for 44 windows,
|
||
1.4× slower, WER 44.7 % vs 31.5 % (duplicated boundary text), and the streamer's
|
||
`on_finalize` fired per pass instead of per window, so `TranscribeProgressTracker`
|
||
saturated the progress bar at ~60 % of the run and the UI looked frozen for the
|
||
remainder (on WASM: many minutes). A window whose last boundary timestamp is 0 can
|
||
even loop forever. Fix: the worker passes `max_new_tokens: 440`, which selects the
|
||
classic single-pass-per-window path (440 ≈ 448 decoder positions minus prompt).
|
||
**Revisit:** when Transformers.js changes the seek-loop condition (currently
|
||
`return_timestamps && !max_new_tokens`, `src/models/whisper/modeling_whisper.js`),
|
||
retest whether the explicit `max_new_tokens` is still needed.
|
||
|
||
## D-022 — Medium models never default to fp16 in the browser
|
||
**Status:** Accepted (2026-07-15)
|
||
Reported symptom: transcribing `en_aiproblem` in-browser with Whisper Medium
|
||
printed just "the". The medium catalog entries defaulted to fp16 on WebNN
|
||
(whole model) and WebGPU (encoder) — and D-017's "auto" order now tries WebNN
|
||
first, so browsers exposing `navigator.ml` silently moved medium onto fp16.
|
||
Two problems with those fp16 conversions
|
||
(`onnx-community/whisper-medium(.en_timestamped|-ONNX)`):
|
||
1. The fp16 ONNX files crash onnxruntime's graph optimizer at session
|
||
creation (`SimplifiedLayerNormFusion` fails on their
|
||
`InsertedPrecisionFreeCast_*` nodes) — reproduced on both onnxruntime-node
|
||
and onnxruntime-web 1.26.0-dev (wasm EP). EPs whose optimizer pipeline
|
||
skips that fusion can load the model and hit problem 2 instead.
|
||
2. Whisper medium/large encoder activations are the classic fp16 overflow
|
||
case — degenerate output with no error raised.
|
||
Meanwhile q4 medium was verified correct end-to-end on the real browser
|
||
runtime (onnxruntime-web wasm EP driven under Bun via the
|
||
`Symbol.for("onnxruntime")` override, `graphOptimizationLevel: "basic"` as in
|
||
D-014, `max_new_tokens: 440` as in D-021; both the `.en` and multilingual
|
||
conversions, auto language detect included). **Fix:** medium's `webgpu` and
|
||
`webnn` defaults are q4 (`ACCEL_Q4` in `src/lib/models.ts`). A backend that
|
||
cannot execute q4 (`MatMulNBits`) throws at pipeline creation and falls down
|
||
the D-017 device chain — strictly better than silently emitting garbage.
|
||
**Scope:** medium only — that's where load-failure evidence exists.
|
||
tiny/base/small keep fp16 on WebNN and small keeps fp16+q4 on WebGPU;
|
||
large-v3-turbo keeps fp16+q4 on WebGPU (the widely-used Transformers.js demo
|
||
configuration). If a one-word/garbage transcript ever shows up on those, suspect
|
||
their fp16 encoders and extend this decision.
|
||
**Revisit:** if the onnx-community medium conversions are re-exported with
|
||
working fp16 files, or once WebNN/WebGPU runs can be exercised in CI.
|
||
|
||
## D-023 — Cross-origin isolation (COOP/COEP) so WASM inference is multithreaded
|
||
**Status:** Accepted (2026-07-15)
|
||
Reported symptom: after D-022 moved medium to q4, a 14 s clip transcribed
|
||
correctly in-browser but "took ages" while native whisper needs seconds.
|
||
Cause: without `crossOriginIsolated` there is no SharedArrayBuffer, and
|
||
onnxruntime-web silently runs its (SIMD-threaded build!) WASM backend on
|
||
**one** thread. Measured on the exact browser runtime (ort-web wasm EP,
|
||
medium q4, the 14.4 s en_aiproblem clip): 52.1 s at 1 thread, 15.3 s at 4
|
||
(ort's default cap once isolation exists), 9.5 s at 12 — a 5.5× speedup.
|
||
**Fix:** `src/index.ts` sends `Cross-Origin-Opener-Policy: same-origin` +
|
||
`Cross-Origin-Embedder-Policy: require-corp` on every response in both serve
|
||
modes (`withIsolationHeaders`). Bun cannot attach headers to an HTMLBundle
|
||
route, so in development the shell moved to an internal `/__shell` route and
|
||
`fetch()` re-serves `/` (and SPA fallbacks) with the headers stamped on. The
|
||
inference worker raises `env.backends.onnx.wasm.numThreads` to
|
||
`min(hardwareConcurrency − 1, 12)` when `crossOriginIsolated` is true (ort
|
||
would otherwise cap at 4; without isolation the default is left alone since
|
||
ort clamps to 1 anyway).
|
||
**Consequences:** COEP `require-corp` blocks cross-origin subresources that
|
||
are neither CORS- nor CORP-approved. Everything the app loads today survives:
|
||
Hub model downloads use `fetch()` in CORS mode, all other assets are
|
||
same-origin, and the service worker preserves the headers it cached, so the
|
||
installed/offline app stays isolated too. Any future cross-origin `<img>`,
|
||
font, or iframe must ship CORS/CORP headers or be proxied same-origin.
|
||
**Verified:** headers present on `/`, worker, sw.js, `/ort/*` in both dev and
|
||
production serve modes; dev HMR asset routes unaffected by the `/__shell`
|
||
indirection. Real-browser thread scaling still deserves a spot check (the
|
||
5.5× number is ort-web under Bun, same runtime but not a browser tab).
|
||
**Amendment (2026-07-15, same day):** threads are per-device now. With
|
||
SharedArrayBuffer newly available, onnxruntime-web's *own* default thread
|
||
count becomes ~4 — and the JSEP accelerator EPs (WebGPU, WebNN) are not
|
||
thread-safe: real-browser WebGPU runs failed at OrtRun with "WebGPU
|
||
validation failed. Buffer with '' label is invalid". The worker now sets
|
||
`numThreads` right before each device attempt (`configureWasmThreads`):
|
||
`min(cores − 1, 12)` for the pure `wasm` device, **explicitly 1** for
|
||
`webgpu`/`webnn` (relying on the default is what broke). onnxruntime-web
|
||
fixes its thread pool at first-session init, so the first attempt in a
|
||
worker decides; a wasm fallback after a failed accelerator attempt may run
|
||
single-threaded until the next worker respawn — correct over fast.
|
||
|
||
## D-024 — Live mode batches queued final passes into merged passes
|
||
**Status:** Accepted (2026-07-17)
|
||
Reported symptom: stopping a live session got stuck on "Finishing up…" while
|
||
the machine sat at full CPU — indistinguishable from a hang, and the saturated
|
||
cores (D-023 uses most of them) made the whole browser feel locked up.
|
||
**Cause — a scheduling backlog, not a deadlock.** Whisper pads every inference
|
||
pass to a full 30 s encoder window, so a pass costs nearly the same whether it
|
||
carries 1 s or 29 s of audio (measured on the browser runtime, ort-web WASM EP
|
||
medium q4: ~4 s per pass at 12 threads, ~9–12 s at 4 threads, for 1–9 s
|
||
segments alike). Segments therefore commit faster than their final passes
|
||
complete whenever the model is slower than speech (medium/large on WASM), the
|
||
committed-segment queue grows all session, and `stop()` used to drain it **one
|
||
pass per segment** — a 60 s session on 4 threads measured a 52 s drain; longer
|
||
sessions scale it into minutes. All the while the UI showed only a static
|
||
"Finishing up…".
|
||
**Fix (two parts):**
|
||
1. `LiveTranscriber.pump()` batches consecutive queued segments into **one
|
||
merged final pass**, capped at the segmenter's `maxSegmentSeconds` so a
|
||
merged pass stays a single Whisper window. Segment audio is concatenated
|
||
(segments end on trailing silence or a soft cut, so splices land in natural
|
||
gaps); the result's chunk timestamps are mapped piecewise back to absolute
|
||
session time via each segment's own start offset. The backlog — and the
|
||
stop drain — is now bounded by ~one pass per 29 s of speech instead of one
|
||
pass per segment (the same 60 s/4-thread scenario drains in ~2 passes).
|
||
2. `App.tsx` surfaces `LiveUpdate.pendingFinalPasses` while stopping:
|
||
"Finishing up… transcribing N remaining segments" counts down instead of
|
||
freezing, on both the browser and the server-live (D-019) paths.
|
||
**Alternatives rejected:** skipping final passes for segments that already had
|
||
tentative text (during a backlog new segments get *no* tentative pass, so
|
||
skipping loses their words entirely); a "stop now" escape hatch (same data
|
||
loss, silently); merging across the 30 s window (impossible — Whisper's
|
||
context is one window).
|
||
**Consequence:** a merged pass attributes its whole text to the batch's first
|
||
segment (per-segment text attribution is meaningless after joining anyway) and
|
||
inter-segment silence is not re-inserted, so chunk timestamps inside a merged
|
||
pass are exact per segment region but a chunk spanning a splice maps to its
|
||
start's region. Covered by `tests/live.test.ts` (batching + cap) and the WER
|
||
gates (`tests/live-accuracy.test.ts`, `tests/live-real-model.test.ts` — the
|
||
real-model test's fast feed makes real inference lag, so merging is exercised
|
||
there with real weights).
|
||
|
||
## D-025 — A degraded-thread WASM fallback restarts the worker; WebGPU is skipped when fp16 needs an absent shader-f16
|
||
**Status:** Accepted (2026-07-17)
|
||
Reported symptom: Firefox performance far below Chrome, live transcription
|
||
worst. Chrome resolves D-017's auto chain to an accelerator; Firefox has no
|
||
WebNN and its WebGPU (where exposed at all) fails onnxruntime-web's
|
||
Chromium-built JSEP EP — notably it lacks the `shader-f16` feature, which the
|
||
catalog's fp16-encoder WebGPU defaults (small, large-v3-turbo) need. So
|
||
Firefox lands on WASM — but *how* it lands there was the real bug: the failed
|
||
WebGPU attempt initializes ort's WASM runtime at the accelerator-safe
|
||
`numThreads: 1` (D-023 amendment), the thread pool is fixed at first init,
|
||
and the fallback session then runs **single-threaded for the worker's whole
|
||
life** (measured 5.5×: 52 s vs 9.5 s on a 14 s clip). Live never respawns the
|
||
worker, and every live pass pads to a full 30 s window (D-024), so passes
|
||
took ~40–60 s: tentative text froze and the stop drain ballooned — D-023
|
||
amendment's "correct over fast" edge case is Firefox's *default* path.
|
||
**Fix (three parts):**
|
||
1. **Restart instead of a degraded fallback.** When `createPipeline` reaches
|
||
the `wasm` attempt after any accelerator attempt ran and multithreading is
|
||
actually available (`desiredWasmThreads() > 1`), the worker posts
|
||
`{type:"load-restart", modelId, preference, device:"wasm"}` and aborts the
|
||
load (sentinel error, never surfaced). The UI terminates the worker,
|
||
respawns it, re-issues the load with the explicit device, and remembers
|
||
`modelId::preference → device` for the session so later loads skip the
|
||
doomed attempt (session-only: a browser update may fix the backend; an
|
||
explicit user backend choice is a different key, so it retries once).
|
||
To make the respawn safe, transcribe posts now wait on a **load gate** in
|
||
`App.tsx` that resolves on `ready` (rejects on load error/cancel/crash):
|
||
no pass is ever queued inside a worker a restart is about to terminate —
|
||
its transferred audio buffer would die with it. The worker serialized
|
||
passes on the load anyway, so nothing starts later than before.
|
||
2. **shader-f16 probe.** The worker's WebGPU probe also records
|
||
`adapter.features.has("shader-f16")`; when the dtype that a WebGPU attempt
|
||
would use requires fp16 (`dtypeRequiresF16`, pure + unit-tested) and the
|
||
feature is absent, WebGPU is skipped outright — Firefox then loads
|
||
straight into multithreaded WASM with no wasted attempt.
|
||
3. **Live-on-WASM hint.** With live active on `wasm` and a heavy model
|
||
(encoder ≥ 300 M params: medium, large-v3-turbo), the UI says the model
|
||
runs slower than real time and suggests a smaller one — WASM lag there is
|
||
inherent (D-024's fixed window cost), not a bug, so tell the user instead
|
||
of looking frozen.
|
||
**Alternatives rejected:** re-posting in-flight passes after respawn (their
|
||
buffers are transferred — gone with the worker); a scout worker pre-flighting
|
||
the accelerator (doubles accelerated-path load time to save the failure
|
||
path); ort-web's proxy-worker mode to escape the pinned pool (undocumented
|
||
interplay with JSEP).
|
||
**Consequence:** the accelerator-failure path costs one worker respawn + a
|
||
pipeline reload from cache (a few seconds, once per model+preference per
|
||
session). A genuine both-attempts-fail load still reports its error
|
||
normally. `deviceAttempts` is unchanged; the new policy pieces
|
||
(`dtypeRequiresF16`, the `load-restart` message guard) are unit-tested in
|
||
`tests/protocol.test.ts`. Real-Firefox verification (badge shows WASM with
|
||
full threads, live keeps up on small models) still needs a manual run — same
|
||
no-browser dev-env caveat as M2/M3.
|
||
**Amendment (2026-07-17, same day — field report: tiny fixed, medium still
|
||
slow in Firefox):** three observability/robustness additions to tell "slow is
|
||
the CPU-vs-GPU physics" apart from "something is still degraded":
|
||
1. The probe **skips software (fallback) WebGPU adapters**
|
||
(`isFallbackAdapter`, both spec locations) — SwiftShader/llvmpipe/WARP
|
||
would *succeed* at session creation and then run slower than multithreaded
|
||
WASM, and the fallback chain only reacts to failures, not slowness.
|
||
onnxruntime-web itself uses whatever adapter it is handed (verified in
|
||
1.26.0-dev: no fallback-adapter or Firefox guard).
|
||
2. The `ready` message now carries **`threads`** — read back from
|
||
`env.backends.onnx.wasm.numThreads` *after* session creation, so ort's own
|
||
clamp-to-1 (missing isolation) is visible. The footer shows it:
|
||
"WASM (on-device, 11 threads)".
|
||
3. The worker **logs the device chain** (`[whisper] …` console lines): each
|
||
skipped backend (software adapter, missing shader-f16), each failed
|
||
attempt with its error, and the final device+threads — previously fallback
|
||
errors were swallowed silently, making field reports unactionable.
|
||
Expected healthy Firefox numbers for medium q4 on WASM (from the D-023/D-024
|
||
measurements): ~4 s per 30 s window at 12 threads, ~9–12 s at 4 — i.e. a
|
||
14 s clip in ~10 s. Chromium's GPU path is a further ~3–5× faster; that
|
||
remaining gap is inherent until Firefox's WebGPU works with ort-web's JSEP
|
||
EP. If the badge shows 1 thread or the console shows a surviving software
|
||
adapter, it is a bug in this decision's scope instead.
|
||
|
||
## D-026 — "Auto" never picks WebGPU on Firefox
|
||
**Status:** Accepted (2026-07-17)
|
||
Field-verified on Firefox (D-025's observability build): explicit
|
||
Backend = WASM runs medium with a full thread pool and healthy speed, while
|
||
explicit Backend = WebGPU (`device=webgpu wasmThreads=1`, correct per D-023)
|
||
**completes but runs far slower** than the same machine's multithreaded WASM.
|
||
That is the worst case for D-017's failure-driven fallback chain: ort-web's
|
||
WebGPU EP (Chromium-developed; no Gecko handling in 1.26.0-dev, verified)
|
||
creates the session *successfully* on Firefox, so nothing ever falls back —
|
||
"auto" happily selects the slowest available backend.
|
||
**Fix:** the worker's capability probe skips the WebGPU probe entirely when
|
||
the preference is `"auto"` and the UA is Firefox (`/Firefox\//` on
|
||
`navigator.userAgent`), with a `[whisper]` console line saying why. Firefox
|
||
auto therefore resolves straight to multithreaded WASM (no accelerator
|
||
attempt → no D-025 restart either). An **explicit** WebGPU selection is still
|
||
honored — that is the deliberate retest hatch: when a future
|
||
Transformers.js/ort-web bump gains real Gecko WebGPU support, verify via the
|
||
explicit path, then delete the UA check.
|
||
**Why UA sniffing:** there is no feature to detect — the sessions create and
|
||
run; only throughput distinguishes them, and a benchmark probe would cost
|
||
more than it saves. The check is one line, logged, scoped to `"auto"`, and
|
||
targets the one engine with field evidence.
|
||
**Revisit:** on any onnxruntime-web bump — retest explicit WebGPU on Firefox
|
||
(file `tests/live.md` note) and drop the check when it reaches parity.
|
||
|
||
## D-027 — Mobile OOM crash on Medium+ loads: sequential session loading (patched) + a warn/confirm gate
|
||
**Status:** Accepted (2026-07-28)
|
||
Reported symptom: downloading Whisper Medium (English) on a phone crashed the
|
||
whole browser — on Android, badly enough to briefly kick the OS back to the
|
||
home screen (consistent with the low-memory killer reaping background
|
||
processes, including the launcher, once the tab's memory spike triggered
|
||
system-wide pressure).
|
||
**Cause.** Whisper Medium's smallest available dtype on any device is q4
|
||
(D-022) — still ~460 MB combined encoder+decoder weights. Two independent,
|
||
compounding sources of transient memory at load time:
|
||
1. `@huggingface/transformers` buffers each ONNX weight file fully into one
|
||
JS-heap `Uint8Array` (`readResponse`/`loadResourceFile`,
|
||
`src/utils/hub.js`/`hub/utils.js`) before it is cached or handed to
|
||
onnxruntime — unavoidable either way (with or without a
|
||
`progress_callback`, and regardless of whether a URL string or a buffer
|
||
is passed to `InferenceSession.create`): onnxruntime-web's WASM backend
|
||
has no streaming ONNX parser, so the whole file must exist as one
|
||
contiguous buffer to construct a session.
|
||
2. `constructSessions` (`src/models/session.js`) loaded the encoder and
|
||
decoder sessions **concurrently** via `Promise.all`, so both full-size
|
||
JS buffers were resident at once — needlessly roughly doubling the
|
||
already-large per-file peak (on top of onnxruntime-web's own
|
||
WASM-heap `_malloc` + `HEAPU8.set` copy inside `copyFromExternalBuffer`,
|
||
made once per file at session-creation time, which is itself already
|
||
serialized across sessions in web environments).
|
||
Net: peak transient memory for Medium approached ~2-3x the ~460 MB combined
|
||
weight size — comfortably inside WASM32's 4 GiB address space, but well
|
||
past the much tighter per-tab memory budgets mobile Safari/Chrome enforce,
|
||
which respond to going over budget with an outright, uncatchable tab kill
|
||
(no JS `error` event fires — `worker.onerror` in `App.tsx` never sees it).
|
||
There was no mobile detection anywhere in the app: Medium downloaded/loaded
|
||
identically on a phone and a workstation.
|
||
**Fix (two parts):**
|
||
1. **Sequential session loading (real fix, no UX cost).** `constructSessions`
|
||
is patched (`bun patch`, `patches/@huggingface%[email protected]`,
|
||
tracked via `patchedDependencies` in `package.json` so it survives
|
||
`bun install`) to load sub-model sessions one at a time instead of via
|
||
`Promise.all`. This lets the previous file's JS buffer become garbage
|
||
before the next file is fetched, roughly halving the load-time peak for
|
||
any multi-file model (Whisper's encoder+decoder) — a strict improvement
|
||
with no downside, so it applies unconditionally, not just on mobile.
|
||
Investigated and rejected as *not* actionable: skipping the
|
||
`progress_callback`, `env.customCache`, and passing a URL string instead
|
||
of a buffer to onnxruntime-web — all three still end up materializing the
|
||
whole file, per the analysis above. Fixing that fully would require
|
||
forking onnxruntime-web itself (out of scope).
|
||
2. **Warn/confirm gate on mobile (`src/lib/mobileMemoryGuard.ts`,
|
||
`estimatePeakLoadBytes` in `src/lib/models.ts`).** Even halved, Medium's
|
||
peak (~555 MB estimated: `2 × max(encoderBytes, decoderBytes)`, reflecting
|
||
the now-sequential loading) can still exceed a constrained phone's tab
|
||
budget. `App.tsx`'s `confirmRiskyMobileLoad` gates all three ways a
|
||
browser-engine load can start — the explicit Download button, and the
|
||
lazy auto-load on file-drop / live-start (D-020) — behind
|
||
`shouldWarnBeforeLoad`: mobile UA detection (Android/iPhone/iPod, plus
|
||
iPadOS's desktop-masquerading Safari UA via touch-point count, same
|
||
UA-sniffing rationale as D-026 — no memory-pressure API exists to probe
|
||
instead) combined with `navigator.deviceMemory` when Chromium reports it,
|
||
falling back to a flat ~400 MB threshold where it doesn't (all of
|
||
iOS/Safari). A native `window.confirm()` states the estimated memory need
|
||
and the crash risk in plain language; declining aborts the load with no
|
||
state changes, accepting remembers the modelId+device combination for the
|
||
rest of the session so the same load doesn't ask twice. Desktop/laptop
|
||
browsers are never gated — no crash evidence exists there.
|
||
**Alternatives rejected:** hard-blocking Medium+ on mobile with no override
|
||
(some phones have enough RAM to be fine; the confirm lets the user judge
|
||
their own device); forking onnxruntime-web for a streaming session-creation
|
||
path (large undertaking for a runtime we don't own upstream).
|
||
**Consequence:** every model load is now faster-to-safe (sequential
|
||
sessions cost a little wall-clock time — files download one after another
|
||
instead of in parallel — in exchange for materially lower peak memory).
|
||
Mobile users choosing Medium/Large now see one extra confirmation step;
|
||
Tiny/Base/Small are unaffected (their estimated peaks stay under the
|
||
warning thresholds).
|
||
**Revisit:** if field reports show the sequential-load patch alone resolves
|
||
mobile crashes in practice, consider loosening the confirm thresholds; if
|
||
onnxruntime-web ever gains a streaming/incremental session-creation API,
|
||
drop the patch and re-measure before touching the thresholds again.
|
||
**Amendment (2026-07-29, D-028):** "a little wall-clock time" undersold the
|
||
cost — see D-028. The sequential patch no longer applies unconditionally;
|
||
`options.sequentialSessionLoad` (set by `App.tsx`, true only when
|
||
`shouldWarnBeforeLoad` is true for the load) now gates it.
|
||
|
||
## D-028 — Sequential session loading gated to actual mobile-crash risk, not unconditional
|
||
**Status:** Accepted (2026-07-29)
|
||
Reported symptom: `encoder_model_q4.onnx` (Whisper Medium) downloaded far
|
||
slower after D-027 landed than before it — reproduced on desktop too, ruling
|
||
out phone CPU/network as the cause. Measured in Chrome DevTools: ~1.6 MB/s
|
||
sustained on a connection that otherwise performs fine, downloading from
|
||
Hugging Face's Xet-backed CDN (`us.aws.cdn.hf.co/xet-bridge-us/...`).
|
||
**Cause.** D-027's `constructSessions` patch made the encoder and decoder
|
||
downloads always sequential — one HTTP request in flight at a time — to keep
|
||
both weight buffers from being resident together. On a long-RTT path (this
|
||
CDN is US-hosted), a single HTTP/2 stream typically can't fill the available
|
||
bandwidth; two concurrent requests (the pre-D-027 `Promise.all`) filled more
|
||
of it. Serializing the downloads dropped every load back to one flow's
|
||
throughput ceiling — not "a little" wall-clock cost as D-027 claimed, but
|
||
multiple times slower on exactly this kind of connection, and paid by every
|
||
user regardless of device.
|
||
**Fix.** The sequential/parallel choice is now conditional, not baked into
|
||
the patch: `constructSessions` branches on a new `options.sequentialSessionLoad`
|
||
flag (`patches/@huggingface%[email protected]`) — parallel
|
||
`Promise.all` (the original, fast behavior) when falsy/absent, the D-027
|
||
for-loop when true. `App.tsx` computes this per load via `isMobileLoadRisk`
|
||
(extracted from `confirmRiskyMobileLoad`'s existing hint/`estimatePeakLoadBytes`
|
||
logic — the same `shouldWarnBeforeLoad` check the crash-warning dialog uses)
|
||
and threads it through a new `sequentialSessionLoad` field on the worker's
|
||
`LoadMessage` (`protocol.ts`) into the `pipeline()` call
|
||
(`inference.worker.ts`). Crucially this is evaluated independently of the
|
||
confirm dialog's answer — even a load the user clicked "Continue anyway" on
|
||
still loads sequentially, since the memory risk that motivated D-027 doesn't
|
||
go away just because the user accepted it.
|
||
**Consequence:** desktop/laptop loads (and any mobile load under the
|
||
size/`deviceMemory` risk threshold) get back the full concurrent-download
|
||
speed that predates D-027; only loads that would already trigger the mobile
|
||
crash warning pay the sequential-loading wall-clock cost, which is exactly
|
||
where D-027's OOM fix is actually needed.
|
||
**Revisit:** if Hugging Face's Xet CDN behavior changes, or if a future model
|
||
in the catalog is large enough to risk mobile OOM at a size where the
|
||
confirm-dialog threshold doesn't yet trigger, re-check that the two
|
||
thresholds (`shouldWarnBeforeLoad`'s and this gate's) are meant to stay
|
||
identical.
|
||
|
||
## D-029 — Mobile crash warning was silently disabled on every phone but the lowest-end
|
||
**Status:** Accepted (2026-07-29)
|
||
Reported symptom: the D-027 mobile crash `window.confirm()` never appeared
|
||
for Whisper Medium (or any model), even on devices it's explicitly meant to
|
||
protect.
|
||
**Cause.** `shouldWarnBeforeLoad`'s `deviceMemory`-scaled branch used
|
||
`SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.5`. Checked against the actual
|
||
catalog (`estimatePeakLoadBytes`) for every `navigator.deviceMemory` value
|
||
Chromium can report:
|
||
|
||
| deviceMemory | "safe" budget (old 0.5) | Medium (556 MB) | Large v3 Turbo (762 MB) |
|
||
|---|---|---|---|
|
||
| 1 | 500 MB | warns | warns |
|
||
| 2 | 1 GB | **silent** | **silent** |
|
||
| 4 | 2 GB | **silent** | **silent** |
|
||
| 8 | 4 GB | **silent** | **silent** |
|
||
|
||
`deviceMemory` of 2 or more is ordinary today, not just flagship-tier — so
|
||
in practice the warning only ever fired on the oldest/cheapest Android
|
||
phones and on iOS (which lacks the API entirely and always used the flat
|
||
400 MB fallback, D-027). The exact mid/low-range phones the original crash
|
||
report was about — the whole reason D-027 exists — cleared the threshold and
|
||
got no warning at all.
|
||
**Fix:** `SAFE_TAB_FRACTION_OF_DEVICE_MEMORY` lowered to 0.1
|
||
(`src/lib/mobileMemoryGuard.ts`). Re-checking the same table: deviceMemory
|
||
2 and 4 now warn for Medium+ (200 MB / 400 MB budgets, both under 556 MB);
|
||
deviceMemory 6-8 (600-800 MB budgets) still don't, treating genuine
|
||
high-end devices as having real headroom. `tests/mobileMemoryGuard.test.ts`
|
||
gained a regression case pinning deviceMemory 2/4 to `true` for Medium's
|
||
peak — the exact gap this decision closes.
|
||
**Alternatives rejected:** dropping the `deviceMemory` scaling entirely in
|
||
favor of always using the flat 400 MB fallback (simpler, but throws away the
|
||
one signal that lets genuinely high-RAM devices skip an unnecessary
|
||
confirm); a hard absolute cap on top of the fraction (more robust long-term,
|
||
but two knobs to calibrate against no real data beats one — revisit if a
|
||
future model's peak lands awkwardly relative to the 0.1 fraction).
|
||
**Revisit:** this is still a guess, same as D-027's original 0.5 was — there
|
||
is no real crash-telemetry feedback loop. If field reports show 0.1 is still
|
||
too generous (or too aggressive) for some device class, adjust the constant
|
||
rather than the shape of the heuristic.
|
||
|
||
## D-030 — Error taxonomy: pattern-matched classification, not structured error types
|
||
**Status:** Accepted (2026-07-29)
|
||
Every failure path (decode, model load, transcription, a crashed worker, a
|
||
live session) used to end at `setError(rawMessage)`, rendered as one plain
|
||
string — no distinction between "this file won't decode" (try another file),
|
||
"the tab ran out of memory" (try a smaller model), a lost WebGPU device
|
||
(switch backend), or a dropped network connection mid-download (retry). The
|
||
WorkBreakdown M6 item asked for exactly those four categories.
|
||
**Shape:** `src/lib/errors.ts` exports a pure `classifyError(raw, context)`:
|
||
`context.stage` (`"decode" | "load" | "transcribe" | "worker-crash"`) plus
|
||
`context.offline` (`navigator.onLine`) disambiguate cases the message text
|
||
alone can't, then substring pattern-matching on the message sorts the rest
|
||
into `unsupported-file` / `out-of-memory` / `device-lost` / `network` /
|
||
`generic`, each with a user-facing `message`, an optional actionable
|
||
`suggestion`, and `retryable`. **Why pattern-matching, not typed errors from
|
||
each layer:** the failures cross Web Audio, Transformers.js, onnxruntime-web
|
||
(WASM and the WebGPU/WebNN JSEP EPs), and `fetch` — none of which expose a
|
||
shared error hierarchy the app controls, so there is nothing to `instanceof`
|
||
against; the only thing every layer reliably produces is an `Error` (or
|
||
DOMException) with a message string. A worker crash (`Worker.onerror`) with
|
||
an **empty** message is itself classified as out-of-memory: D-027 found that
|
||
the browser's silent OOM tab-kill fires no JS error event with details at
|
||
all, so an empty message *is* the OOM signal in that one context — the
|
||
classifier only applies that inference when `context.stage ===
|
||
"worker-crash"`, not to an empty message anywhere else.
|
||
**UI wiring (`src/App.tsx`):** the `error` state changed from a `string` to
|
||
`{ classified: ClassifiedError; retryKind?: "load" | "file" }`. `retryKind`
|
||
is resolved to an actual retry action **at render time** (`onClick={() =>
|
||
startLoad(modelId, devicePreference)}` / replaying `lastFileRef.current`
|
||
through `handleFile`), reading current component state directly — not
|
||
computed inside the worker's `message`/`error` event listeners, which are
|
||
registered once per worker lifetime (`ensureWorker`) and would otherwise
|
||
close over whichever render's `modelId`/`devicePreference` happened to be
|
||
current at registration, going stale on every later model/device switch.
|
||
**Consequence:** only `network` and `device-lost` are `retryable` (an
|
||
unsupported file or an OOM won't succeed by retrying the identical action);
|
||
the error box shows a Retry button only for those, alongside the suggestion
|
||
text. Live-mode errors (mic permission, WS disconnects) still classify
|
||
through the same function for a consistent error box, but never carry
|
||
`retryKind` — the existing Start/Stop controls are already the retry path
|
||
there.
|
||
**Alternatives rejected:** wrapping each subsystem's calls in typed error
|
||
classes at the boundary (real fix, but Transformers.js/onnxruntime-web
|
||
throw plain `Error`s from deep internals — retrofitting types there means
|
||
forking dependencies, out of scope); classifying in the worker instead of
|
||
the UI (the worker already forwards a string message over postMessage —
|
||
classifying there would just move the same pattern-matching one hop over
|
||
for no benefit, and the UI is where `context.offline`/retry actions live).
|
||
**Revisit:** if a future onnxruntime-web/Transformers.js version starts
|
||
surfacing richer structured errors (e.g. a `GPUDeviceLostInfo` object
|
||
instead of a stringified message), prefer inspecting that directly over
|
||
adding more substring patterns.
|
||
|
||
## D-031 — Docker image hardening: exclude local venvs, move Transformers.js out of the runtime image
|
||
**Status:** Accepted (2026-07-29)
|
||
WorkBreakdown's M6 "production build + `bun start` deployment check" item —
|
||
a `docker build .` from this checkout, not a fresh clone, exposed two real
|
||
bugs the documented image sizes (WorkBreakdown M5 notes: "244 MB" /
|
||
"405 MB") had already gone stale against.
|
||
**Bug 1 — local Python venvs leak into every image.** `server-py/venv` and
|
||
`server-py/.venv` (created locally to test the sidecar against a real
|
||
faster-whisper install) are invisible to `git status` — `python -m venv`
|
||
drops its own nested `.gitignore` (`*`) inside the venv directory — but
|
||
Docker does not read `.gitignore`, and `.dockerignore` had no matching
|
||
pattern. `COPY server-py ./server-py` therefore copied both venvs (858 MB
|
||
combined) into the image; measured effect on the Bun-only image: 999 MB
|
||
instead of the intended size, with a completely unused,
|
||
machine-specific Python interpreter baked in at `/app/server-py/venv` and
|
||
`/app/server-py/.venv`. **Fix:** `.dockerignore` gained `**/venv/` and
|
||
`**/.venv/`. Verified: rebuilding after the fix, `find /app/server-py`
|
||
inside the image no longer lists either directory.
|
||
**Bug 2 — `@huggingface/transformers` (and its `onnxruntime-node` +
|
||
`onnxruntime-web` dependencies, 341 MB combined) had no reason to be in the
|
||
*production runtime* image.** Grepped: the only file in `src/` that imports
|
||
`@huggingface/transformers` as a value (not a type) is
|
||
`src/worker/inference.worker.ts`, and that file is bundled once, at build
|
||
time, into the static `dist/inference.worker.js` the production server
|
||
just serves as a file (confirmed: `src/index.ts`'s `Bun.build`-based
|
||
on-the-fly bundling path — needed because Bun doesn't follow `new
|
||
Worker(...)` graphs, per M1's Architecture notes — only runs in
|
||
**development** mode; production never calls it). No server-side module
|
||
(`src/server/*`, `src/index.ts`, `src/lib/models.ts`'s only import from the
|
||
package is `import type`, erased at transpile time) touches the package at
|
||
runtime. Yet it was a `dependencies` entry, so the Dockerfile's runtime
|
||
stage (`bun install --frozen-lockfile --production`) installed it and its
|
||
full dependency tree anyway — dead weight on every deploy. **Fix:**
|
||
`@huggingface/transformers` moved to `devDependencies` in `package.json`.
|
||
The **build** stage's `bun install --frozen-lockfile` (no `--production`)
|
||
still installs devDependencies, so `scripts/build.ts`'s `Bun.build` call
|
||
(which needs the package on disk to bundle the worker) is unaffected; the
|
||
patch in `patches/@huggingface%[email protected]` still applies
|
||
there the same way. Verified: production `node_modules` in the built image
|
||
dropped from 409 MB to 7.9 MB (react/react-dom/passport only); the running
|
||
container still served `/inference.worker.js` correctly (534,714 bytes,
|
||
byte-identical to the pre-change build) and `/api/transcribe/status`
|
||
correctly reported the expected `faster-whisper`/`python3` unavailability
|
||
message — nothing at runtime needed the package.
|
||
**Net effect, measured against a real `docker build .`/`--target
|
||
with-server-engine` in this checkout:**
|
||
| Image | Before both fixes | After |
|
||
| --- | --- | --- |
|
||
| Bun-only (`runtime`) | 2.18 GB (venv leak) | ~421 MB |
|
||
| with-server-engine | not separately measured (inherits the leak) | ~7.1 GB — see below |
|
||
**A third, non-bug finding while re-measuring:** the with-server-engine
|
||
image is legitimately several GB, not the "405 MB" previously written down
|
||
— its last `RUN` layer pre-downloads both server-selectable models
|
||
(`large-v3-turbo`, `medium.en`, matching the `EngineToggle`'s hardcoded
|
||
allow-list in `App.tsx`) into `/opt/faster-whisper/models` at *build* time,
|
||
which alone measured 3.15 GB; the venv/faster-whisper/ctranslate2 install
|
||
layer is a separate 451 MB. This is intentional (comment already in the
|
||
Dockerfile: baking the weights into the image, outside `/app/data`, means
|
||
no first-request cold-download and no dependence on the data volume for
|
||
those two models) — not something to shrink — but the number that made it
|
||
into WorkBreakdown's M5 notes and README's env-var table was never updated
|
||
to match, and the README's Persistence section wrongly implied these
|
||
weights live in the `/app/data` volume (`FASTER_WHISPER_DOWNLOAD_ROOT`'s
|
||
actual default in that image is `/opt/faster-whisper/models`, not
|
||
`/app/data/faster-whisper-models`). **Fix:** README's env-var table,
|
||
Persistence section, and a new "Image sizes" subsection now state the
|
||
verified numbers and the reason for each; WorkBreakdown's M5 notes updated
|
||
to match.
|
||
**Consequence:** none of this is CI-visible — GitLab CI (`.gitlab-ci.yml`)
|
||
builds from a clean checkout that never has `server-py/venv` on disk, so
|
||
Bug 1 only bit a local `docker build .` run from a working directory that
|
||
had one (as this checkout did); the `.dockerignore` fix is still correct
|
||
defense-in-depth regardless. Bug 2 was real everywhere, CI included — every
|
||
pushed Bun-only image has been ~340 MB heavier than necessary since
|
||
`@huggingface/transformers` became a production dependency.
|
||
**Revisit:** if the with-server-engine catalog ever grows beyond the two
|
||
baked-in models, either accept runtime downloads for the extra ones (point
|
||
`FASTER_WHISPER_DOWNLOAD_ROOT` at `/app/data` for those, per the README's
|
||
override note) or extend the Dockerfile's pre-download `RUN` line —
|
||
re-measure and re-document the image size either way.
|