whisper_web/Decisions.md

Decisions

Lightweight ADR log. One entry per decision that shapes the codebase. Statuses: Accepted (act on it), Proposed (default choice, revisit before building on it), Superseded (kept for history, points to replacement).

Add new entries at the bottom with the next D-### id. Never rewrite an Accepted entry to mean something else — supersede it.

---

D-001 — Bun as the only runtime and toolchain

Status: Accepted (mandated by CLAUDE.md) Bun replaces Node, vite/webpack (via bun build + HTML imports), jest/vitest (bun test), express (Bun.serve), better-sqlite3 (bun:sqlite), and dotenv. Consequence: no vite plugin ecosystem; PWA assets (service worker, manifest) are wired by hand instead of via a plugin.

D-002 — Transformers.js (@huggingface/transformers) for all Whisper inference

Status: Accepted for browser inference; the server side is superseded by D-013 (faster-whisper). Same library in the browser (WebGPU/WASM) and on the server (ONNX Runtime under Bun). Why: one model catalog, one pipeline API, no native whisper.cpp build step. Trade-off: server CPU inference is slower than whisper.cpp; acceptable — the server engine is a convenience for logged-in users, not the primary path.

D-003 — Inference runs in a dedicated Web Worker

Status: Accepted Model load + transcription block for seconds-to-minutes; running them on the main thread would freeze the UI. Communication via a small typed postMessage protocol (src/worker/protocol.ts) shared between UI and worker.

D-004 — WebGPU with automatic WASM fallback

Status: Accepted (amended by D-017 — "auto" now probes WebNN first) Detect via navigator.gpu?.requestAdapter() inside the worker; pass device: "webgpu" | "wasm" to the pipeline. The active backend is surfaced in the UI but users don't have to configure anything. Encoder-heavy Whisper benefits ~10-40× from WebGPU where available (Chromium); WASM keeps Firefox/Safari working.

D-005 — Model weights cached via the browser Cache API (Transformers.js default)

Status: Accepted Transformers.js already persists Hub downloads in the Cache API (useBrowserCache). We do not re-cache weights in the service worker — the SW precaches only the app shell. Consequence: offline = installed PWA + at least one previously loaded model; the UI must show per-model "downloaded / not downloaded" state (query caches for hub URLs) so users can prepare for offline use.

D-006 — Quantized model catalog, curated not exhaustive

Status: Proposed Expose a curated list (src/lib/models.ts): whisper-tiny(.en), base(.en), small(.en), medium(.en), large-v3-turbo, each with sensible dtype defaults (e.g. q4/q8 on WASM, fp16/q4 on WebGPU) and a size estimate shown before download. Why not free-form model ids: predictable UX, tested combinations, honest size/VRAM warnings. Revisit if users ask for arbitrary Hub ids.

D-007 — PWA: hand-rolled service worker, app-shell precache

Status: Accepted No Workbox/vite-plugin-pwa (conflicts with D-001 tooling). bun build emits hashed assets to dist/; a small build step writes the precache manifest into sw.ts. Strategy: precache app shell, cache-first for static assets, network-only for /api/*.

D-008 — Passport.js via a connect-compat adapter over Bun.serve

Status: Accepted (Passport mandated by CLAUDE.md; adapter is the how) Passport expects connect/express-style (req, res, next) and req.login(). We are not allowed to use express (D-001), so src/server/auth.ts provides a thin adapter: wrap Bun's Request in a minimal req/res shim, invoke the passport authenticate chain, translate the outcome back to a Response. Local (username/password) strategy first; other strategies plug into the same adapter later. Alternative rejected: dropping Passport for hand-rolled auth — CLAUDE.md names Passport explicitly.

D-009 — Sessions + users in bun:sqlite

Status: Accepted Single-file SQLite DB (path from env, default ./data/app.db). Tables: users (id, username, password_hash — Argon2 via Bun.password), sessions (id, user_id, expires_at). Cookie: HttpOnly, SameSite=Lax, Secure in production. No Redis/Postgres until there's a scaling reason (would be Bun.redis / Bun.sql).

D-010 — Engine abstraction: BrowserEngine and ServerEngine behind one interface

Status: Accepted src/lib/engine.ts defines Engine { load(model), transcribe(audio), onProgress }. The UI never branches on engine type beyond the toggle. Keeps the server engine (milestone M5) from leaking into UI code built in M1–M4.

D-011 — Server engine transport: streaming HTTP response first, WebSocket if needed

Status: Accepted (implemented in M5, 2026-07-10) POST /api/transcribe returns a streamed (chunked JSON-lines) response with progress and partial transcripts. Simpler than WebSockets (no connection lifecycle, works with plain fetch + ReadableStream). Switch to Bun.serve websockets only if we need bidirectional control (e.g. cancel mid-run) that abort signals can't cover. As implemented: application/x-ndjson of segment* then result | error lines (wire types + guards shared client/server via src/lib/engine.ts); the request body is raw float32 mono 16 kHz PCM — the decoded form the browser already has — so the server never decodes containers. Cancel-mid-run needed no WebSocket: fetch's AbortSignal reaches request.signal, which forwards a cancel to the sidecar (it checks between segments).

D-012 — Audio decoding on the main thread via Web Audio API

Status: Accepted decodeAudioData (handles most audio and audio-in-video containers) → mono downmix → resample to 16 kHz → transfer Float32Array to worker (zero-copy). No ffmpeg.wasm unless we hit real container-support gaps; note in src/lib/audio.ts if we do.

D-013 — Server engine runs faster-whisper (Python sidecar)

Status: Accepted (user requirement, 2026-07-10; supersedes the server half of D-002) Server-side inference (M5) uses faster-whisper (CTranslate2) instead of Transformers.js under Bun. Browser inference is untouched — faster-whisper cannot run in a browser (checked 2026-07): it is Python over native C++ CTranslate2, which has no WASM build (OpenNMT/CTranslate2#528 open since 2021, no progress) and relies on native SIMD/threading kernels. So D-002 stays Accepted for the browser, where Transformers.js remains the engine. Shape: Bun stays the only web runtime. src/server/transcribe.ts talks to a small Python sidecar (server-py/, faster-whisper behind a minimal stdio or localhost-HTTP interface) that Bun.spawn starts and supervises. The POST /api/transcribe contract (D-011: streamed JSON-lines) is unchanged — faster-whisper's segment generator maps naturally onto it. Consequence: the "one inference codebase" invariant narrows to "one browser inference codebase + one shared model catalog"; src/lib/models.ts needs a mapping from catalog ids to faster-whisper/CTranslate2 model names (Systran/faster-whisper-*); deployment gains a Python ≥3.9 dependency (server engine only — the anonymous/browser path still deploys with Bun alone, and the app must degrade gracefully when the sidecar is absent). Why: CTranslate2 CPU inference is several times faster and more memory-efficient than ONNX Runtime under Bun; the server engine becomes genuinely useful rather than a slow convenience.

D-014 — WASM sessions run at graphOptimizationLevel "basic"

Status: Accepted (bug workaround, 2026-07-10) On the WASM backend, the inference worker passes session_options: { graphOptimizationLevel: "basic" } to the Transformers.js pipeline. The onnxruntime-web build bundled with Transformers.js 4.2 (1.26.0-dev.20260416) has a broken level-2 QDQ graph transform (qdq_actions.cc TransposeDQWeightsForMatMulNBits) that fails session creation for the 8-bit Whisper decoders (_quantized/_int8/_uint8), whose tied embed_tokens weight is both gathered (embedding) and matmul'd (lm_head): "Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale". Reproduced outside the browser against decoder_model_merged_quantized.onnx of onnx-community/whisper-tiny; levels "all"/"extended" fail, "basic"/ "disabled" load fine, and onnxruntime-node 1.24.3 loads the same file at every level (regression in the ort-web dev build, not the model). Scope: WASM only — WebGPU keeps the default level. The q4 conversions use native MatMulNBits ops (no QDQ), so skipping level-2 costs the larger WASM models nothing. Revisit: when Transformers.js bumps its onnxruntime-web dependency past 1.26.0-dev.20260416, retest q8-on-WASM at the default level and drop this.

D-015 — Runtime feature flags via server env vars; first flag gates auth

Status: Accepted (2026-07-10) Feature flags are read from the server's process environment at startup (src/server/flags.ts, envFlag(): unset/empty → the flag's default; false/0/no/off any case → off; anything else → on). Runtime-only by design: one Docker image serves every configuration — flags are flipped in docker-compose.yml / docker run -e, never at build time, so the client bundle stays identical across deployments. First flag: AUTH_ENABLED (default on) gates the M4 login/register feature. Off, the server 404s every /api/auth/* route. The client is never told separately: fetchAuthStatus (the GET /api/auth/me probe useAuth already makes on mount) reads 404 as "feature off" and <AuthForms /> renders nothing. The Sign in button therefore only appears after the probe confirms the endpoints exist — a disabled deployment never flashes auth UI, and an offline/unreachable server also hides it (signing in would be impossible anyway), preserving the anonymous-first invariant (no error surfaced). Consequence: any future flag that must reach the browser needs a server-derived signal (an endpoint the client can probe, like /me here) — BUN_PUBLIC_* build-time inlining is not used for feature flags.

D-016 — Live microphone transcription via segment re-transcription (browser engine)

Status: Accepted (2026-07-10; promotes "live transcription" out of the WorkBreakdown out-of-scope list → milestone M7) Live (microphone) transcription runs on the browser engine only — same privacy story as file uploads: audio never touches the network, works for anonymous users, works offline. Whisper is not a streaming model (it consumes up-to-30 s windows), so "live" is approximated by a segment loop:

• Capture (main thread): getUserMedia → AudioContext at the device's native rate → an AudioWorklet chunker posts ~100 ms Float32 frames to the main thread, which resamples them to 16 kHz with the pure, stateful StreamingResampler (windowed-sinc low-pass; resampleLinear has no anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above 8 kHz into the speech band and measurably hurt accuracy — see tests/resampler.test.ts). The worklet module is loaded from a Blob URL built from an inline code string — no extra served file, no build/service-worker changes, and it keeps working offline. Where audioWorklet/Blob modules are unavailable (older Safari), fall back to a ScriptProcessorNode. The context is not forced to 16 kHz because Firefox cannot connect a MediaStreamAudioSourceNode across sample rates.
• Segmentation (pure, unit-tested): a simple RMS energy gate (src/lib/live.ts) splits speech into segments: a segment starts at speech onset (with a short pre-roll), commits after a run of trailing silence or at a max length (under Whisper's 30 s window), and pure-silence audio is dropped entirely — Whisper hallucinates on silence, so never transcribe it. Two accuracy refinements (2026-07-14): the gate has hysteresis (once a segment is open, frames stay "speech" down to half the onset threshold, so quiet phonemes don't fragment utterances), and a max-length commit is a soft cut — it splits at the last sub-threshold frame in the trailing window instead of mid-word at the hard cap, seeding the next segment with the remainder (a mid-word cut corrupts the boundary word in both segments). Dropped (near-silent) segments also clear any tentative text they produced, so a hallucinated tentative pass can neither linger on screen nor be adopted as the next segment's provisional text. Production tuning lives in LIVE_TUNING (src/lib/live.ts), shared by src/App.tsx and the accuracy harness (tests/live-accuracy.test.ts — tone-word oracle engine + WER, so segmentation/scheduling/resampling accuracy is regression-tested under bun test without a model). No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
• Inference (existing worker, serialized): the current segment is re-transcribed through the existing inference worker/pipeline on a rolling basis (tentative text, replaced each pass); a committed segment gets one final pass and its text freezes. At most one inference is in flight; final passes take priority over tentative refreshes, which are throttled and always use the latest audio snapshot (intermediate snapshots are skipped, so the loop degrades gracefully when inference is slower than real time).
• Protocol: UI↔worker messages gain an optional requestId echoed on partial/result/error, so the live controller can correlate responses and coexist with the file flow. The file flow keeps working without ids.

Alternatives rejected: streaming into the worker with a custom chunk-stitching decoder (large fork of Transformers.js internals for marginal latency); MediaRecorder + re-decoding compressed blobs (adds codec latency and decodeAudioData round-trips for no benefit over raw PCM taps). Consequence: live quality/latency is bounded by model speed on the user's device — the UI must keep tentative text visibly distinct from committed text. Server-engine live input (WebSocket streaming to faster-whisper) is a separate future decision; D-011's "WebSocket if needed" note is the hook.

D-017 — "Auto" device order: WebNN → WebGPU → WASM

Status: Accepted (2026-07-14; amends D-004, which only knew WebGPU → WASM) With WebNN support added (commit "WebNN support initial"), the "auto" device preference resolves in priority order WebNN → WebGPU → WASM: the worker probes navigator.ml (WebNN) first, then navigator.gpu.requestAdapter() (WebGPU), and always keeps WASM as the last resort. The probe/ordering policy is pure and unit-tested (deviceAttempts in src/worker/protocol.ts, tests/protocol.test.ts); the worker only supplies the capability probe. Pipeline-creation failures walk the same chain — an auto-detected WebNN that fails still tries WebGPU before WASM. An explicit "webnn" or "webgpu" preference deliberately falls back straight to WASM (no detour through the other accelerator — the user asked for a specific backend, so honor it or take the universal fallback). The selected backend stays user-visible: the worker's {status:"ready", device} message drives a "Running on …" badge next to the Backend select (ModelPicker) and the engine line in the footer (D-004's "surfaced in the UI" requirement).