whisper_web/docs/Architecture.md

Architecture

Companion to ../PRODUCT.md (product spec), ../AGENTS.md (agent & tooling rules) and Decisions.md (why things are the way they are). This file describes how the system is structured. Update it whenever the structure changes.

1. System overview

Browser-first speech-to-text app. Whisper inference runs in the browser by default via Transformers.js (@huggingface/transformers); no audio ever leaves the device for anonymous users. Logged-in users may optionally route inference to the server.

┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ LiveControls       (microphone start/stop — D-016)          │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebNN → WebGPU → WASM fallback            │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: faster-whisper (CTranslate2) Python sidecar,  │
│                    spawned & supervised by Bun (D-013)            │
└──────────────────────────────────────────────────────────────────┘

2. Runtime & tooling

• Runtime / bundler / test runner: Bun only (bun --hot src/index.ts for dev, bun build src/index.html for production, bun test). No vite/webpack/jest.
• Frontend: React 19, mounted from src/index.html via Bun HTML imports.
• Server: Bun.serve() with the routes option. No express.
• DB: bun:sqlite. No ORM planned initially.

3. Directory layout (target)

src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, TranscriptionOptions, VocabularyHint, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    live.ts                # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
    mic.ts                 # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
    hint.ts                # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-WS2a)
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
    prompt.ts              # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-WS2a)
  pwa/
    sw.ts                  # service worker (app-shell precache, /ort runtime cache)
    precache.ts            # placeholder-injection helpers shared by build + dev serve
    register.ts            # SW registration + update-toast flow
    manifest.webmanifest
    icons/                 # generated by scripts/generate-icons.ts (checked in; `bun run icons` regenerates)
  fonts/                   # self-hosted Inter, copied from node_modules by scripts/fonts.ts (gitignored, D-037)
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint (bridges the sidecar, M5)
    live.ts                # server-engine live mic over WebSocket (D-019, M7)
server-py/                 # faster-whisper sidecar (Python, D-013; M5)
tests/                     # bun test

(Current state: M0–M6 built — browser transcription with the curated model catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching, language/task options, the installable offline PWA (pwa/, scripts/build.ts production build, dist/ serving in src/index.ts; browser-level offline behavior pending the manual run in tests/offline.md), M4 accounts (server/db.ts, server/auth.ts, /api/auth/*, AuthForms), and the M5 server engine end to end — server-py/sidecar.py, server/transcribe.ts (supervisor + /api/transcribe + /status), lib/engine.ts (both engines, D-010), EngineToggle, and the catalog mapping in lib/models.ts — with graceful degradation when Python/faster-whisper is absent. M7 live microphone transcription is built for both engines: browser (lib/live.ts, lib/mic.ts, LiveControls, D-016) and server (server/live.ts, D-019); in-browser manual verification of both is still pending per tests/live.md (no real browser in the dev environment). See WorkBreakdown.md for the itemized checklist.)

4. Key flows

4.1 Browser transcription (anonymous, default)

1. User picks a model in ModelPicker (curated catalog from lib/models.ts, with size estimates and a Cache-API-backed "downloaded" badge). Selection alone downloads nothing (D-020): the UI posts {type:"load", modelId, device} when a browser-engine transcription starts (file drop / live start) or when the user clicks the explicit Download button on the selected model; server-engine runs never post a load. device is a preference ("auto" | "webnn" | "webgpu" | "wasm", default "auto"). An explicit dtype override is expressible in the message but normally omitted.
2. Worker resolves the device (D-004, D-017): for "auto" it probes navigator.ml (WebNN) first, then navigator.gpu?.requestAdapter() (WebGPU), and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the picked device throws, it retries down the same chain (WASM always last). An explicit "webnn"/"webgpu" preference falls back straight to WASM. Two D-025 refinements: WebGPU is skipped up front when the dtype it would use needs fp16 but the adapter lacks shader-f16 (Firefox), and a WASM fallback after a failed accelerator attempt does not run in-place — the failed attempt pinned ort's thread pool at 1 — but posts load-restart, and the UI respawns the worker with an explicit wasm load (transcribe posts wait on a ready gate in App.tsx, so none are lost). It then resolves the catalog's dtype default for the device actually used and creates the Transformers.js automatic-speech-recognition pipeline. Download progress events are forwarded to the UI; the terminal {status:"ready", device} message reports the backend actually selected, which the UI surfaces. Loading a different model/device disposes the old pipeline (pipeline.dispose()) before creating the new one.
3. Weights are fetched from the Hugging Face Hub once; Transformers.js stores them in the browser Cache API, so subsequent loads (including offline) hit cache (lib/modelCache.ts reads that cache for the downloaded badges).
4. User drops an audio/video file. Main thread decodes it with AudioContext.decodeAudioData, downmixes to mono, resamples to 16 kHz, transfers the Float32Array to the worker. For multilingual models the transcribe message also carries language (or auto-detect) and task (transcribe | translate). For every model it may carry prompt: the optional "Names and terms" hint (VocabularyHint, stored in localStorage by lib/hint.ts), raw as typed (D-WS2a).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results back via {type:"partial"} messages; final result as {type:"result"}. A hint becomes <|startofprev|> hint. once per pass (worker/prompt.ts, cut to Whisper's 223-token prompt budget at a whole word) and is prepended to the decoder input of every 30 s window by a wrapper around the model's generate(), which also strips it from the output again (D-WS2a). Live passes (4.1b) carry the same prompt.

4.1b Live microphone transcription (anonymous, browser engine — D-016)

1. LiveControls starts a session: getUserMedia → AudioContext at the device's native rate → AudioWorklet chunker (module from a Blob URL; ScriptProcessorNode fallback) posts ~100 ms Float32 frames; the main thread resamples each to 16 kHz (lib/mic.ts).
2. LiveTranscriber (lib/live.ts, pure) feeds frames through an RMS energy gate: segments start at speech onset (short pre-roll), commit on trailing silence or at ~25 s; pure silence is never transcribed.
3. The current segment is re-transcribed through the normal inference worker on a rolling basis (throttled, single-flight, latest-snapshot-wins) as tentative text; a committed segment gets one final pass and freezes. When inference runs slower than speech, queued committed segments are batched into merged final passes (one Whisper window each, D-024) so the backlog stays bounded. requestId on the protocol messages correlates responses.
4. Stopping flushes the segmenter, drains pending final passes (merged, with a "N remaining segments" countdown in the UI), and yields a normal transcript (text + absolute-timestamped chunks) in TranscriptView. Everything runs on-device; works offline like the file flow.
5. Server-engine live (logged in, D-019). Same LiveControls UI, but App.tsx opens a GET /api/transcribe/live WebSocket instead of running the segmenter locally: the browser only captures/resamples mic audio (lib/mic.ts) and streams raw PCM frames over the socket; the same pure LiveTranscriber/segmenter (lib/live.ts) runs server-side (server/live.ts), writing each segment to a temp file and bridging it to the faster-whisper sidecar (D-013) the same way POST /api/transcribe does. {type:"update"} frames carry committed/tentative text back; {type:"stop"} drains and closes with a {type:"result"} frame. Selected only when signed in with the server engine available (EngineToggle); anonymous/offline users always get the browser-engine path above.

4.2 Offline (PWA)

• Service worker (pwa/sw.ts, hand-rolled — D-007) precaches the built app shell (hashed JS/CSS, /, worker bundle, manifest, icons, /fonts/* Inter subsets) on install into whisper-shell-<version>, cache-first thereafter; unknown navigations fall back to the cached shell. scripts/build.ts injects the precache list + a content-hash version into dist/sw.js via pwa/precache.ts; the dev server injects an empty list ("dev"), making the dev shell network-first so HMR works.
• onnxruntime's .wasm/.mjs binaries are served from our origin at /ort/* (never the jsdelivr CDN default) and runtime-cached cache-first into whisper-runtime-<version> — they are required for offline inference but too large to precache eagerly.
• Model weights live in the Cache API (transformers-cache) via Transformers.js — independent of the SW, which never touches that cache (D-005). /api/* is network-only.
• Updates: pwa/register.ts surfaces a waiting SW as a toast; accepting posts SKIP_WAITING, activation drops old whisper-* caches, and the page reloads.
• Result: with the app installed and at least one model previously loaded, the full transcription flow works with no network and with our server down (manual verification procedure: tests/offline.md).

4.3 Server engine (logged-in only)

1. EngineToggle is visible only when a session exists.
2. ServerEngine.transcribe() uploads the decoded (or original) audio to POST /api/transcribe with the chosen model id.
3. Server inference runs on faster-whisper (CTranslate2) in a Python sidecar (server-py/) that Bun spawns and supervises (D-013; faster-whisper has no browser/WASM build, so it is server-only — the browser stays on Transformers.js). src/server/transcribe.ts bridges the sidecar's segment stream to the streamed JSON-lines HTTP response (D-011). language, task and the vocabulary hint (prompt) travel as query params (also on the live WebSocket) and are validated by parseLanguageAndTask (D-032, D-WS2a); the sidecar cuts the hint with the same rule as the browser and passes it to faster-whisper (D-WS2a). The shared catalog maps model ids to Systran/faster-whisper-* CTranslate2 conversions. If the sidecar is absent, the server engine reports unavailable; the browser/anonymous flow is unaffected.
4. Both engines implement the same Engine interface in src/lib/engine.ts, so the UI is engine-agnostic.

4.4 Auth

• Passport.js local strategy behind a thin adapter that bridges Bun's Request/Response to Passport's connect-style API (see Decisions.md D-008).
• Sessions in SQLite; session id in an HttpOnly cookie.
• Anonymous users are never blocked from browser inference — auth only unlocks the server engine.

5. Invariants

• Privacy: browser-engine audio and transcripts never touch the network.
• Anonymous-first: every feature except the server engine works logged out.
• Offline-first: app shell + previously-downloaded models must work offline.
• One browser inference codebase: Transformers.js in the browser; the server engine runs faster-whisper (D-013). The model catalog stays shared across both.
• Worker isolation: inference never runs on the main thread.