whisper_web/Architecture.md

Architecture

Companion to CLAUDE.md (product spec & tooling rules) and Decisions.md (why things are the way they are). This file describes how the system is structured. Update it whenever the structure changes.

1. System overview

Browser-first speech-to-text app. Whisper inference runs in the browser by default via Transformers.js (@huggingface/transformers); no audio ever leaves the device for anonymous users. Logged-in users may optionally route inference to the server.

┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebGPU → WASM fallback                    │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: @huggingface/transformers (Node/Bun runtime)  │
└──────────────────────────────────────────────────────────────────┘

2. Runtime & tooling

• Runtime / bundler / test runner: Bun only (bun --hot src/index.ts for dev, bun build src/index.html for production, bun test). No vite/webpack/jest.
• Frontend: React 19, mounted from src/index.html via Bun HTML imports.
• Server: Bun.serve() with the routes option. No express.
• DB: bun:sqlite. No ORM planned initially.

3. Directory layout (target)

src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, TranscriptionOptions, FileDropzone, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
  pwa/
    sw.ts                  # service worker (app-shell precache)
    manifest.webmanifest
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint
tests/                     # bun test

(Current state: M0–M2 built — browser transcription with the curated model catalog, WebGPU→WASM auto-selection, model switching, and language/task options. lib/engine.ts, pwa/, and server/ are TODO stubs for M3–M5; see WorkBreakdown.md.)

4. Key flows

4.1 Browser transcription (anonymous, default)

1. User picks a model in ModelPicker (curated catalog from lib/models.ts, with size estimates and a Cache-API-backed "downloaded" badge); UI posts {type:"load", modelId, device} where device is a preference ("auto" | "webgpu" | "wasm", default "auto"). An explicit dtype override is expressible in the message but normally omitted.
2. Worker resolves the device: navigator.gpu?.requestAdapter() → "webgpu" if an adapter exists (and pipeline creation succeeds — otherwise it retries on WASM), else "wasm" (D-004). It then resolves the catalog's dtype default for that device and creates the Transformers.js automatic-speech-recognition pipeline. Download progress events are forwarded to the UI; the terminal {status:"ready", device} message reports the backend actually selected, which the UI surfaces. Loading a different model/device disposes the old pipeline (pipeline.dispose()) before creating the new one.
3. Weights are fetched from the Hugging Face Hub once; Transformers.js stores them in the browser Cache API, so subsequent loads (including offline) hit cache (lib/modelCache.ts reads that cache for the downloaded badges).
4. User drops an audio/video file. Main thread decodes it with AudioContext.decodeAudioData, downmixes to mono, resamples to 16 kHz, transfers the Float32Array to the worker. For multilingual models the transcribe message also carries language (or auto-detect) and task (transcribe | translate).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results back via {type:"partial"} messages; final result as {type:"result"}.

4.2 Offline (PWA)

• Service worker precaches the built app shell (HTML/JS/CSS/icons) on install.
• Model weights live in the Cache API via Transformers.js — independent of the SW.
• Result: with the app installed and at least one model previously loaded, the full transcription flow works with no network and with our server down.

4.3 Server engine (logged-in only)

1. EngineToggle is visible only when a session exists.
2. ServerEngine.transcribe() uploads the decoded (or original) audio to POST /api/transcribe with the chosen model id.
3. Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU), streams progress/results back (chunked response or WebSocket — see Decisions.md D-011 once settled).
4. Both engines implement the same Engine interface in src/lib/engine.ts, so the UI is engine-agnostic.

4.4 Auth

• Passport.js local strategy behind a thin adapter that bridges Bun's Request/Response to Passport's connect-style API (see Decisions.md D-008).
• Sessions in SQLite; session id in an HttpOnly cookie.
• Anonymous users are never blocked from browser inference — auth only unlocks the server engine.

5. Invariants

• Privacy: browser-engine audio and transcripts never touch the network.
• Anonymous-first: every feature except the server engine works logged out.
• Offline-first: app shell + previously-downloaded models must work offline.
• One inference codebase: Transformers.js on both sides; model catalog shared.
• Worker isolation: inference never runs on the main thread.