whisper_web/Architecture.md

# Architecture

> Companion to `CLAUDE.md` (product spec & tooling rules) and `Decisions.md` (why things
> are the way they are). This file describes **how the system is structured**. Update it
> whenever the structure changes.

## 1. System overview

Browser-first speech-to-text app. Whisper inference runs **in the browser by default**
via [Transformers.js](https://github.com/huggingface/transformers)
(`@huggingface/transformers`); no audio ever leaves the device for anonymous users.
Logged-in users may optionally route inference to the server.

```
┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebGPU → WASM fallback                    │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: @huggingface/transformers (Node/Bun runtime)  │
└──────────────────────────────────────────────────────────────────┘
```

## 2. Runtime & tooling

- **Runtime / bundler / test runner:** Bun only (`bun --hot src/index.ts` for dev,
  `bun build src/index.html` for production, `bun test`). No vite/webpack/jest.
- **Frontend:** React 19, mounted from `src/index.html` via Bun HTML imports.
- **Server:** `Bun.serve()` with the `routes` option. No express.
- **DB:** `bun:sqlite`. No ORM planned initially.

## 3. Directory layout (target)

```
src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, TranscriptionOptions, FileDropzone, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
  pwa/
    sw.ts                  # service worker (app-shell precache)
    manifest.webmanifest
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint
tests/                     # bun test
```

(Current state: M0–M2 built — browser transcription with the curated model
catalog, WebGPU→WASM auto-selection, model switching, and language/task options.
`lib/engine.ts`, `pwa/`, and `server/` are TODO stubs for M3–M5; see
`WorkBreakdown.md`.)

## 4. Key flows

### 4.1 Browser transcription (anonymous, default)
1. User picks a model in `ModelPicker` (curated catalog from `lib/models.ts`, with
   size estimates and a Cache-API-backed "downloaded" badge); UI posts
   `{type:"load", modelId, device}` where `device` is a preference
   (`"auto" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype` override
   is expressible in the message but normally omitted.
2. Worker resolves the device: `navigator.gpu?.requestAdapter()` → `"webgpu"` if an
   adapter exists (and pipeline creation succeeds — otherwise it retries on WASM),
   else `"wasm"` (D-004). It then resolves the catalog's dtype default for that
   device and creates the Transformers.js `automatic-speech-recognition` pipeline.
   Download progress events are forwarded to the UI; the terminal
   `{status:"ready", device}` message reports the backend actually selected, which
   the UI surfaces. Loading a different model/device disposes the old pipeline
   (`pipeline.dispose()`) before creating the new one.
3. Weights are fetched from the Hugging Face Hub **once**; Transformers.js stores them
   in the browser Cache API, so subsequent loads (including offline) hit cache
   (`lib/modelCache.ts` reads that cache for the downloaded badges).
4. User drops an audio/video file. Main thread decodes it with
   `AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
   transfers the `Float32Array` to the worker. For multilingual models the
   `transcribe` message also carries `language` (or auto-detect) and `task`
   (transcribe | translate).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
   back via `{type:"partial"}` messages; final result as `{type:"result"}`.

### 4.2 Offline (PWA)
- Service worker precaches the built app shell (HTML/JS/CSS/icons) on install.
- Model weights live in the Cache API via Transformers.js — independent of the SW.
- Result: with the app installed and at least one model previously loaded, the full
  transcription flow works with no network and with our server down.

### 4.3 Server engine (logged-in only)
1. `EngineToggle` is visible only when a session exists.
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
   `POST /api/transcribe` with the chosen model id.
3. Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU),
   streams progress/results back (chunked response or WebSocket — see Decisions.md
   D-011 once settled).
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
   UI is engine-agnostic.

### 4.4 Auth
- Passport.js **local strategy** behind a thin adapter that bridges Bun's
  `Request`/`Response` to Passport's connect-style API (see Decisions.md D-008).
- Sessions in SQLite; session id in an HttpOnly cookie.
- Anonymous users are never blocked from browser inference — auth only unlocks the
  server engine.

## 5. Invariants

- **Privacy:** browser-engine audio and transcripts never touch the network.
- **Anonymous-first:** every feature except the server engine works logged out.
- **Offline-first:** app shell + previously-downloaded models must work offline.
- **One inference codebase:** Transformers.js on both sides; model catalog shared.
- **Worker isolation:** inference never runs on the main thread.