# Architecture

> Companion to `../PRODUCT.md` (product spec), `../AGENTS.md` (agent & tooling rules) and
> `Decisions.md` (why things are the way they are). This file describes **how the system is structured**. Update it
> whenever the structure changes.

## 1. System overview

Browser-first speech-to-text app. Whisper inference runs **in the browser by default**
via [Transformers.js](https://github.com/huggingface/transformers)
(`@huggingface/transformers`); no audio ever leaves the device for anonymous users.
Logged-in users may optionally route inference to the server.

```
┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ LiveControls       (microphone start/stop — D-016)          │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebNN → WebGPU → WASM fallback            │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Translate Web Worker (D-WS4b)                                   │
│   └─ Transformers.js Opus-MT (MarianMT), WASM, on user request   │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /translate.worker.js → translation worker bundle (same)     │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: faster-whisper (CTranslate2) Python sidecar,  │
│                    spawned & supervised by Bun (D-013)            │
└──────────────────────────────────────────────────────────────────┘
```

## 2. Runtime & tooling

- **Runtime / bundler / test runner:** Bun only (`bun --hot src/index.ts` for dev,
  `bun build src/index.html` for production, `bun test`). No vite/webpack/jest.
- **Frontend:** React 19, mounted from `src/index.html` via Bun HTML imports.
- **Server:** `Bun.serve()` with the `routes` option. No express.
- **DB:** `bun:sqlite`. No ORM planned initially.

## 3. Directory layout (target)

```
src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, TranscriptionOptions, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    live.ts                # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
    mic.ts                 # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
    models.ts              # model catalog: ids, sizes, dtype defaults per device, languages
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
    translation.ts         # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-WS4b)
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
    translate.worker.ts    # transcript translation (Opus-MT), its own worker + ort instance (D-WS4b)
    translateProtocol.ts   # typed messages for the translate worker
    opusMt.ts              # Opus-MT loader shared by the worker, bench script and real-model test
  pwa/
    sw.ts                  # service worker (app-shell precache, /ort runtime cache)
    precache.ts            # placeholder-injection helpers shared by build + dev serve
    register.ts            # SW registration + update-toast flow
    manifest.webmanifest
    icons/                 # generated by scripts/generate-icons.ts (checked in; `bun run icons` regenerates)
  fonts/                   # self-hosted Inter, copied from node_modules by scripts/fonts.ts (gitignored, D-037)
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint (bridges the sidecar, M5)
    live.ts                # server-engine live mic over WebSocket (D-019, M7)
server-py/                 # faster-whisper sidecar (Python, D-013; M5)
tests/                     # bun test
```

(Current state: M0–M6 built — browser transcription with the curated model
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
language/task options, the installable offline PWA (`pwa/`,
`scripts/build.ts` production build, `dist/` serving in `src/index.ts`;
browser-level offline behavior pending the manual run in `tests/offline.md`),
M4 accounts (`server/db.ts`, `server/auth.ts`, `/api/auth/*`, `AuthForms`),
and the M5 server engine end to end — `server-py/sidecar.py`,
`server/transcribe.ts` (supervisor + `/api/transcribe` + `/status`),
`lib/engine.ts` (both engines, D-010), `EngineToggle`, and the catalog
mapping in `lib/models.ts` — with graceful degradation when
Python/faster-whisper is absent. M7 live microphone transcription is built
for both engines: browser (`lib/live.ts`, `lib/mic.ts`, `LiveControls`,
D-016) and server (`server/live.ts`, D-019); in-browser manual verification
of both is still pending per `tests/live.md` (no real browser in the dev
environment). See `WorkBreakdown.md` for the itemized checklist.)

## 4. Key flows

### 4.1 Browser transcription (anonymous, default)
1. User picks a model in `ModelPicker` (curated catalog from `lib/models.ts`, with
   size estimates and a Cache-API-backed "downloaded" badge). Selection alone
   downloads nothing (D-020): the UI posts `{type:"load", modelId, device}` when a
   browser-engine transcription starts (file drop / live start) or when the user
   clicks the explicit **Download** button on the selected model; server-engine
   runs never post a load. `device` is a preference
   (`"auto" | "webnn" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype`
   override is expressible in the message but normally omitted.
2. Worker resolves the device (D-004, D-017): for `"auto"` it probes
   `navigator.ml` (WebNN) first, then `navigator.gpu?.requestAdapter()` (WebGPU),
   and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the
   picked device throws, it retries down the same chain (WASM always last). An
   explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM.
   Two D-025 refinements: WebGPU is skipped up front when the dtype it would
   use needs fp16 but the adapter lacks `shader-f16` (Firefox), and a WASM
   fallback *after* a failed accelerator attempt does not run in-place — the
   failed attempt pinned ort's thread pool at 1 — but posts `load-restart`,
   and the UI respawns the worker with an explicit `wasm` load (transcribe
   posts wait on a ready gate in `App.tsx`, so none are lost). It then
   resolves the catalog's dtype default for the device actually used and creates
   the Transformers.js `automatic-speech-recognition` pipeline.
   Download progress events are forwarded to the UI; the terminal
   `{status:"ready", device}` message reports the backend actually selected, which
   the UI surfaces. Loading a different model/device disposes the old pipeline
   (`pipeline.dispose()`) before creating the new one.
3. Weights are fetched from the Hugging Face Hub **once**; Transformers.js stores them
   in the browser Cache API, so subsequent loads (including offline) hit cache
   (`lib/modelCache.ts` reads that cache for the downloaded badges).
4. User drops an audio/video file. Main thread decodes it with
   `AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
   transfers the `Float32Array` to the worker. For multilingual models the
   `transcribe` message also carries `language` (or auto-detect) and `task`
   (transcribe | translate — translate only for models with `canTranslate`, D-WS4a).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
   back via `{type:"partial"}` messages; final result as `{type:"result"}`.

### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
   device's native rate → AudioWorklet chunker (module from a Blob URL;
   `ScriptProcessorNode` fallback) posts ~100 ms Float32 frames; the main
   thread resamples each to 16 kHz (`lib/mic.ts`).
2. `LiveTranscriber` (`lib/live.ts`, pure) feeds frames through an RMS energy
   gate: segments start at speech onset (short pre-roll), commit on trailing
   silence or at ~25 s; pure silence is never transcribed.
3. The current segment is re-transcribed through the normal inference worker
   on a rolling basis (throttled, single-flight, latest-snapshot-wins) as
   *tentative* text; a committed segment gets one final pass and freezes.
   When inference runs slower than speech, queued committed segments are
   batched into merged final passes (one Whisper window each, D-024) so the
   backlog stays bounded. `requestId` on the protocol messages correlates
   responses.
4. Stopping flushes the segmenter, drains pending final passes (merged, with
   a "N remaining segments" countdown in the UI), and yields a normal
   transcript (text + absolute-timestamped chunks) in `TranscriptView`.
   Everything runs on-device; works offline like the file flow.
5. **Server-engine live (logged in, D-019).** Same `LiveControls` UI, but
   `App.tsx` opens a `GET /api/transcribe/live` WebSocket instead of running
   the segmenter locally: the browser only captures/resamples mic audio
   (`lib/mic.ts`) and streams raw PCM frames over the socket; the *same* pure
   `LiveTranscriber`/segmenter (`lib/live.ts`) runs server-side
   (`server/live.ts`), writing each segment to a temp file and bridging it to
   the faster-whisper sidecar (D-013) the same way `POST /api/transcribe`
   does. `{type:"update"}` frames carry committed/tentative text back;
   `{type:"stop"}` drains and closes with a `{type:"result"}` frame. Selected
   only when signed in with the server engine available (`EngineToggle`);
   anonymous/offline users always get the browser-engine path above.

### 4.1c Transcript translation (any engine — D-WS4b)

1. Any finished transcript (browser file, browser live, server engine) shows a
   "Translate transcript…" disclosure (`TranslatePanel`). Whisper's own
   "Translate to English" task stays in `TranscriptionOptions`, only for
   models with `canTranslate` (D-WS4a).
2. On **Translate**, the UI groups the transcript's chunks into sentence-ending
   units (`buildTranslationUnits`, merged time spans) and posts their texts to
   `translate.worker.js` — a separate worker with its own onnxruntime instance.
3. The worker loads the Opus-MT model (q8, ~113 MB, downloaded on this first
   request and cached by Transformers.js like Whisper weights), splits every
   unit into sentence pieces under a token budget (`splitForTranslation`,
   Opus-MT's 512-token input), translates them one by one, and answers one
   string per unit. The UI reassembles `{ text, chunks }` with the units'
   timestamps and renders it in a second `TranscriptView` with its own
   `.txt` / `.srt` exports (`<file>.<lang>.srt`). The text never leaves the
   device, whichever engine produced it.

### 4.2 Offline (PWA)
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
  (hashed JS/CSS, `/`, worker bundle, manifest, icons, `/fonts/*` Inter subsets) on install into
  `whisper-shell-<version>`, cache-first thereafter; unknown navigations fall back
  to the cached shell. `scripts/build.ts` injects the precache list + a
  content-hash version into `dist/sw.js` via `pwa/precache.ts`; the dev server
  injects an empty list ("dev"), making the dev shell network-first so HMR works.
- onnxruntime's `.wasm`/`.mjs` binaries are served from our origin at `/ort/*`
  (never the jsdelivr CDN default) and runtime-cached cache-first into
  `whisper-runtime-<version>` — they are required for offline inference but too
  large to precache eagerly.
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
  independent of the SW, which never touches that cache (D-005). `/api/*` is
  network-only.
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
  `SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
- Result: with the app installed and at least one model previously loaded, the full
  transcription flow works with no network and with our server down (manual
  verification procedure: `tests/offline.md`).

### 4.3 Server engine (logged-in only)
1. `EngineToggle` is visible only when a session exists.
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
   `POST /api/transcribe` with the chosen model id.
3. Server inference runs on **faster-whisper** (CTranslate2) in a Python sidecar
   (`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
   browser/WASM build, so it is server-only — the browser stays on Transformers.js).
   `src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
   JSON-lines HTTP response (D-011). The shared catalog maps model ids to
   `Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
   the server engine reports unavailable; the browser/anonymous flow is unaffected.
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
   UI is engine-agnostic.

### 4.4 Auth
- Passport.js **local strategy** behind a thin adapter that bridges Bun's
  `Request`/`Response` to Passport's connect-style API (see Decisions.md D-008).
- Sessions in SQLite; session id in an HttpOnly cookie.
- Anonymous users are never blocked from browser inference — auth only unlocks the
  server engine.

## 5. Invariants

- **Privacy:** browser-engine audio and transcripts never touch the network.
- **Anonymous-first:** every feature except the server engine works logged out.
- **Offline-first:** app shell + previously-downloaded models must work offline.
- **One browser inference codebase:** Transformers.js in the browser; the server
  engine runs faster-whisper (D-013). The model catalog stays shared across both.
- **Worker isolation:** inference never runs on the main thread.