Architecture

Companion to CLAUDE.md (product spec & tooling rules) and Decisions.md (why things are the way they are). This file describes how the system is structured. Update it whenever the structure changes.

1. System overview

Browser-first speech-to-text app. Whisper inference runs in the browser by default via Transformers.js (@huggingface/transformers); no audio ever leaves the device for anonymous users. Logged-in users may optionally route inference to the server.

┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebGPU → WASM fallback                    │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: @huggingface/transformers (Node/Bun runtime)  │
└──────────────────────────────────────────────────────────────────┘

2. Runtime & tooling

Runtime / bundler / test runner: Bun only (bun --hot src/index.ts for dev, bun build src/index.html for production, bun test). No vite/webpack/jest.
Frontend: React 19, mounted from src/index.html via Bun HTML imports.
Server: Bun.serve() with the routes option. No express.
DB: bun:sqlite. No ORM planned initially.

3. Directory layout (target)

src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, FileDropzone, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    models.ts              # model catalog: ids, sizes, dtypes, languages
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebGPU/WASM selection
    protocol.ts            # typed postMessage messages (shared with UI)
  pwa/
    sw.ts                  # service worker (app-shell precache)
    manifest.webmanifest
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint
tests/                     # bun test

(Current state: M0–M1 built — browser transcription MVP with one hardcoded model (onnx-community/whisper-tiny.en, WASM). lib/models.ts, lib/engine.ts, pwa/, and server/ are TODO stubs for M2–M5; see WorkBreakdown.md.)

4. Key flows

4.1 Browser transcription (anonymous, default)

User picks a model in ModelPicker; UI posts {type:"load", modelId, dtype, device} to the inference worker.
Worker creates the Transformers.js automatic-speech-recognition pipeline. device: "webgpu" if navigator.gpu adapter is available, else "wasm". Download progress events are forwarded to the UI.
Weights are fetched from the Hugging Face Hub once; Transformers.js stores them in the browser Cache API, so subsequent loads (including offline) hit cache.
User drops an audio/video file. Main thread decodes it with AudioContext.decodeAudioData, downmixes to mono, resamples to 16 kHz, transfers the Float32Array to the worker.
Worker runs the pipeline (chunked, with timestamps), streaming partial results back via {type:"partial"} messages; final result as {type:"result"}.

4.2 Offline (PWA)

Service worker precaches the built app shell (HTML/JS/CSS/icons) on install.
Model weights live in the Cache API via Transformers.js — independent of the SW.
Result: with the app installed and at least one model previously loaded, the full transcription flow works with no network and with our server down.

4.3 Server engine (logged-in only)

EngineToggle is visible only when a session exists.
ServerEngine.transcribe() uploads the decoded (or original) audio to POST /api/transcribe with the chosen model id.
Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU), streams progress/results back (chunked response or WebSocket — see Decisions.md D-011 once settled).
Both engines implement the same Engine interface in src/lib/engine.ts, so the UI is engine-agnostic.

4.4 Auth

Passport.js local strategy behind a thin adapter that bridges Bun's Request/Response to Passport's connect-style API (see Decisions.md D-008).
Sessions in SQLite; session id in an HttpOnly cookie.
Anonymous users are never blocked from browser inference — auth only unlocks the server engine.

5. Invariants

Privacy: browser-engine audio and transcripts never touch the network.
Anonymous-first: every feature except the server engine works logged out.
Offline-first: app shell + previously-downloaded models must work offline.
One inference codebase: Transformers.js on both sides; model catalog shared.
Worker isolation: inference never runs on the main thread.

7.4 KiB Raw Blame History Unescape Escape