whisper_web/docs/Architecture.md

Architecture

Companion to ../PRODUCT.md (product spec), ../AGENTS.md (agent & tooling rules) and Decisions.md (why things are the way they are). This file describes how the system is structured. Update it whenever the structure changes.

1. System overview

Browser-first speech-to-text app. Whisper inference runs in the browser by default via Transformers.js (@huggingface/transformers); no audio ever leaves the device for anonymous users. Logged-in users may optionally route inference to the server.

┌──────────────────────────── Browser ────────────────────────────┐
│                                                                  │
│  React UI (App shell)                                            │
│   ├─ ModelPicker        (tiny → large, .en, quantized variants)  │
│   ├─ FileDropzone       (audio/video upload)                     │
│   ├─ LiveControls       (microphone start/stop — D-016)          │
│   ├─ TranscriptView     (streaming partial results)              │
│   └─ EngineToggle       (device / server — logged-in only)       │
│          │ postMessage protocol                                  │
│          ▼                                                       │
│  Inference Web Worker                                            │
│   ├─ Transformers.js ASR pipeline (Whisper)                      │
│   ├─ Device selection: WebNN → WebGPU → WASM fallback            │
│   └─ Model weights cached via browser Cache API                  │
│                                                                  │
│  Translate Web Worker (D-045)                                   │
│   └─ Transformers.js Opus-MT (MarianMT), WASM, on user request   │
│                                                                  │
│  Service Worker (PWA)                                            │
│   ├─ Precached app shell (offline-first)                         │
│   └─ Works with zero network once installed + model cached       │
└──────────────────┬───────────────────────────────────────────────┘
                   │ HTTPS (only when online / server engine chosen)
                   ▼
┌──────────────────────────── Bun server ─────────────────────────┐
│  Bun.serve() — routes + HTML imports (no vite, no express)       │
│   ├─ /            → src/index.html (bundled React app)           │
│   ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
│   │                 since Bun doesn't bundle `new Worker` graphs) │
│   ├─ /translate.worker.js → translation worker bundle (same)     │
│   ├─ /api/auth/*  → Passport.js (local strategy) + sessions      │
│   ├─ /api/transcribe → server engine (auth required)             │
│   └─ /manifest.webmanifest, /sw.js → PWA assets                  │
│  bun:sqlite → users, sessions                                    │
│  Server inference: faster-whisper (CTranslate2) Python sidecar,  │
│                    spawned & supervised by Bun (D-013)            │
└──────────────────────────────────────────────────────────────────┘

2. Runtime & tooling

• Runtime / bundler / test runner: Bun only (bun --hot src/index.ts for dev, bun build src/index.html for production, bun test). No vite/webpack/jest.
• Frontend: React 19, mounted from src/index.html via Bun HTML imports.
• Server: Bun.serve() with the routes option. No express.
• DB: bun:sqlite. No ORM planned initially.

3. Directory layout (target)

src/
  index.ts                 # Bun.serve entry — routes, static, API
  index.html               # HTML entry, imports frontend.tsx
  frontend.tsx             # React root
  App.tsx                  # top-level UI composition
  components/              # ModelPicker, TranscriptionOptions, VocabularyHint, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
  lib/
    audio.ts               # decode + resample to 16 kHz mono Float32Array (Web Audio API)
    live.ts                # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
    mic.ts                 # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
    models.ts              # model catalog: ids, family (whisper|parakeet), sizes, dtype defaults per device, languages
    parakeet.ts            # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
    modelCache.ts          # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
    engine.ts              # engine abstraction: BrowserEngine | ServerEngine
    translation.ts         # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-045)
    hint.ts                # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-048)
  worker/
    inference.worker.ts    # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
    parakeetEngine.ts      # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-051)
    modelFiles.ts          # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
    ortShim.ts             # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
    protocol.ts            # typed postMessage messages (shared with UI)
    prompt.ts              # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-048)
    translate.worker.ts    # transcript translation (Opus-MT), its own worker + ort instance (D-045)
    translateProtocol.ts   # typed messages for the translate worker
    opusMt.ts              # Opus-MT loader shared by the worker, bench script and real-model test
    longform/              # long-form file pipeline (D-049): pure vad.ts / quality.ts / pipeline.ts
                           # behind transcribeLongForm(audio, deps, options); transformers.ts implements
                           # the deps (Silero VAD + Whisper decodes) on the same Transformers.js runtime
  pwa/
    sw.ts                  # service worker (app-shell precache, /ort runtime cache)
    precache.ts            # placeholder-injection helpers shared by build + dev serve
    register.ts            # SW registration + update-toast flow
    manifest.webmanifest
    icons/                 # generated by scripts/generate-icons.ts (checked in; `bun run icons` regenerates)
  fonts/                   # self-hosted Inter, copied from node_modules by scripts/fonts.ts (gitignored, D-037)
  server/
    auth.ts                # Passport.js wiring + session middleware adapter
    db.ts                  # bun:sqlite schema + queries
    transcribe.ts          # server-side ASR endpoint (bridges the sidecar, M5)
    live.ts                # server-engine live mic over WebSocket (D-019, M7)
server-py/                 # faster-whisper sidecar (Python, D-013; M5)
tests/                     # bun test

(Current state: M0–M6 built — browser transcription with the curated model catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching, language/task options, the installable offline PWA (pwa/, scripts/build.ts production build, dist/ serving in src/index.ts; browser-level offline behavior pending the manual run in tests/offline.md), M4 accounts (server/db.ts, server/auth.ts, /api/auth/*, AuthForms), and the M5 server engine end to end — server-py/sidecar.py, server/transcribe.ts (supervisor + /api/transcribe + /status), lib/engine.ts (both engines, D-010), EngineToggle, and the catalog mapping in lib/models.ts — with graceful degradation when Python/faster-whisper is absent. M7 live microphone transcription is built for both engines: browser (lib/live.ts, lib/mic.ts, LiveControls, D-016) and server (server/live.ts, D-019); in-browser manual verification of both is still pending per tests/live.md (no real browser in the dev environment). See WorkBreakdown.md for the itemized checklist.)

4. Key flows

4.1 Browser transcription (anonymous, default)

1. User picks a model in ModelPicker (curated catalog from lib/models.ts, with size estimates and a Cache-API-backed "downloaded" badge). Selection alone downloads nothing (D-020): the UI posts {type:"load", modelId, device} when a browser-engine transcription starts (file drop / live start) or when the user clicks the explicit Download button on the selected model; server-engine runs never post a load. device is a preference ("auto" | "webnn" | "webgpu" | "wasm", default "auto"). An explicit dtype override is expressible in the message but normally omitted.
2. Worker resolves the device (D-004, D-017): for "auto" it probes navigator.ml (WebNN) first, then navigator.gpu?.requestAdapter() (WebGPU), and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the picked device throws, it retries down the same chain (WASM always last). An explicit "webnn"/"webgpu" preference falls back straight to WASM. Two D-025 refinements: WebGPU is skipped up front when the dtype it would use needs fp16 but the adapter lacks shader-f16 (Firefox), and a WASM fallback after a failed accelerator attempt does not run in-place — the failed attempt pinned ort's thread pool at 1 — but posts load-restart, and the UI respawns the worker with an explicit wasm load (transcribe posts wait on a ready gate in App.tsx, so none are lost). It then resolves the catalog's dtype default for the device actually used and creates the Transformers.js automatic-speech-recognition pipeline. Download progress events are forwarded to the UI; the terminal {status:"ready", device} message reports the backend actually selected, which the UI surfaces. Loading a different model/device disposes the old pipeline (pipeline.dispose()) before creating the new one.
3. Weights are fetched from the Hugging Face Hub once; Transformers.js stores them in the browser Cache API, so subsequent loads (including offline) hit cache (lib/modelCache.ts reads that cache for the downloaded badges).
4. User drops an audio/video file. Main thread decodes it with AudioContext.decodeAudioData, downmixes to mono, resamples to 16 kHz, transfers the Float32Array to the worker. For multilingual models the transcribe message also carries language (or auto-detect) and task (transcribe | translate — translate only for models with canTranslate, D-044). For every model it may carry prompt: the optional "Names and terms" hint (VocabularyHint, stored in localStorage by lib/hint.ts), raw as typed (D-048).
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results back via {type:"partial"} messages; final result as {type:"result"}. With auto-detect on a multilingual model, it first detects the language on the first 30 s (worker/language.ts, D-053) and pins it — Transformers.js itself would default to English. A hint becomes <|startofprev|> hint. once per pass (worker/prompt.ts, cut to a 120-token prompt budget at a whole word) and is prepended to the decoder input of every 30 s window by a wrapper around the model's generate(), which also strips it from the output again (D-048). Live passes (4.1b) carry the same prompt.
6. Long-form branch (D-049/D-050, "Advanced → Skip silence", on by default since D-052). With the option on, every load posted for a file run or the Download button carries vad: true; once the Whisper pipeline is ready the worker loads Silero VAD (onnx-community/silero-vad, 2.2 MB, Cache API like the weights; offline it is used only if already cached). A file pass with longForm: true then runs transcribeLongForm (worker/longform/): VAD speech probabilities over the whole buffer → speech spans (faster-whisper vad.py port) → chunks ≤ 30 s cut on speech boundaries → one Whisper decode per chunk (language detected once on the first chunk and pinned for multilingual models) → timestamps mapped back to the file's timeline; progress follows the variable chunks (TranscribeProgressTracker.forWindows). Same partial / transcribe-progress / result messages as step 5. choosePipeline keeps the chunked path whenever the VAD model isn't loaded, and live passes never take this branch (live keeps its RMS segmenter, D-016/D-024). A finished transcript with no words shows "No speech was found in this recording." The names-and-terms hint (prompt) is applied to every chunk's decode the same way (D-048).

4.1b Live microphone transcription (anonymous, browser engine — D-016)

1. LiveControls starts a session: getUserMedia → AudioContext at the device's native rate → AudioWorklet chunker (module from a Blob URL; ScriptProcessorNode fallback) posts ~100 ms Float32 frames; the main thread resamples each to 16 kHz (lib/mic.ts).
2. LiveTranscriber (lib/live.ts, pure) feeds frames through an RMS energy gate: segments start at speech onset (short pre-roll), commit on trailing silence or at ~25 s; pure silence is never transcribed.
3. The current segment is re-transcribed through the normal inference worker on a rolling basis (throttled, single-flight, latest-snapshot-wins) as tentative text; a committed segment gets one final pass and freezes. When inference runs slower than speech, queued committed segments are batched into merged final passes (one Whisper window each, D-024) so the backlog stays bounded. requestId on the protocol messages correlates responses.
4. Stopping flushes the segmenter, drains pending final passes (merged, with a "N remaining segments" countdown in the UI), and yields a normal transcript (text + absolute-timestamped chunks) in TranscriptView. Everything runs on-device; works offline like the file flow.
5. Server-engine live (logged in, D-019). Same LiveControls UI, but App.tsx opens a GET /api/transcribe/live WebSocket instead of running the segmenter locally: the browser only captures/resamples mic audio (lib/mic.ts) and streams raw PCM frames over the socket; the same pure LiveTranscriber/segmenter (lib/live.ts) runs server-side (server/live.ts), writing each segment to a temp file and bridging it to the faster-whisper sidecar (D-013) the same way POST /api/transcribe does. {type:"update"} frames carry committed/tentative text back; {type:"stop"} drains and closes with a {type:"result"} frame. Selected only when signed in with the server engine available (EngineToggle); anonymous/offline users always get the browser-engine path above.

4.1c Transcript translation (any engine — D-045)

1. Any finished transcript (browser file, browser live, server engine) shows a "Translate transcript…" disclosure (TranslatePanel). Whisper's own "Translate to English" task stays in TranscriptionOptions, only for models with canTranslate (D-044).
2. On Translate, the UI groups the transcript's chunks into sentence-ending units (buildTranslationUnits, merged time spans) and posts their texts to translate.worker.js — a separate worker with its own onnxruntime instance.
3. The worker loads the Opus-MT model (q8, ~113 MB, downloaded on this first request and cached by Transformers.js like Whisper weights), splits every unit into sentence pieces under a token budget (splitForTranslation, Opus-MT's 512-token input), translates them one by one, and answers one string per unit. The UI reassembles { text, chunks } with the units' timestamps and renders it in a second TranscriptView with its own .txt / .srt exports (<file>.<lang>.srt). The text never leaves the device, whichever engine produced it.

4.2 Offline (PWA)

• Service worker (pwa/sw.ts, hand-rolled — D-007) precaches the built app shell (hashed JS/CSS, /, worker bundle, manifest, icons, /fonts/* Inter subsets) on install into whisper-shell-<version>, cache-first thereafter; unknown navigations fall back to the cached shell. scripts/build.ts injects the precache list + a content-hash version into dist/sw.js via pwa/precache.ts; the dev server injects an empty list ("dev"), making the dev shell network-first so HMR works.
• onnxruntime's .wasm/.mjs binaries are served from our origin at /ort/* (never the jsdelivr CDN default) and runtime-cached cache-first into whisper-runtime-<version> — they are required for offline inference but too large to precache eagerly.
• Model weights live in the Cache API (transformers-cache) via Transformers.js — independent of the SW, which never touches that cache (D-005). /api/* is network-only. The Silero VAD model of the long-form branch lives there too (D-050); offline, the worker checks that cache before loading it and never fetches, falling back to the chunked path if it is missing.
• Updates: pwa/register.ts surfaces a waiting SW as a toast; accepting posts SKIP_WAITING, activation drops old whisper-* caches, and the page reloads.
• Result: with the app installed and at least one model previously loaded, the full transcription flow works with no network and with our server down (manual verification procedure: tests/offline.md).

4.3 Server engine (logged-in only)

1. EngineToggle is visible only when a session exists.
2. ServerEngine.transcribe() uploads the decoded (or original) audio to POST /api/transcribe with the chosen model id.
3. Server inference runs on faster-whisper (CTranslate2) in a Python sidecar (server-py/) that Bun spawns and supervises (D-013; faster-whisper has no browser/WASM build, so it is server-only — the browser stays on Transformers.js). src/server/transcribe.ts bridges the sidecar's segment stream to the streamed JSON-lines HTTP response (D-011). language, task and the vocabulary hint (prompt) travel as query params (also on the live WebSocket) and are validated by parseLanguageAndTask (D-032, D-048); the sidecar cuts the hint with the same rule as the browser and passes it to faster-whisper as initial_prompt (D-048). File requests (mode: "file") decode with faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live passes send mode: "live" and stay sequential (D-047). The shared catalog maps model ids to Systran/faster-whisper-* CTranslate2 conversions. If the sidecar is absent, the server engine reports unavailable; the browser/anonymous flow is unaffected.
4. Both engines implement the same Engine interface in src/lib/engine.ts, so the UI is engine-agnostic.

4.4 Auth

• Passport.js local strategy behind a thin adapter that bridges Bun's Request/Response to Passport's connect-style API (see Decisions.md D-008).
• Sessions in SQLite; session id in an HttpOnly cookie.
• Anonymous users are never blocked from browser inference — auth only unlocks the server engine.

4.1c Parakeet (English-only, desktop-only browser model — D-051)

Catalog entries carry family?: "whisper" | "parakeet" (absent ⇒ whisper). For family: "parakeet" the worker skips the Transformers.js pipeline:

1. Load: modelFiles.ts fetches vocab.txt, the decoder_joint and the encoder from the Hub (revision-pinned URLs, lib/parakeet.ts) into the same transformers-cache Cache Storage cache Transformers.js uses, posting the same load-progress messages — so badges, the storage panel, delete and ModelLoadProgress work unchanged. A file is cached only once complete. Devices: parakeetDeviceAttempts — WASM int8 unless WebGPU is chosen explicitly (fp16 encoder on WebGPU + int8 decoder on WASM, needs shader-f16); the D-025 restart rule applies.
2. Runtime: parakeetEngine.ts builds parakeet.js's ParakeetModel from our own sessions on onnxruntime-web/webgpu — the module Transformers.js imports, so there is one runtime and one /ort/ wasm. parakeet.js's initOrt (own ort copy, CDN wasm path) is never called; scripts/ortAlias.ts points its import at worker/ortShim.ts so the bundle carries one ort (tests/workerBundle.test.ts).
3. Transcribe: audio is planned into ≤ 30 s windows cut at the quietest 100 ms near each limit (subarray views, no copies); each window's word timestamps become sentence-sized TranscriptChunks with absolute times. File passes post one partial + transcribe-progress per window; live passes resolve via result like Whisper's, so LiveTranscriber, TranscriptView and export are untouched. language/task are ignored (English only).
4. UI: the picker lists it under English-only with a one-line note, hides it on phones/tablets (modelsForDevice), and the server-engine list never includes it (no faster-whisper name). With it selected, a note replaces the language/task controls.

5. Invariants

• Privacy: browser-engine audio and transcripts never touch the network.
• Anonymous-first: every feature except the server engine works logged out.
• Offline-first: app shell + previously-downloaded models must work offline.
• One browser inference runtime: onnxruntime-web, driven by Transformers.js for Whisper and by parakeet.js for Parakeet (D-051) — the same ort instance, never a second runtime; the server engine runs faster-whisper (D-013). The model catalog stays shared across both engines (browser-only entries have no faster-whisper name).
• Worker isolation: inference never runs on the main thread.