323 lines
22 KiB
Markdown
323 lines
22 KiB
Markdown
# Architecture
|
||
|
||
> Companion to `../PRODUCT.md` (product spec), `../AGENTS.md` (agent & tooling rules) and
|
||
> `Decisions.md` (why things are the way they are). This file describes **how the system is structured**. Update it
|
||
> whenever the structure changes.
|
||
|
||
## 1. System overview
|
||
|
||
Browser-first speech-to-text app. Whisper inference runs **in the browser by default**
|
||
via [Transformers.js](https://github.com/huggingface/transformers)
|
||
(`@huggingface/transformers`); no audio ever leaves the device for anonymous users.
|
||
Logged-in users may optionally route inference to the server.
|
||
|
||
```
|
||
┌──────────────────────────── Browser ────────────────────────────┐
|
||
│ │
|
||
│ React UI (App shell) │
|
||
│ ├─ ModelPicker (tiny → large, .en, quantized variants) │
|
||
│ ├─ FileDropzone (audio/video upload) │
|
||
│ ├─ LiveControls (microphone start/stop — D-016) │
|
||
│ ├─ TranscriptView (streaming partial results) │
|
||
│ └─ EngineToggle (device / server — logged-in only) │
|
||
│ │ postMessage protocol │
|
||
│ ▼ │
|
||
│ Inference Web Worker │
|
||
│ ├─ Transformers.js ASR pipeline (Whisper) │
|
||
│ ├─ Device selection: WebNN → WebGPU → WASM fallback │
|
||
│ └─ Model weights cached via browser Cache API │
|
||
│ │
|
||
│ Translate Web Worker (D-045) │
|
||
│ └─ Transformers.js Opus-MT (MarianMT), WASM, on user request │
|
||
│ │
|
||
│ Service Worker (PWA) │
|
||
│ ├─ Precached app shell (offline-first) │
|
||
│ └─ Works with zero network once installed + model cached │
|
||
└──────────────────┬───────────────────────────────────────────────┘
|
||
│ HTTPS (only when online / server engine chosen)
|
||
▼
|
||
┌──────────────────────────── Bun server ─────────────────────────┐
|
||
│ Bun.serve() — routes + HTML imports (no vite, no express) │
|
||
│ ├─ / → src/index.html (bundled React app) │
|
||
│ ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
|
||
│ │ since Bun doesn't bundle `new Worker` graphs) │
|
||
│ ├─ /translate.worker.js → translation worker bundle (same) │
|
||
│ ├─ /api/auth/* → Passport.js (local strategy) + sessions │
|
||
│ ├─ /api/transcribe → server engine (auth required) │
|
||
│ └─ /manifest.webmanifest, /sw.js → PWA assets │
|
||
│ bun:sqlite → users, sessions │
|
||
│ Server inference: faster-whisper (CTranslate2) Python sidecar, │
|
||
│ spawned & supervised by Bun (D-013) │
|
||
└──────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## 2. Runtime & tooling
|
||
|
||
- **Runtime / bundler / test runner:** Bun only (`bun --hot src/index.ts` for dev,
|
||
`bun build src/index.html` for production, `bun test`). No vite/webpack/jest.
|
||
- **Frontend:** React 19, mounted from `src/index.html` via Bun HTML imports.
|
||
- **Server:** `Bun.serve()` with the `routes` option. No express.
|
||
- **DB:** `bun:sqlite`. No ORM planned initially.
|
||
|
||
## 3. Directory layout (target)
|
||
|
||
```
|
||
src/
|
||
index.ts # Bun.serve entry — routes, static, API
|
||
index.html # HTML entry, imports frontend.tsx
|
||
frontend.tsx # React root
|
||
App.tsx # top-level UI composition
|
||
components/ # ModelPicker, TranscriptionOptions, VocabularyHint, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
|
||
lib/
|
||
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
|
||
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
|
||
mic.ts # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
|
||
models.ts # model catalog: ids, family (whisper|parakeet), sizes, dtype defaults per device, languages
|
||
parakeet.ts # pure Parakeet logic: ≤30 s window plan, words→chunks, vocab tokenizer, pinned URLs
|
||
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
|
||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||
translation.ts # transcript translation planning: Opus-MT catalog, units, sentence pieces (pure, D-045)
|
||
hint.ts # "Names and terms" hint: normalize, localStorage, prompt truncation rule (D-048)
|
||
worker/
|
||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection; dispatches on catalog `family`
|
||
parakeetEngine.ts # Parakeet TDT path: parakeet.js ParakeetModel on the shared onnxruntime-web (D-051)
|
||
modelFiles.ts # Cache-API weight download for non-Transformers.js paths (same cache, same progress msgs)
|
||
ortShim.ts # re-exports onnxruntime-web/webgpu; parakeet.js's own ort import is aliased here
|
||
protocol.ts # typed postMessage messages (shared with UI)
|
||
prompt.ts # decoder prompt (<|startofprev|> hint) via a generate() wrapper (D-048)
|
||
translate.worker.ts # transcript translation (Opus-MT), its own worker + ort instance (D-045)
|
||
translateProtocol.ts # typed messages for the translate worker
|
||
opusMt.ts # Opus-MT loader shared by the worker, bench script and real-model test
|
||
longform/ # long-form file pipeline (D-049): pure vad.ts / quality.ts / pipeline.ts
|
||
# behind transcribeLongForm(audio, deps, options); transformers.ts implements
|
||
# the deps (Silero VAD + Whisper decodes) on the same Transformers.js runtime
|
||
pwa/
|
||
sw.ts # service worker (app-shell precache, /ort runtime cache)
|
||
precache.ts # placeholder-injection helpers shared by build + dev serve
|
||
register.ts # SW registration + update-toast flow
|
||
manifest.webmanifest
|
||
icons/ # generated by scripts/generate-icons.ts (checked in; `bun run icons` regenerates)
|
||
fonts/ # self-hosted Inter, copied from node_modules by scripts/fonts.ts (gitignored, D-037)
|
||
server/
|
||
auth.ts # Passport.js wiring + session middleware adapter
|
||
db.ts # bun:sqlite schema + queries
|
||
transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5)
|
||
live.ts # server-engine live mic over WebSocket (D-019, M7)
|
||
server-py/ # faster-whisper sidecar (Python, D-013; M5)
|
||
tests/ # bun test
|
||
```
|
||
|
||
(Current state: M0–M6 built — browser transcription with the curated model
|
||
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
|
||
language/task options, the installable offline PWA (`pwa/`,
|
||
`scripts/build.ts` production build, `dist/` serving in `src/index.ts`;
|
||
browser-level offline behavior pending the manual run in `tests/offline.md`),
|
||
M4 accounts (`server/db.ts`, `server/auth.ts`, `/api/auth/*`, `AuthForms`),
|
||
and the M5 server engine end to end — `server-py/sidecar.py`,
|
||
`server/transcribe.ts` (supervisor + `/api/transcribe` + `/status`),
|
||
`lib/engine.ts` (both engines, D-010), `EngineToggle`, and the catalog
|
||
mapping in `lib/models.ts` — with graceful degradation when
|
||
Python/faster-whisper is absent. M7 live microphone transcription is built
|
||
for both engines: browser (`lib/live.ts`, `lib/mic.ts`, `LiveControls`,
|
||
D-016) and server (`server/live.ts`, D-019); in-browser manual verification
|
||
of both is still pending per `tests/live.md` (no real browser in the dev
|
||
environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||
|
||
## 4. Key flows
|
||
|
||
### 4.1 Browser transcription (anonymous, default)
|
||
1. User picks a model in `ModelPicker` (curated catalog from `lib/models.ts`, with
|
||
size estimates and a Cache-API-backed "downloaded" badge). Selection alone
|
||
downloads nothing (D-020): the UI posts `{type:"load", modelId, device}` when a
|
||
browser-engine transcription starts (file drop / live start) or when the user
|
||
clicks the explicit **Download** button on the selected model; server-engine
|
||
runs never post a load. `device` is a preference
|
||
(`"auto" | "webnn" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype`
|
||
override is expressible in the message but normally omitted.
|
||
2. Worker resolves the device (D-004, D-017): for `"auto"` it probes
|
||
`navigator.ml` (WebNN) first, then `navigator.gpu?.requestAdapter()` (WebGPU),
|
||
and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the
|
||
picked device throws, it retries down the same chain (WASM always last). An
|
||
explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM.
|
||
Two D-025 refinements: WebGPU is skipped up front when the dtype it would
|
||
use needs fp16 but the adapter lacks `shader-f16` (Firefox), and a WASM
|
||
fallback *after* a failed accelerator attempt does not run in-place — the
|
||
failed attempt pinned ort's thread pool at 1 — but posts `load-restart`,
|
||
and the UI respawns the worker with an explicit `wasm` load (transcribe
|
||
posts wait on a ready gate in `App.tsx`, so none are lost). It then
|
||
resolves the catalog's dtype default for the device actually used and creates
|
||
the Transformers.js `automatic-speech-recognition` pipeline.
|
||
Download progress events are forwarded to the UI; the terminal
|
||
`{status:"ready", device}` message reports the backend actually selected, which
|
||
the UI surfaces. Loading a different model/device disposes the old pipeline
|
||
(`pipeline.dispose()`) before creating the new one.
|
||
3. Weights are fetched from the Hugging Face Hub **once**; Transformers.js stores them
|
||
in the browser Cache API, so subsequent loads (including offline) hit cache
|
||
(`lib/modelCache.ts` reads that cache for the downloaded badges).
|
||
4. User drops an audio/video file. Main thread decodes it with
|
||
`AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
|
||
transfers the `Float32Array` to the worker. For multilingual models the
|
||
`transcribe` message also carries `language` (or auto-detect) and `task`
|
||
(transcribe | translate — translate only for models with `canTranslate`, D-044).
|
||
For every model it may carry `prompt`: the optional "Names and terms" hint
|
||
(`VocabularyHint`, stored in localStorage by `lib/hint.ts`), raw as typed (D-048).
|
||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||
A hint becomes `<|startofprev|> hint.` once per pass (`worker/prompt.ts`,
|
||
cut to a 120-token prompt budget at a whole word) and is prepended
|
||
to the decoder input of every 30 s window by a wrapper around the model's
|
||
`generate()`, which also strips it from the output again (D-048).
|
||
Live passes (4.1b) carry the same `prompt`.
|
||
6. **Long-form branch (D-049/D-050, "Advanced → Skip silence", off by default).**
|
||
With the option on, every load posted for a file run or the Download button carries
|
||
`vad: true`; once the Whisper pipeline is ready the worker loads Silero VAD
|
||
(`onnx-community/silero-vad`, 2.2 MB, Cache API like the weights; offline it is
|
||
used only if already cached). A file pass with `longForm: true` then runs
|
||
`transcribeLongForm` (`worker/longform/`): VAD speech probabilities over the
|
||
whole buffer → speech spans (faster-whisper `vad.py` port) → chunks ≤ 30 s cut on
|
||
speech boundaries → one Whisper decode per chunk (language detected once on the
|
||
first chunk and pinned for multilingual models) → timestamps mapped back to the
|
||
file's timeline; progress follows the variable chunks
|
||
(`TranscribeProgressTracker.forWindows`). Same `partial` /
|
||
`transcribe-progress` / `result` messages as step 5. `choosePipeline` keeps the
|
||
chunked path whenever the VAD model isn't loaded, and live passes never take
|
||
this branch (live keeps its RMS segmenter, D-016/D-024). A finished transcript
|
||
with no words shows "No speech was found in this recording." The names-and-terms
|
||
hint (`prompt`) is applied to every chunk's decode the same way (D-048).
|
||
|
||
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
|
||
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
|
||
device's native rate → AudioWorklet chunker (module from a Blob URL;
|
||
`ScriptProcessorNode` fallback) posts ~100 ms Float32 frames; the main
|
||
thread resamples each to 16 kHz (`lib/mic.ts`).
|
||
2. `LiveTranscriber` (`lib/live.ts`, pure) feeds frames through an RMS energy
|
||
gate: segments start at speech onset (short pre-roll), commit on trailing
|
||
silence or at ~25 s; pure silence is never transcribed.
|
||
3. The current segment is re-transcribed through the normal inference worker
|
||
on a rolling basis (throttled, single-flight, latest-snapshot-wins) as
|
||
*tentative* text; a committed segment gets one final pass and freezes.
|
||
When inference runs slower than speech, queued committed segments are
|
||
batched into merged final passes (one Whisper window each, D-024) so the
|
||
backlog stays bounded. `requestId` on the protocol messages correlates
|
||
responses.
|
||
4. Stopping flushes the segmenter, drains pending final passes (merged, with
|
||
a "N remaining segments" countdown in the UI), and yields a normal
|
||
transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
||
Everything runs on-device; works offline like the file flow.
|
||
5. **Server-engine live (logged in, D-019).** Same `LiveControls` UI, but
|
||
`App.tsx` opens a `GET /api/transcribe/live` WebSocket instead of running
|
||
the segmenter locally: the browser only captures/resamples mic audio
|
||
(`lib/mic.ts`) and streams raw PCM frames over the socket; the *same* pure
|
||
`LiveTranscriber`/segmenter (`lib/live.ts`) runs server-side
|
||
(`server/live.ts`), writing each segment to a temp file and bridging it to
|
||
the faster-whisper sidecar (D-013) the same way `POST /api/transcribe`
|
||
does. `{type:"update"}` frames carry committed/tentative text back;
|
||
`{type:"stop"}` drains and closes with a `{type:"result"}` frame. Selected
|
||
only when signed in with the server engine available (`EngineToggle`);
|
||
anonymous/offline users always get the browser-engine path above.
|
||
|
||
### 4.1c Transcript translation (any engine — D-045)
|
||
|
||
1. Any finished transcript (browser file, browser live, server engine) shows a
|
||
"Translate transcript…" disclosure (`TranslatePanel`). Whisper's own
|
||
"Translate to English" task stays in `TranscriptionOptions`, only for
|
||
models with `canTranslate` (D-044).
|
||
2. On **Translate**, the UI groups the transcript's chunks into sentence-ending
|
||
units (`buildTranslationUnits`, merged time spans) and posts their texts to
|
||
`translate.worker.js` — a separate worker with its own onnxruntime instance.
|
||
3. The worker loads the Opus-MT model (q8, ~113 MB, downloaded on this first
|
||
request and cached by Transformers.js like Whisper weights), splits every
|
||
unit into sentence pieces under a token budget (`splitForTranslation`,
|
||
Opus-MT's 512-token input), translates them one by one, and answers one
|
||
string per unit. The UI reassembles `{ text, chunks }` with the units'
|
||
timestamps and renders it in a second `TranscriptView` with its own
|
||
`.txt` / `.srt` exports (`<file>.<lang>.srt`). The text never leaves the
|
||
device, whichever engine produced it.
|
||
|
||
### 4.2 Offline (PWA)
|
||
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
|
||
(hashed JS/CSS, `/`, worker bundle, manifest, icons, `/fonts/*` Inter subsets) on install into
|
||
`whisper-shell-<version>`, cache-first thereafter; unknown navigations fall back
|
||
to the cached shell. `scripts/build.ts` injects the precache list + a
|
||
content-hash version into `dist/sw.js` via `pwa/precache.ts`; the dev server
|
||
injects an empty list ("dev"), making the dev shell network-first so HMR works.
|
||
- onnxruntime's `.wasm`/`.mjs` binaries are served from our origin at `/ort/*`
|
||
(never the jsdelivr CDN default) and runtime-cached cache-first into
|
||
`whisper-runtime-<version>` — they are required for offline inference but too
|
||
large to precache eagerly.
|
||
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
|
||
independent of the SW, which never touches that cache (D-005). `/api/*` is
|
||
network-only. The Silero VAD model of the long-form branch lives there too
|
||
(D-050); offline, the worker checks that cache before loading it and never
|
||
fetches, falling back to the chunked path if it is missing.
|
||
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
|
||
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
|
||
- Result: with the app installed and at least one model previously loaded, the full
|
||
transcription flow works with no network and with our server down (manual
|
||
verification procedure: `tests/offline.md`).
|
||
|
||
### 4.3 Server engine (logged-in only)
|
||
1. `EngineToggle` is visible only when a session exists.
|
||
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
|
||
`POST /api/transcribe` with the chosen model id.
|
||
3. Server inference runs on **faster-whisper** (CTranslate2) in a Python sidecar
|
||
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
|
||
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
|
||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||
JSON-lines HTTP response (D-011). `language`, `task` and the vocabulary hint
|
||
(`prompt`) travel as query params (also on the live WebSocket) and are
|
||
validated by `parseLanguageAndTask` (D-032, D-048); the sidecar cuts the
|
||
hint with the same rule as the browser and passes it to faster-whisper as
|
||
`initial_prompt` (D-048). File requests (`mode: "file"`) decode with
|
||
faster-whisper's batched pipeline over Silero-VAD speech chunks; server-live
|
||
passes send `mode: "live"` and stay sequential (D-047). The shared catalog maps model ids to
|
||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||
the server engine reports unavailable; the browser/anonymous flow is unaffected.
|
||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||
UI is engine-agnostic.
|
||
|
||
### 4.4 Auth
|
||
- Passport.js **local strategy** behind a thin adapter that bridges Bun's
|
||
`Request`/`Response` to Passport's connect-style API (see Decisions.md D-008).
|
||
- Sessions in SQLite; session id in an HttpOnly cookie.
|
||
- Anonymous users are never blocked from browser inference — auth only unlocks the
|
||
server engine.
|
||
|
||
### 4.1c Parakeet (English-only, desktop-only browser model — D-051)
|
||
Catalog entries carry `family?: "whisper" | "parakeet"` (absent ⇒ whisper). For
|
||
`family: "parakeet"` the worker skips the Transformers.js pipeline:
|
||
1. **Load:** `modelFiles.ts` fetches `vocab.txt`, the decoder_joint and the
|
||
encoder from the Hub (revision-pinned URLs, `lib/parakeet.ts`) into the same
|
||
`transformers-cache` Cache Storage cache Transformers.js uses, posting the same
|
||
`load-progress` messages — so badges, the storage panel, delete and
|
||
ModelLoadProgress work unchanged. A file is cached only once complete.
|
||
Devices: `parakeetDeviceAttempts` — WASM int8 unless WebGPU is chosen
|
||
explicitly (fp16 encoder on WebGPU + int8 decoder on WASM, needs
|
||
`shader-f16`); the D-025 restart rule applies.
|
||
2. **Runtime:** `parakeetEngine.ts` builds parakeet.js's `ParakeetModel` from our
|
||
own sessions on `onnxruntime-web/webgpu` — the module Transformers.js imports,
|
||
so there is one runtime and one `/ort/` wasm. parakeet.js's `initOrt` (own ort
|
||
copy, CDN wasm path) is never called; `scripts/ortAlias.ts` points its import
|
||
at `worker/ortShim.ts` so the bundle carries one ort (`tests/workerBundle.test.ts`).
|
||
3. **Transcribe:** audio is planned into ≤ 30 s windows cut at the quietest 100 ms
|
||
near each limit (`subarray` views, no copies); each window's word timestamps
|
||
become sentence-sized `TranscriptChunk`s with absolute times. File passes post
|
||
one `partial` + `transcribe-progress` per window; live passes resolve via
|
||
`result` like Whisper's, so `LiveTranscriber`, TranscriptView and export are
|
||
untouched. `language`/`task` are ignored (English only).
|
||
4. **UI:** the picker lists it under English-only with a one-line note, hides it on
|
||
phones/tablets (`modelsForDevice`), and the server-engine list never includes
|
||
it (no faster-whisper name). With it selected, a note replaces the language/task
|
||
controls.
|
||
|
||
## 5. Invariants
|
||
|
||
- **Privacy:** browser-engine audio and transcripts never touch the network.
|
||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||
- **One browser inference runtime:** onnxruntime-web, driven by Transformers.js for
|
||
Whisper and by parakeet.js for Parakeet (D-051) — the same ort instance, never a
|
||
second runtime; the server engine runs faster-whisper (D-013). The model catalog
|
||
stays shared across both engines (browser-only entries have no faster-whisper name).
|
||
- **Worker isolation:** inference never runs on the main thread.
|