Agent completed work jubii
This commit is contained in:
Lucas Winther committed
1 parent
6466d3a45c
commit
eb72a2e646
29 files changed
+2051
No files matched your search
+131
@@ -0,0 +1,131 @@
|
||||
# Architecture
|
||||
|
||||
> Companion to `CLAUDE.md` (product spec & tooling rules) and `Decisions.md` (why things
|
||||
> are the way they are). This file describes **how the system is structured**. Update it
|
||||
> whenever the structure changes.
|
||||
|
||||
## 1. System overview
|
||||
|
||||
Browser-first speech-to-text app. Whisper inference runs **in the browser by default**
|
||||
via [Transformers.js](https://github.com/huggingface/transformers)
|
||||
(`@huggingface/transformers`); no audio ever leaves the device for anonymous users.
|
||||
Logged-in users may optionally route inference to the server.
|
||||
|
||||
```
|
||||
┌──────────────────────────── Browser ────────────────────────────┐
|
||||
│ │
|
||||
│ React UI (App shell) │
|
||||
│ ├─ ModelPicker (tiny → large, .en, quantized variants) │
|
||||
│ ├─ FileDropzone (audio/video upload) │
|
||||
│ ├─ TranscriptView (streaming partial results) │
|
||||
│ └─ EngineToggle (device / server — logged-in only) │
|
||||
│ │ postMessage protocol │
|
||||
│ ▼ │
|
||||
│ Inference Web Worker │
|
||||
│ ├─ Transformers.js ASR pipeline (Whisper) │
|
||||
│ ├─ Device selection: WebGPU → WASM fallback │
|
||||
│ └─ Model weights cached via browser Cache API │
|
||||
│ │
|
||||
│ Service Worker (PWA) │
|
||||
│ ├─ Precached app shell (offline-first) │
|
||||
│ └─ Works with zero network once installed + model cached │
|
||||
└──────────────────┬───────────────────────────────────────────────┘
|
||||
│ HTTPS (only when online / server engine chosen)
|
||||
▼
|
||||
┌──────────────────────────── Bun server ─────────────────────────┐
|
||||
│ Bun.serve() — routes + HTML imports (no vite, no express) │
|
||||
│ ├─ / → src/index.html (bundled React app) │
|
||||
│ ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
|
||||
│ │ since Bun doesn't bundle `new Worker` graphs) │
|
||||
│ ├─ /api/auth/* → Passport.js (local strategy) + sessions │
|
||||
│ ├─ /api/transcribe → server engine (auth required) │
|
||||
│ └─ /manifest.webmanifest, /sw.js → PWA assets │
|
||||
│ bun:sqlite → users, sessions │
|
||||
│ Server inference: @huggingface/transformers (Node/Bun runtime) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2. Runtime & tooling
|
||||
|
||||
- **Runtime / bundler / test runner:** Bun only (`bun --hot src/index.ts` for dev,
|
||||
`bun build src/index.html` for production, `bun test`). No vite/webpack/jest.
|
||||
- **Frontend:** React 19, mounted from `src/index.html` via Bun HTML imports.
|
||||
- **Server:** `Bun.serve()` with the `routes` option. No express.
|
||||
- **DB:** `bun:sqlite`. No ORM planned initially.
|
||||
|
||||
## 3. Directory layout (target)
|
||||
|
||||
```
|
||||
src/
|
||||
index.ts # Bun.serve entry — routes, static, API
|
||||
index.html # HTML entry, imports frontend.tsx
|
||||
frontend.tsx # React root
|
||||
App.tsx # top-level UI composition
|
||||
components/ # ModelPicker, FileDropzone, TranscriptView, EngineToggle, AuthForms
|
||||
lib/
|
||||
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
|
||||
models.ts # model catalog: ids, sizes, dtypes, languages
|
||||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||||
worker/
|
||||
inference.worker.ts # Transformers.js pipeline, WebGPU/WASM selection
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
pwa/
|
||||
sw.ts # service worker (app-shell precache)
|
||||
manifest.webmanifest
|
||||
server/
|
||||
auth.ts # Passport.js wiring + session middleware adapter
|
||||
db.ts # bun:sqlite schema + queries
|
||||
transcribe.ts # server-side ASR endpoint
|
||||
tests/ # bun test
|
||||
```
|
||||
|
||||
(Current state: M0–M1 built — browser transcription MVP with one hardcoded model
|
||||
(`onnx-community/whisper-tiny.en`, WASM). `lib/models.ts`, `lib/engine.ts`,
|
||||
`pwa/`, and `server/` are TODO stubs for M2–M5; see `WorkBreakdown.md`.)
|
||||
|
||||
## 4. Key flows
|
||||
|
||||
### 4.1 Browser transcription (anonymous, default)
|
||||
1. User picks a model in `ModelPicker`; UI posts `{type:"load", modelId, dtype, device}`
|
||||
to the inference worker.
|
||||
2. Worker creates the Transformers.js `automatic-speech-recognition` pipeline.
|
||||
`device: "webgpu"` if `navigator.gpu` adapter is available, else `"wasm"`.
|
||||
Download progress events are forwarded to the UI.
|
||||
3. Weights are fetched from the Hugging Face Hub **once**; Transformers.js stores them
|
||||
in the browser Cache API, so subsequent loads (including offline) hit cache.
|
||||
4. User drops an audio/video file. Main thread decodes it with
|
||||
`AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
|
||||
transfers the `Float32Array` to the worker.
|
||||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
|
||||
### 4.2 Offline (PWA)
|
||||
- Service worker precaches the built app shell (HTML/JS/CSS/icons) on install.
|
||||
- Model weights live in the Cache API via Transformers.js — independent of the SW.
|
||||
- Result: with the app installed and at least one model previously loaded, the full
|
||||
transcription flow works with no network and with our server down.
|
||||
|
||||
### 4.3 Server engine (logged-in only)
|
||||
1. `EngineToggle` is visible only when a session exists.
|
||||
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
|
||||
`POST /api/transcribe` with the chosen model id.
|
||||
3. Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU),
|
||||
streams progress/results back (chunked response or WebSocket — see Decisions.md
|
||||
D-011 once settled).
|
||||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||||
UI is engine-agnostic.
|
||||
|
||||
### 4.4 Auth
|
||||
- Passport.js **local strategy** behind a thin adapter that bridges Bun's
|
||||
`Request`/`Response` to Passport's connect-style API (see Decisions.md D-008).
|
||||
- Sessions in SQLite; session id in an HttpOnly cookie.
|
||||
- Anonymous users are never blocked from browser inference — auth only unlocks the
|
||||
server engine.
|
||||
|
||||
## 5. Invariants
|
||||
|
||||
- **Privacy:** browser-engine audio and transcripts never touch the network.
|
||||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||||
- **One inference codebase:** Transformers.js on both sides; model catalog shared.
|
||||
- **Worker isolation:** inference never runs on the main thread.
|
||||
Reference in new issue
Block a user