diff --git a/Architecture.md b/Architecture.md index 18ae90c..5416e60 100644 --- a/Architecture.md +++ b/Architecture.md @@ -41,7 +41,8 @@ Logged-in users may optionally route inference to the server. │ ├─ /api/transcribe → server engine (auth required) │ │ └─ /manifest.webmanifest, /sw.js → PWA assets │ │ bun:sqlite → users, sessions │ -│ Server inference: @huggingface/transformers (Node/Bun runtime) │ +│ Server inference: faster-whisper (CTranslate2) Python sidecar, │ +│ spawned & supervised by Bun (D-013) │ └──────────────────────────────────────────────────────────────────┘ ``` @@ -71,18 +72,26 @@ src/ inference.worker.ts # Transformers.js pipeline, WebGPU/WASM selection protocol.ts # typed postMessage messages (shared with UI) pwa/ - sw.ts # service worker (app-shell precache) + sw.ts # service worker (app-shell precache, /ort runtime cache) + precache.ts # placeholder-injection helpers shared by build + dev serve + register.ts # SW registration + update-toast flow manifest.webmanifest + icons/ # generated by scripts/generate-icons.ts (not checked in) server/ auth.ts # Passport.js wiring + session middleware adapter db.ts # bun:sqlite schema + queries - transcribe.ts # server-side ASR endpoint + transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5) +server-py/ # faster-whisper sidecar (Python, D-013; M5) tests/ # bun test ``` -(Current state: M0–M2 built — browser transcription with the curated model -catalog, WebGPU→WASM auto-selection, model switching, and language/task options. -`lib/engine.ts`, `pwa/`, and `server/` are TODO stubs for M3–M5; see +(Current state: M0–M3 built — browser transcription with the curated model +catalog, WebGPU→WASM auto-selection, model switching, language/task options, +and the installable offline PWA (`pwa/`, `scripts/build.ts` production build, +`dist/` serving in `src/index.ts`; browser-level offline behavior pending the +manual run in `tests/offline.md`). M4 accounts are built (`server/db.ts`, +`server/auth.ts`, `/api/auth/*`, `AuthForms`). `lib/engine.ts`, +`server/transcribe.ts`, and `server-py/` are TODO for M5; see `WorkBreakdown.md`.) ## 4. Key flows @@ -113,18 +122,36 @@ catalog, WebGPU→WASM auto-selection, model switching, and language/task option back via `{type:"partial"}` messages; final result as `{type:"result"}`. ### 4.2 Offline (PWA) -- Service worker precaches the built app shell (HTML/JS/CSS/icons) on install. -- Model weights live in the Cache API via Transformers.js — independent of the SW. +- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell + (hashed JS/CSS, `/`, worker bundle, manifest, icons) on install into + `whisper-shell-`, cache-first thereafter; unknown navigations fall back + to the cached shell. `scripts/build.ts` injects the precache list + a + content-hash version into `dist/sw.js` via `pwa/precache.ts`; the dev server + injects an empty list ("dev"), making the dev shell network-first so HMR works. +- onnxruntime's `.wasm`/`.mjs` binaries are served from our origin at `/ort/*` + (never the jsdelivr CDN default) and runtime-cached cache-first into + `whisper-runtime-` — they are required for offline inference but too + large to precache eagerly. +- Model weights live in the Cache API (`transformers-cache`) via Transformers.js — + independent of the SW, which never touches that cache (D-005). `/api/*` is + network-only. +- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts + `SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads. - Result: with the app installed and at least one model previously loaded, the full - transcription flow works with no network and with our server down. + transcription flow works with no network and with our server down (manual + verification procedure: `tests/offline.md`). ### 4.3 Server engine (logged-in only) 1. `EngineToggle` is visible only when a session exists. 2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to `POST /api/transcribe` with the chosen model id. -3. Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU), - streams progress/results back (chunked response or WebSocket — see Decisions.md - D-011 once settled). +3. Server inference runs on **faster-whisper** (CTranslate2) in a Python sidecar + (`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no + browser/WASM build, so it is server-only — the browser stays on Transformers.js). + `src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed + JSON-lines HTTP response (D-011). The shared catalog maps model ids to + `Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent, + the server engine reports unavailable; the browser/anonymous flow is unaffected. 4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the UI is engine-agnostic. @@ -140,5 +167,6 @@ catalog, WebGPU→WASM auto-selection, model switching, and language/task option - **Privacy:** browser-engine audio and transcripts never touch the network. - **Anonymous-first:** every feature except the server engine works logged out. - **Offline-first:** app shell + previously-downloaded models must work offline. -- **One inference codebase:** Transformers.js on both sides; model catalog shared. +- **One browser inference codebase:** Transformers.js in the browser; the server + engine runs faster-whisper (D-013). The model catalog stays shared across both. - **Worker isolation:** inference never runs on the main thread. diff --git a/CLAUDE.md b/CLAUDE.md index 98b31c1..91433ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,6 +17,7 @@ A browser-first speech-to-text web app powered by OpenAI Whisper, running throug **Logged-in users (Passport.js) get all of the above, plus:** - A toggle to choose where inference runs: **their own device** (browser engine) or **the server** the app is deployed on (server engine). +- The server engine MUST use [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CTranslate2) for inference — not Transformers.js (see Decisions.md D-013). faster-whisper cannot run in a browser (no WASM build of CTranslate2), so the browser engine stays on Transformers.js. ## 2 How to run things diff --git a/WorkBreakdown.md b/WorkBreakdown.md index 3be099f..451933d 100644 --- a/WorkBreakdown.md +++ b/WorkBreakdown.md @@ -68,26 +68,72 @@ Notes (M2): indicator should reuse `getDownloadedModels`. ## M3 — PWA & offline (core) -- [ ] `manifest.webmanifest` + icons; installability -- [ ] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007) -- [ ] Build step: emit precache manifest from `bun build` output into the SW -- [ ] SW registration + update flow (new-version toast) -- [ ] Offline verification: airplane-mode transcription with a previously +- [x] `manifest.webmanifest` + icons; installability +- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007) +- [x] Build step: emit precache manifest from `bun build` output into the SW +- [x] SW registration + update flow (new-version toast) +- [~] Offline verification: airplane-mode transcription with a previously downloaded model; document the manual test procedure in `tests/offline.md` -- [ ] "Available offline" indicator per model in `ModelPicker` + — procedure written (`tests/offline.md`), pending a manual run in a real + browser (none in the dev environment; same caveat as M2's variants) +- [x] "Available offline" indicator per model in `ModelPicker` + +Notes (M3): +- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string + placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real + values. Production: `scripts/build.ts` injects the hashed `dist/` asset list + and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an + empty list + `"dev"`, so the shell is runtime-cached network-first and HMR + keeps working. SW bundles are never minified — the placeholders must survive + verbatim. +- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev: + straight from `node_modules/onnxruntime-web/dist`; prod: copied to + `dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them + cache-first into `whisper-runtime-*` — required for offline inference, too + big to precache. Model weights stay in Transformers.js's own + `transformers-cache`, which the SW never touches (D-005). +- Bun's HTML loader must resolve every `` to a real file, so + `index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...` + relatively (absolute `/...` URLs fail the build — bug found while verifying + M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and + `/icons/*` URLs are still served for the manifest's internal icon refs. + `ensureIcons()` must run before the HTML is bundled (icons are generated, + not checked in). +- Production smoke-tested via curl: `/` and deep links → shell (no-cache), + hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` → + `application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` → + 404 network-only. Browser-level behavior (install, airplane mode, update + toast) is manual — see `tests/offline.md`. ## M4 — Accounts (Passport.js) -- [ ] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009) -- [ ] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008) -- [ ] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me` -- [ ] Password hashing via `Bun.password` (Argon2) -- [ ] Frontend: login/register forms, session-aware header, logged-out ⇒ no +- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009) +- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008) +- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me` +- [x] Password hashing via `Bun.password` (Argon2) +- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no behavioral change (anonymous-first invariant) -- [ ] Auth tests (register/login/logout, session expiry, wrong password) +- [x] Auth tests (register/login/logout, session expiry, wrong password) + +Notes (M4): +- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests + inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit + base64url; expiry is enforced server-side via `sessions.expires_at`, not just + the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in + production. +- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes` + table and the production `fetch` handler — new API routes go there. +- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded + `/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the + framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or + offline) fail silent to anonymous — the app is offline-first. ## M5 — Server engine + toggle -- [ ] Server-side pipeline in `src/server/transcribe.ts` (Transformers.js under Bun, - CPU; reuse the shared model catalog) +- [ ] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper** + (CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by + Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared + model catalog to `Systran/faster-whisper-*` model ids in `src/lib/models.ts` +- [ ] Degrade gracefully when Python/the sidecar is absent (server engine + unavailable; browser engine untouched — anonymous path deploys with Bun alone) - [ ] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011), upload size limit, request abort handling - [ ] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and