Close out M3 docs, tick M4, spec faster-whisper server engine (D-013)
- Fix: index.html <link> hrefs must be relative for Bun's HTML loader (absolute URLs failed the build and dev serve); ensureIcons() now runs before bundling in both scripts/build.ts and src/index.ts - tests/offline.md: manual airplane-mode verification procedure - WorkBreakdown: M3 checked ([~] on the manual browser run), M4 all done with notes; M5 retargeted to faster-whisper sidecar - Architecture/CLAUDE.md/Decisions: server engine = faster-whisper (CTranslate2 Python sidecar, D-013); browser stays Transformers.js (no WASM build of CTranslate2 exists) - bun test: 83 pass, tsc clean, production curl smoke-test green Co-Authored-By: Claude Fable 5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Fable 5
Claude Fable 5
parent
9f39a82982
commit
1abbc9e737
+41
-13
@@ -41,7 +41,8 @@ Logged-in users may optionally route inference to the server.
|
||||
│ ├─ /api/transcribe → server engine (auth required) │
|
||||
│ └─ /manifest.webmanifest, /sw.js → PWA assets │
|
||||
│ bun:sqlite → users, sessions │
|
||||
│ Server inference: @huggingface/transformers (Node/Bun runtime) │
|
||||
│ Server inference: faster-whisper (CTranslate2) Python sidecar, │
|
||||
│ spawned & supervised by Bun (D-013) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
@@ -71,18 +72,26 @@ src/
|
||||
inference.worker.ts # Transformers.js pipeline, WebGPU/WASM selection
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
pwa/
|
||||
sw.ts # service worker (app-shell precache)
|
||||
sw.ts # service worker (app-shell precache, /ort runtime cache)
|
||||
precache.ts # placeholder-injection helpers shared by build + dev serve
|
||||
register.ts # SW registration + update-toast flow
|
||||
manifest.webmanifest
|
||||
icons/ # generated by scripts/generate-icons.ts (not checked in)
|
||||
server/
|
||||
auth.ts # Passport.js wiring + session middleware adapter
|
||||
db.ts # bun:sqlite schema + queries
|
||||
transcribe.ts # server-side ASR endpoint
|
||||
transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5)
|
||||
server-py/ # faster-whisper sidecar (Python, D-013; M5)
|
||||
tests/ # bun test
|
||||
```
|
||||
|
||||
(Current state: M0–M2 built — browser transcription with the curated model
|
||||
catalog, WebGPU→WASM auto-selection, model switching, and language/task options.
|
||||
`lib/engine.ts`, `pwa/`, and `server/` are TODO stubs for M3–M5; see
|
||||
(Current state: M0–M3 built — browser transcription with the curated model
|
||||
catalog, WebGPU→WASM auto-selection, model switching, language/task options,
|
||||
and the installable offline PWA (`pwa/`, `scripts/build.ts` production build,
|
||||
`dist/` serving in `src/index.ts`; browser-level offline behavior pending the
|
||||
manual run in `tests/offline.md`). M4 accounts are built (`server/db.ts`,
|
||||
`server/auth.ts`, `/api/auth/*`, `AuthForms`). `lib/engine.ts`,
|
||||
`server/transcribe.ts`, and `server-py/` are TODO for M5; see
|
||||
`WorkBreakdown.md`.)
|
||||
|
||||
## 4. Key flows
|
||||
@@ -113,18 +122,36 @@ catalog, WebGPU→WASM auto-selection, model switching, and language/task option
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
|
||||
### 4.2 Offline (PWA)
|
||||
- Service worker precaches the built app shell (HTML/JS/CSS/icons) on install.
|
||||
- Model weights live in the Cache API via Transformers.js — independent of the SW.
|
||||
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
|
||||
(hashed JS/CSS, `/`, worker bundle, manifest, icons) on install into
|
||||
`whisper-shell-<version>`, cache-first thereafter; unknown navigations fall back
|
||||
to the cached shell. `scripts/build.ts` injects the precache list + a
|
||||
content-hash version into `dist/sw.js` via `pwa/precache.ts`; the dev server
|
||||
injects an empty list ("dev"), making the dev shell network-first so HMR works.
|
||||
- onnxruntime's `.wasm`/`.mjs` binaries are served from our origin at `/ort/*`
|
||||
(never the jsdelivr CDN default) and runtime-cached cache-first into
|
||||
`whisper-runtime-<version>` — they are required for offline inference but too
|
||||
large to precache eagerly.
|
||||
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
|
||||
independent of the SW, which never touches that cache (D-005). `/api/*` is
|
||||
network-only.
|
||||
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
|
||||
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
|
||||
- Result: with the app installed and at least one model previously loaded, the full
|
||||
transcription flow works with no network and with our server down.
|
||||
transcription flow works with no network and with our server down (manual
|
||||
verification procedure: `tests/offline.md`).
|
||||
|
||||
### 4.3 Server engine (logged-in only)
|
||||
1. `EngineToggle` is visible only when a session exists.
|
||||
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
|
||||
`POST /api/transcribe` with the chosen model id.
|
||||
3. Server runs the same Transformers.js pipeline under Bun (ONNX Runtime, CPU),
|
||||
streams progress/results back (chunked response or WebSocket — see Decisions.md
|
||||
D-011 once settled).
|
||||
3. Server inference runs on **faster-whisper** (CTranslate2) in a Python sidecar
|
||||
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
|
||||
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
|
||||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||||
JSON-lines HTTP response (D-011). The shared catalog maps model ids to
|
||||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||||
the server engine reports unavailable; the browser/anonymous flow is unaffected.
|
||||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||||
UI is engine-agnostic.
|
||||
|
||||
@@ -140,5 +167,6 @@ catalog, WebGPU→WASM auto-selection, model switching, and language/task option
|
||||
- **Privacy:** browser-engine audio and transcripts never touch the network.
|
||||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||||
- **One inference codebase:** Transformers.js on both sides; model catalog shared.
|
||||
- **One browser inference codebase:** Transformers.js in the browser; the server
|
||||
engine runs faster-whisper (D-013). The model catalog stays shared across both.
|
||||
- **Worker isolation:** inference never runs on the main thread.
|
||||
|
||||
@@ -17,6 +17,7 @@ A browser-first speech-to-text web app powered by OpenAI Whisper, running throug
|
||||
|
||||
**Logged-in users (Passport.js) get all of the above, plus:**
|
||||
- A toggle to choose where inference runs: **their own device** (browser engine) or **the server** the app is deployed on (server engine).
|
||||
- The server engine MUST use [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CTranslate2) for inference — not Transformers.js (see Decisions.md D-013). faster-whisper cannot run in a browser (no WASM build of CTranslate2), so the browser engine stays on Transformers.js.
|
||||
|
||||
## 2 How to run things
|
||||
|
||||
|
||||
+60
-14
@@ -68,26 +68,72 @@ Notes (M2):
|
||||
indicator should reuse `getDownloadedModels`.
|
||||
|
||||
## M3 — PWA & offline (core)
|
||||
- [ ] `manifest.webmanifest` + icons; installability
|
||||
- [ ] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
|
||||
- [ ] Build step: emit precache manifest from `bun build` output into the SW
|
||||
- [ ] SW registration + update flow (new-version toast)
|
||||
- [ ] Offline verification: airplane-mode transcription with a previously
|
||||
- [x] `manifest.webmanifest` + icons; installability
|
||||
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
|
||||
- [x] Build step: emit precache manifest from `bun build` output into the SW
|
||||
- [x] SW registration + update flow (new-version toast)
|
||||
- [~] Offline verification: airplane-mode transcription with a previously
|
||||
downloaded model; document the manual test procedure in `tests/offline.md`
|
||||
- [ ] "Available offline" indicator per model in `ModelPicker`
|
||||
— procedure written (`tests/offline.md`), pending a manual run in a real
|
||||
browser (none in the dev environment; same caveat as M2's variants)
|
||||
- [x] "Available offline" indicator per model in `ModelPicker`
|
||||
|
||||
Notes (M3):
|
||||
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
|
||||
placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
|
||||
values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
|
||||
and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
|
||||
empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
|
||||
keeps working. SW bundles are never minified — the placeholders must survive
|
||||
verbatim.
|
||||
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
|
||||
straight from `node_modules/onnxruntime-web/dist`; prod: copied to
|
||||
`dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
|
||||
cache-first into `whisper-runtime-*` — required for offline inference, too
|
||||
big to precache. Model weights stay in Transformers.js's own
|
||||
`transformers-cache`, which the SW never touches (D-005).
|
||||
- Bun's HTML loader must resolve every `<link href>` to a real file, so
|
||||
`index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
|
||||
relatively (absolute `/...` URLs fail the build — bug found while verifying
|
||||
M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
|
||||
`/icons/*` URLs are still served for the manifest's internal icon refs.
|
||||
`ensureIcons()` must run before the HTML is bundled (icons are generated,
|
||||
not checked in).
|
||||
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
|
||||
hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
|
||||
`application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
|
||||
404 network-only. Browser-level behavior (install, airplane mode, update
|
||||
toast) is manual — see `tests/offline.md`.
|
||||
|
||||
## M4 — Accounts (Passport.js)
|
||||
- [ ] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
|
||||
- [ ] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
|
||||
- [ ] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
|
||||
- [ ] Password hashing via `Bun.password` (Argon2)
|
||||
- [ ] Frontend: login/register forms, session-aware header, logged-out ⇒ no
|
||||
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
|
||||
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
|
||||
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
|
||||
- [x] Password hashing via `Bun.password` (Argon2)
|
||||
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
|
||||
behavioral change (anonymous-first invariant)
|
||||
- [ ] Auth tests (register/login/logout, session expiry, wrong password)
|
||||
- [x] Auth tests (register/login/logout, session expiry, wrong password)
|
||||
|
||||
Notes (M4):
|
||||
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
|
||||
inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
|
||||
base64url; expiry is enforced server-side via `sessions.expires_at`, not just
|
||||
the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
|
||||
production.
|
||||
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
|
||||
table and the production `fetch` handler — new API routes go there.
|
||||
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
|
||||
`/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
|
||||
framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
|
||||
offline) fail silent to anonymous — the app is offline-first.
|
||||
|
||||
## M5 — Server engine + toggle
|
||||
- [ ] Server-side pipeline in `src/server/transcribe.ts` (Transformers.js under Bun,
|
||||
CPU; reuse the shared model catalog)
|
||||
- [ ] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
|
||||
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
|
||||
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
|
||||
model catalog to `Systran/faster-whisper-*` model ids in `src/lib/models.ts`
|
||||
- [ ] Degrade gracefully when Python/the sidecar is absent (server engine
|
||||
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
|
||||
- [ ] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
|
||||
upload size limit, request abort handling
|
||||
- [ ] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
|
||||
|
||||
Reference in New Issue
Block a user