whisper_web/WorkBreakdown.md

# Work Breakdown

Milestones are ordered so that **every milestone ends in a demoable, working app**.
Within a milestone, tasks are roughly in dependency order. Check items off as they
land; add follow-up tasks where they belong rather than in a separate backlog.

Legend: `[ ]` todo · `[x]` done · `[~]` in progress

---

## M0 — Scaffold cleanup & docs (small)
Current state is the raw `bun init` React template.

- [x] `Architecture.md`, `Decisions.md`, `WorkBreakdown.md` created
- [x] Remove template cruft: `APITester.tsx`, hello routes in `src/index.ts`,
      Bun/React logo branding in `App.tsx` / `index.html`
- [x] Add `@huggingface/transformers` dependency
- [x] Target directory layout from `Architecture.md` §3 (empty modules with TODOs are fine)
- [x] Smoke test in `tests/` proving `bun test` runs

## M1 — Browser transcription MVP (core)
Anonymous user can transcribe an uploaded file with one hardcoded model. **This is the
riskiest milestone — do it first, learn from it.**

- [x] `src/worker/protocol.ts` — typed messages: load / load-progress / transcribe /
      partial / result / error
- [x] `src/worker/inference.worker.ts` — Transformers.js ASR pipeline
      (`onnx-community/whisper-tiny.en` hardcoded, WASM only for now), chunked
      with timestamps, partials streamed via `WhisperTextStreamer`
- [x] `src/lib/audio.ts` — decode upload → 16 kHz mono Float32Array (D-012)
- [x] `FileDropzone` component (drag-drop + file picker, audio/* and video/*)
- [x] `TranscriptView` component — streaming partials, final text, copy button
- [x] Model download progress UI (bytes / %, per-file)
- [x] Wire into `App.tsx`; delete remaining template UI
- [x] Unit tests for audio resampling and protocol handling

Notes (M1):
- Bun 1.3 does not bundle `new Worker(new URL(...))` graphs (verified against
  both `bun build` and the dev server), so `src/index.ts` bundles the worker
  with `Bun.build` at startup and serves it at `/inference.worker.js`; the UI
  spawns `new Worker("/inference.worker.js", { type: "module" })`. Revisit if
  Bun gains native worker bundling.
- onnxruntime-web fetches its `.wasm` binaries from the jsdelivr CDN
  (Transformers.js default). Fine for M1; M3 (offline) must serve/cache them
  locally.

## M2 — Model choice + WebGPU (core)
- [x] `src/lib/models.ts` — curated catalog with sizes and dtype defaults (D-006)
- [x] `ModelPicker` component — size estimates, English-only vs multilingual,
      downloaded-state badge (query Cache API)
- [x] WebGPU detection in worker, `device` selection with WASM fallback (D-004);
      surface active backend in UI
- [x] Handle model switching (dispose old pipeline, load new)
- [x] Language / task options for multilingual models (transcribe vs translate)
- [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
      combinations in `models.ts`

Notes (M2):
- Quantized-variant verification is pending manual in-browser testing (no real
  browser in the dev environment). `models.ts` encodes the community-known-good
  combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU:
  fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16
  encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a
  `verified` flag and note. Only `whisper-tiny.en @ wasm/q8` (exercised in M1)
  is marked `verified: true`; flip the flags as manual testing confirms each.
- Downloaded badge (D-005): `src/lib/modelCache.ts` inspects Cache Storage for
  transformers-cache entries with `.onnx` hits per model id; M3's offline
  indicator should reuse `getDownloadedModels`.

## M3 — PWA & offline (core)
- [x] `manifest.webmanifest` + icons; installability
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
- [x] Build step: emit precache manifest from `bun build` output into the SW
- [x] SW registration + update flow (new-version toast)
- [~] Offline verification: airplane-mode transcription with a previously
      downloaded model; document the manual test procedure in `tests/offline.md`
      — procedure written (`tests/offline.md`), pending a manual run in a real
      browser (none in the dev environment; same caveat as M2's variants)
- [x] "Available offline" indicator per model in `ModelPicker`

Notes (M3):
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
  placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
  values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
  and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
  empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
  keeps working. SW bundles are never minified — the placeholders must survive
  verbatim.
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
  straight from `node_modules/onnxruntime-web/dist`; prod: copied to
  `dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
  cache-first into `whisper-runtime-*` — required for offline inference, too
  big to precache. Model weights stay in Transformers.js's own
  `transformers-cache`, which the SW never touches (D-005).
- Bun's HTML loader must resolve every `<link href>` to a real file, so
  `index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
  relatively (absolute `/...` URLs fail the build — bug found while verifying
  M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
  `/icons/*` URLs are still served for the manifest's internal icon refs.
  `ensureIcons()` must run before the HTML is bundled (icons are generated,
  not checked in).
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
  hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
  `application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
  404 network-only. Browser-level behavior (install, airplane mode, update
  toast) is manual — see `tests/offline.md`.

## M4 — Accounts (Passport.js)
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
- [x] Password hashing via `Bun.password` (Argon2)
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
      behavioral change (anonymous-first invariant)
- [x] Auth tests (register/login/logout, session expiry, wrong password)

Notes (M4):
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
  inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
  base64url; expiry is enforced server-side via `sessions.expires_at`, not just
  the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
  production.
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
  table and the production `fetch` handler — new API routes go there.
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
  `/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
  framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
  offline) fail silent to anonymous — the app is offline-first.

## M5 — Server engine + toggle
- [ ] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
      (CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
      Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
      model catalog to `Systran/faster-whisper-*` model ids in `src/lib/models.ts`
- [ ] Degrade gracefully when Python/the sidecar is absent (server engine
      unavailable; browser engine untouched — anonymous path deploys with Bun alone)
- [ ] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
      upload size limit, request abort handling
- [ ] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
      `ServerEngine` (wraps fetch stream) (D-010)
- [ ] `EngineToggle` component — visible only with a session; persists choice
- [ ] Server model caching/warm-up strategy (don't reload weights per request)
- [ ] Integration test: same fixture audio through both engines, transcripts agree

## M6 — Hardening & polish
- [ ] Error taxonomy: unsupported file, out-of-memory (suggest smaller model),
      WebGPU device-lost, network loss mid-download (resume or clear retry)
- [ ] Cancel in-flight transcription (both engines)
- [ ] Export transcript: .txt and .srt (timestamps already available)
- [ ] Storage management UI: list cached models, per-model delete
- [ ] Basic a11y pass (keyboard, focus, aria on progress/live transcript)
- [ ] Production build + `bun start` deployment check; README for deployers

---

## Deliberately out of scope (for now)
Microphone/live transcription, arbitrary Hub model ids, OAuth providers,
multi-user transcript storage, whisper.cpp server backend, i18n of the UI.
Promote to a milestone via a new entry in `Decisions.md` first.