whisper_web/WorkBreakdown.md

Work Breakdown

Milestones are ordered so that every milestone ends in a demoable, working app. Within a milestone, tasks are roughly in dependency order. Check items off as they land; add follow-up tasks where they belong rather than in a separate backlog.

Legend: [ ] todo · [x] done · [~] in progress

---

M0 — Scaffold cleanup & docs (small)

Current state is the raw bun init React template.

• Architecture.md, Decisions.md, WorkBreakdown.md created
• Remove template cruft: APITester.tsx, hello routes in src/index.ts, Bun/React logo branding in App.tsx / index.html
• Add @huggingface/transformers dependency
• Target directory layout from Architecture.md §3 (empty modules with TODOs are fine)
• Smoke test in tests/ proving bun test runs

M1 — Browser transcription MVP (core)

Anonymous user can transcribe an uploaded file with one hardcoded model. This is the riskiest milestone — do it first, learn from it.

• src/worker/protocol.ts — typed messages: load / load-progress / transcribe / partial / result / error
• src/worker/inference.worker.ts — Transformers.js ASR pipeline (onnx-community/whisper-tiny.en hardcoded, WASM only for now), chunked with timestamps, partials streamed via WhisperTextStreamer
• src/lib/audio.ts — decode upload → 16 kHz mono Float32Array (D-012)
• FileDropzone component (drag-drop + file picker, audio/* and video/*)
• TranscriptView component — streaming partials, final text, copy button
• Model download progress UI (bytes / %, per-file)
• Wire into App.tsx; delete remaining template UI
• Unit tests for audio resampling and protocol handling

Notes (M1):

• Bun 1.3 does not bundle new Worker(new URL(...)) graphs (verified against both bun build and the dev server), so src/index.ts bundles the worker with Bun.build at startup and serves it at /inference.worker.js; the UI spawns new Worker("/inference.worker.js", { type: "module" }). Revisit if Bun gains native worker bundling.
• onnxruntime-web fetches its .wasm binaries from the jsdelivr CDN (Transformers.js default). Fine for M1; M3 (offline) must serve/cache them locally.

M2 — Model choice + WebGPU (core)

• src/lib/models.ts — curated catalog with sizes and dtype defaults (D-006)
• ModelPicker component — size estimates, English-only vs multilingual, downloaded-state badge (query Cache API)
• WebGPU detection in worker, device selection with WASM fallback (D-004); surface active backend in UI
• Handle model switching (dispose old pipeline, load new)
• Language / task options for multilingual models (transcribe vs translate)
• [~] Verify quantized variants (q4/q8/fp16) on both backends; record working combinations in models.ts

Notes (M2):

• Quantized-variant verification is pending manual in-browser testing (no real browser in the dev environment). models.ts encodes the community-known-good combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU: fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16 encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a verified flag and note. Only whisper-tiny.en @ wasm/q8 (exercised in M1) is marked verified: true; flip the flags as manual testing confirms each.
• Downloaded badge (D-005): src/lib/modelCache.ts inspects Cache Storage for transformers-cache entries with .onnx hits per model id; M3's offline indicator should reuse getDownloadedModels.

M3 — PWA & offline (core)

• manifest.webmanifest + icons; installability
• src/pwa/sw.ts — app-shell precache, cache-first static, network-only API (D-007)
• Build step: emit precache manifest from bun build output into the SW
• SW registration + update flow (new-version toast)
• [~] Offline verification: airplane-mode transcription with a previously downloaded model; document the manual test procedure in tests/offline.md — procedure written (tests/offline.md), pending a manual run in a real browser (none in the dev environment; same caveat as M2's variants)
• "Available offline" indicator per model in ModelPicker

Notes (M3):

• sw.ts ships with "__PRECACHE_MANIFEST__" / __CACHE_VERSION__ string placeholders; src/pwa/precache.ts (pure, unit-tested) splices in the real values. Production: scripts/build.ts injects the hashed dist/ asset list and a content-hash version into dist/sw.js. Dev: src/index.ts injects an empty list + "dev", so the shell is runtime-cached network-first and HMR keeps working. SW bundles are never minified — the placeholders must survive verbatim.
• onnxruntime's .wasm/.mjs are served from our origin at /ort/* (dev: straight from node_modules/onnxruntime-web/dist; prod: copied to dist/ort/) instead of the jsdelivr CDN, and the SW runtime-caches them cache-first into whisper-runtime-* — required for offline inference, too big to precache. Model weights stay in Transformers.js's own transformers-cache, which the SW never touches (D-005).
• Bun's HTML loader must resolve every <link href> to a real file, so index.html references ./pwa/manifest.webmanifest / ./pwa/icons/... relatively (absolute /... URLs fail the build — bug found while verifying M3). The bundler emits hashed copies; the fixed /manifest.webmanifest and /icons/* URLs are still served for the manifest's internal icon refs. ensureIcons() must run before the HTML is bundled (icons are generated, not checked in).
• Production smoke-tested via curl: / and deep links → shell (no-cache), hashed chunks → immutable, /sw.js → no-cache, /manifest.webmanifest → application/manifest+json, /ort/*.wasm → application/wasm, /api/* → 404 network-only. Browser-level behavior (install, airplane mode, update toast) is manual — see tests/offline.md.

M4 — Accounts (Passport.js)

• src/server/db.ts — bun:sqlite schema: users, sessions (D-009)
• Connect-compat adapter + Passport local strategy in src/server/auth.ts (D-008)
• Routes: POST /api/auth/register|login|logout, GET /api/auth/me
• Password hashing via Bun.password (Argon2)
• Frontend: login/register forms, session-aware header, logged-out ⇒ no behavioral change (anonymous-first invariant)
• Auth tests (register/login/logout, session expiry, wrong password)

Notes (M4):

• DB path env var is DB_PATH (default ./data/app.db, gitignored); tests inject createDb(":memory:") and never touch disk. Session ids are 256-bit base64url; expiry is enforced server-side via sessions.expires_at, not just the cookie. Cookie: sid, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in production.
• One shared /api/* dispatcher in src/index.ts serves both the dev routes table and the production fetch handler — new API routes go there.
• src/server/auth.ts exports getSessionUser(request) for M5's auth-guarded /api/transcribe. Frontend auth state lives in src/lib/useAuth.ts over the framework-free src/lib/authClient.ts; /api/auth/me failures (401 or offline) fail silent to anonymous — the app is offline-first.

M5 — Server engine + toggle

• Server-side pipeline in src/server/transcribe.ts — faster-whisper (CTranslate2) via a Python sidecar in server-py/, spawned/supervised by Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared model catalog to Systran/faster-whisper-* model ids in src/lib/models.ts
• Degrade gracefully when Python/the sidecar is absent (server engine unavailable; browser engine untouched — anonymous path deploys with Bun alone)
• POST /api/transcribe — auth-guarded, streamed JSON-lines response (D-011), upload size limit, request abort handling
• src/lib/engine.ts — Engine interface; BrowserEngine (wraps worker) and ServerEngine (wraps fetch stream) (D-010)
• EngineToggle component — visible only with a session; persists choice
• Server model caching/warm-up strategy (don't reload weights per request)
• Integration test: same fixture audio through both engines, transcripts agree

M6 — Hardening & polish

• Error taxonomy: unsupported file, out-of-memory (suggest smaller model), WebGPU device-lost, network loss mid-download (resume or clear retry)
• Cancel in-flight transcription (both engines)
• Export transcript: .txt and .srt (timestamps already available)
• Storage management UI: list cached models, per-model delete
• Basic a11y pass (keyboard, focus, aria on progress/live transcript)
• Production build + bun start deployment check; README for deployers

---

Deliberately out of scope (for now)

Microphone/live transcription, arbitrary Hub model ids, OAuth providers, multi-user transcript storage, whisper.cpp server backend, i18n of the UI. Promote to a milestone via a new entry in Decisions.md first.