whisper_web/WorkBreakdown.md

Work Breakdown

Milestones are ordered so that every milestone ends in a demoable, working app. Within a milestone, tasks are roughly in dependency order. Check items off as they land; add follow-up tasks where they belong rather than in a separate backlog.

Legend: [ ] todo · [x] done · [~] in progress

---

M0 — Scaffold cleanup & docs (small)

Current state is the raw bun init React template.

• Architecture.md, Decisions.md, WorkBreakdown.md created
• Remove template cruft: APITester.tsx, hello routes in src/index.ts, Bun/React logo branding in App.tsx / index.html
• Add @huggingface/transformers dependency
• Target directory layout from Architecture.md §3 (empty modules with TODOs are fine)
• Smoke test in tests/ proving bun test runs

M1 — Browser transcription MVP (core)

Anonymous user can transcribe an uploaded file with one hardcoded model. This is the riskiest milestone — do it first, learn from it.

• src/worker/protocol.ts — typed messages: load / load-progress / transcribe / partial / result / error
• src/worker/inference.worker.ts — Transformers.js ASR pipeline (onnx-community/whisper-tiny.en hardcoded, WASM only for now), chunked with timestamps, partials streamed via WhisperTextStreamer
• src/lib/audio.ts — decode upload → 16 kHz mono Float32Array (D-012)
• FileDropzone component (drag-drop + file picker, audio/* and video/*)
• TranscriptView component — streaming partials, final text, copy button
• Model download progress UI (bytes / %, per-file)
• Wire into App.tsx; delete remaining template UI
• Unit tests for audio resampling and protocol handling

Notes (M1):

• Bun 1.3 does not bundle new Worker(new URL(...)) graphs (verified against both bun build and the dev server), so src/index.ts bundles the worker with Bun.build at startup and serves it at /inference.worker.js; the UI spawns new Worker("/inference.worker.js", { type: "module" }). Revisit if Bun gains native worker bundling.
• onnxruntime-web fetches its .wasm binaries from the jsdelivr CDN (Transformers.js default). Fine for M1; M3 (offline) must serve/cache them locally.

M2 — Model choice + WebGPU (core)

• src/lib/models.ts — curated catalog with sizes and dtype defaults (D-006)
• ModelPicker component — size estimates, English-only vs multilingual, downloaded-state badge (query Cache API)
• WebGPU detection in worker, device selection with WASM fallback (D-004); surface active backend in UI
• Handle model switching (dispose old pipeline, load new)
• Language / task options for multilingual models (transcribe vs translate)
• [~] Verify quantized variants (q4/q8/fp16) on both backends; record working combinations in models.ts

Notes (M2):

• Quantized-variant verification is pending manual in-browser testing (no real browser in the dev environment). models.ts encodes the community-known-good combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU: fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16 encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a verified flag and note. Only whisper-tiny.en @ wasm/q8 (exercised in M1) is marked verified: true; flip the flags as manual testing confirms each.
• Downloaded badge (D-005): src/lib/modelCache.ts inspects Cache Storage for transformers-cache entries with .onnx hits per model id; M3's offline indicator should reuse getDownloadedModels.

M3 — PWA & offline (core)

• manifest.webmanifest + icons; installability
• src/pwa/sw.ts — app-shell precache, cache-first static, network-only API (D-007)
• Build step: emit precache manifest from bun build output into the SW
• SW registration + update flow (new-version toast)
• Offline verification: airplane-mode transcription with a previously downloaded model; document the manual test procedure in tests/offline.md
• "Available offline" indicator per model in ModelPicker

M4 — Accounts (Passport.js)

• src/server/db.ts — bun:sqlite schema: users, sessions (D-009)
• Connect-compat adapter + Passport local strategy in src/server/auth.ts (D-008)
• Routes: POST /api/auth/register|login|logout, GET /api/auth/me
• Password hashing via Bun.password (Argon2)
• Frontend: login/register forms, session-aware header, logged-out ⇒ no behavioral change (anonymous-first invariant)
• Auth tests (register/login/logout, session expiry, wrong password)

M5 — Server engine + toggle

• Server-side pipeline in src/server/transcribe.ts (Transformers.js under Bun, CPU; reuse the shared model catalog)
• POST /api/transcribe — auth-guarded, streamed JSON-lines response (D-011), upload size limit, request abort handling
• src/lib/engine.ts — Engine interface; BrowserEngine (wraps worker) and ServerEngine (wraps fetch stream) (D-010)
• EngineToggle component — visible only with a session; persists choice
• Server model caching/warm-up strategy (don't reload weights per request)
• Integration test: same fixture audio through both engines, transcripts agree

M6 — Hardening & polish

• Error taxonomy: unsupported file, out-of-memory (suggest smaller model), WebGPU device-lost, network loss mid-download (resume or clear retry)
• Cancel in-flight transcription (both engines)
• Export transcript: .txt and .srt (timestamps already available)
• Storage management UI: list cached models, per-model delete
• Basic a11y pass (keyboard, focus, aria on progress/live transcript)
• Production build + bun start deployment check; README for deployers

---

Deliberately out of scope (for now)

Microphone/live transcription, arbitrary Hub model ids, OAuth providers, multi-user transcript storage, whisper.cpp server backend, i18n of the UI. Promote to a milestone via a new entry in Decisions.md first.