whisper_web/docs/WorkBreakdown.md

Work Breakdown

Milestones are ordered so that every milestone ends in a demoable, working app. Within a milestone, tasks are roughly in dependency order. Check items off as they land; add follow-up tasks where they belong rather than in a separate backlog.

Legend: [ ] todo · [x] done · [~] in progress

---

WIP Tasks before usage limit cutoff:

• Allow users to cancel current transcription. — Cancel button on the file line, wired to worker termination (D-018); covers decode, pending model download, and the pass itself. Server-engine cancel is tracked in M6 (needs the M5 EngineToggle first).
• Better user-feedback while transcription is in progress (better than "Listening for the first few words") — the worker now posts transcribe-progress (seconds processed / total, from the streamer's chunk timestamps via the pure TranscribeProgressTracker); the UI shows a percent in the heading plus a progress bar with mm:ss, and the file flow's placeholder reads "Processing audio…" (the live wording stays live-only).

M0 — Scaffold cleanup & docs (small)

Current state is the raw bun init React template.

• Architecture.md, Decisions.md, WorkBreakdown.md created
• Remove template cruft: APITester.tsx, hello routes in src/index.ts, Bun/React logo branding in App.tsx / index.html
• Add @huggingface/transformers dependency
• Target directory layout from Architecture.md §3 (empty modules with TODOs are fine)
• Smoke test in tests/ proving bun test runs

M1 — Browser transcription MVP (core)

Anonymous user can transcribe an uploaded file with one hardcoded model. This is the riskiest milestone — do it first, learn from it.

• src/worker/protocol.ts — typed messages: load / load-progress / transcribe / partial / result / error
• src/worker/inference.worker.ts — Transformers.js ASR pipeline (onnx-community/whisper-tiny.en hardcoded, WASM only for now), chunked with timestamps, partials streamed via WhisperTextStreamer
• src/lib/audio.ts — decode upload → 16 kHz mono Float32Array (D-012)
• FileDropzone component (drag-drop + file picker, audio/* and video/*)
• TranscriptView component — streaming partials, final text, copy button
• Model download progress UI (bytes / %, per-file)
• Wire into App.tsx; delete remaining template UI
• Unit tests for audio resampling and protocol handling

Notes (M1):

• Bun 1.3 does not bundle new Worker(new URL(...)) graphs (verified against both bun build and the dev server), so src/index.ts bundles the worker with Bun.build at startup and serves it at /inference.worker.js; the UI spawns new Worker("/inference.worker.js", { type: "module" }). Revisit if Bun gains native worker bundling.
• onnxruntime-web fetches its .wasm binaries from the jsdelivr CDN (Transformers.js default). Fine for M1; M3 (offline) must serve/cache them locally.

M2 — Model choice + WebGPU (core)

• src/lib/models.ts — curated catalog with sizes and dtype defaults (D-006)
• ModelPicker component — size estimates, English-only vs multilingual, downloaded-state badge (query Cache API)
• WebGPU detection in worker, device selection with WASM fallback (D-004); surface active backend in UI
• Handle model switching (dispose old pipeline, load new)
• Language / task options for multilingual models (transcribe vs translate)
• [~] Verify quantized variants (q4/q8/fp16) on both backends; record working combinations in models.ts

Notes (M2):

• Quantized-variant verification is pending manual in-browser testing (no real browser in the dev environment). models.ts encodes the community-known-good combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU: fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16 encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a verified flag and note. Only whisper-tiny.en @ wasm/q8 (exercised in M1) is marked verified: true; flip the flags as manual testing confirms each.
• Downloaded badge (D-005): src/lib/modelCache.ts inspects Cache Storage for transformers-cache entries with .onnx hits per model id; M3's offline indicator should reuse getDownloadedModels.

M3 — PWA & offline (core)

• manifest.webmanifest + icons; installability
• src/pwa/sw.ts — app-shell precache, cache-first static, network-only API (D-007)
• Build step: emit precache manifest from bun build output into the SW
• SW registration + update flow (new-version toast)
• [~] Offline verification: airplane-mode transcription with a previously downloaded model; document the manual test procedure in tests/offline.md — procedure written (tests/offline.md), pending a manual run in a real browser (none in the dev environment; same caveat as M2's variants)
• "Available offline" indicator per model in ModelPicker

Notes (M3):

• sw.ts ships with "__PRECACHE_MANIFEST__" / __CACHE_VERSION__ string placeholders; src/pwa/precache.ts (pure, unit-tested) splices in the real values. Production: scripts/build.ts injects the hashed dist/ asset list and a content-hash version into dist/sw.js. Dev: src/index.ts injects an empty list + "dev", so the shell is runtime-cached network-first and HMR keeps working. SW bundles are never minified — the placeholders must survive verbatim.
• onnxruntime's .wasm/.mjs are served from our origin at /ort/* (dev: straight from node_modules/onnxruntime-web/dist; prod: copied to dist/ort/) instead of the jsdelivr CDN, and the SW runtime-caches them cache-first into whisper-runtime-* — required for offline inference, too big to precache. Model weights stay in Transformers.js's own transformers-cache, which the SW never touches (D-005).
• Bun's HTML loader must resolve every <link href> to a real file, so index.html references ./pwa/manifest.webmanifest / ./pwa/icons/... relatively (absolute /... URLs fail the build — bug found while verifying M3). The bundler emits hashed copies; the fixed /manifest.webmanifest and /icons/* URLs are still served for the manifest's internal icon refs. ensureIcons() must run before the HTML is bundled (icons are generated, not checked in).
• Production smoke-tested via curl: / and deep links → shell (no-cache), hashed chunks → immutable, /sw.js → no-cache, /manifest.webmanifest → application/manifest+json, /ort/*.wasm → application/wasm, /api/* → 404 network-only. Browser-level behavior (install, airplane mode, update toast) is manual — see tests/offline.md.

M4 — Accounts (Passport.js)

• src/server/db.ts — bun:sqlite schema: users, sessions (D-009)
• Connect-compat adapter + Passport local strategy in src/server/auth.ts (D-008)
• Routes: POST /api/auth/register|login|logout, GET /api/auth/me
• Password hashing via Bun.password (Argon2)
• Frontend: login/register forms, session-aware header, logged-out ⇒ no behavioral change (anonymous-first invariant)
• Auth tests (register/login/logout, session expiry, wrong password)

Notes (M4):

• DB path env var is DB_PATH (default ./data/app.db, gitignored); tests inject createDb(":memory:") and never touch disk. Session ids are 256-bit base64url; expiry is enforced server-side via sessions.expires_at, not just the cookie. Cookie: sid, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in production.
• One shared /api/* dispatcher in src/index.ts serves both the dev routes table and the production fetch handler — new API routes go there.
• src/server/auth.ts exports getSessionUser(request) for M5's auth-guarded /api/transcribe. Frontend auth state lives in src/lib/useAuth.ts over the framework-free src/lib/authClient.ts; /api/auth/me failures (401 or offline) fail silent to anonymous — the app is offline-first.

M5 — Server engine + toggle

• Server-side pipeline in src/server/transcribe.ts — faster-whisper (CTranslate2) via a Python sidecar in server-py/, spawned/supervised by Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared model catalog to faster-whisper model ids in src/lib/models.ts
• Degrade gracefully when Python/the sidecar is absent (server engine unavailable; browser engine untouched — anonymous path deploys with Bun alone)
• POST /api/transcribe — auth-guarded, streamed JSON-lines response (D-011), upload size limit, request abort handling; plus GET /api/transcribe/status (availability probe for the toggle)
• src/lib/engine.ts — Engine interface; BrowserEngine (wraps worker) and ServerEngine (wraps fetch stream) (D-010)
• EngineToggle component — visible only with a session; persists choice (persistence helpers + status probe already in lib/engine.ts; remaining: the component, lifting useAuth out of AuthForms into App so both see one session state, and routing the file flow through the selected engine — live stays on BrowserEngine per D-016)
• Server model caching/warm-up strategy (don't reload weights per request)
• [~] Integration test: same fixture audio through both engines, transcripts agree — supervisor/HTTP layers covered in tests/transcribe.test.ts (real Bun.spawn against tests/fixtures/fake_sidecar.py); real inference is the opt-in FW_INTEGRATION=1 test + manual run in tests/server-engine.md (no pip/browser in the dev environment; same caveat as M2/M3)
• Deployment: optional Python/faster-whisper layer for the Docker image + deployer note (base image stays Bun-only; install server-py/requirements.txt to enable the server engine)

Notes (M5):

• Sidecar protocol (server-py/sidecar.py): JSON-lines over stdio — ready/fatal handshake, transcribe → segment* → done, cancel between segments, shutdown/EOF exits. Audio is handed over as a temp file of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes), read with np.fromfile — no container decoding server-side. The last-used model stays loaded (replaced on change, so memory is bounded to one model); device/compute via FASTER_WHISPER_DEVICE/FASTER_WHISPER_COMPUTE (default auto/int8).
• Supervision (createSidecar): lazy spawn on first use; a crash rejects in-flight requests and the next request respawns; a missing runtime (no python3 / fatal handshake) is remembered as a permanent unavailable reason — surfaced verbatim by GET /api/transcribe/status and 503s — so a Bun-only deployment never respawn-storms. Tests force this path with python3 -S -I (site-packages off) against the real sidecar.
• Upload limit: TRANSCRIBE_MAX_UPLOAD_BYTES (default 512 MiB ≈ 2¼ h of raw f32 PCM), enforced pre-read via Content-Length and while streaming the body to the temp file. Errors after the 200 header travel in-stream as an {"type":"error"} line (wire format shared with the client via src/lib/engine.ts, so server and ServerEngine cannot drift).
• AUTH_ENABLED=false (D-015) 404s /api/transcribe* along with the auth routes — the server engine is session-gated, so it disappears with auth.
• Smoke-tested live against the dev server: / + auth unchanged; status 401 → (register) → 200 {available:false, reason:"faster-whisper is not installed…"}; POST → 503 with the same reason, body untouched.
• EngineToggle: one useAuth() lives in App (passed into AuthForms and gating the toggle); the status probe runs on sign-in and the server option renders disabled-with-reason when unavailable. The file flow now goes through the Engine interface for both paths (D-010): BrowserEngine shares the worker with the live loop (distinct requestId namespaces) and a new stream flag on the transcribe message keeps token partials + transcribe-progress flowing for id-carrying engine passes (live passes stay non-streaming). Server runs skip the browser model load, show segment partials without a progress bar (faster-whisper's stream carries no window boundaries), and cancel by aborting the fetch (D-018 amendment) — browser runs keep worker-termination cancel. The choice persists in localStorage but only takes effect while signed in with the engine available.
• Deployment: multi-stage Dockerfile — runtime (Bun-only, ~400-450 MB, see D-031) is the default target so plain docker build . (existing CI) is unchanged; --target with-server-engine (several GB — two Whisper models pre-downloaded at build time, D-031) adds python3 + faster-whisper in a venv at /opt/faster-whisper (wired via SIDECAR_PYTHON), with FASTER_WHISPER_DOWNLOAD_ROOT pointed at /opt/faster-whisper/models (deliberately outside the /app/data volume, D-031) so weights persist without a volume and there's no first-request download. README.md documents both flavors, all runtime env vars, the DB volume, and the HTTPS/reverse-proxy requirement. Verified live in Docker: Bun-only image degrades per D-013 (available:false, "python3 is not available…"); server-engine image ran a real POST /api/transcribe of en_aiproblem.m4a end-to-end, streamed segments matching the fixture transcript — the server half of the M5 integration-test item is now exercised for real (browser half still needs the manual run).

M6 — Hardening & polish

• Error taxonomy: unsupported file, out-of-memory (suggest smaller model), WebGPU device-lost, network loss mid-download (resume or clear retry) — pure classifyError in src/lib/errors.ts (D-030) sorts every failure path (decode, load, transcribe, a crashed worker) into one of the four categories plus a generic fallback, each with a user-facing message and an actionable suggestion; the error box in App.tsx adds a Retry button for the two retryable kinds (network, device-lost), replaying the failed load or the last file. Unit-tested in tests/errors.test.ts.
• Cancel in-flight transcription (both engines) — browser engine via worker termination (D-018), server engine via aborting the /api/transcribe fetch (D-018 amendment; landed with M5's EngineToggle)
• Export transcript: .txt and .srt (timestamps already available) — pure formatters in src/lib/export.ts (unit-tested: SRT cue numbering, HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start or fall back to +2 s, filename derived from the source file); .txt/.srt download buttons next to Copy in TranscriptView. Live-mode results reuse the same result state with absolute timestamps (D-016), so both flows export identically.
• Storage management UI: list cached models, per-model delete — collapsible "Storage" panel (StorageManager) listing cached models with exact on-disk size (summed blob().size, measured lazily on open) and per-model delete; delete is confirm-free (weights are just cache — they re-download on next use) and refreshes the picker's downloaded/offline badges. Helpers in src/lib/modelCache.ts take an injectable Cache-Storage-shaped object (mocked in tests/modelCache.test.ts). Scope is the transformers-cache only: the SW's whisper-runtime-* cache (onnxruntime .wasm, shared by all models, versioned/cleaned by the SW itself) is deliberately not surfaced — deleting it per-model makes no sense and the SW would just re-fetch it.
• Model downloads are user-initiated (D-020): selecting a model/backend no longer auto-downloads weights; the load starts on browser-engine transcription start (file drop / live start) or via an explicit per-model Download button in ModelPicker — and never when the engine toggle is set to "server".
• Firefox performance: never run the WASM fallback single-threaded after a failed accelerator attempt (worker posts load-restart, UI respawns with explicit wasm — D-025); skip WebGPU when fp16 weights need the absent shader-f16 feature; hint that medium/large lag in live mode on WASM. Field-verified: WASM now gets the full thread pool; ort-web's WebGPU EP on Firefox completes but runs far slower than WASM, so "auto" skips WebGPU on Firefox (D-026, explicit selection still honored — retest on ort bumps).
• Mobile OOM crash on Medium+ model loads (D-027) — patched @huggingface/transformers (bun patch) to load the encoder/decoder sessions sequentially instead of concurrently, roughly halving load-time peak memory for every model; added a mobile UA + deviceMemory warn/confirm gate (src/lib/mobileMemoryGuard.ts) before any browser-engine load whose estimated peak (estimatePeakLoadBytes) is risky on a phone/tablet, covering the explicit Download button and the lazy auto-load on file-drop/live-start alike.
• Basic a11y pass (keyboard, focus, aria on progress/live transcript) — the model-download and transcription <progress> bars gained aria-labels (previously announced as an unnamed percentage); TranscriptView's heading is now a programmatic focus target (tabIndex={-1}) that receives focus the moment a final result lands, so keyboard/screen-reader users land on new content instead of hunting for it; the Copy button's "Copied!" state and LiveControls' status transitions (requesting mic / listening / finishing) now also announce via a visually-hidden role="status" region for users not focused on the button. Keyboard operability was already in place everywhere in scope (native buttons/inputs, FileDropzone's role="button" + onKeyDown) and needed no changes. bun test unaffected (233 pass, 1 pre-existing unrelated WER-gate failure, see Decisions.md).
• Production build + bun start deployment check; README for deployers — bun run build + bun start smoke-tested live (headers, hashed-asset immutability, /api/* routing, AUTH_ENABLED=false, server-engine graceful degradation all matched the README's claims). A real docker build . from this checkout also caught two real bugs (D-031): local Python venvs leaking into the image (858 MB, via a nested .gitignore .dockerignore doesn't read) and @huggingface/transformers +onnxruntime-node/-web (341 MB) sitting in the production dependencies despite never being imported at server runtime — both fixed, bringing the Bun-only image from a buggy 2.18 GB down to ~421 MB. Also corrected stale docs: the with-server-engine image is legitimately several GB (it bakes in two Whisper models at build time), and FASTER_WHISPER_DOWNLOAD_ROOT's real default in that image is /opt/faster-whisper/models, not under /app/data as previously written.

M7 — Live microphone transcription (browser engine, D-016)

Promoted out of the out-of-scope list via D-016. Browser engine only: audio never touches the network, anonymous-first, works offline once a model is cached.

• Protocol: optional requestId on transcribe, echoed on partial/result/error so the live loop can correlate responses (file flow unchanged without ids)
• src/lib/live.ts — pure, unit-tested segment loop: RMS energy gate, LiveSegmenter (speech onset + pre-roll, silence/max-length commits, pure silence dropped), LiveTranscriber (single-flight scheduling, final passes before throttled tentative refreshes, stop/drain, absolute-timestamped chunks)
• src/lib/mic.ts — getUserMedia + AudioWorklet chunker from a Blob URL (ScriptProcessorNode fallback), native-rate capture resampled to 16 kHz on the main thread (D-016)
• LiveControls component — start/stop button + recording indicator; mutually exclusive with the file flow
• Live transcript UI: committed text + visually-distinct tentative tail in TranscriptView; final stop() result reuses the normal transcript view (copy button, timestamps)
• Unit tests: segmenter behavior, transcriber scheduling with a fake engine and injected clock, protocol requestId guards
• [~] Manual in-browser verification (mic permission, live text, offline run) — procedure in tests/live.md, pending a real browser (same dev-env caveat as M2/M3)
• Live transcription on the server engine (D-019) — GET /api/transcribe/live upgrades to a WebSocket (src/server/live.ts, wired in src/index.ts); the client streams mic PCM and runs the browser-engine UI unchanged otherwise (App.tsx's startLive/stopLive branch on useServer). Landed already but was undocumented here and untested; brought up to the rest of the codebase's bar (D-032): createLiveWebSocket now takes the sidecar as an injected dependency like createTranscribeHandlers, the WS route validates language/task the same way the HTTP route does, and tests/server-live.test.ts covers the bridge against a fake sidecar.

M8 — Transcription quality and speed (spec 001)

• Transformers.js 4.2.0 → 4.3.2, sequential-session patch regenerated for 4.3.2 and guarded by tests/transformers-runtime.test.ts (D-WS1a).

---

Deliberately out of scope (for now)

Arbitrary Hub model ids, OAuth providers, multi-user transcript storage, whisper.cpp server backend, i18n of the UI. Promote to a milestone via a new entry in Decisions.md first.

---

Noticed, not fixed

Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29.

• bun test tests/live-real-model.test.ts fails: the en_alien (en) case times out at its 900 s limit (2026-10-29, Linux CPU/WASM, reproduced in isolation; da_festuge and en_aiproblem pass). — Not a hang; budget and WER bar were calibrated for short clips. Fixed (D-039).
• isolate_wer2.ts (repo root) is marked "TEMP (delete after use)" and writes to a stale /tmp/claude-1000/... scratchpad path. — Deleted.
• docs/Architecture.md §3 and scripts/build.ts say src/pwa/icons/ is "not checked in", but icon-192.png/icon-512.png are tracked. — Kept tracked (ensureIcons() is idempotent); comments + docs fixed.
• No typecheck or lint: tsconfig.json is strict but typescript is not installed, so nothing checks types (bun strips them). — bun run typecheck (D-038). Still no linter.
• .gitlab-ci.yml only builds Docker images; bun test never runs in CI. — check stage: typecheck + fast suite gate the build (D-038).
• README "Graceful degradation" says server-engine weights download into the data volume; the env table, Dockerfile and D-031 say they are baked into the image at /opt/faster-whisper/models. — README fixed.
• src/index.css declares color-scheme: light dark but only has a dark palette; Inter is listed first in the font stack but never loaded. — Light theme via tokens + self-hosted Inter (D-037). Also fixed: the Backend select overflowed its panel on 390 px-wide phones.

Recurring

• Maintain AGENTS.md:
  • Add a rule only in response to a real failure. When an agent repeatedly gets something wrong, add one line, ideally with a code example. Do not pre-empt hypothetical mistakes.
  • Update in the same commit as the convention change. A stale rules file actively misleads.
  • Prune quarterly. Delete rules whose convention changed or whose mistake no longer occurs. Keep it under 150 lines.
  • Optional validation: run the same task twice in clean sessions, with and without the file, and compare steps and token use.