# Work Breakdown
Milestones are ordered so that **every milestone ends in a demoable, working app**.
Within a milestone, tasks are roughly in dependency order. Check items off as they
land; add follow-up tasks where they belong rather than in a separate backlog.
Legend: `[ ]` todo · `[x]` done · `[~]` in progress
---
## WIP Tasks before usage limit cutoff:
- [x] Allow users to cancel current transcription. — Cancel button on the
file line, wired to worker termination (D-018); covers decode, pending
model download, and the pass itself. Server-engine cancel is tracked in
M6 (needs the M5 EngineToggle first).
- [x] Better user-feedback while transcription is in progress (better than
"Listening for the first few words") — the worker now posts
`transcribe-progress` (seconds processed / total, from the streamer's
chunk timestamps via the pure `TranscribeProgressTracker`); the UI shows
a percent in the heading plus a progress bar with mm:ss, and the file
flow's placeholder reads "Processing audio…" (the live wording stays
live-only).
## M0 — Scaffold cleanup & docs (small)
Current state is the raw `bun init` React template.
- [x] `Architecture.md`, `Decisions.md`, `WorkBreakdown.md` created
- [x] Remove template cruft: `APITester.tsx`, hello routes in `src/index.ts`,
Bun/React logo branding in `App.tsx` / `index.html`
- [x] Add `@huggingface/transformers` dependency
- [x] Target directory layout from `Architecture.md` §3 (empty modules with TODOs are fine)
- [x] Smoke test in `tests/` proving `bun test` runs
## M1 — Browser transcription MVP (core)
Anonymous user can transcribe an uploaded file with one hardcoded model. **This is the
riskiest milestone — do it first, learn from it.**
- [x] `src/worker/protocol.ts` — typed messages: load / load-progress / transcribe /
partial / result / error
- [x] `src/worker/inference.worker.ts` — Transformers.js ASR pipeline
(`onnx-community/whisper-tiny.en` hardcoded, WASM only for now), chunked
with timestamps, partials streamed via `WhisperTextStreamer`
- [x] `src/lib/audio.ts` — decode upload → 16 kHz mono Float32Array (D-012)
- [x] `FileDropzone` component (drag-drop + file picker, audio/* and video/*)
- [x] `TranscriptView` component — streaming partials, final text, copy button
- [x] Model download progress UI (bytes / %, per-file)
- [x] Wire into `App.tsx`; delete remaining template UI
- [x] Unit tests for audio resampling and protocol handling
Notes (M1):
- Bun 1.3 does not bundle `new Worker(new URL(...))` graphs (verified against
both `bun build` and the dev server), so `src/index.ts` bundles the worker
with `Bun.build` at startup and serves it at `/inference.worker.js`; the UI
spawns `new Worker("/inference.worker.js", { type: "module" })`. Revisit if
Bun gains native worker bundling.
- onnxruntime-web fetches its `.wasm` binaries from the jsdelivr CDN
(Transformers.js default). Fine for M1; M3 (offline) must serve/cache them
locally.
## M2 — Model choice + WebGPU (core)
- [x] `src/lib/models.ts` — curated catalog with sizes and dtype defaults (D-006)
- [x] `ModelPicker` component — size estimates, English-only vs multilingual,
downloaded-state badge (query Cache API)
- [x] WebGPU detection in worker, `device` selection with WASM fallback (D-004);
surface active backend in UI
- [x] Handle model switching (dispose old pipeline, load new)
- [x] Language / task options for multilingual models (transcribe vs translate)
- [x] Whisper translate only on models trained for it (`canTranslate`; hidden for
large-v3-turbo and `.en`, stale "translate" resets to transcribe — D-WS4a)
- [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
combinations in `models.ts`
Notes (M2):
- Quantized-variant verification is pending manual in-browser testing (no real
browser in the dev environment). `models.ts` encodes the community-known-good
combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU:
fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16
encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a
`verified` flag and note. Only `whisper-tiny.en @ wasm/q8` (exercised in M1)
is marked `verified: true`; flip the flags as manual testing confirms each.
- Downloaded badge (D-005): `src/lib/modelCache.ts` inspects Cache Storage for
transformers-cache entries with `.onnx` hits per model id; M3's offline
indicator should reuse `getDownloadedModels`.
## M3 — PWA & offline (core)
- [x] `manifest.webmanifest` + icons; installability
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
- [x] Build step: emit precache manifest from `bun build` output into the SW
- [x] SW registration + update flow (new-version toast)
- [~] Offline verification: airplane-mode transcription with a previously
downloaded model; document the manual test procedure in `tests/offline.md`
— procedure written (`tests/offline.md`), pending a manual run in a real
browser (none in the dev environment; same caveat as M2's variants)
- [x] "Available offline" indicator per model in `ModelPicker`
Notes (M3):
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
keeps working. SW bundles are never minified — the placeholders must survive
verbatim.
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
straight from `node_modules/onnxruntime-web/dist`; prod: copied to
`dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
cache-first into `whisper-runtime-*` — required for offline inference, too
big to precache. Model weights stay in Transformers.js's own
`transformers-cache`, which the SW never touches (D-005).
- Bun's HTML loader must resolve every `` to a real file, so
`index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
relatively (absolute `/...` URLs fail the build — bug found while verifying
M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
`/icons/*` URLs are still served for the manifest's internal icon refs.
`ensureIcons()` must run before the HTML is bundled (icons are generated,
not checked in).
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
`application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
404 network-only. Browser-level behavior (install, airplane mode, update
toast) is manual — see `tests/offline.md`.
## M4 — Accounts (Passport.js)
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
- [x] Password hashing via `Bun.password` (Argon2)
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
behavioral change (anonymous-first invariant)
- [x] Auth tests (register/login/logout, session expiry, wrong password)
Notes (M4):
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
base64url; expiry is enforced server-side via `sessions.expires_at`, not just
the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
production.
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
table and the production `fetch` handler — new API routes go there.
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
`/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
offline) fail silent to anonymous — the app is offline-first.
## M5 — Server engine + toggle
- [x] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
model catalog to faster-whisper model ids in `src/lib/models.ts`
- [x] Degrade gracefully when Python/the sidecar is absent (server engine
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
- [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
upload size limit, request abort handling; plus `GET /api/transcribe/status`
(availability probe for the toggle)
- [x] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
`ServerEngine` (wraps fetch stream) (D-010)
- [x] `EngineToggle` component — visible only with a session; persists choice
(persistence helpers + status probe already in `lib/engine.ts`; remaining:
the component, lifting `useAuth` out of `AuthForms` into `App` so both see
one session state, and routing the file flow through the selected engine —
live stays on `BrowserEngine` per D-016)
- [x] Server model caching/warm-up strategy (don't reload weights per request)
- [~] Integration test: same fixture audio through both engines, transcripts agree
— supervisor/HTTP layers covered in `tests/transcribe.test.ts` (real
`Bun.spawn` against `tests/fixtures/fake_sidecar.py`); real inference is
the opt-in `FW_INTEGRATION=1` test + manual run in `tests/server-engine.md`
(no pip/browser in the dev environment; same caveat as M2/M3)
- [x] Deployment: optional Python/faster-whisper layer for the Docker image +
deployer note (base image stays Bun-only; install
`server-py/requirements.txt` to enable the server engine)
Notes (M5):
- Sidecar protocol (`server-py/sidecar.py`): JSON-lines over stdio —
`ready`/`fatal` handshake, `transcribe` → `segment`* → `done`, `cancel`
between segments, `shutdown`/EOF exits. Audio is handed over as a temp file
of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes),
read with `np.fromfile` — no container decoding server-side. The last-used
model stays loaded (replaced on change, so memory is bounded to one model);
device/compute via `FASTER_WHISPER_DEVICE`/`FASTER_WHISPER_COMPUTE`
(default auto/int8).
- Supervision (`createSidecar`): lazy spawn on first use; a crash rejects
in-flight requests and the next request respawns; a missing runtime
(no python3 / `fatal` handshake) is remembered as a *permanent* unavailable
reason — surfaced verbatim by `GET /api/transcribe/status` and 503s — so a
Bun-only deployment never respawn-storms. Tests force this path with
`python3 -S -I` (site-packages off) against the real sidecar.
- Upload limit: `TRANSCRIBE_MAX_UPLOAD_BYTES` (default 512 MiB ≈ 2¼ h of raw
f32 PCM), enforced pre-read via Content-Length *and* while streaming the
body to the temp file. Errors after the 200 header travel in-stream as an
`{"type":"error"}` line (wire format shared with the client via
`src/lib/engine.ts`, so server and `ServerEngine` cannot drift).
- `AUTH_ENABLED=false` (D-015) 404s `/api/transcribe*` along with the auth
routes — the server engine is session-gated, so it disappears with auth.
- Smoke-tested live against the dev server: `/` + auth unchanged; status
401 → (register) → 200 `{available:false, reason:"faster-whisper is not
installed…"}`; POST → 503 with the same reason, body untouched.
- EngineToggle: one `useAuth()` lives in `App` (passed into `AuthForms` and
gating the toggle); the status probe runs on sign-in and the server option
renders disabled-with-reason when unavailable. The file flow now goes
through the `Engine` interface for **both** paths (D-010): `BrowserEngine`
shares the worker with the live loop (distinct requestId namespaces) and a
new `stream` flag on the transcribe message keeps token partials +
transcribe-progress flowing for id-carrying engine passes (live passes stay
non-streaming). Server runs skip the browser model load, show segment
partials without a progress bar (faster-whisper's stream carries no window
boundaries), and cancel by aborting the fetch (D-018 amendment) — browser
runs keep worker-termination cancel. The choice persists in localStorage
but only takes effect while signed in with the engine available.
- Deployment: multi-stage `Dockerfile` — `runtime` (Bun-only, ~400-450 MB, see
D-031) is the default target so plain `docker build .` (existing CI) is
unchanged; `--target with-server-engine` (several GB — two Whisper models
pre-downloaded at build time, D-031) adds python3 + faster-whisper in a venv
at `/opt/faster-whisper` (wired via `SIDECAR_PYTHON`), with
`FASTER_WHISPER_DOWNLOAD_ROOT` pointed at `/opt/faster-whisper/models`
(deliberately outside the `/app/data` volume, D-031) so weights persist
without a volume and there's no first-request download. README.md documents
both flavors, all runtime env vars, the DB volume, and the HTTPS/reverse-proxy
requirement. Verified live in Docker:
Bun-only image degrades per D-013 (`available:false`, "python3 is not
available…"); server-engine image ran a real `POST /api/transcribe` of
`en_aiproblem.m4a` end-to-end, streamed segments matching the fixture
transcript — the server half of the M5 integration-test item is now
exercised for real (browser half still needs the manual run).
## M6 — Hardening & polish
- [x] Error taxonomy: unsupported file, out-of-memory (suggest smaller model),
WebGPU device-lost, network loss mid-download (resume or clear retry) —
pure `classifyError` in `src/lib/errors.ts` (D-030) sorts every failure
path (decode, load, transcribe, a crashed worker) into one of the four
categories plus a generic fallback, each with a user-facing message and
an actionable suggestion; the error box in `App.tsx` adds a Retry
button for the two retryable kinds (network, device-lost), replaying
the failed load or the last file. Unit-tested in `tests/errors.test.ts`.
- [x] Cancel in-flight transcription (both engines) — browser engine via
worker termination (D-018), server engine via aborting the
`/api/transcribe` fetch (D-018 amendment; landed with M5's EngineToggle)
- [x] Export transcript: .txt and .srt (timestamps already available) —
pure formatters in `src/lib/export.ts` (unit-tested: SRT cue numbering,
HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start
or fall back to +2 s, filename derived from the source file); .txt/.srt
download buttons next to Copy in `TranscriptView`. Live-mode results
reuse the same `result` state with absolute timestamps (D-016), so both
flows export identically.
- [x] Storage management UI: list cached models, per-model delete —
collapsible "Storage" panel (`StorageManager`) listing cached models
with exact on-disk size (summed `blob().size`, measured lazily on
open) and per-model delete; delete is confirm-free (weights are just
cache — they re-download on next use) and refreshes the picker's
downloaded/offline badges. Helpers in `src/lib/modelCache.ts` take an
injectable Cache-Storage-shaped object (mocked in
`tests/modelCache.test.ts`). Scope is the transformers-cache only:
the SW's `whisper-runtime-*` cache (onnxruntime .wasm, shared by all
models, versioned/cleaned by the SW itself) is deliberately not
surfaced — deleting it per-model makes no sense and the SW would
just re-fetch it.
- [x] Model downloads are user-initiated (D-020): selecting a model/backend no
longer auto-downloads weights; the load starts on browser-engine
transcription start (file drop / live start) or via an explicit
per-model Download button in `ModelPicker` — and never when the engine
toggle is set to "server".
- [x] Firefox performance: never run the WASM fallback single-threaded after
a failed accelerator attempt (worker posts `load-restart`, UI respawns
with explicit wasm — D-025); skip WebGPU when fp16 weights need the
absent `shader-f16` feature; hint that medium/large lag in live mode on
WASM. Field-verified: WASM now gets the full thread pool; ort-web's
WebGPU EP on Firefox completes but runs far slower than WASM, so
"auto" skips WebGPU on Firefox (D-026, explicit selection still
honored — retest on ort bumps).
- [x] Mobile OOM crash on Medium+ model loads (D-027) — patched
`@huggingface/transformers` (`bun patch`) to load the encoder/decoder
sessions sequentially instead of concurrently, roughly halving
load-time peak memory for every model; added a mobile UA +
`deviceMemory` warn/confirm gate (`src/lib/mobileMemoryGuard.ts`)
before any browser-engine load whose estimated peak
(`estimatePeakLoadBytes`) is risky on a phone/tablet, covering the
explicit Download button and the lazy auto-load on file-drop/live-start
alike.
- [x] Basic a11y pass (keyboard, focus, aria on progress/live transcript) —
the model-download and transcription `