Work Breakdown

Milestones are ordered so that every milestone ends in a demoable, working app. Within a milestone, tasks are roughly in dependency order. Check items off as they land; add follow-up tasks where they belong rather than in a separate backlog.

Legend: [ ] todo · [x] done · [~] in progress

WIP Tasks before usage limit cutoff:

Allow users to cancel current transcription. — Cancel button on the file line, wired to worker termination (D-018); covers decode, pending model download, and the pass itself. Server-engine cancel is tracked in M6 (needs the M5 EngineToggle first).
Better user-feedback while transcription is in progress (better than "Listening for the first few words") — the worker now posts transcribe-progress (seconds processed / total, from the streamer's chunk timestamps via the pure TranscribeProgressTracker); the UI shows a percent in the heading plus a progress bar with mm:ss, and the file flow's placeholder reads "Processing audio…" (the live wording stays live-only).

M0 — Scaffold cleanup & docs (small)

Current state is the raw bun init React template.

Architecture.md, Decisions.md, WorkBreakdown.md created
Remove template cruft: APITester.tsx, hello routes in src/index.ts, Bun/React logo branding in App.tsx / index.html
Add @huggingface/transformers dependency
Target directory layout from Architecture.md §3 (empty modules with TODOs are fine)
Smoke test in tests/ proving bun test runs

M1 — Browser transcription MVP (core)

Anonymous user can transcribe an uploaded file with one hardcoded model. This is the riskiest milestone — do it first, learn from it.

src/worker/protocol.ts — typed messages: load / load-progress / transcribe / partial / result / error
src/worker/inference.worker.ts — Transformers.js ASR pipeline (onnx-community/whisper-tiny.en hardcoded, WASM only for now), chunked with timestamps, partials streamed via WhisperTextStreamer
src/lib/audio.ts — decode upload → 16 kHz mono Float32Array (D-012)
FileDropzone component (drag-drop + file picker, audio/* and video/*)
TranscriptView component — streaming partials, final text, copy button
Model download progress UI (bytes / %, per-file)
Wire into App.tsx; delete remaining template UI
Unit tests for audio resampling and protocol handling

Notes (M1):

Bun 1.3 does not bundle new Worker(new URL(...)) graphs (verified against both bun build and the dev server), so src/index.ts bundles the worker with Bun.build at startup and serves it at /inference.worker.js; the UI spawns new Worker("/inference.worker.js", { type: "module" }). Revisit if Bun gains native worker bundling.
onnxruntime-web fetches its .wasm binaries from the jsdelivr CDN (Transformers.js default). Fine for M1; M3 (offline) must serve/cache them locally.

M2 — Model choice + WebGPU (core)

src/lib/models.ts — curated catalog with sizes and dtype defaults (D-006)
ModelPicker component — size estimates, English-only vs multilingual, downloaded-state badge (query Cache API)
WebGPU detection in worker, device selection with WASM fallback (D-004); surface active backend in UI
Handle model switching (dispose old pipeline, load new)
Language / task options for multilingual models (transcribe vs translate)
[~] Verify quantized variants (q4/q8/fp16) on both backends; record working combinations in models.ts

Notes (M2):

Quantized-variant verification is pending manual in-browser testing (no real browser in the dev environment). models.ts encodes the community-known-good combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU: fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16 encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a verified flag and note. Only whisper-tiny.en @ wasm/q8 (exercised in M1) is marked verified: true; flip the flags as manual testing confirms each.
Downloaded badge (D-005): src/lib/modelCache.ts inspects Cache Storage for transformers-cache entries with .onnx hits per model id; M3's offline indicator should reuse getDownloadedModels.

M3 — PWA & offline (core)

manifest.webmanifest + icons; installability
src/pwa/sw.ts — app-shell precache, cache-first static, network-only API (D-007)
Build step: emit precache manifest from bun build output into the SW
SW registration + update flow (new-version toast)
[~] Offline verification: airplane-mode transcription with a previously downloaded model; document the manual test procedure in tests/offline.md — procedure written (tests/offline.md), pending a manual run in a real browser (none in the dev environment; same caveat as M2's variants)
"Available offline" indicator per model in ModelPicker

Notes (M3):

sw.ts ships with "__PRECACHE_MANIFEST__" / __CACHE_VERSION__ string placeholders; src/pwa/precache.ts (pure, unit-tested) splices in the real values. Production: scripts/build.ts injects the hashed dist/ asset list and a content-hash version into dist/sw.js. Dev: src/index.ts injects an empty list + "dev", so the shell is runtime-cached network-first and HMR keeps working. SW bundles are never minified — the placeholders must survive verbatim.
onnxruntime's .wasm/.mjs are served from our origin at /ort/* (dev: straight from node_modules/onnxruntime-web/dist; prod: copied to dist/ort/) instead of the jsdelivr CDN, and the SW runtime-caches them cache-first into whisper-runtime-* — required for offline inference, too big to precache. Model weights stay in Transformers.js's own transformers-cache, which the SW never touches (D-005).
Bun's HTML loader must resolve every <link href> to a real file, so index.html references ./pwa/manifest.webmanifest / ./pwa/icons/... relatively (absolute /... URLs fail the build — bug found while verifying M3). The bundler emits hashed copies; the fixed /manifest.webmanifest and /icons/* URLs are still served for the manifest's internal icon refs. ensureIcons() must run before the HTML is bundled (icons are generated, not checked in).
Production smoke-tested via curl: / and deep links → shell (no-cache), hashed chunks → immutable, /sw.js → no-cache, /manifest.webmanifest → application/manifest+json, /ort/*.wasm → application/wasm, /api/* → 404 network-only. Browser-level behavior (install, airplane mode, update toast) is manual — see tests/offline.md.

M4 — Accounts (Passport.js)

src/server/db.ts — bun:sqlite schema: users, sessions (D-009)
Connect-compat adapter + Passport local strategy in src/server/auth.ts (D-008)
Routes: POST /api/auth/register|login|logout, GET /api/auth/me
Password hashing via Bun.password (Argon2)
Frontend: login/register forms, session-aware header, logged-out ⇒ no behavioral change (anonymous-first invariant)
Auth tests (register/login/logout, session expiry, wrong password)

Notes (M4):

DB path env var is DB_PATH (default ./data/app.db, gitignored); tests inject createDb(":memory:") and never touch disk. Session ids are 256-bit base64url; expiry is enforced server-side via sessions.expires_at, not just the cookie. Cookie: sid, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in production.
One shared /api/* dispatcher in src/index.ts serves both the dev routes table and the production fetch handler — new API routes go there.
src/server/auth.ts exports getSessionUser(request) for M5's auth-guarded /api/transcribe. Frontend auth state lives in src/lib/useAuth.ts over the framework-free src/lib/authClient.ts; /api/auth/me failures (401 or offline) fail silent to anonymous — the app is offline-first.

M5 — Server engine + toggle

Server-side pipeline in src/server/transcribe.ts — faster-whisper (CTranslate2) via a Python sidecar in server-py/, spawned/supervised by Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared model catalog to faster-whisper model ids in src/lib/models.ts
Degrade gracefully when Python/the sidecar is absent (server engine unavailable; browser engine untouched — anonymous path deploys with Bun alone)
POST /api/transcribe — auth-guarded, streamed JSON-lines response (D-011), upload size limit, request abort handling; plus GET /api/transcribe/status (availability probe for the toggle)
src/lib/engine.ts — Engine interface; BrowserEngine (wraps worker) and ServerEngine (wraps fetch stream) (D-010)
EngineToggle component — visible only with a session; persists choice (persistence helpers + status probe already in lib/engine.ts; remaining: the component, lifting useAuth out of AuthForms into App so both see one session state, and routing the file flow through the selected engine — live stays on BrowserEngine per D-016)
Server model caching/warm-up strategy (don't reload weights per request)
[~] Integration test: same fixture audio through both engines, transcripts agree — supervisor/HTTP layers covered in tests/transcribe.test.ts (real Bun.spawn against tests/fixtures/fake_sidecar.py); real inference is the opt-in FW_INTEGRATION=1 test + manual run in tests/server-engine.md (no pip/browser in the dev environment; same caveat as M2/M3)
Deployment: optional Python/faster-whisper layer for the Docker image + deployer note (base image stays Bun-only; install server-py/requirements.txt to enable the server engine)

Notes (M5):

Sidecar protocol (server-py/sidecar.py): JSON-lines over stdio — ready/fatal handshake, transcribe → segment* → done, cancel between segments, shutdown/EOF exits. Audio is handed over as a temp file of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes), read with np.fromfile — no container decoding server-side. The last-used model stays loaded (replaced on change, so memory is bounded to one model); device/compute via FASTER_WHISPER_DEVICE/FASTER_WHISPER_COMPUTE (default auto/int8).
Supervision (createSidecar): lazy spawn on first use; a crash rejects in-flight requests and the next request respawns; a missing runtime (no python3 / fatal handshake) is remembered as a permanent unavailable reason — surfaced verbatim by GET /api/transcribe/status and 503s — so a Bun-only deployment never respawn-storms. Tests force this path with python3 -S -I (site-packages off) against the real sidecar.
Upload limit: TRANSCRIBE_MAX_UPLOAD_BYTES (default 512 MiB ≈ 2¼ h of raw f32 PCM), enforced pre-read via Content-Length and while streaming the body to the temp file. Errors after the 200 header travel in-stream as an {"type":"error"} line (wire format shared with the client via src/lib/engine.ts, so server and ServerEngine cannot drift).
AUTH_ENABLED=false (D-015) 404s /api/transcribe* along with the auth routes — the server engine is session-gated, so it disappears with auth.
Smoke-tested live against the dev server: / + auth unchanged; status 401 → (register) → 200 {available:false, reason:"faster-whisper is not installed…"}; POST → 503 with the same reason, body untouched.
EngineToggle: one useAuth() lives in App (passed into AuthForms and gating the toggle); the status probe runs on sign-in and the server option renders disabled-with-reason when unavailable. The file flow now goes through the Engine interface for both paths (D-010): BrowserEngine shares the worker with the live loop (distinct requestId namespaces) and a new stream flag on the transcribe message keeps token partials + transcribe-progress flowing for id-carrying engine passes (live passes stay non-streaming). Server runs skip the browser model load, show segment partials without a progress bar (faster-whisper's stream carries no window boundaries), and cancel by aborting the fetch (D-018 amendment) — browser runs keep worker-termination cancel. The choice persists in localStorage but only takes effect while signed in with the engine available.
Deployment: multi-stage Dockerfile — runtime (Bun-only, 244 MB) is the default target so plain docker build . (existing CI) is unchanged; --target with-server-engine (405 MB) adds python3 + faster-whisper in a venv at /opt/faster-whisper (wired via SIDECAR_PYTHON), with FASTER_WHISPER_DOWNLOAD_ROOT under the /app/data volume so weights persist. README.md documents both flavors, all runtime env vars, the DB volume, and the HTTPS/reverse-proxy requirement. Verified live in Docker: Bun-only image degrades per D-013 (available:false, "python3 is not available…"); server-engine image ran a real POST /api/transcribe of en_aiproblem.m4a end-to-end, streamed segments matching the fixture transcript — the server half of the M5 integration-test item is now exercised for real (browser half still needs the manual run).

M6 — Hardening & polish

Error taxonomy: unsupported file, out-of-memory (suggest smaller model), WebGPU device-lost, network loss mid-download (resume or clear retry)
Cancel in-flight transcription (both engines) — browser engine via worker termination (D-018), server engine via aborting the /api/transcribe fetch (D-018 amendment; landed with M5's EngineToggle)
Export transcript: .txt and .srt (timestamps already available) — pure formatters in src/lib/export.ts (unit-tested: SRT cue numbering, HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start or fall back to +2 s, filename derived from the source file); .txt/.srt download buttons next to Copy in TranscriptView. Live-mode results reuse the same result state with absolute timestamps (D-016), so both flows export identically.
Storage management UI: list cached models, per-model delete — collapsible "Storage" panel (StorageManager) listing cached models with exact on-disk size (summed blob().size, measured lazily on open) and per-model delete; delete is confirm-free (weights are just cache — they re-download on next use) and refreshes the picker's downloaded/offline badges. Helpers in src/lib/modelCache.ts take an injectable Cache-Storage-shaped object (mocked in tests/modelCache.test.ts). Scope is the transformers-cache only: the SW's whisper-runtime-* cache (onnxruntime .wasm, shared by all models, versioned/cleaned by the SW itself) is deliberately not surfaced — deleting it per-model makes no sense and the SW would just re-fetch it.
Basic a11y pass (keyboard, focus, aria on progress/live transcript)
Production build + bun start deployment check; README for deployers

M7 — Live microphone transcription (browser engine, D-016)

Promoted out of the out-of-scope list via D-016. Browser engine only: audio never touches the network, anonymous-first, works offline once a model is cached.

Protocol: optional requestId on transcribe, echoed on partial/result/error so the live loop can correlate responses (file flow unchanged without ids)
src/lib/live.ts — pure, unit-tested segment loop: RMS energy gate, LiveSegmenter (speech onset + pre-roll, silence/max-length commits, pure silence dropped), LiveTranscriber (single-flight scheduling, final passes before throttled tentative refreshes, stop/drain, absolute-timestamped chunks)
src/lib/mic.ts — getUserMedia + AudioWorklet chunker from a Blob URL (ScriptProcessorNode fallback), native-rate capture resampled to 16 kHz on the main thread (D-016)
LiveControls component — start/stop button + recording indicator; mutually exclusive with the file flow
Live transcript UI: committed text + visually-distinct tentative tail in TranscriptView; final stop() result reuses the normal transcript view (copy button, timestamps)
Unit tests: segmenter behavior, transcriber scheduling with a fake engine and injected clock, protocol requestId guards
[~] Manual in-browser verification (mic permission, live text, offline run) — procedure in tests/live.md, pending a real browser (same dev-env caveat as M2/M3)
Future want: live transcription on the server engine — stream mic audio over a WebSocket (D-011's escape hatch) to faster-whisper (D-013) for logged-in users on weak devices. Needs its own decision entry; blocked on M5 landing first.

Deliberately out of scope (for now)

Arbitrary Hub model ids, OAuth providers, multi-user transcript storage, whisper.cpp server backend, i18n of the UI. Promote to a milestone via a new entry in Decisions.md first.

17 KiB Raw Blame History