Files
whisper_web/docs/WorkBreakdown.md
T

389 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Work Breakdown
Milestones are ordered so that **every milestone ends in a demoable, working app**.
Within a milestone, tasks are roughly in dependency order. Check items off as they
land; add follow-up tasks where they belong rather than in a separate backlog.
Legend: `[ ]` todo · `[x]` done · `[~]` in progress
---
## WIP Tasks before usage limit cutoff:
- [x] Allow users to cancel current transcription. — Cancel button on the
file line, wired to worker termination (D-018); covers decode, pending
model download, and the pass itself. Server-engine cancel is tracked in
M6 (needs the M5 EngineToggle first).
- [x] Better user-feedback while transcription is in progress (better than
"Listening for the first few words") — the worker now posts
`transcribe-progress` (seconds processed / total, from the streamer's
chunk timestamps via the pure `TranscribeProgressTracker`); the UI shows
a percent in the heading plus a progress bar with mm:ss, and the file
flow's placeholder reads "Processing audio…" (the live wording stays
live-only).
## M0 — Scaffold cleanup & docs (small)
Current state is the raw `bun init` React template.
- [x] `Architecture.md`, `Decisions.md`, `WorkBreakdown.md` created
- [x] Remove template cruft: `APITester.tsx`, hello routes in `src/index.ts`,
Bun/React logo branding in `App.tsx` / `index.html`
- [x] Add `@huggingface/transformers` dependency
- [x] Target directory layout from `Architecture.md` §3 (empty modules with TODOs are fine)
- [x] Smoke test in `tests/` proving `bun test` runs
## M1 — Browser transcription MVP (core)
Anonymous user can transcribe an uploaded file with one hardcoded model. **This is the
riskiest milestone — do it first, learn from it.**
- [x] `src/worker/protocol.ts` — typed messages: load / load-progress / transcribe /
partial / result / error
- [x] `src/worker/inference.worker.ts` — Transformers.js ASR pipeline
(`onnx-community/whisper-tiny.en` hardcoded, WASM only for now), chunked
with timestamps, partials streamed via `WhisperTextStreamer`
- [x] `src/lib/audio.ts` — decode upload → 16 kHz mono Float32Array (D-012)
- [x] `FileDropzone` component (drag-drop + file picker, audio/* and video/*)
- [x] `TranscriptView` component — streaming partials, final text, copy button
- [x] Model download progress UI (bytes / %, per-file)
- [x] Wire into `App.tsx`; delete remaining template UI
- [x] Unit tests for audio resampling and protocol handling
Notes (M1):
- Bun 1.3 does not bundle `new Worker(new URL(...))` graphs (verified against
both `bun build` and the dev server), so `src/index.ts` bundles the worker
with `Bun.build` at startup and serves it at `/inference.worker.js`; the UI
spawns `new Worker("/inference.worker.js", { type: "module" })`. Revisit if
Bun gains native worker bundling.
- onnxruntime-web fetches its `.wasm` binaries from the jsdelivr CDN
(Transformers.js default). Fine for M1; M3 (offline) must serve/cache them
locally.
## M2 — Model choice + WebGPU (core)
- [x] `src/lib/models.ts` — curated catalog with sizes and dtype defaults (D-006)
- [x] `ModelPicker` component — size estimates, English-only vs multilingual,
downloaded-state badge (query Cache API)
- [x] WebGPU detection in worker, `device` selection with WASM fallback (D-004);
surface active backend in UI
- [x] Handle model switching (dispose old pipeline, load new)
- [x] Language / task options for multilingual models (transcribe vs translate)
- [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
combinations in `models.ts`
Notes (M2):
- Quantized-variant verification is pending manual in-browser testing (no real
browser in the dev environment). `models.ts` encodes the community-known-good
combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU:
fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16
encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a
`verified` flag and note. Only `whisper-tiny.en @ wasm/q8` (exercised in M1)
is marked `verified: true`; flip the flags as manual testing confirms each.
- Downloaded badge (D-005): `src/lib/modelCache.ts` inspects Cache Storage for
transformers-cache entries with `.onnx` hits per model id; M3's offline
indicator should reuse `getDownloadedModels`.
## M3 — PWA & offline (core)
- [x] `manifest.webmanifest` + icons; installability
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
- [x] Build step: emit precache manifest from `bun build` output into the SW
- [x] SW registration + update flow (new-version toast)
- [~] Offline verification: airplane-mode transcription with a previously
downloaded model; document the manual test procedure in `tests/offline.md`
— procedure written (`tests/offline.md`), pending a manual run in a real
browser (none in the dev environment; same caveat as M2's variants)
- [x] "Available offline" indicator per model in `ModelPicker`
Notes (M3):
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
keeps working. SW bundles are never minified — the placeholders must survive
verbatim.
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
straight from `node_modules/onnxruntime-web/dist`; prod: copied to
`dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
cache-first into `whisper-runtime-*` — required for offline inference, too
big to precache. Model weights stay in Transformers.js's own
`transformers-cache`, which the SW never touches (D-005).
- Bun's HTML loader must resolve every `<link href>` to a real file, so
`index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
relatively (absolute `/...` URLs fail the build — bug found while verifying
M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
`/icons/*` URLs are still served for the manifest's internal icon refs.
`ensureIcons()` must run before the HTML is bundled (icons are generated,
not checked in).
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
`application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
404 network-only. Browser-level behavior (install, airplane mode, update
toast) is manual — see `tests/offline.md`.
## M4 — Accounts (Passport.js)
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
- [x] Password hashing via `Bun.password` (Argon2)
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
behavioral change (anonymous-first invariant)
- [x] Auth tests (register/login/logout, session expiry, wrong password)
Notes (M4):
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
base64url; expiry is enforced server-side via `sessions.expires_at`, not just
the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
production.
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
table and the production `fetch` handler — new API routes go there.
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
`/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
offline) fail silent to anonymous — the app is offline-first.
## M5 — Server engine + toggle
- [x] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
model catalog to faster-whisper model ids in `src/lib/models.ts`
- [x] Batched inference + tuned Silero VAD for server-engine file
transcriptions, live passes sequential (spec 001 Goal 5, D-WS5a):
≈ 2× faster on CPU int8 at equal-or-better WER; segments ≤ 30 s chunks
- [x] Degrade gracefully when Python/the sidecar is absent (server engine
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
- [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
upload size limit, request abort handling; plus `GET /api/transcribe/status`
(availability probe for the toggle)
- [x] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
`ServerEngine` (wraps fetch stream) (D-010)
- [x] `EngineToggle` component — visible only with a session; persists choice
(persistence helpers + status probe already in `lib/engine.ts`; remaining:
the component, lifting `useAuth` out of `AuthForms` into `App` so both see
one session state, and routing the file flow through the selected engine —
live stays on `BrowserEngine` per D-016)
- [x] Server model caching/warm-up strategy (don't reload weights per request)
- [~] Integration test: same fixture audio through both engines, transcripts agree
— supervisor/HTTP layers covered in `tests/transcribe.test.ts` (real
`Bun.spawn` against `tests/fixtures/fake_sidecar.py`); real inference is
the opt-in `FW_INTEGRATION=1` test + manual run in `tests/server-engine.md`
(no pip/browser in the dev environment; same caveat as M2/M3)
- [x] Deployment: optional Python/faster-whisper layer for the Docker image +
deployer note (base image stays Bun-only; install
`server-py/requirements.txt` to enable the server engine)
Notes (M5):
- Sidecar protocol (`server-py/sidecar.py`): JSON-lines over stdio —
`ready`/`fatal` handshake, `transcribe` → `segment`* → `done`, `cancel`
between segments, `shutdown`/EOF exits. Audio is handed over as a temp file
of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes),
read with `np.fromfile` — no container decoding server-side. The last-used
model stays loaded (replaced on change, so memory is bounded to one model);
device/compute via `FASTER_WHISPER_DEVICE`/`FASTER_WHISPER_COMPUTE`
(default auto/int8).
- Supervision (`createSidecar`): lazy spawn on first use; a crash rejects
in-flight requests and the next request respawns; a missing runtime
(no python3 / `fatal` handshake) is remembered as a *permanent* unavailable
reason — surfaced verbatim by `GET /api/transcribe/status` and 503s — so a
Bun-only deployment never respawn-storms. Tests force this path with
`python3 -S -I` (site-packages off) against the real sidecar.
- Upload limit: `TRANSCRIBE_MAX_UPLOAD_BYTES` (default 512 MiB ≈ 2¼ h of raw
f32 PCM), enforced pre-read via Content-Length *and* while streaming the
body to the temp file. Errors after the 200 header travel in-stream as an
`{"type":"error"}` line (wire format shared with the client via
`src/lib/engine.ts`, so server and `ServerEngine` cannot drift).
- `AUTH_ENABLED=false` (D-015) 404s `/api/transcribe*` along with the auth
routes — the server engine is session-gated, so it disappears with auth.
- Smoke-tested live against the dev server: `/` + auth unchanged; status
401 → (register) → 200 `{available:false, reason:"faster-whisper is not
installed…"}`; POST → 503 with the same reason, body untouched.
- EngineToggle: one `useAuth()` lives in `App` (passed into `AuthForms` and
gating the toggle); the status probe runs on sign-in and the server option
renders disabled-with-reason when unavailable. The file flow now goes
through the `Engine` interface for **both** paths (D-010): `BrowserEngine`
shares the worker with the live loop (distinct requestId namespaces) and a
new `stream` flag on the transcribe message keeps token partials +
transcribe-progress flowing for id-carrying engine passes (live passes stay
non-streaming). Server runs skip the browser model load, show segment
partials without a progress bar (faster-whisper's stream carries no window
boundaries), and cancel by aborting the fetch (D-018 amendment) — browser
runs keep worker-termination cancel. The choice persists in localStorage
but only takes effect while signed in with the engine available.
- Deployment: multi-stage `Dockerfile` — `runtime` (Bun-only, ~400-450 MB, see
D-031) is the default target so plain `docker build .` (existing CI) is
unchanged; `--target with-server-engine` (several GB — two Whisper models
pre-downloaded at build time, D-031) adds python3 + faster-whisper in a venv
at `/opt/faster-whisper` (wired via `SIDECAR_PYTHON`), with
`FASTER_WHISPER_DOWNLOAD_ROOT` pointed at `/opt/faster-whisper/models`
(deliberately outside the `/app/data` volume, D-031) so weights persist
without a volume and there's no first-request download. README.md documents
both flavors, all runtime env vars, the DB volume, and the HTTPS/reverse-proxy
requirement. Verified live in Docker:
Bun-only image degrades per D-013 (`available:false`, "python3 is not
available…"); server-engine image ran a real `POST /api/transcribe` of
`en_aiproblem.m4a` end-to-end, streamed segments matching the fixture
transcript — the server half of the M5 integration-test item is now
exercised for real (browser half still needs the manual run).
## M6 — Hardening & polish
- [x] Error taxonomy: unsupported file, out-of-memory (suggest smaller model),
WebGPU device-lost, network loss mid-download (resume or clear retry) —
pure `classifyError` in `src/lib/errors.ts` (D-030) sorts every failure
path (decode, load, transcribe, a crashed worker) into one of the four
categories plus a generic fallback, each with a user-facing message and
an actionable suggestion; the error box in `App.tsx` adds a Retry
button for the two retryable kinds (network, device-lost), replaying
the failed load or the last file. Unit-tested in `tests/errors.test.ts`.
- [x] Cancel in-flight transcription (both engines) — browser engine via
worker termination (D-018), server engine via aborting the
`/api/transcribe` fetch (D-018 amendment; landed with M5's EngineToggle)
- [x] Export transcript: .txt and .srt (timestamps already available) —
pure formatters in `src/lib/export.ts` (unit-tested: SRT cue numbering,
HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start
or fall back to +2 s, filename derived from the source file); .txt/.srt
download buttons next to Copy in `TranscriptView`. Live-mode results
reuse the same `result` state with absolute timestamps (D-016), so both
flows export identically.
- [x] Storage management UI: list cached models, per-model delete —
collapsible "Storage" panel (`StorageManager`) listing cached models
with exact on-disk size (summed `blob().size`, measured lazily on
open) and per-model delete; delete is confirm-free (weights are just
cache — they re-download on next use) and refreshes the picker's
downloaded/offline badges. Helpers in `src/lib/modelCache.ts` take an
injectable Cache-Storage-shaped object (mocked in
`tests/modelCache.test.ts`). Scope is the transformers-cache only:
the SW's `whisper-runtime-*` cache (onnxruntime .wasm, shared by all
models, versioned/cleaned by the SW itself) is deliberately not
surfaced — deleting it per-model makes no sense and the SW would
just re-fetch it.
- [x] Model downloads are user-initiated (D-020): selecting a model/backend no
longer auto-downloads weights; the load starts on browser-engine
transcription start (file drop / live start) or via an explicit
per-model Download button in `ModelPicker` — and never when the engine
toggle is set to "server".
- [x] Firefox performance: never run the WASM fallback single-threaded after
a failed accelerator attempt (worker posts `load-restart`, UI respawns
with explicit wasm — D-025); skip WebGPU when fp16 weights need the
absent `shader-f16` feature; hint that medium/large lag in live mode on
WASM. Field-verified: WASM now gets the full thread pool; ort-web's
WebGPU EP on Firefox completes but runs far slower than WASM, so
"auto" skips WebGPU on Firefox (D-026, explicit selection still
honored — retest on ort bumps).
- [x] Mobile OOM crash on Medium+ model loads (D-027) — patched
`@huggingface/transformers` (`bun patch`) to load the encoder/decoder
sessions sequentially instead of concurrently, roughly halving
load-time peak memory for every model; added a mobile UA +
`deviceMemory` warn/confirm gate (`src/lib/mobileMemoryGuard.ts`)
before any browser-engine load whose estimated peak
(`estimatePeakLoadBytes`) is risky on a phone/tablet, covering the
explicit Download button and the lazy auto-load on file-drop/live-start
alike.
- [x] Basic a11y pass (keyboard, focus, aria on progress/live transcript) —
the model-download and transcription `<progress>` bars gained
`aria-label`s (previously announced as an unnamed percentage);
`TranscriptView`'s heading is now a programmatic focus target
(`tabIndex={-1}`) that receives focus the moment a final result lands,
so keyboard/screen-reader users land on new content instead of hunting
for it; the Copy button's "Copied!" state and `LiveControls`' status
transitions (requesting mic / listening / finishing) now also announce
via a visually-hidden `role="status"` region for users not focused on
the button. Keyboard operability was already in place everywhere in
scope (native buttons/inputs, `FileDropzone`'s `role="button"` +
`onKeyDown`) and needed no changes. `bun test` unaffected (233 pass, 1
pre-existing unrelated WER-gate failure, see Decisions.md).
- [x] Production build + `bun start` deployment check; README for deployers —
`bun run build` + `bun start` smoke-tested live (headers, hashed-asset
immutability, `/api/*` routing, `AUTH_ENABLED=false`, server-engine
graceful degradation all matched the README's claims). A real
`docker build .` from this checkout also caught two real bugs (D-031):
local Python venvs leaking into the image (858 MB, via a nested
`.gitignore` `.dockerignore` doesn't read) and `@huggingface/transformers`
+`onnxruntime-node`/`-web` (341 MB) sitting in the production
`dependencies` despite never being imported at server runtime — both
fixed, bringing the Bun-only image from a buggy 2.18 GB down to ~421 MB.
Also corrected stale docs: the with-server-engine image is legitimately
several GB (it bakes in two Whisper models at build time), and
`FASTER_WHISPER_DOWNLOAD_ROOT`'s real default in that image is
`/opt/faster-whisper/models`, not under `/app/data` as previously
written.
## M7 — Live microphone transcription (browser engine, D-016)
Promoted out of the out-of-scope list via D-016. Browser engine only: audio
never touches the network, anonymous-first, works offline once a model is
cached.
- [x] Protocol: optional `requestId` on `transcribe`, echoed on
`partial`/`result`/`error` so the live loop can correlate responses
(file flow unchanged without ids)
- [x] `src/lib/live.ts` — pure, unit-tested segment loop: RMS energy gate,
`LiveSegmenter` (speech onset + pre-roll, silence/max-length commits,
pure silence dropped), `LiveTranscriber` (single-flight scheduling,
final passes before throttled tentative refreshes, stop/drain,
absolute-timestamped chunks)
- [x] `src/lib/mic.ts` — `getUserMedia` + AudioWorklet chunker from a Blob URL
(ScriptProcessorNode fallback), native-rate capture resampled to 16 kHz
on the main thread (D-016)
- [x] `LiveControls` component — start/stop button + recording indicator;
mutually exclusive with the file flow
- [x] Live transcript UI: committed text + visually-distinct tentative tail in
`TranscriptView`; final stop() result reuses the normal transcript view
(copy button, timestamps)
- [x] Unit tests: segmenter behavior, transcriber scheduling with a fake
engine and injected clock, protocol `requestId` guards
- [~] Manual in-browser verification (mic permission, live text, offline run)
— procedure in `tests/live.md`, pending a real browser (same dev-env
caveat as M2/M3)
- [x] Live transcription on the **server engine** (D-019) — `GET
/api/transcribe/live` upgrades to a WebSocket (`src/server/live.ts`,
wired in `src/index.ts`); the client streams mic PCM and runs the
browser-engine UI unchanged otherwise (`App.tsx`'s `startLive`/`stopLive`
branch on `useServer`). Landed already but was undocumented here and
untested; brought up to the rest of the codebase's bar (D-032):
`createLiveWebSocket` now takes the sidecar as an injected dependency
like `createTranscribeHandlers`, the WS route validates
`language`/`task` the same way the HTTP route does, and
`tests/server-live.test.ts` covers the bridge against a fake sidecar.
---
## Deliberately out of scope (for now)
Arbitrary Hub model ids, OAuth providers, multi-user transcript storage,
whisper.cpp server backend, i18n of the UI.
Promote to a milestone via a new entry in `Decisions.md` first.
---
## Noticed, not fixed
Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29.
- [x] `bun test tests/live-real-model.test.ts` fails: the `en_alien (en)` case
times out at its 900 s limit (2026-10-29, Linux CPU/WASM, reproduced in
isolation; `da_festuge` and `en_aiproblem` pass). — Not a hang; budget
and WER bar were calibrated for short clips. Fixed (D-039).
- [x] `isolate_wer2.ts` (repo root) is marked "TEMP (delete after use)" and
writes to a stale `/tmp/claude-1000/...` scratchpad path. — Deleted.
- [x] `docs/Architecture.md` §3 and `scripts/build.ts` say `src/pwa/icons/`
is "not checked in", but `icon-192.png`/`icon-512.png` are tracked.
— Kept tracked (`ensureIcons()` is idempotent); comments + docs fixed.
- [x] No typecheck or lint: `tsconfig.json` is strict but `typescript` is not
installed, so nothing checks types (`bun` strips them). — `bun run
typecheck` (D-038). Still no linter.
- [x] `.gitlab-ci.yml` only builds Docker images; `bun test` never runs in CI.
— `check` stage: typecheck + fast suite gate the build (D-038).
- [x] README "Graceful degradation" says server-engine weights download into
the data volume; the env table, Dockerfile and D-031 say they are baked
into the image at `/opt/faster-whisper/models`. — README fixed.
- [x] `src/index.css` declares `color-scheme: light dark` but only has a dark
palette; `Inter` is listed first in the font stack but never loaded.
— Light theme via tokens + self-hosted Inter (D-037). Also fixed: the
Backend select overflowed its panel on 390 px-wide phones.
## Recurring
- [ ] Maintain `AGENTS.md`:
- **Add a rule only in response to a real failure.** When an agent repeatedly
gets something wrong, add one line, ideally with a code example. Do not
pre-empt hypothetical mistakes.
- **Update in the same commit as the convention change.** A stale rules file
actively misleads.
- **Prune quarterly.** Delete rules whose convention changed or whose mistake
no longer occurs. Keep it under 150 lines.
- **Optional validation:** run the same task twice in clean sessions, with and
without the file, and compare steps and token use.