481 lines
30 KiB
Markdown
481 lines
30 KiB
Markdown
# Work Breakdown
|
||
|
||
Milestones are ordered so that **every milestone ends in a demoable, working app**.
|
||
Within a milestone, tasks are roughly in dependency order. Check items off as they
|
||
land; add follow-up tasks where they belong rather than in a separate backlog.
|
||
|
||
Legend: `[ ]` todo · `[x]` done · `[~]` in progress
|
||
|
||
---
|
||
|
||
## WIP Tasks before usage limit cutoff:
|
||
- [x] Allow users to cancel current transcription. — Cancel button on the
|
||
file line, wired to worker termination (D-018); covers decode, pending
|
||
model download, and the pass itself. Server-engine cancel is tracked in
|
||
M6 (needs the M5 EngineToggle first).
|
||
- [x] Better user-feedback while transcription is in progress (better than
|
||
"Listening for the first few words") — the worker now posts
|
||
`transcribe-progress` (seconds processed / total, from the streamer's
|
||
chunk timestamps via the pure `TranscribeProgressTracker`); the UI shows
|
||
a percent in the heading plus a progress bar with mm:ss, and the file
|
||
flow's placeholder reads "Processing audio…" (the live wording stays
|
||
live-only).
|
||
|
||
## M0 — Scaffold cleanup & docs (small)
|
||
Current state is the raw `bun init` React template.
|
||
|
||
- [x] `Architecture.md`, `Decisions.md`, `WorkBreakdown.md` created
|
||
- [x] Remove template cruft: `APITester.tsx`, hello routes in `src/index.ts`,
|
||
Bun/React logo branding in `App.tsx` / `index.html`
|
||
- [x] Add `@huggingface/transformers` dependency
|
||
- [x] Target directory layout from `Architecture.md` §3 (empty modules with TODOs are fine)
|
||
- [x] Smoke test in `tests/` proving `bun test` runs
|
||
|
||
## M1 — Browser transcription MVP (core)
|
||
Anonymous user can transcribe an uploaded file with one hardcoded model. **This is the
|
||
riskiest milestone — do it first, learn from it.**
|
||
|
||
- [x] `src/worker/protocol.ts` — typed messages: load / load-progress / transcribe /
|
||
partial / result / error
|
||
- [x] `src/worker/inference.worker.ts` — Transformers.js ASR pipeline
|
||
(`onnx-community/whisper-tiny.en` hardcoded, WASM only for now), chunked
|
||
with timestamps, partials streamed via `WhisperTextStreamer`
|
||
- [x] `src/lib/audio.ts` — decode upload → 16 kHz mono Float32Array (D-012)
|
||
- [x] `FileDropzone` component (drag-drop + file picker, audio/* and video/*)
|
||
- [x] `TranscriptView` component — streaming partials, final text, copy button
|
||
- [x] Model download progress UI (bytes / %, per-file)
|
||
- [x] Wire into `App.tsx`; delete remaining template UI
|
||
- [x] Unit tests for audio resampling and protocol handling
|
||
|
||
Notes (M1):
|
||
- Bun 1.3 does not bundle `new Worker(new URL(...))` graphs (verified against
|
||
both `bun build` and the dev server), so `src/index.ts` bundles the worker
|
||
with `Bun.build` at startup and serves it at `/inference.worker.js`; the UI
|
||
spawns `new Worker("/inference.worker.js", { type: "module" })`. Revisit if
|
||
Bun gains native worker bundling.
|
||
- onnxruntime-web fetches its `.wasm` binaries from the jsdelivr CDN
|
||
(Transformers.js default). Fine for M1; M3 (offline) must serve/cache them
|
||
locally.
|
||
|
||
## M2 — Model choice + WebGPU (core)
|
||
- [x] `src/lib/models.ts` — curated catalog with sizes and dtype defaults (D-006)
|
||
- [x] `ModelPicker` component — size estimates, English-only vs multilingual,
|
||
downloaded-state badge (query Cache API)
|
||
- [x] WebGPU detection in worker, `device` selection with WASM fallback (D-004);
|
||
surface active backend in UI
|
||
- [x] Handle model switching (dispose old pipeline, load new)
|
||
- [x] Language / task options for multilingual models (transcribe vs translate)
|
||
- [x] Whisper translate only on models trained for it (`canTranslate`; hidden for
|
||
large-v3-turbo and `.en`, stale "translate" resets to transcribe — D-044)
|
||
- [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
|
||
combinations in `models.ts`
|
||
- [~] Danish fine-tunes (spec 001 Goal 6, D-046): `whisper-small-danish`,
|
||
`whisper-large-v3-turbo-danish`, `edda-v0.2`. We converted them to ONNX
|
||
(fp32/fp16/q8/q4) and CT2 int8 and verified them on CPU. The catalog
|
||
entries hang off `DANISH_MODELS_HF_OWNER`. Still pending: the owner's go
|
||
for the Hugging Face upload (folders prepared, not uploaded), then the
|
||
owner constant, then browser checks of WebGPU/WebNN fp16.
|
||
|
||
Notes (M2):
|
||
- Quantized-variant verification is pending manual in-browser testing (no real
|
||
browser in the dev environment). `models.ts` encodes the community-known-good
|
||
combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU:
|
||
fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16
|
||
encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a
|
||
`verified` flag and note. Only `whisper-tiny.en @ wasm/q8` (exercised in M1)
|
||
is marked `verified: true`; flip the flags as manual testing confirms each.
|
||
- Downloaded badge (D-005): `src/lib/modelCache.ts` inspects Cache Storage for
|
||
transformers-cache entries with `.onnx` hits per model id; M3's offline
|
||
indicator should reuse `getDownloadedModels`.
|
||
|
||
## M3 — PWA & offline (core)
|
||
- [x] `manifest.webmanifest` + icons; installability
|
||
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
|
||
- [x] Build step: emit precache manifest from `bun build` output into the SW
|
||
- [x] SW registration + update flow (new-version toast)
|
||
- [~] Offline verification: airplane-mode transcription with a previously
|
||
downloaded model; document the manual test procedure in `tests/offline.md`
|
||
— procedure written (`tests/offline.md`), pending a manual run in a real
|
||
browser (none in the dev environment; same caveat as M2's variants)
|
||
- [x] "Available offline" indicator per model in `ModelPicker`
|
||
|
||
Notes (M3):
|
||
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
|
||
placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
|
||
values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
|
||
and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
|
||
empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
|
||
keeps working. SW bundles are never minified — the placeholders must survive
|
||
verbatim.
|
||
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
|
||
straight from `node_modules/onnxruntime-web/dist`; prod: copied to
|
||
`dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
|
||
cache-first into `whisper-runtime-*` — required for offline inference, too
|
||
big to precache. Model weights stay in Transformers.js's own
|
||
`transformers-cache`, which the SW never touches (D-005).
|
||
- Bun's HTML loader must resolve every `<link href>` to a real file, so
|
||
`index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
|
||
relatively (absolute `/...` URLs fail the build — bug found while verifying
|
||
M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
|
||
`/icons/*` URLs are still served for the manifest's internal icon refs.
|
||
`ensureIcons()` must run before the HTML is bundled (icons are generated,
|
||
not checked in).
|
||
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
|
||
hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
|
||
`application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
|
||
404 network-only. Browser-level behavior (install, airplane mode, update
|
||
toast) is manual — see `tests/offline.md`.
|
||
|
||
## M4 — Accounts (Passport.js)
|
||
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
|
||
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
|
||
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
|
||
- [x] Password hashing via `Bun.password` (Argon2)
|
||
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
|
||
behavioral change (anonymous-first invariant)
|
||
- [x] Auth tests (register/login/logout, session expiry, wrong password)
|
||
|
||
Notes (M4):
|
||
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
|
||
inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
|
||
base64url; expiry is enforced server-side via `sessions.expires_at`, not just
|
||
the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
|
||
production.
|
||
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
|
||
table and the production `fetch` handler — new API routes go there.
|
||
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
|
||
`/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
|
||
framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
|
||
offline) fail silent to anonymous — the app is offline-first.
|
||
|
||
## M5 — Server engine + toggle
|
||
- [x] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
|
||
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
|
||
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
|
||
model catalog to faster-whisper model ids in `src/lib/models.ts`
|
||
- [x] Batched inference + tuned Silero VAD for server-engine file
|
||
transcriptions, live passes sequential (spec 001 Goal 5, D-047):
|
||
≈ 2× faster on CPU int8 at equal-or-better WER; segments ≤ 30 s chunks
|
||
- [x] Degrade gracefully when Python/the sidecar is absent (server engine
|
||
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
|
||
- [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
|
||
upload size limit, request abort handling; plus `GET /api/transcribe/status`
|
||
(availability probe for the toggle)
|
||
- [x] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
|
||
`ServerEngine` (wraps fetch stream) (D-010)
|
||
- [x] `EngineToggle` component — visible only with a session; persists choice
|
||
(persistence helpers + status probe already in `lib/engine.ts`; remaining:
|
||
the component, lifting `useAuth` out of `AuthForms` into `App` so both see
|
||
one session state, and routing the file flow through the selected engine —
|
||
live stays on `BrowserEngine` per D-016)
|
||
- [x] Server model caching/warm-up strategy (don't reload weights per request)
|
||
- [~] Integration test: same fixture audio through both engines, transcripts agree
|
||
— supervisor/HTTP layers covered in `tests/transcribe.test.ts` (real
|
||
`Bun.spawn` against `tests/fixtures/fake_sidecar.py`); real inference is
|
||
the opt-in `FW_INTEGRATION=1` test + manual run in `tests/server-engine.md`
|
||
(no pip/browser in the dev environment; same caveat as M2/M3)
|
||
- [x] Deployment: optional Python/faster-whisper layer for the Docker image +
|
||
deployer note (base image stays Bun-only; install
|
||
`server-py/requirements.txt` to enable the server engine)
|
||
|
||
Notes (M5):
|
||
- Sidecar protocol (`server-py/sidecar.py`): JSON-lines over stdio —
|
||
`ready`/`fatal` handshake, `transcribe` → `segment`* → `done`, `cancel`
|
||
between segments, `shutdown`/EOF exits. Audio is handed over as a temp file
|
||
of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes),
|
||
read with `np.fromfile` — no container decoding server-side. The last-used
|
||
model stays loaded (replaced on change, so memory is bounded to one model);
|
||
device/compute via `FASTER_WHISPER_DEVICE`/`FASTER_WHISPER_COMPUTE`
|
||
(default auto/int8).
|
||
- Supervision (`createSidecar`): lazy spawn on first use; a crash rejects
|
||
in-flight requests and the next request respawns; a missing runtime
|
||
(no python3 / `fatal` handshake) is remembered as a *permanent* unavailable
|
||
reason — surfaced verbatim by `GET /api/transcribe/status` and 503s — so a
|
||
Bun-only deployment never respawn-storms. Tests force this path with
|
||
`python3 -S -I` (site-packages off) against the real sidecar.
|
||
- Upload limit: `TRANSCRIBE_MAX_UPLOAD_BYTES` (default 512 MiB ≈ 2¼ h of raw
|
||
f32 PCM), enforced pre-read via Content-Length *and* while streaming the
|
||
body to the temp file. Errors after the 200 header travel in-stream as an
|
||
`{"type":"error"}` line (wire format shared with the client via
|
||
`src/lib/engine.ts`, so server and `ServerEngine` cannot drift).
|
||
- `AUTH_ENABLED=false` (D-015) 404s `/api/transcribe*` along with the auth
|
||
routes — the server engine is session-gated, so it disappears with auth.
|
||
- Smoke-tested live against the dev server: `/` + auth unchanged; status
|
||
401 → (register) → 200 `{available:false, reason:"faster-whisper is not
|
||
installed…"}`; POST → 503 with the same reason, body untouched.
|
||
- EngineToggle: one `useAuth()` lives in `App` (passed into `AuthForms` and
|
||
gating the toggle); the status probe runs on sign-in and the server option
|
||
renders disabled-with-reason when unavailable. The file flow now goes
|
||
through the `Engine` interface for **both** paths (D-010): `BrowserEngine`
|
||
shares the worker with the live loop (distinct requestId namespaces) and a
|
||
new `stream` flag on the transcribe message keeps token partials +
|
||
transcribe-progress flowing for id-carrying engine passes (live passes stay
|
||
non-streaming). Server runs skip the browser model load, show segment
|
||
partials without a progress bar (faster-whisper's stream carries no window
|
||
boundaries), and cancel by aborting the fetch (D-018 amendment) — browser
|
||
runs keep worker-termination cancel. The choice persists in localStorage
|
||
but only takes effect while signed in with the engine available.
|
||
- Deployment: multi-stage `Dockerfile` — `runtime` (Bun-only, ~400-450 MB, see
|
||
D-031) is the default target so plain `docker build .` (existing CI) is
|
||
unchanged; `--target with-server-engine` (several GB — two Whisper models
|
||
pre-downloaded at build time, D-031) adds python3 + faster-whisper in a venv
|
||
at `/opt/faster-whisper` (wired via `SIDECAR_PYTHON`), with
|
||
`FASTER_WHISPER_DOWNLOAD_ROOT` pointed at `/opt/faster-whisper/models`
|
||
(deliberately outside the `/app/data` volume, D-031) so weights persist
|
||
without a volume and there's no first-request download. README.md documents
|
||
both flavors, all runtime env vars, the DB volume, and the HTTPS/reverse-proxy
|
||
requirement. Verified live in Docker:
|
||
Bun-only image degrades per D-013 (`available:false`, "python3 is not
|
||
available…"); server-engine image ran a real `POST /api/transcribe` of
|
||
`en_aiproblem.m4a` end-to-end, streamed segments matching the fixture
|
||
transcript — the server half of the M5 integration-test item is now
|
||
exercised for real (browser half still needs the manual run).
|
||
|
||
## M6 — Hardening & polish
|
||
- [x] Error taxonomy: unsupported file, out-of-memory (suggest smaller model),
|
||
WebGPU device-lost, network loss mid-download (resume or clear retry) —
|
||
pure `classifyError` in `src/lib/errors.ts` (D-030) sorts every failure
|
||
path (decode, load, transcribe, a crashed worker) into one of the four
|
||
categories plus a generic fallback, each with a user-facing message and
|
||
an actionable suggestion; the error box in `App.tsx` adds a Retry
|
||
button for the two retryable kinds (network, device-lost), replaying
|
||
the failed load or the last file. Unit-tested in `tests/errors.test.ts`.
|
||
- [x] Cancel in-flight transcription (both engines) — browser engine via
|
||
worker termination (D-018), server engine via aborting the
|
||
`/api/transcribe` fetch (D-018 amendment; landed with M5's EngineToggle)
|
||
- [x] Export transcript: .txt and .srt (timestamps already available) —
|
||
pure formatters in `src/lib/export.ts` (unit-tested: SRT cue numbering,
|
||
HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start
|
||
or fall back to +2 s, filename derived from the source file); .txt/.srt
|
||
download buttons next to Copy in `TranscriptView`. Live-mode results
|
||
reuse the same `result` state with absolute timestamps (D-016), so both
|
||
flows export identically.
|
||
- [x] Storage management UI: list cached models, per-model delete —
|
||
collapsible "Storage" panel (`StorageManager`) listing cached models
|
||
with exact on-disk size (summed `blob().size`, measured lazily on
|
||
open) and per-model delete; delete is confirm-free (weights are just
|
||
cache — they re-download on next use) and refreshes the picker's
|
||
downloaded/offline badges. Helpers in `src/lib/modelCache.ts` take an
|
||
injectable Cache-Storage-shaped object (mocked in
|
||
`tests/modelCache.test.ts`). Scope is the transformers-cache only:
|
||
the SW's `whisper-runtime-*` cache (onnxruntime .wasm, shared by all
|
||
models, versioned/cleaned by the SW itself) is deliberately not
|
||
surfaced — deleting it per-model makes no sense and the SW would
|
||
just re-fetch it.
|
||
- [x] Model downloads are user-initiated (D-020): selecting a model/backend no
|
||
longer auto-downloads weights; the load starts on browser-engine
|
||
transcription start (file drop / live start) or via an explicit
|
||
per-model Download button in `ModelPicker` — and never when the engine
|
||
toggle is set to "server".
|
||
- [x] Firefox performance: never run the WASM fallback single-threaded after
|
||
a failed accelerator attempt (worker posts `load-restart`, UI respawns
|
||
with explicit wasm — D-025); skip WebGPU when fp16 weights need the
|
||
absent `shader-f16` feature; hint that medium/large lag in live mode on
|
||
WASM. Field-verified: WASM now gets the full thread pool; ort-web's
|
||
WebGPU EP on Firefox completes but runs far slower than WASM, so
|
||
"auto" skips WebGPU on Firefox (D-026, explicit selection still
|
||
honored — retest on ort bumps).
|
||
- [x] Mobile OOM crash on Medium+ model loads (D-027) — patched
|
||
`@huggingface/transformers` (`bun patch`) to load the encoder/decoder
|
||
sessions sequentially instead of concurrently, roughly halving
|
||
load-time peak memory for every model; added a mobile UA +
|
||
`deviceMemory` warn/confirm gate (`src/lib/mobileMemoryGuard.ts`)
|
||
before any browser-engine load whose estimated peak
|
||
(`estimatePeakLoadBytes`) is risky on a phone/tablet, covering the
|
||
explicit Download button and the lazy auto-load on file-drop/live-start
|
||
alike.
|
||
- [x] Basic a11y pass (keyboard, focus, aria on progress/live transcript) —
|
||
the model-download and transcription `<progress>` bars gained
|
||
`aria-label`s (previously announced as an unnamed percentage);
|
||
`TranscriptView`'s heading is now a programmatic focus target
|
||
(`tabIndex={-1}`) that receives focus the moment a final result lands,
|
||
so keyboard/screen-reader users land on new content instead of hunting
|
||
for it; the Copy button's "Copied!" state and `LiveControls`' status
|
||
transitions (requesting mic / listening / finishing) now also announce
|
||
via a visually-hidden `role="status"` region for users not focused on
|
||
the button. Keyboard operability was already in place everywhere in
|
||
scope (native buttons/inputs, `FileDropzone`'s `role="button"` +
|
||
`onKeyDown`) and needed no changes. `bun test` unaffected (233 pass, 1
|
||
pre-existing unrelated WER-gate failure, see Decisions.md).
|
||
- [x] Production build + `bun start` deployment check; README for deployers —
|
||
`bun run build` + `bun start` smoke-tested live (headers, hashed-asset
|
||
immutability, `/api/*` routing, `AUTH_ENABLED=false`, server-engine
|
||
graceful degradation all matched the README's claims). A real
|
||
`docker build .` from this checkout also caught two real bugs (D-031):
|
||
local Python venvs leaking into the image (858 MB, via a nested
|
||
`.gitignore` `.dockerignore` doesn't read) and `@huggingface/transformers`
|
||
+`onnxruntime-node`/`-web` (341 MB) sitting in the production
|
||
`dependencies` despite never being imported at server runtime — both
|
||
fixed, bringing the Bun-only image from a buggy 2.18 GB down to ~421 MB.
|
||
Also corrected stale docs: the with-server-engine image is legitimately
|
||
several GB (it bakes in two Whisper models at build time), and
|
||
`FASTER_WHISPER_DOWNLOAD_ROOT`'s real default in that image is
|
||
`/opt/faster-whisper/models`, not under `/app/data` as previously
|
||
written.
|
||
|
||
## M7 — Live microphone transcription (browser engine, D-016)
|
||
Promoted out of the out-of-scope list via D-016. Browser engine only: audio
|
||
never touches the network, anonymous-first, works offline once a model is
|
||
cached.
|
||
|
||
- [x] Protocol: optional `requestId` on `transcribe`, echoed on
|
||
`partial`/`result`/`error` so the live loop can correlate responses
|
||
(file flow unchanged without ids)
|
||
- [x] `src/lib/live.ts` — pure, unit-tested segment loop: RMS energy gate,
|
||
`LiveSegmenter` (speech onset + pre-roll, silence/max-length commits,
|
||
pure silence dropped), `LiveTranscriber` (single-flight scheduling,
|
||
final passes before throttled tentative refreshes, stop/drain,
|
||
absolute-timestamped chunks)
|
||
- [x] `src/lib/mic.ts` — `getUserMedia` + AudioWorklet chunker from a Blob URL
|
||
(ScriptProcessorNode fallback), native-rate capture resampled to 16 kHz
|
||
on the main thread (D-016)
|
||
- [x] `LiveControls` component — start/stop button + recording indicator;
|
||
mutually exclusive with the file flow
|
||
- [x] Live transcript UI: committed text + visually-distinct tentative tail in
|
||
`TranscriptView`; final stop() result reuses the normal transcript view
|
||
(copy button, timestamps)
|
||
- [x] Unit tests: segmenter behavior, transcriber scheduling with a fake
|
||
engine and injected clock, protocol `requestId` guards
|
||
- [~] Manual in-browser verification (mic permission, live text, offline run)
|
||
— procedure in `tests/live.md`, pending a real browser (same dev-env
|
||
caveat as M2/M3)
|
||
- [x] Live transcription on the **server engine** (D-019) — `GET
|
||
/api/transcribe/live` upgrades to a WebSocket (`src/server/live.ts`,
|
||
wired in `src/index.ts`); the client streams mic PCM and runs the
|
||
browser-engine UI unchanged otherwise (`App.tsx`'s `startLive`/`stopLive`
|
||
branch on `useServer`). Landed already but was undocumented here and
|
||
untested; brought up to the rest of the codebase's bar (D-032):
|
||
`createLiveWebSocket` now takes the sidecar as an injected dependency
|
||
like `createTranscribeHandlers`, the WS route validates
|
||
`language`/`task` the same way the HTTP route does, and
|
||
`tests/server-live.test.ts` covers the bridge against a fake sidecar.
|
||
|
||
## M8 — Transcription quality and speed (spec 001)
|
||
- [x] Transformers.js 4.2.0 → 4.3.2, sequential-session patch regenerated
|
||
for 4.3.2 and guarded by `tests/transformers-runtime.test.ts` (D-040).
|
||
- [x] WASM sessions back at the default graph optimization level: ort-web
|
||
1.31.0-dev loads the q8 decoders at every level (D-043 supersedes D-014).
|
||
- [x] Vocabulary hint "Names and terms" (D-048) — optional field for every
|
||
model, remembered in localStorage, sent as `prompt` on file and live
|
||
passes (browser and server engines); cut to 120 tokens at a whole word.
|
||
faster-whisper gets `initial_prompt` only (hotwords measured, no gain).
|
||
|
||
---
|
||
|
||
## Spec 001 — Translate (workstream 4)
|
||
- [x] `canTranslate` per model; Task control hidden where false; stale
|
||
"translate" resets to transcribe (D-044)
|
||
- [x] Text translation of any finished transcript, da→en / en→da, Opus-MT
|
||
in its own worker, sentence-chunked, timestamps kept for exports (D-045)
|
||
- [~] Real-browser checks: offline translation (`tests/offline.md` §5),
|
||
WebGPU-machine speed (owner)
|
||
|
||
---
|
||
|
||
## Spec 001 Goal 7 — Parakeet TDT 0.6B v3 (English, desktop, D-051)
|
||
- [x] Runtime decision: Transformers.js 4.2.0/4.3.2 have CTC-only Parakeet → parakeet.js on the
|
||
shared onnxruntime-web (one runtime in the bundle, `tests/workerBundle.test.ts`).
|
||
- [x] Catalog `family` + exact sizes; worker engine path (≤ 30 s windows, word → chunk
|
||
timestamps, progress/partials); Cache-API downloader (`modelFiles.ts`).
|
||
- [x] File and live mode; WASM int8 baseline; explicit WebGPU = fp16 encoder (opt-in only).
|
||
- [x] Hidden on phones/tablets; English-only note instead of language/task controls.
|
||
- [x] Real-model WER/wall-clock vs Large v3 Turbo (CPU), opt-in `PARAKEET_REAL=1` test.
|
||
- [x] Headless Chrome (no GPU, fresh profile, prod build, COOP/COEP on): picking Parakeet and
|
||
dropping `en_aiproblem.m4a` downloaded the weights, cached them ("available offline"),
|
||
ran on WASM (12 threads) and showed the transcript, 2 timestamp segments and the
|
||
.txt/.srt/.txt+ts/Copy exports — 15 s end to end including the ~670 MB download. Picker
|
||
screenshots: desktop light/dark show the row + English-only note; a phone UA hides it.
|
||
- [ ] Owner's browser check: real Chrome WASM load/transcribe (speed, memory), explicit
|
||
WebGPU on a real GPU (works? faster than WASM? fallback restart), Firefox/Safari WASM.
|
||
|
||
## Deliberately out of scope (for now)
|
||
Arbitrary Hub model ids, OAuth providers, multi-user transcript storage,
|
||
whisper.cpp server backend, i18n of the UI.
|
||
Promote to a milestone via a new entry in `Decisions.md` first.
|
||
|
||
---
|
||
|
||
## Noticed, not fixed
|
||
Found while bootstrapping agent docs (D-036); all fixed on 2026-10-29.
|
||
- [x] `bun test tests/live-real-model.test.ts` fails: the `en_alien (en)` case
|
||
times out at its 900 s limit (2026-10-29, Linux CPU/WASM, reproduced in
|
||
isolation; `da_festuge` and `en_aiproblem` pass). — Not a hang; budget
|
||
and WER bar were calibrated for short clips. Fixed (D-039).
|
||
- [x] `isolate_wer2.ts` (repo root) is marked "TEMP (delete after use)" and
|
||
writes to a stale `/tmp/claude-1000/...` scratchpad path. — Deleted.
|
||
- [x] `docs/Architecture.md` §3 and `scripts/build.ts` say `src/pwa/icons/`
|
||
is "not checked in", but `icon-192.png`/`icon-512.png` are tracked.
|
||
— Kept tracked (`ensureIcons()` is idempotent); comments + docs fixed.
|
||
- [x] No typecheck or lint: `tsconfig.json` is strict but `typescript` is not
|
||
installed, so nothing checks types (`bun` strips them). — `bun run
|
||
typecheck` (D-038). Still no linter.
|
||
- [x] `.gitlab-ci.yml` only builds Docker images; `bun test` never runs in CI.
|
||
— `check` stage: typecheck + fast suite gate the build (D-038).
|
||
- [x] README "Graceful degradation" says server-engine weights download into
|
||
the data volume; the env table, Dockerfile and D-031 say they are baked
|
||
into the image at `/opt/faster-whisper/models`. — README fixed.
|
||
- [x] `src/index.css` declares `color-scheme: light dark` but only has a dark
|
||
palette; `Inter` is listed first in the font stack but never loaded.
|
||
— Light theme via tokens + self-hosted Inter (D-037). Also fixed: the
|
||
Backend select overflowed its panel on 390 px-wide phones.
|
||
- [ ] An English "Names and terms" hint can pull a short Danish live window
|
||
toward English words (one 2 s da_festuge snippet on turbo, D-048).
|
||
Not addressed: suggest same-language hints if it shows up in use.
|
||
|
||
Found while building transcript translation (D-045), 2026-10-29:
|
||
- [ ] Transformers.js 4.2 batched Marian generation returns row 0's output for
|
||
every row of a padded batch, so translation runs one sentence per call
|
||
(~4× slower than batch 8 would be). Re-check after the 4.3 upgrade with
|
||
`bun scripts/bench-translate.ts` ("output identical" across batch sizes).
|
||
**Re-checked on 4.3.2 (2026-10-30): still broken** — a batch of 4 Danish
|
||
sentences returned row 0's translation for all 4 (our `translateBatch`
|
||
passes the batch straight to the translation pipeline, so it is
|
||
upstream). Batch 8 would be ~4× faster (12 s vs 49 s for en_alien.txt).
|
||
- [ ] `src/worker/opusMt.ts` still forces `graphOptimizationLevel: "basic"`
|
||
(D-014) for Marian; D-043 dropped it for Whisper on ort-web 1.31. Check
|
||
whether the q8 Marian decoder now loads at the default level.
|
||
- [x] The server API still accepts `task: "translate"` with large-v3-turbo
|
||
(only the UI hides it, D-044). — Fixed: `POST /api/transcribe` and the
|
||
live WebSocket answer 400 via `translateNotSupported` (src/server/transcribe.ts).
|
||
|
||
Found in spec 001 workstream 6 (D-046), open:
|
||
- [ ] Transformers.js 4.2 has no beam search: `num_beams > 1` in a model's
|
||
`generation_config.json` (edda-v0.2 ships 5) silently decodes greedily
|
||
and does a full-vocabulary top-k per step (D-046). Not fixed: it would
|
||
mean overriding `num_beams: 1` in the worker, or a patch.
|
||
- [ ] On `da_festuge`, neither Danish turbo fine-tune beats the stock turbo
|
||
(D-046). Judging them needs a longer, harder Danish reference clip than
|
||
this repo has.
|
||
|
||
Found while building the long-form pipeline (spec 001 Goal 3, D-049), 2026-10-29:
|
||
- [x] "Auto-detect" on the default (chunked) file path does not detect: with no
|
||
`language`, Transformers.js 4.2 logs "No language specified - defaulting to
|
||
English" and decodes Danish audio as an English translation (da_festuge WER
|
||
0.94 vs 0.14 with `da`). Long-form ("Skip silence") detects once and pins
|
||
the language; the chunked path is unchanged.
|
||
- [ ] Spec 001's measurement fallback mentions a hint list of the names in
|
||
`en_alien`'s reference; `tests/meetingFixture.ts` returns audio + reference
|
||
only. Add the list when the vocabulary hint (workstream 2) lands.
|
||
— Fixed for file passes: detect once on the first 30 s and pin (D-053).
|
||
- [ ] "Auto-detect" on **live** passes (browser engine) still decodes as English:
|
||
each live pass would need its own detection (an extra encoder run per
|
||
pass). Needs a once-per-session design (detect on the first committed
|
||
segment, then pin) — D-053.
|
||
|
||
- [ ] Parakeet v3 also transcribes 24 other European languages, Danish included, but the app
|
||
offers it English-only (spec 001) — unmeasured on `da_festuge`. Candidate for a later spec.
|
||
- [ ] parakeet.js still logs `[Parakeet.js]` lines to the worker console on some paths; harmless,
|
||
not silenced (no logger hook in the library).
|
||
|
||
## Recurring
|
||
- [ ] Maintain `AGENTS.md`:
|
||
- **Add a rule only in response to a real failure.** When an agent repeatedly
|
||
gets something wrong, add one line, ideally with a code example. Do not
|
||
pre-empt hypothetical mistakes.
|
||
- **Update in the same commit as the convention change.** A stale rules file
|
||
actively misleads.
|
||
- **Prune quarterly.** Delete rules whose convention changed or whose mistake
|
||
no longer occurs. Keep it under 150 lines.
|
||
- **Optional validation:** run the same task twice in clean sessions, with and
|
||
without the file, and compare steps and token use.
|