Add live transcription via remote server.
This commit is contained in:
Lucas Winther committed
1 parent
695b97be76
commit
42fe27d2dd
9 files changed
+227
-21
No files matched your search
@@ -0,0 +1,201 @@
|
||||
# Architecture
|
||||
|
||||
> Companion to `CLAUDE.md` (product spec & tooling rules) and `Decisions.md` (why things
|
||||
> are the way they are). This file describes **how the system is structured**. Update it
|
||||
> whenever the structure changes.
|
||||
|
||||
## 1. System overview
|
||||
|
||||
Browser-first speech-to-text app. Whisper inference runs **in the browser by default**
|
||||
via [Transformers.js](https://github.com/huggingface/transformers)
|
||||
(`@huggingface/transformers`); no audio ever leaves the device for anonymous users.
|
||||
Logged-in users may optionally route inference to the server.
|
||||
|
||||
```
|
||||
┌──────────────────────────── Browser ────────────────────────────┐
|
||||
│ │
|
||||
│ React UI (App shell) │
|
||||
│ ├─ ModelPicker (tiny → large, .en, quantized variants) │
|
||||
│ ├─ FileDropzone (audio/video upload) │
|
||||
│ ├─ LiveControls (microphone start/stop — D-016) │
|
||||
│ ├─ TranscriptView (streaming partial results) │
|
||||
│ └─ EngineToggle (device / server — logged-in only) │
|
||||
│ │ postMessage protocol │
|
||||
│ ▼ │
|
||||
│ Inference Web Worker │
|
||||
│ ├─ Transformers.js ASR pipeline (Whisper) │
|
||||
│ ├─ Device selection: WebNN → WebGPU → WASM fallback │
|
||||
│ └─ Model weights cached via browser Cache API │
|
||||
│ │
|
||||
│ Service Worker (PWA) │
|
||||
│ ├─ Precached app shell (offline-first) │
|
||||
│ └─ Works with zero network once installed + model cached │
|
||||
└──────────────────┬───────────────────────────────────────────────┘
|
||||
│ HTTPS (only when online / server engine chosen)
|
||||
▼
|
||||
┌──────────────────────────── Bun server ─────────────────────────┐
|
||||
│ Bun.serve() — routes + HTML imports (no vite, no express) │
|
||||
│ ├─ / → src/index.html (bundled React app) │
|
||||
│ ├─ /inference.worker.js → worker bundle (Bun.build at startup, │
|
||||
│ │ since Bun doesn't bundle `new Worker` graphs) │
|
||||
│ ├─ /api/auth/* → Passport.js (local strategy) + sessions │
|
||||
│ ├─ /api/transcribe → server engine (auth required) │
|
||||
│ └─ /manifest.webmanifest, /sw.js → PWA assets │
|
||||
│ bun:sqlite → users, sessions │
|
||||
│ Server inference: faster-whisper (CTranslate2) Python sidecar, │
|
||||
│ spawned & supervised by Bun (D-013) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2. Runtime & tooling
|
||||
|
||||
- **Runtime / bundler / test runner:** Bun only (`bun --hot src/index.ts` for dev,
|
||||
`bun build src/index.html` for production, `bun test`). No vite/webpack/jest.
|
||||
- **Frontend:** React 19, mounted from `src/index.html` via Bun HTML imports.
|
||||
- **Server:** `Bun.serve()` with the `routes` option. No express.
|
||||
- **DB:** `bun:sqlite`. No ORM planned initially.
|
||||
|
||||
## 3. Directory layout (target)
|
||||
|
||||
```
|
||||
src/
|
||||
index.ts # Bun.serve entry — routes, static, API
|
||||
index.html # HTML entry, imports frontend.tsx
|
||||
frontend.tsx # React root
|
||||
App.tsx # top-level UI composition
|
||||
components/ # ModelPicker, TranscriptionOptions, FileDropzone, LiveControls, TranscriptView, EngineToggle, AuthForms
|
||||
lib/
|
||||
audio.ts # decode + resample to 16 kHz mono Float32Array (Web Audio API)
|
||||
live.ts # live-mode segment loop: RMS gate, LiveSegmenter, LiveTranscriber (pure, D-016)
|
||||
mic.ts # microphone capture: AudioWorklet chunker (Blob URL) + fallback (D-016)
|
||||
models.ts # model catalog: ids, sizes, dtype defaults per device, languages
|
||||
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
|
||||
engine.ts # engine abstraction: BrowserEngine | ServerEngine
|
||||
worker/
|
||||
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
|
||||
protocol.ts # typed postMessage messages (shared with UI)
|
||||
pwa/
|
||||
sw.ts # service worker (app-shell precache, /ort runtime cache)
|
||||
precache.ts # placeholder-injection helpers shared by build + dev serve
|
||||
register.ts # SW registration + update-toast flow
|
||||
manifest.webmanifest
|
||||
icons/ # generated by scripts/generate-icons.ts (not checked in)
|
||||
server/
|
||||
auth.ts # Passport.js wiring + session middleware adapter
|
||||
db.ts # bun:sqlite schema + queries
|
||||
transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5)
|
||||
server-py/ # faster-whisper sidecar (Python, D-013; M5)
|
||||
tests/ # bun test
|
||||
```
|
||||
|
||||
(Current state: M0–M3 built — browser transcription with the curated model
|
||||
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
|
||||
language/task options,
|
||||
and the installable offline PWA (`pwa/`, `scripts/build.ts` production build,
|
||||
`dist/` serving in `src/index.ts`; browser-level offline behavior pending the
|
||||
manual run in `tests/offline.md`). M4 accounts are built (`server/db.ts`,
|
||||
`server/auth.ts`, `/api/auth/*`, `AuthForms`). M7 live microphone
|
||||
transcription is built (`lib/live.ts`, `lib/mic.ts`, `LiveControls`, D-016;
|
||||
in-browser run pending per `tests/live.md`). M5's server side is built —
|
||||
`server-py/sidecar.py`, `server/transcribe.ts` (supervisor +
|
||||
`/api/transcribe` + `/status`), `lib/engine.ts` (both engines, D-010), and
|
||||
the catalog mapping in `lib/models.ts` — with graceful degradation when
|
||||
Python/faster-whisper is absent; the `EngineToggle` component and App wiring
|
||||
are the remaining M5 work; see `WorkBreakdown.md`.)
|
||||
|
||||
## 4. Key flows
|
||||
|
||||
### 4.1 Browser transcription (anonymous, default)
|
||||
1. User picks a model in `ModelPicker` (curated catalog from `lib/models.ts`, with
|
||||
size estimates and a Cache-API-backed "downloaded" badge); UI posts
|
||||
`{type:"load", modelId, device}` where `device` is a preference
|
||||
(`"auto" | "webnn" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype`
|
||||
override is expressible in the message but normally omitted.
|
||||
2. Worker resolves the device (D-004, D-017): for `"auto"` it probes
|
||||
`navigator.ml` (WebNN) first, then `navigator.gpu?.requestAdapter()` (WebGPU),
|
||||
and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the
|
||||
picked device throws, it retries down the same chain (WASM always last). An
|
||||
explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM. It then
|
||||
resolves the catalog's dtype default for the device actually used and creates
|
||||
the Transformers.js `automatic-speech-recognition` pipeline.
|
||||
Download progress events are forwarded to the UI; the terminal
|
||||
`{status:"ready", device}` message reports the backend actually selected, which
|
||||
the UI surfaces. Loading a different model/device disposes the old pipeline
|
||||
(`pipeline.dispose()`) before creating the new one.
|
||||
3. Weights are fetched from the Hugging Face Hub **once**; Transformers.js stores them
|
||||
in the browser Cache API, so subsequent loads (including offline) hit cache
|
||||
(`lib/modelCache.ts` reads that cache for the downloaded badges).
|
||||
4. User drops an audio/video file. Main thread decodes it with
|
||||
`AudioContext.decodeAudioData`, downmixes to mono, resamples to 16 kHz,
|
||||
transfers the `Float32Array` to the worker. For multilingual models the
|
||||
`transcribe` message also carries `language` (or auto-detect) and `task`
|
||||
(transcribe | translate).
|
||||
5. Worker runs the pipeline (chunked, with timestamps), streaming partial results
|
||||
back via `{type:"partial"}` messages; final result as `{type:"result"}`.
|
||||
|
||||
### 4.1b Live microphone transcription (anonymous, browser engine — D-016)
|
||||
1. `LiveControls` starts a session: `getUserMedia` → `AudioContext` at the
|
||||
device's native rate → AudioWorklet chunker (module from a Blob URL;
|
||||
`ScriptProcessorNode` fallback) posts ~100 ms Float32 frames; the main
|
||||
thread resamples each to 16 kHz (`lib/mic.ts`).
|
||||
2. `LiveTranscriber` (`lib/live.ts`, pure) feeds frames through an RMS energy
|
||||
gate: segments start at speech onset (short pre-roll), commit on trailing
|
||||
silence or at ~25 s; pure silence is never transcribed.
|
||||
3. The current segment is re-transcribed through the normal inference worker
|
||||
on a rolling basis (throttled, single-flight, latest-snapshot-wins) as
|
||||
*tentative* text; a committed segment gets one final pass and freezes.
|
||||
`requestId` on the protocol messages correlates responses.
|
||||
4. Stopping flushes the segmenter, drains pending final passes, and yields a
|
||||
normal transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
||||
Everything runs on-device; works offline like the file flow. Server-engine
|
||||
live input is a recorded future want (WorkBreakdown M7, D-016).
|
||||
|
||||
### 4.2 Offline (PWA)
|
||||
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
|
||||
(hashed JS/CSS, `/`, worker bundle, manifest, icons) on install into
|
||||
`whisper-shell-<version>`, cache-first thereafter; unknown navigations fall back
|
||||
to the cached shell. `scripts/build.ts` injects the precache list + a
|
||||
content-hash version into `dist/sw.js` via `pwa/precache.ts`; the dev server
|
||||
injects an empty list ("dev"), making the dev shell network-first so HMR works.
|
||||
- onnxruntime's `.wasm`/`.mjs` binaries are served from our origin at `/ort/*`
|
||||
(never the jsdelivr CDN default) and runtime-cached cache-first into
|
||||
`whisper-runtime-<version>` — they are required for offline inference but too
|
||||
large to precache eagerly.
|
||||
- Model weights live in the Cache API (`transformers-cache`) via Transformers.js —
|
||||
independent of the SW, which never touches that cache (D-005). `/api/*` is
|
||||
network-only.
|
||||
- Updates: `pwa/register.ts` surfaces a waiting SW as a toast; accepting posts
|
||||
`SKIP_WAITING`, activation drops old `whisper-*` caches, and the page reloads.
|
||||
- Result: with the app installed and at least one model previously loaded, the full
|
||||
transcription flow works with no network and with our server down (manual
|
||||
verification procedure: `tests/offline.md`).
|
||||
|
||||
### 4.3 Server engine (logged-in only)
|
||||
1. `EngineToggle` is visible only when a session exists.
|
||||
2. `ServerEngine.transcribe()` uploads the decoded (or original) audio to
|
||||
`POST /api/transcribe` with the chosen model id.
|
||||
3. Server inference runs on **faster-whisper** (CTranslate2) in a Python sidecar
|
||||
(`server-py/`) that Bun spawns and supervises (D-013; faster-whisper has no
|
||||
browser/WASM build, so it is server-only — the browser stays on Transformers.js).
|
||||
`src/server/transcribe.ts` bridges the sidecar's segment stream to the streamed
|
||||
JSON-lines HTTP response (D-011). The shared catalog maps model ids to
|
||||
`Systran/faster-whisper-*` CTranslate2 conversions. If the sidecar is absent,
|
||||
the server engine reports unavailable; the browser/anonymous flow is unaffected.
|
||||
4. Both engines implement the same `Engine` interface in `src/lib/engine.ts`, so the
|
||||
UI is engine-agnostic.
|
||||
|
||||
### 4.4 Auth
|
||||
- Passport.js **local strategy** behind a thin adapter that bridges Bun's
|
||||
`Request`/`Response` to Passport's connect-style API (see Decisions.md D-008).
|
||||
- Sessions in SQLite; session id in an HttpOnly cookie.
|
||||
- Anonymous users are never blocked from browser inference — auth only unlocks the
|
||||
server engine.
|
||||
|
||||
## 5. Invariants
|
||||
|
||||
- **Privacy:** browser-engine audio and transcripts never touch the network.
|
||||
- **Anonymous-first:** every feature except the server engine works logged out.
|
||||
- **Offline-first:** app shell + previously-downloaded models must work offline.
|
||||
- **One browser inference codebase:** Transformers.js in the browser; the server
|
||||
engine runs faster-whisper (D-013). The model catalog stays shared across both.
|
||||
- **Worker isolation:** inference never runs on the main thread.
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
# CLAUDE.md — Read this first, every session
|
||||
|
||||
This is the single source of truth for any agent working in this repo. If anything
|
||||
you're told conflicts with this file, this file wins.
|
||||
|
||||
## 1. What we are building
|
||||
|
||||
A browser-first speech-to-text web app powered by OpenAI Whisper, running through
|
||||
[Transformers.js](https://github.com/huggingface/transformers) (`@huggingface/transformers`).
|
||||
|
||||
**Anonymous users (no login) can:**
|
||||
- Run Whisper models fully in the browser (no data leaves the device).
|
||||
- Choose which Whisper model to use (tiny → large, English-only and multilingual, quantized variants).
|
||||
- Keep working when the network — or our own server — is offline (installable PWA + cached model weights).
|
||||
- Accelerate inference with WebNN or WebGPU (auto-selected in that order, D-017), falling back to WASM automatically.
|
||||
- Transcribe from an uploaded audio/video file.
|
||||
|
||||
**Logged-in users (Passport.js) get all of the above, plus:**
|
||||
- A toggle to choose where inference runs: **their own device** (browser engine) or **the server** the app is deployed on (server engine).
|
||||
- The server engine MUST use [faster-whisper](https://github.com/SYSTRAN/faster-whisper) (CTranslate2) for inference — not Transformers.js (see Decisions.md D-013). faster-whisper cannot run in a browser (no WASM build of CTranslate2), so the browser engine stays on Transformers.js.
|
||||
|
||||
## 2 How to run things
|
||||
|
||||
|
||||
Default to using Bun instead of Node.js.
|
||||
|
||||
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
||||
- Use `bun test` instead of `jest` or `vitest`
|
||||
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
||||
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
||||
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
||||
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
||||
- Bun automatically loads .env, so don't use dotenv.
|
||||
|
||||
## APIs
|
||||
|
||||
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
||||
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
||||
- `Bun.redis` for Redis. Don't use `ioredis`.
|
||||
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
||||
- `WebSocket` is built-in. Don't use `ws`.
|
||||
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
||||
- Bun.$`ls` instead of execa.
|
||||
|
||||
## Testing
|
||||
|
||||
Use `bun test` to run tests.
|
||||
|
||||
```ts#index.test.ts
|
||||
import { test, expect } from "bun:test";
|
||||
|
||||
test("hello world", () => {
|
||||
expect(1).toBe(1);
|
||||
});
|
||||
```
|
||||
|
||||
## Frontend
|
||||
|
||||
Use HTML imports with `Bun.serve()`. Don't use `vite`. HTML imports fully support React, CSS, Tailwind.
|
||||
|
||||
Server:
|
||||
|
||||
```ts#index.ts
|
||||
import index from "./index.html"
|
||||
|
||||
Bun.serve({
|
||||
routes: {
|
||||
"/": index,
|
||||
"/api/users/:id": {
|
||||
GET: (req) => {
|
||||
return new Response(JSON.stringify({ id: req.params.id }));
|
||||
},
|
||||
},
|
||||
},
|
||||
// optional websocket support
|
||||
websocket: {
|
||||
open: (ws) => {
|
||||
ws.send("Hello, world!");
|
||||
},
|
||||
message: (ws, message) => {
|
||||
ws.send(message);
|
||||
},
|
||||
close: (ws) => {
|
||||
// handle close
|
||||
}
|
||||
},
|
||||
development: {
|
||||
hmr: true,
|
||||
console: true,
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
HTML files can import .tsx, .jsx or .js files directly and Bun's bundler will transpile & bundle automatically. `<link>` tags can point to stylesheets and Bun's CSS bundler will bundle.
|
||||
|
||||
```html#index.html
|
||||
<html>
|
||||
<body>
|
||||
<h1>Hello, world!</h1>
|
||||
<script type="module" src="./frontend.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
With the following `frontend.tsx`:
|
||||
|
||||
```tsx#frontend.tsx
|
||||
import React from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
|
||||
// import .css files directly and it works
|
||||
import './index.css';
|
||||
|
||||
const root = createRoot(document.body);
|
||||
|
||||
export default function Frontend() {
|
||||
return <h1>Hello, world!</h1>;
|
||||
}
|
||||
|
||||
root.render(<Frontend />);
|
||||
```
|
||||
|
||||
Then, run index.ts
|
||||
|
||||
```sh
|
||||
bun --hot ./index.ts
|
||||
```
|
||||
|
||||
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
||||
@@ -0,0 +1,274 @@
|
||||
# Decisions
|
||||
|
||||
Lightweight ADR log. One entry per decision that shapes the codebase. Statuses:
|
||||
**Accepted** (act on it), **Proposed** (default choice, revisit before building on it),
|
||||
**Superseded** (kept for history, points to replacement).
|
||||
|
||||
Add new entries at the bottom with the next `D-###` id. Never rewrite an Accepted
|
||||
entry to mean something else — supersede it.
|
||||
|
||||
---
|
||||
|
||||
## D-001 — Bun as the only runtime and toolchain
|
||||
**Status:** Accepted (mandated by CLAUDE.md)
|
||||
Bun replaces Node, vite/webpack (via `bun build` + HTML imports), jest/vitest
|
||||
(`bun test`), express (`Bun.serve`), better-sqlite3 (`bun:sqlite`), and dotenv.
|
||||
**Consequence:** no vite plugin ecosystem; PWA assets (service worker, manifest) are
|
||||
wired by hand instead of via a plugin.
|
||||
|
||||
## D-002 — Transformers.js (`@huggingface/transformers`) for all Whisper inference
|
||||
**Status:** Accepted for browser inference; the server side is **superseded by D-013**
|
||||
(faster-whisper).
|
||||
Same library in the browser (WebGPU/WASM) and on the server (ONNX Runtime under Bun).
|
||||
**Why:** one model catalog, one pipeline API, no native whisper.cpp build step.
|
||||
**Trade-off:** server CPU inference is slower than whisper.cpp; acceptable — the server
|
||||
engine is a convenience for logged-in users, not the primary path.
|
||||
|
||||
## D-003 — Inference runs in a dedicated Web Worker
|
||||
**Status:** Accepted
|
||||
Model load + transcription block for seconds-to-minutes; running them on the main
|
||||
thread would freeze the UI. Communication via a small typed postMessage protocol
|
||||
(`src/worker/protocol.ts`) shared between UI and worker.
|
||||
|
||||
## D-004 — WebGPU with automatic WASM fallback
|
||||
**Status:** Accepted (amended by D-017 — "auto" now probes WebNN first)
|
||||
Detect via `navigator.gpu?.requestAdapter()` inside the worker; pass
|
||||
`device: "webgpu" | "wasm"` to the pipeline. The active backend is surfaced in the UI
|
||||
but users don't have to configure anything. Encoder-heavy Whisper benefits ~10-40× from
|
||||
WebGPU where available (Chromium); WASM keeps Firefox/Safari working.
|
||||
|
||||
## D-005 — Model weights cached via the browser Cache API (Transformers.js default)
|
||||
**Status:** Accepted
|
||||
Transformers.js already persists Hub downloads in the Cache API (`useBrowserCache`).
|
||||
We do **not** re-cache weights in the service worker — the SW precaches only the app
|
||||
shell. **Consequence:** offline = installed PWA + at least one previously loaded model;
|
||||
the UI must show per-model "downloaded / not downloaded" state (query
|
||||
`caches` for hub URLs) so users can prepare for offline use.
|
||||
|
||||
## D-006 — Quantized model catalog, curated not exhaustive
|
||||
**Status:** Proposed
|
||||
Expose a curated list (`src/lib/models.ts`): `whisper-tiny(.en)`, `base(.en)`,
|
||||
`small(.en)`, `medium(.en)`, `large-v3-turbo`, each with sensible dtype defaults
|
||||
(e.g. q4/q8 on WASM, fp16/q4 on WebGPU) and a size estimate shown before download.
|
||||
**Why not free-form model ids:** predictable UX, tested combinations, honest
|
||||
size/VRAM warnings. Revisit if users ask for arbitrary Hub ids.
|
||||
|
||||
## D-007 — PWA: hand-rolled service worker, app-shell precache
|
||||
**Status:** Accepted
|
||||
No Workbox/vite-plugin-pwa (conflicts with D-001 tooling). `bun build` emits hashed
|
||||
assets to `dist/`; a small build step writes the precache manifest into `sw.ts`.
|
||||
Strategy: precache app shell, cache-first for static assets, network-only for `/api/*`.
|
||||
|
||||
## D-008 — Passport.js via a connect-compat adapter over Bun.serve
|
||||
**Status:** Accepted (Passport mandated by CLAUDE.md; adapter is the how)
|
||||
Passport expects connect/express-style `(req, res, next)` and `req.login()`. We are
|
||||
not allowed to use express (D-001), so `src/server/auth.ts` provides a thin adapter:
|
||||
wrap Bun's `Request` in a minimal req/res shim, invoke the passport authenticate
|
||||
chain, translate the outcome back to a `Response`. Local (username/password)
|
||||
strategy first; other strategies plug into the same adapter later.
|
||||
**Alternative rejected:** dropping Passport for hand-rolled auth — CLAUDE.md names
|
||||
Passport explicitly.
|
||||
|
||||
## D-009 — Sessions + users in `bun:sqlite`
|
||||
**Status:** Accepted
|
||||
Single-file SQLite DB (path from env, default `./data/app.db`). Tables: `users`
|
||||
(id, username, password_hash — Argon2 via `Bun.password`), `sessions`
|
||||
(id, user_id, expires_at). Cookie: HttpOnly, SameSite=Lax, Secure in production.
|
||||
No Redis/Postgres until there's a scaling reason (would be `Bun.redis` / `Bun.sql`).
|
||||
|
||||
## D-010 — Engine abstraction: `BrowserEngine` and `ServerEngine` behind one interface
|
||||
**Status:** Accepted
|
||||
`src/lib/engine.ts` defines `Engine { load(model), transcribe(audio), onProgress }`.
|
||||
The UI never branches on engine type beyond the toggle. Keeps the server engine
|
||||
(milestone M5) from leaking into UI code built in M1–M4.
|
||||
|
||||
## D-011 — Server engine transport: streaming HTTP response first, WebSocket if needed
|
||||
**Status:** Accepted (implemented in M5, 2026-07-10)
|
||||
`POST /api/transcribe` returns a streamed (chunked JSON-lines) response with progress
|
||||
and partial transcripts. Simpler than WebSockets (no connection lifecycle, works with
|
||||
plain fetch + ReadableStream). Switch to `Bun.serve` websockets only if we need
|
||||
bidirectional control (e.g. cancel mid-run) that abort signals can't cover.
|
||||
**As implemented:** `application/x-ndjson` of `segment`* then `result` | `error`
|
||||
lines (wire types + guards shared client/server via `src/lib/engine.ts`); the
|
||||
request body is raw float32 mono 16 kHz PCM — the decoded form the browser
|
||||
already has — so the server never decodes containers. Cancel-mid-run needed no
|
||||
WebSocket: `fetch`'s AbortSignal reaches `request.signal`, which forwards a
|
||||
`cancel` to the sidecar (it checks between segments).
|
||||
|
||||
## D-012 — Audio decoding on the main thread via Web Audio API
|
||||
**Status:** Accepted
|
||||
`decodeAudioData` (handles most audio and audio-in-video containers) → mono downmix →
|
||||
resample to 16 kHz → transfer Float32Array to worker (zero-copy). No ffmpeg.wasm
|
||||
unless we hit real container-support gaps; note in `src/lib/audio.ts` if we do.
|
||||
|
||||
## D-013 — Server engine runs faster-whisper (Python sidecar)
|
||||
**Status:** Accepted (user requirement, 2026-07-10; supersedes the server half of D-002)
|
||||
Server-side inference (M5) uses [faster-whisper](https://github.com/SYSTRAN/faster-whisper)
|
||||
(CTranslate2) instead of Transformers.js under Bun. Browser inference is untouched —
|
||||
faster-whisper **cannot** run in a browser (checked 2026-07): it is Python over native
|
||||
C++ CTranslate2, which has no WASM build (OpenNMT/CTranslate2#528 open since 2021, no
|
||||
progress) and relies on native SIMD/threading kernels. So D-002 stays Accepted for the
|
||||
browser, where Transformers.js remains the engine.
|
||||
**Shape:** Bun stays the only web runtime. `src/server/transcribe.ts` talks to a small
|
||||
Python sidecar (`server-py/`, faster-whisper behind a minimal stdio or localhost-HTTP
|
||||
interface) that `Bun.spawn` starts and supervises. The `POST /api/transcribe` contract
|
||||
(D-011: streamed JSON-lines) is unchanged — faster-whisper's segment generator maps
|
||||
naturally onto it.
|
||||
**Consequence:** the "one inference codebase" invariant narrows to "one *browser*
|
||||
inference codebase + one shared model catalog"; `src/lib/models.ts` needs a mapping
|
||||
from catalog ids to faster-whisper/CTranslate2 model names (`Systran/faster-whisper-*`);
|
||||
deployment gains a Python ≥3.9 dependency (server engine only — the anonymous/browser
|
||||
path still deploys with Bun alone, and the app must degrade gracefully when the
|
||||
sidecar is absent).
|
||||
**Why:** CTranslate2 CPU inference is several times faster and more memory-efficient
|
||||
than ONNX Runtime under Bun; the server engine becomes genuinely useful rather than a
|
||||
slow convenience.
|
||||
|
||||
## D-014 — WASM sessions run at graphOptimizationLevel "basic"
|
||||
**Status:** Accepted (bug workaround, 2026-07-10)
|
||||
On the WASM backend, the inference worker passes
|
||||
`session_options: { graphOptimizationLevel: "basic" }` to the Transformers.js
|
||||
pipeline. The onnxruntime-web build bundled with Transformers.js 4.2
|
||||
(1.26.0-dev.20260416) has a broken level-2 QDQ graph transform
|
||||
(`qdq_actions.cc` `TransposeDQWeightsForMatMulNBits`) that fails session
|
||||
creation for the 8-bit Whisper decoders (`_quantized`/`_int8`/`_uint8`), whose
|
||||
tied `embed_tokens` weight is both gathered (embedding) and matmul'd (lm_head):
|
||||
"Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale".
|
||||
Reproduced outside the browser against `decoder_model_merged_quantized.onnx`
|
||||
of `onnx-community/whisper-tiny`; levels "all"/"extended" fail, "basic"/
|
||||
"disabled" load fine, and onnxruntime-node 1.24.3 loads the same file at every
|
||||
level (regression in the ort-web dev build, not the model).
|
||||
**Scope:** WASM only — WebGPU keeps the default level. The q4 conversions use
|
||||
native `MatMulNBits` ops (no QDQ), so skipping level-2 costs the larger WASM
|
||||
models nothing.
|
||||
**Revisit:** when Transformers.js bumps its onnxruntime-web dependency past
|
||||
1.26.0-dev.20260416, retest q8-on-WASM at the default level and drop this.
|
||||
|
||||
## D-015 — Runtime feature flags via server env vars; first flag gates auth
|
||||
**Status:** Accepted (2026-07-10)
|
||||
Feature flags are read from the **server's** process environment at startup
|
||||
(`src/server/flags.ts`, `envFlag()`: unset/empty → the flag's default;
|
||||
`false`/`0`/`no`/`off` any case → off; anything else → on). Runtime-only by
|
||||
design: one Docker image serves every configuration — flags are flipped in
|
||||
docker-compose.yml / `docker run -e`, never at build time, so the client
|
||||
bundle stays identical across deployments.
|
||||
**First flag:** `AUTH_ENABLED` (default **on**) gates the M4 login/register
|
||||
feature. Off, the server 404s every `/api/auth/*` route. The client is never
|
||||
told separately: `fetchAuthStatus` (the `GET /api/auth/me` probe `useAuth`
|
||||
already makes on mount) reads 404 as "feature off" and `<AuthForms />` renders
|
||||
nothing. The Sign in button therefore only appears after the probe confirms
|
||||
the endpoints exist — a disabled deployment never flashes auth UI, and an
|
||||
offline/unreachable server also hides it (signing in would be impossible
|
||||
anyway), preserving the anonymous-first invariant (no error surfaced).
|
||||
**Consequence:** any future flag that must reach the browser needs a
|
||||
server-derived signal (an endpoint the client can probe, like /me here) —
|
||||
`BUN_PUBLIC_*` build-time inlining is not used for feature flags.
|
||||
|
||||
## D-016 — Live microphone transcription via segment re-transcription (browser engine)
|
||||
**Status:** Accepted (2026-07-10; promotes "live transcription" out of the
|
||||
WorkBreakdown out-of-scope list → milestone M7)
|
||||
Live (microphone) transcription runs on the **browser engine only** — same
|
||||
privacy story as file uploads: audio never touches the network, works for
|
||||
anonymous users, works offline. Whisper is not a streaming model (it consumes
|
||||
up-to-30 s windows), so "live" is approximated by a **segment loop**:
|
||||
|
||||
- **Capture (main thread):** `getUserMedia` → `AudioContext` at the device's
|
||||
native rate → an `AudioWorklet` chunker posts ~100 ms Float32 frames to the
|
||||
main thread, which resamples them to 16 kHz with the pure, stateful
|
||||
`StreamingResampler` (windowed-sinc low-pass; `resampleLinear` has no
|
||||
anti-aliasing filter, so 48 kHz → 16 kHz decimation folded everything above
|
||||
8 kHz into the speech band and measurably hurt accuracy — see
|
||||
`tests/resampler.test.ts`). The worklet module is loaded from a **Blob URL**
|
||||
built from an inline code string — no extra served file, no
|
||||
build/service-worker changes, and it keeps working offline. Where
|
||||
`audioWorklet`/Blob modules are unavailable (older Safari), fall back to a
|
||||
`ScriptProcessorNode`. The context is *not* forced to 16 kHz because Firefox
|
||||
cannot connect a `MediaStreamAudioSourceNode` across sample rates.
|
||||
- **Segmentation (pure, unit-tested):** a simple RMS energy gate
|
||||
(`src/lib/live.ts`) splits speech into segments: a segment starts at speech
|
||||
onset (with a short pre-roll), commits after a run of trailing silence or at
|
||||
a max length (under Whisper's 30 s window), and pure-silence audio is
|
||||
dropped entirely — Whisper hallucinates on silence, so never transcribe it.
|
||||
Two accuracy refinements (2026-07-14): the gate has **hysteresis** (once a
|
||||
segment is open, frames stay "speech" down to half the onset threshold, so
|
||||
quiet phonemes don't fragment utterances), and a max-length commit is a
|
||||
**soft cut** — it splits at the last sub-threshold frame in the trailing
|
||||
window instead of mid-word at the hard cap, seeding the next segment with
|
||||
the remainder (a mid-word cut corrupts the boundary word in both segments).
|
||||
Dropped (near-silent) segments also clear any tentative text they produced,
|
||||
so a hallucinated tentative pass can neither linger on screen nor be adopted
|
||||
as the next segment's provisional text. Production tuning lives in
|
||||
`LIVE_TUNING` (`src/lib/live.ts`), shared by `src/App.tsx` and the accuracy
|
||||
harness (`tests/live-accuracy.test.ts` — tone-word oracle engine + WER, so
|
||||
segmentation/scheduling/resampling accuracy is regression-tested under
|
||||
`bun test` without a model).
|
||||
No ML VAD (e.g. Silero) until the energy gate proves insufficient in real use.
|
||||
- **Inference (existing worker, serialized):** the current segment is
|
||||
re-transcribed through the **existing** inference worker/pipeline on a
|
||||
rolling basis (tentative text, replaced each pass); a committed segment gets
|
||||
one final pass and its text freezes. At most one inference is in flight;
|
||||
final passes take priority over tentative refreshes, which are throttled and
|
||||
always use the latest audio snapshot (intermediate snapshots are skipped, so
|
||||
the loop degrades gracefully when inference is slower than real time).
|
||||
- **Protocol:** UI↔worker messages gain an optional `requestId` echoed on
|
||||
`partial`/`result`/`error`, so the live controller can correlate responses
|
||||
and coexist with the file flow. The file flow keeps working without ids.
|
||||
|
||||
**Alternatives rejected:** streaming into the worker with a custom
|
||||
chunk-stitching decoder (large fork of Transformers.js internals for marginal
|
||||
latency); MediaRecorder + re-decoding compressed blobs (adds codec latency and
|
||||
`decodeAudioData` round-trips for no benefit over raw PCM taps).
|
||||
**Consequence:** live quality/latency is bounded by model speed on the user's
|
||||
device — the UI must keep tentative text visibly distinct from committed text.
|
||||
Server-engine live input (WebSocket streaming to faster-whisper) is a separate
|
||||
future decision; D-011's "WebSocket if needed" note is the hook.
|
||||
|
||||
## D-017 — "Auto" device order: WebNN → WebGPU → WASM
|
||||
**Status:** Accepted (2026-07-14; amends D-004, which only knew WebGPU → WASM)
|
||||
With WebNN support added (commit "WebNN support initial"), the `"auto"` device
|
||||
preference resolves in priority order **WebNN → WebGPU → WASM**: the worker
|
||||
probes `navigator.ml` (WebNN) first, then `navigator.gpu.requestAdapter()`
|
||||
(WebGPU), and always keeps WASM as the last resort. The probe/ordering policy
|
||||
is pure and unit-tested (`deviceAttempts` in `src/worker/protocol.ts`,
|
||||
`tests/protocol.test.ts`); the worker only supplies the capability probe.
|
||||
Pipeline-creation failures walk the same chain — an auto-detected WebNN that
|
||||
fails still tries WebGPU before WASM. An **explicit** `"webnn"` or `"webgpu"`
|
||||
preference deliberately falls back straight to WASM (no detour through the
|
||||
other accelerator — the user asked for a specific backend, so honor it or take
|
||||
the universal fallback). The selected backend stays user-visible: the worker's
|
||||
`{status:"ready", device}` message drives a "Running on …" badge next to the
|
||||
Backend select (`ModelPicker`) and the engine line in the footer (D-004's
|
||||
"surfaced in the UI" requirement).
|
||||
|
||||
## D-018 — Cancel a file transcription by terminating the worker
|
||||
**Status:** Accepted (2026-07-14)
|
||||
Users can cancel an in-flight file run (decode, model download kicked off by
|
||||
the drop, or the transcription pass itself) via a Cancel button next to the
|
||||
file name. Transformers.js exposes no abort API for a running `asr()` call,
|
||||
and on the WASM backend the pass blocks the worker's event loop — a "cancel"
|
||||
postMessage would sit unread until the pass finished anyway. So cancellation
|
||||
**terminates the inference worker** outright; it is the only reliable
|
||||
preemption and also aborts an in-flight weight download. A `fileRun` counter
|
||||
in `App.tsx` marks the decode that was running when the user cancelled as
|
||||
stale (`decodeAudioData` itself cannot be aborted), so it never posts audio to
|
||||
the fresh worker. The next run respawns the worker and reloads the pipeline —
|
||||
cheap, because the weights stay in the browser Cache API (D-005); the UI
|
||||
resets `loadStarted`/`loadProgress` so the reload starts from a clean slate.
|
||||
Live mode is untouched: it has its own stop/drain semantics (D-016), and a
|
||||
live session never coexists with a file run (mutually exclusive in the UI).
|
||||
**Alternatives rejected:** a cancel message handled between chunks (dead on
|
||||
WASM, where the event loop is blocked; and still seconds of latency
|
||||
elsewhere); `AbortSignal` plumbed into Transformers.js (no upstream support).
|
||||
**Consequence:** cancel throws away the loaded pipeline even when only the
|
||||
transcription was unwanted — an accepted cost since reloading from cache takes
|
||||
a few seconds.
|
||||
**Amendment (2026-07-14, EngineToggle landing):** a server-engine run cancels
|
||||
by aborting the `/api/transcribe` fetch instead (`AbortController`; the server
|
||||
cancels the sidecar pass between segments, D-011/D-013) — the inference worker
|
||||
and its loaded pipeline are left untouched in that path.
|
||||
|
||||
## D-019 — Server-based live microphone transcription
|
||||
**Status:** Accepted
|
||||
**Why:** M7 future want. Enables logged-in users on weak devices to transcribe live microphone audio using the server engine.
|
||||
**Shape:** The client opens a WebSocket connection to `/api/transcribe/live`. The client streams 16kHz Float32 PCM chunks (from `MicCapture`) over the WebSocket. The server runs `LiveTranscriber` and `LiveSegmenter` (from `src/lib/live.ts`, which is pure and runs perfectly in Node/Bun). For each segment/pass, the server's `LiveTranscriber` writes the audio to a temporary file and invokes the Python sidecar. The server sends `LiveUpdate` objects back to the client over the WebSocket.
|
||||
**Consequence:** The client does zero ML and minimal buffering, acting only as a mic-streamer and UI-renderer for live passes on the server.
|
||||
@@ -0,0 +1,284 @@
|
||||
# Work Breakdown
|
||||
|
||||
Milestones are ordered so that **every milestone ends in a demoable, working app**.
|
||||
Within a milestone, tasks are roughly in dependency order. Check items off as they
|
||||
land; add follow-up tasks where they belong rather than in a separate backlog.
|
||||
|
||||
Legend: `[ ]` todo · `[x]` done · `[~]` in progress
|
||||
|
||||
---
|
||||
|
||||
## WIP Tasks before usage limit cutoff:
|
||||
- [x] Allow users to cancel current transcription. — Cancel button on the
|
||||
file line, wired to worker termination (D-018); covers decode, pending
|
||||
model download, and the pass itself. Server-engine cancel is tracked in
|
||||
M6 (needs the M5 EngineToggle first).
|
||||
- [x] Better user-feedback while transcription is in progress (better than
|
||||
"Listening for the first few words") — the worker now posts
|
||||
`transcribe-progress` (seconds processed / total, from the streamer's
|
||||
chunk timestamps via the pure `TranscribeProgressTracker`); the UI shows
|
||||
a percent in the heading plus a progress bar with mm:ss, and the file
|
||||
flow's placeholder reads "Processing audio…" (the live wording stays
|
||||
live-only).
|
||||
|
||||
## M0 — Scaffold cleanup & docs (small)
|
||||
Current state is the raw `bun init` React template.
|
||||
|
||||
- [x] `Architecture.md`, `Decisions.md`, `WorkBreakdown.md` created
|
||||
- [x] Remove template cruft: `APITester.tsx`, hello routes in `src/index.ts`,
|
||||
Bun/React logo branding in `App.tsx` / `index.html`
|
||||
- [x] Add `@huggingface/transformers` dependency
|
||||
- [x] Target directory layout from `Architecture.md` §3 (empty modules with TODOs are fine)
|
||||
- [x] Smoke test in `tests/` proving `bun test` runs
|
||||
|
||||
## M1 — Browser transcription MVP (core)
|
||||
Anonymous user can transcribe an uploaded file with one hardcoded model. **This is the
|
||||
riskiest milestone — do it first, learn from it.**
|
||||
|
||||
- [x] `src/worker/protocol.ts` — typed messages: load / load-progress / transcribe /
|
||||
partial / result / error
|
||||
- [x] `src/worker/inference.worker.ts` — Transformers.js ASR pipeline
|
||||
(`onnx-community/whisper-tiny.en` hardcoded, WASM only for now), chunked
|
||||
with timestamps, partials streamed via `WhisperTextStreamer`
|
||||
- [x] `src/lib/audio.ts` — decode upload → 16 kHz mono Float32Array (D-012)
|
||||
- [x] `FileDropzone` component (drag-drop + file picker, audio/* and video/*)
|
||||
- [x] `TranscriptView` component — streaming partials, final text, copy button
|
||||
- [x] Model download progress UI (bytes / %, per-file)
|
||||
- [x] Wire into `App.tsx`; delete remaining template UI
|
||||
- [x] Unit tests for audio resampling and protocol handling
|
||||
|
||||
Notes (M1):
|
||||
- Bun 1.3 does not bundle `new Worker(new URL(...))` graphs (verified against
|
||||
both `bun build` and the dev server), so `src/index.ts` bundles the worker
|
||||
with `Bun.build` at startup and serves it at `/inference.worker.js`; the UI
|
||||
spawns `new Worker("/inference.worker.js", { type: "module" })`. Revisit if
|
||||
Bun gains native worker bundling.
|
||||
- onnxruntime-web fetches its `.wasm` binaries from the jsdelivr CDN
|
||||
(Transformers.js default). Fine for M1; M3 (offline) must serve/cache them
|
||||
locally.
|
||||
|
||||
## M2 — Model choice + WebGPU (core)
|
||||
- [x] `src/lib/models.ts` — curated catalog with sizes and dtype defaults (D-006)
|
||||
- [x] `ModelPicker` component — size estimates, English-only vs multilingual,
|
||||
downloaded-state badge (query Cache API)
|
||||
- [x] WebGPU detection in worker, `device` selection with WASM fallback (D-004);
|
||||
surface active backend in UI
|
||||
- [x] Handle model switching (dispose old pipeline, load new)
|
||||
- [x] Language / task options for multilingual models (transcribe vs translate)
|
||||
- [~] Verify quantized variants (q4/q8/fp16) on both backends; record working
|
||||
combinations in `models.ts`
|
||||
|
||||
Notes (M2):
|
||||
- Quantized-variant verification is pending manual in-browser testing (no real
|
||||
browser in the dev environment). `models.ts` encodes the community-known-good
|
||||
combinations — WASM: q8 (tiny→small) / q4 (medium, large-v3-turbo); WebGPU:
|
||||
fp32 encoder+decoder for tiny/base (fp16 encoders known-bad there), fp16
|
||||
encoder + q4 merged decoder for small/medium/large-v3-turbo — each with a
|
||||
`verified` flag and note. Only `whisper-tiny.en @ wasm/q8` (exercised in M1)
|
||||
is marked `verified: true`; flip the flags as manual testing confirms each.
|
||||
- Downloaded badge (D-005): `src/lib/modelCache.ts` inspects Cache Storage for
|
||||
transformers-cache entries with `.onnx` hits per model id; M3's offline
|
||||
indicator should reuse `getDownloadedModels`.
|
||||
|
||||
## M3 — PWA & offline (core)
|
||||
- [x] `manifest.webmanifest` + icons; installability
|
||||
- [x] `src/pwa/sw.ts` — app-shell precache, cache-first static, network-only API (D-007)
|
||||
- [x] Build step: emit precache manifest from `bun build` output into the SW
|
||||
- [x] SW registration + update flow (new-version toast)
|
||||
- [~] Offline verification: airplane-mode transcription with a previously
|
||||
downloaded model; document the manual test procedure in `tests/offline.md`
|
||||
— procedure written (`tests/offline.md`), pending a manual run in a real
|
||||
browser (none in the dev environment; same caveat as M2's variants)
|
||||
- [x] "Available offline" indicator per model in `ModelPicker`
|
||||
|
||||
Notes (M3):
|
||||
- `sw.ts` ships with `"__PRECACHE_MANIFEST__"` / `__CACHE_VERSION__` string
|
||||
placeholders; `src/pwa/precache.ts` (pure, unit-tested) splices in the real
|
||||
values. Production: `scripts/build.ts` injects the hashed `dist/` asset list
|
||||
and a content-hash version into `dist/sw.js`. Dev: `src/index.ts` injects an
|
||||
empty list + `"dev"`, so the shell is runtime-cached network-first and HMR
|
||||
keeps working. SW bundles are never minified — the placeholders must survive
|
||||
verbatim.
|
||||
- onnxruntime's `.wasm`/`.mjs` are served from our origin at `/ort/*` (dev:
|
||||
straight from `node_modules/onnxruntime-web/dist`; prod: copied to
|
||||
`dist/ort/`) instead of the jsdelivr CDN, and the SW runtime-caches them
|
||||
cache-first into `whisper-runtime-*` — required for offline inference, too
|
||||
big to precache. Model weights stay in Transformers.js's own
|
||||
`transformers-cache`, which the SW never touches (D-005).
|
||||
- Bun's HTML loader must resolve every `<link href>` to a real file, so
|
||||
`index.html` references `./pwa/manifest.webmanifest` / `./pwa/icons/...`
|
||||
relatively (absolute `/...` URLs fail the build — bug found while verifying
|
||||
M3). The bundler emits hashed copies; the fixed `/manifest.webmanifest` and
|
||||
`/icons/*` URLs are still served for the manifest's internal icon refs.
|
||||
`ensureIcons()` must run before the HTML is bundled (icons are generated,
|
||||
not checked in).
|
||||
- Production smoke-tested via curl: `/` and deep links → shell (no-cache),
|
||||
hashed chunks → immutable, `/sw.js` → no-cache, `/manifest.webmanifest` →
|
||||
`application/manifest+json`, `/ort/*.wasm` → `application/wasm`, `/api/*` →
|
||||
404 network-only. Browser-level behavior (install, airplane mode, update
|
||||
toast) is manual — see `tests/offline.md`.
|
||||
|
||||
## M4 — Accounts (Passport.js)
|
||||
- [x] `src/server/db.ts` — bun:sqlite schema: users, sessions (D-009)
|
||||
- [x] Connect-compat adapter + Passport local strategy in `src/server/auth.ts` (D-008)
|
||||
- [x] Routes: `POST /api/auth/register|login|logout`, `GET /api/auth/me`
|
||||
- [x] Password hashing via `Bun.password` (Argon2)
|
||||
- [x] Frontend: login/register forms, session-aware header, logged-out ⇒ no
|
||||
behavioral change (anonymous-first invariant)
|
||||
- [x] Auth tests (register/login/logout, session expiry, wrong password)
|
||||
|
||||
Notes (M4):
|
||||
- DB path env var is `DB_PATH` (default `./data/app.db`, gitignored); tests
|
||||
inject `createDb(":memory:")` and never touch disk. Session ids are 256-bit
|
||||
base64url; expiry is enforced server-side via `sessions.expires_at`, not just
|
||||
the cookie. Cookie: `sid`, HttpOnly, SameSite=Lax, Max-Age 7d, Secure in
|
||||
production.
|
||||
- One shared `/api/*` dispatcher in `src/index.ts` serves both the dev `routes`
|
||||
table and the production `fetch` handler — new API routes go there.
|
||||
- `src/server/auth.ts` exports `getSessionUser(request)` for M5's auth-guarded
|
||||
`/api/transcribe`. Frontend auth state lives in `src/lib/useAuth.ts` over the
|
||||
framework-free `src/lib/authClient.ts`; `/api/auth/me` failures (401 or
|
||||
offline) fail silent to anonymous — the app is offline-first.
|
||||
|
||||
## M5 — Server engine + toggle
|
||||
- [x] Server-side pipeline in `src/server/transcribe.ts` — **faster-whisper**
|
||||
(CTranslate2) via a Python sidecar in `server-py/`, spawned/supervised by
|
||||
Bun (D-013 supersedes the Transformers.js-under-Bun plan); map the shared
|
||||
model catalog to faster-whisper model ids in `src/lib/models.ts`
|
||||
- [x] Degrade gracefully when Python/the sidecar is absent (server engine
|
||||
unavailable; browser engine untouched — anonymous path deploys with Bun alone)
|
||||
- [x] `POST /api/transcribe` — auth-guarded, streamed JSON-lines response (D-011),
|
||||
upload size limit, request abort handling; plus `GET /api/transcribe/status`
|
||||
(availability probe for the toggle)
|
||||
- [x] `src/lib/engine.ts` — `Engine` interface; `BrowserEngine` (wraps worker) and
|
||||
`ServerEngine` (wraps fetch stream) (D-010)
|
||||
- [x] `EngineToggle` component — visible only with a session; persists choice
|
||||
(persistence helpers + status probe already in `lib/engine.ts`; remaining:
|
||||
the component, lifting `useAuth` out of `AuthForms` into `App` so both see
|
||||
one session state, and routing the file flow through the selected engine —
|
||||
live stays on `BrowserEngine` per D-016)
|
||||
- [x] Server model caching/warm-up strategy (don't reload weights per request)
|
||||
- [~] Integration test: same fixture audio through both engines, transcripts agree
|
||||
— supervisor/HTTP layers covered in `tests/transcribe.test.ts` (real
|
||||
`Bun.spawn` against `tests/fixtures/fake_sidecar.py`); real inference is
|
||||
the opt-in `FW_INTEGRATION=1` test + manual run in `tests/server-engine.md`
|
||||
(no pip/browser in the dev environment; same caveat as M2/M3)
|
||||
- [x] Deployment: optional Python/faster-whisper layer for the Docker image +
|
||||
deployer note (base image stays Bun-only; install
|
||||
`server-py/requirements.txt` to enable the server engine)
|
||||
|
||||
Notes (M5):
|
||||
- Sidecar protocol (`server-py/sidecar.py`): JSON-lines over stdio —
|
||||
`ready`/`fatal` handshake, `transcribe` → `segment`* → `done`, `cancel`
|
||||
between segments, `shutdown`/EOF exits. Audio is handed over as a temp file
|
||||
of raw float32 mono 16 kHz PCM (exactly what the browser engine consumes),
|
||||
read with `np.fromfile` — no container decoding server-side. The last-used
|
||||
model stays loaded (replaced on change, so memory is bounded to one model);
|
||||
device/compute via `FASTER_WHISPER_DEVICE`/`FASTER_WHISPER_COMPUTE`
|
||||
(default auto/int8).
|
||||
- Supervision (`createSidecar`): lazy spawn on first use; a crash rejects
|
||||
in-flight requests and the next request respawns; a missing runtime
|
||||
(no python3 / `fatal` handshake) is remembered as a *permanent* unavailable
|
||||
reason — surfaced verbatim by `GET /api/transcribe/status` and 503s — so a
|
||||
Bun-only deployment never respawn-storms. Tests force this path with
|
||||
`python3 -S -I` (site-packages off) against the real sidecar.
|
||||
- Upload limit: `TRANSCRIBE_MAX_UPLOAD_BYTES` (default 512 MiB ≈ 2¼ h of raw
|
||||
f32 PCM), enforced pre-read via Content-Length *and* while streaming the
|
||||
body to the temp file. Errors after the 200 header travel in-stream as an
|
||||
`{"type":"error"}` line (wire format shared with the client via
|
||||
`src/lib/engine.ts`, so server and `ServerEngine` cannot drift).
|
||||
- `AUTH_ENABLED=false` (D-015) 404s `/api/transcribe*` along with the auth
|
||||
routes — the server engine is session-gated, so it disappears with auth.
|
||||
- Smoke-tested live against the dev server: `/` + auth unchanged; status
|
||||
401 → (register) → 200 `{available:false, reason:"faster-whisper is not
|
||||
installed…"}`; POST → 503 with the same reason, body untouched.
|
||||
- EngineToggle: one `useAuth()` lives in `App` (passed into `AuthForms` and
|
||||
gating the toggle); the status probe runs on sign-in and the server option
|
||||
renders disabled-with-reason when unavailable. The file flow now goes
|
||||
through the `Engine` interface for **both** paths (D-010): `BrowserEngine`
|
||||
shares the worker with the live loop (distinct requestId namespaces) and a
|
||||
new `stream` flag on the transcribe message keeps token partials +
|
||||
transcribe-progress flowing for id-carrying engine passes (live passes stay
|
||||
non-streaming). Server runs skip the browser model load, show segment
|
||||
partials without a progress bar (faster-whisper's stream carries no window
|
||||
boundaries), and cancel by aborting the fetch (D-018 amendment) — browser
|
||||
runs keep worker-termination cancel. The choice persists in localStorage
|
||||
but only takes effect while signed in with the engine available.
|
||||
- Deployment: multi-stage `Dockerfile` — `runtime` (Bun-only, 244 MB) is the
|
||||
default target so plain `docker build .` (existing CI) is unchanged;
|
||||
`--target with-server-engine` (405 MB) adds python3 + faster-whisper in a
|
||||
venv at `/opt/faster-whisper` (wired via `SIDECAR_PYTHON`), with
|
||||
`FASTER_WHISPER_DOWNLOAD_ROOT` under the `/app/data` volume so weights
|
||||
persist. README.md documents both flavors, all runtime env vars, the DB
|
||||
volume, and the HTTPS/reverse-proxy requirement. Verified live in Docker:
|
||||
Bun-only image degrades per D-013 (`available:false`, "python3 is not
|
||||
available…"); server-engine image ran a real `POST /api/transcribe` of
|
||||
`en_aiproblem.m4a` end-to-end, streamed segments matching the fixture
|
||||
transcript — the server half of the M5 integration-test item is now
|
||||
exercised for real (browser half still needs the manual run).
|
||||
|
||||
## M6 — Hardening & polish
|
||||
- [ ] Error taxonomy: unsupported file, out-of-memory (suggest smaller model),
|
||||
WebGPU device-lost, network loss mid-download (resume or clear retry)
|
||||
- [x] Cancel in-flight transcription (both engines) — browser engine via
|
||||
worker termination (D-018), server engine via aborting the
|
||||
`/api/transcribe` fetch (D-018 amendment; landed with M5's EngineToggle)
|
||||
- [x] Export transcript: .txt and .srt (timestamps already available) —
|
||||
pure formatters in `src/lib/export.ts` (unit-tested: SRT cue numbering,
|
||||
HH:MM:SS,mmm timestamps, null-end segments borrow the next cue's start
|
||||
or fall back to +2 s, filename derived from the source file); .txt/.srt
|
||||
download buttons next to Copy in `TranscriptView`. Live-mode results
|
||||
reuse the same `result` state with absolute timestamps (D-016), so both
|
||||
flows export identically.
|
||||
- [x] Storage management UI: list cached models, per-model delete —
|
||||
collapsible "Storage" panel (`StorageManager`) listing cached models
|
||||
with exact on-disk size (summed `blob().size`, measured lazily on
|
||||
open) and per-model delete; delete is confirm-free (weights are just
|
||||
cache — they re-download on next use) and refreshes the picker's
|
||||
downloaded/offline badges. Helpers in `src/lib/modelCache.ts` take an
|
||||
injectable Cache-Storage-shaped object (mocked in
|
||||
`tests/modelCache.test.ts`). Scope is the transformers-cache only:
|
||||
the SW's `whisper-runtime-*` cache (onnxruntime .wasm, shared by all
|
||||
models, versioned/cleaned by the SW itself) is deliberately not
|
||||
surfaced — deleting it per-model makes no sense and the SW would
|
||||
just re-fetch it.
|
||||
- [ ] Basic a11y pass (keyboard, focus, aria on progress/live transcript)
|
||||
- [ ] Production build + `bun start` deployment check; README for deployers
|
||||
|
||||
## M7 — Live microphone transcription (browser engine, D-016)
|
||||
Promoted out of the out-of-scope list via D-016. Browser engine only: audio
|
||||
never touches the network, anonymous-first, works offline once a model is
|
||||
cached.
|
||||
|
||||
- [x] Protocol: optional `requestId` on `transcribe`, echoed on
|
||||
`partial`/`result`/`error` so the live loop can correlate responses
|
||||
(file flow unchanged without ids)
|
||||
- [x] `src/lib/live.ts` — pure, unit-tested segment loop: RMS energy gate,
|
||||
`LiveSegmenter` (speech onset + pre-roll, silence/max-length commits,
|
||||
pure silence dropped), `LiveTranscriber` (single-flight scheduling,
|
||||
final passes before throttled tentative refreshes, stop/drain,
|
||||
absolute-timestamped chunks)
|
||||
- [x] `src/lib/mic.ts` — `getUserMedia` + AudioWorklet chunker from a Blob URL
|
||||
(ScriptProcessorNode fallback), native-rate capture resampled to 16 kHz
|
||||
on the main thread (D-016)
|
||||
- [x] `LiveControls` component — start/stop button + recording indicator;
|
||||
mutually exclusive with the file flow
|
||||
- [x] Live transcript UI: committed text + visually-distinct tentative tail in
|
||||
`TranscriptView`; final stop() result reuses the normal transcript view
|
||||
(copy button, timestamps)
|
||||
- [x] Unit tests: segmenter behavior, transcriber scheduling with a fake
|
||||
engine and injected clock, protocol `requestId` guards
|
||||
- [~] Manual in-browser verification (mic permission, live text, offline run)
|
||||
— procedure in `tests/live.md`, pending a real browser (same dev-env
|
||||
caveat as M2/M3)
|
||||
- [ ] **Future want:** live transcription on the **server engine** — stream
|
||||
mic audio over a WebSocket (D-011's escape hatch) to faster-whisper
|
||||
(D-013) for logged-in users on weak devices. Needs its own decision
|
||||
entry; blocked on M5 landing first.
|
||||
|
||||
---
|
||||
|
||||
## Deliberately out of scope (for now)
|
||||
Arbitrary Hub model ids, OAuth providers, multi-user transcript storage,
|
||||
whisper.cpp server backend, i18n of the UI.
|
||||
Promote to a milestone via a new entry in `Decisions.md` first.
|
||||
Reference in new issue
Block a user