Better tests and more user feedback while doing transcription; WIP.

This commit is contained in:
Lucas Winther committed 2026-07-14 14:44:01 +02:00
1 parent 18acead43c
commit be82ecfe57
19 files changed
+447 -138

No files matched your search

+13 -9
View File
@@ -24,7 +24,7 @@ Logged-in users may optionally route inference to the server.
│ ▼ │
│ Inference Web Worker │
│ ├─ Transformers.js ASR pipeline (Whisper) │
│ ├─ Device selection: WebGPU → WASM fallback │
│ ├─ Device selection: WebNN → WebGPU → WASM fallback │
│ └─ Model weights cached via browser Cache API │
│ │
│ Service Worker (PWA) │
@@ -72,7 +72,7 @@ src/
modelCache.ts # "downloaded?" per model via Cache Storage (D-005; reused by M3 offline indicator)
engine.ts # engine abstraction: BrowserEngine | ServerEngine
worker/
inference.worker.ts # Transformers.js pipeline, WebGPU/WASM selection
inference.worker.ts # Transformers.js pipeline, WebNN/WebGPU/WASM selection
protocol.ts # typed postMessage messages (shared with UI)
pwa/
sw.ts # service worker (app-shell precache, /ort runtime cache)
@@ -89,7 +89,8 @@ tests/ # bun test
```
(Current state: M0–M3 built — browser transcription with the curated model
catalog, WebGPU→WASM auto-selection, model switching, language/task options,
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
language/task options,
and the installable offline PWA (`pwa/`, `scripts/build.ts` production build,
`dist/` serving in `src/index.ts`; browser-level offline behavior pending the
manual run in `tests/offline.md`). M4 accounts are built (`server/db.ts`,
@@ -108,12 +109,15 @@ are the remaining M5 work; see `WorkBreakdown.md`.)
1. User picks a model in `ModelPicker` (curated catalog from `lib/models.ts`, with
size estimates and a Cache-API-backed "downloaded" badge); UI posts
`{type:"load", modelId, device}` where `device` is a preference
(`"auto" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype` override
is expressible in the message but normally omitted.
2. Worker resolves the device: `navigator.gpu?.requestAdapter()` → `"webgpu"` if an
adapter exists (and pipeline creation succeeds — otherwise it retries on WASM),
else `"wasm"` (D-004). It then resolves the catalog's dtype default for that
device and creates the Transformers.js `automatic-speech-recognition` pipeline.
(`"auto" | "webnn" | "webgpu" | "wasm"`, default `"auto"`). An explicit `dtype`
override is expressible in the message but normally omitted.
2. Worker resolves the device (D-004, D-017): for `"auto"` it probes
`navigator.ml` (WebNN) first, then `navigator.gpu?.requestAdapter()` (WebGPU),
and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the
picked device throws, it retries down the same chain (WASM always last). An
explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM. It then
resolves the catalog's dtype default for the device actually used and creates
the Transformers.js `automatic-speech-recognition` pipeline.
Download progress events are forwarded to the UI; the terminal
`{status:"ready", device}` message reports the backend actually selected, which
the UI surfaces. Loading a different model/device disposes the old pipeline