diff --git a/docs/Architecture.md b/docs/Architecture.md index 36df6c9..71b1362 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -118,7 +118,13 @@ are the remaining M5 work; see `WorkBreakdown.md`.) `navigator.ml` (WebNN) first, then `navigator.gpu?.requestAdapter()` (WebGPU), and picks in priority order WebNN → WebGPU → WASM; if pipeline creation on the picked device throws, it retries down the same chain (WASM always last). An - explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM. It then + explicit `"webnn"`/`"webgpu"` preference falls back straight to WASM. + Two D-025 refinements: WebGPU is skipped up front when the dtype it would + use needs fp16 but the adapter lacks `shader-f16` (Firefox), and a WASM + fallback *after* a failed accelerator attempt does not run in-place — the + failed attempt pinned ort's thread pool at 1 — but posts `load-restart`, + and the UI respawns the worker with an explicit `wasm` load (transcribe + posts wait on a ready gate in `App.tsx`, so none are lost). It then resolves the catalog's dtype default for the device actually used and creates the Transformers.js `automatic-speech-recognition` pipeline. Download progress events are forwarded to the UI; the terminal diff --git a/docs/Decisions.md b/docs/Decisions.md index 81862c1..69b6897 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -437,3 +437,105 @@ start's region. Covered by `tests/live.test.ts` (batching + cap) and the WER gates (`tests/live-accuracy.test.ts`, `tests/live-real-model.test.ts` — the real-model test's fast feed makes real inference lag, so merging is exercised there with real weights). + +## D-025 — A degraded-thread WASM fallback restarts the worker; WebGPU is skipped when fp16 needs an absent shader-f16 +**Status:** Accepted (2026-07-17) +Reported symptom: Firefox performance far below Chrome, live transcription +worst. Chrome resolves D-017's auto chain to an accelerator; Firefox has no +WebNN and its WebGPU (where exposed at all) fails onnxruntime-web's +Chromium-built JSEP EP — notably it lacks the `shader-f16` feature, which the +catalog's fp16-encoder WebGPU defaults (small, large-v3-turbo) need. So +Firefox lands on WASM — but *how* it lands there was the real bug: the failed +WebGPU attempt initializes ort's WASM runtime at the accelerator-safe +`numThreads: 1` (D-023 amendment), the thread pool is fixed at first init, +and the fallback session then runs **single-threaded for the worker's whole +life** (measured 5.5×: 52 s vs 9.5 s on a 14 s clip). Live never respawns the +worker, and every live pass pads to a full 30 s window (D-024), so passes +took ~40–60 s: tentative text froze and the stop drain ballooned — D-023 +amendment's "correct over fast" edge case is Firefox's *default* path. +**Fix (three parts):** +1. **Restart instead of a degraded fallback.** When `createPipeline` reaches + the `wasm` attempt after any accelerator attempt ran and multithreading is + actually available (`desiredWasmThreads() > 1`), the worker posts + `{type:"load-restart", modelId, preference, device:"wasm"}` and aborts the + load (sentinel error, never surfaced). The UI terminates the worker, + respawns it, re-issues the load with the explicit device, and remembers + `modelId::preference → device` for the session so later loads skip the + doomed attempt (session-only: a browser update may fix the backend; an + explicit user backend choice is a different key, so it retries once). + To make the respawn safe, transcribe posts now wait on a **load gate** in + `App.tsx` that resolves on `ready` (rejects on load error/cancel/crash): + no pass is ever queued inside a worker a restart is about to terminate — + its transferred audio buffer would die with it. The worker serialized + passes on the load anyway, so nothing starts later than before. +2. **shader-f16 probe.** The worker's WebGPU probe also records + `adapter.features.has("shader-f16")`; when the dtype that a WebGPU attempt + would use requires fp16 (`dtypeRequiresF16`, pure + unit-tested) and the + feature is absent, WebGPU is skipped outright — Firefox then loads + straight into multithreaded WASM with no wasted attempt. +3. **Live-on-WASM hint.** With live active on `wasm` and a heavy model + (encoder ≥ 300 M params: medium, large-v3-turbo), the UI says the model + runs slower than real time and suggests a smaller one — WASM lag there is + inherent (D-024's fixed window cost), not a bug, so tell the user instead + of looking frozen. +**Alternatives rejected:** re-posting in-flight passes after respawn (their +buffers are transferred — gone with the worker); a scout worker pre-flighting +the accelerator (doubles accelerated-path load time to save the failure +path); ort-web's proxy-worker mode to escape the pinned pool (undocumented +interplay with JSEP). +**Consequence:** the accelerator-failure path costs one worker respawn + a +pipeline reload from cache (a few seconds, once per model+preference per +session). A genuine both-attempts-fail load still reports its error +normally. `deviceAttempts` is unchanged; the new policy pieces +(`dtypeRequiresF16`, the `load-restart` message guard) are unit-tested in +`tests/protocol.test.ts`. Real-Firefox verification (badge shows WASM with +full threads, live keeps up on small models) still needs a manual run — same +no-browser dev-env caveat as M2/M3. +**Amendment (2026-07-17, same day — field report: tiny fixed, medium still +slow in Firefox):** three observability/robustness additions to tell "slow is +the CPU-vs-GPU physics" apart from "something is still degraded": +1. The probe **skips software (fallback) WebGPU adapters** + (`isFallbackAdapter`, both spec locations) — SwiftShader/llvmpipe/WARP + would *succeed* at session creation and then run slower than multithreaded + WASM, and the fallback chain only reacts to failures, not slowness. + onnxruntime-web itself uses whatever adapter it is handed (verified in + 1.26.0-dev: no fallback-adapter or Firefox guard). +2. The `ready` message now carries **`threads`** — read back from + `env.backends.onnx.wasm.numThreads` *after* session creation, so ort's own + clamp-to-1 (missing isolation) is visible. The footer shows it: + "WASM (on-device, 11 threads)". +3. The worker **logs the device chain** (`[whisper] …` console lines): each + skipped backend (software adapter, missing shader-f16), each failed + attempt with its error, and the final device+threads — previously fallback + errors were swallowed silently, making field reports unactionable. +Expected healthy Firefox numbers for medium q4 on WASM (from the D-023/D-024 +measurements): ~4 s per 30 s window at 12 threads, ~9–12 s at 4 — i.e. a +14 s clip in ~10 s. Chromium's GPU path is a further ~3–5× faster; that +remaining gap is inherent until Firefox's WebGPU works with ort-web's JSEP +EP. If the badge shows 1 thread or the console shows a surviving software +adapter, it is a bug in this decision's scope instead. + +## D-026 — "Auto" never picks WebGPU on Firefox +**Status:** Accepted (2026-07-17) +Field-verified on Firefox (D-025's observability build): explicit +Backend = WASM runs medium with a full thread pool and healthy speed, while +explicit Backend = WebGPU (`device=webgpu wasmThreads=1`, correct per D-023) +**completes but runs far slower** than the same machine's multithreaded WASM. +That is the worst case for D-017's failure-driven fallback chain: ort-web's +WebGPU EP (Chromium-developed; no Gecko handling in 1.26.0-dev, verified) +creates the session *successfully* on Firefox, so nothing ever falls back — +"auto" happily selects the slowest available backend. +**Fix:** the worker's capability probe skips the WebGPU probe entirely when +the preference is `"auto"` and the UA is Firefox (`/Firefox\//` on +`navigator.userAgent`), with a `[whisper]` console line saying why. Firefox +auto therefore resolves straight to multithreaded WASM (no accelerator +attempt → no D-025 restart either). An **explicit** WebGPU selection is still +honored — that is the deliberate retest hatch: when a future +Transformers.js/ort-web bump gains real Gecko WebGPU support, verify via the +explicit path, then delete the UA check. +**Why UA sniffing:** there is no feature to detect — the sessions create and +run; only throughput distinguishes them, and a benchmark probe would cost +more than it saves. The check is one line, logged, scoped to `"auto"`, and +targets the one engine with field evidence. +**Revisit:** on any onnxruntime-web bump — retest explicit WebGPU on Firefox +(file `tests/live.md` note) and drop the check when it reaches parity. diff --git a/docs/WorkBreakdown.md b/docs/WorkBreakdown.md index 0e4ed7b..d3eff25 100644 --- a/docs/WorkBreakdown.md +++ b/docs/WorkBreakdown.md @@ -247,6 +247,14 @@ Notes (M5): transcription start (file drop / live start) or via an explicit per-model Download button in `ModelPicker` — and never when the engine toggle is set to "server". +- [x] Firefox performance: never run the WASM fallback single-threaded after + a failed accelerator attempt (worker posts `load-restart`, UI respawns + with explicit wasm — D-025); skip WebGPU when fp16 weights need the + absent `shader-f16` feature; hint that medium/large lag in live mode on + WASM. Field-verified: WASM now gets the full thread pool; ort-web's + WebGPU EP on Firefox completes but runs far slower than WASM, so + "auto" skips WebGPU on Firefox (D-026, explicit selection still + honored — retest on ort bumps). - [ ] Basic a11y pass (keyboard, focus, aria on progress/live transcript) - [ ] Production build + `bun start` deployment check; README for deployers diff --git a/src/App.tsx b/src/App.tsx index ead5c6a..0ba9ef6 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -78,6 +78,22 @@ export function App() { // audio to the (fresh) worker. Decoding itself cannot be aborted. const fileRun = useRef(0); + // Accelerator failures learned this session (D-025), keyed + // "modelId::preference" → the explicit device to load with instead, so + // repeat loads skip the doomed accelerator attempt (and its restart) + // entirely. Session-only on purpose: a browser update may fix the backend. + const deviceOverrides = useRef(new Map()); + + // Resolves when the current browser-engine load reports ready (rejects on + // load failure/cancel). Transcribe posts await it, so no request is ever + // queued inside a worker that a load-restart (D-025) is about to terminate + // — its audio buffer would die with the worker. + const loadGate = useRef<{ + promise: Promise; + resolve: () => void; + reject: (error: Error) => void; + } | null>(null); + // Engine plumbing (M5, D-010). The BrowserEngine is bound to the worker's // lifetime (recreated together on cancel); the ServerEngine is stateless // and lives for the app. A server run's fetch is cancelled via abortRef. @@ -113,6 +129,8 @@ export function App() { DEFAULT_DEVICE_PREFERENCE, ); const [activeDevice, setActiveDevice] = useState(null); + // WASM thread count ort actually got (D-025 observability); null until ready. + const [activeThreads, setActiveThreads] = useState(null); const [language, setLanguage] = useState(AUTO_LANGUAGE); const [task, setTask] = useState(DEFAULT_TASK); const [downloaded, setDownloaded] = useState>(new Set()); @@ -173,6 +191,36 @@ export function App() { }; }, [auth.user]); + function overrideKey(forModelId: string, preference: DevicePreference): string { + return `${forModelId}::${preference}`; + } + + /** Device to actually load with — the preference unless D-025 learned better. */ + function effectiveDevice( + forModelId: string, + preference: DevicePreference, + ): DevicePreference { + return deviceOverrides.current.get(overrideKey(forModelId, preference)) ?? preference; + } + + function newLoadGate(): void { + let resolve!: () => void; + let reject!: (error: Error) => void; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + // Waiters attach their own handlers; this swallows the "unhandled + // rejection" a gate rejected with nobody waiting would otherwise raise. + promise.catch(() => {}); + loadGate.current = { promise, resolve, reject }; + } + + /** Wait until the browser pipeline is ready (or its load failed/cancelled). */ + function browserModelReady(): Promise { + return loadGate.current?.promise ?? Promise.resolve(); + } + function takeLiveRequest(requestId: string | undefined) { if (requestId === undefined) return undefined; const pending = pendingLive.current.get(requestId); @@ -197,7 +245,10 @@ export function App() { // Surface the backend the worker actually selected (D-004) and // refresh the downloaded badges now that weights are cached. setActiveDevice(message.device); + setActiveThreads(message.threads ?? null); refreshDownloaded(); + // Release any transcribe posts waiting on this load (D-025). + loadGate.current?.resolve(); } break; case "partial": @@ -235,6 +286,9 @@ export function App() { } else if (message.requestId === undefined) { // Load-stage failures and legacy id-less passes. Engine-routed // file passes carry an id and surface through the engine promise. + if (message.stage === "load") { + loadGate.current?.reject(new Error(message.message)); + } setError(message.message); setTranscribeProgress(null); setBusy(false); @@ -242,6 +296,42 @@ export function App() { } break; } + case "load-restart": { + // The accelerator attempt failed after it had already initialized + // onnxruntime's WASM runtime single-threaded (D-025). This worker + // can never run multithreaded WASM again, so: remember the outcome + // for this session, replace the worker, and reload with the + // explicit fallback device. The load gate stays pending — the fresh + // worker's "ready" releases the waiting transcribe posts, which + // then flow to the new worker. + deviceOverrides.current.set( + overrideKey(message.modelId, message.preference), + message.device, + ); + // Defensive: the gate means nothing should be in flight, but never + // leave a promise awaiting a worker we are about to terminate. + const restart = new Error( + "Inference restarted to use all CPU cores — this pass was skipped.", + ); + browserEngineRef.current?.failAll(restart); + for (const pendingPass of pendingLive.current.values()) { + pendingPass.reject(restart); + } + pendingLive.current.clear(); + workerRef.current?.terminate(); + workerRef.current = null; + browserEngineRef.current = null; + setLoadProgress(initialLoadProgress); + setActiveDevice(null); + setActiveThreads(null); + const fresh = ensureWorker(); + postToWorker(fresh, { + type: "load", + modelId: message.modelId, + device: message.device, + }); + break; + } } }); worker.addEventListener("error", (event) => { @@ -251,6 +341,7 @@ export function App() { // Unblock any live passes so a live session can drain and stop. for (const pending of pendingLive.current.values()) pending.reject(failure); pendingLive.current.clear(); + loadGate.current?.reject(failure); setError(failure.message); setBusy(false); }); @@ -275,7 +366,12 @@ export function App() { setLoadProgress(initialLoadProgress); setLoadStarted(true); setError(null); - postToWorker(worker, { type: "load", modelId: nextModelId, device: preference }); + newLoadGate(); + postToWorker(worker, { + type: "load", + modelId: nextModelId, + device: effectiveDevice(nextModelId, preference), + }); return worker; } @@ -283,6 +379,7 @@ export function App() { if (busy || liveStatus !== "idle" || nextModelId === modelId) return; setModelId(nextModelId); setActiveDevice(null); + setActiveThreads(null); setPartialText(""); setTranscribeProgress(null); setResult(null); @@ -292,15 +389,18 @@ export function App() { // reports fresh progress for this model. setLoadStarted(false); setLoadProgress(initialLoadProgress); + loadGate.current = null; } function handleDevicePreference(preference: DevicePreference) { if (busy || liveStatus !== "idle" || preference === devicePreference) return; setDevicePreference(preference); setActiveDevice(null); + setActiveThreads(null); // No eager reload (D-020) — the next load picks up the new preference. setLoadStarted(false); setLoadProgress(initialLoadProgress); + loadGate.current = null; } /** Explicit model download (D-020) — pre-caches weights for offline use. */ @@ -334,9 +434,13 @@ export function App() { workerRef.current = null; // The worker (and any pipeline / partial download) is gone — reset the // load state so the next transcription reloads from a clean slate. + // Rejecting the gate unblocks a transcribe post still waiting on it. + loadGate.current?.reject(cancelled); + loadGate.current = null; setLoadStarted(false); setLoadProgress(initialLoadProgress); setActiveDevice(null); + setActiveThreads(null); } // The file flow never has live passes pending, but if anything is in the // map (defensive), unblock it so nothing awaits a dead worker forever. @@ -368,13 +472,15 @@ export function App() { setResult(null); setFileName(file.name); - let engine: Engine; + let engine: Engine | null = null; if (useServer) { engine = serverEngineRef.current ??= new ServerEngine(); } else { // Kick off the (idempotent) model load immediately so the download runs // while we decode the audio locally. Server runs skip this — they don't - // need the browser pipeline. + // need the browser pipeline. The engine itself is resolved after the + // load gate below: a load-restart (D-025) may replace the worker (and + // its BrowserEngine) while the audio decodes. ensureWorker(); if (!loadStarted) { startLoad(modelId, devicePreference); @@ -382,10 +488,9 @@ export function App() { postToWorker(workerRef.current!, { type: "load", modelId, - device: devicePreference, + device: effectiveDevice(modelId, devicePreference), }); } - engine = browserEngineRef.current!; } const multilingual = getModel(modelId)?.multilingual ?? false; @@ -411,6 +516,24 @@ export function App() { } if (fileRun.current !== run) return; // cancelled while decoding (D-018) + if (engine === null) { + // Browser run: wait for the pipeline before posting the pass, so the + // audio never sits queued in a worker a load-restart may terminate + // (D-025). The worker serialized the pass on the load anyway, so this + // changes nothing about when inference actually starts. + try { + await browserModelReady(); + } catch (loadError) { + if (fileRun.current !== run) return; // cancelled while loading + setError(loadError instanceof Error ? loadError.message : String(loadError)); + setBusy(false); + setRunEngine(null); + return; + } + if (fileRun.current !== run) return; + engine = browserEngineRef.current!; + } + // Server runs cancel by aborting the fetch (D-018 amendment); browser // runs cancel by terminating the worker, which rejects the promise via // BrowserEngine.failAll. @@ -457,11 +580,17 @@ export function App() { // ------------------------------------------------------------------------- /** One inference pass over a live segment, correlated via requestId. */ - function transcribeLiveSegment( - worker: Worker, + async function transcribeLiveSegment( audio: Float32Array, options: { language?: string; task?: WhisperTask }, ): Promise { + // Wait for the pipeline before posting: a load-restart (D-025) may swap + // the worker while the model loads, and a pass queued in the old worker + // would die with it (its audio buffer is transferred, so it cannot be + // re-posted). The worker is re-read after the wait for the same reason. + await browserModelReady(); + const worker = workerRef.current; + if (!worker) throw new Error("The inference worker is not running."); const requestId = `live-${++nextLiveRequest.current}`; return new Promise((resolve, reject) => { pendingLive.current.set(requestId, { resolve, reject }); @@ -561,17 +690,21 @@ export function App() { } // Idempotent model load, like the file flow — a pending download runs - // while the mic opens; segment passes queue in the worker until ready. + // while the mic opens; segment passes wait on the load gate until ready. const worker = ensureWorker(); if (!loadStarted) { startLoad(modelId, devicePreference); } else { - postToWorker(worker, { type: "load", modelId, device: devicePreference }); + postToWorker(worker, { + type: "load", + modelId, + device: effectiveDevice(modelId, devicePreference), + }); } const transcriber = new LiveTranscriber({ ...LIVE_TUNING, - transcribe: (audio) => transcribeLiveSegment(worker, audio, options), + transcribe: (audio) => transcribeLiveSegment(audio, options), onUpdate: (update) => { setLiveCommitted(update.committedText); setLiveTentative(update.tentativeText); @@ -639,6 +772,7 @@ export function App() { if (!allowed.includes(modelId) && !busy && liveStatus === "idle") { setModelId("onnx-community/whisper-large-v3-turbo"); setActiveDevice(null); + setActiveThreads(null); setPartialText(""); setTranscribeProgress(null); setResult(null); @@ -713,6 +847,18 @@ export function App() { disabled={busy} /> + {/* Live on the WASM backend cannot keep up with real time for the big + models — every pass pads to a full 30 s Whisper window (D-024), so + tentative text lags and the stop drain grows. Say so instead of + looking frozen. */} + {liveActive && activeDevice === "wasm" && (model?.encoderParamsM ?? 0) >= 300 && ( +

+ {model?.displayName} runs slower than real time on the WASM backend, so + live text will lag behind speech. A smaller model (Tiny, Base or Small) + keeps up better on this device. +

+ )} + {fileName && (

File: {fileName} @@ -797,7 +943,11 @@ export function App() { : activeDevice === "webnn" ? "WebNN (on-device)" : activeDevice === "wasm" - ? "WASM (on-device)" + ? `WASM (on-device${ + activeThreads !== null + ? `, ${activeThreads} thread${activeThreads === 1 ? "" : "s"}` + : "" + })` : "on-device — backend selected on first load"}

diff --git a/src/worker/inference.worker.ts b/src/worker/inference.worker.ts index 9b95916..663ca49 100644 --- a/src/worker/inference.worker.ts +++ b/src/worker/inference.worker.ts @@ -18,6 +18,7 @@ import { import { resolveDefaultDtype } from "../lib/models"; import { deviceAttempts, + dtypeRequiresF16, isUiToWorkerMessage, TranscribeProgressTracker, type DeviceCapabilities, @@ -54,13 +55,27 @@ if (env.backends.onnx.wasm) { * therefore may run single-threaded — correct, just slower; the next worker * respawn (cancel does one, D-018) picks the full count up again. */ +function desiredWasmThreads(): number { + return globalThis.crossOriginIsolated + ? Math.max(1, Math.min((navigator.hardwareConcurrency || 4) - 1, 12)) + : 1; +} + function configureWasmThreads(device: InferenceDevice): void { const wasm = env.backends.onnx.wasm; if (!wasm) return; - wasm.numThreads = - device === "wasm" && globalThis.crossOriginIsolated - ? Math.max(1, Math.min((navigator.hardwareConcurrency || 4) - 1, 12)) - : 1; + wasm.numThreads = device === "wasm" ? desiredWasmThreads() : 1; +} + +/** + * Thrown to abort a load whose WASM fallback would be stuck single-threaded + * (D-025). The worker has already posted `load-restart`; the UI terminates + * this worker and retries in a fresh one, so nothing is reported as an error. + */ +class RestartRequested extends Error { + constructor() { + super("load aborted pending worker restart (D-025)"); + } } // `lib: ["DOM"]` types `self` as Window; give it the worker-scope shape we use. @@ -117,8 +132,22 @@ function forwardProgress(info: ProgressInfo): void { // fallback policy live in the pure `deviceAttempts` (protocol.ts, unit-tested). // --------------------------------------------------------------------------- +interface GpuAdapterLike { + features?: { has(name: string): boolean }; + /** Pre-2024 spec location of the software-adapter flag. */ + isFallbackAdapter?: boolean; + /** Current spec location (GPUAdapterInfo). */ + info?: { isFallbackAdapter?: boolean }; +} + +/** True for software adapters (SwiftShader, llvmpipe, WARP) — GPU-shaped but + * slower than multithreaded WASM, so never worth an attempt (D-025). */ +function isSoftwareAdapter(adapter: GpuAdapterLike): boolean { + return Boolean(adapter.isFallbackAdapter ?? adapter.info?.isFallbackAdapter); +} + interface NavigatorWithHardware { - gpu?: { requestAdapter(): Promise }; + gpu?: { requestAdapter(): Promise }; ml?: unknown; } @@ -134,9 +163,30 @@ async function probeCapabilities( // Probing threw — treat WebNN as unavailable. } } - if (preference === "auto" || preference === "webgpu") { + if (preference === "auto" && /Firefox\//.test(navigator.userAgent)) { + // Firefox creates ort-web WebGPU sessions fine but runs them far slower + // than this machine's multithreaded WASM (field-verified 2026-07-17, + // D-026) — and a *successful* slow session is invisible to the + // failure-driven fallback chain. Never auto-pick WebGPU on Firefox; an + // explicit selection is honored (the retest hatch for future ort/Firefox). + console.info( + "[whisper] Firefox: skipping WebGPU for \"auto\" — onnxruntime-web's WebGPU backend " + + "runs much slower than multithreaded WASM on Firefox (D-026). " + + "Select WebGPU explicitly to retest.", + ); + } else if (preference === "auto" || preference === "webgpu") { try { - capabilities.webgpu = Boolean(await hardware.gpu?.requestAdapter()); + const adapter = (await hardware.gpu?.requestAdapter()) ?? null; + if (adapter && isSoftwareAdapter(adapter)) { + // A fallback (software-rendered) adapter would "succeed" and then run + // far slower than multithreaded WASM — never attempt it (D-025). + console.info("[whisper] WebGPU adapter is software-rendered — skipping WebGPU."); + } else { + capabilities.webgpu = adapter !== null; + // fp16 weights need the "shader-f16" feature (D-025); Firefox ships + // WebGPU without it, so createPipeline gates on this. + capabilities.webgpuF16 = Boolean(adapter?.features?.has("shader-f16")); + } } catch { // Probing threw — treat WebGPU as unavailable. } @@ -167,17 +217,42 @@ function loadKey(message: LoadMessage): string { async function createPipeline( message: LoadMessage, ): Promise { + const capabilities = await probeCapabilities(message.device); + // WebGPU without "shader-f16" (Firefox) cannot run fp16 weights — skip the + // doomed attempt instead of failing into a degraded-thread fallback (D-025). + if (capabilities.webgpu && !capabilities.webgpuF16) { + const webgpuDtype = + message.dtype ?? resolveDefaultDtype(message.modelId, "webgpu"); + if (dtypeRequiresF16(webgpuDtype)) { + capabilities.webgpu = false; + console.info( + "[whisper] WebGPU adapter lacks shader-f16 but the model dtype needs fp16 — skipping WebGPU.", + ); + } + } // Ordered devices to try (D-004/D-017): the first is the selected device; // if pipeline creation throws there (driver quirks, unsupported ops, // out-of-memory), the next one is tried automatically. For "auto" the chain // is WebNN → WebGPU → WASM (filtered by the probe); WASM always terminates it. - const attempts: InferenceDevice[] = deviceAttempts( - message.device, - await probeCapabilities(message.device), - ); + const attempts: InferenceDevice[] = deviceAttempts(message.device, capabilities); let lastError: unknown; + let acceleratorAttempted = false; for (const device of attempts) { + // A failed WebGPU/WebNN attempt has already initialized onnxruntime's + // WASM runtime single-threaded, and the thread pool is fixed at first + // init (D-023) — a WASM session created now would run on one thread for + // this worker's whole life (measured 5.5× slower). When multithreading is + // actually available, hand the fallback to a fresh worker instead (D-025). + if (device === "wasm" && acceleratorAttempted && desiredWasmThreads() > 1) { + post({ + type: "load-restart", + modelId: message.modelId, + preference: message.device, + device: "wasm", + }); + throw new RestartRequested(); + } configureWasmThreads(device); const dtype = message.dtype ?? resolveDefaultDtype(message.modelId, device); try { @@ -194,10 +269,18 @@ async function createPipeline( }), progress_callback: forwardProgress, }); - post({ type: "load-progress", status: "ready", device }); + // What ort actually ended up with (it clamps to 1 without isolation) — + // surfaced to the UI so degraded threading is visible in the field. + const threads = env.backends.onnx.wasm?.numThreads ?? 1; + console.info(`[whisper] pipeline ready: device=${device} wasmThreads=${threads}`); + post({ type: "load-progress", status: "ready", device, threads }); return asr; } catch (error) { lastError = error; + if (device !== "wasm") acceleratorAttempted = true; + console.warn( + `[whisper] pipeline creation failed on ${device}: ${errorMessage(error)}`, + ); } } throw lastError; @@ -350,11 +433,15 @@ scope.addEventListener("message", (event) => { switch (data.type) { case "load": load(data).catch((error: unknown) => { + // A restart-aborted load is not an error — the UI already got the + // load-restart message and is respawning this worker (D-025). + if (error instanceof RestartRequested) return; post({ type: "error", stage: "load", message: errorMessage(error) }); }); break; case "transcribe": transcribe(data).catch((error: unknown) => { + if (error instanceof RestartRequested) return; post({ type: "error", stage: "transcribe", diff --git a/src/worker/protocol.ts b/src/worker/protocol.ts index 61864f4..485f8c2 100644 --- a/src/worker/protocol.ts +++ b/src/worker/protocol.ts @@ -106,7 +106,18 @@ export type LoadProgressMessage = progress: number; } | { type: "load-progress"; status: "done"; file: string } - | { type: "load-progress"; status: "ready"; device: InferenceDevice }; + | { + type: "load-progress"; + status: "ready"; + device: InferenceDevice; + /** + * WASM thread count onnxruntime actually ended up with (D-025) — ort + * clamps to 1 when cross-origin isolation is missing, so this is field + * observability for "is multithreading really on?". Accelerator devices + * report 1 (their WASM side must be single-threaded, D-023). + */ + threads?: number; + }; /** Incremental transcript fragment — append to the text streamed so far. */ export interface PartialMessage { @@ -155,12 +166,32 @@ export interface ErrorMessage { requestId?: string; } +/** + * The worker cannot finish this load usefully in-place and asks the UI to + * respawn it (D-025). Posted when an accelerator attempt already initialized + * onnxruntime's WASM runtime — whose thread pool is fixed at first init — + * single-threaded, so the WASM fallback in *this* worker would run on one + * thread for the worker's whole life. The UI terminates the worker, spawns a + * fresh one, and re-issues the load with the explicit `device`, remembering + * `modelId`+`preference` so later loads skip the doomed attempt entirely. + */ +export interface LoadRestartMessage { + type: "load-restart"; + /** Echo of the aborted load's model id. */ + modelId: string; + /** Echo of the aborted load's device preference. */ + preference: DevicePreference; + /** Explicit device the fresh worker should load with. */ + device: InferenceDevice; +} + export type WorkerToUiMessage = | LoadProgressMessage | PartialMessage | TranscribeProgressMessage | ResultMessage - | ErrorMessage; + | ErrorMessage + | LoadRestartMessage; // --------------------------------------------------------------------------- // Device selection policy (D-004, D-017) — pure so it is unit-testable; the @@ -173,6 +204,23 @@ export interface DeviceCapabilities { webnn: boolean; /** `navigator.gpu.requestAdapter()` returned an adapter (WebGPU). */ webgpu: boolean; + /** + * The WebGPU adapter supports the "shader-f16" feature. Firefox ships + * WebGPU without it, so fp16 weights cannot run there (D-025); the worker + * skips the WebGPU attempt when the resolved dtype needs fp16 and this is + * false. Optional so `deviceAttempts` callers that don't probe it (tests) + * stay valid. + */ + webgpuF16?: boolean; +} + +/** + * Whether a dtype spec involves fp16 weights anywhere — such a model cannot + * run on a WebGPU adapter without the "shader-f16" feature (D-025). + */ +export function dtypeRequiresF16(dtype: DtypeSpec): boolean { + if (typeof dtype === "string") return dtype === "fp16"; + return dtype.encoder_model === "fp16" || dtype.decoder_model_merged === "fp16"; } /** @@ -284,7 +332,10 @@ export function isWorkerToUiMessage(value: unknown): value is WorkerToUiMessage typeof value.progress === "number" ); case "ready": - return isInferenceDevice(value.device); + return ( + isInferenceDevice(value.device) && + (value.threads === undefined || typeof value.threads === "number") + ); default: return false; } @@ -308,6 +359,12 @@ export function isWorkerToUiMessage(value: unknown): value is WorkerToUiMessage (value.stage === "load" || value.stage === "transcribe") && isOptionalString(value.requestId) ); + case "load-restart": + return ( + typeof value.modelId === "string" && + isDevicePreference(value.preference) && + isInferenceDevice(value.device) + ); default: return false; } diff --git a/tests/protocol.test.ts b/tests/protocol.test.ts index 83d9d96..43bf74c 100644 --- a/tests/protocol.test.ts +++ b/tests/protocol.test.ts @@ -2,6 +2,7 @@ import { describe, test, expect } from "bun:test"; import { applyLoadProgress, deviceAttempts, + dtypeRequiresF16, initialLoadProgress, isUiToWorkerMessage, isWorkerToUiMessage, @@ -83,7 +84,9 @@ describe("isWorkerToUiMessage", () => { }, { type: "load-progress", status: "done", file: "model.onnx" }, { type: "load-progress", status: "ready", device: "wasm" }, + { type: "load-progress", status: "ready", device: "wasm", threads: 11 }, { type: "load-progress", status: "ready", device: "webgpu" }, + { type: "load-progress", status: "ready", device: "webgpu", threads: 1 }, { type: "partial", text: " hello" }, { type: "partial", text: " hello", requestId: "live-1" }, { type: "transcribe-progress", processedSeconds: 0, totalSeconds: 90 }, @@ -92,6 +95,18 @@ describe("isWorkerToUiMessage", () => { { type: "result", text: "hello world", chunks: [], requestId: "live-1" }, { type: "error", stage: "load", message: "boom" }, { type: "error", stage: "transcribe", message: "boom", requestId: "live-2" }, + { + type: "load-restart", + modelId: "onnx-community/whisper-medium-ONNX", + preference: "auto", + device: "wasm", + }, + { + type: "load-restart", + modelId: "onnx-community/whisper-tiny.en", + preference: "webgpu", + device: "wasm", + }, ]; for (const message of messages) { expect(isWorkerToUiMessage(message)).toBe(true); @@ -104,6 +119,9 @@ describe("isWorkerToUiMessage", () => { expect(isWorkerToUiMessage({ type: "load-progress", status: "progress", file: "f" })).toBe(false); expect(isWorkerToUiMessage({ type: "load-progress", status: "ready" })).toBe(false); expect(isWorkerToUiMessage({ type: "load-progress", status: "ready", device: "cuda" })).toBe(false); + expect( + isWorkerToUiMessage({ type: "load-progress", status: "ready", device: "wasm", threads: "12" }), + ).toBe(false); expect(isWorkerToUiMessage({ type: "partial" })).toBe(false); expect(isWorkerToUiMessage({ type: "partial", text: "x", requestId: 1 })).toBe(false); expect(isWorkerToUiMessage({ type: "transcribe-progress", processedSeconds: 1 })).toBe(false); @@ -121,6 +139,37 @@ describe("isWorkerToUiMessage", () => { expect(isWorkerToUiMessage({ type: "result", text: "x", chunks: "nope" })).toBe(false); expect(isWorkerToUiMessage({ type: "error", stage: "decode", message: "m" })).toBe(false); expect(isWorkerToUiMessage({ type: "load", modelId: "x", device: "wasm" })).toBe(false); + expect(isWorkerToUiMessage({ type: "load-restart", modelId: "x" })).toBe(false); + expect( + isWorkerToUiMessage({ type: "load-restart", modelId: "x", preference: "auto", device: "auto" }), + ).toBe(false); + expect( + isWorkerToUiMessage({ type: "load-restart", preference: "auto", device: "wasm" }), + ).toBe(false); + }); +}); + +describe("dtypeRequiresF16 (D-025)", () => { + test("plain dtypes: only fp16 needs the shader-f16 feature", () => { + expect(dtypeRequiresF16("fp16")).toBe(true); + expect(dtypeRequiresF16("fp32")).toBe(false); + expect(dtypeRequiresF16("q8")).toBe(false); + expect(dtypeRequiresF16("q4")).toBe(false); + }); + + test("per-part dtypes: fp16 in either part needs it", () => { + expect( + dtypeRequiresF16({ encoder_model: "fp16", decoder_model_merged: "q4" }), + ).toBe(true); + expect( + dtypeRequiresF16({ encoder_model: "fp32", decoder_model_merged: "fp16" }), + ).toBe(true); + expect( + dtypeRequiresF16({ encoder_model: "fp32", decoder_model_merged: "fp32" }), + ).toBe(false); + expect( + dtypeRequiresF16({ encoder_model: "fp32", decoder_model_merged: "q4" }), + ).toBe(false); }); });