diff --git a/docs/Decisions.md b/docs/Decisions.md index a78c6e1..aeca5a1 100644 --- a/docs/Decisions.md +++ b/docs/Decisions.md @@ -617,3 +617,88 @@ warning thresholds). mobile crashes in practice, consider loosening the confirm thresholds; if onnxruntime-web ever gains a streaming/incremental session-creation API, drop the patch and re-measure before touching the thresholds again. +**Amendment (2026-07-29, D-028):** "a little wall-clock time" undersold the +cost — see D-028. The sequential patch no longer applies unconditionally; +`options.sequentialSessionLoad` (set by `App.tsx`, true only when +`shouldWarnBeforeLoad` is true for the load) now gates it. + +## D-028 — Sequential session loading gated to actual mobile-crash risk, not unconditional +**Status:** Accepted (2026-07-29) +Reported symptom: `encoder_model_q4.onnx` (Whisper Medium) downloaded far +slower after D-027 landed than before it — reproduced on desktop too, ruling +out phone CPU/network as the cause. Measured in Chrome DevTools: ~1.6 MB/s +sustained on a connection that otherwise performs fine, downloading from +Hugging Face's Xet-backed CDN (`us.aws.cdn.hf.co/xet-bridge-us/...`). +**Cause.** D-027's `constructSessions` patch made the encoder and decoder +downloads always sequential — one HTTP request in flight at a time — to keep +both weight buffers from being resident together. On a long-RTT path (this +CDN is US-hosted), a single HTTP/2 stream typically can't fill the available +bandwidth; two concurrent requests (the pre-D-027 `Promise.all`) filled more +of it. Serializing the downloads dropped every load back to one flow's +throughput ceiling — not "a little" wall-clock cost as D-027 claimed, but +multiple times slower on exactly this kind of connection, and paid by every +user regardless of device. +**Fix.** The sequential/parallel choice is now conditional, not baked into +the patch: `constructSessions` branches on a new `options.sequentialSessionLoad` +flag (`patches/@huggingface%2Ftransformers@4.2.0.patch`) — parallel +`Promise.all` (the original, fast behavior) when falsy/absent, the D-027 +for-loop when true. `App.tsx` computes this per load via `isMobileLoadRisk` +(extracted from `confirmRiskyMobileLoad`'s existing hint/`estimatePeakLoadBytes` +logic — the same `shouldWarnBeforeLoad` check the crash-warning dialog uses) +and threads it through a new `sequentialSessionLoad` field on the worker's +`LoadMessage` (`protocol.ts`) into the `pipeline()` call +(`inference.worker.ts`). Crucially this is evaluated independently of the +confirm dialog's answer — even a load the user clicked "Continue anyway" on +still loads sequentially, since the memory risk that motivated D-027 doesn't +go away just because the user accepted it. +**Consequence:** desktop/laptop loads (and any mobile load under the +size/`deviceMemory` risk threshold) get back the full concurrent-download +speed that predates D-027; only loads that would already trigger the mobile +crash warning pay the sequential-loading wall-clock cost, which is exactly +where D-027's OOM fix is actually needed. +**Revisit:** if Hugging Face's Xet CDN behavior changes, or if a future model +in the catalog is large enough to risk mobile OOM at a size where the +confirm-dialog threshold doesn't yet trigger, re-check that the two +thresholds (`shouldWarnBeforeLoad`'s and this gate's) are meant to stay +identical. + +## D-029 — Mobile crash warning was silently disabled on every phone but the lowest-end +**Status:** Accepted (2026-07-29) +Reported symptom: the D-027 mobile crash `window.confirm()` never appeared +for Whisper Medium (or any model), even on devices it's explicitly meant to +protect. +**Cause.** `shouldWarnBeforeLoad`'s `deviceMemory`-scaled branch used +`SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.5`. Checked against the actual +catalog (`estimatePeakLoadBytes`) for every `navigator.deviceMemory` value +Chromium can report: + +| deviceMemory | "safe" budget (old 0.5) | Medium (556 MB) | Large v3 Turbo (762 MB) | +|---|---|---|---| +| 1 | 500 MB | warns | warns | +| 2 | 1 GB | **silent** | **silent** | +| 4 | 2 GB | **silent** | **silent** | +| 8 | 4 GB | **silent** | **silent** | + +`deviceMemory` of 2 or more is ordinary today, not just flagship-tier — so +in practice the warning only ever fired on the oldest/cheapest Android +phones and on iOS (which lacks the API entirely and always used the flat +400 MB fallback, D-027). The exact mid/low-range phones the original crash +report was about — the whole reason D-027 exists — cleared the threshold and +got no warning at all. +**Fix:** `SAFE_TAB_FRACTION_OF_DEVICE_MEMORY` lowered to 0.1 +(`src/lib/mobileMemoryGuard.ts`). Re-checking the same table: deviceMemory +2 and 4 now warn for Medium+ (200 MB / 400 MB budgets, both under 556 MB); +deviceMemory 6-8 (600-800 MB budgets) still don't, treating genuine +high-end devices as having real headroom. `tests/mobileMemoryGuard.test.ts` +gained a regression case pinning deviceMemory 2/4 to `true` for Medium's +peak — the exact gap this decision closes. +**Alternatives rejected:** dropping the `deviceMemory` scaling entirely in +favor of always using the flat 400 MB fallback (simpler, but throws away the +one signal that lets genuinely high-RAM devices skip an unnecessary +confirm); a hard absolute cap on top of the fraction (more robust long-term, +but two knobs to calibrate against no real data beats one — revisit if a +future model's peak lands awkwardly relative to the 0.1 fraction). +**Revisit:** this is still a guess, same as D-027's original 0.5 was — there +is no real crash-telemetry feedback loop. If field reports show 0.1 is still +too generous (or too aggressive) for some device class, adjust the constant +rather than the shape of the heuristic. diff --git a/patches/@huggingface%2Ftransformers@4.2.0.patch b/patches/@huggingface%2Ftransformers@4.2.0.patch index f7d8a20..1d1293a 100644 --- a/patches/@huggingface%2Ftransformers@4.2.0.patch +++ b/patches/@huggingface%2Ftransformers@4.2.0.patch @@ -1,8 +1,11 @@ +diff --git a/node_modules/@huggingface/transformers/.bun-tag-8719566ebeed846f b/.bun-tag-8719566ebeed846f +new file mode 100644 +index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/dist/transformers.web.js b/dist/transformers.web.js -index bb7b32717a42475a5f66411a33d5e5964dfda3e2..799cb402e98d8d0d58bf59629c4c2992407477e0 100644 +index bb7b32717a42475a5f66411a33d5e5964dfda3e2..03de63967e99f8f3c847858621d72577ef037391 100644 --- a/dist/transformers.web.js +++ b/dist/transformers.web.js -@@ -18558,22 +18558,28 @@ async function getSession(pretrained_model_name_or_path, fileName, options, cach +@@ -18558,22 +18558,49 @@ async function getSession(pretrained_model_name_or_path, fileName, options, cach return { buffer_or_path, session_options, session_config }; } async function constructSessions(pretrained_model_name_or_path, names, options, cache_sessions = void 0) { @@ -22,14 +25,35 @@ index bb7b32717a42475a5f66411a33d5e5964dfda3e2..799cb402e98d8d0d58bf59629c4c2992 - }) - ) - ); -+ // Patched (au_whisper_web): load sub-model sessions one at a time instead of -+ // Promise.all-ing them. Each entry briefly holds a full-size decoded model -+ // buffer in JS-heap memory (on top of onnxruntime-web's own WASM-heap copy -+ // during session creation); loading Whisper's encoder and decoder ++ // Patched (au_whisper_web): each entry briefly holds a full-size decoded ++ // model buffer in JS-heap memory (on top of onnxruntime-web's own WASM-heap ++ // copy during session creation); loading Whisper's encoder and decoder + // concurrently keeps both buffers alive at once and roughly doubles the + // load-time memory peak, which is enough to OOM-crash mobile browsers on -+ // larger models (e.g. medium, ~460 MB combined). Sequential loading lets the -+ // previous entry's buffer become garbage before the next one is fetched. ++ // larger models (e.g. medium, ~460 MB combined). `options.sequentialSessionLoad` ++ // (set by the caller only when that risk is real, D-027/D-029) loads ++ // sub-model sessions one at a time instead of Promise.all-ing them, so the ++ // previous entry's buffer becomes garbage before the next one is fetched — ++ // at the cost of losing the download concurrency a fast connection could ++ // otherwise use, which is the default (and always was, pre-D-027). ++ if (!options?.sequentialSessionLoad) { ++ return Object.fromEntries( ++ await Promise.all( ++ Object.keys(names).map(async (name) => { ++ const cache_config = cache_sessions?.[name] ?? false; ++ const { buffer_or_path, session_options, session_config } = await getSession( ++ pretrained_model_name_or_path, ++ names[name], ++ options, ++ cache_config, ++ name ++ ); ++ const session = await createInferenceSession(buffer_or_path, session_options, session_config); ++ return [name, session]; ++ }) ++ ) ++ ); ++ } + const entries = []; + for (const name of Object.keys(names)) { + const cache_config = cache_sessions?.[name] ?? false; diff --git a/src/App.tsx b/src/App.tsx index f0f8182..7e8bd8c 100644 --- a/src/App.tsx +++ b/src/App.tsx @@ -213,6 +213,38 @@ export function App() { return deviceOverrides.current.get(overrideKey(forModelId, preference)) ?? preference; } + /** Mirrors ModelPicker's size-estimate fallback: until the worker reports an + * active device, assume WASM unless the user asked for an accelerator. */ + function guessDevice(preference: DevicePreference): InferenceDevice { + return ( + activeDevice ?? + (preference === "webgpu" ? "webgpu" : preference === "webnn" ? "webnn" : "wasm") + ); + } + + /** + * Whether loading `forModelId` on `device` risks crashing a mobile tab + * (D-027) — independent of whether the user has already been asked. Used + * both to decide whether to show the confirm dialog and, regardless of the + * user's answer, to decide whether the worker should load sub-model + * sessions sequentially (memory-safe but slower) or in parallel (fast, + * safe everywhere this isn't true). + */ + function isMobileLoadRisk(forModelId: string, device: InferenceDevice): boolean { + const targetModel = getModel(forModelId); + if (!targetModel) return false; + const peakBytes = estimatePeakLoadBytes(targetModel, resolveDefaultDtype(forModelId, device)); + const hint: MobileMemoryHint = { + userAgent: typeof navigator !== "undefined" ? navigator.userAgent : undefined, + maxTouchPoints: typeof navigator !== "undefined" ? navigator.maxTouchPoints : undefined, + deviceMemoryGB: + typeof navigator !== "undefined" + ? (navigator as Navigator & { deviceMemory?: number }).deviceMemory + : undefined, + }; + return shouldWarnBeforeLoad(hint, peakBytes); + } + /** * Confirm before a browser-engine load that risks crashing the tab on * mobile (D-027). Returns true if the load should proceed — either it @@ -223,25 +255,12 @@ export function App() { function confirmRiskyMobileLoad(forModelId: string, preference: DevicePreference): boolean { const targetModel = getModel(forModelId); if (!targetModel) return true; - // Mirrors ModelPicker's size-estimate fallback: until the worker reports - // an active device, assume WASM unless the user asked for an accelerator. - const device: InferenceDevice = - activeDevice ?? - (preference === "webgpu" ? "webgpu" : preference === "webnn" ? "webnn" : "wasm"); + const device = guessDevice(preference); const key = overrideKey(forModelId, device); if (mobileWarningAcknowledged.current.has(key)) return true; + if (!isMobileLoadRisk(forModelId, device)) return true; const peakBytes = estimatePeakLoadBytes(targetModel, resolveDefaultDtype(forModelId, device)); - const hint: MobileMemoryHint = { - userAgent: typeof navigator !== "undefined" ? navigator.userAgent : undefined, - maxTouchPoints: typeof navigator !== "undefined" ? navigator.maxTouchPoints : undefined, - deviceMemoryGB: - typeof navigator !== "undefined" - ? (navigator as Navigator & { deviceMemory?: number }).deviceMemory - : undefined, - }; - if (!shouldWarnBeforeLoad(hint, peakBytes)) return true; - const proceed = window.confirm( `${targetModel.displayName} needs roughly ${formatModelSize(peakBytes)} of memory at ` + "once to load — more than many phones and tablets allow a browser tab to use " + @@ -379,6 +398,7 @@ export function App() { type: "load", modelId: message.modelId, device: message.device, + sequentialSessionLoad: isMobileLoadRisk(message.modelId, message.device), }); break; } @@ -417,10 +437,16 @@ export function App() { setLoadStarted(true); setError(null); newLoadGate(); + const device = effectiveDevice(nextModelId, preference); postToWorker(worker, { type: "load", modelId: nextModelId, - device: effectiveDevice(nextModelId, preference), + device, + // D-027/D-029: sub-model sessions load sequentially (memory-safe, but + // slower — halves the CDN concurrency a fast connection could otherwise + // use) only where the peak-memory crash risk is real; everywhere else + // (desktop, low-risk models) the worker loads them in parallel. + sequentialSessionLoad: isMobileLoadRisk(nextModelId, guessDevice(device)), }); return worker; } @@ -540,10 +566,12 @@ export function App() { if (!loadStarted) { startLoad(modelId, devicePreference); } else { + const device = effectiveDevice(modelId, devicePreference); postToWorker(workerRef.current!, { type: "load", modelId, - device: effectiveDevice(modelId, devicePreference), + device, + sequentialSessionLoad: isMobileLoadRisk(modelId, guessDevice(device)), }); } } @@ -788,10 +816,12 @@ export function App() { if (!loadStarted) { startLoad(modelId, devicePreference); } else { + const device = effectiveDevice(modelId, devicePreference); postToWorker(worker, { type: "load", modelId, - device: effectiveDevice(modelId, devicePreference), + device, + sequentialSessionLoad: isMobileLoadRisk(modelId, guessDevice(device)), }); } diff --git a/src/lib/mobileMemoryGuard.ts b/src/lib/mobileMemoryGuard.ts index b8b2549..10095c2 100644 --- a/src/lib/mobileMemoryGuard.ts +++ b/src/lib/mobileMemoryGuard.ts @@ -52,11 +52,18 @@ export function isLikelyMobile( /** * Fraction of reported device RAM a single tab can safely peak at before - * mobile OSes intervene. Conservative on purpose — `deviceMemory` already - * caps at 8 regardless of actual RAM, and the tab competes with the rest of - * the OS, the browser process itself, and other open tabs. + * mobile OSes intervene. + * + * D-029: the original 0.5 was not conservative enough — `deviceMemory` is + * commonly 2-4 on ordinary (not just flagship) Android phones today, and at + * 0.5 that already yields a 1-2 GB "safe" budget, clearing every model in + * the catalog including Medium (~556 MB peak) and Large v3 Turbo (~762 MB) — + * silently skipping the warning for exactly the phones D-027's crash report + * was about. 0.1 keeps mid/low-end phones (deviceMemory ≤ 4) warned for + * Medium+ while still exempting genuine high-end devices (deviceMemory ≥ 6, + * 600+ MB budget) that have real headroom to spare. */ -const SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.5; +const SAFE_TAB_FRACTION_OF_DEVICE_MEMORY = 0.1; /** * Fallback risk threshold when `deviceMemory` isn't reported (notably all of diff --git a/src/worker/inference.worker.ts b/src/worker/inference.worker.ts index 663ca49..2d5b4b2 100644 --- a/src/worker/inference.worker.ts +++ b/src/worker/inference.worker.ts @@ -13,6 +13,7 @@ import { pipeline, WhisperTextStreamer, type AutomaticSpeechRecognitionPipeline, + type PretrainedModelOptions, type ProgressInfo, } from "@huggingface/transformers"; import { resolveDefaultDtype } from "../lib/models"; @@ -256,7 +257,7 @@ async function createPipeline( configureWasmThreads(device); const dtype = message.dtype ?? resolveDefaultDtype(message.modelId, device); try { - const asr = await pipeline("automatic-speech-recognition", message.modelId, { + const pipelineOptions: PretrainedModelOptions & { sequentialSessionLoad?: boolean } = { device, dtype, // WASM sessions cap graph optimization at "basic" (D-014): the level-2 @@ -268,7 +269,12 @@ async function createPipeline( session_options: { graphOptimizationLevel: "basic" as const }, }), progress_callback: forwardProgress, - }); + // D-027/D-029: sequential sub-model session loading trades load speed + // for a lower peak-memory footprint — only worth it where the UI + // determined this load risks crashing a mobile tab. + sequentialSessionLoad: message.sequentialSessionLoad ?? false, + }; + const asr = await pipeline("automatic-speech-recognition", message.modelId, pipelineOptions); // What ort actually ended up with (it clamps to 1 without isolation) — // surfaced to the UI so degraded threading is visible in the field. const threads = env.backends.onnx.wasm?.numThreads ?? 1; diff --git a/src/worker/protocol.ts b/src/worker/protocol.ts index 485f8c2..688333a 100644 --- a/src/worker/protocol.ts +++ b/src/worker/protocol.ts @@ -47,6 +47,16 @@ export interface LoadMessage { * default for the device it actually selected (src/lib/models.ts). */ dtype?: DtypeSpec; + /** + * Load the encoder/decoder ONNX sessions one at a time instead of + * concurrently (D-027/D-029). Sequential halves the peak JS-heap/WASM + * memory a load needs (avoiding mobile tab crashes on larger models) at + * the cost of losing the CDN-concurrency benefit a fast connection would + * otherwise get from two simultaneous downloads — worth paying only where + * the memory-crash risk is real. The UI computes this (`isMobileLoadRisk` + * in App.tsx); omitted/false loads in parallel. + */ + sequentialSessionLoad?: boolean; } export interface TranscribeMessage { diff --git a/tests/mobileMemoryGuard.test.ts b/tests/mobileMemoryGuard.test.ts index 4815ed9..2772467 100644 --- a/tests/mobileMemoryGuard.test.ts +++ b/tests/mobileMemoryGuard.test.ts @@ -70,6 +70,20 @@ describe("shouldWarnBeforeLoad", () => { ).toBe(false); }); + // D-029: deviceMemoryGB 2-4 is the ordinary (not just budget) Android + // range today — these must still warn for Medium+, the exact class of + // phone the original D-027 crash report was about. + test("warns on mobile mid-range devices (deviceMemory 2-4) for Medium+", () => { + for (const deviceMemoryGB of [2, 4]) { + expect( + shouldWarnBeforeLoad( + { userAgent: ANDROID_UA, maxTouchPoints: 5, deviceMemoryGB }, + MEDIUM_PEAK_BYTES, + ), + ).toBe(true); + } + }); + test("falls back to a flat threshold when deviceMemory is unreported (iOS)", () => { expect( shouldWarnBeforeLoad(