Usability and test infrastructure updates.
This commit is contained in:
+27
-15
@@ -84,24 +84,26 @@ src/
|
|||||||
auth.ts # Passport.js wiring + session middleware adapter
|
auth.ts # Passport.js wiring + session middleware adapter
|
||||||
db.ts # bun:sqlite schema + queries
|
db.ts # bun:sqlite schema + queries
|
||||||
transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5)
|
transcribe.ts # server-side ASR endpoint (bridges the sidecar, M5)
|
||||||
|
live.ts # server-engine live mic over WebSocket (D-019, M7)
|
||||||
server-py/ # faster-whisper sidecar (Python, D-013; M5)
|
server-py/ # faster-whisper sidecar (Python, D-013; M5)
|
||||||
tests/ # bun test
|
tests/ # bun test
|
||||||
```
|
```
|
||||||
|
|
||||||
(Current state: M0–M3 built — browser transcription with the curated model
|
(Current state: M0–M6 built — browser transcription with the curated model
|
||||||
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
|
catalog, WebNN→WebGPU→WASM auto-selection (D-017), model switching,
|
||||||
language/task options,
|
language/task options, the installable offline PWA (`pwa/`,
|
||||||
and the installable offline PWA (`pwa/`, `scripts/build.ts` production build,
|
`scripts/build.ts` production build, `dist/` serving in `src/index.ts`;
|
||||||
`dist/` serving in `src/index.ts`; browser-level offline behavior pending the
|
browser-level offline behavior pending the manual run in `tests/offline.md`),
|
||||||
manual run in `tests/offline.md`). M4 accounts are built (`server/db.ts`,
|
M4 accounts (`server/db.ts`, `server/auth.ts`, `/api/auth/*`, `AuthForms`),
|
||||||
`server/auth.ts`, `/api/auth/*`, `AuthForms`). M7 live microphone
|
and the M5 server engine end to end — `server-py/sidecar.py`,
|
||||||
transcription is built (`lib/live.ts`, `lib/mic.ts`, `LiveControls`, D-016;
|
`server/transcribe.ts` (supervisor + `/api/transcribe` + `/status`),
|
||||||
in-browser run pending per `tests/live.md`). M5's server side is built —
|
`lib/engine.ts` (both engines, D-010), `EngineToggle`, and the catalog
|
||||||
`server-py/sidecar.py`, `server/transcribe.ts` (supervisor +
|
mapping in `lib/models.ts` — with graceful degradation when
|
||||||
`/api/transcribe` + `/status`), `lib/engine.ts` (both engines, D-010), and
|
Python/faster-whisper is absent. M7 live microphone transcription is built
|
||||||
the catalog mapping in `lib/models.ts` — with graceful degradation when
|
for both engines: browser (`lib/live.ts`, `lib/mic.ts`, `LiveControls`,
|
||||||
Python/faster-whisper is absent; the `EngineToggle` component and App wiring
|
D-016) and server (`server/live.ts`, D-019); in-browser manual verification
|
||||||
are the remaining M5 work; see `WorkBreakdown.md`.)
|
of both is still pending per `tests/live.md` (no real browser in the dev
|
||||||
|
environment). See `WorkBreakdown.md` for the itemized checklist.)
|
||||||
|
|
||||||
## 4. Key flows
|
## 4. Key flows
|
||||||
|
|
||||||
@@ -160,8 +162,18 @@ are the remaining M5 work; see `WorkBreakdown.md`.)
|
|||||||
4. Stopping flushes the segmenter, drains pending final passes (merged, with
|
4. Stopping flushes the segmenter, drains pending final passes (merged, with
|
||||||
a "N remaining segments" countdown in the UI), and yields a normal
|
a "N remaining segments" countdown in the UI), and yields a normal
|
||||||
transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
transcript (text + absolute-timestamped chunks) in `TranscriptView`.
|
||||||
Everything runs on-device; works offline like the file flow. Server-engine
|
Everything runs on-device; works offline like the file flow.
|
||||||
live input is a recorded future want (WorkBreakdown M7, D-016).
|
5. **Server-engine live (logged in, D-019).** Same `LiveControls` UI, but
|
||||||
|
`App.tsx` opens a `GET /api/transcribe/live` WebSocket instead of running
|
||||||
|
the segmenter locally: the browser only captures/resamples mic audio
|
||||||
|
(`lib/mic.ts`) and streams raw PCM frames over the socket; the *same* pure
|
||||||
|
`LiveTranscriber`/segmenter (`lib/live.ts`) runs server-side
|
||||||
|
(`server/live.ts`), writing each segment to a temp file and bridging it to
|
||||||
|
the faster-whisper sidecar (D-013) the same way `POST /api/transcribe`
|
||||||
|
does. `{type:"update"}` frames carry committed/tentative text back;
|
||||||
|
`{type:"stop"}` drains and closes with a `{type:"result"}` frame. Selected
|
||||||
|
only when signed in with the server engine available (`EngineToggle`);
|
||||||
|
anonymous/offline users always get the browser-engine path above.
|
||||||
|
|
||||||
### 4.2 Offline (PWA)
|
### 4.2 Offline (PWA)
|
||||||
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
|
- Service worker (`pwa/sw.ts`, hand-rolled — D-007) precaches the built app shell
|
||||||
|
|||||||
@@ -836,3 +836,72 @@ baked-in models, either accept runtime downloads for the extra ones (point
|
|||||||
`FASTER_WHISPER_DOWNLOAD_ROOT` at `/app/data` for those, per the README's
|
`FASTER_WHISPER_DOWNLOAD_ROOT` at `/app/data` for those, per the README's
|
||||||
override note) or extend the Dockerfile's pre-download `RUN` line —
|
override note) or extend the Dockerfile's pre-download `RUN` line —
|
||||||
re-measure and re-document the image size either way.
|
re-measure and re-document the image size either way.
|
||||||
|
|
||||||
|
## D-032 — Server-live WebSocket brought in line with the HTTP engine's DI/validation/test conventions
|
||||||
|
**Status:** Accepted (2026-07-29)
|
||||||
|
D-019's server-engine live transcription (`GET /api/transcribe/live` →
|
||||||
|
`src/server/live.ts`) had landed and was wired end-to-end in `App.tsx`, but
|
||||||
|
three things about it lagged the standard the rest of M5 set with
|
||||||
|
`src/server/transcribe.ts`:
|
||||||
|
1. **Not injectable.** `live.ts` imported the real `sidecar` singleton
|
||||||
|
directly and called it from inside the `LiveTranscriber`'s `transcribe`
|
||||||
|
closure, unlike `createTranscribeHandlers(deps)`, whose whole point (per
|
||||||
|
its own comment) is that `Sidecar` is "injectable — tests use a fake".
|
||||||
|
There was no way to unit-test the WebSocket bridge without a real Python
|
||||||
|
sidecar.
|
||||||
|
2. **Unvalidated query params.** `POST /api/transcribe`'s `parseParams`
|
||||||
|
validates `language` against `LANGUAGE_RE` and `task` against
|
||||||
|
`"transcribe" | "translate"` before it ever reaches the sidecar protocol.
|
||||||
|
The WS upgrade route in `src/index.ts` skipped both checks —
|
||||||
|
`url.searchParams.get("task") || undefined` forwarded an arbitrary string
|
||||||
|
straight into `LiveSessionData.task` and on to the Python sidecar's
|
||||||
|
JSON-lines protocol. Also skipped: the upfront `sidecar.status()` check the HTTP
|
||||||
|
route makes before doing any work (D-013's "degrade gracefully" —
|
||||||
|
opening a socket that can only report unavailability after the first
|
||||||
|
segment reaches the sidecar is a worse failure mode than a prompt 503).
|
||||||
|
3. **Zero test coverage.** No `tests/server-live.*` file existed; every
|
||||||
|
other stateful server module in the app (`transcribe.ts`, `auth.ts`,
|
||||||
|
`db.ts`) has one.
|
||||||
|
**Fix:**
|
||||||
|
1. `src/server/transcribe.ts` exports `parseLanguageAndTask(url)` (extracted
|
||||||
|
from `parseParams`, same validation, same 400-text-on-string-return
|
||||||
|
contract); `src/index.ts`'s `GET /api/transcribe/live` case now calls it
|
||||||
|
and 400s on an invalid value, then calls `sidecar.status()` and 503s with
|
||||||
|
the reason before upgrading — matching `POST /api/transcribe` exactly.
|
||||||
|
2. `src/server/live.ts` exports `createLiveWebSocket(deps: { sidecar:
|
||||||
|
Sidecar })`; `liveWebSocket` (the instance `src/index.ts` wires into
|
||||||
|
`Bun.serve`) is just `createLiveWebSocket({ sidecar })` against the real
|
||||||
|
singleton. `LiveSessionData.user`/`task` are now typed `SessionUser`/
|
||||||
|
`WhisperTask` instead of `any`/`string | undefined` (task always has a
|
||||||
|
value now that the WS route defaults it like the HTTP route does), which
|
||||||
|
let the pre-existing `options.websocket = liveWebSocket as any` cast in
|
||||||
|
`src/index.ts` drop to a plain assignment — it was papering over the
|
||||||
|
untyped `user`/`task` fields, not a real incompatibility with `Bun.serve`'s
|
||||||
|
generic `websocket` option.
|
||||||
|
3. `tests/server-live.test.ts` exercises `createLiveWebSocket` against a fake
|
||||||
|
`Sidecar` (same shape as `tests/transcribe.test.ts`'s): a committed
|
||||||
|
segment writes exactly its own PCM bytes to the temp file and cleans up
|
||||||
|
after, model/language/task pass through, a sidecar error surfaces via the
|
||||||
|
`update` message's `error` field instead of crashing the session, a
|
||||||
|
`{"type":"stop"}` frame drains and closes with a `result` message, and a
|
||||||
|
malformed text frame is ignored. `tests/transcribe.test.ts` gained direct
|
||||||
|
cases for `parseLanguageAndTask`.
|
||||||
|
**Incidental correctness fix found while rewriting the `transcribe` closure:**
|
||||||
|
the original code wrote `new Uint8Array(audio.buffer as ArrayBuffer)` to the
|
||||||
|
temp file — the whole underlying `ArrayBuffer`, ignoring `audio.byteOffset`/
|
||||||
|
`audio.byteLength`. Every real caller today happens to pass a fresh
|
||||||
|
`Float32Array` from `lib/live.ts`'s `concat()` helper (always offset 0, full
|
||||||
|
length), so this never manifested, but `LiveTranscribeFn`'s type signature
|
||||||
|
(`(audio: Float32Array) => ...`) makes no such promise to callers. Restored
|
||||||
|
the original's `audio.byteOffset, audio.byteLength` slice (which the
|
||||||
|
pre-refactor code already had — it would have been an actual regression to
|
||||||
|
drop it during the DI rewrite).
|
||||||
|
**Consequence:** the server-live path now has the same testability and
|
||||||
|
input-validation posture as the HTTP path; no behavior change for a valid
|
||||||
|
client, since `App.tsx` already only ever sends catalog model ids and
|
||||||
|
`"transcribe" | "translate"` task values.
|
||||||
|
**Alternatives rejected:** leaving `live.ts` untested and testing only
|
||||||
|
through a real spawned sidecar (matches `createSidecar`'s own integration
|
||||||
|
tests, but that suite already covers the sidecar process itself — the gap
|
||||||
|
was specifically the WebSocket/`LiveTranscriber` glue, which needs no Python
|
||||||
|
process to exercise).
|
||||||
|
|||||||
+23
-5
@@ -273,7 +273,19 @@ Notes (M5):
|
|||||||
(`estimatePeakLoadBytes`) is risky on a phone/tablet, covering the
|
(`estimatePeakLoadBytes`) is risky on a phone/tablet, covering the
|
||||||
explicit Download button and the lazy auto-load on file-drop/live-start
|
explicit Download button and the lazy auto-load on file-drop/live-start
|
||||||
alike.
|
alike.
|
||||||
- [ ] Basic a11y pass (keyboard, focus, aria on progress/live transcript)
|
- [x] Basic a11y pass (keyboard, focus, aria on progress/live transcript) —
|
||||||
|
the model-download and transcription `<progress>` bars gained
|
||||||
|
`aria-label`s (previously announced as an unnamed percentage);
|
||||||
|
`TranscriptView`'s heading is now a programmatic focus target
|
||||||
|
(`tabIndex={-1}`) that receives focus the moment a final result lands,
|
||||||
|
so keyboard/screen-reader users land on new content instead of hunting
|
||||||
|
for it; the Copy button's "Copied!" state and `LiveControls`' status
|
||||||
|
transitions (requesting mic / listening / finishing) now also announce
|
||||||
|
via a visually-hidden `role="status"` region for users not focused on
|
||||||
|
the button. Keyboard operability was already in place everywhere in
|
||||||
|
scope (native buttons/inputs, `FileDropzone`'s `role="button"` +
|
||||||
|
`onKeyDown`) and needed no changes. `bun test` unaffected (233 pass, 1
|
||||||
|
pre-existing unrelated WER-gate failure, see Decisions.md).
|
||||||
- [x] Production build + `bun start` deployment check; README for deployers —
|
- [x] Production build + `bun start` deployment check; README for deployers —
|
||||||
`bun run build` + `bun start` smoke-tested live (headers, hashed-asset
|
`bun run build` + `bun start` smoke-tested live (headers, hashed-asset
|
||||||
immutability, `/api/*` routing, `AUTH_ENABLED=false`, server-engine
|
immutability, `/api/*` routing, `AUTH_ENABLED=false`, server-engine
|
||||||
@@ -316,10 +328,16 @@ cached.
|
|||||||
- [~] Manual in-browser verification (mic permission, live text, offline run)
|
- [~] Manual in-browser verification (mic permission, live text, offline run)
|
||||||
— procedure in `tests/live.md`, pending a real browser (same dev-env
|
— procedure in `tests/live.md`, pending a real browser (same dev-env
|
||||||
caveat as M2/M3)
|
caveat as M2/M3)
|
||||||
- [ ] **Future want:** live transcription on the **server engine** — stream
|
- [x] Live transcription on the **server engine** (D-019) — `GET
|
||||||
mic audio over a WebSocket (D-011's escape hatch) to faster-whisper
|
/api/transcribe/live` upgrades to a WebSocket (`src/server/live.ts`,
|
||||||
(D-013) for logged-in users on weak devices. Needs its own decision
|
wired in `src/index.ts`); the client streams mic PCM and runs the
|
||||||
entry; blocked on M5 landing first.
|
browser-engine UI unchanged otherwise (`App.tsx`'s `startLive`/`stopLive`
|
||||||
|
branch on `useServer`). Landed already but was undocumented here and
|
||||||
|
untested; brought up to the rest of the codebase's bar (D-032):
|
||||||
|
`createLiveWebSocket` now takes the sidecar as an injected dependency
|
||||||
|
like `createTranscribeHandlers`, the WS route validates
|
||||||
|
`language`/`task` the same way the HTTP route does, and
|
||||||
|
`tests/server-live.test.ts` covers the bridge against a fake sidecar.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -41,6 +41,14 @@ export function LiveControls({
|
|||||||
? "Listening — like everything else, your audio stays on this device."
|
? "Listening — like everything else, your audio stays on this device."
|
||||||
: "…or transcribe live from your microphone."}
|
: "…or transcribe live from your microphone."}
|
||||||
</p>
|
</p>
|
||||||
|
{/* Announce mode transitions for screen-reader users not focused on the
|
||||||
|
button — the button's own label change is only reliably heard by
|
||||||
|
whoever triggered it. */}
|
||||||
|
<span className="visually-hidden" role="status">
|
||||||
|
{status === "starting" && "Requesting microphone access"}
|
||||||
|
{status === "listening" && "Live transcription started"}
|
||||||
|
{status === "stopping" && "Finishing live transcription"}
|
||||||
|
</span>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -39,7 +39,11 @@ export function ModelLoadProgress({ modelId, progress }: ModelLoadProgressProps)
|
|||||||
: "…"}
|
: "…"}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<progress max={100} value={percent} />
|
<progress
|
||||||
|
max={100}
|
||||||
|
value={percent}
|
||||||
|
aria-label={`Download progress for ${file.file}`}
|
||||||
|
/>
|
||||||
</li>
|
</li>
|
||||||
);
|
);
|
||||||
})}
|
})}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { useEffect, useState } from "react";
|
import { useEffect, useRef, useState } from "react";
|
||||||
import {
|
import {
|
||||||
exportFilename,
|
exportFilename,
|
||||||
transcriptToSrt,
|
transcriptToSrt,
|
||||||
@@ -64,6 +64,8 @@ export function TranscriptView({
|
|||||||
exportName = null,
|
exportName = null,
|
||||||
}: TranscriptViewProps) {
|
}: TranscriptViewProps) {
|
||||||
const [copied, setCopied] = useState(false);
|
const [copied, setCopied] = useState(false);
|
||||||
|
const headingRef = useRef<HTMLHeadingElement>(null);
|
||||||
|
const prevResultRef = useRef<typeof result>(null);
|
||||||
const text = result?.text ?? partialText;
|
const text = result?.text ?? partialText;
|
||||||
const copyText = `${text}${tentativeText}`.trim();
|
const copyText = `${text}${tentativeText}`.trim();
|
||||||
const showProgress =
|
const showProgress =
|
||||||
@@ -78,6 +80,16 @@ export function TranscriptView({
|
|||||||
return () => clearTimeout(timer);
|
return () => clearTimeout(timer);
|
||||||
}, [copied]);
|
}, [copied]);
|
||||||
|
|
||||||
|
// Move focus to the transcript heading the moment a final result lands, so
|
||||||
|
// keyboard/screen-reader users land on the new content instead of having to
|
||||||
|
// hunt for it after a file or live run finishes.
|
||||||
|
useEffect(() => {
|
||||||
|
if (result && !prevResultRef.current) {
|
||||||
|
headingRef.current?.focus();
|
||||||
|
}
|
||||||
|
prevResultRef.current = result;
|
||||||
|
}, [result]);
|
||||||
|
|
||||||
if (!transcribing && !text && !tentativeText) return null;
|
if (!transcribing && !text && !tentativeText) return null;
|
||||||
|
|
||||||
async function copy() {
|
async function copy() {
|
||||||
@@ -89,7 +101,7 @@ export function TranscriptView({
|
|||||||
return (
|
return (
|
||||||
<section className="transcript">
|
<section className="transcript">
|
||||||
<div className="transcript-header">
|
<div className="transcript-header">
|
||||||
<h2>
|
<h2 ref={headingRef} tabIndex={-1}>
|
||||||
{result ? "Transcript" : activeLabel}
|
{result ? "Transcript" : activeLabel}
|
||||||
{percent !== null && (
|
{percent !== null && (
|
||||||
<span className="transcript-percent"> {percent}%</span>
|
<span className="transcript-percent"> {percent}%</span>
|
||||||
@@ -150,6 +162,9 @@ export function TranscriptView({
|
|||||||
>
|
>
|
||||||
{copied ? "Copied!" : "Copy"}
|
{copied ? "Copied!" : "Copy"}
|
||||||
</button>
|
</button>
|
||||||
|
<span className="visually-hidden" role="status">
|
||||||
|
{copied ? "Copied to clipboard" : ""}
|
||||||
|
</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{showProgress && (
|
{showProgress && (
|
||||||
@@ -157,6 +172,7 @@ export function TranscriptView({
|
|||||||
<progress
|
<progress
|
||||||
max={progress.totalSeconds}
|
max={progress.totalSeconds}
|
||||||
value={Math.min(progress.processedSeconds, progress.totalSeconds)}
|
value={Math.min(progress.processedSeconds, progress.totalSeconds)}
|
||||||
|
aria-label="Transcription progress"
|
||||||
/>
|
/>
|
||||||
<span className="transcript-progress-time">
|
<span className="transcript-progress-time">
|
||||||
{formatTimestamp(progress.processedSeconds)} /{" "}
|
{formatTimestamp(progress.processedSeconds)} /{" "}
|
||||||
|
|||||||
@@ -16,6 +16,18 @@ body {
|
|||||||
min-height: 100vh;
|
min-height: 100vh;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.visually-hidden {
|
||||||
|
position: absolute;
|
||||||
|
width: 1px;
|
||||||
|
height: 1px;
|
||||||
|
padding: 0;
|
||||||
|
margin: -1px;
|
||||||
|
overflow: hidden;
|
||||||
|
clip: rect(0, 0, 0, 0);
|
||||||
|
white-space: nowrap;
|
||||||
|
border: 0;
|
||||||
|
}
|
||||||
|
|
||||||
.app {
|
.app {
|
||||||
max-width: 720px;
|
max-width: 720px;
|
||||||
margin: 0 auto;
|
margin: 0 auto;
|
||||||
|
|||||||
+20
-7
@@ -1,10 +1,11 @@
|
|||||||
import { serve } from "bun";
|
import { serve } from "bun";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
import { join, normalize, sep } from "node:path";
|
import { join, normalize, sep } from "node:path";
|
||||||
import { ensureIcons, ICONS_DIR } from "../scripts/generate-icons.ts";
|
import { ensureIcons, ICONS_DIR } from "../scripts/generate-icons.ts";
|
||||||
import { injectPrecacheManifest } from "./pwa/precache.ts";
|
import { injectPrecacheManifest } from "./pwa/precache.ts";
|
||||||
import { auth } from "./server/auth.ts";
|
import { auth } from "./server/auth.ts";
|
||||||
import { AUTH_ENABLED } from "./server/flags.ts";
|
import { AUTH_ENABLED } from "./server/flags.ts";
|
||||||
import { MAX_UPLOAD_BYTES, transcribe } from "./server/transcribe.ts";
|
import { MAX_UPLOAD_BYTES, parseLanguageAndTask, sidecar, transcribe } from "./server/transcribe.ts";
|
||||||
import { liveWebSocket } from "./server/live.ts";
|
import { liveWebSocket } from "./server/live.ts";
|
||||||
import { getFasterWhisperModel } from "./lib/models.ts";
|
import { getFasterWhisperModel } from "./lib/models.ts";
|
||||||
|
|
||||||
@@ -132,15 +133,27 @@ async function handleApiRequest(request: Request, server: any): Promise<Response
|
|||||||
if (!modelId) return new Response("Missing model", { status: 400 });
|
if (!modelId) return new Response("Missing model", { status: 400 });
|
||||||
const fasterWhisperModel = getFasterWhisperModel(modelId);
|
const fasterWhisperModel = getFasterWhisperModel(modelId);
|
||||||
if (!fasterWhisperModel) return new Response("Unknown model", { status: 400 });
|
if (!fasterWhisperModel) return new Response("Unknown model", { status: 400 });
|
||||||
|
// Same validation as POST /api/transcribe (D-030-adjacent: never forward
|
||||||
|
// an unvalidated task/language string into the sidecar protocol just
|
||||||
|
// because this entry point is a WebSocket, not JSON over HTTP).
|
||||||
|
const options = parseLanguageAndTask(url);
|
||||||
|
if (typeof options === "string") return new Response(options, { status: 400 });
|
||||||
|
// Fail fast like the HTTP route (D-013 "degrade gracefully") instead of
|
||||||
|
// opening a socket that can only report unavailability once the first
|
||||||
|
// segment reaches the sidecar.
|
||||||
|
const status = await sidecar.status();
|
||||||
|
if (!status.available) {
|
||||||
|
return new Response(status.reason ?? "server engine unavailable", { status: 503 });
|
||||||
|
}
|
||||||
|
|
||||||
const success = server.upgrade(request, {
|
const success = server.upgrade(request, {
|
||||||
data: {
|
data: {
|
||||||
user,
|
user,
|
||||||
model: fasterWhisperModel,
|
model: fasterWhisperModel,
|
||||||
language: url.searchParams.get("language") || undefined,
|
language: options.language,
|
||||||
task: url.searchParams.get("task") || undefined,
|
task: options.task,
|
||||||
tempDir: require("node:os").tmpdir()
|
tempDir: tmpdir(),
|
||||||
}
|
},
|
||||||
});
|
});
|
||||||
if (success) return undefined;
|
if (success) return undefined;
|
||||||
return new Response("WebSocket upgrade failed", { status: 500 });
|
return new Response("WebSocket upgrade failed", { status: 500 });
|
||||||
@@ -310,7 +323,7 @@ options.port = process.env.PORT || 3000;
|
|||||||
// Bun's default cap is 128 MB, which would 4xx a /api/transcribe upload of
|
// Bun's default cap is 128 MB, which would 4xx a /api/transcribe upload of
|
||||||
// more than ~35 minutes of raw PCM before the route's own limit applies.
|
// more than ~35 minutes of raw PCM before the route's own limit applies.
|
||||||
options.maxRequestBodySize = MAX_UPLOAD_BYTES;
|
options.maxRequestBodySize = MAX_UPLOAD_BYTES;
|
||||||
options.websocket = liveWebSocket as any;
|
options.websocket = liveWebSocket;
|
||||||
|
|
||||||
const server = serve(options);
|
const server = serve(options);
|
||||||
|
|
||||||
|
|||||||
+116
-88
@@ -1,103 +1,131 @@
|
|||||||
import { tmpdir } from "node:os";
|
/**
|
||||||
|
* Server-side live microphone transcription over WebSocket (D-019): the
|
||||||
|
* browser streams raw float32 16 kHz mono PCM frames, the server runs the
|
||||||
|
* same pure `LiveTranscriber`/segmenter as the browser engine (src/lib/live.ts)
|
||||||
|
* and bridges each pass to the faster-whisper sidecar (src/server/transcribe.ts,
|
||||||
|
* D-013) by writing the segment audio to a temp file. Wired at
|
||||||
|
* `GET /api/transcribe/live` in src/index.ts.
|
||||||
|
*
|
||||||
|
* `createLiveWebSocket` takes the `Sidecar` as an injected dependency —
|
||||||
|
* mirroring `createTranscribeHandlers` in transcribe.ts — so tests exercise
|
||||||
|
* this module against a fake sidecar instead of spawning a real Python
|
||||||
|
* process (see tests/server-live.test.ts).
|
||||||
|
*/
|
||||||
|
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import type { ServerWebSocket } from "bun";
|
import type { ServerWebSocket } from "bun";
|
||||||
import { LIVE_TUNING, LiveTranscriber, type LiveUpdate } from "../lib/live";
|
import { LIVE_TUNING, LiveTranscriber, type LiveUpdate } from "../lib/live";
|
||||||
import { sidecar } from "./transcribe";
|
import { sidecar, type Sidecar } from "./transcribe";
|
||||||
import { getSessionUser } from "./auth";
|
import type { SessionUser } from "./auth";
|
||||||
import { getFasterWhisperModel } from "../lib/models";
|
|
||||||
import type { WhisperTask } from "../worker/protocol";
|
import type { WhisperTask } from "../worker/protocol";
|
||||||
|
|
||||||
interface LiveSessionData {
|
export interface LiveSessionData {
|
||||||
user: any;
|
user: SessionUser;
|
||||||
|
/** faster-whisper model name (catalog mapping, src/lib/models.ts). */
|
||||||
model: string;
|
model: string;
|
||||||
language?: string;
|
language?: string;
|
||||||
task?: WhisperTask;
|
task: WhisperTask;
|
||||||
tempDir: string;
|
tempDir: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export const liveWebSocket = {
|
export interface LiveWebSocketDeps {
|
||||||
open(ws: ServerWebSocket<LiveSessionData>) {
|
sidecar: Sidecar;
|
||||||
// Initialize the transcriber for this connection.
|
}
|
||||||
const transcriber = new LiveTranscriber({
|
|
||||||
...LIVE_TUNING,
|
|
||||||
transcribe: async (audio: Float32Array) => {
|
|
||||||
const audioPath = join(
|
|
||||||
ws.data.tempDir,
|
|
||||||
`whisper-live-${crypto.randomUUID()}.f32`
|
|
||||||
);
|
|
||||||
const sink = Bun.file(audioPath).writer();
|
|
||||||
// audio.buffer is an ArrayBuffer, we need a Uint8Array view
|
|
||||||
sink.write(new Uint8Array(audio.buffer as ArrayBuffer, audio.byteOffset, audio.byteLength));
|
|
||||||
await sink.end();
|
|
||||||
|
|
||||||
return new Promise((resolve, reject) => {
|
/** Exported for tests (tests/server-live.test.ts), which fake this shape. */
|
||||||
let text = "";
|
export type LiveSocket = ServerWebSocket<LiveSessionData> & { transcriber?: LiveTranscriber };
|
||||||
let chunks: any[] = [];
|
|
||||||
|
|
||||||
sidecar.transcribe(
|
|
||||||
{
|
|
||||||
model: ws.data.model,
|
|
||||||
audioPath,
|
|
||||||
language: ws.data.language,
|
|
||||||
task: ws.data.task,
|
|
||||||
},
|
|
||||||
(segment) => {
|
|
||||||
text += segment.text;
|
|
||||||
chunks.push({
|
|
||||||
text: segment.text,
|
|
||||||
timestamp: [segment.start, segment.end],
|
|
||||||
});
|
|
||||||
}
|
|
||||||
)
|
|
||||||
.then(() => resolve({ text: text.trim(), chunks }))
|
|
||||||
.catch(reject)
|
|
||||||
.finally(() => Bun.file(audioPath).unlink().catch(() => {}));
|
|
||||||
});
|
|
||||||
},
|
|
||||||
onUpdate: (update: LiveUpdate) => {
|
|
||||||
ws.send(JSON.stringify({ type: "update", update }));
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
// Store it on the websocket
|
|
||||||
(ws as any).transcriber = transcriber;
|
|
||||||
},
|
|
||||||
|
|
||||||
message(ws: ServerWebSocket<LiveSessionData>, message: string | Buffer | Uint8Array) {
|
export function createLiveWebSocket(deps: LiveWebSocketDeps) {
|
||||||
if (typeof message === "string") {
|
return {
|
||||||
let parsed: any;
|
open(ws: LiveSocket) {
|
||||||
try {
|
const transcriber = new LiveTranscriber({
|
||||||
parsed = JSON.parse(message);
|
...LIVE_TUNING,
|
||||||
} catch {
|
transcribe: async (audio) => {
|
||||||
// Not a frame we understand — ignore rather than crash the handler
|
const audioPath = join(
|
||||||
// (which would otherwise take the whole connection down for every
|
ws.data.tempDir,
|
||||||
// client sharing this Bun.serve process, not just this session).
|
`whisper-live-${crypto.randomUUID()}.f32`,
|
||||||
|
);
|
||||||
|
// audio may be a view into a larger buffer — byteOffset/byteLength
|
||||||
|
// must be preserved, not just the underlying buffer.
|
||||||
|
await Bun.write(
|
||||||
|
audioPath,
|
||||||
|
new Uint8Array(audio.buffer as ArrayBuffer, audio.byteOffset, audio.byteLength),
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
const text: string[] = [];
|
||||||
|
const chunks: { text: string; timestamp: [number, number | null] }[] = [];
|
||||||
|
await deps.sidecar.transcribe(
|
||||||
|
{
|
||||||
|
model: ws.data.model,
|
||||||
|
audioPath,
|
||||||
|
language: ws.data.language,
|
||||||
|
task: ws.data.task,
|
||||||
|
},
|
||||||
|
(segment) => {
|
||||||
|
text.push(segment.text);
|
||||||
|
chunks.push({ text: segment.text, timestamp: [segment.start, segment.end] });
|
||||||
|
},
|
||||||
|
);
|
||||||
|
return { text: text.join("").trim(), chunks };
|
||||||
|
} finally {
|
||||||
|
await Bun.file(audioPath).unlink().catch(() => {});
|
||||||
|
}
|
||||||
|
},
|
||||||
|
onUpdate: (update: LiveUpdate) => {
|
||||||
|
ws.send(JSON.stringify({ type: "update", update }));
|
||||||
|
},
|
||||||
|
});
|
||||||
|
ws.transcriber = transcriber;
|
||||||
|
},
|
||||||
|
|
||||||
|
message(ws: LiveSocket, message: string | Buffer) {
|
||||||
|
if (typeof message === "string") {
|
||||||
|
let parsed: { type?: string };
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(message);
|
||||||
|
} catch {
|
||||||
|
// Not a frame we understand — ignore rather than crash the handler
|
||||||
|
// (which would otherwise take the whole connection down for every
|
||||||
|
// client sharing this Bun.serve process, not just this session).
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (parsed.type === "stop") {
|
||||||
|
ws.transcriber
|
||||||
|
?.stop()
|
||||||
|
.then((result) => {
|
||||||
|
ws.send(JSON.stringify({ type: "result", result }));
|
||||||
|
ws.close();
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
ws.send(
|
||||||
|
JSON.stringify({
|
||||||
|
type: "error",
|
||||||
|
error: err instanceof Error ? err.message : String(err),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
ws.close();
|
||||||
|
});
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (parsed.type === "stop") {
|
// Binary frame: a raw float32 mono 16 kHz PCM chunk (lib/mic.ts already
|
||||||
const transcriber = (ws as any).transcriber as LiveTranscriber;
|
// resampled it on the main thread, same contract as the browser engine).
|
||||||
transcriber.stop().then((result) => {
|
const audio = new Float32Array(
|
||||||
ws.send(JSON.stringify({ type: "result", result }));
|
message.buffer,
|
||||||
ws.close();
|
message.byteOffset,
|
||||||
}).catch((err) => {
|
message.byteLength / 4,
|
||||||
ws.send(JSON.stringify({ type: "error", error: err instanceof Error ? err.message : String(err) }));
|
);
|
||||||
ws.close();
|
ws.transcriber?.push(audio);
|
||||||
});
|
},
|
||||||
}
|
|
||||||
} else {
|
|
||||||
// Audio chunk arrived
|
|
||||||
const transcriber = (ws as any).transcriber as LiveTranscriber;
|
|
||||||
// Convert raw bytes to Float32Array (16kHz mono)
|
|
||||||
const buffer = message instanceof Buffer ? message.buffer : message.buffer;
|
|
||||||
const byteOffset = message instanceof Buffer ? message.byteOffset : (message as Uint8Array).byteOffset;
|
|
||||||
const byteLength = message.byteLength;
|
|
||||||
const audio = new Float32Array(buffer, byteOffset, byteLength / 4);
|
|
||||||
transcriber.push(audio);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
|
|
||||||
close(ws: ServerWebSocket<LiveSessionData>) {
|
close(ws: LiveSocket) {
|
||||||
const transcriber = (ws as any).transcriber as LiveTranscriber;
|
ws.transcriber?.stop().catch(() => {});
|
||||||
transcriber?.stop().catch(() => {});
|
},
|
||||||
},
|
};
|
||||||
};
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Default instance (real sidecar), wired in src/index.ts
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export const liveWebSocket = createLiveWebSocket({ sidecar });
|
||||||
|
|||||||
@@ -324,14 +324,16 @@ interface TranscribeParams {
|
|||||||
task: WhisperTask;
|
task: WhisperTask;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Parse + validate query params; a string is the 400 error text. */
|
/**
|
||||||
function parseParams(url: URL): TranscribeParams | string {
|
* Parse + validate the `language`/`task` query params shared by both the
|
||||||
const modelId = url.searchParams.get("model");
|
* HTTP (`POST /api/transcribe`) and WebSocket (`GET /api/transcribe/live`,
|
||||||
if (!modelId) return "missing model parameter";
|
* src/server/live.ts) entry points, so a live session can't forward an
|
||||||
const fasterWhisperModel = getFasterWhisperModel(modelId);
|
* unvalidated string into the sidecar protocol just because it skips
|
||||||
// Catalog-only (D-006/D-013): never forward arbitrary ids to the sidecar.
|
* `parseParams` below. A string return is the 400 error text.
|
||||||
if (!fasterWhisperModel) return `unknown model: ${modelId}`;
|
*/
|
||||||
|
export function parseLanguageAndTask(
|
||||||
|
url: URL,
|
||||||
|
): { language?: string; task: WhisperTask } | string {
|
||||||
const rawLanguage = url.searchParams.get("language");
|
const rawLanguage = url.searchParams.get("language");
|
||||||
let language: string | undefined;
|
let language: string | undefined;
|
||||||
if (rawLanguage !== null && rawLanguage !== "" && rawLanguage !== "auto") {
|
if (rawLanguage !== null && rawLanguage !== "" && rawLanguage !== "auto") {
|
||||||
@@ -344,7 +346,21 @@ function parseParams(url: URL): TranscribeParams | string {
|
|||||||
return `invalid task: ${rawTask}`;
|
return `invalid task: ${rawTask}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
return { modelId, fasterWhisperModel, language, task: rawTask };
|
return { language, task: rawTask };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Parse + validate query params; a string is the 400 error text. */
|
||||||
|
function parseParams(url: URL): TranscribeParams | string {
|
||||||
|
const modelId = url.searchParams.get("model");
|
||||||
|
if (!modelId) return "missing model parameter";
|
||||||
|
const fasterWhisperModel = getFasterWhisperModel(modelId);
|
||||||
|
// Catalog-only (D-006/D-013): never forward arbitrary ids to the sidecar.
|
||||||
|
if (!fasterWhisperModel) return `unknown model: ${modelId}`;
|
||||||
|
|
||||||
|
const options = parseLanguageAndTask(url);
|
||||||
|
if (typeof options === "string") return options;
|
||||||
|
|
||||||
|
return { modelId, fasterWhisperModel, ...options };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import { join } from "node:path";
|
|||||||
import {
|
import {
|
||||||
createSidecar,
|
createSidecar,
|
||||||
createTranscribeHandlers,
|
createTranscribeHandlers,
|
||||||
|
parseLanguageAndTask,
|
||||||
type Sidecar,
|
type Sidecar,
|
||||||
type SidecarSegment,
|
type SidecarSegment,
|
||||||
type SidecarTranscribeRequest,
|
type SidecarTranscribeRequest,
|
||||||
@@ -124,6 +125,46 @@ async function readStreamLines(response: Response): Promise<TranscribeStreamLine
|
|||||||
// HTTP handlers (fake sidecar)
|
// HTTP handlers (fake sidecar)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// parseLanguageAndTask — shared with the WS live route (src/server/live.ts,
|
||||||
|
// src/index.ts) so it can't skip the validation the HTTP route enforces.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
describe("parseLanguageAndTask", () => {
|
||||||
|
test("defaults to transcribe with no language", () => {
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x"))).toEqual({
|
||||||
|
language: undefined,
|
||||||
|
task: "transcribe",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("accepts a valid language and task", () => {
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x?language=da&task=translate"))).toEqual(
|
||||||
|
{ language: "da", task: "translate" },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('"auto"/empty language means no language override', () => {
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x?language=auto"))).toEqual({
|
||||||
|
language: undefined,
|
||||||
|
task: "transcribe",
|
||||||
|
});
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x?language="))).toEqual({
|
||||||
|
language: undefined,
|
||||||
|
task: "transcribe",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("rejects an invalid language or task", () => {
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x?language=Danish!"))).toBe(
|
||||||
|
"invalid language: Danish!",
|
||||||
|
);
|
||||||
|
expect(parseLanguageAndTask(new URL("http://localhost/x?task=summarize"))).toBe(
|
||||||
|
"invalid task: summarize",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("GET /api/transcribe/status", () => {
|
describe("GET /api/transcribe/status", () => {
|
||||||
test("401 without a session", async () => {
|
test("401 without a session", async () => {
|
||||||
const handlers = createTranscribeHandlers({
|
const handlers = createTranscribeHandlers({
|
||||||
|
|||||||
Reference in New Issue
Block a user