diff --git a/docs/superpowers/plans/2026-10-30-stcourses.md b/docs/superpowers/plans/2026-10-30-stcourses.md new file mode 100644 index 0000000..cdab0b4 --- /dev/null +++ b/docs/superpowers/plans/2026-10-30-stcourses.md @@ -0,0 +1,572 @@ +# STCourses Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build STCourses, a self-hosted course platform. Course content is files and user data lives in SQLite or Postgres. The build is delivered as a shared contract (Phase 0), four parallel work packages (Phase 1) and an integration pass (Phase 2). + +**Architecture:** +- One Bun process (`Bun.serve`) serves a React single-page app through HTML imports, plus a JSON API. +- Course folders, described by `stcourse.yaml` manifests (in the course repo or as an overlay), are loaded into an in-memory course model. +- User data goes through one `Bun.SQL` data layer with plain-SQL migrations that run unchanged on SQLite and Postgres. +- Features (learner, import, studio, AI) are packages that export a `Feature` (routes plus job handlers) and a set of web routes. They are composed by the server and the web shell. + +**Tech Stack:** +- Bun 1.3.x and TypeScript (strict). +- React 18 and React Router 6 (library mode). +- zod for validation, `Bun.YAML` for manifests, `Bun.SQL` for databases. +- @fontsource packages for IBM Plex Sans, IBM Plex Mono and Literata. +- `bun test` for tests. + +**Spec:** `docs/superpowers/specs/2026-10-30-stcourses-design.md` (sections 1–13). Visual reference: `docs/design/mockups/platform-v1.html`. + +## Global Constraints + +These apply project-wide; every task inherits them. + +- **Bun conventions** (from `bun init`'s CLAUDE.md): + - use `Bun.serve` routes and WebSockets, `bun:sqlite` or `Bun.sql`, `Bun.file`, `` Bun.$ ``, `bun test`, `bun install`, `bunx`; + - never use express, vite, dotenv, pg, postgres.js, better-sqlite3, jest or vitest. +- **Frontend:** a React SPA served via HTML imports. No Next.js. Plain CSS with custom properties; no Tailwind. +- **Course content is never stored in the database.** Deleting the DB loses user data only. +- **Database URL:** `STC_DATABASE_URL`, either `sqlite://` (default `sqlite://./data/stcourses.db`) or `postgres://…`. Every migration and query must run on both. +- **Idle memory:** at most 2 GB for the deployment (checked in Phase 2). +- **License:** AGPL-3.0-only. Every package's `package.json` has `"license": "AGPL-3.0-only"`. +- **Never write into `/home/user/functional-programming-course` or `/home/user/systemic-design-course`.** Their manifests are overlays in `overlays/`. +- **Never commit anything from `/home/user/example_course_content`.** It is licensed material, used only for local test runs. +- **Design tokens** (spec section 4.1): + - ground `#12141a`, side `#181b22`, raised `#232733`, line `#262a33`, line2 `#343947`; + - text `#d9dce3`, reading text `#d9d4c8`, heading `#f2efe8`, muted `#a4abb8`, faint `#8c93a0`; + - accent `#f1b44c`, accent-dim `#8c7440`; + - fonts: IBM Plex Sans (UI), Literata (lesson titles and reading text, 19px / 1.75 in a 660px column), IBM Plex Mono (code); + - dark theme only in v1. +- **Keyboard:** arrow keys in the slide viewer, Escape closes overlays, normal Tab focus. No other shortcuts. +- **Commits:** author is the git identity configured on the machine, and every message ends with `Co-Authored-By: Claude Opus 5.5 `. + +## Review Focus + +1. **Paths with spaces and non-ASCII characters** (for example `S08 - Containerizing Applications/`, `Thinknyx_Mastering_Docker_Essentials_Handson_DevOps.pdf`) must load, serve and download correctly. Tests: Task 4 (loader) and Task 9 (media route). +2. **An overlay whose `root` repo is missing or moved** must give a course with `status: "error"` and a readable diagnostic. The server must keep serving every other course. Tests: Task 4 and Task 9. +3. **A manifest that references a missing file, heading anchor or deck id** must give a diagnostic naming the lesson and the path, not throw. Test: Task 4. +4. **Video seeking:** `Range: bytes=…` requests get `206` with correct `Content-Range`. An unsatisfiable range gets `416`. Test: Task 9. +5. **Media paths with `..`, absolute paths or symlinks that escape the course root** must get `404` and never read outside the root. Test: Task 9. + +--- + +## Repository layout (locked in by Phase 0) + +``` +package.json # bun workspaces: ["packages/*", "apps/*"] +tsconfig.json CLAUDE.md LICENSE README.md .gitignore +packages/ + contracts/ # shared TS types and interfaces only (no runtime deps) + src/index.ts src/server.ts src/exercise.ts src/ai.ts src/import.ts src/web.ts + course-format/ # manifest schema, markdown extraction, discovery, CourseRepository + src/schema.ts src/markdown.ts src/load.ts src/repository.ts src/index.ts test/ + db/ # Bun.SQL connection, migrations, repositories + src/connect.ts src/migrate.ts migrations/0001_init.sql src/repos/*.ts src/index.ts test/ + core/ # config, job queue, feature composition, media serving + src/config.ts src/jobs.ts src/media.ts src/app.ts src/index.ts test/ + ui/ # tokens.css, fonts, shared React primitives (AppShell, Pill, ProgressBar) + src/tokens.css src/fonts.css src/AppShell.tsx src/primitives.tsx src/index.ts + feature-learner/ feature-import/ feature-studio/ feature-ai/ # Phase 1 (one per work package) +apps/ + server/ # entry: src/main.ts — builds ServerContext, composes features, Bun.serve + web/ # index.html, src/main.tsx (router built from feature route registries) +overlays/ + pure-snake/stcourse.yaml # root: /home/user/functional-programming-course + systemic-design/stcourse.yaml # root: /home/user/systemic-design-course +fixtures/courses/sample/ # tiny self-contained course used by all tests +scripts/test-pg.sh # starts postgres:17 in docker, runs DB tests with STC_TEST_PG_URL +``` + +--- + +# Phase 0 — Contract (sequential, on `main`) + +### Task 1: Workspace scaffold + +**Files:** +- Create: `package.json`, `tsconfig.json`, `CLAUDE.md`, `LICENSE`, `README.md`. +- Create: one `package.json` per package and app in the layout above (`name: "@stc/"`, `type: "module"`, `license: "AGPL-3.0-only"`). +- Modify: `.gitignore`, adding `data/` and `.cache/`. + +**Interfaces:** +- Produces: workspace package names `@stc/contracts`, `@stc/course-format`, `@stc/db`, `@stc/core`, `@stc/ui`, `@stc/server`, `@stc/web`, `@stc/feature-learner`, `@stc/feature-import`, `@stc/feature-studio`, `@stc/feature-ai`. + +- [ ] **Step 1:** In a scratch directory, run `bun init -y` and copy its generated `CLAUDE.md` text into the root `CLAUDE.md`. Then add a "Project rules" section that repeats the Global Constraints above and points to the spec and plan. +- [ ] **Step 2:** Write the root `package.json` with `"workspaces": ["packages/*","apps/*"]` and these scripts: + - `test`: `bun test` + - `typecheck`: `bunx tsc --noEmit -p .` + - `dev`: `bun --hot apps/server/src/main.ts` + + Write the root `tsconfig.json` with `strict: true`, `jsx: "react-jsx"`, `moduleResolution: "bundler"`, `types: ["bun-types"]`, and `paths` mapping `@stc/*` to `packages/*/src`. +- [ ] **Step 3:** Write the full AGPL-3.0 text to `LICENSE`. Write `README.md` with one paragraph from spec section 1 and the dev commands. +- [ ] **Step 4:** Add `packages/contracts/test/smoke.test.ts` asserting `expect(1).toBe(1)`. Run `bun install && bun test && bun run typecheck`. Expected: 1 pass, no type errors. +- [ ] **Step 5:** Commit: `chore: bun workspace scaffold`. + +### Task 2: Contracts package (types only) + +**Files:** +- Create: `packages/contracts/src/{index.ts,server.ts,exercise.ts,ai.ts,import.ts,web.ts}` +- Test: `packages/contracts/test/types.test.ts` + +**Interfaces (Produces). This exact code is the contract; copy it verbatim:** + +```ts +// server.ts +export type Role = "owner" | "author" | "learner"; +export interface AuthUser { id: string; email: string; name: string; roles: Role[]; groups: string[] } +export interface RequestContext { user: AuthUser | null; params: Record; url: URL } +export type RouteHandler = (req: Request, rc: RequestContext) => Response | Promise; +export interface Feature { + name: string; // "learner" | "import" | "studio" | "ai" | "core" + routes: Record>>; // paths start with /api/ + jobs?: Record; // job kind -> handler +} +export interface JobContext { reportProgress(fraction: number, message?: string): Promise; signal: AbortSignal } +export type JobHandler = (payload: unknown, jc: JobContext) => Promise; +export type JobStatus = "queued" | "running" | "done" | "failed"; +export interface JobRecord { id: string; kind: string; status: JobStatus; progress: number; message: string | null; payload: unknown; result: unknown; error: string | null; createdAt: string; updatedAt: string } +export interface JobQueue { enqueue(kind: string, payload: unknown): Promise; get(id: string): Promise; list(kind?: string): Promise } +export interface AuthService { userFromRequest(req: Request): Promise } +export interface ServerContext { + config: AppConfig; db: import("@stc/db").Db; courses: import("@stc/course-format").CourseRepository; + jobs: JobQueue; auth: AuthService; +} +export interface AppConfig { + port: number; databaseUrl: string; dataDir: string; // dataDir holds cache/, media/, worktrees/ + courseDirs: string[]; // folders scanned for stcourse.yaml (course repos and overlays/) + secret: string; // STC_SECRET, used to encrypt BYOK keys + publicUrl: string; +} +// exercise.ts — browser-runnable exercises (spec 4.3 type 6) +export type ExerciseLanguage = "js" | "ts"; +export interface Assert { equal(actual: unknown, expected: unknown, message?: string): void; deepEqual(actual: unknown, expected: unknown, message?: string): void; ok(value: unknown, message?: string): void } +export interface ExerciseTest { name: string; run(mod: Record, assert: Assert): void | Promise } +// A tests file is an ES module: `export const tests: ExerciseTest[]`. Learner code is an ES module. +export interface TestResult { name: string; passed: boolean; message?: string; expected?: string; actual?: string } +export interface ExerciseRunner { language: ExerciseLanguage; run(learnerCode: string, testsModule: string, timeoutMs: number): Promise } +// ai.ts +export interface ChatMessage { role: "system" | "user" | "assistant" | "tool"; content: string; toolCallId?: string; name?: string } +export interface ToolSpec { name: string; description: string; inputSchema: Record } +export interface AiProvider { id: string; chat(messages: ChatMessage[], opts: { tools?: ToolSpec[]; signal?: AbortSignal }): AsyncIterable<{ text?: string; toolCall?: { id: string; name: string; input: unknown } }> } +// import.ts +export interface ImportSource { kind: "packt-video" | "pdf-deck" | "code-repo" | "video-folder"; path: string; options?: Record } +export interface ImportResult { courseId: string; manifestPath: string; warnings: string[] } +// web.ts +export interface WebRoute { path: string; element: () => Promise<{ default: React.ComponentType }>; nav?: { label: string; area: "header" | "admin" } } +export interface WebFeature { name: string; routes: WebRoute[] } +``` + +- [ ] **Step 1:** Write `types.test.ts`. It builds one object literal of each interface above, typed by the interface (so it fails to compile if a field is wrong), and runs `expect(typeof feature.routes).toBe("object")`. +- [ ] **Step 2:** Run `bun run typecheck`. Expected: errors, because the modules don't exist yet. +- [ ] **Step 3:** Create the files with the code above. `index.ts` re-exports everything. `web.ts` uses `import type React from "react"`. +- [ ] **Step 4:** Run `bun test packages/contracts && bun run typecheck`. Expected: PASS. (The `@stc/db` and `@stc/course-format` type imports resolve once Tasks 4 and 7 exist. Until then, temporarily declare `Db` and `CourseRepository` as `unknown` placeholders in `server.ts`, and switch them in Task 7.) +- [ ] **Step 5:** Commit: `feat(contracts): shared server, exercise, ai, import and web interfaces`. + +### Task 3: Manifest schema + +**Files:** +- Create: `packages/course-format/src/schema.ts` +- Test: `packages/course-format/test/schema.test.ts` + +**Interfaces (Produces):** +- `parseManifest(yamlText: string): { ok: true; manifest: Manifest } | { ok: false; errors: string[] }`. It uses `Bun.YAML.parse`, then the zod `ManifestSchema`. Each error string reads `": "`. +- `type Manifest = z.infer`, plus the exported types `Module`, `Lesson`, `Deck`, `Attachment`, `Track`. + +Schema (fields are required unless marked `?`): + +```yaml +schema: 1 # literal 1 +id: string # /^[a-z0-9][a-z0-9-]*$/ +title: string +pitch?: string +authors?: string[] +root?: string # overlay only: absolute path, or path relative to this manifest, of the course repo; default "." +cover?: string # path relative to root +accent?: string # /^#[0-9a-fA-F]{6}$/ +collections?: string[] +visibility: public | signed-in | restricted # default restricted +gating?: none | sequential # default none +tracks?: [{ id: string, label: string }] +mentor?: string # path to mentor instructions file +source?: { kind: authored | packt-video | pdf-deck | code-repo | video-folder, origin?: string, importedAt?: string } +decks?: [{ id: string, file: string, title?: string, attachTo: string }] # attachTo = a lesson id or module id +attachments?: [{ file: string, title?: string, module?: string }] +modules: + - id: string + title: string + code?: { dir: string, command?: string } # "Files for this section" + lessons: + - id: string # unique across the course + title: string + type: text | video | exercise + file?: string # required for text and video + anchor?: string # heading slug inside file + duration?: number # seconds + transcript?: string # video only + tracks?: { : { file?: string, command?: string } } # per-track file shown or command to run + hints?: { file: string, match: string } # exercise:
blocks whose summary starts with match + browser?: { language: js | ts, starter: string, tests: string, solution?: string } # exercise runnable in browser + localCommand?: string | { : string } +``` + +- [ ] **Step 1: Write the failing tests:** + - `parses a minimal manifest`: id `x`, one module with one text lesson. Expect `visibility` to default to `"restricted"` and `gating` to `"none"`. + - `rejects a bad accent`: `accent: "orange"`. Expect an error containing `accent`. + - `rejects duplicate lesson ids across modules`. Expect an error containing the duplicate id. + - `requires file for text and video lessons`. + - `rejects a deck whose attachTo is not a lesson or module id`. + - `rejects schema: 2`. +- [ ] **Step 2:** Run `bun test packages/course-format/test/schema.test.ts`. Expected: FAIL (module not found). +- [ ] **Step 3:** Implement `ManifestSchema`. Cross-field checks (duplicate ids, `attachTo`, `file` required by type) go in a zod `.superRefine`. Add the zod dependency to `@stc/course-format`. +- [ ] **Step 4:** Run the same command. Expected: 6 pass. +- [ ] **Step 5:** Commit: `feat(course-format): stcourse.yaml schema`. + +### Task 4: Markdown extraction, discovery and loading + +**Files:** +- Create: `packages/course-format/src/{markdown.ts,load.ts,repository.ts,index.ts}` +- Test: `packages/course-format/test/{markdown.test.ts,load.test.ts}` + +**Interfaces (Produces):** +- `slugify(heading: string): string`. GitHub style: lowercase, strip markdown and backticks, drop punctuation except `-`, turn spaces into `-`. `"Exercise 3.1 — `move`"` becomes `"exercise-31--move"`. Copy GitHub's exact algorithm (em dash removed, so the two spaces leave a double hyphen). +- `extractSection(markdown: string, anchor: string): string | null`. Returns the heading line plus everything up to the next heading of the same or a higher level. +- `extractHints(markdown: string, match: string): { label: string; body: string }[]`. Returns `
……
` blocks whose summary text starts with `match`, in document order. `label` is the summary text after `match`, with leading `" — "`, `" - "` or `" ("` trimmed. +- `type Diagnostic = { level: "error" | "warning"; courseId?: string; lessonId?: string; path?: string; message: string }` +- `interface LoadedCourse { id: string; manifestPath: string; root: string; manifest: Manifest | null; status: "ok" | "error"; diagnostics: Diagnostic[]; lessonIndex: Map }` +- `discoverManifests(courseDirs: string[]): Promise`. Finds `stcourse.yaml` at depth 0–2 under each dir. Duplicate ids are resolved in `loadAll`. +- `loadCourse(manifestPath: string): Promise`. Resolves `root`, then checks that every referenced file exists under root, every `anchor` resolves through `extractSection`, and every `hints.match` finds at least one block. Each failure becomes a diagnostic; nothing throws. A missing root gives `status: "error"`. +- `loadAll(courseDirs: string[]): Promise`. When two manifests share an id, the one whose `root` points outside its own directory (the overlay) wins, and a warning diagnostic records the shadowed path (spec section 13). +- `interface CourseRepository { list(): LoadedCourse[]; get(id: string): LoadedCourse | null; reload(id?: string): Promise; readLessonMarkdown(courseId: string, lessonId: string): Promise; resolvePath(courseId: string, relPath: string): string | null }` + - `resolvePath` returns an absolute path only if the real path (following symlinks) is inside the course root. Otherwise it returns `null`. +- `createCourseRepository(courseDirs: string[]): Promise`. + +- [ ] **Step 1: Write the failing tests in `markdown.test.ts`:** + - `slugify` of `"Exercise 3.1 — \`move\`"` equals `"exercise-31--move"`, and `"Why this module"` equals `"why-this-module"`. + - `extractSection` on a 3-heading sample returns only the middle `###` section. + - `extractHints` on a sample containing `Exercise 3.1 (move) — nudge` and `Exercise 3.1 (move) — stronger hint`, with match `"Exercise 3.1 (move)"`, returns labels `["nudge", "stronger hint"]`. +- [ ] **Step 2: Write the failing tests in `load.test.ts`**, using temp dirs built with `Bun.write`: + - `loads the sample fixture with no errors` (Task 5 fixture). + - `overlay with missing root gives status error`: diagnostic message contains the root path, and `loadAll` still returns the other course. + - `missing lesson file, anchor and hints each produce one error diagnostic naming lessonId`. + - `handles spaces and unicode in paths`: lesson file `S08 - Containerizing Applications/Æble notes.md`. + - `overlay wins over in-repo manifest with same id` (spec section 13). + - `resolvePath rejects ../ and symlink escape`: create a symlink inside root pointing at `/etc`, and expect `null`. +- [ ] **Step 3:** Run `bun test packages/course-format`. Expected: FAIL. +- [ ] **Step 4:** Implement. Use `fs.realpath` from `node:fs/promises` for the containment check (`Bun.file` has no realpath). For file reads, use `Bun.file(...).text()`. +- [ ] **Step 5:** Run `bun test packages/course-format`. Expected: all pass. Commit: `feat(course-format): markdown extraction, discovery, loading, repository`. + +### Task 5: Sample fixture course + +**Files:** +- Create: `fixtures/courses/sample/stcourse.yaml` +- Create: `fixtures/courses/sample/lessons/{01-intro.md,02-video.md,03-code.md,04-exercise.md}` +- Create: `fixtures/courses/sample/media/clip.mp4` (a 2-second, 160×90 test pattern made with ffmpeg, under 50 KB) +- Create: `fixtures/courses/sample/media/deck.pdf` (3 pages, made with pymupdf or by hand, under 20 KB) +- Create: `fixtures/courses/sample/exercises/add/{starter.js,tests.js,solution.js}` +- Create: `fixtures/courses/sample/code/S01 - First steps/hello.sh` +- Create: `fixtures/courses/sample/MENTOR.md` +- Create: `scripts/make-fixtures.sh` (regenerates the binaries) + +**Interfaces (Produces):** course id `sample`, `visibility: public`, `tracks: [{id: js, label: JavaScript}]`, and two modules: +- `m1`, with lessons `m1-intro` (text), `m1-video` (video, `file: media/clip.mp4`, `duration: 2`) and `m1-code` (text). Module `code: { dir: "code/S01 - First steps", command: "sh hello.sh" }`. +- `m2`, with lessons `m2-text` (text, containing `### Exercise 2.1 — \`add\`` and a `## Hints` section with two `
` for `Exercise 2.1 (add)`) and `m2-ex-add` (type exercise, `anchor: exercise-21--add`, `hints: { file: lessons/04-exercise.md, match: "Exercise 2.1 (add)" }`, `browser: { language: js, starter: exercises/add/starter.js, tests: exercises/add/tests.js, solution: exercises/add/solution.js }`, `localCommand: "node --test"`). + +Also `decks: [{ id: d1, file: media/deck.pdf, attachTo: m1-video }]` and `mentor: MENTOR.md`. `tests.js` exports `tests: ExerciseTest[]` checking `add(2,3) === 5` and `add(-1,1) === 0`. `starter.js` returns `a - b`, so both tests fail at the start. + +- [ ] **Step 1:** Write `scripts/make-fixtures.sh` (ffmpeg `testsrc` for 2s at 160x90 with `-pix_fmt yuv420p`; the PDF via `/home/user/.venvs/pdf/bin/python -I` with pymupdf). Run it and check that both files exist and are under their size limits. +- [ ] **Step 2:** Write the manifest and the Markdown/JS files above. +- [ ] **Step 3:** Run `bun test packages/course-format` (the Task 4 fixture test). Expected: `loads the sample fixture with no errors` passes. +- [ ] **Step 4:** Commit: `test: sample fixture course`. + +### Task 6: Reference course overlays + +**Files:** +- Create: `overlays/pure-snake/stcourse.yaml`, `overlays/systemic-design/stcourse.yaml` +- Test: `packages/course-format/test/reference.test.ts` + +**Interfaces (Produces):** course ids `pure-snake` and `systemic-design`, both `visibility: public`. + +- **Pure Snake** (`root: /home/user/functional-programming-course`): + - 12 modules, `m00`…`m11`, one per `course/NN-*.md`. + - Each module has a text lesson (`mNN-text`, the whole file) and one exercise lesson per `### Exercise N.M — …` heading: id `mNN-ex-` (for example `m03-ex-move`), with `anchor` set to the slug, `hints.match` set to `"Exercise N.M ()"` (check the exact summary text in each file), `localCommand` per track (`js`: `cd js && node --test "test/mNN-*.test.js"`, `scala`: `cd scala && ./test.sh MNN`, `fsharp`: `cd fsharp && ./test.sh MNN`), and no `browser` (spec section 13). + - `tracks`: js / JavaScript, scala / Scala 3, fsharp / F#. `mentor: MENTOR.md`. + - `method.md` and `reference.md` become text lessons in an extra module `ref`, titled "Method and reference". +- **Systemic Design** (`root: /home/user/systemic-design-course`): + - Modules `00`…`10`, one per `course/NN-*/`. + - Each module has `lesson.md` as a text lesson and `scenarios.md` as a text lesson titled "Scenarios". + - `gating: sequential`; tracks rust / Rust and ts / TypeScript; `mentor: mentor/SKILL.md`. + - `localCommand` is `node tools/start-module.mjs NN `. + +Read each module's real headings with `grep -n '^#'` before writing anchors. + +- [ ] **Step 1:** Write the failing `reference.test.ts`. If a reference root directory doesn't exist, skip with `test.skipIf`. Assertions: + - `pure-snake` loads with zero error diagnostics, has 13 modules, and includes lesson `m03-ex-move` with exactly 3 hints; + - `systemic-design` loads with zero error diagnostics, has 11 modules, and has `gating === "sequential"`. +- [ ] **Step 2:** Run `bun test packages/course-format/test/reference.test.ts`. Expected: FAIL. +- [ ] **Step 3:** Write both overlay manifests. +- [ ] **Step 4:** Run the test. Expected: PASS. Confirm `git -C /home/user/functional-programming-course status --short` and the same for systemic-design are unchanged from before the task. +- [ ] **Step 5:** Commit: `feat: overlay manifests for the two reference courses`. + +### Task 7: Database connection and migrations + +**Files:** +- Create: `packages/db/src/{connect.ts,migrate.ts,index.ts}`, `packages/db/migrations/0001_init.sql`, `scripts/test-pg.sh` +- Test: `packages/db/test/migrate.test.ts`, `packages/db/test/helpers.ts` + +**Interfaces (Produces):** +- `type Db = { sql: Bun.SQL; dialect: "sqlite" | "postgres"; close(): Promise }` +- `connect(url: string): Promise`. `sqlite://` creates parent directories and enables WAL and foreign keys. `postgres://` passes through to `new Bun.SQL(url)`. +- `migrate(db: Db): Promise`. Applies `migrations/*.sql` in name order that aren't yet in `schema_migrations(name TEXT PRIMARY KEY, applied_at TEXT)`, each in a transaction, and returns the names applied. +- `test/helpers.ts`: `forEachDb(name: string, fn: (db: Db) => Promise)`. Registers the test against a fresh in-memory SQLite, and also against Postgres when `STC_TEST_PG_URL` is set, using a fresh schema per test via `SET search_path`. +- In `contracts/server.ts`, replace the placeholder with `import type { Db } from "@stc/db"`. + +**Portable SQL rules (all migrations):** +- ids are `TEXT` (generated in the app with `Bun.randomUUIDv7()`); +- timestamps are `TEXT` in ISO-8601 UTC; +- booleans are `INTEGER` 0/1; +- JSON is `TEXT`; +- no `AUTOINCREMENT`, `SERIAL`, `RETURNING` or dialect-only functions. + +`0001_init.sql` tables (columns are `NOT NULL` unless marked `?`): + +``` +user_roles(user_id, role) PK(user_id, role) +groups(id PK, name UNIQUE) +group_members(group_id, user_id) PK(group_id, user_id) +course_authors(course_id, user_id) PK(course_id, user_id) +course_acl(course_id, principal_type ['user'|'group'], principal_id) PK(all three) +progress(user_id, course_id, lesson_id, state ['not-started'|'in-progress'|'done'], position_seconds? REAL, updated_at) PK(user_id, course_id, lesson_id) +notes(id PK, user_id, course_id, lesson_id, anchor_kind ['none'|'video'|'slide'], anchor_value? TEXT, deck_id? TEXT, body, created_at, updated_at) INDEX(user_id, course_id, lesson_id) +exercise_attempts(user_id, course_id, lesson_id, track, code, result_json?, passed INTEGER, updated_at) PK(user_id, course_id, lesson_id, track) +user_api_keys(user_id, provider, ciphertext, created_at) PK(user_id, provider) +ai_settings(scope ['server'|'user'], owner_id, mentor_off INTEGER, config_json, updated_at) PK(scope, owner_id) -- owner_id '' for server +ai_conversations(id PK, user_id, course_id, title?, created_at) +ai_messages(id PK, conversation_id, role, content, created_at) INDEX(conversation_id) +library_settings(key PK, value) -- e.g. featured_course_id +collections(id PK, title, sort INTEGER) +collection_courses(collection_id, course_id, sort INTEGER) PK(collection_id, course_id) +jobs(id PK, kind, status, progress REAL, message?, payload_json, result_json?, error?, created_at, updated_at) INDEX(status) +``` + +Auth tables belong to the auth library (Phase 1, learner package). They live in their own migration files there, `user_id` is `TEXT`, and no foreign keys point from these tables to them. + +- [ ] **Step 1:** Write `migrate.test.ts` using `forEachDb`: + - `applies 0001 once`: the first run returns `["0001_init.sql"]` and the second returns `[]`; + - `all tables exist`: insert and select one row in each table. +- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL. +- [ ] **Step 3:** Implement `connect`, `migrate` and the SQL. Write `scripts/test-pg.sh`: it runs `docker run -d --rm --name stc-pg -e POSTGRES_PASSWORD=stc -p 55432:5432 postgres:17`, waits for `pg_isready`, runs `STC_TEST_PG_URL=postgres://postgres:stc@127.0.0.1:55432/postgres bun test packages`, then stops the container. +- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS on both dialects. +- [ ] **Step 5:** Commit: `feat(db): Bun.SQL connection and portable migrations`. + +### Task 8: Data repositories + +**Files:** +- Create: `packages/db/src/repos/{progress.ts,notes.ts,attempts.ts,library.ts,jobs.ts}`, exported from `index.ts` +- Test: `packages/db/test/repos.test.ts` + +**Interfaces (Produces):** each is a factory taking `Db`. + +- `progressRepo(db)`: + - `get(userId, courseId): Promise>` + - `set(userId, courseId, lessonId, state, positionSeconds?): Promise`. An upsert written as select-then-insert-or-update inside a transaction, so it stays portable. +- `notesRepo(db)`: + - `list(userId, courseId, lessonId?): Promise`, ordered by `anchor_kind`, then by `anchor_value` read numerically, then `created_at` + - `create(input: {userId, courseId, lessonId, anchorKind, anchorValue?, deckId?, body}): Promise` + - `update(id, userId, body): Promise` (only the owner can update) + - `remove(id, userId): Promise` + - `type Note = {id, userId, courseId, lessonId, anchorKind: "none"|"video"|"slide", anchorValue: string|null, deckId: string|null, body, createdAt, updatedAt}` +- `attemptsRepo(db)`: + - `get(userId, courseId, lessonId, track): Promise` + - `save(a: Attempt): Promise` +- `libraryRepo(db)`: + - `getFeatured(): Promise`, `setFeatured(courseId|null)` + - `listCollections(): Promise<{id,title,sort,courseIds: string[]}[]>`, `saveCollection(c)`, `deleteCollection(id)` +- `jobsRepo(db)`: + - `insert(kind, payload): Promise` + - `claimNext(kinds: string[]): Promise`. Atomically moves one queued job to running: `UPDATE … WHERE id = (SELECT …) AND status='queued'`, then reads it back. + - `update(id, patch: Partial>)` + - `get(id)`, `list(kind?)` + +- [ ] **Step 1:** Write `repos.test.ts` with `forEachDb`, one test per method group: + - progress upsert keeps one row; + - notes order puts a slide note on page 2 before page 10; + - another user can't update or delete a note; + - attempts round-trip; + - collections keep their order; + - `claimNext` returns null when nothing is queued, and two concurrent `claimNext` calls never claim the same job. +- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL. +- [ ] **Step 3:** Implement the repositories. +- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS. +- [ ] **Step 5:** Commit: `feat(db): progress, notes, attempts, library, jobs repositories`. + +### Task 9: Core: config, job queue, media serving, feature composition + +**Files:** +- Create: `packages/core/src/{config.ts,jobs.ts,media.ts,app.ts,index.ts}` +- Test: `packages/core/test/{config.test.ts,jobs.test.ts,media.test.ts,app.test.ts}` + +**Interfaces (Produces):** +- `loadConfig(env = process.env): AppConfig`. Reads these variables: + - `STC_PORT`: default 3000 + - `STC_DATABASE_URL`: default `sqlite://./data/stcourses.db` + - `STC_DATA_DIR`: default `./data` + - `STC_COURSE_DIRS`: `:`-separated, default `./overlays:./courses` + - `STC_SECRET`: required when `NODE_ENV=production`; otherwise a dev default with a printed warning + - `STC_PUBLIC_URL`: default `http://localhost:` +- `createJobQueue(db: Db, handlers: Record, opts?: {pollMs?: number}): JobQueue & { start(): void; stop(): Promise }`. A single worker loop (one job at a time) that uses `jobsRepo.claimNext`. Handler errors give `status: "failed"` with the error message. On startup, jobs left `running` from a previous process are reset to `queued`. +- `serveMedia(req: Request, absPath: string): Promise`. Uses `Bun.file`. Handles `Range: bytes=a-b`, `a-` and `-n`, returning 206 with `Content-Range`, `Accept-Ranges: bytes` and `Content-Length`. Unsatisfiable ranges get 416. Content type comes from the extension (mp4, webm, pdf, png, jpg, svg, md, txt, zip, js, ts). Downloads use `Content-Disposition` with RFC 5987 `filename*` for non-ASCII names. +- `createApp(ctx: ServerContext, features: Feature[]): { routes: Record; fetch(req: Request): Promise }`, which composes: + - each feature's routes, wrapped with a `RequestContext` built by `ctx.auth.userFromRequest`. Duplicate path plus method across features throws at startup. + - core routes: + - `GET /api/health`, returning `{ ok: true, courses: n, db: dialect }`; + - `GET /api/media/:courseId/*`, which resolves via `ctx.courses.resolvePath`. A null result gives 404. A course the user cannot see gives 404 (visibility check: `canView(user, course)`). + - `canView(user: AuthUser|null, course: LoadedCourse, acl: {principal_type, principal_id}[]): boolean` is exported: + - public: anyone; + - signed-in: any user; + - restricted: owner role, a course author, or a matching ACL user or group. + +- [ ] **Step 1: Write the failing tests:** + - **config:** defaults; colon splitting; production without a secret throws. + - **jobs:** an enqueued job runs and reaches `done` with its result; a throwing handler gives `failed`; progress updates persist; a stale `running` job is requeued at start. + - **media:** the full file gives 200; `bytes=0-99` gives 206 with `Content-Range: bytes 0-99/`; `bytes=999999999-` gives 416; a filename `Æble notes.pdf` gives a valid `filename*`. + - **app:** + - health returns ok; + - a media request with `..%2F..%2Fetc%2Fpasswd` gives 404; + - a restricted course gives 404 when anonymous; + - a course with `status: "error"` doesn't break `/api/health`; + - two features registering `GET /api/x` throw. +- [ ] **Step 2:** Run `bun test packages/core`. Expected: FAIL. +- [ ] **Step 3:** Implement. +- [ ] **Step 4:** Run `bun test packages/core`. Expected: PASS. +- [ ] **Step 5:** Commit: `feat(core): config, job queue, range media serving, feature composition`. + +### Task 10: Server and web shell + +**Files:** +- Create: `apps/server/src/main.ts` +- Create: `packages/ui/src/{tokens.css,fonts.css,AppShell.tsx,primitives.tsx,index.ts}` +- Create: `apps/web/index.html`, `apps/web/src/main.tsx`, `apps/web/src/features.ts` +- Test: `apps/server/test/server.test.ts`, `packages/ui/test/primitives.test.tsx` + +**Interfaces:** +- **Consumes:** everything above. +- **Produces:** + - `apps/server/src/main.ts` exports `startServer(config: AppConfig, features: Feature[], webFeatures?: unknown): Promise<{ server: Bun.Server; stop(): Promise }>`. It connects the DB, migrates, creates the course repository and job queue, then calls `Bun.serve({ routes: { ...app.routes, "/*": indexHtml }, fetch: app.fetch })` so the SPA handles every non-API path. Run directly, it calls `startServer(loadConfig(), [])`. + - A temporary `AuthService` returns `null`; the learner package replaces it. + - `apps/web/src/features.ts` exports `webFeatures: WebFeature[]`, empty in Phase 0. Each Phase 1 package appends one import line, which is the only shared line they touch. + - `main.tsx` builds a `createBrowserRouter` from `webFeatures` routes (lazy `element`), inside ``, with a fallback home page reading "No features installed yet". + - `@stc/ui`: + - `AppShell({ header?: ReactNode, sidebar?: ReactNode, children })`; + - `Pill({ variant?: "default"|"primary"|"ghost", ...buttonProps })`; + - `SegmentedToggle({ options: {value,label}[], value, onChange, label })`; + - `ProgressBar({ value: number /*0..1*/, dim?: boolean, label: string })`, which renders `role="progressbar"`; + - `FloatingActionBar({ children })`; + - `CourseCover({ title, image?, accent? })`, a monogram fallback. + - `tokens.css` defines `--ground --side --raised --line --line2 --text --read --head --muted --faint --accent --accent-dim --sans --serif --mono`, plus `.reading` (Literata 19px / 1.75, max-width 660px) and `.work` (Plex Sans 15.5px, max-width 760px) classes. The values are listed in Global Constraints. The mockup at `docs/design/mockups/platform-v1.html` is the visual reference for the primitives. + - `fonts.css` imports `@fontsource/ibm-plex-sans/{400,500,600}.css`, `@fontsource/ibm-plex-mono/400.css` and `@fontsource/literata/{400,600,400-italic}.css`. Nothing loads from googleapis. + +- [ ] **Step 1: Write the failing tests:** + - **server.test.ts:** + - start on port 0 with `STC_COURSE_DIRS=fixtures/courses` and an in-memory SQLite; + - `GET /api/health` reports `courses: 1`; + - `GET /some/spa/path` returns HTML containing `
`; + - `GET /api/media/sample/media/clip.mp4` with `Range: bytes=0-9` gives 206. + - **primitives.test.tsx:** use `react-dom/server` `renderToString` to check that `ProgressBar` gives `aria-valuenow="40"` for value 0.4 and that `CourseCover` without an image shows the two-letter monogram. +- [ ] **Step 2:** Run `bun test apps packages/ui`. Expected: FAIL. +- [ ] **Step 3:** Implement. Add the dependencies `react`, `react-dom`, `react-router-dom@6` and the three `@fontsource` packages. +- [ ] **Step 4:** Run `bun test && bun run typecheck`. Then run `STC_COURSE_DIRS=./fixtures/courses:./overlays bun apps/server/src/main.ts & sleep 2; curl -s localhost:3000/api/health; kill %1`. Expected: `{"ok":true,"courses":3,"db":"sqlite"}`. +- [ ] **Step 5:** Commit: `feat: server entry and web shell with design tokens`. Then push `main`. Phase 0 is done. + +--- + +# Phase 1 — Four parallel work packages + +Each package runs as one agent in its own git worktree on branch `feat/`, created from `main` after Phase 0. + +Each agent starts by writing its own detailed plan, `docs/superpowers/plans/2026-10-30-.md`, in the format of this plan. The plan is built from the spec sections listed for that package and the Phase 0 interfaces. The agent then executes it with TDD and frequent commits. + +Each agent owns only its listed paths. The only shared files it may touch are: +- one appended line in `apps/web/src/features.ts`; +- one appended entry in the feature list in `apps/server/src/main.ts`; +- dependency additions to its own `package.json`. + +Each agent must not change `packages/contracts`, `packages/course-format`, `packages/db/migrations/0001_init.sql` or `packages/core`. If it needs a contract change, it writes the proposal to `docs/contract-requests/.md` and works around it locally. Phase 2 resolves these proposals. + +New database tables go in the package's own migrations folder, named `NNNN__*.sql` and registered through a `migrationsDirs` export. Phase 2 wires these into `migrate()`. + +### WP-A Learner app (`packages/feature-learner`, web routes under `/`, `/c/:courseId/*`, `/account`) + +Spec sections 4.1–4.4, 6, 7 and 13. Scope: +- **Auth:** Better Auth on Bun, with email/password, generic OIDC, social providers and SAML. If Better Auth cannot use `Bun.SQL`/`bun:sqlite` for both dialects, write a thin adapter or choose an equivalent, and record why in the package plan. The package implements `AuthService`, plus the roles and groups admin API. Groups can be filled from OIDC/SAML claims. +- **Library:** featured course, continue row, browsable rows per collection, filters, search and covers (spec section 4.2). +- **Lesson reader:** text lessons with the track switcher, Reading/Work toggle, hint tiers and "think about it first" folds; video lessons with companion decks; deck viewer with per-slide notes; notes list; code files view with a zip download; exercise pages for runner and local-only exercises; JS/TS browser runner in a Web Worker following the `ExerciseRunner` contract; progress, gating and the floating action bar. +- **Acceptance:** + - all lesson types render for `sample`, `pure-snake` and `systemic-design`; + - the `sample` exercise passes after editing; + - notes persist on SQLite and Postgres. + +### WP-B Importer (`packages/feature-import`, web routes `/admin/import`) + +Spec section 8. Scope: +- job handlers for the `ImportSource` kinds `packt-video`, `code-repo`, `pdf-deck` and `video-folder`; +- generated course folders under `/courses//` with `stcourse.yaml` (`visibility: restricted`, `source` filled in); +- cover candidates (an ffmpeg frame, the first PDF page); +- deck page images and text into `/cache/`; +- an optional transcription job that calls an external Whisper-compatible HTTP endpoint (`STC_TRANSCRIBE_URL`), skipped when it isn't set; +- the admin import UI with job progress. + +Acceptance: importing `/home/user/example_course_content` locally produces: +- a Docker course with 9 sections and the correct clip titles; +- S08 code attached to section 8; +- the serverless deck attached to a chosen lesson. + +None of this licensed content may be committed. Tests use tiny synthetic fixtures built by the package's own scripts. + +### WP-C Studio (`packages/feature-studio`, web routes `/studio/*`) + +Spec sections 9.1 and 12 (git model). Scope: +- a published worktree per course under `/worktrees//published`, plus per-author draft worktrees; +- editors for the manifest, lessons (with live preview using the learner renderer, imported from `@stc/feature-learner` only through its exported `LessonView`, or through a `docs/contract-requests` note if it isn't exported yet), deck links, attachments and exercises; +- save as a commit, publish as a merge, an explicit Sync for courses with a remote, and conflict display; +- reloading the `CourseRepository` when the published worktree changes. + +Overlay courses (whose root is an outside repo) are read-only in the studio in v1, with the message "This course is managed outside STCourses". + +Acceptance: create a course in the studio, edit a lesson, publish, and see the change as a learner. A conflicting external commit is shown, not overwritten. + +### WP-D AI (`packages/feature-ai`, web routes `/c/:courseId/assistant` panel and `/studio/generate`) + +Spec section 9.2. Scope: +- `AiProvider` implementations for Anthropic (default model `claude-opus-5-5`) and OpenAI-compatible endpoints (covers Ollama); +- server configuration (admin) and BYOK keys, encrypted with AES-GCM using a key derived from `STC_SECRET`; +- retrieval over course files, transcripts, deck text, code and the user's notes; +- an MCP client for admin-configured servers (global or per course); +- mentor mode from the manifest's `mentor` file, on by default, with a learner toggle and a visible "Mentor mode off" banner; +- a studio generator that writes drafts into a draft worktree (through the studio's exported API, or a contract request) following the house-style patterns in spec section 9.2. + +AI is off unless it is configured. Every test uses a fake `AiProvider`; no real API calls run in tests. + +Acceptance: +- with a fake provider, an assistant answer cites course files; +- mentor rules appear in the system prompt unless the learner switched them off; +- generation produces draft files, not published ones. + +--- + +# Phase 2 — Integration (sequential, on `main`) + +### Task I1: Merge and resolve contracts +- [ ] Merge `feat/learner`, then `feat/import`, `feat/studio` and `feat/ai`. After each merge run `bun install && bun test && bun run typecheck && bash scripts/test-pg.sh`, and fix failures before the next merge. +- [ ] Apply the accepted `docs/contract-requests/*.md` changes to contracts/core and update the consumers. Wire each package's `migrationsDirs` into `migrate()`: core migrations first, then packages in alphabetical order. Commit after each merge: `merge: `. + +### Task I2: Packaging and memory budget +- [ ] Add a `Dockerfile` (`oven/bun:1.3` base, `bun install --production`, `ffmpeg` and `git` installed, non-root user, `EXPOSE 3000`, volume `/data`) and a `docker-compose.yml` with an optional `postgres:17` service behind a `postgres` profile. +- [ ] Build and run with all three courses plus the locally imported example content (mounted, not copied). After 5 minutes idle, read `docker stats --no-stream`. Expected: the total of the app and Postgres is under 2 GB. Record the numbers in `docs/operations.md`. +- [ ] Commit: `build: docker image and compose`. + +### Task I3: End-to-end check and final review +- [ ] Add a `bun test` end-to-end test that starts the server against the fixtures, then signs up a user, marks a lesson done, creates a slide note, runs the sample exercise and reads progress back. Run it on both databases. +- [ ] Manually check every lesson type in a browser against the mockup on the LAN host, with a screenshot of each. +- [ ] Run a whole-branch review with a fresh reviewer on the most capable model. Fix the findings, push `main`, and write the final decision log for the user.