Add STCourses implementation plan

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
2026-10-30 02:46:28 +01:00
co-authored by Claude Opus 5.5
parent ab96addbab
commit 29ba043de7
@@ -0,0 +1,572 @@
# STCourses Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build STCourses, a self-hosted course platform. Course content is files and user data lives in SQLite or Postgres. The build is delivered as a shared contract (Phase 0), four parallel work packages (Phase 1) and an integration pass (Phase 2).
**Architecture:**
- One Bun process (`Bun.serve`) serves a React single-page app through HTML imports, plus a JSON API.
- Course folders, described by `stcourse.yaml` manifests (in the course repo or as an overlay), are loaded into an in-memory course model.
- User data goes through one `Bun.SQL` data layer with plain-SQL migrations that run unchanged on SQLite and Postgres.
- Features (learner, import, studio, AI) are packages that export a `Feature` (routes plus job handlers) and a set of web routes. They are composed by the server and the web shell.
**Tech Stack:**
- Bun 1.3.x and TypeScript (strict).
- React 18 and React Router 6 (library mode).
- zod for validation, `Bun.YAML` for manifests, `Bun.SQL` for databases.
- @fontsource packages for IBM Plex Sans, IBM Plex Mono and Literata.
- `bun test` for tests.
**Spec:** `docs/superpowers/specs/2026-10-30-stcourses-design.md` (sections 1–13). Visual reference: `docs/design/mockups/platform-v1.html`.
## Global Constraints
These apply project-wide; every task inherits them.
- **Bun conventions** (from `bun init`'s CLAUDE.md):
- use `Bun.serve` routes and WebSockets, `bun:sqlite` or `Bun.sql`, `Bun.file`, `` Bun.$ ``, `bun test`, `bun install`, `bunx`;
- never use express, vite, dotenv, pg, postgres.js, better-sqlite3, jest or vitest.
- **Frontend:** a React SPA served via HTML imports. No Next.js. Plain CSS with custom properties; no Tailwind.
- **Course content is never stored in the database.** Deleting the DB loses user data only.
- **Database URL:** `STC_DATABASE_URL`, either `sqlite://<path>` (default `sqlite://./data/stcourses.db`) or `postgres://…`. Every migration and query must run on both.
- **Idle memory:** at most 2 GB for the deployment (checked in Phase 2).
- **License:** AGPL-3.0-only. Every package's `package.json` has `"license": "AGPL-3.0-only"`.
- **Never write into `/home/user/functional-programming-course` or `/home/user/systemic-design-course`.** Their manifests are overlays in `overlays/`.
- **Never commit anything from `/home/user/example_course_content`.** It is licensed material, used only for local test runs.
- **Design tokens** (spec section 4.1):
- ground `#12141a`, side `#181b22`, raised `#232733`, line `#262a33`, line2 `#343947`;
- text `#d9dce3`, reading text `#d9d4c8`, heading `#f2efe8`, muted `#a4abb8`, faint `#8c93a0`;
- accent `#f1b44c`, accent-dim `#8c7440`;
- fonts: IBM Plex Sans (UI), Literata (lesson titles and reading text, 19px / 1.75 in a 660px column), IBM Plex Mono (code);
- dark theme only in v1.
- **Keyboard:** arrow keys in the slide viewer, Escape closes overlays, normal Tab focus. No other shortcuts.
- **Commits:** author is the git identity configured on the machine, and every message ends with `Co-Authored-By: Claude Opus 5.5 <[email protected]>`.
## Review Focus
1. **Paths with spaces and non-ASCII characters** (for example `S08 - Containerizing Applications/`, `Thinknyx_Mastering_Docker_Essentials_Handson_DevOps.pdf`) must load, serve and download correctly. Tests: Task 4 (loader) and Task 9 (media route).
2. **An overlay whose `root` repo is missing or moved** must give a course with `status: "error"` and a readable diagnostic. The server must keep serving every other course. Tests: Task 4 and Task 9.
3. **A manifest that references a missing file, heading anchor or deck id** must give a diagnostic naming the lesson and the path, not throw. Test: Task 4.
4. **Video seeking:** `Range: bytes=…` requests get `206` with correct `Content-Range`. An unsatisfiable range gets `416`. Test: Task 9.
5. **Media paths with `..`, absolute paths or symlinks that escape the course root** must get `404` and never read outside the root. Test: Task 9.
---
## Repository layout (locked in by Phase 0)
```
package.json # bun workspaces: ["packages/*", "apps/*"]
tsconfig.json CLAUDE.md LICENSE README.md .gitignore
packages/
contracts/ # shared TS types and interfaces only (no runtime deps)
src/index.ts src/server.ts src/exercise.ts src/ai.ts src/import.ts src/web.ts
course-format/ # manifest schema, markdown extraction, discovery, CourseRepository
src/schema.ts src/markdown.ts src/load.ts src/repository.ts src/index.ts test/
db/ # Bun.SQL connection, migrations, repositories
src/connect.ts src/migrate.ts migrations/0001_init.sql src/repos/*.ts src/index.ts test/
core/ # config, job queue, feature composition, media serving
src/config.ts src/jobs.ts src/media.ts src/app.ts src/index.ts test/
ui/ # tokens.css, fonts, shared React primitives (AppShell, Pill, ProgressBar)
src/tokens.css src/fonts.css src/AppShell.tsx src/primitives.tsx src/index.ts
feature-learner/ feature-import/ feature-studio/ feature-ai/ # Phase 1 (one per work package)
apps/
server/ # entry: src/main.ts — builds ServerContext, composes features, Bun.serve
web/ # index.html, src/main.tsx (router built from feature route registries)
overlays/
pure-snake/stcourse.yaml # root: /home/user/functional-programming-course
systemic-design/stcourse.yaml # root: /home/user/systemic-design-course
fixtures/courses/sample/ # tiny self-contained course used by all tests
scripts/test-pg.sh # starts postgres:17 in docker, runs DB tests with STC_TEST_PG_URL
```
---
# Phase 0 — Contract (sequential, on `main`)
### Task 1: Workspace scaffold
**Files:**
- Create: `package.json`, `tsconfig.json`, `CLAUDE.md`, `LICENSE`, `README.md`.
- Create: one `package.json` per package and app in the layout above (`name: "@stc/<dir>"`, `type: "module"`, `license: "AGPL-3.0-only"`).
- Modify: `.gitignore`, adding `data/` and `.cache/`.
**Interfaces:**
- Produces: workspace package names `@stc/contracts`, `@stc/course-format`, `@stc/db`, `@stc/core`, `@stc/ui`, `@stc/server`, `@stc/web`, `@stc/feature-learner`, `@stc/feature-import`, `@stc/feature-studio`, `@stc/feature-ai`.
- [ ] **Step 1:** In a scratch directory, run `bun init -y` and copy its generated `CLAUDE.md` text into the root `CLAUDE.md`. Then add a "Project rules" section that repeats the Global Constraints above and points to the spec and plan.
- [ ] **Step 2:** Write the root `package.json` with `"workspaces": ["packages/*","apps/*"]` and these scripts:
- `test`: `bun test`
- `typecheck`: `bunx tsc --noEmit -p .`
- `dev`: `bun --hot apps/server/src/main.ts`
Write the root `tsconfig.json` with `strict: true`, `jsx: "react-jsx"`, `moduleResolution: "bundler"`, `types: ["bun-types"]`, and `paths` mapping `@stc/*` to `packages/*/src`.
- [ ] **Step 3:** Write the full AGPL-3.0 text to `LICENSE`. Write `README.md` with one paragraph from spec section 1 and the dev commands.
- [ ] **Step 4:** Add `packages/contracts/test/smoke.test.ts` asserting `expect(1).toBe(1)`. Run `bun install && bun test && bun run typecheck`. Expected: 1 pass, no type errors.
- [ ] **Step 5:** Commit: `chore: bun workspace scaffold`.
### Task 2: Contracts package (types only)
**Files:**
- Create: `packages/contracts/src/{index.ts,server.ts,exercise.ts,ai.ts,import.ts,web.ts}`
- Test: `packages/contracts/test/types.test.ts`
**Interfaces (Produces). This exact code is the contract; copy it verbatim:**
```ts
// server.ts
export type Role = "owner" | "author" | "learner";
export interface AuthUser { id: string; email: string; name: string; roles: Role[]; groups: string[] }
export interface RequestContext { user: AuthUser | null; params: Record<string, string>; url: URL }
export type RouteHandler = (req: Request, rc: RequestContext) => Response | Promise<Response>;
export interface Feature {
name: string; // "learner" | "import" | "studio" | "ai" | "core"
routes: Record<string, Partial<Record<"GET"|"POST"|"PUT"|"PATCH"|"DELETE", RouteHandler>>>; // paths start with /api/
jobs?: Record<string, JobHandler>; // job kind -> handler
}
export interface JobContext { reportProgress(fraction: number, message?: string): Promise<void>; signal: AbortSignal }
export type JobHandler = (payload: unknown, jc: JobContext) => Promise<unknown>;
export type JobStatus = "queued" | "running" | "done" | "failed";
export interface JobRecord { id: string; kind: string; status: JobStatus; progress: number; message: string | null; payload: unknown; result: unknown; error: string | null; createdAt: string; updatedAt: string }
export interface JobQueue { enqueue(kind: string, payload: unknown): Promise<string>; get(id: string): Promise<JobRecord | null>; list(kind?: string): Promise<JobRecord[]> }
export interface AuthService { userFromRequest(req: Request): Promise<AuthUser | null> }
export interface ServerContext {
config: AppConfig; db: import("@stc/db").Db; courses: import("@stc/course-format").CourseRepository;
jobs: JobQueue; auth: AuthService;
}
export interface AppConfig {
port: number; databaseUrl: string; dataDir: string; // dataDir holds cache/, media/, worktrees/
courseDirs: string[]; // folders scanned for stcourse.yaml (course repos and overlays/)
secret: string; // STC_SECRET, used to encrypt BYOK keys
publicUrl: string;
}
// exercise.ts — browser-runnable exercises (spec 4.3 type 6)
export type ExerciseLanguage = "js" | "ts";
export interface Assert { equal(actual: unknown, expected: unknown, message?: string): void; deepEqual(actual: unknown, expected: unknown, message?: string): void; ok(value: unknown, message?: string): void }
export interface ExerciseTest { name: string; run(mod: Record<string, unknown>, assert: Assert): void | Promise<void> }
// A tests file is an ES module: `export const tests: ExerciseTest[]`. Learner code is an ES module.
export interface TestResult { name: string; passed: boolean; message?: string; expected?: string; actual?: string }
export interface ExerciseRunner { language: ExerciseLanguage; run(learnerCode: string, testsModule: string, timeoutMs: number): Promise<TestResult[]> }
// ai.ts
export interface ChatMessage { role: "system" | "user" | "assistant" | "tool"; content: string; toolCallId?: string; name?: string }
export interface ToolSpec { name: string; description: string; inputSchema: Record<string, unknown> }
export interface AiProvider { id: string; chat(messages: ChatMessage[], opts: { tools?: ToolSpec[]; signal?: AbortSignal }): AsyncIterable<{ text?: string; toolCall?: { id: string; name: string; input: unknown } }> }
// import.ts
export interface ImportSource { kind: "packt-video" | "pdf-deck" | "code-repo" | "video-folder"; path: string; options?: Record<string, unknown> }
export interface ImportResult { courseId: string; manifestPath: string; warnings: string[] }
// web.ts
export interface WebRoute { path: string; element: () => Promise<{ default: React.ComponentType }>; nav?: { label: string; area: "header" | "admin" } }
export interface WebFeature { name: string; routes: WebRoute[] }
```
- [ ] **Step 1:** Write `types.test.ts`. It builds one object literal of each interface above, typed by the interface (so it fails to compile if a field is wrong), and runs `expect(typeof feature.routes).toBe("object")`.
- [ ] **Step 2:** Run `bun run typecheck`. Expected: errors, because the modules don't exist yet.
- [ ] **Step 3:** Create the files with the code above. `index.ts` re-exports everything. `web.ts` uses `import type React from "react"`.
- [ ] **Step 4:** Run `bun test packages/contracts && bun run typecheck`. Expected: PASS. (The `@stc/db` and `@stc/course-format` type imports resolve once Tasks 4 and 7 exist. Until then, temporarily declare `Db` and `CourseRepository` as `unknown` placeholders in `server.ts`, and switch them in Task 7.)
- [ ] **Step 5:** Commit: `feat(contracts): shared server, exercise, ai, import and web interfaces`.
### Task 3: Manifest schema
**Files:**
- Create: `packages/course-format/src/schema.ts`
- Test: `packages/course-format/test/schema.test.ts`
**Interfaces (Produces):**
- `parseManifest(yamlText: string): { ok: true; manifest: Manifest } | { ok: false; errors: string[] }`. It uses `Bun.YAML.parse`, then the zod `ManifestSchema`. Each error string reads `"<dotted.path>: <message>"`.
- `type Manifest = z.infer<typeof ManifestSchema>`, plus the exported types `Module`, `Lesson`, `Deck`, `Attachment`, `Track`.
Schema (fields are required unless marked `?`):
```yaml
schema: 1 # literal 1
id: string # /^[a-z0-9][a-z0-9-]*$/
title: string
pitch?: string
authors?: string[]
root?: string # overlay only: absolute path, or path relative to this manifest, of the course repo; default "."
cover?: string # path relative to root
accent?: string # /^#[0-9a-fA-F]{6}$/
collections?: string[]
visibility: public | signed-in | restricted # default restricted
gating?: none | sequential # default none
tracks?: [{ id: string, label: string }]
mentor?: string # path to mentor instructions file
source?: { kind: authored | packt-video | pdf-deck | code-repo | video-folder, origin?: string, importedAt?: string }
decks?: [{ id: string, file: string, title?: string, attachTo: string }] # attachTo = a lesson id or module id
attachments?: [{ file: string, title?: string, module?: string }]
modules:
- id: string
title: string
code?: { dir: string, command?: string } # "Files for this section"
lessons:
- id: string # unique across the course
title: string
type: text | video | exercise
file?: string # required for text and video
anchor?: string # heading slug inside file
duration?: number # seconds
transcript?: string # video only
tracks?: { <trackId>: { file?: string, command?: string } } # per-track file shown or command to run
hints?: { file: string, match: string } # exercise: <details> blocks whose summary starts with match
browser?: { language: js | ts, starter: string, tests: string, solution?: string } # exercise runnable in browser
localCommand?: string | { <trackId>: string }
```
- [ ] **Step 1: Write the failing tests:**
- `parses a minimal manifest`: id `x`, one module with one text lesson. Expect `visibility` to default to `"restricted"` and `gating` to `"none"`.
- `rejects a bad accent`: `accent: "orange"`. Expect an error containing `accent`.
- `rejects duplicate lesson ids across modules`. Expect an error containing the duplicate id.
- `requires file for text and video lessons`.
- `rejects a deck whose attachTo is not a lesson or module id`.
- `rejects schema: 2`.
- [ ] **Step 2:** Run `bun test packages/course-format/test/schema.test.ts`. Expected: FAIL (module not found).
- [ ] **Step 3:** Implement `ManifestSchema`. Cross-field checks (duplicate ids, `attachTo`, `file` required by type) go in a zod `.superRefine`. Add the zod dependency to `@stc/course-format`.
- [ ] **Step 4:** Run the same command. Expected: 6 pass.
- [ ] **Step 5:** Commit: `feat(course-format): stcourse.yaml schema`.
### Task 4: Markdown extraction, discovery and loading
**Files:**
- Create: `packages/course-format/src/{markdown.ts,load.ts,repository.ts,index.ts}`
- Test: `packages/course-format/test/{markdown.test.ts,load.test.ts}`
**Interfaces (Produces):**
- `slugify(heading: string): string`. GitHub style: lowercase, strip markdown and backticks, drop punctuation except `-`, turn spaces into `-`. `"Exercise 3.1 — `move`"` becomes `"exercise-31--move"`. Copy GitHub's exact algorithm (em dash removed, so the two spaces leave a double hyphen).
- `extractSection(markdown: string, anchor: string): string | null`. Returns the heading line plus everything up to the next heading of the same or a higher level.
- `extractHints(markdown: string, match: string): { label: string; body: string }[]`. Returns `<details><summary>…</summary>…</details>` blocks whose summary text starts with `match`, in document order. `label` is the summary text after `match`, with leading `" — "`, `" - "` or `" ("` trimmed.
- `type Diagnostic = { level: "error" | "warning"; courseId?: string; lessonId?: string; path?: string; message: string }`
- `interface LoadedCourse { id: string; manifestPath: string; root: string; manifest: Manifest | null; status: "ok" | "error"; diagnostics: Diagnostic[]; lessonIndex: Map<string, { moduleId: string; lesson: Lesson }> }`
- `discoverManifests(courseDirs: string[]): Promise<string[]>`. Finds `stcourse.yaml` at depth 0–2 under each dir. Duplicate ids are resolved in `loadAll`.
- `loadCourse(manifestPath: string): Promise<LoadedCourse>`. Resolves `root`, then checks that every referenced file exists under root, every `anchor` resolves through `extractSection`, and every `hints.match` finds at least one block. Each failure becomes a diagnostic; nothing throws. A missing root gives `status: "error"`.
- `loadAll(courseDirs: string[]): Promise<LoadedCourse[]>`. When two manifests share an id, the one whose `root` points outside its own directory (the overlay) wins, and a warning diagnostic records the shadowed path (spec section 13).
- `interface CourseRepository { list(): LoadedCourse[]; get(id: string): LoadedCourse | null; reload(id?: string): Promise<void>; readLessonMarkdown(courseId: string, lessonId: string): Promise<string | null>; resolvePath(courseId: string, relPath: string): string | null }`
- `resolvePath` returns an absolute path only if the real path (following symlinks) is inside the course root. Otherwise it returns `null`.
- `createCourseRepository(courseDirs: string[]): Promise<CourseRepository>`.
- [ ] **Step 1: Write the failing tests in `markdown.test.ts`:**
- `slugify` of `"Exercise 3.1 — \`move\`"` equals `"exercise-31--move"`, and `"Why this module"` equals `"why-this-module"`.
- `extractSection` on a 3-heading sample returns only the middle `###` section.
- `extractHints` on a sample containing `<summary>Exercise 3.1 (move) — nudge</summary>` and `<summary>Exercise 3.1 (move) — stronger hint</summary>`, with match `"Exercise 3.1 (move)"`, returns labels `["nudge", "stronger hint"]`.
- [ ] **Step 2: Write the failing tests in `load.test.ts`**, using temp dirs built with `Bun.write`:
- `loads the sample fixture with no errors` (Task 5 fixture).
- `overlay with missing root gives status error`: diagnostic message contains the root path, and `loadAll` still returns the other course.
- `missing lesson file, anchor and hints each produce one error diagnostic naming lessonId`.
- `handles spaces and unicode in paths`: lesson file `S08 - Containerizing Applications/Æble notes.md`.
- `overlay wins over in-repo manifest with same id` (spec section 13).
- `resolvePath rejects ../ and symlink escape`: create a symlink inside root pointing at `/etc`, and expect `null`.
- [ ] **Step 3:** Run `bun test packages/course-format`. Expected: FAIL.
- [ ] **Step 4:** Implement. Use `fs.realpath` from `node:fs/promises` for the containment check (`Bun.file` has no realpath). For file reads, use `Bun.file(...).text()`.
- [ ] **Step 5:** Run `bun test packages/course-format`. Expected: all pass. Commit: `feat(course-format): markdown extraction, discovery, loading, repository`.
### Task 5: Sample fixture course
**Files:**
- Create: `fixtures/courses/sample/stcourse.yaml`
- Create: `fixtures/courses/sample/lessons/{01-intro.md,02-video.md,03-code.md,04-exercise.md}`
- Create: `fixtures/courses/sample/media/clip.mp4` (a 2-second, 160×90 test pattern made with ffmpeg, under 50 KB)
- Create: `fixtures/courses/sample/media/deck.pdf` (3 pages, made with pymupdf or by hand, under 20 KB)
- Create: `fixtures/courses/sample/exercises/add/{starter.js,tests.js,solution.js}`
- Create: `fixtures/courses/sample/code/S01 - First steps/hello.sh`
- Create: `fixtures/courses/sample/MENTOR.md`
- Create: `scripts/make-fixtures.sh` (regenerates the binaries)
**Interfaces (Produces):** course id `sample`, `visibility: public`, `tracks: [{id: js, label: JavaScript}]`, and two modules:
- `m1`, with lessons `m1-intro` (text), `m1-video` (video, `file: media/clip.mp4`, `duration: 2`) and `m1-code` (text). Module `code: { dir: "code/S01 - First steps", command: "sh hello.sh" }`.
- `m2`, with lessons `m2-text` (text, containing `### Exercise 2.1 — \`add\`` and a `## Hints` section with two `<details>` for `Exercise 2.1 (add)`) and `m2-ex-add` (type exercise, `anchor: exercise-21--add`, `hints: { file: lessons/04-exercise.md, match: "Exercise 2.1 (add)" }`, `browser: { language: js, starter: exercises/add/starter.js, tests: exercises/add/tests.js, solution: exercises/add/solution.js }`, `localCommand: "node --test"`).
Also `decks: [{ id: d1, file: media/deck.pdf, attachTo: m1-video }]` and `mentor: MENTOR.md`. `tests.js` exports `tests: ExerciseTest[]` checking `add(2,3) === 5` and `add(-1,1) === 0`. `starter.js` returns `a - b`, so both tests fail at the start.
- [ ] **Step 1:** Write `scripts/make-fixtures.sh` (ffmpeg `testsrc` for 2s at 160x90 with `-pix_fmt yuv420p`; the PDF via `/home/user/.venvs/pdf/bin/python -I` with pymupdf). Run it and check that both files exist and are under their size limits.
- [ ] **Step 2:** Write the manifest and the Markdown/JS files above.
- [ ] **Step 3:** Run `bun test packages/course-format` (the Task 4 fixture test). Expected: `loads the sample fixture with no errors` passes.
- [ ] **Step 4:** Commit: `test: sample fixture course`.
### Task 6: Reference course overlays
**Files:**
- Create: `overlays/pure-snake/stcourse.yaml`, `overlays/systemic-design/stcourse.yaml`
- Test: `packages/course-format/test/reference.test.ts`
**Interfaces (Produces):** course ids `pure-snake` and `systemic-design`, both `visibility: public`.
- **Pure Snake** (`root: /home/user/functional-programming-course`):
- 12 modules, `m00`…`m11`, one per `course/NN-*.md`.
- Each module has a text lesson (`mNN-text`, the whole file) and one exercise lesson per `### Exercise N.M — …` heading: id `mNN-ex-<name>` (for example `m03-ex-move`), with `anchor` set to the slug, `hints.match` set to `"Exercise N.M (<name>)"` (check the exact summary text in each file), `localCommand` per track (`js`: `cd js && node --test "test/mNN-*.test.js"`, `scala`: `cd scala && ./test.sh MNN`, `fsharp`: `cd fsharp && ./test.sh MNN`), and no `browser` (spec section 13).
- `tracks`: js / JavaScript, scala / Scala 3, fsharp / F#. `mentor: MENTOR.md`.
- `method.md` and `reference.md` become text lessons in an extra module `ref`, titled "Method and reference".
- **Systemic Design** (`root: /home/user/systemic-design-course`):
- Modules `00`…`10`, one per `course/NN-*/`.
- Each module has `lesson.md` as a text lesson and `scenarios.md` as a text lesson titled "Scenarios".
- `gating: sequential`; tracks rust / Rust and ts / TypeScript; `mentor: mentor/SKILL.md`.
- `localCommand` is `node tools/start-module.mjs NN <your-workspace>`.
Read each module's real headings with `grep -n '^#'` before writing anchors.
- [ ] **Step 1:** Write the failing `reference.test.ts`. If a reference root directory doesn't exist, skip with `test.skipIf`. Assertions:
- `pure-snake` loads with zero error diagnostics, has 13 modules, and includes lesson `m03-ex-move` with exactly 3 hints;
- `systemic-design` loads with zero error diagnostics, has 11 modules, and has `gating === "sequential"`.
- [ ] **Step 2:** Run `bun test packages/course-format/test/reference.test.ts`. Expected: FAIL.
- [ ] **Step 3:** Write both overlay manifests.
- [ ] **Step 4:** Run the test. Expected: PASS. Confirm `git -C /home/user/functional-programming-course status --short` and the same for systemic-design are unchanged from before the task.
- [ ] **Step 5:** Commit: `feat: overlay manifests for the two reference courses`.
### Task 7: Database connection and migrations
**Files:**
- Create: `packages/db/src/{connect.ts,migrate.ts,index.ts}`, `packages/db/migrations/0001_init.sql`, `scripts/test-pg.sh`
- Test: `packages/db/test/migrate.test.ts`, `packages/db/test/helpers.ts`
**Interfaces (Produces):**
- `type Db = { sql: Bun.SQL; dialect: "sqlite" | "postgres"; close(): Promise<void> }`
- `connect(url: string): Promise<Db>`. `sqlite://` creates parent directories and enables WAL and foreign keys. `postgres://` passes through to `new Bun.SQL(url)`.
- `migrate(db: Db): Promise<string[]>`. Applies `migrations/*.sql` in name order that aren't yet in `schema_migrations(name TEXT PRIMARY KEY, applied_at TEXT)`, each in a transaction, and returns the names applied.
- `test/helpers.ts`: `forEachDb(name: string, fn: (db: Db) => Promise<void>)`. Registers the test against a fresh in-memory SQLite, and also against Postgres when `STC_TEST_PG_URL` is set, using a fresh schema per test via `SET search_path`.
- In `contracts/server.ts`, replace the placeholder with `import type { Db } from "@stc/db"`.
**Portable SQL rules (all migrations):**
- ids are `TEXT` (generated in the app with `Bun.randomUUIDv7()`);
- timestamps are `TEXT` in ISO-8601 UTC;
- booleans are `INTEGER` 0/1;
- JSON is `TEXT`;
- no `AUTOINCREMENT`, `SERIAL`, `RETURNING` or dialect-only functions.
`0001_init.sql` tables (columns are `NOT NULL` unless marked `?`):
```
user_roles(user_id, role) PK(user_id, role)
groups(id PK, name UNIQUE)
group_members(group_id, user_id) PK(group_id, user_id)
course_authors(course_id, user_id) PK(course_id, user_id)
course_acl(course_id, principal_type ['user'|'group'], principal_id) PK(all three)
progress(user_id, course_id, lesson_id, state ['not-started'|'in-progress'|'done'], position_seconds? REAL, updated_at) PK(user_id, course_id, lesson_id)
notes(id PK, user_id, course_id, lesson_id, anchor_kind ['none'|'video'|'slide'], anchor_value? TEXT, deck_id? TEXT, body, created_at, updated_at) INDEX(user_id, course_id, lesson_id)
exercise_attempts(user_id, course_id, lesson_id, track, code, result_json?, passed INTEGER, updated_at) PK(user_id, course_id, lesson_id, track)
user_api_keys(user_id, provider, ciphertext, created_at) PK(user_id, provider)
ai_settings(scope ['server'|'user'], owner_id, mentor_off INTEGER, config_json, updated_at) PK(scope, owner_id) -- owner_id '' for server
ai_conversations(id PK, user_id, course_id, title?, created_at)
ai_messages(id PK, conversation_id, role, content, created_at) INDEX(conversation_id)
library_settings(key PK, value) -- e.g. featured_course_id
collections(id PK, title, sort INTEGER)
collection_courses(collection_id, course_id, sort INTEGER) PK(collection_id, course_id)
jobs(id PK, kind, status, progress REAL, message?, payload_json, result_json?, error?, created_at, updated_at) INDEX(status)
```
Auth tables belong to the auth library (Phase 1, learner package). They live in their own migration files there, `user_id` is `TEXT`, and no foreign keys point from these tables to them.
- [ ] **Step 1:** Write `migrate.test.ts` using `forEachDb`:
- `applies 0001 once`: the first run returns `["0001_init.sql"]` and the second returns `[]`;
- `all tables exist`: insert and select one row in each table.
- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL.
- [ ] **Step 3:** Implement `connect`, `migrate` and the SQL. Write `scripts/test-pg.sh`: it runs `docker run -d --rm --name stc-pg -e POSTGRES_PASSWORD=stc -p 55432:5432 postgres:17`, waits for `pg_isready`, runs `STC_TEST_PG_URL=postgres://postgres:[email protected]:55432/postgres bun test packages`, then stops the container.
- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS on both dialects.
- [ ] **Step 5:** Commit: `feat(db): Bun.SQL connection and portable migrations`.
### Task 8: Data repositories
**Files:**
- Create: `packages/db/src/repos/{progress.ts,notes.ts,attempts.ts,library.ts,jobs.ts}`, exported from `index.ts`
- Test: `packages/db/test/repos.test.ts`
**Interfaces (Produces):** each is a factory taking `Db`.
- `progressRepo(db)`:
- `get(userId, courseId): Promise<Record<lessonId, {state, positionSeconds: number|null, updatedAt}>>`
- `set(userId, courseId, lessonId, state, positionSeconds?): Promise<void>`. An upsert written as select-then-insert-or-update inside a transaction, so it stays portable.
- `notesRepo(db)`:
- `list(userId, courseId, lessonId?): Promise<Note[]>`, ordered by `anchor_kind`, then by `anchor_value` read numerically, then `created_at`
- `create(input: {userId, courseId, lessonId, anchorKind, anchorValue?, deckId?, body}): Promise<Note>`
- `update(id, userId, body): Promise<Note|null>` (only the owner can update)
- `remove(id, userId): Promise<boolean>`
- `type Note = {id, userId, courseId, lessonId, anchorKind: "none"|"video"|"slide", anchorValue: string|null, deckId: string|null, body, createdAt, updatedAt}`
- `attemptsRepo(db)`:
- `get(userId, courseId, lessonId, track): Promise<Attempt|null>`
- `save(a: Attempt): Promise<void>`
- `libraryRepo(db)`:
- `getFeatured(): Promise<string|null>`, `setFeatured(courseId|null)`
- `listCollections(): Promise<{id,title,sort,courseIds: string[]}[]>`, `saveCollection(c)`, `deleteCollection(id)`
- `jobsRepo(db)`:
- `insert(kind, payload): Promise<string>`
- `claimNext(kinds: string[]): Promise<JobRecord|null>`. Atomically moves one queued job to running: `UPDATE … WHERE id = (SELECT …) AND status='queued'`, then reads it back.
- `update(id, patch: Partial<Pick<JobRecord,"status"|"progress"|"message"|"result"|"error">>)`
- `get(id)`, `list(kind?)`
- [ ] **Step 1:** Write `repos.test.ts` with `forEachDb`, one test per method group:
- progress upsert keeps one row;
- notes order puts a slide note on page 2 before page 10;
- another user can't update or delete a note;
- attempts round-trip;
- collections keep their order;
- `claimNext` returns null when nothing is queued, and two concurrent `claimNext` calls never claim the same job.
- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL.
- [ ] **Step 3:** Implement the repositories.
- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS.
- [ ] **Step 5:** Commit: `feat(db): progress, notes, attempts, library, jobs repositories`.
### Task 9: Core: config, job queue, media serving, feature composition
**Files:**
- Create: `packages/core/src/{config.ts,jobs.ts,media.ts,app.ts,index.ts}`
- Test: `packages/core/test/{config.test.ts,jobs.test.ts,media.test.ts,app.test.ts}`
**Interfaces (Produces):**
- `loadConfig(env = process.env): AppConfig`. Reads these variables:
- `STC_PORT`: default 3000
- `STC_DATABASE_URL`: default `sqlite://./data/stcourses.db`
- `STC_DATA_DIR`: default `./data`
- `STC_COURSE_DIRS`: `:`-separated, default `./overlays:./courses`
- `STC_SECRET`: required when `NODE_ENV=production`; otherwise a dev default with a printed warning
- `STC_PUBLIC_URL`: default `http://localhost:<port>`
- `createJobQueue(db: Db, handlers: Record<string, JobHandler>, opts?: {pollMs?: number}): JobQueue & { start(): void; stop(): Promise<void> }`. A single worker loop (one job at a time) that uses `jobsRepo.claimNext`. Handler errors give `status: "failed"` with the error message. On startup, jobs left `running` from a previous process are reset to `queued`.
- `serveMedia(req: Request, absPath: string): Promise<Response>`. Uses `Bun.file`. Handles `Range: bytes=a-b`, `a-` and `-n`, returning 206 with `Content-Range`, `Accept-Ranges: bytes` and `Content-Length`. Unsatisfiable ranges get 416. Content type comes from the extension (mp4, webm, pdf, png, jpg, svg, md, txt, zip, js, ts). Downloads use `Content-Disposition` with RFC 5987 `filename*` for non-ASCII names.
- `createApp(ctx: ServerContext, features: Feature[]): { routes: Record<string, …>; fetch(req: Request): Promise<Response> }`, which composes:
- each feature's routes, wrapped with a `RequestContext` built by `ctx.auth.userFromRequest`. Duplicate path plus method across features throws at startup.
- core routes:
- `GET /api/health`, returning `{ ok: true, courses: n, db: dialect }`;
- `GET /api/media/:courseId/*`, which resolves via `ctx.courses.resolvePath`. A null result gives 404. A course the user cannot see gives 404 (visibility check: `canView(user, course)`).
- `canView(user: AuthUser|null, course: LoadedCourse, acl: {principal_type, principal_id}[]): boolean` is exported:
- public: anyone;
- signed-in: any user;
- restricted: owner role, a course author, or a matching ACL user or group.
- [ ] **Step 1: Write the failing tests:**
- **config:** defaults; colon splitting; production without a secret throws.
- **jobs:** an enqueued job runs and reaches `done` with its result; a throwing handler gives `failed`; progress updates persist; a stale `running` job is requeued at start.
- **media:** the full file gives 200; `bytes=0-99` gives 206 with `Content-Range: bytes 0-99/<size>`; `bytes=999999999-` gives 416; a filename `Æble notes.pdf` gives a valid `filename*`.
- **app:**
- health returns ok;
- a media request with `..%2F..%2Fetc%2Fpasswd` gives 404;
- a restricted course gives 404 when anonymous;
- a course with `status: "error"` doesn't break `/api/health`;
- two features registering `GET /api/x` throw.
- [ ] **Step 2:** Run `bun test packages/core`. Expected: FAIL.
- [ ] **Step 3:** Implement.
- [ ] **Step 4:** Run `bun test packages/core`. Expected: PASS.
- [ ] **Step 5:** Commit: `feat(core): config, job queue, range media serving, feature composition`.
### Task 10: Server and web shell
**Files:**
- Create: `apps/server/src/main.ts`
- Create: `packages/ui/src/{tokens.css,fonts.css,AppShell.tsx,primitives.tsx,index.ts}`
- Create: `apps/web/index.html`, `apps/web/src/main.tsx`, `apps/web/src/features.ts`
- Test: `apps/server/test/server.test.ts`, `packages/ui/test/primitives.test.tsx`
**Interfaces:**
- **Consumes:** everything above.
- **Produces:**
- `apps/server/src/main.ts` exports `startServer(config: AppConfig, features: Feature[], webFeatures?: unknown): Promise<{ server: Bun.Server; stop(): Promise<void> }>`. It connects the DB, migrates, creates the course repository and job queue, then calls `Bun.serve({ routes: { ...app.routes, "/*": indexHtml }, fetch: app.fetch })` so the SPA handles every non-API path. Run directly, it calls `startServer(loadConfig(), [])`.
- A temporary `AuthService` returns `null`; the learner package replaces it.
- `apps/web/src/features.ts` exports `webFeatures: WebFeature[]`, empty in Phase 0. Each Phase 1 package appends one import line, which is the only shared line they touch.
- `main.tsx` builds a `createBrowserRouter` from `webFeatures` routes (lazy `element`), inside `<AppShell>`, with a fallback home page reading "No features installed yet".
- `@stc/ui`:
- `AppShell({ header?: ReactNode, sidebar?: ReactNode, children })`;
- `Pill({ variant?: "default"|"primary"|"ghost", ...buttonProps })`;
- `SegmentedToggle({ options: {value,label}[], value, onChange, label })`;
- `ProgressBar({ value: number /*0..1*/, dim?: boolean, label: string })`, which renders `role="progressbar"`;
- `FloatingActionBar({ children })`;
- `CourseCover({ title, image?, accent? })`, a monogram fallback.
- `tokens.css` defines `--ground --side --raised --line --line2 --text --read --head --muted --faint --accent --accent-dim --sans --serif --mono`, plus `.reading` (Literata 19px / 1.75, max-width 660px) and `.work` (Plex Sans 15.5px, max-width 760px) classes. The values are listed in Global Constraints. The mockup at `docs/design/mockups/platform-v1.html` is the visual reference for the primitives.
- `fonts.css` imports `@fontsource/ibm-plex-sans/{400,500,600}.css`, `@fontsource/ibm-plex-mono/400.css` and `@fontsource/literata/{400,600,400-italic}.css`. Nothing loads from googleapis.
- [ ] **Step 1: Write the failing tests:**
- **server.test.ts:**
- start on port 0 with `STC_COURSE_DIRS=fixtures/courses` and an in-memory SQLite;
- `GET /api/health` reports `courses: 1`;
- `GET /some/spa/path` returns HTML containing `<div id="root">`;
- `GET /api/media/sample/media/clip.mp4` with `Range: bytes=0-9` gives 206.
- **primitives.test.tsx:** use `react-dom/server` `renderToString` to check that `ProgressBar` gives `aria-valuenow="40"` for value 0.4 and that `CourseCover` without an image shows the two-letter monogram.
- [ ] **Step 2:** Run `bun test apps packages/ui`. Expected: FAIL.
- [ ] **Step 3:** Implement. Add the dependencies `react`, `react-dom`, `react-router-dom@6` and the three `@fontsource` packages.
- [ ] **Step 4:** Run `bun test && bun run typecheck`. Then run `STC_COURSE_DIRS=./fixtures/courses:./overlays bun apps/server/src/main.ts & sleep 2; curl -s localhost:3000/api/health; kill %1`. Expected: `{"ok":true,"courses":3,"db":"sqlite"}`.
- [ ] **Step 5:** Commit: `feat: server entry and web shell with design tokens`. Then push `main`. Phase 0 is done.
---
# Phase 1 — Four parallel work packages
Each package runs as one agent in its own git worktree on branch `feat/<package>`, created from `main` after Phase 0.
Each agent starts by writing its own detailed plan, `docs/superpowers/plans/2026-10-30-<package>.md`, in the format of this plan. The plan is built from the spec sections listed for that package and the Phase 0 interfaces. The agent then executes it with TDD and frequent commits.
Each agent owns only its listed paths. The only shared files it may touch are:
- one appended line in `apps/web/src/features.ts`;
- one appended entry in the feature list in `apps/server/src/main.ts`;
- dependency additions to its own `package.json`.
Each agent must not change `packages/contracts`, `packages/course-format`, `packages/db/migrations/0001_init.sql` or `packages/core`. If it needs a contract change, it writes the proposal to `docs/contract-requests/<package>.md` and works around it locally. Phase 2 resolves these proposals.
New database tables go in the package's own migrations folder, named `NNNN_<package>_*.sql` and registered through a `migrationsDirs` export. Phase 2 wires these into `migrate()`.
### WP-A Learner app (`packages/feature-learner`, web routes under `/`, `/c/:courseId/*`, `/account`)
Spec sections 4.1–4.4, 6, 7 and 13. Scope:
- **Auth:** Better Auth on Bun, with email/password, generic OIDC, social providers and SAML. If Better Auth cannot use `Bun.SQL`/`bun:sqlite` for both dialects, write a thin adapter or choose an equivalent, and record why in the package plan. The package implements `AuthService`, plus the roles and groups admin API. Groups can be filled from OIDC/SAML claims.
- **Library:** featured course, continue row, browsable rows per collection, filters, search and covers (spec section 4.2).
- **Lesson reader:** text lessons with the track switcher, Reading/Work toggle, hint tiers and "think about it first" folds; video lessons with companion decks; deck viewer with per-slide notes; notes list; code files view with a zip download; exercise pages for runner and local-only exercises; JS/TS browser runner in a Web Worker following the `ExerciseRunner` contract; progress, gating and the floating action bar.
- **Acceptance:**
- all lesson types render for `sample`, `pure-snake` and `systemic-design`;
- the `sample` exercise passes after editing;
- notes persist on SQLite and Postgres.
### WP-B Importer (`packages/feature-import`, web routes `/admin/import`)
Spec section 8. Scope:
- job handlers for the `ImportSource` kinds `packt-video`, `code-repo`, `pdf-deck` and `video-folder`;
- generated course folders under `<dataDir>/courses/<id>/` with `stcourse.yaml` (`visibility: restricted`, `source` filled in);
- cover candidates (an ffmpeg frame, the first PDF page);
- deck page images and text into `<dataDir>/cache/`;
- an optional transcription job that calls an external Whisper-compatible HTTP endpoint (`STC_TRANSCRIBE_URL`), skipped when it isn't set;
- the admin import UI with job progress.
Acceptance: importing `/home/user/example_course_content` locally produces:
- a Docker course with 9 sections and the correct clip titles;
- S08 code attached to section 8;
- the serverless deck attached to a chosen lesson.
None of this licensed content may be committed. Tests use tiny synthetic fixtures built by the package's own scripts.
### WP-C Studio (`packages/feature-studio`, web routes `/studio/*`)
Spec sections 9.1 and 12 (git model). Scope:
- a published worktree per course under `<dataDir>/worktrees/<id>/published`, plus per-author draft worktrees;
- editors for the manifest, lessons (with live preview using the learner renderer, imported from `@stc/feature-learner` only through its exported `LessonView`, or through a `docs/contract-requests` note if it isn't exported yet), deck links, attachments and exercises;
- save as a commit, publish as a merge, an explicit Sync for courses with a remote, and conflict display;
- reloading the `CourseRepository` when the published worktree changes.
Overlay courses (whose root is an outside repo) are read-only in the studio in v1, with the message "This course is managed outside STCourses".
Acceptance: create a course in the studio, edit a lesson, publish, and see the change as a learner. A conflicting external commit is shown, not overwritten.
### WP-D AI (`packages/feature-ai`, web routes `/c/:courseId/assistant` panel and `/studio/generate`)
Spec section 9.2. Scope:
- `AiProvider` implementations for Anthropic (default model `claude-opus-5-5`) and OpenAI-compatible endpoints (covers Ollama);
- server configuration (admin) and BYOK keys, encrypted with AES-GCM using a key derived from `STC_SECRET`;
- retrieval over course files, transcripts, deck text, code and the user's notes;
- an MCP client for admin-configured servers (global or per course);
- mentor mode from the manifest's `mentor` file, on by default, with a learner toggle and a visible "Mentor mode off" banner;
- a studio generator that writes drafts into a draft worktree (through the studio's exported API, or a contract request) following the house-style patterns in spec section 9.2.
AI is off unless it is configured. Every test uses a fake `AiProvider`; no real API calls run in tests.
Acceptance:
- with a fake provider, an assistant answer cites course files;
- mentor rules appear in the system prompt unless the learner switched them off;
- generation produces draft files, not published ones.
---
# Phase 2 — Integration (sequential, on `main`)
### Task I1: Merge and resolve contracts
- [ ] Merge `feat/learner`, then `feat/import`, `feat/studio` and `feat/ai`. After each merge run `bun install && bun test && bun run typecheck && bash scripts/test-pg.sh`, and fix failures before the next merge.
- [ ] Apply the accepted `docs/contract-requests/*.md` changes to contracts/core and update the consumers. Wire each package's `migrationsDirs` into `migrate()`: core migrations first, then packages in alphabetical order. Commit after each merge: `merge: <package>`.
### Task I2: Packaging and memory budget
- [ ] Add a `Dockerfile` (`oven/bun:1.3` base, `bun install --production`, `ffmpeg` and `git` installed, non-root user, `EXPOSE 3000`, volume `/data`) and a `docker-compose.yml` with an optional `postgres:17` service behind a `postgres` profile.
- [ ] Build and run with all three courses plus the locally imported example content (mounted, not copied). After 5 minutes idle, read `docker stats --no-stream`. Expected: the total of the app and Postgres is under 2 GB. Record the numbers in `docs/operations.md`.
- [ ] Commit: `build: docker image and compose`.
### Task I3: End-to-end check and final review
- [ ] Add a `bun test` end-to-end test that starts the server against the fixtures, then signs up a user, marks a lesson done, creates a slide note, runs the sample exercise and reads progress back. Run it on both databases.
- [ ] Manually check every lesson type in a browser against the mockup on the LAN host, with a screenshot of each.
- [ ] Run a whole-branch review with a fresh reviewer on the most capable model. Fix the findings, push `main`, and write the final decision log for the user.