lucasw89/STCourses
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ea18f18b4fdf1e8dfd4e50d23ab6f29a25bcb0ff
STCourses/docs/superpowers/plans/2026-10-30-stcourses.md
T
Lucas WintherandClaude Opus 5.5 29ba043de7 Add STCourses implementation plan 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-30 02:46:28 +01:00

573 lines
44 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# STCourses Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build STCourses, a self-hosted course platform. Course content is files and user data lives in SQLite or Postgres. The build is delivered as a shared contract (Phase 0), four parallel work packages (Phase 1) and an integration pass (Phase 2).
**Architecture:**
- One Bun process (`Bun.serve`) serves a React single-page app through HTML imports, plus a JSON API.
- Course folders, described by `stcourse.yaml` manifests (in the course repo or as an overlay), are loaded into an in-memory course model.
- User data goes through one `Bun.SQL` data layer with plain-SQL migrations that run unchanged on SQLite and Postgres.
- Features (learner, import, studio, AI) are packages that export a `Feature` (routes plus job handlers) and a set of web routes. They are composed by the server and the web shell.
**Tech Stack:**
- Bun 1.3.x and TypeScript (strict).
- React 18 and React Router 6 (library mode).
- zod for validation, `Bun.YAML` for manifests, `Bun.SQL` for databases.
- @fontsource packages for IBM Plex Sans, IBM Plex Mono and Literata.
- `bun test` for tests.
**Spec:** `docs/superpowers/specs/2026-10-30-stcourses-design.md` (sections 1–13). Visual reference: `docs/design/mockups/platform-v1.html`.
## Global Constraints
These apply project-wide; every task inherits them.
- **Bun conventions** (from `bun init`'s CLAUDE.md):
- use `Bun.serve` routes and WebSockets, `bun:sqlite` or `Bun.sql`, `Bun.file`, `` Bun.$ ``, `bun test`, `bun install`, `bunx`;
- never use express, vite, dotenv, pg, postgres.js, better-sqlite3, jest or vitest.
- **Frontend:** a React SPA served via HTML imports. No Next.js. Plain CSS with custom properties; no Tailwind.
- **Course content is never stored in the database.** Deleting the DB loses user data only.
- **Database URL:** `STC_DATABASE_URL`, either `sqlite://<path>` (default `sqlite://./data/stcourses.db`) or `postgres://…`. Every migration and query must run on both.
- **Idle memory:** at most 2 GB for the deployment (checked in Phase 2).
- **License:** AGPL-3.0-only. Every package's `package.json` has `"license": "AGPL-3.0-only"`.
- **Never write into `/home/user/functional-programming-course` or `/home/user/systemic-design-course`.** Their manifests are overlays in `overlays/`.
- **Never commit anything from `/home/user/example_course_content`.** It is licensed material, used only for local test runs.
- **Design tokens** (spec section 4.1):
- ground `#12141a`, side `#181b22`, raised `#232733`, line `#262a33`, line2 `#343947`;
- text `#d9dce3`, reading text `#d9d4c8`, heading `#f2efe8`, muted `#a4abb8`, faint `#8c93a0`;
- accent `#f1b44c`, accent-dim `#8c7440`;
- fonts: IBM Plex Sans (UI), Literata (lesson titles and reading text, 19px / 1.75 in a 660px column), IBM Plex Mono (code);
- dark theme only in v1.
- **Keyboard:** arrow keys in the slide viewer, Escape closes overlays, normal Tab focus. No other shortcuts.
- **Commits:** author is the git identity configured on the machine, and every message ends with `Co-Authored-By: Claude Opus 5.5 <[email protected]>`.
## Review Focus
1. **Paths with spaces and non-ASCII characters** (for example `S08 - Containerizing Applications/`, `Thinknyx_Mastering_Docker_Essentials_Handson_DevOps.pdf`) must load, serve and download correctly. Tests: Task 4 (loader) and Task 9 (media route).
2. **An overlay whose `root` repo is missing or moved** must give a course with `status: "error"` and a readable diagnostic. The server must keep serving every other course. Tests: Task 4 and Task 9.
3. **A manifest that references a missing file, heading anchor or deck id** must give a diagnostic naming the lesson and the path, not throw. Test: Task 4.
4. **Video seeking:** `Range: bytes=…` requests get `206` with correct `Content-Range`. An unsatisfiable range gets `416`. Test: Task 9.
5. **Media paths with `..`, absolute paths or symlinks that escape the course root** must get `404` and never read outside the root. Test: Task 9.
---
## Repository layout (locked in by Phase 0)
```
package.json # bun workspaces: ["packages/*", "apps/*"]
tsconfig.json CLAUDE.md LICENSE README.md .gitignore
packages/
contracts/ # shared TS types and interfaces only (no runtime deps)
src/index.ts src/server.ts src/exercise.ts src/ai.ts src/import.ts src/web.ts
course-format/ # manifest schema, markdown extraction, discovery, CourseRepository
src/schema.ts src/markdown.ts src/load.ts src/repository.ts src/index.ts test/
db/ # Bun.SQL connection, migrations, repositories
src/connect.ts src/migrate.ts migrations/0001_init.sql src/repos/*.ts src/index.ts test/
core/ # config, job queue, feature composition, media serving
src/config.ts src/jobs.ts src/media.ts src/app.ts src/index.ts test/
ui/ # tokens.css, fonts, shared React primitives (AppShell, Pill, ProgressBar)
src/tokens.css src/fonts.css src/AppShell.tsx src/primitives.tsx src/index.ts
feature-learner/ feature-import/ feature-studio/ feature-ai/ # Phase 1 (one per work package)
apps/
server/ # entry: src/main.ts — builds ServerContext, composes features, Bun.serve
web/ # index.html, src/main.tsx (router built from feature route registries)
overlays/
pure-snake/stcourse.yaml # root: /home/user/functional-programming-course
systemic-design/stcourse.yaml # root: /home/user/systemic-design-course
fixtures/courses/sample/ # tiny self-contained course used by all tests
scripts/test-pg.sh # starts postgres:17 in docker, runs DB tests with STC_TEST_PG_URL
```
---
# Phase 0 — Contract (sequential, on `main`)
### Task 1: Workspace scaffold
**Files:**
- Create: `package.json`, `tsconfig.json`, `CLAUDE.md`, `LICENSE`, `README.md`.
- Create: one `package.json` per package and app in the layout above (`name: "@stc/<dir>"`, `type: "module"`, `license: "AGPL-3.0-only"`).
- Modify: `.gitignore`, adding `data/` and `.cache/`.
**Interfaces:**
- Produces: workspace package names `@stc/contracts`, `@stc/course-format`, `@stc/db`, `@stc/core`, `@stc/ui`, `@stc/server`, `@stc/web`, `@stc/feature-learner`, `@stc/feature-import`, `@stc/feature-studio`, `@stc/feature-ai`.
- [ ] **Step 1:** In a scratch directory, run `bun init -y` and copy its generated `CLAUDE.md` text into the root `CLAUDE.md`. Then add a "Project rules" section that repeats the Global Constraints above and points to the spec and plan.
- [ ] **Step 2:** Write the root `package.json` with `"workspaces": ["packages/*","apps/*"]` and these scripts:
- `test`: `bun test`
- `typecheck`: `bunx tsc --noEmit -p .`
- `dev`: `bun --hot apps/server/src/main.ts`
Write the root `tsconfig.json` with `strict: true`, `jsx: "react-jsx"`, `moduleResolution: "bundler"`, `types: ["bun-types"]`, and `paths` mapping `@stc/*` to `packages/*/src`.
- [ ] **Step 3:** Write the full AGPL-3.0 text to `LICENSE`. Write `README.md` with one paragraph from spec section 1 and the dev commands.
- [ ] **Step 4:** Add `packages/contracts/test/smoke.test.ts` asserting `expect(1).toBe(1)`. Run `bun install && bun test && bun run typecheck`. Expected: 1 pass, no type errors.
- [ ] **Step 5:** Commit: `chore: bun workspace scaffold`.
### Task 2: Contracts package (types only)
**Files:**
- Create: `packages/contracts/src/{index.ts,server.ts,exercise.ts,ai.ts,import.ts,web.ts}`
- Test: `packages/contracts/test/types.test.ts`
**Interfaces (Produces). This exact code is the contract; copy it verbatim:**
```ts
// server.ts
export type Role = "owner" | "author" | "learner";
export interface AuthUser { id: string; email: string; name: string; roles: Role[]; groups: string[] }
export interface RequestContext { user: AuthUser | null; params: Record<string, string>; url: URL }
export type RouteHandler = (req: Request, rc: RequestContext) => Response | Promise<Response>;
export interface Feature {
name: string; // "learner" | "import" | "studio" | "ai" | "core"
routes: Record<string, Partial<Record<"GET"|"POST"|"PUT"|"PATCH"|"DELETE", RouteHandler>>>; // paths start with /api/
jobs?: Record<string, JobHandler>; // job kind -> handler
}
export interface JobContext { reportProgress(fraction: number, message?: string): Promise<void>; signal: AbortSignal }
export type JobHandler = (payload: unknown, jc: JobContext) => Promise<unknown>;
export type JobStatus = "queued" | "running" | "done" | "failed";
export interface JobRecord { id: string; kind: string; status: JobStatus; progress: number; message: string | null; payload: unknown; result: unknown; error: string | null; createdAt: string; updatedAt: string }
export interface JobQueue { enqueue(kind: string, payload: unknown): Promise<string>; get(id: string): Promise<JobRecord | null>; list(kind?: string): Promise<JobRecord[]> }
export interface AuthService { userFromRequest(req: Request): Promise<AuthUser | null> }
export interface ServerContext {
config: AppConfig; db: import("@stc/db").Db; courses: import("@stc/course-format").CourseRepository;
jobs: JobQueue; auth: AuthService;
}
export interface AppConfig {
port: number; databaseUrl: string; dataDir: string; // dataDir holds cache/, media/, worktrees/
courseDirs: string[]; // folders scanned for stcourse.yaml (course repos and overlays/)
secret: string; // STC_SECRET, used to encrypt BYOK keys
publicUrl: string;
}
// exercise.ts — browser-runnable exercises (spec 4.3 type 6)
export type ExerciseLanguage = "js" | "ts";
export interface Assert { equal(actual: unknown, expected: unknown, message?: string): void; deepEqual(actual: unknown, expected: unknown, message?: string): void; ok(value: unknown, message?: string): void }
export interface ExerciseTest { name: string; run(mod: Record<string, unknown>, assert: Assert): void | Promise<void> }
// A tests file is an ES module: `export const tests: ExerciseTest[]`. Learner code is an ES module.
export interface TestResult { name: string; passed: boolean; message?: string; expected?: string; actual?: string }
export interface ExerciseRunner { language: ExerciseLanguage; run(learnerCode: string, testsModule: string, timeoutMs: number): Promise<TestResult[]> }
// ai.ts
export interface ChatMessage { role: "system" | "user" | "assistant" | "tool"; content: string; toolCallId?: string; name?: string }
export interface ToolSpec { name: string; description: string; inputSchema: Record<string, unknown> }
export interface AiProvider { id: string; chat(messages: ChatMessage[], opts: { tools?: ToolSpec[]; signal?: AbortSignal }): AsyncIterable<{ text?: string; toolCall?: { id: string; name: string; input: unknown } }> }
// import.ts
export interface ImportSource { kind: "packt-video" | "pdf-deck" | "code-repo" | "video-folder"; path: string; options?: Record<string, unknown> }
export interface ImportResult { courseId: string; manifestPath: string; warnings: string[] }
// web.ts
export interface WebRoute { path: string; element: () => Promise<{ default: React.ComponentType }>; nav?: { label: string; area: "header" | "admin" } }
export interface WebFeature { name: string; routes: WebRoute[] }
```
- [ ] **Step 1:** Write `types.test.ts`. It builds one object literal of each interface above, typed by the interface (so it fails to compile if a field is wrong), and runs `expect(typeof feature.routes).toBe("object")`.
- [ ] **Step 2:** Run `bun run typecheck`. Expected: errors, because the modules don't exist yet.
- [ ] **Step 3:** Create the files with the code above. `index.ts` re-exports everything. `web.ts` uses `import type React from "react"`.
- [ ] **Step 4:** Run `bun test packages/contracts && bun run typecheck`. Expected: PASS. (The `@stc/db` and `@stc/course-format` type imports resolve once Tasks 4 and 7 exist. Until then, temporarily declare `Db` and `CourseRepository` as `unknown` placeholders in `server.ts`, and switch them in Task 7.)
- [ ] **Step 5:** Commit: `feat(contracts): shared server, exercise, ai, import and web interfaces`.
### Task 3: Manifest schema
**Files:**
- Create: `packages/course-format/src/schema.ts`
- Test: `packages/course-format/test/schema.test.ts`
**Interfaces (Produces):**
- `parseManifest(yamlText: string): { ok: true; manifest: Manifest } | { ok: false; errors: string[] }`. It uses `Bun.YAML.parse`, then the zod `ManifestSchema`. Each error string reads `"<dotted.path>: <message>"`.
- `type Manifest = z.infer<typeof ManifestSchema>`, plus the exported types `Module`, `Lesson`, `Deck`, `Attachment`, `Track`.
Schema (fields are required unless marked `?`):
```yaml
schema: 1 # literal 1
id: string # /^[a-z0-9][a-z0-9-]*$/
title: string
pitch?: string
authors?: string[]
root?: string # overlay only: absolute path, or path relative to this manifest, of the course repo; default "."
cover?: string # path relative to root
accent?: string # /^#[0-9a-fA-F]{6}$/
collections?: string[]
visibility: public | signed-in | restricted # default restricted
gating?: none | sequential # default none
tracks?: [{ id: string, label: string }]
mentor?: string # path to mentor instructions file
source?: { kind: authored | packt-video | pdf-deck | code-repo | video-folder, origin?: string, importedAt?: string }
decks?: [{ id: string, file: string, title?: string, attachTo: string }] # attachTo = a lesson id or module id
attachments?: [{ file: string, title?: string, module?: string }]
modules:
- id: string
title: string
code?: { dir: string, command?: string } # "Files for this section"
lessons:
- id: string # unique across the course
title: string
type: text | video | exercise
file?: string # required for text and video
anchor?: string # heading slug inside file
duration?: number # seconds
transcript?: string # video only
tracks?: { <trackId>: { file?: string, command?: string } } # per-track file shown or command to run
hints?: { file: string, match: string } # exercise: <details> blocks whose summary starts with match
browser?: { language: js | ts, starter: string, tests: string, solution?: string } # exercise runnable in browser
localCommand?: string | { <trackId>: string }
```
- [ ] **Step 1: Write the failing tests:**
- `parses a minimal manifest`: id `x`, one module with one text lesson. Expect `visibility` to default to `"restricted"` and `gating` to `"none"`.
- `rejects a bad accent`: `accent: "orange"`. Expect an error containing `accent`.
- `rejects duplicate lesson ids across modules`. Expect an error containing the duplicate id.
- `requires file for text and video lessons`.
- `rejects a deck whose attachTo is not a lesson or module id`.
- `rejects schema: 2`.
- [ ] **Step 2:** Run `bun test packages/course-format/test/schema.test.ts`. Expected: FAIL (module not found).
- [ ] **Step 3:** Implement `ManifestSchema`. Cross-field checks (duplicate ids, `attachTo`, `file` required by type) go in a zod `.superRefine`. Add the zod dependency to `@stc/course-format`.
- [ ] **Step 4:** Run the same command. Expected: 6 pass.
- [ ] **Step 5:** Commit: `feat(course-format): stcourse.yaml schema`.
### Task 4: Markdown extraction, discovery and loading
**Files:**
- Create: `packages/course-format/src/{markdown.ts,load.ts,repository.ts,index.ts}`
- Test: `packages/course-format/test/{markdown.test.ts,load.test.ts}`
**Interfaces (Produces):**
- `slugify(heading: string): string`. GitHub style: lowercase, strip markdown and backticks, drop punctuation except `-`, turn spaces into `-`. `"Exercise 3.1 — `move`"` becomes `"exercise-31--move"`. Copy GitHub's exact algorithm (em dash removed, so the two spaces leave a double hyphen).
- `extractSection(markdown: string, anchor: string): string | null`. Returns the heading line plus everything up to the next heading of the same or a higher level.
- `extractHints(markdown: string, match: string): { label: string; body: string }[]`. Returns `<details><summary>…</summary>…</details>` blocks whose summary text starts with `match`, in document order. `label` is the summary text after `match`, with leading `" — "`, `" - "` or `" ("` trimmed.
- `type Diagnostic = { level: "error" | "warning"; courseId?: string; lessonId?: string; path?: string; message: string }`
- `interface LoadedCourse { id: string; manifestPath: string; root: string; manifest: Manifest | null; status: "ok" | "error"; diagnostics: Diagnostic[]; lessonIndex: Map<string, { moduleId: string; lesson: Lesson }> }`
- `discoverManifests(courseDirs: string[]): Promise<string[]>`. Finds `stcourse.yaml` at depth 0–2 under each dir. Duplicate ids are resolved in `loadAll`.
- `loadCourse(manifestPath: string): Promise<LoadedCourse>`. Resolves `root`, then checks that every referenced file exists under root, every `anchor` resolves through `extractSection`, and every `hints.match` finds at least one block. Each failure becomes a diagnostic; nothing throws. A missing root gives `status: "error"`.
- `loadAll(courseDirs: string[]): Promise<LoadedCourse[]>`. When two manifests share an id, the one whose `root` points outside its own directory (the overlay) wins, and a warning diagnostic records the shadowed path (spec section 13).
- `interface CourseRepository { list(): LoadedCourse[]; get(id: string): LoadedCourse | null; reload(id?: string): Promise<void>; readLessonMarkdown(courseId: string, lessonId: string): Promise<string | null>; resolvePath(courseId: string, relPath: string): string | null }`
- `resolvePath` returns an absolute path only if the real path (following symlinks) is inside the course root. Otherwise it returns `null`.
- `createCourseRepository(courseDirs: string[]): Promise<CourseRepository>`.
- [ ] **Step 1: Write the failing tests in `markdown.test.ts`:**
- `slugify` of `"Exercise 3.1 — \`move\`"` equals `"exercise-31--move"`, and `"Why this module"` equals `"why-this-module"`.
- `extractSection` on a 3-heading sample returns only the middle `###` section.
- `extractHints` on a sample containing `<summary>Exercise 3.1 (move) — nudge</summary>` and `<summary>Exercise 3.1 (move) — stronger hint</summary>`, with match `"Exercise 3.1 (move)"`, returns labels `["nudge", "stronger hint"]`.
- [ ] **Step 2: Write the failing tests in `load.test.ts`**, using temp dirs built with `Bun.write`:
- `loads the sample fixture with no errors` (Task 5 fixture).
- `overlay with missing root gives status error`: diagnostic message contains the root path, and `loadAll` still returns the other course.
- `missing lesson file, anchor and hints each produce one error diagnostic naming lessonId`.
- `handles spaces and unicode in paths`: lesson file `S08 - Containerizing Applications/Æble notes.md`.
- `overlay wins over in-repo manifest with same id` (spec section 13).
- `resolvePath rejects ../ and symlink escape`: create a symlink inside root pointing at `/etc`, and expect `null`.
- [ ] **Step 3:** Run `bun test packages/course-format`. Expected: FAIL.
- [ ] **Step 4:** Implement. Use `fs.realpath` from `node:fs/promises` for the containment check (`Bun.file` has no realpath). For file reads, use `Bun.file(...).text()`.
- [ ] **Step 5:** Run `bun test packages/course-format`. Expected: all pass. Commit: `feat(course-format): markdown extraction, discovery, loading, repository`.
### Task 5: Sample fixture course
**Files:**
- Create: `fixtures/courses/sample/stcourse.yaml`
- Create: `fixtures/courses/sample/lessons/{01-intro.md,02-video.md,03-code.md,04-exercise.md}`
- Create: `fixtures/courses/sample/media/clip.mp4` (a 2-second, 160×90 test pattern made with ffmpeg, under 50 KB)
- Create: `fixtures/courses/sample/media/deck.pdf` (3 pages, made with pymupdf or by hand, under 20 KB)
- Create: `fixtures/courses/sample/exercises/add/{starter.js,tests.js,solution.js}`
- Create: `fixtures/courses/sample/code/S01 - First steps/hello.sh`
- Create: `fixtures/courses/sample/MENTOR.md`
- Create: `scripts/make-fixtures.sh` (regenerates the binaries)
**Interfaces (Produces):** course id `sample`, `visibility: public`, `tracks: [{id: js, label: JavaScript}]`, and two modules:
- `m1`, with lessons `m1-intro` (text), `m1-video` (video, `file: media/clip.mp4`, `duration: 2`) and `m1-code` (text). Module `code: { dir: "code/S01 - First steps", command: "sh hello.sh" }`.
- `m2`, with lessons `m2-text` (text, containing `### Exercise 2.1 — \`add\`` and a `## Hints` section with two `<details>` for `Exercise 2.1 (add)`) and `m2-ex-add` (type exercise, `anchor: exercise-21--add`, `hints: { file: lessons/04-exercise.md, match: "Exercise 2.1 (add)" }`, `browser: { language: js, starter: exercises/add/starter.js, tests: exercises/add/tests.js, solution: exercises/add/solution.js }`, `localCommand: "node --test"`).
Also `decks: [{ id: d1, file: media/deck.pdf, attachTo: m1-video }]` and `mentor: MENTOR.md`. `tests.js` exports `tests: ExerciseTest[]` checking `add(2,3) === 5` and `add(-1,1) === 0`. `starter.js` returns `a - b`, so both tests fail at the start.
- [ ] **Step 1:** Write `scripts/make-fixtures.sh` (ffmpeg `testsrc` for 2s at 160x90 with `-pix_fmt yuv420p`; the PDF via `/home/user/.venvs/pdf/bin/python -I` with pymupdf). Run it and check that both files exist and are under their size limits.
- [ ] **Step 2:** Write the manifest and the Markdown/JS files above.
- [ ] **Step 3:** Run `bun test packages/course-format` (the Task 4 fixture test). Expected: `loads the sample fixture with no errors` passes.
- [ ] **Step 4:** Commit: `test: sample fixture course`.
### Task 6: Reference course overlays
**Files:**
- Create: `overlays/pure-snake/stcourse.yaml`, `overlays/systemic-design/stcourse.yaml`
- Test: `packages/course-format/test/reference.test.ts`
**Interfaces (Produces):** course ids `pure-snake` and `systemic-design`, both `visibility: public`.
- **Pure Snake** (`root: /home/user/functional-programming-course`):
- 12 modules, `m00`…`m11`, one per `course/NN-*.md`.
- Each module has a text lesson (`mNN-text`, the whole file) and one exercise lesson per `### Exercise N.M — …` heading: id `mNN-ex-<name>` (for example `m03-ex-move`), with `anchor` set to the slug, `hints.match` set to `"Exercise N.M (<name>)"` (check the exact summary text in each file), `localCommand` per track (`js`: `cd js && node --test "test/mNN-*.test.js"`, `scala`: `cd scala && ./test.sh MNN`, `fsharp`: `cd fsharp && ./test.sh MNN`), and no `browser` (spec section 13).
- `tracks`: js / JavaScript, scala / Scala 3, fsharp / F#. `mentor: MENTOR.md`.
- `method.md` and `reference.md` become text lessons in an extra module `ref`, titled "Method and reference".
- **Systemic Design** (`root: /home/user/systemic-design-course`):
- Modules `00`…`10`, one per `course/NN-*/`.
- Each module has `lesson.md` as a text lesson and `scenarios.md` as a text lesson titled "Scenarios".
- `gating: sequential`; tracks rust / Rust and ts / TypeScript; `mentor: mentor/SKILL.md`.
- `localCommand` is `node tools/start-module.mjs NN <your-workspace>`.
Read each module's real headings with `grep -n '^#'` before writing anchors.
- [ ] **Step 1:** Write the failing `reference.test.ts`. If a reference root directory doesn't exist, skip with `test.skipIf`. Assertions:
- `pure-snake` loads with zero error diagnostics, has 13 modules, and includes lesson `m03-ex-move` with exactly 3 hints;
- `systemic-design` loads with zero error diagnostics, has 11 modules, and has `gating === "sequential"`.
- [ ] **Step 2:** Run `bun test packages/course-format/test/reference.test.ts`. Expected: FAIL.
- [ ] **Step 3:** Write both overlay manifests.
- [ ] **Step 4:** Run the test. Expected: PASS. Confirm `git -C /home/user/functional-programming-course status --short` and the same for systemic-design are unchanged from before the task.
- [ ] **Step 5:** Commit: `feat: overlay manifests for the two reference courses`.
### Task 7: Database connection and migrations
**Files:**
- Create: `packages/db/src/{connect.ts,migrate.ts,index.ts}`, `packages/db/migrations/0001_init.sql`, `scripts/test-pg.sh`
- Test: `packages/db/test/migrate.test.ts`, `packages/db/test/helpers.ts`
**Interfaces (Produces):**
- `type Db = { sql: Bun.SQL; dialect: "sqlite" | "postgres"; close(): Promise<void> }`
- `connect(url: string): Promise<Db>`. `sqlite://` creates parent directories and enables WAL and foreign keys. `postgres://` passes through to `new Bun.SQL(url)`.
- `migrate(db: Db): Promise<string[]>`. Applies `migrations/*.sql` in name order that aren't yet in `schema_migrations(name TEXT PRIMARY KEY, applied_at TEXT)`, each in a transaction, and returns the names applied.
- `test/helpers.ts`: `forEachDb(name: string, fn: (db: Db) => Promise<void>)`. Registers the test against a fresh in-memory SQLite, and also against Postgres when `STC_TEST_PG_URL` is set, using a fresh schema per test via `SET search_path`.
- In `contracts/server.ts`, replace the placeholder with `import type { Db } from "@stc/db"`.
**Portable SQL rules (all migrations):**
- ids are `TEXT` (generated in the app with `Bun.randomUUIDv7()`);
- timestamps are `TEXT` in ISO-8601 UTC;
- booleans are `INTEGER` 0/1;
- JSON is `TEXT`;
- no `AUTOINCREMENT`, `SERIAL`, `RETURNING` or dialect-only functions.
`0001_init.sql` tables (columns are `NOT NULL` unless marked `?`):
```
user_roles(user_id, role) PK(user_id, role)
groups(id PK, name UNIQUE)
group_members(group_id, user_id) PK(group_id, user_id)
course_authors(course_id, user_id) PK(course_id, user_id)
course_acl(course_id, principal_type ['user'|'group'], principal_id) PK(all three)
progress(user_id, course_id, lesson_id, state ['not-started'|'in-progress'|'done'], position_seconds? REAL, updated_at) PK(user_id, course_id, lesson_id)
notes(id PK, user_id, course_id, lesson_id, anchor_kind ['none'|'video'|'slide'], anchor_value? TEXT, deck_id? TEXT, body, created_at, updated_at) INDEX(user_id, course_id, lesson_id)
exercise_attempts(user_id, course_id, lesson_id, track, code, result_json?, passed INTEGER, updated_at) PK(user_id, course_id, lesson_id, track)
user_api_keys(user_id, provider, ciphertext, created_at) PK(user_id, provider)
ai_settings(scope ['server'|'user'], owner_id, mentor_off INTEGER, config_json, updated_at) PK(scope, owner_id) -- owner_id '' for server
ai_conversations(id PK, user_id, course_id, title?, created_at)
ai_messages(id PK, conversation_id, role, content, created_at) INDEX(conversation_id)
library_settings(key PK, value) -- e.g. featured_course_id
collections(id PK, title, sort INTEGER)
collection_courses(collection_id, course_id, sort INTEGER) PK(collection_id, course_id)
jobs(id PK, kind, status, progress REAL, message?, payload_json, result_json?, error?, created_at, updated_at) INDEX(status)
```
Auth tables belong to the auth library (Phase 1, learner package). They live in their own migration files there, `user_id` is `TEXT`, and no foreign keys point from these tables to them.
- [ ] **Step 1:** Write `migrate.test.ts` using `forEachDb`:
- `applies 0001 once`: the first run returns `["0001_init.sql"]` and the second returns `[]`;
- `all tables exist`: insert and select one row in each table.
- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL.
- [ ] **Step 3:** Implement `connect`, `migrate` and the SQL. Write `scripts/test-pg.sh`: it runs `docker run -d --rm --name stc-pg -e POSTGRES_PASSWORD=stc -p 55432:5432 postgres:17`, waits for `pg_isready`, runs `STC_TEST_PG_URL=postgres://postgres:[email protected]:55432/postgres bun test packages`, then stops the container.
- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS on both dialects.
- [ ] **Step 5:** Commit: `feat(db): Bun.SQL connection and portable migrations`.
### Task 8: Data repositories
**Files:**
- Create: `packages/db/src/repos/{progress.ts,notes.ts,attempts.ts,library.ts,jobs.ts}`, exported from `index.ts`
- Test: `packages/db/test/repos.test.ts`
**Interfaces (Produces):** each is a factory taking `Db`.
- `progressRepo(db)`:
- `get(userId, courseId): Promise<Record<lessonId, {state, positionSeconds: number|null, updatedAt}>>`
- `set(userId, courseId, lessonId, state, positionSeconds?): Promise<void>`. An upsert written as select-then-insert-or-update inside a transaction, so it stays portable.
- `notesRepo(db)`:
- `list(userId, courseId, lessonId?): Promise<Note[]>`, ordered by `anchor_kind`, then by `anchor_value` read numerically, then `created_at`
- `create(input: {userId, courseId, lessonId, anchorKind, anchorValue?, deckId?, body}): Promise<Note>`
- `update(id, userId, body): Promise<Note|null>` (only the owner can update)
- `remove(id, userId): Promise<boolean>`
- `type Note = {id, userId, courseId, lessonId, anchorKind: "none"|"video"|"slide", anchorValue: string|null, deckId: string|null, body, createdAt, updatedAt}`
- `attemptsRepo(db)`:
- `get(userId, courseId, lessonId, track): Promise<Attempt|null>`
- `save(a: Attempt): Promise<void>`
- `libraryRepo(db)`:
- `getFeatured(): Promise<string|null>`, `setFeatured(courseId|null)`
- `listCollections(): Promise<{id,title,sort,courseIds: string[]}[]>`, `saveCollection(c)`, `deleteCollection(id)`
- `jobsRepo(db)`:
- `insert(kind, payload): Promise<string>`
- `claimNext(kinds: string[]): Promise<JobRecord|null>`. Atomically moves one queued job to running: `UPDATE … WHERE id = (SELECT …) AND status='queued'`, then reads it back.
- `update(id, patch: Partial<Pick<JobRecord,"status"|"progress"|"message"|"result"|"error">>)`
- `get(id)`, `list(kind?)`
- [ ] **Step 1:** Write `repos.test.ts` with `forEachDb`, one test per method group:
- progress upsert keeps one row;
- notes order puts a slide note on page 2 before page 10;
- another user can't update or delete a note;
- attempts round-trip;
- collections keep their order;
- `claimNext` returns null when nothing is queued, and two concurrent `claimNext` calls never claim the same job.
- [ ] **Step 2:** Run `bun test packages/db`. Expected: FAIL.
- [ ] **Step 3:** Implement the repositories.
- [ ] **Step 4:** Run `bun test packages/db && bash scripts/test-pg.sh`. Expected: PASS.
- [ ] **Step 5:** Commit: `feat(db): progress, notes, attempts, library, jobs repositories`.
### Task 9: Core: config, job queue, media serving, feature composition
**Files:**
- Create: `packages/core/src/{config.ts,jobs.ts,media.ts,app.ts,index.ts}`
- Test: `packages/core/test/{config.test.ts,jobs.test.ts,media.test.ts,app.test.ts}`
**Interfaces (Produces):**
- `loadConfig(env = process.env): AppConfig`. Reads these variables:
- `STC_PORT`: default 3000
- `STC_DATABASE_URL`: default `sqlite://./data/stcourses.db`
- `STC_DATA_DIR`: default `./data`
- `STC_COURSE_DIRS`: `:`-separated, default `./overlays:./courses`
- `STC_SECRET`: required when `NODE_ENV=production`; otherwise a dev default with a printed warning
- `STC_PUBLIC_URL`: default `http://localhost:<port>`
- `createJobQueue(db: Db, handlers: Record<string, JobHandler>, opts?: {pollMs?: number}): JobQueue & { start(): void; stop(): Promise<void> }`. A single worker loop (one job at a time) that uses `jobsRepo.claimNext`. Handler errors give `status: "failed"` with the error message. On startup, jobs left `running` from a previous process are reset to `queued`.
- `serveMedia(req: Request, absPath: string): Promise<Response>`. Uses `Bun.file`. Handles `Range: bytes=a-b`, `a-` and `-n`, returning 206 with `Content-Range`, `Accept-Ranges: bytes` and `Content-Length`. Unsatisfiable ranges get 416. Content type comes from the extension (mp4, webm, pdf, png, jpg, svg, md, txt, zip, js, ts). Downloads use `Content-Disposition` with RFC 5987 `filename*` for non-ASCII names.
- `createApp(ctx: ServerContext, features: Feature[]): { routes: Record<string, …>; fetch(req: Request): Promise<Response> }`, which composes:
- each feature's routes, wrapped with a `RequestContext` built by `ctx.auth.userFromRequest`. Duplicate path plus method across features throws at startup.
- core routes:
- `GET /api/health`, returning `{ ok: true, courses: n, db: dialect }`;
- `GET /api/media/:courseId/*`, which resolves via `ctx.courses.resolvePath`. A null result gives 404. A course the user cannot see gives 404 (visibility check: `canView(user, course)`).
- `canView(user: AuthUser|null, course: LoadedCourse, acl: {principal_type, principal_id}[]): boolean` is exported:
- public: anyone;
- signed-in: any user;
- restricted: owner role, a course author, or a matching ACL user or group.
- [ ] **Step 1: Write the failing tests:**
- **config:** defaults; colon splitting; production without a secret throws.
- **jobs:** an enqueued job runs and reaches `done` with its result; a throwing handler gives `failed`; progress updates persist; a stale `running` job is requeued at start.
- **media:** the full file gives 200; `bytes=0-99` gives 206 with `Content-Range: bytes 0-99/<size>`; `bytes=999999999-` gives 416; a filename `Æble notes.pdf` gives a valid `filename*`.
- **app:**
- health returns ok;
- a media request with `..%2F..%2Fetc%2Fpasswd` gives 404;
- a restricted course gives 404 when anonymous;
- a course with `status: "error"` doesn't break `/api/health`;
- two features registering `GET /api/x` throw.
- [ ] **Step 2:** Run `bun test packages/core`. Expected: FAIL.
- [ ] **Step 3:** Implement.
- [ ] **Step 4:** Run `bun test packages/core`. Expected: PASS.
- [ ] **Step 5:** Commit: `feat(core): config, job queue, range media serving, feature composition`.
### Task 10: Server and web shell
**Files:**
- Create: `apps/server/src/main.ts`
- Create: `packages/ui/src/{tokens.css,fonts.css,AppShell.tsx,primitives.tsx,index.ts}`
- Create: `apps/web/index.html`, `apps/web/src/main.tsx`, `apps/web/src/features.ts`
- Test: `apps/server/test/server.test.ts`, `packages/ui/test/primitives.test.tsx`
**Interfaces:**
- **Consumes:** everything above.
- **Produces:**
- `apps/server/src/main.ts` exports `startServer(config: AppConfig, features: Feature[], webFeatures?: unknown): Promise<{ server: Bun.Server; stop(): Promise<void> }>`. It connects the DB, migrates, creates the course repository and job queue, then calls `Bun.serve({ routes: { ...app.routes, "/*": indexHtml }, fetch: app.fetch })` so the SPA handles every non-API path. Run directly, it calls `startServer(loadConfig(), [])`.
- A temporary `AuthService` returns `null`; the learner package replaces it.
- `apps/web/src/features.ts` exports `webFeatures: WebFeature[]`, empty in Phase 0. Each Phase 1 package appends one import line, which is the only shared line they touch.
- `main.tsx` builds a `createBrowserRouter` from `webFeatures` routes (lazy `element`), inside `<AppShell>`, with a fallback home page reading "No features installed yet".
- `@stc/ui`:
- `AppShell({ header?: ReactNode, sidebar?: ReactNode, children })`;
- `Pill({ variant?: "default"|"primary"|"ghost", ...buttonProps })`;
- `SegmentedToggle({ options: {value,label}[], value, onChange, label })`;
- `ProgressBar({ value: number /*0..1*/, dim?: boolean, label: string })`, which renders `role="progressbar"`;
- `FloatingActionBar({ children })`;
- `CourseCover({ title, image?, accent? })`, a monogram fallback.
- `tokens.css` defines `--ground --side --raised --line --line2 --text --read --head --muted --faint --accent --accent-dim --sans --serif --mono`, plus `.reading` (Literata 19px / 1.75, max-width 660px) and `.work` (Plex Sans 15.5px, max-width 760px) classes. The values are listed in Global Constraints. The mockup at `docs/design/mockups/platform-v1.html` is the visual reference for the primitives.
- `fonts.css` imports `@fontsource/ibm-plex-sans/{400,500,600}.css`, `@fontsource/ibm-plex-mono/400.css` and `@fontsource/literata/{400,600,400-italic}.css`. Nothing loads from googleapis.
- [ ] **Step 1: Write the failing tests:**
- **server.test.ts:**
- start on port 0 with `STC_COURSE_DIRS=fixtures/courses` and an in-memory SQLite;
- `GET /api/health` reports `courses: 1`;
- `GET /some/spa/path` returns HTML containing `<div id="root">`;
- `GET /api/media/sample/media/clip.mp4` with `Range: bytes=0-9` gives 206.
- **primitives.test.tsx:** use `react-dom/server` `renderToString` to check that `ProgressBar` gives `aria-valuenow="40"` for value 0.4 and that `CourseCover` without an image shows the two-letter monogram.
- [ ] **Step 2:** Run `bun test apps packages/ui`. Expected: FAIL.
- [ ] **Step 3:** Implement. Add the dependencies `react`, `react-dom`, `react-router-dom@6` and the three `@fontsource` packages.
- [ ] **Step 4:** Run `bun test && bun run typecheck`. Then run `STC_COURSE_DIRS=./fixtures/courses:./overlays bun apps/server/src/main.ts & sleep 2; curl -s localhost:3000/api/health; kill %1`. Expected: `{"ok":true,"courses":3,"db":"sqlite"}`.
- [ ] **Step 5:** Commit: `feat: server entry and web shell with design tokens`. Then push `main`. Phase 0 is done.
---
# Phase 1 — Four parallel work packages
Each package runs as one agent in its own git worktree on branch `feat/<package>`, created from `main` after Phase 0.
Each agent starts by writing its own detailed plan, `docs/superpowers/plans/2026-10-30-<package>.md`, in the format of this plan. The plan is built from the spec sections listed for that package and the Phase 0 interfaces. The agent then executes it with TDD and frequent commits.
Each agent owns only its listed paths. The only shared files it may touch are:
- one appended line in `apps/web/src/features.ts`;
- one appended entry in the feature list in `apps/server/src/main.ts`;
- dependency additions to its own `package.json`.
Each agent must not change `packages/contracts`, `packages/course-format`, `packages/db/migrations/0001_init.sql` or `packages/core`. If it needs a contract change, it writes the proposal to `docs/contract-requests/<package>.md` and works around it locally. Phase 2 resolves these proposals.
New database tables go in the package's own migrations folder, named `NNNN_<package>_*.sql` and registered through a `migrationsDirs` export. Phase 2 wires these into `migrate()`.
### WP-A Learner app (`packages/feature-learner`, web routes under `/`, `/c/:courseId/*`, `/account`)
Spec sections 4.1–4.4, 6, 7 and 13. Scope:
- **Auth:** Better Auth on Bun, with email/password, generic OIDC, social providers and SAML. If Better Auth cannot use `Bun.SQL`/`bun:sqlite` for both dialects, write a thin adapter or choose an equivalent, and record why in the package plan. The package implements `AuthService`, plus the roles and groups admin API. Groups can be filled from OIDC/SAML claims.
- **Library:** featured course, continue row, browsable rows per collection, filters, search and covers (spec section 4.2).
- **Lesson reader:** text lessons with the track switcher, Reading/Work toggle, hint tiers and "think about it first" folds; video lessons with companion decks; deck viewer with per-slide notes; notes list; code files view with a zip download; exercise pages for runner and local-only exercises; JS/TS browser runner in a Web Worker following the `ExerciseRunner` contract; progress, gating and the floating action bar.
- **Acceptance:**
- all lesson types render for `sample`, `pure-snake` and `systemic-design`;
- the `sample` exercise passes after editing;
- notes persist on SQLite and Postgres.
### WP-B Importer (`packages/feature-import`, web routes `/admin/import`)
Spec section 8. Scope:
- job handlers for the `ImportSource` kinds `packt-video`, `code-repo`, `pdf-deck` and `video-folder`;
- generated course folders under `<dataDir>/courses/<id>/` with `stcourse.yaml` (`visibility: restricted`, `source` filled in);
- cover candidates (an ffmpeg frame, the first PDF page);
- deck page images and text into `<dataDir>/cache/`;
- an optional transcription job that calls an external Whisper-compatible HTTP endpoint (`STC_TRANSCRIBE_URL`), skipped when it isn't set;
- the admin import UI with job progress.
Acceptance: importing `/home/user/example_course_content` locally produces:
- a Docker course with 9 sections and the correct clip titles;
- S08 code attached to section 8;
- the serverless deck attached to a chosen lesson.
None of this licensed content may be committed. Tests use tiny synthetic fixtures built by the package's own scripts.
### WP-C Studio (`packages/feature-studio`, web routes `/studio/*`)
Spec sections 9.1 and 12 (git model). Scope:
- a published worktree per course under `<dataDir>/worktrees/<id>/published`, plus per-author draft worktrees;
- editors for the manifest, lessons (with live preview using the learner renderer, imported from `@stc/feature-learner` only through its exported `LessonView`, or through a `docs/contract-requests` note if it isn't exported yet), deck links, attachments and exercises;
- save as a commit, publish as a merge, an explicit Sync for courses with a remote, and conflict display;
- reloading the `CourseRepository` when the published worktree changes.
Overlay courses (whose root is an outside repo) are read-only in the studio in v1, with the message "This course is managed outside STCourses".
Acceptance: create a course in the studio, edit a lesson, publish, and see the change as a learner. A conflicting external commit is shown, not overwritten.
### WP-D AI (`packages/feature-ai`, web routes `/c/:courseId/assistant` panel and `/studio/generate`)
Spec section 9.2. Scope:
- `AiProvider` implementations for Anthropic (default model `claude-opus-5-5`) and OpenAI-compatible endpoints (covers Ollama);
- server configuration (admin) and BYOK keys, encrypted with AES-GCM using a key derived from `STC_SECRET`;
- retrieval over course files, transcripts, deck text, code and the user's notes;
- an MCP client for admin-configured servers (global or per course);
- mentor mode from the manifest's `mentor` file, on by default, with a learner toggle and a visible "Mentor mode off" banner;
- a studio generator that writes drafts into a draft worktree (through the studio's exported API, or a contract request) following the house-style patterns in spec section 9.2.
AI is off unless it is configured. Every test uses a fake `AiProvider`; no real API calls run in tests.
Acceptance:
- with a fake provider, an assistant answer cites course files;
- mentor rules appear in the system prompt unless the learner switched them off;
- generation produces draft files, not published ones.
---
# Phase 2 — Integration (sequential, on `main`)
### Task I1: Merge and resolve contracts
- [ ] Merge `feat/learner`, then `feat/import`, `feat/studio` and `feat/ai`. After each merge run `bun install && bun test && bun run typecheck && bash scripts/test-pg.sh`, and fix failures before the next merge.
- [ ] Apply the accepted `docs/contract-requests/*.md` changes to contracts/core and update the consumers. Wire each package's `migrationsDirs` into `migrate()`: core migrations first, then packages in alphabetical order. Commit after each merge: `merge: <package>`.
### Task I2: Packaging and memory budget
- [ ] Add a `Dockerfile` (`oven/bun:1.3` base, `bun install --production`, `ffmpeg` and `git` installed, non-root user, `EXPOSE 3000`, volume `/data`) and a `docker-compose.yml` with an optional `postgres:17` service behind a `postgres` profile.
- [ ] Build and run with all three courses plus the locally imported example content (mounted, not copied). After 5 minutes idle, read `docker stats --no-stream`. Expected: the total of the app and Postgres is under 2 GB. Record the numbers in `docs/operations.md`.
- [ ] Commit: `build: docker image and compose`.
### Task I3: End-to-end check and final review
- [ ] Add a `bun test` end-to-end test that starts the server against the fixtures, then signs up a user, marks a lesson done, creates a slide note, runs the sample exercise and reads progress back. Run it on both databases.
- [ ] Manually check every lesson type in a browser against the mockup on the LAN host, with a screenshot of each.
- [ ] Run a whole-branch review with a fresh reviewer on the most capable model. Fix the findings, push `main`, and write the final decision log for the user.
Reference in New Issue View Git Blame Copy Permalink