Lucas WintherandClaude Opus 5.5 committed
commit
627ebe0029
40 files changed
+2146
-2
No files matched your search
@@ -0,0 +1,12 @@
|
||||
# Contract requests from WP-B (Importer)
|
||||
|
||||
1. **`loadConfig` should add `<dataDir>/courses` to `courseDirs`.** The importer writes courses there. Worked around in `apps/server/src/main.ts` (the main guard pushes it) and in `packages/feature-import/dev/serve.ts`. `/api/import/info` reports `coursesDirServed` and the UI warns when it is false.
|
||||
2. **`startServer` accepts feature factories** (`Feature | ((ctx) => Feature)`). A `Feature` needs `ServerContext`, but `ServerContext.jobs` is built from the features' job handlers. `importFeature(ctx)` only reads `ctx` while a job or request runs, so `main.ts` now builds `ctx` first, resolves the factories, then assigns `ctx.jobs`. The edit is about ten lines in `apps/server/src/main.ts`; Phase 2 should keep or replace it.
|
||||
3. **Web entry for features.** `apps/web` must not pull a package's server code into the browser bundle. `@stc/feature-import` exposes a browser-safe entry as `./web` (package.json `exports`), but the root tsconfig `paths` (`@stc/*` to `packages/*/src`) cannot resolve `@stc/feature-import/web`, so `apps/web/src/features.ts` imports it by relative path. Suggest a tsconfig `paths` entry per feature, or a `src/web.ts` convention.
|
||||
4. **Derived cache layout (no contract defines one).** Written by the importer, to be read by the learner and AI packages:
|
||||
- `<dataDir>/cache/decks/<courseId>/<deckId>/p-001.jpg ...` (1280px wide JPEGs) and `text.json` (`[{ page, text }]`);
|
||||
- `<dataDir>/cache/covers/<courseId>/frame-N.jpg` and `deck-<deckId>.jpg` (cover candidates);
|
||||
- `<dataDir>/cache/transcripts/<courseId>/<lessonId>.json` (`{ lessonId, text, segments: [{ start, end, text }] }`).
|
||||
Suggest a `cacheDir`/`deckPageUrl` helper in core, and a media route for `cache/decks` so the deck viewer can load pages.
|
||||
5. **Video lessons have no description field.** Packt clips come with a one-line description. It is dropped, because `transcript` is the only Markdown a video lesson can carry and the transcription job writes it.
|
||||
6. **`STC_IMPORT_ROOTS`** (":"-separated, default `<dataDir>/import-inbox`, created when `importFeature` is built) is read inside the import package. Core config could own it later.
|
||||
@@ -0,0 +1,56 @@
|
||||
# WP-B Importer Implementation Plan
|
||||
|
||||
> Executed inline by the WP-B agent, task by task, TDD. Spec: `docs/superpowers/specs/2026-10-30-stcourses-design.md` sections 5, 8, 13.
|
||||
|
||||
**Goal:** `packages/feature-import` turns a Packt video zip, a companion code repo, a slide PDF or a folder of videos into an ordinary course folder under `<dataDir>/courses/<id>/`, as background jobs with visible progress, plus an owner-only admin UI at `/admin/import`.
|
||||
|
||||
**Architecture:**
|
||||
- `importFeature(ctx)` returns a `Feature` with job handlers `import.packt-video`, `import.code-repo`, `import.pdf-deck`, `import.video-folder`, `import.transcribe` and owner-only `/api/import/*` routes. `ctx` is read lazily (inside handlers), because the job queue is built from the handlers.
|
||||
- Courses are built in `<dataDir>/tmp/<job>` and renamed into `<dataDir>/courses/<id>` in one step. Attach jobs (code, deck) only touch courses under `<dataDir>/courses` and write the manifest last, through temp file + rename.
|
||||
- Media: **copied** (`cp --reflink=auto`, so instant on CoW filesystems), or hard-linked with `options.media = "link"` (falls back to copy across filesystems). Symlinks out of the course are not an option: the media route 404s on paths that escape the course root. The originals are only ever read.
|
||||
- PDFs: PyMuPDF through `py/pdf_tool.py` run as `python -I` in a subprocess (`STC_PYTHON`, default `python3`). Chosen over pdfjs/mupdf-wasm because it rasterises and extracts text natively, is already installed, is AGPL-compatible and costs no idle memory in the server. If it is missing, the deck is still attached and the job reports a warning.
|
||||
- Derived data goes to `<dataDir>/cache/`: `decks/<course>/<deck>/p-001.jpg…` + `text.json`, `covers/<course>/*.jpg`, `transcripts/<course>/<lesson>.json`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
Inherited unchanged from `docs/superpowers/plans/2026-10-30-stcourses.md` (Bun conventions, AGPL, design tokens, no licensed content in the repo, commit trailer).
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. Original files are never modified; attach jobs refuse courses outside `<dataDir>/courses` (overlays, authored repos).
|
||||
2. Paths with spaces/non-ASCII (`S08 - Containerizing Applications`) survive unzip, copy and manifest round trip.
|
||||
3. Packt parsing is structural (chapter blocks plus the section summary lists), not regexes on "Video N.M:" prefixes.
|
||||
4. Failure leaves no half-built course visible; a failed PDF/ffmpeg step degrades to a warning where the course is still valid.
|
||||
5. Routes: 401 without a user, 403 without the owner role.
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: Utilities (slug, HTML entities, natural sort, exec, manifest IO, sources)
|
||||
- Files: `src/util/*.ts`, `test/util.test.ts`
|
||||
- Interfaces: `slugify`, `uniqueCourseId`, `decodeEntities`, `naturalSort`, `run`, `readRawManifest`, `writeManifest` (validates with `ManifestSchema`), `openSource(path)` (zip via `unzip`, or directory).
|
||||
- [ ] Failing tests for each helper; [ ] implement; [ ] commit.
|
||||
|
||||
### Task 2: Fixtures and media tooling
|
||||
- Files: `test/fixtures.ts` (generates fake Packt zip, repo zip, PDF, video folder in a temp dir), `src/media.ts` (ffprobe duration, ffmpeg frame, copy/link), `src/pdf.ts`, `py/pdf_tool.py`.
|
||||
- [ ] Failing tests (frame extraction, duration, PDF pages+text); [ ] implement; [ ] commit.
|
||||
|
||||
### Task 3: Packt parser and `import.packt-video`
|
||||
- Files: `src/packt.ts`, `src/jobs/packt-video.ts`, `test/packt.test.ts`
|
||||
- [ ] Parser test on the synthetic index (summary-only titles, missing chapter); [ ] job test (2 sections, 3 clips, restricted manifest, source filled, course loads `ok`, cover candidates); [ ] implement; [ ] commit.
|
||||
|
||||
### Task 4: `import.video-folder`, `import.pdf-deck`
|
||||
- [ ] Tests: natural order, deck cache pages/text, attach to lesson and module, refuses unmanaged course; [ ] implement; [ ] commit.
|
||||
|
||||
### Task 5: `import.code-repo`
|
||||
- [ ] Tests: folder `S02 - …` attaches to module 2 by number, by name when no number, PDFs become deck/attachment, git URL validation; [ ] implement; [ ] commit.
|
||||
|
||||
### Task 6: `import.transcribe`
|
||||
- [ ] Tests with a fake Whisper server; skipped when `STC_TRANSCRIBE_URL` unset; [ ] implement; [ ] commit.
|
||||
|
||||
### Task 7: Routes and `importFeature`
|
||||
- [ ] Tests: 401/403/owner, enqueue validation, queue integration (SQLite and Postgres), covers; [ ] implement; wire `main.ts`; [ ] commit.
|
||||
|
||||
### Task 8: Web UI `importWeb`
|
||||
- [ ] Page with form, job list with progress, cover picker; register in `apps/web/src/features.ts`; screenshot check; [ ] commit.
|
||||
|
||||
### Task 9: Manual check against `/home/user/example_course_content`, report.
|
||||
Reference in new issue
Block a user