docs(import): WP-B plan

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
2026-10-30 03:12:03 +01:00
co-authored by Claude Opus 5.5
parent 3ca05fcfbc
commit 10cf93c27e
@@ -0,0 +1,56 @@
# WP-B Importer Implementation Plan
> Executed inline by the WP-B agent, task by task, TDD. Spec: `docs/superpowers/specs/2026-10-30-stcourses-design.md` sections 5, 8, 13.
**Goal:** `packages/feature-import` turns a Packt video zip, a companion code repo, a slide PDF or a folder of videos into an ordinary course folder under `<dataDir>/courses/<id>/`, as background jobs with visible progress, plus an owner-only admin UI at `/admin/import`.
**Architecture:**
- `importFeature(ctx)` returns a `Feature` with job handlers `import.packt-video`, `import.code-repo`, `import.pdf-deck`, `import.video-folder`, `import.transcribe` and owner-only `/api/import/*` routes. `ctx` is read lazily (inside handlers), because the job queue is built from the handlers.
- Courses are built in `<dataDir>/tmp/<job>` and renamed into `<dataDir>/courses/<id>` in one step. Attach jobs (code, deck) only touch courses under `<dataDir>/courses` and write the manifest last, through temp file + rename.
- Media: **copied** (`cp --reflink=auto`, so instant on CoW filesystems), or hard-linked with `options.media = "link"` (falls back to copy across filesystems). Symlinks out of the course are not an option: the media route 404s on paths that escape the course root. The originals are only ever read.
- PDFs: PyMuPDF through `py/pdf_tool.py` run as `python -I` in a subprocess (`STC_PYTHON`, default `python3`). Chosen over pdfjs/mupdf-wasm because it rasterises and extracts text natively, is already installed, is AGPL-compatible and costs no idle memory in the server. If it is missing, the deck is still attached and the job reports a warning.
- Derived data goes to `<dataDir>/cache/`: `decks/<course>/<deck>/p-001.jpg…` + `text.json`, `covers/<course>/*.jpg`, `transcripts/<course>/<lesson>.json`.
## Global Constraints
Inherited unchanged from `docs/superpowers/plans/2026-10-30-stcourses.md` (Bun conventions, AGPL, design tokens, no licensed content in the repo, commit trailer).
## Review Focus
1. Original files are never modified; attach jobs refuse courses outside `<dataDir>/courses` (overlays, authored repos).
2. Paths with spaces/non-ASCII (`S08 - Containerizing Applications`) survive unzip, copy and manifest round trip.
3. Packt parsing is structural (chapter blocks plus the section summary lists), not regexes on "Video N.M:" prefixes.
4. Failure leaves no half-built course visible; a failed PDF/ffmpeg step degrades to a warning where the course is still valid.
5. Routes: 401 without a user, 403 without the owner role.
## Tasks
### Task 1: Utilities (slug, HTML entities, natural sort, exec, manifest IO, sources)
- Files: `src/util/*.ts`, `test/util.test.ts`
- Interfaces: `slugify`, `uniqueCourseId`, `decodeEntities`, `naturalSort`, `run`, `readRawManifest`, `writeManifest` (validates with `ManifestSchema`), `openSource(path)` (zip via `unzip`, or directory).
- [ ] Failing tests for each helper; [ ] implement; [ ] commit.
### Task 2: Fixtures and media tooling
- Files: `test/fixtures.ts` (generates fake Packt zip, repo zip, PDF, video folder in a temp dir), `src/media.ts` (ffprobe duration, ffmpeg frame, copy/link), `src/pdf.ts`, `py/pdf_tool.py`.
- [ ] Failing tests (frame extraction, duration, PDF pages+text); [ ] implement; [ ] commit.
### Task 3: Packt parser and `import.packt-video`
- Files: `src/packt.ts`, `src/jobs/packt-video.ts`, `test/packt.test.ts`
- [ ] Parser test on the synthetic index (summary-only titles, missing chapter); [ ] job test (2 sections, 3 clips, restricted manifest, source filled, course loads `ok`, cover candidates); [ ] implement; [ ] commit.
### Task 4: `import.video-folder`, `import.pdf-deck`
- [ ] Tests: natural order, deck cache pages/text, attach to lesson and module, refuses unmanaged course; [ ] implement; [ ] commit.
### Task 5: `import.code-repo`
- [ ] Tests: folder `S02 - …` attaches to module 2 by number, by name when no number, PDFs become deck/attachment, git URL validation; [ ] implement; [ ] commit.
### Task 6: `import.transcribe`
- [ ] Tests with a fake Whisper server; skipped when `STC_TRANSCRIBE_URL` unset; [ ] implement; [ ] commit.
### Task 7: Routes and `importFeature`
- [ ] Tests: 401/403/owner, enqueue validation, queue integration (SQLite and Postgres), covers; [ ] implement; wire `main.ts`; [ ] commit.
### Task 8: Web UI `importWeb`
- [ ] Page with form, job list with progress, cover picker; register in `apps/web/src/features.ts`; screenshot check; [ ] commit.
### Task 9: Manual check against `/home/user/example_course_content`, report.