From 10cf93c27eec12759cda0dfa0668d04d4093da46 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 03:11:53 +0100 Subject: [PATCH] docs(import): WP-B plan Co-Authored-By: Claude Opus 5.5 --- docs/superpowers/plans/2026-10-30-import.md | 56 +++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-30-import.md diff --git a/docs/superpowers/plans/2026-10-30-import.md b/docs/superpowers/plans/2026-10-30-import.md new file mode 100644 index 0000000..ba21859 --- /dev/null +++ b/docs/superpowers/plans/2026-10-30-import.md @@ -0,0 +1,56 @@ +# WP-B Importer Implementation Plan + +> Executed inline by the WP-B agent, task by task, TDD. Spec: `docs/superpowers/specs/2026-10-30-stcourses-design.md` sections 5, 8, 13. + +**Goal:** `packages/feature-import` turns a Packt video zip, a companion code repo, a slide PDF or a folder of videos into an ordinary course folder under `/courses//`, as background jobs with visible progress, plus an owner-only admin UI at `/admin/import`. + +**Architecture:** +- `importFeature(ctx)` returns a `Feature` with job handlers `import.packt-video`, `import.code-repo`, `import.pdf-deck`, `import.video-folder`, `import.transcribe` and owner-only `/api/import/*` routes. `ctx` is read lazily (inside handlers), because the job queue is built from the handlers. +- Courses are built in `/tmp/` and renamed into `/courses/` in one step. Attach jobs (code, deck) only touch courses under `/courses` and write the manifest last, through temp file + rename. +- Media: **copied** (`cp --reflink=auto`, so instant on CoW filesystems), or hard-linked with `options.media = "link"` (falls back to copy across filesystems). Symlinks out of the course are not an option: the media route 404s on paths that escape the course root. The originals are only ever read. +- PDFs: PyMuPDF through `py/pdf_tool.py` run as `python -I` in a subprocess (`STC_PYTHON`, default `python3`). Chosen over pdfjs/mupdf-wasm because it rasterises and extracts text natively, is already installed, is AGPL-compatible and costs no idle memory in the server. If it is missing, the deck is still attached and the job reports a warning. +- Derived data goes to `/cache/`: `decks///p-001.jpg…` + `text.json`, `covers//*.jpg`, `transcripts//.json`. + +## Global Constraints + +Inherited unchanged from `docs/superpowers/plans/2026-10-30-stcourses.md` (Bun conventions, AGPL, design tokens, no licensed content in the repo, commit trailer). + +## Review Focus + +1. Original files are never modified; attach jobs refuse courses outside `/courses` (overlays, authored repos). +2. Paths with spaces/non-ASCII (`S08 - Containerizing Applications`) survive unzip, copy and manifest round trip. +3. Packt parsing is structural (chapter blocks plus the section summary lists), not regexes on "Video N.M:" prefixes. +4. Failure leaves no half-built course visible; a failed PDF/ffmpeg step degrades to a warning where the course is still valid. +5. Routes: 401 without a user, 403 without the owner role. + +## Tasks + +### Task 1: Utilities (slug, HTML entities, natural sort, exec, manifest IO, sources) +- Files: `src/util/*.ts`, `test/util.test.ts` +- Interfaces: `slugify`, `uniqueCourseId`, `decodeEntities`, `naturalSort`, `run`, `readRawManifest`, `writeManifest` (validates with `ManifestSchema`), `openSource(path)` (zip via `unzip`, or directory). +- [ ] Failing tests for each helper; [ ] implement; [ ] commit. + +### Task 2: Fixtures and media tooling +- Files: `test/fixtures.ts` (generates fake Packt zip, repo zip, PDF, video folder in a temp dir), `src/media.ts` (ffprobe duration, ffmpeg frame, copy/link), `src/pdf.ts`, `py/pdf_tool.py`. +- [ ] Failing tests (frame extraction, duration, PDF pages+text); [ ] implement; [ ] commit. + +### Task 3: Packt parser and `import.packt-video` +- Files: `src/packt.ts`, `src/jobs/packt-video.ts`, `test/packt.test.ts` +- [ ] Parser test on the synthetic index (summary-only titles, missing chapter); [ ] job test (2 sections, 3 clips, restricted manifest, source filled, course loads `ok`, cover candidates); [ ] implement; [ ] commit. + +### Task 4: `import.video-folder`, `import.pdf-deck` +- [ ] Tests: natural order, deck cache pages/text, attach to lesson and module, refuses unmanaged course; [ ] implement; [ ] commit. + +### Task 5: `import.code-repo` +- [ ] Tests: folder `S02 - …` attaches to module 2 by number, by name when no number, PDFs become deck/attachment, git URL validation; [ ] implement; [ ] commit. + +### Task 6: `import.transcribe` +- [ ] Tests with a fake Whisper server; skipped when `STC_TRANSCRIBE_URL` unset; [ ] implement; [ ] commit. + +### Task 7: Routes and `importFeature` +- [ ] Tests: 401/403/owner, enqueue validation, queue integration (SQLite and Postgres), covers; [ ] implement; wire `main.ts`; [ ] commit. + +### Task 8: Web UI `importWeb` +- [ ] Page with form, job list with progress, cover picker; register in `apps/web/src/features.ts`; screenshot check; [ ] commit. + +### Task 9: Manual check against `/home/user/example_course_content`, report.