STCourses/docs/superpowers/plans/2026-10-30-import.md

WP-B Importer Implementation Plan

Executed inline by the WP-B agent, task by task, TDD. Spec: docs/superpowers/specs/2026-10-30-stcourses-design.md sections 5, 8, 13.

Goal: packages/feature-import turns a Packt video zip, a companion code repo, a slide PDF or a folder of videos into an ordinary course folder under <dataDir>/courses/<id>/, as background jobs with visible progress, plus an owner-only admin UI at /admin/import.

Architecture:

• importFeature(ctx) returns a Feature with job handlers import.packt-video, import.code-repo, import.pdf-deck, import.video-folder, import.transcribe and owner-only /api/import/* routes. ctx is read lazily (inside handlers), because the job queue is built from the handlers.
• Courses are built in <dataDir>/tmp/<job> and renamed into <dataDir>/courses/<id> in one step. Attach jobs (code, deck) only touch courses under <dataDir>/courses and write the manifest last, through temp file + rename.
• Media: copied (cp --reflink=auto, so instant on CoW filesystems), or hard-linked with options.media = "link" (falls back to copy across filesystems). Symlinks out of the course are not an option: the media route 404s on paths that escape the course root. The originals are only ever read.
• PDFs: PyMuPDF through py/pdf_tool.py run as python -I in a subprocess (STC_PYTHON, default python3). Chosen over pdfjs/mupdf-wasm because it rasterises and extracts text natively, is already installed, is AGPL-compatible and costs no idle memory in the server. If it is missing, the deck is still attached and the job reports a warning.
• Derived data goes to <dataDir>/cache/: decks/<course>/<deck>/p-001.jpg… + text.json, covers/<course>/*.jpg, transcripts/<course>/<lesson>.json.

Global Constraints

Inherited unchanged from docs/superpowers/plans/2026-10-30-stcourses.md (Bun conventions, AGPL, design tokens, no licensed content in the repo, commit trailer).

Review Focus

1. Original files are never modified; attach jobs refuse courses outside <dataDir>/courses (overlays, authored repos).
2. Paths with spaces/non-ASCII (S08 - Containerizing Applications) survive unzip, copy and manifest round trip.
3. Packt parsing is structural (chapter blocks plus the section summary lists), not regexes on "Video N.M:" prefixes.
4. Failure leaves no half-built course visible; a failed PDF/ffmpeg step degrades to a warning where the course is still valid.
5. Routes: 401 without a user, 403 without the owner role.

Tasks

Task 1: Utilities (slug, HTML entities, natural sort, exec, manifest IO, sources)

• Files: src/util/*.ts, test/util.test.ts
• Interfaces: slugify, uniqueCourseId, decodeEntities, naturalSort, run, readRawManifest, writeManifest (validates with ManifestSchema), openSource(path) (zip via unzip, or directory).
• Failing tests for each helper; [ ] implement; [ ] commit.

Task 2: Fixtures and media tooling

• Files: test/fixtures.ts (generates fake Packt zip, repo zip, PDF, video folder in a temp dir), src/media.ts (ffprobe duration, ffmpeg frame, copy/link), src/pdf.ts, py/pdf_tool.py.
• Failing tests (frame extraction, duration, PDF pages+text); [ ] implement; [ ] commit.

Task 3: Packt parser and import.packt-video

• Files: src/packt.ts, src/jobs/packt-video.ts, test/packt.test.ts
• Parser test on the synthetic index (summary-only titles, missing chapter); [ ] job test (2 sections, 3 clips, restricted manifest, source filled, course loads ok, cover candidates); [ ] implement; [ ] commit.

Task 4: import.video-folder, import.pdf-deck

• Tests: natural order, deck cache pages/text, attach to lesson and module, refuses unmanaged course; [ ] implement; [ ] commit.

Task 5: import.code-repo

• Tests: folder S02 - … attaches to module 2 by number, by name when no number, PDFs become deck/attachment, git URL validation; [ ] implement; [ ] commit.

Task 6: import.transcribe

• Tests with a fake Whisper server; skipped when STC_TRANSCRIBE_URL unset; [ ] implement; [ ] commit.

Task 7: Routes and importFeature

• Tests: 401/403/owner, enqueue validation, queue integration (SQLite and Postgres), covers; [ ] implement; wire main.ts; [ ] commit.

Task 8: Web UI importWeb

• Page with form, job list with progress, cover picker; register in apps/web/src/features.ts; screenshot check; [ ] commit.

Task 9: Manual check against /home/user/example_course_content, report.