merge: v1.1 edit imported courses in the studio
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas Winther
@@ -22,7 +22,7 @@ Layout after the review (controller ruling on I4):
|
||||
|
||||
Proposal: main.ts calls `createCourseRepository(studioCourseDirs(config))`, or `loadConfig` appends both dirs.
|
||||
|
||||
Imported courses (`source.kind` other than `authored`, with their root under `<dataDir>/courses`) are read-only in the studio for v1, with the message "Imported course: read-only in the studio for now". There is a `TODO(studio)` in `feature.ts` (`readOnlyMessage`) for converting one into a studio-managed repo. The studio refuses to create a course whose id already exists as `<dataDir>/courses/<id>`. WP-B should likewise refuse an import id that `<dataDir>/studio/repos/<id>.git` already uses.
|
||||
Imported courses (`source.kind` other than `authored`, with their root under `<dataDir>/courses`) are read-only in the studio for v1, with the message "Imported course: read-only in the studio for now". There was a `TODO(studio)` in `feature.ts` (`readOnlyMessage`) for converting one into a studio-managed repo; v1.1 S resolved it with the owner-only "Make editable in the studio" action (see `docs/operations.md`, "Imported courses in the studio"). The studio refuses to create a course whose id already exists as `<dataDir>/courses/<id>`. WP-B should likewise refuse an import id that `<dataDir>/studio/repos/<id>.git` already uses.
|
||||
|
||||
## 3. Live preview: swap in the learner `LessonView`
|
||||
|
||||
|
||||
+76
-1
@@ -22,7 +22,7 @@ Mounts (all course mounts are read-only; STCourses never changes your originals)
|
||||
|
||||
| Host | Container | What |
|
||||
|---|---|---|
|
||||
| volume `stc-data` | `/data` | database (SQLite), imported courses (`/data/courses`), studio repos (`/data/studio/repos`) and their published worktrees (`/data/worktrees`), caches (`/data/cache`), AI drafts for non-studio courses (`/data/drafts`) |
|
||||
| volume `stc-data` | `/data` | database (SQLite), imported courses (`/data/courses`), studio repos (`/data/studio/repos`) and their published worktrees (`/data/worktrees`), media of imports made editable in the studio (`/data/media`), caches (`/data/cache`), AI drafts for non-studio courses (`/data/drafts`) |
|
||||
| `STC_COURSES_HOST` (default `./courses`) | `/courses` | course repos, one folder with a `stcourse.yaml` per course |
|
||||
| `STC_IMPORT_HOST` (default `./import-inbox`) | `/import` | files the owner may import (Packt zips, video folders, PDFs, code zips) |
|
||||
|
||||
@@ -78,6 +78,81 @@ Without Docker: `bun install`, then `STC_SECRET=… NODE_ENV=production bun apps
|
||||
- Course media (`/api/media/…`) is sent with `X-Content-Type-Options: nosniff`, and SVG, HTML and JavaScript files
|
||||
with `Content-Security-Policy: sandbox`, so author content cannot run script on the app's origin.
|
||||
|
||||
## Imported courses in the studio
|
||||
|
||||
Imported courses (`<data>/courses/<id>`) are read-only in the studio until the owner chooses **Make editable in the
|
||||
studio** on the course's studio page. That runs a background job (`studio.convert-import`) which:
|
||||
|
||||
1. commits the manifest, lessons, transcripts, small images, small PDFs and code into a new studio repo,
|
||||
`<data>/studio/repos/<id>.git`, checked out as `<data>/worktrees/<id>/published`. The manifest is committed
|
||||
unchanged: the course keeps its id, its visibility (restricted after an import) and its `source.kind`, so only the
|
||||
owner can later make it public or signed-in. ACL entries, progress and notes are keyed by course and lesson id
|
||||
and are kept;
|
||||
2. keeps **large media out of git**: every video or audio file (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus
|
||||
flac`) and any other file over 8 MiB. Each is listed with its path, size and SHA-256 in `.stcourses/media.json`
|
||||
in the repo. They are kept out of `git status` by the repo's `info/exclude` (shared by every worktree), not by
|
||||
the course's `.gitignore`, and the studio never counts a listed file as a change. So an author or a remote
|
||||
rewriting `.gitignore` cannot block publishing or Sync. Git LFS is not used: it is not in the image and would
|
||||
need an LFS server;
|
||||
3. **moves the import folder to `<data>/media/<id>`** (decided over keeping it in place: a manifest left in
|
||||
`<data>/courses/<id>` would be found as a second course with the same id, and the importer's "add to course" and
|
||||
transcription jobs would keep changing a folder learners no longer see). `<data>/media` is never scanned for
|
||||
courses and nothing writes to it afterwards. File permissions are left alone, because an import made in `link`
|
||||
mode shares its files with your originals;
|
||||
4. puts every referenced file into the published worktree, and later into each author's draft worktree, as a
|
||||
**hard link** to `<data>/media/<id>/<path>`. The manifest paths are unchanged, so `/api/media/<id>/…` serves the
|
||||
same URLs with Range support, and no extra disk space is used. If `<data>/worktrees` and `<data>/media` are on
|
||||
different filesystems, each worktree gets a **full copy** instead: the published one plus one per author who
|
||||
opens a draft. Keep the data directory on one filesystem.
|
||||
|
||||
In the studio, referenced media cannot be replaced or deleted (save, upload and conflict resolution refuse them with
|
||||
409); upload a new file under another name instead. A studio write never goes through a hard link: a file with more
|
||||
than one link is replaced, not overwritten. Publish and Sync refuse a commit that tracks a file at a referenced media
|
||||
path, since it would silently replace the media.
|
||||
|
||||
**Importer jobs and conversion.** Both use the same per-course lock. Conversion is refused (409, "an import job for
|
||||
this course is still running") while an importer job for that course is queued or running. An importer job that
|
||||
starts after the conversion fails with "this course is now edited in the studio" and writes nothing.
|
||||
|
||||
**Crash safety.** The import folder is first renamed to `<data>/courses/.converting-<id>` (dot-folders are never
|
||||
scanned), before the published worktree is checked out, so the course is never loaded twice. Until that checkout is
|
||||
done, a failure undoes everything. After it, the conversion is committed and only the move remains:
|
||||
|
||||
- on one filesystem, one `rename` to `<data>/media/<id>`;
|
||||
- across filesystems, a copy into `<data>/media/.staging-<id>.<n>`, `fsync`, a size and SHA-256 check against
|
||||
`media.json`, a `rename` into place, and then the source is renamed to `<data>/courses/.converted-<id>.<n>` and
|
||||
deleted.
|
||||
|
||||
At server start and before every conversion, STCourses recovers each course deterministically:
|
||||
|
||||
- staging folders are deleted;
|
||||
- a `.converting-` folder without a finished repo is renamed back (the next attempt starts over);
|
||||
- a `.converting-` folder with a finished repo has its move finished;
|
||||
- `.converted-` folders are deleted once `<data>/media/<id>` is complete.
|
||||
|
||||
A job left running by a restart is requeued, and re-running a finished conversion reports success.
|
||||
|
||||
Things to know:
|
||||
|
||||
- Back up `<data>/media` together with `<data>/studio`: the repo alone does not contain the videos.
|
||||
- **Sync** pushes only the git repo. A clone of the remote has the manifest, lessons and `.stcourses/media.json`, but
|
||||
not the media files themselves; copy `<data>/media/<id>` alongside if you move the course to another server.
|
||||
- If a worktree loses a media file (for example after a manual `git clean -x`), the next publish links it back into
|
||||
the published worktree, and a new draft worktree gets every file. A file missing from `<data>/media/<id>` shows
|
||||
as a course diagnostic ("lesson file not found"), never a crash.
|
||||
- After conversion the importer can no longer add decks, code or transcripts to that course (it only edits
|
||||
`<data>/courses`). Add them in the studio.
|
||||
- `<data>/media` is fixed under the data directory (there is no separate setting).
|
||||
- **Discovery skips dot-folders.** Every course directory (`STC_COURSE_DIRS`, `<data>/courses`, `<data>/worktrees`)
|
||||
is scanned to depth 2 for `stcourse.yaml`, and folders whose name starts with `.` are never entered. Do not give a
|
||||
course folder a name that starts with a dot. The conversion's `.converting-<id>` and `.converted-<id>.<n>` folders
|
||||
rely on this.
|
||||
- **A failed conversion.** If the media move fails after the commit (disk full, or a copy that fails its checksum),
|
||||
learners are switched to the studio course right away; its media are hard links into the hidden import folder.
|
||||
The course's studio page shows **Conversion incomplete** with a **Retry** button for the owner. A retry (or the
|
||||
next server start) finishes the move. A failure before the commit is rolled back, and learners keep the import
|
||||
unchanged.
|
||||
|
||||
## Memory budget
|
||||
|
||||
The deployment must stay under 2 GB idle. Measured with this compose file (`--profile postgres`), the image built
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# STCourses v1.1 — S: Edit imported courses in the studio
|
||||
|
||||
> **For agentic workers:** executed inline (superpowers:executing-plans) by the S package owner, TDD, one commit per task. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** an owner-only action, "Make editable in the studio", turns an imported course (`<dataDir>/courses/<id>`) into a studio course (`<dataDir>/studio/repos/<id>.git` plus `<dataDir>/worktrees/<id>/published`) with the same id, so authors can edit and publish it and learners keep their progress and notes. Resolves `TODO(studio): convert imported course to a studio-managed repo` (ruling R9).
|
||||
|
||||
**Requirements:** `docs/superpowers/plans/2026-10-30-v1-1.md` ("Global Constraints", package S); spec sections 5, 8, 9.1, 12, 13; `CLAUDE.md` project rules.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. **Large media stay out of git, referenced by path, size and checksum (spec 5).** No Git LFS (not installed in the image, and it would need a remote). A file is "large media" when its extension is video/audio (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus flac`) or it is over 8 MiB. Everything else (manifest, Markdown, small images, small PDFs, code) is committed.
|
||||
- The repo gets `.stcourses/media.json` (`{ schema: 1, files: [{ path, size, sha256 }] }`) and a root `.gitignore` with one anchored, escaped line per referenced path (an existing root `.gitignore` is kept above our block).
|
||||
- Every studio worktree (published and each author's draft) gets the referenced files materialised at their manifest paths as **hard links** to the media store (copy with `--reflink=auto` across filesystems). Manifest paths do not change; core's media route (Range 206/416, 404 outside the root) serves them unchanged. No symlinks: the loader's `resolveInside` and the studio's write rules both refuse paths that leave the course root.
|
||||
2. **The import folder is moved to `<dataDir>/media/<id>`** (a `rename`, same filesystem) rather than kept in place. Reasons: a manifest left in `<dataDir>/courses/<id>` would be discovered as a second course with the same id (and would win: imported dirs are scanned before worktrees); the importer's "add to course" and transcription jobs would keep editing a folder learners no longer see. `<dataDir>/media` is never scanned. Nothing writes there after the move; permissions are not changed, because the importer's `link` mode shares inodes with the owner's originals.
|
||||
3. **Conversion runs as a job** (`studio.convert-import`), because hashing gigabytes of video can exceed Bun's 10 s idle timeout. The route checks everything it can first and answers `202 { jobId }`; the UI polls `GET /api/studio/jobs/:jobId`. The job is restart-safe: a conversion interrupted before the move is rolled back and redone; one interrupted after the repo exists only finishes the move.
|
||||
4. **Same id, same manifest.** The manifest is committed unchanged: `visibility` (restricted by default for imports) and `source.kind` stay, so the v1 owner-only rule at publish still applies (only the owner can make it public or signed-in, or change `source.kind`). ACL rows, progress and notes are keyed by course and lesson ids, which do not change.
|
||||
5. **Referenced media are read-only in the studio.** Saves, uploads, deletes and conflict resolutions that touch a referenced path or `.stcourses/media.json` get 409 with a message to upload under a new name. As defence in depth, any studio write to a file with more than one hard link unlinks it first, so a write can never reach a shared inode.
|
||||
6. **Hostile input:** symlinks and special files in the import folder are skipped (never followed), `.git` folders are skipped, a large file whose name contains a newline (inexpressible in `.gitignore`) refuses the conversion, and the published worktree must be clean after checkout or the conversion rolls back. Materialisation checks each reference with `checkRelPath`, requires a regular file whose real path is inside `<dataDir>/media/<id>`, and writes through `safeWritePath` (refuses symlinks in the destination path).
|
||||
|
||||
## Global Constraints
|
||||
- Everything in `CLAUDE.md` "Project rules" and the v1.1 "Global Constraints" applies. No DB change is needed (no migration).
|
||||
- v1 safeguards stay: symlink-refusing writes, the remote allowlist, the owner-only visibility rule at publish, core's CSRF Origin check (applies to the new POST route automatically), repos at `<dataDir>/studio/repos/<id>.git`.
|
||||
- Synthetic fixtures only. `bun test`, `bun run typecheck` and the Postgres run (`STC_TEST_PG_URL`) stay green.
|
||||
|
||||
## File structure
|
||||
- Create `packages/feature-studio/src/server/media-refs.ts`: classification, `.gitignore` escaping, hashing, reference file read/write.
|
||||
- Create `packages/feature-studio/src/server/convert.ts`: `convertImported` (stage, commit, worktree, materialise, move).
|
||||
- Modify `packages/feature-studio/src/server/store.ts`: `mediaDir` option, `materializeMedia`, draft materialisation, write guards, `listFiles` includes referenced media, `convertImported` wiring.
|
||||
- Modify `packages/feature-studio/src/server/feature.ts`: `POST /api/studio/courses/:courseId/convert`, `GET /api/studio/jobs/:jobId`, job handler, `convertible` flag; remove the TODO.
|
||||
- Modify `packages/feature-studio/src/ui/*`: convert panel in the read-only view, imported tag on the studio home.
|
||||
- Create tests: `packages/feature-studio/test/media-refs.test.ts`, `convert.test.ts`, `convert-routes.test.ts`, `convert-ui.test.tsx`; `apps/server/test/studio-import-e2e.test.ts`.
|
||||
- Modify `docs/operations.md` (media decision, layout).
|
||||
|
||||
### Task 1: Media references
|
||||
- [ ] Tests: `isLargeMedia` by extension and size; `gitignoreLine` escapes `\ * ? [ ] # !` and trailing spaces, and `git check-ignore` matches exactly the listed paths for names with spaces, brackets, `#`, `!`, non-ASCII and trailing spaces (and does not match siblings); `sha256File` matches `Bun.CryptoHasher`; `parseMediaRefs` rejects bad shapes.
|
||||
- [ ] Implement `media-refs.ts`. Run tests, typecheck. Commit.
|
||||
|
||||
### Task 2: Conversion in the store
|
||||
- [ ] Tests (`convert.test.ts`, synthetic import folder built in the test: manifest, lesson Markdown, a fake `.mp4` with space and non-ASCII name, a > 8 MiB PDF, a nested `.gitignore` that ignores a lesson file, a symlink pointing outside, a `.git` dir):
|
||||
- the bare repo, published worktree and media store exist; the import folder is gone; `isManaged`;
|
||||
- git tracks the manifest, lessons (including the nested-ignored one) and small files, never the video or big PDF; the symlink and `.git` contents are not tracked;
|
||||
- `.stcourses/media.json` lists path, size, sha256; the published worktree is clean; the video in the worktree is a hard link (same inode) to `<dataDir>/media/<id>/…`;
|
||||
- a draft worktree gets the media too; `listFiles` includes them;
|
||||
- saving, uploading or deleting a referenced path or the reference file is refused (409); a write to a hard-linked file never changes the other link;
|
||||
- refusals: not under the imports dir, `source.kind: authored`, manifest id mismatch, `root:` set, already a studio course, media store already present; each leaves the import folder untouched and no repo behind;
|
||||
- resume: a leftover partial repo is rebuilt; a managed course whose import folder still exists only finishes the move;
|
||||
- publish after an edit keeps the media in the published worktree.
|
||||
- [ ] Implement `convert.ts` and the store changes. Run studio tests, typecheck. Commit.
|
||||
|
||||
### Task 3: Routes and job
|
||||
- [ ] Tests (`convert-routes.test.ts`, real job queue from `@stc/core`):
|
||||
- owner `POST …/convert` gives 202 and a job id; polling reaches `done`; the course is then studio-managed, served from the published worktree, still restricted, with ACL rows unchanged;
|
||||
- an author or learner gets 403; a foreign Origin gets 403 (CSRF); an overlay or studio course gets 409; an unknown course 404; a second request while one is queued gets 409;
|
||||
- `GET /api/studio/courses/:id` for an imported course says `convertible: true` for the owner only;
|
||||
- after conversion a non-owner author can edit and publish a lesson title, but cannot publish `visibility: public` (403); the owner can.
|
||||
- [ ] Implement. Run tests, typecheck. Commit.
|
||||
|
||||
### Task 4: Studio UI
|
||||
- [ ] Test (`convert-ui.test.tsx`, server render): the read-only panel shows "Make editable in the studio" and the explanation only when `convertible`; progress and error states render.
|
||||
- [ ] Implement `ConvertPanel` (button, job polling, then reload into the workbench) and the imported tag on the home list. Existing tokens and classes only; no new colours, fonts or motion. Commit.
|
||||
|
||||
### Task 5: End-to-end
|
||||
- [ ] `apps/server/test/studio-import-e2e.test.ts`, SQLite and Postgres: production feature set; owner signs up; a synthetic video folder is imported through `/api/import/jobs`; a learner is granted access by ACL, marks a lesson done and writes a note; owner converts; edits a lesson title in the studio; publishes. The learner sees the new title, the same progress and notes, media answer `Range` with 206 and a bad range with 416; anonymous users still get 404.
|
||||
- [ ] Commit.
|
||||
|
||||
### Task 6: Docs, smoke run, report
|
||||
- [ ] `docs/operations.md`: the media decision and the new `<dataDir>/media` folder; backups; that media are not pushed to studio remotes.
|
||||
- [ ] Dev server on port 3492 with `STC_DATA_DIR=/tmp/stc-v11-studio`: import, convert, edit, publish by curl.
|
||||
- [ ] Full `bun test`, typecheck, Postgres run. Commit. Write the report.
|
||||
@@ -181,6 +181,7 @@ Phase 0 (section 10) pins down the exact schema. These principles are fixed:
|
||||
- The two reference courses must load by adding only a manifest and front matter, without moving or rewriting their existing files.
|
||||
- Their existing tooling (test scripts, `tools/*.mjs`) must keep working.
|
||||
- **Derived data lives in a cache directory that can be rebuilt from the files:** search index, slide images and text, transcripts, video posters and AI embeddings.
|
||||
- **Discovery** (v1.1 note): the platform finds `stcourse.yaml` at depth 0–2 under each course directory and skips folders whose name starts with `.` (and `node_modules`). A course folder must therefore not be a dot-folder. The studio relies on this to hide an import folder while it is converted (`.converting-<id>`, `.converted-<id>.<n>`).
|
||||
|
||||
## 6. Accounts, roles and visibility
|
||||
|
||||
|
||||
Reference in New Issue
Block a user