fix(studio): review round 1 for imported-course editing

- Media exclusions move from the editable root .gitignore to the repo's
  info/exclude, and dirtyFiles never counts files listed in media.json, so an
  author's or remote's .gitignore cannot block publish or Sync. Publish and
  Sync refuse commits that track a file over referenced media.
- Importer jobs (code-repo, pdf-deck, transcribe) and the cover route run
  under the course lock now shared with the studio (core withCourseLock) and
  re-check the course inside it; after conversion they fail with "this course
  is now edited in the studio". Convert answers 409 while an import job for
  the course is queued or running, and the job re-checks.
- Crash-safe conversion: the import folder is hidden (.converting-<id>) before
  the published worktree exists; cross-filesystem moves copy into a staging
  dir, fsync, verify size and sha256, then rename; the source is retired as
  .converted-<id>.<n>. Recovery runs at server start and before every convert;
  a re-run of a finished conversion reports success. The loader skips
  dot-folders.
- Minor: no raw NUL bytes in source, link fallback only on EXDEV/EPERM/EMLINK,
  referenced media are not read whole by the editor, missing media are logged,
  mediaDir JSDoc, docs on per-author copies across filesystems.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
2026-10-30 20:53:50 +01:00
co-authored by Claude Opus 5.5
parent 08e62ff683
commit 771d755da4
17 changed files with 618 additions and 88 deletions
+32 -8
View File
@@ -90,24 +90,47 @@ studio** on the course's studio page. That runs a background job (`studio.conver
and are kept;
2. keeps **large media out of git**: every video or audio file (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus
flac`) and any other file over 8 MiB. Each is listed with its path, size and SHA-256 in `.stcourses/media.json`
in the repo and excluded by one anchored line in the repo's root `.gitignore` (a `.gitignore` the course already
had is kept above that block). Git LFS is not used: it is not in the image and would need an LFS server;
in the repo. They are kept out of `git status` by the repo's `info/exclude` (shared by every worktree), not by
the course's `.gitignore`, and the studio never counts a listed file as a change. So an author or a remote
rewriting `.gitignore` cannot block publishing or Sync. Git LFS is not used: it is not in the image and would
need an LFS server;
3. **moves the import folder to `<data>/media/<id>`** (decided over keeping it in place: a manifest left in
`<data>/courses/<id>` would be found as a second course with the same id, and the importer's "add to course" and
transcription jobs would keep changing a folder learners no longer see). `<data>/media` is never scanned for
courses and nothing writes to it afterwards. File permissions are left alone, because an import made in `link`
mode shares its files with your originals;
4. puts every referenced file into the published worktree, and later into each author's draft worktree, as a
**hard link** to `<data>/media/<id>/<path>` (a copy if they are on different filesystems). The manifest paths are
unchanged, so `/api/media/<id>/…` serves the same URLs with Range support, and no extra disk space is used.
**hard link** to `<data>/media/<id>/<path>`. The manifest paths are unchanged, so `/api/media/<id>/…` serves the
same URLs with Range support, and no extra disk space is used. If `<data>/worktrees` and `<data>/media` are on
different filesystems, each worktree gets a **full copy** instead: the published one plus one per author who
opens a draft. Keep the data directory on one filesystem.
In the studio, referenced media cannot be replaced or deleted (save, upload and conflict resolution refuse them with
409); upload a new file under another name instead. A studio write never goes through a hard link: a file with more
than one link is replaced, not overwritten.
than one link is replaced, not overwritten. Publish and Sync refuse a commit that tracks a file at a referenced media
path, since it would silently replace the media.
Interrupted conversions are safe to retry, and a job left running by a restart is retried automatically: if no
published worktree was made, the partial repo is removed and the conversion starts again; if the repo exists but the
import folder was not moved yet, only the move is done.
**Importer jobs and conversion.** Both use the same per-course lock. Conversion is refused (409, "an import job for
this course is still running") while an importer job for that course is queued or running. An importer job that
starts after the conversion fails with "this course is now edited in the studio" and writes nothing.
**Crash safety.** The import folder is first renamed to `<data>/courses/.converting-<id>` (dot-folders are never
scanned), before the published worktree is checked out, so the course is never loaded twice. Until that checkout is
done, a failure undoes everything. After it, the conversion is committed and only the move remains:
- on one filesystem, one `rename` to `<data>/media/<id>`;
- across filesystems, a copy into `<data>/media/.staging-<id>.<n>`, `fsync`, a size and SHA-256 check against
`media.json`, a `rename` into place, and then the source is renamed to `<data>/courses/.converted-<id>.<n>` and
deleted.
At server start and before every conversion, STCourses recovers each course deterministically:
- staging folders are deleted;
- a `.converting-` folder without a finished repo is renamed back (the next attempt starts over);
- a `.converting-` folder with a finished repo has its move finished;
- `.converted-` folders are deleted once `<data>/media/<id>` is complete.
A job left running by a restart is requeued, and re-running a finished conversion reports success.
Things to know:
@@ -119,6 +142,7 @@ Things to know:
as a course diagnostic ("lesson file not found"), never a crash.
- After conversion the importer can no longer add decks, code or transcripts to that course (it only edits
`<data>/courses`). Add them in the studio.
- `<data>/media` is fixed under the data directory (there is no separate setting).
## Memory budget