fix(studio): review round 1 for imported-course editing
- Media exclusions move from the editable root .gitignore to the repo's info/exclude, and dirtyFiles never counts files listed in media.json, so an author's or remote's .gitignore cannot block publish or Sync. Publish and Sync refuse commits that track a file over referenced media. - Importer jobs (code-repo, pdf-deck, transcribe) and the cover route run under the course lock now shared with the studio (core withCourseLock) and re-check the course inside it; after conversion they fail with "this course is now edited in the studio". Convert answers 409 while an import job for the course is queued or running, and the job re-checks. - Crash-safe conversion: the import folder is hidden (.converting-<id>) before the published worktree exists; cross-filesystem moves copy into a staging dir, fsync, verify size and sha256, then rename; the source is retired as .converted-<id>.<n>. Recovery runs at server start and before every convert; a re-run of a finished conversion reports success. The loader skips dot-folders. - Minor: no raw NUL bytes in source, link fallback only on EXDEV/EPERM/EMLINK, referenced media are not read whole by the editor, missing media are logged, mediaDir JSDoc, docs on per-author copies across filesystems. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
08e62ff683
commit
771d755da4
+32
-8
@@ -90,24 +90,47 @@ studio** on the course's studio page. That runs a background job (`studio.conver
|
||||
and are kept;
|
||||
2. keeps **large media out of git**: every video or audio file (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus
|
||||
flac`) and any other file over 8 MiB. Each is listed with its path, size and SHA-256 in `.stcourses/media.json`
|
||||
in the repo and excluded by one anchored line in the repo's root `.gitignore` (a `.gitignore` the course already
|
||||
had is kept above that block). Git LFS is not used: it is not in the image and would need an LFS server;
|
||||
in the repo. They are kept out of `git status` by the repo's `info/exclude` (shared by every worktree), not by
|
||||
the course's `.gitignore`, and the studio never counts a listed file as a change. So an author or a remote
|
||||
rewriting `.gitignore` cannot block publishing or Sync. Git LFS is not used: it is not in the image and would
|
||||
need an LFS server;
|
||||
3. **moves the import folder to `<data>/media/<id>`** (decided over keeping it in place: a manifest left in
|
||||
`<data>/courses/<id>` would be found as a second course with the same id, and the importer's "add to course" and
|
||||
transcription jobs would keep changing a folder learners no longer see). `<data>/media` is never scanned for
|
||||
courses and nothing writes to it afterwards. File permissions are left alone, because an import made in `link`
|
||||
mode shares its files with your originals;
|
||||
4. puts every referenced file into the published worktree, and later into each author's draft worktree, as a
|
||||
**hard link** to `<data>/media/<id>/<path>` (a copy if they are on different filesystems). The manifest paths are
|
||||
unchanged, so `/api/media/<id>/…` serves the same URLs with Range support, and no extra disk space is used.
|
||||
**hard link** to `<data>/media/<id>/<path>`. The manifest paths are unchanged, so `/api/media/<id>/…` serves the
|
||||
same URLs with Range support, and no extra disk space is used. If `<data>/worktrees` and `<data>/media` are on
|
||||
different filesystems, each worktree gets a **full copy** instead: the published one plus one per author who
|
||||
opens a draft. Keep the data directory on one filesystem.
|
||||
|
||||
In the studio, referenced media cannot be replaced or deleted (save, upload and conflict resolution refuse them with
|
||||
409); upload a new file under another name instead. A studio write never goes through a hard link: a file with more
|
||||
than one link is replaced, not overwritten.
|
||||
than one link is replaced, not overwritten. Publish and Sync refuse a commit that tracks a file at a referenced media
|
||||
path, since it would silently replace the media.
|
||||
|
||||
Interrupted conversions are safe to retry, and a job left running by a restart is retried automatically: if no
|
||||
published worktree was made, the partial repo is removed and the conversion starts again; if the repo exists but the
|
||||
import folder was not moved yet, only the move is done.
|
||||
**Importer jobs and conversion.** Both use the same per-course lock. Conversion is refused (409, "an import job for
|
||||
this course is still running") while an importer job for that course is queued or running. An importer job that
|
||||
starts after the conversion fails with "this course is now edited in the studio" and writes nothing.
|
||||
|
||||
**Crash safety.** The import folder is first renamed to `<data>/courses/.converting-<id>` (dot-folders are never
|
||||
scanned), before the published worktree is checked out, so the course is never loaded twice. Until that checkout is
|
||||
done, a failure undoes everything. After it, the conversion is committed and only the move remains:
|
||||
|
||||
- on one filesystem, one `rename` to `<data>/media/<id>`;
|
||||
- across filesystems, a copy into `<data>/media/.staging-<id>.<n>`, `fsync`, a size and SHA-256 check against
|
||||
`media.json`, a `rename` into place, and then the source is renamed to `<data>/courses/.converted-<id>.<n>` and
|
||||
deleted.
|
||||
|
||||
At server start and before every conversion, STCourses recovers each course deterministically:
|
||||
|
||||
- staging folders are deleted;
|
||||
- a `.converting-` folder without a finished repo is renamed back (the next attempt starts over);
|
||||
- a `.converting-` folder with a finished repo has its move finished;
|
||||
- `.converted-` folders are deleted once `<data>/media/<id>` is complete.
|
||||
|
||||
A job left running by a restart is requeued, and re-running a finished conversion reports success.
|
||||
|
||||
Things to know:
|
||||
|
||||
@@ -119,6 +142,7 @@ Things to know:
|
||||
as a course diagnostic ("lesson file not found"), never a crash.
|
||||
- After conversion the importer can no longer add decks, code or transcripts to that course (it only edits
|
||||
`<data>/courses`). Add them in the studio.
|
||||
- `<data>/media` is fixed under the data directory (there is no separate setting).
|
||||
|
||||
## Memory budget
|
||||
|
||||
|
||||
Reference in New Issue
Block a user