Lucas WintherandClaude Opus 5.5
771d755da4
- Media exclusions move from the editable root .gitignore to the repo's info/exclude, and dirtyFiles never counts files listed in media.json, so an author's or remote's .gitignore cannot block publish or Sync. Publish and Sync refuse commits that track a file over referenced media. - Importer jobs (code-repo, pdf-deck, transcribe) and the cover route run under the course lock now shared with the studio (core withCourseLock) and re-check the course inside it; after conversion they fail with "this course is now edited in the studio". Convert answers 409 while an import job for the course is queued or running, and the job re-checks. - Crash-safe conversion: the import folder is hidden (.converting-<id>) before the published worktree exists; cross-filesystem moves copy into a staging dir, fsync, verify size and sha256, then rename; the source is retired as .converted-<id>.<n>. Recovery runs at server start and before every convert; a re-run of a finished conversion reports success. The loader skips dot-folders. - Minor: no raw NUL bytes in source, link fallback only on EXDEV/EPERM/EMLINK, referenced media are not read whole by the editor, missing media are logged, mediaDir JSDoc, docs on per-author copies across filesystems. Co-Authored-By: Claude Opus 5.5 <[email protected]>
167 lines
12 KiB
Markdown
167 lines
12 KiB
Markdown
# Operating STCourses
|
|
|
|
One server process (Bun) serves the API and the web app. Course content lives in folders; the database holds user
|
|
data only (accounts, progress, notes, attempts, settings), so deleting the database loses user data but never a
|
|
course.
|
|
|
|
## Run with Docker Compose
|
|
|
|
```sh
|
|
# SQLite in the data volume (the default)
|
|
STC_SECRET="$(openssl rand -base64 48)" docker compose up -d
|
|
|
|
# Postgres instead
|
|
STC_SECRET="$(openssl rand -base64 48)" \
|
|
STC_DATABASE_URL=postgres://stc:stc@postgres:5432/stc \
|
|
docker compose --profile postgres up -d
|
|
```
|
|
|
|
The first account that signs up becomes the owner. Open `http://localhost:3000` (or `STC_HOST_PORT`).
|
|
|
|
Mounts (all course mounts are read-only; STCourses never changes your originals):
|
|
|
|
| Host | Container | What |
|
|
|---|---|---|
|
|
| volume `stc-data` | `/data` | database (SQLite), imported courses (`/data/courses`), studio repos (`/data/studio/repos`) and their published worktrees (`/data/worktrees`), media of imports made editable in the studio (`/data/media`), caches (`/data/cache`), AI drafts for non-studio courses (`/data/drafts`) |
|
|
| `STC_COURSES_HOST` (default `./courses`) | `/courses` | course repos, one folder with a `stcourse.yaml` per course |
|
|
| `STC_IMPORT_HOST` (default `./import-inbox`) | `/import` | files the owner may import (Packt zips, video folders, PDFs, code zips) |
|
|
|
|
An overlay in `overlays/` (baked into the image) names its course repo by absolute path in `root:`. Mount that repo
|
|
read-only at the same path inside the container, for example
|
|
`/home/me/functional-programming-course:/home/me/functional-programming-course:ro`. If the repo is missing, the
|
|
course shows as broken with a diagnostic and every other course keeps working.
|
|
|
|
Back up the `stc-data` volume (and the Postgres volume `stc-pg` when you use Postgres).
|
|
|
|
## Configuration
|
|
|
|
| Variable | Default | Meaning |
|
|
|---|---|---|
|
|
| `STC_SECRET` | none (required in production) | encrypts users' own AI keys and signs sessions; use 32+ random characters |
|
|
| `STC_PUBLIC_URL` | `http://localhost:<port>` | the URL users open; sign-in checks the request origin against it |
|
|
| `STC_PORT` | `3000` | listen port |
|
|
| `STC_DATABASE_URL` | `sqlite://./data/stcourses.db` (image: `sqlite:///data/stcourses.db`) | `sqlite://<path>` or `postgres://…` |
|
|
| `STC_DATA_DIR` | `./data` (image: `/data`) | see the table above |
|
|
| `STC_COURSE_DIRS` | `./overlays:./courses` (image: `/app/overlays:/courses`) | `:`-separated folders scanned for courses; `<data>/courses` and `<data>/worktrees` are always added |
|
|
| `STC_IMPORT_ROOTS` | `<data>/import-inbox` (image: `/import`) | `:`-separated folders the importer may read |
|
|
| `STC_UPLOAD_MAX_MB` | `100` | studio upload cap |
|
|
| `STC_PYTHON` | `python3` | Python with PyMuPDF, for PDF page images and text (installed in the image) |
|
|
| `STC_TRANSCRIBE_URL`, `STC_TRANSCRIBE_KEY`, `STC_TRANSCRIBE_MODEL` | unset | Whisper-compatible endpoint for video transcripts; unset disables transcription |
|
|
| `STC_AI_ALLOWED_BASE_URLS` | unset | `,`-separated origins learners' own AI keys may use besides api.anthropic.com and api.openai.com |
|
|
| `STC_AUTH_<GITHUB/GOOGLE/MICROSOFT/GITLAB/DISCORD>_ID`, `_SECRET` | unset | social sign-in |
|
|
| `STC_AUTH_OIDC_ISSUER`, `_ID`, `_SECRET`, `_NAME`, `_SCOPES`, `_DISCOVERY_URL`, `STC_AUTH_GROUPS_CLAIM` | unset | generic OIDC sign-in; group claims drive restricted visibility |
|
|
| `STC_ALLOWED_ORIGINS` | unset | extra `,`-separated origins (e.g. `http://192.168.1.5:3000`) allowed to send writes and sign in, besides `STC_PUBLIC_URL`'s origin. `STC_TRUSTED_ORIGINS` is read as an alias |
|
|
|
|
Without Docker: `bun install`, then `STC_SECRET=… NODE_ENV=production bun apps/server/src/main.ts`. The importer needs
|
|
`ffmpeg`, `ffprobe`, `git`, `unzip` and Python with PyMuPDF on the host.
|
|
|
|
## Cross-site requests and the session cookie
|
|
|
|
- Every state-changing request (POST, PUT, PATCH, DELETE), sign-in included, must carry an `Origin` (or, without
|
|
one, a `Referer`) equal to `STC_PUBLIC_URL`'s origin or one in `STC_ALLOWED_ORIGINS`. Anything else gets 403.
|
|
Open STCourses at exactly that URL. If people reach it by several names (a LAN IP and a hostname), list the others
|
|
in `STC_ALLOWED_ORIGINS`.
|
|
- A write with a body must be `application/json` (uploads: `multipart/form-data`). HTML forms and `text/plain`
|
|
bodies, which another page can send without a CORS preflight, are refused.
|
|
- Exempt: the identity-provider callbacks (SAML ACS and SLO, OAuth/OIDC callbacks under `/api/auth`). The provider
|
|
posts those from its own origin, and Better Auth verifies the signed assertion or the OAuth state itself.
|
|
- The session cookie stays `SameSite=Lax` (Better Auth's default), not `Strict`. OAuth/OIDC sign-in returns from
|
|
the provider by a cross-site top-level navigation, and Better Auth needs its state cookie on that request. Under
|
|
`Strict` the browser withholds it, and social, OIDC and SAML sign-in break. The Origin check above covers the
|
|
same-site cases that `Lax` allows (other ports on the same host, sibling subdomains).
|
|
|
|
## Security headers
|
|
|
|
- The web app carries its Content-Security-Policy as a `<meta>` tag in `apps/web/index.html` (no inline script).
|
|
A meta tag cannot set `frame-ancestors`; if you need clickjacking protection, add
|
|
`Content-Security-Policy: frame-ancestors 'self'` (or `X-Frame-Options: SAMEORIGIN`) in your reverse proxy.
|
|
- Course media (`/api/media/…`) is sent with `X-Content-Type-Options: nosniff`, and SVG, HTML and JavaScript files
|
|
with `Content-Security-Policy: sandbox`, so author content cannot run script on the app's origin.
|
|
|
|
## Imported courses in the studio
|
|
|
|
Imported courses (`<data>/courses/<id>`) are read-only in the studio until the owner chooses **Make editable in the
|
|
studio** on the course's studio page. That runs a background job (`studio.convert-import`) which:
|
|
|
|
1. commits the manifest, lessons, transcripts, small images, small PDFs and code into a new studio repo,
|
|
`<data>/studio/repos/<id>.git`, checked out as `<data>/worktrees/<id>/published`. The manifest is committed
|
|
unchanged: the course keeps its id, its visibility (restricted after an import) and its `source.kind`, so only the
|
|
owner can later make it public or signed-in. ACL entries, progress and notes are keyed by course and lesson id
|
|
and are kept;
|
|
2. keeps **large media out of git**: every video or audio file (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus
|
|
flac`) and any other file over 8 MiB. Each is listed with its path, size and SHA-256 in `.stcourses/media.json`
|
|
in the repo. They are kept out of `git status` by the repo's `info/exclude` (shared by every worktree), not by
|
|
the course's `.gitignore`, and the studio never counts a listed file as a change. So an author or a remote
|
|
rewriting `.gitignore` cannot block publishing or Sync. Git LFS is not used: it is not in the image and would
|
|
need an LFS server;
|
|
3. **moves the import folder to `<data>/media/<id>`** (decided over keeping it in place: a manifest left in
|
|
`<data>/courses/<id>` would be found as a second course with the same id, and the importer's "add to course" and
|
|
transcription jobs would keep changing a folder learners no longer see). `<data>/media` is never scanned for
|
|
courses and nothing writes to it afterwards. File permissions are left alone, because an import made in `link`
|
|
mode shares its files with your originals;
|
|
4. puts every referenced file into the published worktree, and later into each author's draft worktree, as a
|
|
**hard link** to `<data>/media/<id>/<path>`. The manifest paths are unchanged, so `/api/media/<id>/…` serves the
|
|
same URLs with Range support, and no extra disk space is used. If `<data>/worktrees` and `<data>/media` are on
|
|
different filesystems, each worktree gets a **full copy** instead: the published one plus one per author who
|
|
opens a draft. Keep the data directory on one filesystem.
|
|
|
|
In the studio, referenced media cannot be replaced or deleted (save, upload and conflict resolution refuse them with
|
|
409); upload a new file under another name instead. A studio write never goes through a hard link: a file with more
|
|
than one link is replaced, not overwritten. Publish and Sync refuse a commit that tracks a file at a referenced media
|
|
path, since it would silently replace the media.
|
|
|
|
**Importer jobs and conversion.** Both use the same per-course lock. Conversion is refused (409, "an import job for
|
|
this course is still running") while an importer job for that course is queued or running. An importer job that
|
|
starts after the conversion fails with "this course is now edited in the studio" and writes nothing.
|
|
|
|
**Crash safety.** The import folder is first renamed to `<data>/courses/.converting-<id>` (dot-folders are never
|
|
scanned), before the published worktree is checked out, so the course is never loaded twice. Until that checkout is
|
|
done, a failure undoes everything. After it, the conversion is committed and only the move remains:
|
|
|
|
- on one filesystem, one `rename` to `<data>/media/<id>`;
|
|
- across filesystems, a copy into `<data>/media/.staging-<id>.<n>`, `fsync`, a size and SHA-256 check against
|
|
`media.json`, a `rename` into place, and then the source is renamed to `<data>/courses/.converted-<id>.<n>` and
|
|
deleted.
|
|
|
|
At server start and before every conversion, STCourses recovers each course deterministically:
|
|
|
|
- staging folders are deleted;
|
|
- a `.converting-` folder without a finished repo is renamed back (the next attempt starts over);
|
|
- a `.converting-` folder with a finished repo has its move finished;
|
|
- `.converted-` folders are deleted once `<data>/media/<id>` is complete.
|
|
|
|
A job left running by a restart is requeued, and re-running a finished conversion reports success.
|
|
|
|
Things to know:
|
|
|
|
- Back up `<data>/media` together with `<data>/studio`: the repo alone does not contain the videos.
|
|
- **Sync** pushes only the git repo. A clone of the remote has the manifest, lessons and `.stcourses/media.json`, but
|
|
not the media files themselves; copy `<data>/media/<id>` alongside if you move the course to another server.
|
|
- If a worktree loses a media file (for example after a manual `git clean -x`), the next publish links it back into
|
|
the published worktree, and a new draft worktree gets every file. A file missing from `<data>/media/<id>` shows
|
|
as a course diagnostic ("lesson file not found"), never a crash.
|
|
- After conversion the importer can no longer add decks, code or transcripts to that course (it only edits
|
|
`<data>/courses`). Add them in the studio.
|
|
- `<data>/media` is fixed under the data directory (there is no separate setting).
|
|
|
|
## Memory budget
|
|
|
|
The deployment must stay under 2 GB idle. Measured with this compose file (`--profile postgres`), the image built
|
|
from this repository, and four courses mounted read-only: the sample course, Pure Snake and Systemic Design (both
|
|
overlays, repos mounted at their own paths), and the imported Packt course "Mastering Docker Essentials" (68 videos,
|
|
9 modules, a 59-slide deck, code files; about 800 MB of media). Before idling, one owner signed up and opened the
|
|
library, a video lesson with its deck, a Pure Snake lesson and the sample exercise in headless Chrome. Then nothing
|
|
for 5 minutes, and `docker stats --no-stream`:
|
|
|
|
| Container | Memory (idle, after 5 min) | CPU |
|
|
|---|---|---|
|
|
| `stc-app-1` (Bun server, SPA bundle built and cached) | 84.7 MiB | 1.00% |
|
|
| `stc-postgres-1` (postgres:17) | 16.67 MiB | 3.52% |
|
|
| **Total** | **101.4 MiB** (about 5% of the 2 GB budget) | |
|
|
|
|
Measured 2026-10-30 on Linux 6.12 (Docker Compose v5.5.1, `oven/bun:1.3` = Bun 1.3.14). With SQLite there is
|
|
no Postgres container, so the total is the app's figure alone.
|
|
|
|
Video and PDF files are streamed from disk with Range requests; they never sit in memory. The SPA bundle is built
|
|
once, on the first page request, and cached.
|