STCourses/docs/operations.md

# Operating STCourses

One server process (Bun) serves the API and the web app. Course content lives in folders; the database holds user
data only (accounts, progress, notes, attempts, settings), so deleting the database loses user data but never a
course.

## Run with Docker Compose

```sh
# SQLite in the data volume (the default)
STC_SECRET="$(openssl rand -base64 48)" docker compose up -d

# Postgres instead
STC_SECRET="$(openssl rand -base64 48)" \
STC_DATABASE_URL=postgres://stc:stc@postgres:5432/stc \
  docker compose --profile postgres up -d
```

The first account that signs up becomes the owner. Open `http://localhost:3000` (or `STC_HOST_PORT`).

Mounts (all course mounts are read-only; STCourses never changes your originals):

| Host | Container | What |
|---|---|---|
| volume `stc-data` | `/data` | database (SQLite), imported courses (`/data/courses`), studio repos (`/data/studio/repos`) and their published worktrees (`/data/worktrees`), media of imports made editable in the studio (`/data/media`), caches (`/data/cache`), AI drafts for non-studio courses (`/data/drafts`) |
| `STC_COURSES_HOST` (default `./courses`) | `/courses` | course repos, one folder with a `stcourse.yaml` per course |
| `STC_IMPORT_HOST` (default `./import-inbox`) | `/import` | files the owner may import (Packt zips, video folders, PDFs, code zips) |

An overlay in `overlays/` (baked into the image) names its course repo by absolute path in `root:`. Mount that repo
read-only at the same path inside the container, for example
`/home/me/functional-programming-course:/home/me/functional-programming-course:ro`. If the repo is missing, the
course shows as broken with a diagnostic and every other course keeps working.

Back up the `stc-data` volume (and the Postgres volume `stc-pg` when you use Postgres).

## Configuration

| Variable | Default | Meaning |
|---|---|---|
| `STC_SECRET` | none (required in production) | encrypts users' own AI keys and signs sessions; use 32+ random characters |
| `STC_PUBLIC_URL` | `http://localhost:<port>` | the URL users open; sign-in checks the request origin against it |
| `STC_PORT` | `3000` | listen port |
| `STC_DATABASE_URL` | `sqlite://./data/stcourses.db` (image: `sqlite:///data/stcourses.db`) | `sqlite://<path>` or `postgres://…` |
| `STC_DATA_DIR` | `./data` (image: `/data`) | see the table above |
| `STC_COURSE_DIRS` | `./overlays:./courses` (image: `/app/overlays:/courses`) | `:`-separated folders scanned for courses; `<data>/courses` and `<data>/worktrees` are always added |
| `STC_IMPORT_ROOTS` | `<data>/import-inbox` (image: `/import`) | `:`-separated folders the importer may read |
| `STC_UPLOAD_MAX_MB` | `100` | studio upload cap |
| `STC_PYTHON` | `python3` | Python with PyMuPDF, for PDF page images and text (installed in the image) |
| `STC_TRANSCRIBE_URL`, `STC_TRANSCRIBE_KEY`, `STC_TRANSCRIBE_MODEL` | unset | Whisper-compatible endpoint for video transcripts; unset disables transcription |
| `STC_AI_ALLOWED_BASE_URLS` | unset | `,`-separated origins learners' own AI keys may use besides api.anthropic.com and api.openai.com |
| `STC_AUTH_<GITHUB/GOOGLE/MICROSOFT/GITLAB/DISCORD>_ID`, `_SECRET` | unset | social sign-in |
| `STC_AUTH_OIDC_ISSUER`, `_ID`, `_SECRET`, `_NAME`, `_SCOPES`, `_DISCOVERY_URL`, `STC_AUTH_GROUPS_CLAIM` | unset | generic OIDC sign-in; group claims drive restricted visibility |
| `STC_ALLOWED_ORIGINS` | unset | extra `,`-separated origins (e.g. `http://192.168.1.5:3000`) allowed to send writes and sign in, besides `STC_PUBLIC_URL`'s origin. `STC_TRUSTED_ORIGINS` is read as an alias |

Without Docker: `bun install`, then `STC_SECRET=… NODE_ENV=production bun apps/server/src/main.ts`. The importer needs
`ffmpeg`, `ffprobe`, `git`, `unzip` and Python with PyMuPDF on the host.

## Cross-site requests and the session cookie

- Every state-changing request (POST, PUT, PATCH, DELETE), sign-in included, must carry an `Origin` (or, without
  one, a `Referer`) equal to `STC_PUBLIC_URL`'s origin or one in `STC_ALLOWED_ORIGINS`. Anything else gets 403.
  Open STCourses at exactly that URL. If people reach it by several names (a LAN IP and a hostname), list the others
  in `STC_ALLOWED_ORIGINS`.
- A write with a body must be `application/json` (uploads: `multipart/form-data`). HTML forms and `text/plain`
  bodies, which another page can send without a CORS preflight, are refused.
- Exempt: the identity-provider callbacks (SAML ACS and SLO, OAuth/OIDC callbacks under `/api/auth`). The provider
  posts those from its own origin, and Better Auth verifies the signed assertion or the OAuth state itself.
- The session cookie stays `SameSite=Lax` (Better Auth's default), not `Strict`. OAuth/OIDC sign-in returns from
  the provider by a cross-site top-level navigation, and Better Auth needs its state cookie on that request. Under
  `Strict` the browser withholds it, and social, OIDC and SAML sign-in break. The Origin check above covers the
  same-site cases that `Lax` allows (other ports on the same host, sibling subdomains).

## Security headers

- The web app carries its Content-Security-Policy as a `<meta>` tag in `apps/web/index.html` (no inline script).
  A meta tag cannot set `frame-ancestors`; if you need clickjacking protection, add
  `Content-Security-Policy: frame-ancestors 'self'` (or `X-Frame-Options: SAMEORIGIN`) in your reverse proxy.
- Course media (`/api/media/…`) is sent with `X-Content-Type-Options: nosniff`, and SVG, HTML and JavaScript files
  with `Content-Security-Policy: sandbox`, so author content cannot run script on the app's origin.

## Imported courses in the studio

Imported courses (`<data>/courses/<id>`) are read-only in the studio until the owner chooses **Make editable in the
studio** on the course's studio page. That runs a background job (`studio.convert-import`) which:

1. commits the manifest, lessons, transcripts, small images, small PDFs and code into a new studio repo,
   `<data>/studio/repos/<id>.git`, checked out as `<data>/worktrees/<id>/published`. The manifest is committed
   unchanged: the course keeps its id, its visibility (restricted after an import) and its `source.kind`, so only the
   owner can later make it public or signed-in. ACL entries, progress and notes are keyed by course and lesson id
   and are kept;
2. keeps **large media out of git**: every video or audio file (`mp4 m4v webm mov mkv avi mp3 m4a wav ogg oga opus
   flac`) and any other file over 8 MiB. Each is listed with its path, size and SHA-256 in `.stcourses/media.json`
   in the repo. They are kept out of `git status` by the repo's `info/exclude` (shared by every worktree), not by
   the course's `.gitignore`, and the studio never counts a listed file as a change. So an author or a remote
   rewriting `.gitignore` cannot block publishing or Sync. Git LFS is not used: it is not in the image and would
   need an LFS server;
3. **moves the import folder to `<data>/media/<id>`** (decided over keeping it in place: a manifest left in
   `<data>/courses/<id>` would be found as a second course with the same id, and the importer's "add to course" and
   transcription jobs would keep changing a folder learners no longer see). `<data>/media` is never scanned for
   courses and nothing writes to it afterwards. File permissions are left alone, because an import made in `link`
   mode shares its files with your originals;
4. puts every referenced file into the published worktree, and later into each author's draft worktree, as a
   **hard link** to `<data>/media/<id>/<path>`. The manifest paths are unchanged, so `/api/media/<id>/…` serves the
   same URLs with Range support, and no extra disk space is used. If `<data>/worktrees` and `<data>/media` are on
   different filesystems, each worktree gets a **full copy** instead: the published one plus one per author who
   opens a draft. Keep the data directory on one filesystem.

In the studio, referenced media cannot be replaced or deleted (save, upload and conflict resolution refuse them with
409); upload a new file under another name instead. A studio write never goes through a hard link: a file with more
than one link is replaced, not overwritten. Publish and Sync refuse a commit that tracks a file at a referenced media
path, since it would silently replace the media.

**Importer jobs and conversion.** Both use the same per-course lock. Conversion is refused (409, "an import job for
this course is still running") while an importer job for that course is queued or running. An importer job that
starts after the conversion fails with "this course is now edited in the studio" and writes nothing.

**Crash safety.** The import folder is first renamed to `<data>/courses/.converting-<id>` (dot-folders are never
scanned), before the published worktree is checked out, so the course is never loaded twice. Until that checkout is
done, a failure undoes everything. After it, the conversion is committed and only the move remains:

- on one filesystem, one `rename` to `<data>/media/<id>`;
- across filesystems, a copy into `<data>/media/.staging-<id>.<n>`, `fsync`, a size and SHA-256 check against
  `media.json`, a `rename` into place, and then the source is renamed to `<data>/courses/.converted-<id>.<n>` and
  deleted.

At server start and before every conversion, STCourses recovers each course deterministically:

- staging folders are deleted;
- a `.converting-` folder without a finished repo is renamed back (the next attempt starts over);
- a `.converting-` folder with a finished repo has its move finished;
- `.converted-` folders are deleted once `<data>/media/<id>` is complete.

A job left running by a restart is requeued, and re-running a finished conversion reports success.

Things to know:

- Back up `<data>/media` together with `<data>/studio`: the repo alone does not contain the videos.
- **Sync** pushes only the git repo. A clone of the remote has the manifest, lessons and `.stcourses/media.json`, but
  not the media files themselves; copy `<data>/media/<id>` alongside if you move the course to another server.
- If a worktree loses a media file (for example after a manual `git clean -x`), the next publish links it back into
  the published worktree, and a new draft worktree gets every file. A file missing from `<data>/media/<id>` shows
  as a course diagnostic ("lesson file not found"), never a crash.
- After conversion the importer can no longer add decks, code or transcripts to that course (it only edits
  `<data>/courses`). Add them in the studio.
- `<data>/media` is fixed under the data directory (there is no separate setting).
- **Discovery skips dot-folders.** Every course directory (`STC_COURSE_DIRS`, `<data>/courses`, `<data>/worktrees`)
  is scanned to depth 2 for `stcourse.yaml`, and folders whose name starts with `.` are never entered. Do not give a
  course folder a name that starts with a dot. The conversion's `.converting-<id>` and `.converted-<id>.<n>` folders
  rely on this.
- **A failed conversion.** If the media move fails after the commit (disk full, or a copy that fails its checksum),
  learners are switched to the studio course right away; its media are hard links into the hidden import folder.
  The course's studio page shows **Conversion incomplete** with a **Retry** button for the owner. A retry (or the
  next server start) finishes the move. A failure before the commit is rolled back, and learners keep the import
  unchanged.

## Memory budget

The deployment must stay under 2 GB idle. Measured with this compose file (`--profile postgres`), the image built
from this repository, and four courses mounted read-only: the sample course, Pure Snake and Systemic Design (both
overlays, repos mounted at their own paths), and the imported Packt course "Mastering Docker Essentials" (68 videos,
9 modules, a 59-slide deck, code files; about 800 MB of media). Before idling, one owner signed up and opened the
library, a video lesson with its deck, a Pure Snake lesson and the sample exercise in headless Chrome. Then nothing
for 5 minutes, and `docker stats --no-stream`:

| Container | Memory (idle, after 5 min) | CPU |
|---|---|---|
| `stc-app-1` (Bun server, SPA bundle built and cached) | 84.7 MiB | 1.00% |
| `stc-postgres-1` (postgres:17) | 16.67 MiB | 3.52% |
| **Total** | **101.4 MiB** (about 5% of the 2 GB budget) | |

Measured 2026-10-30 on Linux 6.12 (Docker Compose v5.5.1, `oven/bun:1.3` = Bun 1.3.14). With SQLite there is
no Postgres container, so the total is the app's figure alone.

Video and PDF files are streamed from disk with Range requests; they never sit in memory. The SPA bundle is built
once, on the first page request, and cached.