STTracker/docs/ARCHITECTURE.md

Architecture

How STTracker is built today. Update in the same commit as any structural change.

Overview

Status 2026-10-30: only the workspace skeleton exists (root package.json, tsconfig.json). The target design is in docs/specs/001-sttracker-v1-design.md §11. Replace each "planned" marker below as the code lands.

One Bun process serves the REST API, the React web app and plugin ingest endpoints, and runs a database-backed job worker. Source plugins pull data from platforms and return normalised SourceRecords. The core snapshots them, diffs against the previous run, matches them to works, and either applies them to the user layer or files a Changes-inbox event.

Map

Path	Responsibility	State
apps/server	main.ts boots (config → createDb → migrateToLatest → loadPlugins → worker + scheduler + daily backup → Bun.serve). app.ts createApp(deps) = /api/v1 route table + /ingest/* mount (404 in Phase 1), testable via app.fetch. http.ts = route(), auth (session cookie / Bearer app token), CSRF header, body caps, rate limit, error mapping. routes/<area>.ts one file per area; openapi.ts builds /api/v1/openapi.json from the table	exists
apps/web	React 19 app served by the server as an HTML import (index.html → src/main.tsx, src/theme.css). wouter routes in main.tsx (/login, /register, /, /entries/:id; other sections are placeholders until Task 18); api.ts = fetch client (request, ApiError, cookie auth, CSRF header, mirrored wire types); format.ts = pure display helpers and friendlyError; edits.ts = next-value builders for entry edits; hooks.ts = TanStack Query hooks incl. optimistic useEntryPatch; settings.ts = preferences; components/ (shell, detail boxes, history drawer, rating control) and routes/. Never imports packages/* at runtime	exists
packages/core	domain services, one module per area (import by path, no barrel): auth, crypto, works, matching, entries (entries, library listing, quick add, per-user moves, match suggestions), engagements, history (user_history, restore), lists, copies, custom-fields, attachments, preferences (per-user rating scale and default visibility), export (per-user bundle, CSV, restore), backup (scheduled bundles + SQLite VACUUM INTO), sync/ (the change engine: snapshot hashing and versions, apply per-record diff and apply, run = runSync, changes = Changes inbox listChanges/resolveChange, listens = ingestListens), jobs (queue, worker, scheduler), registry (loadPlugins), connections (CRUD, plugin context, syncHandler)	exists (partial)
packages/db	Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, forEachDialect test harness	exists (schema pending)
packages/plugin-sdk	plugin contract, SourceRecord, http/rate-limit helpers, fixture test harness	planned
plugins/<id>	one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv)	planned
docs/design/mockups	approved UI mockups (reference only)	exists

Data flow

HTTP request (apps/server):

1. Bun.serve routes /api/* and /ingest/* to app.fetch; everything else gets the web app (HTML import).
2. apiHandler matches method + path in the route table, applies the per-IP rate limit (login/register), authenticates (session cookie stt_session, Bearer app token with a scope, or either), requires X-Requested-With: sttracker on every non-Bearer write (login included), reads the body (JSON ≤ 1 MB with Content-Type: application/json, multipart ≤ 50 MB) and validates it with zod.
3. The handler calls one core function with the caller's userId and actor (token:<name> for Bearer, else "user"; field_sources stores both as "user"). Core enforces ownership (NotFoundError → 404). Errors become { error: { code, message } }; unknown errors are logged and return 500 internal.
4. POST sources/:id/run|import only enqueue a sync job (lockKey sync:<id>, 409 when held); the worker runs it.

Sync lifecycle (packages/core/src/sync/; steps 1–2 are jobs.ts and connections.ts):

1. The scheduler enqueues a sync job (lockKey sync:<connectionId>) for each non-paused connection that is due.
2. The worker claims the job; syncHandler loads the connection and plugin and calls runSync(deps, plugin, connectionId, mode), injecting makeContext (http client, credentials, cursor). mode picks pull, pullChanges or importFile.
3. runSync opens a source_runs row and iterates the plugin's SourceRecords one at a time.
   • listen records are batched into ingestListens (insert-or-ignore into listens; no snapshot, no events).
   • Every other record goes through applyRecord: hash its normalised form; same hash as source_records.current_hash → only last_seen moves. Otherwise a new source_record_versions row (gzipped {normalized, raw}) is written, and:
     • new record: matchWork (own transaction, result committed on the record row so retries reuse it), then in one apply transaction create/patch the entry, engagement (de-duplicating plays across sources) or list, and write an added auto_applied event;
     • changed record: fields that differ from the previous version are decided against field_sources → applied (changed auto_applied) or filed as pending changed/conflict events. Mapped records are updated through source_records.target_*, never re-matched;
     • deleted: true → pending removed; a new mergedIntoKey → pending merged.
4. After a full pull that completed and applied every record, records not seen are marked removed (source_records.deleted_at + pending removed).
5. The run's counts, log and error land in source_runs; the connection gets last_run_at and status.
6. The user resolves pending events with resolveChange (accept / ignore / keep_mine).
7. Every user-layer write appends to user_history with the plugin actor plugin:<pluginId>:<connectionId>.

External dependencies

• Database: SQLite file under DATA_DIR (default) or Postgres via DATABASE_URL. If unavailable, nothing works.
• Platform APIs (Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm). If unavailable, only that source's sync fails; the run is logged and retried.

Invariants

• A sync never overwrites or hard-deletes user-entered data (spec §8.3).
• Plugins never touch the database.
• All domain behaviour is identical on SQLite and Postgres.

Known rough edges

None yet. See docs/TODO.md.