STTracker/docs/ARCHITECTURE.md

Architecture

How STTracker is built today. Update in the same commit as any structural change.

Overview

Status 2026-10-30: only the workspace skeleton exists (root package.json, tsconfig.json). The target design is in docs/specs/001-sttracker-v1-design.md §11. Replace each "planned" marker below as the code lands.

One Bun process serves the REST API, the React web app and plugin ingest endpoints, and runs a database-backed job worker. Source plugins pull data from platforms and return normalised SourceRecords. The core snapshots them, diffs against the previous run, matches them to works, and either applies them to the user layer or files a Changes-inbox event.

Map

Path	Responsibility	State
apps/server	Bun.serve: /api/v1, /ingest/<plugin>, job worker, static web	planned
apps/web	React app (HTML imports)	planned
packages/core	domain services, one module per area (import by path, no barrel): auth, crypto, works, matching, entries (entries, library listing, quick add, per-user moves, match suggestions), engagements, history (user_history, restore), lists, copies, custom-fields, attachments, export (per-user bundle, CSV, restore), backup (scheduled bundles + SQLite VACUUM INTO), and sync/ (the change engine: snapshot hashing and versions, apply per-record diff and apply, run = runSync, changes = Changes inbox listChanges/resolveChange, listens = ingestListens), jobs (queue, worker, scheduler), registry (loadPlugins), connections (CRUD, plugin context, syncHandler)	exists (partial)
packages/db	Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, forEachDialect test harness	exists (schema pending)
packages/plugin-sdk	plugin contract, SourceRecord, http/rate-limit helpers, fixture test harness	planned
plugins/<id>	one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv)	planned
docs/design/mockups	approved UI mockups (reference only)	exists

Data flow

Sync lifecycle (packages/core/src/sync/; steps 1–2 are jobs.ts and connections.ts):

1. The scheduler enqueues a sync job (lockKey sync:<connectionId>) for each non-paused connection that is due.
2. The worker claims the job; syncHandler loads the connection and plugin and calls runSync(deps, plugin, connectionId, mode), injecting makeContext (http client, credentials, cursor). mode picks pull, pullChanges or importFile.
3. runSync opens a source_runs row and iterates the plugin's SourceRecords one at a time.
   • listen records are batched into ingestListens (insert-or-ignore into listens; no snapshot, no events).
   • Every other record goes through applyRecord: hash its normalised form; same hash as source_records.current_hash → only last_seen moves. Otherwise a new source_record_versions row (gzipped {normalized, raw}) is written, and:
     • new record: matchWork (own transaction), then in one apply transaction create/patch the entry or engagement (de-duplicating plays across sources) and write an added auto_applied event;
     • changed record: fields that differ from the previous version are decided against field_sources → applied (changed auto_applied) or filed as pending changed/conflict events. Mapped records are updated through source_records.target_*, never re-matched;
     • deleted: true → pending removed; a new mergedIntoKey → pending merged.
4. After a full pull that completed and applied every record, records not seen are marked removed (source_records.deleted_at + pending removed).
5. The run's counts, log and error land in source_runs; the connection gets last_run_at and status.
6. The user resolves pending events with resolveChange (accept / ignore / keep_mine).
7. Every user-layer write appends to user_history with the plugin actor plugin:<pluginId>:<connectionId>.

External dependencies

• Database: SQLite file under DATA_DIR (default) or Postgres via DATABASE_URL. If unavailable, nothing works.
• Platform APIs (Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm). If unavailable, only that source's sync fails; the run is logged and retried.

Invariants

• A sync never overwrites or hard-deletes user-entered data (spec §8.3).
• Plugins never touch the database.
• All domain behaviour is identical on SQLite and Postgres.

Known rough edges

None yet. See docs/TODO.md.