58 lines
4.5 KiB
Markdown
58 lines
4.5 KiB
Markdown
# Architecture
|
||
|
||
How STTracker is built today. Update in the same commit as any structural change.
|
||
|
||
## Overview
|
||
Status 2026-10-30: only the workspace skeleton exists (root `package.json`, `tsconfig.json`).
|
||
The target design is in `docs/specs/001-sttracker-v1-design.md` §11. Replace each "planned" marker below as the code lands.
|
||
|
||
One Bun process serves the REST API, the React web app and plugin ingest endpoints, and runs a
|
||
database-backed job worker. Source plugins pull data from platforms and return normalised `SourceRecord`s.
|
||
The core snapshots them, diffs against the previous run, matches them to works, and either applies them to the
|
||
user layer or files a Changes-inbox event.
|
||
|
||
## Map
|
||
| Path | Responsibility | State |
|
||
|---|---|---|
|
||
| `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
|
||
| `apps/web` | React app (HTML imports) | planned |
|
||
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`), `jobs` (queue, worker, scheduler), `registry` (`loadPlugins`), `connections` (CRUD, plugin context, `syncHandler`) | exists (partial) |
|
||
| `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
|
||
| `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
|
||
| `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |
|
||
| `docs/design/mockups` | approved UI mockups (reference only) | exists |
|
||
|
||
## Data flow
|
||
Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are `jobs.ts` and `connections.ts`):
|
||
1. The scheduler enqueues a `sync` job (lockKey `sync:<connectionId>`) for each non-paused connection that is due.
|
||
2. The worker claims the job; `syncHandler` loads the connection and plugin and calls `runSync(deps, plugin, connectionId, mode)`, injecting
|
||
`makeContext` (http client, credentials, cursor). `mode` picks `pull`, `pullChanges` or `importFile`.
|
||
3. `runSync` opens a `source_runs` row and iterates the plugin's `SourceRecord`s one at a time.
|
||
- `listen` records are batched into `ingestListens` (insert-or-ignore into `listens`; no snapshot, no events).
|
||
- Every other record goes through `applyRecord`: hash its normalised form; same hash as
|
||
`source_records.current_hash` → only `last_seen` moves. Otherwise a new `source_record_versions` row
|
||
(gzipped `{normalized, raw}`) is written, and:
|
||
- **new record**: `matchWork` (own transaction), then in one apply transaction create/patch the entry or
|
||
engagement (de-duplicating plays across sources) and write an `added` `auto_applied` event;
|
||
- **changed record**: fields that differ from the previous version are decided against `field_sources`
|
||
→ applied (`changed` `auto_applied`) or filed as `pending` `changed`/`conflict` events. Mapped records
|
||
are updated through `source_records.target_*`, never re-matched;
|
||
- `deleted: true` → `pending` `removed`; a new `mergedIntoKey` → `pending` `merged`.
|
||
4. After a `full` pull that completed and applied every record, records not seen are marked removed
|
||
(`source_records.deleted_at` + `pending` `removed`).
|
||
5. The run's counts, log and error land in `source_runs`; the connection gets `last_run_at` and `status`.
|
||
6. The user resolves `pending` events with `resolveChange` (accept / ignore / keep_mine).
|
||
7. Every user-layer write appends to `user_history` with the plugin actor `plugin:<pluginId>:<connectionId>`.
|
||
|
||
## External dependencies
|
||
- Database: SQLite file under `DATA_DIR` (default) or Postgres via `DATABASE_URL`. If unavailable, nothing works.
|
||
- Platform APIs (Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm). If unavailable, only that source's sync fails; the run is logged and retried.
|
||
|
||
## Invariants
|
||
- A sync never overwrites or hard-deletes user-entered data (spec §8.3).
|
||
- Plugins never touch the database.
|
||
- All domain behaviour is identical on SQLite and Postgres.
|
||
|
||
## Known rough edges
|
||
None yet. See `docs/TODO.md`.
|