lucasw89/STTracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

merge: task 8 sync engine

Browse Source
This commit is contained in:
Lucas Winther
2026-10-30 20:35:37 +01:00
parent c8ce107f73 1975e971f9
commit 207c92bb4d
12 changed files with 1861 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+20 -7
View File
@@ -16,20 +16,33 @@ user layer or files a Changes-inbox event.
|---|---|---|
| `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
| `apps/web` | React app (HTML imports) | planned |
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`). Change engine planned | exists (partial) |
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`) | exists (partial) |
| `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
| `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
| `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |
| `docs/design/mockups` | approved UI mockups (reference only) | exists |
## Data flow
Planned sync lifecycle:
Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are Task 9's job worker, planned):
1. The scheduler enqueues a job.
2. The worker loads the plugin and calls `pull` or `pullChanges` with an SDK context.
3. The plugin returns `SourceRecord[]`.
4. The core stores a new `source_record_versions` row for each changed hash and diffs the records into `change_events`.
5. Each event is either auto-applied (new record, or a field never edited by the user) or sent to the inbox.
6. Every user-layer write appends to `user_history`.
2. The worker loads the connection and plugin and calls `runSync(deps, plugin, connectionId, mode)`, injecting
`makeContext` (http client, credentials, cursor). `mode` picks `pull`, `pullChanges` or `importFile`.
3. `runSync` opens a `source_runs` row and iterates the plugin's `SourceRecord`s one at a time.
- `listen` records are batched into `ingestListens` (insert-or-ignore into `listens`; no snapshot, no events).
- Every other record goes through `applyRecord`: hash its normalised form; same hash as
`source_records.current_hash` → only `last_seen` moves. Otherwise a new `source_record_versions` row
(gzipped `{normalized, raw}`) is written, and:
- **new record**: `matchWork` (own transaction), then in one apply transaction create/patch the entry or
engagement (de-duplicating plays across sources) and write an `added` `auto_applied` event;
- **changed record**: fields that differ from the previous version are decided against `field_sources`
→ applied (`changed` `auto_applied`) or filed as `pending` `changed`/`conflict` events. Mapped records
are updated through `source_records.target_*`, never re-matched;
- `deleted: true` → `pending` `removed`; a new `mergedIntoKey` → `pending` `merged`.
4. After a `full` pull that completed and applied every record, records not seen are marked removed
(`source_records.deleted_at` + `pending` `removed`).
5. The run's counts, log and error land in `source_runs`; the connection gets `last_run_at` and `status`.
6. The user resolves `pending` events with `resolveChange` (accept / ignore / keep_mine).
7. Every user-layer write appends to `user_history` with the plugin actor `plugin:<pluginId>:<connectionId>`.
## External dependencies
- Database: SQLite file under `DATA_DIR` (default) or Postgres via `DATABASE_URL`. If unavailable, nothing works.