Files

T

Lucas WintherandClaude Opus 5.5 a003c3eb7d fix(core): sync review round 1: lists, supersede, dedupe conflicts, retries

- dedupe files conflicts for disagreeing fields instead of dropping them
- merged-away records never file removals
- matched work committed before apply so retries reuse it (no draft leaks)
- stale pending events superseded by newer values
- multi-source engagement removal only detaches the source
- list records applied with history-derived ownership
- gone targets resolve as ignored; counts only for committed records

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-10-30 20:43:08 +01:00

4.3 KiB

Raw Blame History

Architecture

How STTracker is built today. Update in the same commit as any structural change.

Overview

Status 2026-10-30: only the workspace skeleton exists (root package.json, tsconfig.json). The target design is in docs/specs/001-sttracker-v1-design.md §11. Replace each "planned" marker below as the code lands.

One Bun process serves the REST API, the React web app and plugin ingest endpoints, and runs a database-backed job worker. Source plugins pull data from platforms and return normalised SourceRecords. The core snapshots them, diffs against the previous run, matches them to works, and either applies them to the user layer or files a Changes-inbox event.

Map

Path	Responsibility	State
`apps/server`	`Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web	planned
`apps/web`	React app (HTML imports)	planned
`packages/core`	domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`)	exists (partial)
`packages/db`	Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness	exists (schema pending)
`packages/plugin-sdk`	plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness	planned
`plugins/<id>`	one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv)	planned
`docs/design/mockups`	approved UI mockups (reference only)	exists

Data flow

Sync lifecycle (packages/core/src/sync/; steps 1–2 are Task 9's job worker, planned):

The scheduler enqueues a job.
The worker loads the connection and plugin and calls runSync(deps, plugin, connectionId, mode), injecting makeContext (http client, credentials, cursor). mode picks pull, pullChanges or importFile.
runSync opens a source_runs row and iterates the plugin's SourceRecords one at a time.
- listen records are batched into ingestListens (insert-or-ignore into listens; no snapshot, no events).
- Every other record goes through applyRecord: hash its normalised form; same hash as source_records.current_hash → only last_seen moves. Otherwise a new source_record_versions row (gzipped {normalized, raw}) is written, and:
  - new record: matchWork (own transaction, result committed on the record row so retries reuse it), then in one apply transaction create/patch the entry, engagement (de-duplicating plays across sources) or list, and write an added auto_applied event;
  - changed record: fields that differ from the previous version are decided against field_sources → applied (changed auto_applied) or filed as pending changed/conflict events. Mapped records are updated through source_records.target_*, never re-matched;
  - deleted: true → pending removed; a new mergedIntoKey → pending merged.
After a full pull that completed and applied every record, records not seen are marked removed (source_records.deleted_at + pending removed).
The run's counts, log and error land in source_runs; the connection gets last_run_at and status.
The user resolves pending events with resolveChange (accept / ignore / keep_mine).
Every user-layer write appends to user_history with the plugin actor plugin:<pluginId>:<connectionId>.

External dependencies

Database: SQLite file under DATA_DIR (default) or Postgres via DATABASE_URL. If unavailable, nothing works.
Platform APIs (Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm). If unavailable, only that source's sync fails; the run is logged and retried.

Invariants

A sync never overwrites or hard-deletes user-entered data (spec §8.3).
Plugins never touch the database.
All domain behaviour is identical on SQLite and Postgres.

Known rough edges

None yet. See docs/TODO.md.

4.3 KiB Raw Blame History Unescape Escape