merge: task 8 sync engine

docs/ARCHITECTURE.md

@@ -16,20 +16,33 @@ user layer or files a Changes-inbox event.
 |---|---|---|
 | `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
 | `apps/web` | React app (HTML imports) | planned |
-| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`). Change engine planned | exists (partial) |
+| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`) | exists (partial) |
 | `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
 | `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
 | `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |
 | `docs/design/mockups` | approved UI mockups (reference only) | exists |
 
 ## Data flow
-Planned sync lifecycle:
+Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are Task 9's job worker, planned):
 1. The scheduler enqueues a job.
-2. The worker loads the plugin and calls `pull` or `pullChanges` with an SDK context.
-3. The plugin returns `SourceRecord[]`.
-4. The core stores a new `source_record_versions` row for each changed hash and diffs the records into `change_events`.
-5. Each event is either auto-applied (new record, or a field never edited by the user) or sent to the inbox.
-6. Every user-layer write appends to `user_history`.
+2. The worker loads the connection and plugin and calls `runSync(deps, plugin, connectionId, mode)`, injecting
+   `makeContext` (http client, credentials, cursor). `mode` picks `pull`, `pullChanges` or `importFile`.
+3. `runSync` opens a `source_runs` row and iterates the plugin's `SourceRecord`s one at a time.
+   - `listen` records are batched into `ingestListens` (insert-or-ignore into `listens`; no snapshot, no events).
+   - Every other record goes through `applyRecord`: hash its normalised form; same hash as
+     `source_records.current_hash` → only `last_seen` moves. Otherwise a new `source_record_versions` row
+     (gzipped `{normalized, raw}`) is written, and:
+     - **new record**: `matchWork` (own transaction), then in one apply transaction create/patch the entry or
+       engagement (de-duplicating plays across sources) and write an `added` `auto_applied` event;
+     - **changed record**: fields that differ from the previous version are decided against `field_sources`
+       → applied (`changed` `auto_applied`) or filed as `pending` `changed`/`conflict` events. Mapped records
+       are updated through `source_records.target_*`, never re-matched;
+     - `deleted: true` → `pending` `removed`; a new `mergedIntoKey` → `pending` `merged`.
+4. After a `full` pull that completed and applied every record, records not seen are marked removed
+   (`source_records.deleted_at` + `pending` `removed`).
+5. The run's counts, log and error land in `source_runs`; the connection gets `last_run_at` and `status`.
+6. The user resolves `pending` events with `resolveChange` (accept / ignore / keep_mine).
+7. Every user-layer write appends to `user_history` with the plugin actor `plugin:<pluginId>:<connectionId>`.
 
 ## External dependencies
 - Database: SQLite file under `DATA_DIR` (default) or Postgres via `DATABASE_URL`. If unavailable, nothing works.

docs/DECISIONS.md

@@ -30,6 +30,54 @@ No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, lin
   are built in `<stamp>.tmp` then renamed (modes 0700/0600), and prune to the newest `keep` (default 14).
 **Consequences:** Any change to these files after release needs `v2` and a migration path for old bundles.
 
+## 2026-10-30: Sync engine: hash composition, anti-flip-flop, no revive, root ancestor, dedupe
+**Status:** accepted
+**Context:** Task 8 built the change engine (`packages/core/src/sync/`). Spec §8.3 fixes the outcomes; these
+are the mechanics a maintainer would otherwise reverse-engineer.
+**Decision:**
+- **Hash composition.** sha256 of the stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list,
+  deleted, mergedIntoKey}`. `key` and `raw` are excluded; `deleted` is always a boolean and `mergedIntoKey`
+  always present (null), so an omitted flag and an explicit `false` hash alike. A version row stores
+  base64(`Bun.gzipSync`) of `{normalized, raw}`: the normalised record is what the next change is diffed
+  against (field-level diff = previous version vs. incoming), the raw payload is kept for the record.
+- **Per-field decision.** Only fields that differ from the previous version are considered, and of those
+  only values that differ from the current STTracker value. `field_sources[f]` unset or this connection's
+  actor → applied; `"user"` → `pending changed {field, from: current, to: incoming}`; another source →
+  `pending conflict {field, current, currentSource, incoming, incomingSource}`. A new record landing on an
+  entry that already exists goes through the same per-field decision, so a first sync never overwrites
+  what the user or another source wrote. Unchanged values are never sent, so a plugin never takes
+  ownership of a field it did not change.
+- **Anti-flip-flop.** A pending `changed`/`conflict` for the same (record, field, incoming value) is never
+  filed twice. `keep_mine` re-stamps `field_sources[f] = "user"`, so later values from any source become a
+  user-decided `changed`, never another source-vs-source conflict.
+- **No revive.** If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing
+  is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing.
+  Only the user revives.
+- **Root ancestor.** Creating an entry on a child work also creates a `status "none"` entry (plugin actor,
+  empty `field_sources`) on the root ancestor work, unless the user has one, live or deleted.
+- **Mapped records are never re-matched.** Only new records go through `matchWork`. A changed work hint
+  refreshes the mapped work via `updateWorkFromHint` (shared works only).
+- **Engagement dedupe.** Among live engagements on the entry not already sourced by this connection:
+  finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and
+  their dates (first 10 chars, computed with `Date.UTC`, no timezone shift) are ≤ 1 day apart; or start
+  datetimes are the same instant. A match attaches an `engagement_sources` row and fills only fields that
+  are empty and that no source has set; the first source's values otherwise stand.
+- **Removal.** Absence counts only after a `full` pull whose iteration finished and whose records all
+  applied without error; any record error fails the run and skips removal. A removed record that comes
+  back clears `deleted_at` and retires its still-pending `removed` event as `ignored`.
+- **Listens skip snapshots.** Listen records are not stored as `source_records` (they are high-volume and
+  append-only). `listens` insert-or-ignore is the idempotence; they are never removed.
+- **Merge accept.** The target work is the merge target record's mapped entry's work, else `matchWork` on
+  its latest hint. Admin-merge events (`toWorkId`, no connection) resolve through the same `resolveChange`.
+
+**Alternatives:**
+- Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
+- Hashing `raw` would turn every cosmetic API change into a new version.
+- Keeping one run "ok" despite failing records risks marking those records removed.
+
+**Consequences:** `list` records are snapshotted but not applied, and removal of a de-duplicated engagement
+deletes it even if another source still reports it (both in `docs/TODO.md`).
+
 ## 2026-10-30: User layer: per-user moves merge entries, keep aliases, and roll the library up by hierarchy
 **Status:** accepted
 **Context:** Task 6 built the user layer (`packages/core/src/{entries,engagements,history,lists,copies}.ts`).

docs/TODO.md

@@ -29,6 +29,15 @@ Small, specific, independently actionable. Remove only when actually done.
 - [ ] Restore: an unlink of identical attachment bytes between the pre-write and the commit could remove the file;
   mitigated by re-writing after commit, not locked.
 - [ ] Export bundles hold attachment bytes in memory; very large libraries need a streaming zip.
+- [ ] Sync: `list` records are snapshotted but not applied to `lists` (Trakt/Hardcover emit them). Decide
+  the mapping (one list per record, items by matched work) and apply it through the change engine.
+- [ ] Sync: accepting a `removed` event on an engagement de-duplicated across sources soft-deletes it even if
+  another connection still reports it; consider detaching that connection's `engagement_sources` row instead.
+- [ ] Sync: pending `changed`/`conflict` events for a value the source has since moved past stay pending;
+  consider superseding them when a newer value for the same field arrives.
+- [ ] Sync: two concurrent runs of the same connection can both insert a new `source_records` row for one key
+  (one fails on the unique key and is logged as a record error). The job worker should lock per connection.
+- [ ] Sync: `ingestListens` does one lookup and one insert per listen; batch it for large backfills.
 - [ ] Admin merge does not merge colliding children recursively; colliding children stay under the tombstoned parent.
 - [ ] Matching: a shared parent resolved without external IDs or position (title/year reuse in
   `packages/core/src/matching.ts`) has no unique key, so concurrent matches can create it twice.