fix(core): sync review round 1: lists, supersede, dedupe conflicts, retries

- dedupe files conflicts for disagreeing fields instead of dropping them
- merged-away records never file removals
- matched work committed before apply so retries reuse it (no draft leaks)
- stale pending events superseded by newer values
- multi-source engagement removal only detaches the source
- list records applied with history-derived ownership
- gone targets resolve as ignored; counts only for committed records

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-10-30 20:43:08 +01:00
1 parent 0604c323c3
commit a003c3eb7d
8 files changed
+675 -196

No files matched your search

+3 -2
View File
@@ -32,8 +32,9 @@ Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are Task 9's job worker,
- Every other record goes through `applyRecord`: hash its normalised form; same hash as
`source_records.current_hash` → only `last_seen` moves. Otherwise a new `source_record_versions` row
(gzipped `{normalized, raw}`) is written, and:
- **new record**: `matchWork` (own transaction), then in one apply transaction create/patch the entry or
engagement (de-duplicating plays across sources) and write an `added` `auto_applied` event;
- **new record**: `matchWork` (own transaction, result committed on the record row so retries reuse it),
then in one apply transaction create/patch the entry, engagement (de-duplicating plays across sources)
or list, and write an `added` `auto_applied` event;
- **changed record**: fields that differ from the previous version are decided against `field_sources`
→ applied (`changed` `auto_applied`) or filed as `pending` `changed`/`conflict` events. Mapped records
are updated through `source_records.target_*`, never re-matched;
+32 -11
View File
@@ -21,36 +21,57 @@ are the mechanics a maintainer would otherwise reverse-engineer.
entry that already exists goes through the same per-field decision, so a first sync never overwrites
what the user or another source wrote. Unchanged values are never sent, so a plugin never takes
ownership of a field it did not change.
- **Anti-flip-flop.** A pending `changed`/`conflict` for the same (record, field, incoming value) is never
filed twice. `keep_mine` re-stamps `field_sources[f] = "user"`, so later values from any source become a
user-decided `changed`, never another source-vs-source conflict.
- **Anti-flip-flop and superseding.** A pending `changed`/`conflict` for the same (record, field, incoming
value) is never filed twice. A different newer value for the same (record, field) resolves the older
pending event as `ignored` with `payload.supersededBy = <new event id>`, so at most one is pending per
field. If the source comes round to the value STTracker already holds, its pending events for that field
are resolved `ignored` with `payload.reason = "source_agrees"`. `keep_mine` re-stamps
`field_sources[f] = "user"`, so later values from any source become a user-decided `changed`, never
another source-vs-source conflict.
- **No revive.** If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing
is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing.
Only the user revives.
- **Root ancestor.** Creating an entry on a child work also creates a `status "none"` entry (plugin actor,
empty `field_sources`) on the root ancestor work, unless the user has one, live or deleted.
- **Mapped records are never re-matched.** Only new records go through `matchWork`. A changed work hint
refreshes the mapped work via `updateWorkFromHint` (shared works only).
- **Mapped records are never re-matched.** Only new records go through `matchWork`. Its result is committed
at once as the record's row (`current_hash ""`, `target_entity "work"`), so a failed apply is retried
against the same work and an ID-less hint never leaks a second draft. A changed work hint refreshes the
mapped work via `updateWorkFromHint` (shared works only). `moveEntryToWork` re-points `source_records`
from a merged-away entry to the survivor in the same transaction, so a user's merge never orphans a source.
- **Engagement dedupe.** Among live engagements on the entry not already sourced by this connection:
finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and
their dates (first 10 chars, computed with `Date.UTC`, no timezone shift) are ≤ 1 day apart; or start
datetimes are the same instant. A match attaches an `engagement_sources` row and fills only fields that
are empty and that no source has set; the first source's values otherwise stand.
datetimes are the same instant. A match attaches an `engagement_sources` row and fills fields that are
empty and that no source has set. Every other differing field goes through the per-field decision
(so a disagreeing rating is a `conflict`), except the date that identified the play, whose difference is
within the tolerance by definition.
- **Removal.** Absence counts only after a `full` pull whose iteration finished and whose records all
applied without error; any record error fails the run and skips removal. A removed record that comes
back clears `deleted_at` and retires its still-pending `removed` event as `ignored`.
back clears `deleted_at` and retires its still-pending `removed` event as `ignored`. A merged-away record
(`merged_into_key` set, or a pending `merged` event) is expected to vanish: no removal, ever. Accepting
`removed` on an engagement other sources still report only deletes this connection's
`engagement_sources` row; the engagement is soft-deleted when no source remains. Run counts include only
records whose transaction committed.
- **Listens skip snapshots.** Listen records are not stored as `source_records` (they are high-volume and
append-only). `listens` insert-or-ignore is the idempotence; they are never removed.
append-only). `listens` insert-or-ignore is the idempotence; they are never removed. Per batch of 500:
one lookup of existing keys, each distinct recording resolved once, one multi-row insert.
- **Lists.** A new `list` record creates a user list (kind `list`, description `Imported from <plugin id>`),
items matched outside the transaction (an ID-less item first reuses the user's own private work with the
same media type and title). Lists have no `field_sources`; ownership is derived from `user_history`:
`name`/`description` belong to whoever last changed them, `items` to the user once any user edit exists.
A user-owned field becomes a pending `changed` (for items, the whole incoming item set); otherwise the
fields are applied and items replaced (`replaceListItems`, one history row).
- **Merge accept.** The target work is the merge target record's mapped entry's work, else `matchWork` on
its latest hint. Admin-merge events (`toWorkId`, no connection) resolve through the same `resolveChange`.
- **Gone targets.** Accept/keep_mine on an event whose target is deleted (or merged away with no live
survivor, found through the user's work alias) resolves it `ignored` with `payload.reason = "target_gone"`.
**Alternatives:**
- Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
- Hashing `raw` would turn every cosmetic API change into a new version.
- Keeping one run "ok" despite failing records risks marking those records removed.
**Consequences:** `list` records are snapshotted but not applied, and removal of a de-duplicated engagement
deletes it even if another source still reports it (both in `docs/TODO.md`).
**Consequences:** Concurrent runs of one connection are not locked here (Task 9's worker; `docs/TODO.md`).
## 2026-10-30: Export format `sttracker-export/v1`: per-user zip, camelCase JSONL, deleted rows included, no secrets
**Status:** accepted