merge: feat/phase1 into task/9
This commit is contained in:
Lucas Winther
@@ -16,7 +16,7 @@ user layer or files a Changes-inbox event.
|
||||
|---|---|---|
|
||||
| `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
|
||||
| `apps/web` | React app (HTML imports) | planned |
|
||||
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`), `jobs` (queue, worker, scheduler), `registry` (`loadPlugins`), `connections` (CRUD, plugin context, `syncHandler`) | exists (partial) |
|
||||
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`), `jobs` (queue, worker, scheduler), `registry` (`loadPlugins`), `connections` (CRUD, plugin context, `syncHandler`) | exists (partial) |
|
||||
| `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
|
||||
| `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
|
||||
| `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |
|
||||
@@ -32,8 +32,9 @@ Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are `jobs.ts` and `connec
|
||||
- Every other record goes through `applyRecord`: hash its normalised form; same hash as
|
||||
`source_records.current_hash` → only `last_seen` moves. Otherwise a new `source_record_versions` row
|
||||
(gzipped `{normalized, raw}`) is written, and:
|
||||
- **new record**: `matchWork` (own transaction), then in one apply transaction create/patch the entry or
|
||||
engagement (de-duplicating plays across sources) and write an `added` `auto_applied` event;
|
||||
- **new record**: `matchWork` (own transaction, result committed on the record row so retries reuse it),
|
||||
then in one apply transaction create/patch the entry, engagement (de-duplicating plays across sources)
|
||||
or list, and write an `added` `auto_applied` event;
|
||||
- **changed record**: fields that differ from the previous version are decided against `field_sources`
|
||||
→ applied (`changed` `auto_applied`) or filed as `pending` `changed`/`conflict` events. Mapped records
|
||||
are updated through `source_records.target_*`, never re-matched;
|
||||
|
||||
+70
-48
@@ -41,6 +41,76 @@ No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, lin
|
||||
within 7 days; else `changes`. A failed run is rethrown so the worker retries it.
|
||||
**Consequences:** Reload re-imports plugin modules with `?v=<Date.now()>`; old module instances are not freed by Bun.
|
||||
|
||||
|
||||
## 2026-10-30: Sync engine: hash composition, anti-flip-flop, no revive, root ancestor, dedupe
|
||||
**Status:** accepted
|
||||
**Context:** Task 8 built the change engine (`packages/core/src/sync/`). Spec §8.3 fixes the outcomes; these
|
||||
are the mechanics a maintainer would otherwise reverse-engineer.
|
||||
**Decision:**
|
||||
- **Hash composition.** sha256 of the stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list,
|
||||
deleted, mergedIntoKey}`. `key` and `raw` are excluded; `deleted` is always a boolean and `mergedIntoKey`
|
||||
always present (null), so an omitted flag and an explicit `false` hash alike. A version row stores
|
||||
base64(`Bun.gzipSync`) of `{normalized, raw}`: the normalised record is what the next change is diffed
|
||||
against (field-level diff = previous version vs. incoming), the raw payload is kept for the record.
|
||||
- **Per-field decision.** Only fields that differ from the previous version are considered, and of those
|
||||
only values that differ from the current STTracker value. `field_sources[f]` unset or this connection's
|
||||
actor → applied; `"user"` → `pending changed {field, from: current, to: incoming}`; another source →
|
||||
`pending conflict {field, current, currentSource, incoming, incomingSource}`. A new record landing on an
|
||||
entry that already exists goes through the same per-field decision, so a first sync never overwrites
|
||||
what the user or another source wrote. Unchanged values are never sent, so a plugin never takes
|
||||
ownership of a field it did not change.
|
||||
- **Anti-flip-flop and superseding.** A pending `changed`/`conflict` for the same (record, field, incoming
|
||||
value) is never filed twice. A different newer value for the same (record, field) resolves the older
|
||||
pending event as `ignored` with `payload.supersededBy = <new event id>`, so at most one is pending per
|
||||
field. If the source comes round to the value STTracker already holds, its pending events for that field
|
||||
are resolved `ignored` with `payload.reason = "source_agrees"`. `keep_mine` re-stamps
|
||||
`field_sources[f] = "user"`, so later values from any source become a user-decided `changed`, never
|
||||
another source-vs-source conflict.
|
||||
- **No revive.** If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing
|
||||
is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing.
|
||||
Only the user revives.
|
||||
- **Root ancestor.** Creating an entry on a child work also creates a `status "none"` entry (plugin actor,
|
||||
empty `field_sources`) on the root ancestor work, unless the user has one, live or deleted.
|
||||
- **Mapped records are never re-matched.** Only new records go through `matchWork`. Its result is committed
|
||||
at once as the record's row (`current_hash ""`, `target_entity "work"`), so a failed apply is retried
|
||||
against the same work and an ID-less hint never leaks a second draft. A changed work hint refreshes the
|
||||
mapped work via `updateWorkFromHint` (shared works only). `moveEntryToWork` re-points `source_records`
|
||||
from a merged-away entry to the survivor in the same transaction, so a user's merge never orphans a source.
|
||||
- **Engagement dedupe.** Among live engagements on the entry not already sourced by this connection:
|
||||
finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and
|
||||
their dates (first 10 chars, computed with `Date.UTC`, no timezone shift) are ≤ 1 day apart; or start
|
||||
datetimes are the same instant. A match attaches an `engagement_sources` row and fills fields that are
|
||||
empty and that no source has set. Every other differing field goes through the per-field decision
|
||||
(so a disagreeing rating is a `conflict`), except the date that identified the play, whose difference is
|
||||
within the tolerance by definition.
|
||||
- **Removal.** Absence counts only after a `full` pull whose iteration finished and whose records all
|
||||
applied without error; any record error fails the run and skips removal. A removed record that comes
|
||||
back clears `deleted_at` and retires its still-pending `removed` event as `ignored`. A merged-away record
|
||||
(`merged_into_key` set, or a pending `merged` event) is expected to vanish: no removal, ever. Accepting
|
||||
`removed` on an engagement other sources still report only deletes this connection's
|
||||
`engagement_sources` row; the engagement is soft-deleted when no source remains. Run counts include only
|
||||
records whose transaction committed.
|
||||
- **Listens skip snapshots.** Listen records are not stored as `source_records` (they are high-volume and
|
||||
append-only). `listens` insert-or-ignore is the idempotence; they are never removed. Per batch of 500:
|
||||
one lookup of existing keys, each distinct recording resolved once, one multi-row insert.
|
||||
- **Lists.** A new `list` record creates a user list (kind `list`, description `Imported from <plugin id>`),
|
||||
items matched outside the transaction (an ID-less item first reuses the user's own private work with the
|
||||
same media type and title). Lists have no `field_sources`; ownership is derived from `user_history`:
|
||||
`name`/`description` belong to whoever last changed them, `items` to the user once any user edit exists.
|
||||
A user-owned field becomes a pending `changed` (for items, the whole incoming item set); otherwise the
|
||||
fields are applied and items replaced (`replaceListItems`, one history row).
|
||||
- **Merge accept.** The target work is the merge target record's mapped entry's work, else `matchWork` on
|
||||
its latest hint. Admin-merge events (`toWorkId`, no connection) resolve through the same `resolveChange`.
|
||||
- **Gone targets.** Accept/keep_mine on an event whose target is deleted (or merged away with no live
|
||||
survivor, found through the user's work alias) resolves it `ignored` with `payload.reason = "target_gone"`.
|
||||
|
||||
**Alternatives:**
|
||||
- Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
|
||||
- Hashing `raw` would turn every cosmetic API change into a new version.
|
||||
- Keeping one run "ok" despite failing records risks marking those records removed.
|
||||
|
||||
**Consequences:** Concurrent runs of one connection are not locked here (Task 9's worker; `docs/TODO.md`).
|
||||
|
||||
## 2026-10-30: Export format `sttracker-export/v1`: per-user zip, camelCase JSONL, deleted rows included, no secrets
|
||||
**Status:** accepted
|
||||
**Context:** Task 15 (`packages/core/src/{export,backup}.ts`) fixed the on-disk bundle format that backups and restore share.
|
||||
@@ -67,54 +137,6 @@ No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, lin
|
||||
are built in `<stamp>.tmp` then renamed (modes 0700/0600), and prune to the newest `keep` (default 14).
|
||||
**Consequences:** Any change to these files after release needs `v2` and a migration path for old bundles.
|
||||
|
||||
## 2026-10-30: Sync engine: hash composition, anti-flip-flop, no revive, root ancestor, dedupe
|
||||
**Status:** accepted
|
||||
**Context:** Task 8 built the change engine (`packages/core/src/sync/`). Spec §8.3 fixes the outcomes; these
|
||||
are the mechanics a maintainer would otherwise reverse-engineer.
|
||||
**Decision:**
|
||||
- **Hash composition.** sha256 of the stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list,
|
||||
deleted, mergedIntoKey}`. `key` and `raw` are excluded; `deleted` is always a boolean and `mergedIntoKey`
|
||||
always present (null), so an omitted flag and an explicit `false` hash alike. A version row stores
|
||||
base64(`Bun.gzipSync`) of `{normalized, raw}`: the normalised record is what the next change is diffed
|
||||
against (field-level diff = previous version vs. incoming), the raw payload is kept for the record.
|
||||
- **Per-field decision.** Only fields that differ from the previous version are considered, and of those
|
||||
only values that differ from the current STTracker value. `field_sources[f]` unset or this connection's
|
||||
actor → applied; `"user"` → `pending changed {field, from: current, to: incoming}`; another source →
|
||||
`pending conflict {field, current, currentSource, incoming, incomingSource}`. A new record landing on an
|
||||
entry that already exists goes through the same per-field decision, so a first sync never overwrites
|
||||
what the user or another source wrote. Unchanged values are never sent, so a plugin never takes
|
||||
ownership of a field it did not change.
|
||||
- **Anti-flip-flop.** A pending `changed`/`conflict` for the same (record, field, incoming value) is never
|
||||
filed twice. `keep_mine` re-stamps `field_sources[f] = "user"`, so later values from any source become a
|
||||
user-decided `changed`, never another source-vs-source conflict.
|
||||
- **No revive.** If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing
|
||||
is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing.
|
||||
Only the user revives.
|
||||
- **Root ancestor.** Creating an entry on a child work also creates a `status "none"` entry (plugin actor,
|
||||
empty `field_sources`) on the root ancestor work, unless the user has one, live or deleted.
|
||||
- **Mapped records are never re-matched.** Only new records go through `matchWork`. A changed work hint
|
||||
refreshes the mapped work via `updateWorkFromHint` (shared works only).
|
||||
- **Engagement dedupe.** Among live engagements on the entry not already sourced by this connection:
|
||||
finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and
|
||||
their dates (first 10 chars, computed with `Date.UTC`, no timezone shift) are ≤ 1 day apart; or start
|
||||
datetimes are the same instant. A match attaches an `engagement_sources` row and fills only fields that
|
||||
are empty and that no source has set; the first source's values otherwise stand.
|
||||
- **Removal.** Absence counts only after a `full` pull whose iteration finished and whose records all
|
||||
applied without error; any record error fails the run and skips removal. A removed record that comes
|
||||
back clears `deleted_at` and retires its still-pending `removed` event as `ignored`.
|
||||
- **Listens skip snapshots.** Listen records are not stored as `source_records` (they are high-volume and
|
||||
append-only). `listens` insert-or-ignore is the idempotence; they are never removed.
|
||||
- **Merge accept.** The target work is the merge target record's mapped entry's work, else `matchWork` on
|
||||
its latest hint. Admin-merge events (`toWorkId`, no connection) resolve through the same `resolveChange`.
|
||||
|
||||
**Alternatives:**
|
||||
- Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
|
||||
- Hashing `raw` would turn every cosmetic API change into a new version.
|
||||
- Keeping one run "ok" despite failing records risks marking those records removed.
|
||||
|
||||
**Consequences:** `list` records are snapshotted but not applied, and removal of a de-duplicated engagement
|
||||
deletes it even if another source still reports it (both in `docs/TODO.md`).
|
||||
|
||||
## 2026-10-30: User layer: per-user moves merge entries, keep aliases, and roll the library up by hierarchy
|
||||
**Status:** accepted
|
||||
**Context:** Task 6 built the user layer (`packages/core/src/{entries,engagements,history,lists,copies}.ts`).
|
||||
|
||||
+2
-9
@@ -27,20 +27,13 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
- [ ] Plugins: reload re-imports entry files only; edits to a plugin's other modules need a server restart.
|
||||
- [ ] Plugins: rate limiting is per run (each run builds its own client), not per plugin (spec §11); share a limiter per plugin in Phase 2.
|
||||
- [ ] Plugins: `ctx.log` reaches `source_runs.log` only after the run ends, so a crash mid-run loses it.
|
||||
- [ ] Sync: two concurrent runs of one connection outside the job queue (e.g. a manual run) can still race on `source_records`.
|
||||
Queued syncs are serialised by `lockKey`.
|
||||
- [ ] Restore: a new child work whose parent+position collides with an existing child of a re-matched parent fails
|
||||
with `ImportError("conflict")` instead of being merged or repositioned.
|
||||
- [ ] Restore: an unlink of identical attachment bytes between the pre-write and the commit could remove the file;
|
||||
mitigated by re-writing after commit, not locked.
|
||||
- [ ] Export bundles hold attachment bytes in memory; very large libraries need a streaming zip.
|
||||
- [ ] Sync: `list` records are snapshotted but not applied to `lists` (Trakt/Hardcover emit them). Decide
|
||||
the mapping (one list per record, items by matched work) and apply it through the change engine.
|
||||
- [ ] Sync: accepting a `removed` event on an engagement de-duplicated across sources soft-deletes it even if
|
||||
another connection still reports it; consider detaching that connection's `engagement_sources` row instead.
|
||||
- [ ] Sync: pending `changed`/`conflict` events for a value the source has since moved past stay pending;
|
||||
consider superseding them when a newer value for the same field arrives.
|
||||
- [ ] Sync: two concurrent runs of the same connection (outside the job queue, e.g. a manual run) can both insert a new
|
||||
`source_records` row for one key. Queued syncs are serialised by `lockKey`.
|
||||
- [ ] Sync: `ingestListens` does one lookup and one insert per listen; batch it for large backfills.
|
||||
- [ ] Admin merge does not merge colliding children recursively; colliding children stay under the tombstoned parent.
|
||||
- [ ] Matching: a shared parent resolved without external IDs or position (title/year reuse in
|
||||
`packages/core/src/matching.ts`) has no unique key, so concurrent matches can create it twice.
|
||||
|
||||
Reference in New Issue
Block a user