STTracker/docs/DECISIONS.md

# Decisions

Append-only. Never edit or delete an entry. Supersede it with a new one that links back.
One entry per decision a future maintainer would otherwise have to reverse-engineer.
No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, line number).

## 2026-10-30: Job failure semantics and plugin reload limits
**Status:** accepted
**Context:** Task 9 fix round 1.
**Decision:**
- **Failures before `runSync` never retry.** A missing/paused connection drops the job (`DropJobError`: `done`, message in
  `last_error`). A missing plugin or undecryptable credentials (`CryptoError`) insert a failed `source_runs` row, set the
  connection to `error`, advance `last_run_at` and throw `NonRetryableError` (job `failed`, no backoff). The scheduler still
  picks `error` connections, one interval later, so fixing the cause heals them.
- **Stale locks.** Every worker tick re-queues `running` jobs whose `locked_at` is over an hour old (attempts unchanged), so a
  crash cannot hold a lockKey forever. Job bookkeeping errors are logged (`[jobs]`), never unhandled; `stop()` never rejects
  and aborts the `signal` passed to handlers (`handler(payload, { signal })`).
- **Reload** evicts the entry file from `require.cache` and imports with `?v=<Date.now()>` (Bun ignores the query alone for
  cached `.ts`). Only entry files reload: changes in a plugin's other modules need a server restart.
- `credentials.save` and `updateConnection` read-merge-write in one transaction (row lock on Postgres). `ctx.log` is buffered and
  appended to the run's `source_runs.log` when the run ends.

## 2026-10-30: Jobs, plugin registry, connections: rate limit in the manifest, secret split, per-connection lock
**Status:** accepted
**Context:** Task 9 (`packages/core/src/{jobs,registry,connections}.ts`).
**Decision:**
- **Rate limit lives in the manifest** (`PluginManifest.rateLimit?: { perMinute, burst? }`), not in plugin code. The host builds
  `ctx.http` from it (default 60/min, burst 5, UA `STTracker/0.1 (+self-hosted)`), so every plugin is throttled the same way
  and the per-plugin `RATE_PER_MINUTE` constants are gone.
- **Secrets are split off config.** Config fields typed `"secret"` in the manifest are stored only in the encrypted
  `credentials_enc` blob (with OAuth tokens a plugin saves), never in `config`. `makePluginContext` merges them back into
  `ctx.config`; `listConnections` returns plain config and `hasCredentials` only. `updateConnection` keeps secrets that are
  omitted and replaces plain config wholesale. `credentials.save` merges and persists immediately, so a token refreshed
  mid-pull survives a crash.
- **Constructor deviation.** `createConnection`/`updateConnection` take the plugin `manifest` as a trailing argument (needed
  for the split), and `makePluginContext` is async (it decrypts) and takes the manifest before an optional `http`.
- **One sync per connection.** Sync jobs carry `lockKey = "sync:<connectionId>"`; `enqueue` returns null while a
  `queued|running` job holds the key. On Postgres the check and insert run under `pg_advisory_xact_lock(hashtext(key))`.
  Claiming is `update … where status = 'queued'` checked on affected rows. Retries back off `2^attempts` minutes, 5 attempts, then `failed`.
- **Mode choice.** The sync handler runs `full` when asked, when the plugin lacks `pullChanges`, or when no `ok` full run started
  within 7 days; else `changes`. A failed run is rethrown so the worker retries it.
**Consequences:** Reload re-imports plugin modules with `?v=<Date.now()>`; old module instances are not freed by Bun.

## 2026-10-30: Export format `sttracker-export/v1`: per-user zip, camelCase JSONL, deleted rows included, no secrets
**Status:** accepted
**Context:** Task 15 (`packages/core/src/{export,backup}.ts`) fixed the on-disk bundle format that backups and restore share.
**Decision:**
- **Files.** A zip (`fflate`) holding `manifest.json` (`{ format, exportedAt, user: { username, displayName } }`),
  `works.jsonl`, `entries.jsonl`, `engagements.jsonl`, `listens.jsonl`, `lists.jsonl` (items nested),
  `copies.jsonl`, `custom_fields.jsonl` (field definitions), `history.jsonl`, `attachments.jsonl` (links nested)
  and `attachments/<sha256>` bytes. One JSON object per line, `\n` separated.
- **Keys.** camelCase; JSON columns are parsed into real JSON; `favorite` is a boolean. `user_id` columns are omitted
  (the importing user owns everything). `works.jsonl` adds `private` and `externalIds`.
- **Deleted rows are included** with their `deletedAt`, and `fieldSources`/timestamps are kept, so a restore is lossless.
- **No secrets.** No users row beyond username/display name, no sessions, tokens, connections, credentials or source snapshots.
- **Merges and aliases.** `works.jsonl` rows carry `mergedIntoId` (the survivor and ancestors are exported too) and
  `work_aliases.jsonl` holds the user's `{ fromWorkId, toWorkId }` rows. Both are additive to v1. Restore re-creates
  tombstones (for works it creates) and aliases, so a later write to the old work still resolves to the moved entry.
- **Export is one snapshot:** all reads run in a single transaction (repeatable read on Postgres).
- **Restore** (`importUserBundle`) needs an account with no entries (`account_not_empty`) and `format === "sttracker-export/v1"`
  (`unsupported_format`). Works with external IDs are re-matched (else created shared); works without become private.
  Ids are remapped (UUIDs inside history `before`/`after` and custom-field value keys too). Rows are inserted directly, so no
  new history is written. The import validates first (attachment mime/size/sha256, rating 0-100, status, visibility, 2 GB
  uncompressed cap) and writes all rows, attachments included, in one transaction; errors are
  `invalid_attachment`, `corrupt_bundle`, `bundle_too_large`, `conflict`. Works merged away at export time are exported as the tombstone row without `mergedIntoId`.
- **Backups** write `<dataDir>/backups/<ISO with ':'->'-'>/<username>.zip` per user plus `sttracker.db` (`VACUUM INTO`, SQLite only),
  are built in `<stamp>.tmp` then renamed (modes 0700/0600), and prune to the newest `keep` (default 14).
**Consequences:** Any change to these files after release needs `v2` and a migration path for old bundles.

## 2026-10-30: Sync engine: hash composition, anti-flip-flop, no revive, root ancestor, dedupe
**Status:** accepted
**Context:** Task 8 built the change engine (`packages/core/src/sync/`). Spec §8.3 fixes the outcomes; these
are the mechanics a maintainer would otherwise reverse-engineer.
**Decision:**
- **Hash composition.** sha256 of the stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list,
  deleted, mergedIntoKey}`. `key` and `raw` are excluded; `deleted` is always a boolean and `mergedIntoKey`
  always present (null), so an omitted flag and an explicit `false` hash alike. A version row stores
  base64(`Bun.gzipSync`) of `{normalized, raw}`: the normalised record is what the next change is diffed
  against (field-level diff = previous version vs. incoming), the raw payload is kept for the record.
- **Per-field decision.** Only fields that differ from the previous version are considered, and of those
  only values that differ from the current STTracker value. `field_sources[f]` unset or this connection's
  actor → applied; `"user"` → `pending changed {field, from: current, to: incoming}`; another source →
  `pending conflict {field, current, currentSource, incoming, incomingSource}`. A new record landing on an
  entry that already exists goes through the same per-field decision, so a first sync never overwrites
  what the user or another source wrote. Unchanged values are never sent, so a plugin never takes
  ownership of a field it did not change.
- **Anti-flip-flop.** A pending `changed`/`conflict` for the same (record, field, incoming value) is never
  filed twice. `keep_mine` re-stamps `field_sources[f] = "user"`, so later values from any source become a
  user-decided `changed`, never another source-vs-source conflict.
- **No revive.** If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing
  is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing.
  Only the user revives.
- **Root ancestor.** Creating an entry on a child work also creates a `status "none"` entry (plugin actor,
  empty `field_sources`) on the root ancestor work, unless the user has one, live or deleted.
- **Mapped records are never re-matched.** Only new records go through `matchWork`. A changed work hint
  refreshes the mapped work via `updateWorkFromHint` (shared works only).
- **Engagement dedupe.** Among live engagements on the entry not already sourced by this connection:
  finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and
  their dates (first 10 chars, computed with `Date.UTC`, no timezone shift) are ≤ 1 day apart; or start
  datetimes are the same instant. A match attaches an `engagement_sources` row and fills only fields that
  are empty and that no source has set; the first source's values otherwise stand.
- **Removal.** Absence counts only after a `full` pull whose iteration finished and whose records all
  applied without error; any record error fails the run and skips removal. A removed record that comes
  back clears `deleted_at` and retires its still-pending `removed` event as `ignored`.
- **Listens skip snapshots.** Listen records are not stored as `source_records` (they are high-volume and
  append-only). `listens` insert-or-ignore is the idempotence; they are never removed.
- **Merge accept.** The target work is the merge target record's mapped entry's work, else `matchWork` on
  its latest hint. Admin-merge events (`toWorkId`, no connection) resolve through the same `resolveChange`.

**Alternatives:**
- Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
- Hashing `raw` would turn every cosmetic API change into a new version.
- Keeping one run "ok" despite failing records risks marking those records removed.

**Consequences:** `list` records are snapshotted but not applied, and removal of a de-duplicated engagement
deletes it even if another source still reports it (both in `docs/TODO.md`).

## 2026-10-30: User layer: per-user moves merge entries, keep aliases, and roll the library up by hierarchy
**Status:** accepted
**Context:** Task 6 built the user layer (`packages/core/src/{entries,engagements,history,lists,copies}.ts`).
Several behaviours are not obvious from the spec and would otherwise have to be reverse-engineered.
**Decision:**
- **Merge on move.** `moveEntryToWork` onto a work where the user already has an entry merges the two:
  - Live target: the target's values win. Empty target fields are filled from the moved entry, carrying
    the moved entry's `field_sources`. Tags are unioned and custom fields and overrides merged; their
    `field_sources` is `"user"` if either side's was, otherwise the target's.
  - Differing non-empty `review`/`private_notes` are concatenated (target, `\n\n---\n\n`, moved).
  - A conflicting rating keeps the target's; the moved rating is recorded in the history row's `before`.
  - Engagements are re-parented, each with an `entryId` history row (restorable), and the moved entry is
    soft-deleted.
  - Soft-deleted target: revived with the moved entry's values and `field_sources` as-is.
- **Alias bookkeeping.** A move writes `work_aliases (user, old → target)`, re-points aliases that led to
  the old work, drops any alias leaving the target (no cycles), tombstones the user's own private draft,
  and re-points this user's copies (with history) and list items. Writes to an aliased or merged work
  (`upsertEntry`, copies, list items) resolve to the target. **Undoing a move** via `restoreVersion` deletes
  the forward alias and writes no reverse alias, so later syncs onto the target land on the target.
- **Library cursor.** `listEntries` cursors are base64 JSON `{ sort, sortValue, id }`. The sort is carried
  so a cursor cannot be replayed under another sort. For the title sort, `sortValue` is the SQL
  `lower(w.title)` of the last row, never a JS lowercase, so SQL and cursor agree on non-ASCII titles.
- **Custom fields patches replace the whole map** (`customFields` in an entry, engagement or copy patch).
  This makes restore exact. `field_sources` tracks the key `customFields` as a whole. Restore does not
  re-check values against today's definitions, because it is the user's own past data.
- **List items are hard-deleted.** `list_items` has no `deleted_at`. The list's `user_history` snapshot
  holds the ordered items, so `restoreVersion` brings removed items back.
- **Hierarchy visibility.** `childSummary` counts the user's live entries up to three levels below the
  entry's work (child, grandchild, great-grandchild). The default library view (`parentId: null`) lists
  entries with no ancestor, up to three levels, on which the user has a live entry. An episode whose show
  is untracked therefore stays visible, and is rolled up once the show gets an entry. Drilling in
  (`parentId: X`) applies the same rule relative to X: descendants up to three levels with no live-entry
  work strictly between them and X. `childSummary` still counts every descendant entry.
- **Concurrent creates.** `upsertEntry` inserts with `ON CONFLICT (user_id, work_id) DO NOTHING`, then
  patches through the update path, so a concurrent creator never aborts the caller's transaction.

**Alternatives:**
- Refusing a move onto an existing entry pushes the merge onto the user.
- Recursive CTEs for hierarchy are portable but heavier; three levels cover show → season → episode.
- A JS-lowercased cursor breaks on SQLite, whose `lower()` is ASCII-only.

**Consequences:** Undoing a move does not move copies or list items back, and hierarchy deeper than three
levels is not rolled up (both in `docs/TODO.md`).

## 2026-10-30: A single Kysely dialect over Bun.SQL for SQLite and Postgres
**Status:** accepted
**Context:** Kysely ships no driver for Bun. `Bun.SQL` speaks both `sqlite://` and `postgres://` behind one API.
**Decision:**
- `packages/db/src/dialect.ts` is one Kysely `Dialect` whose driver wraps `new SQL(url)`, choosing Kysely's
  SQLite or Postgres adapter, compiler and introspector by URL scheme.
- Postgres connections come from `sql.reserve()`. Bun's SQLite adapter does not support `reserve()`, so SQLite
  uses the single `SQL` instance as its only connection, serialised by a mutex so transactions never interleave.
- Transactions are plain `begin`/`commit`/`rollback` on the held connection.

**Alternatives:**
- Separate dialects over `bun:sqlite` and `Bun.sql`: two drivers to maintain for the same job.
- Community Bun dialect packages: an extra dependency, and none covers both databases.

**Consequences:** SQLite is effectively single-connection (fine for a self-hosted app). Savepoints and real
streaming are not implemented; `streamQuery` returns the whole result as one chunk.

## 2026-10-30: JSON columns are stored as text on both dialects
**Status:** accepted
**Context:** Spec §11 calls for `jsonb`. SQLite has no JSON type, so `jsonb` on Postgres would need
per-dialect read/write code in every service that touches a JSON column.
**Decision:** JSON columns are `text` everywhere, written with `toJson()` and read with `fromJson()` from
`@sttracker/db/json`. `fromJson` returns a caller-supplied fallback for null or invalid text.
**Alternatives:** `jsonb` on Postgres with per-dialect read/write code.
**Consequences:** No JSON operators or indexes in SQL; filtering on JSON content happens in app code. Can be
superseded later by a migration to `jsonb` if a query needs it.

## 2026-10-30: UUIDv7 instead of ULID for IDs
**Status:** accepted
**Context:** Spec and `AGENTS.md` say IDs are ULIDs generated in app code. Bun has `Bun.randomUUIDv7()` built in.
**Decision:** `newId()` in `@sttracker/db/ids` returns `Bun.randomUUIDv7()`. IDs are stored as `text`.
**Alternatives:** ULID, which needs a dependency or hand-written encoder.
**Consequences:** IDs are equally time-sortable as strings with no extra dependency. They are 36 characters
instead of 26. Where `AGENTS.md` or the spec say "ULID", read "UUIDv7".

## 2026-10-30: Product design decisions recorded in spec 001
**Status:** accepted
**Context:** Brainstorming with the owner produced the v1 design.
**Decision:** See `docs/specs/001-sttracker-v1-design.md` §13 for the designer's calls. Owner decisions:
- Phase 1 is read-only.
- Write-back is opt-in per connection.
- Conflicts between sources go to the Changes inbox.
- Backloggd waits for its official export.
- The app UI is Hardcover-style; the public page is scrapbook + river.
**Alternatives:** Recorded in the spec.
**Consequences:** Phase 1 has no `push` implementations.

## 2026-10-30: Bun workspaces monorepo with Kysely for both databases
**Status:** accepted
**Context:** The owner requires Bun with SQLite and Postgres options.
**Decision:**
- Bun workspaces (`apps/*`, `packages/*`, `plugins/*`).
- Kysely is the single query layer, with portable migrations.
- SQLite runs through `bun:sqlite` and Postgres through `Bun.sql`.

**Alternatives:**
- Drizzle needs a separate schema per dialect.
- Prisma does not fit Bun-native drivers.
- Raw SQL would duplicate every query per dialect.

**Consequences:** We may need a small custom Kysely dialect for `Bun.sql` or `bun:sqlite`. Every DB test runs on both databases.

## 2026-10-30: Adopt AGENTS.md as single source of truth
**Status:** accepted
**Context:** Agents had no durable project context and no consistent workflow between sessions.
**Decision:**
- `AGENTS.md` is canonical; `CLAUDE.md` and `GEMINI.md` are pointers.
- Specs are permanent in `docs/specs/`.
- Plans live in `docs/plans/` only while work is in progress and are removed before the merge request.
- Both locations override the superpowers default paths.

**Alternatives:**
- Per-tool rule files: they drift and create synchronisation debt.
- No rules file: agents re-explore the repo every session, which is measurably slower and costs more tokens.

**Consequences:** Rule changes happen in one file, and the pointer files must stay thin. Plans are working
artefacts: kept after merge, they go stale and contradict the code, so the spec is the record.

## 2026-10-30: Catalogue matching is race-safe through unique keys plus whole-transaction retry
**Status:** accepted
**Context:** Two syncs matching the same new work at once both miss the lookup and both insert. On
Postgres the loser hit the `external_ids (namespace, value)` unique key and crashed the sync; seasons and
episodes had no key at all and were duplicated.
**Decision:**
- Migration `0002_works_position_unique` adds a unique index on `works (parent_id, position)`. NULLs are
  distinct on both dialects, so top-level works and position-less children are unaffected.
- `matchWork` re-runs the whole resolve in a fresh transaction on a unique violation (SQLite
  "UNIQUE constraint failed", Postgres SQLSTATE 23505), up to 3 attempts. Postgres aborts the transaction
  on the error, so a statement-level retry is not possible.
- Adding missing external IDs to an existing work uses `ON CONFLICT DO NOTHING`.
- An ID match only counts if the work is shared or owned by the matching user, and private works can
  never carry external IDs (`createWork` refuses; global merges accept only shared works).
- A global merge repoints children of `from` to `to` unless `to` already has a child at that position.

**Alternatives:**
- Advisory or table locks: dialect-specific, and SQLite already serialises writes.
- Upserts throughout `createWork`: would hide genuine conflicts and need per-dialect `RETURNING` handling.

**Consequences:** Parents resolved without IDs or position (title/year reuse) still have no unique key
and can be duplicated under concurrency; tracked in `docs/TODO.md`.