STTracker/docs/DECISIONS.md

Decisions

Append-only. Never edit or delete an entry. Supersede it with a new one that links back. One entry per decision a future maintainer would otherwise have to reverse-engineer. No index: grep -n '^## ' docs/DECISIONS.md lists every entry (date, title, line number).

2026-10-30: Deployment: one image on oven/bun:1.3, source run directly, compose with an optional Postgres profile

Status: accepted Context: Task 19 packages the server. Bun.serve bundles the apps/web HTML import at runtime, so there is no build step to ship. Decision:

• The image is oven/bun:1.3 (tag, not digest) running bun apps/server/src/main.ts from source, as the non-root bun user. It installs production dependencies only and copies apps/web too, because the server bundles it at startup.
• /data is the volume (DATA_DIR=/data): SQLite file, attachments, backups, user plugins. It is created and owned by bun in the image.
• docker-compose.yml has the sttracker service (./data:/data, env_file: .env) and a postgres:17-alpine service under the postgres profile with a named volume. SQLite stays the default.
• CI is .gitea/workflows/ci.yml (GitHub-compatible syntax): typecheck, then bun test against a Postgres service, plus the AGENTS.md 150-line guard. Alternatives: A compiled single binary (bun build --compile) was rejected: user plugins load from disk at runtime and the web bundle is built on demand. Consequences: Image start time includes bundling the web app. Agent permission files for the new Docker commands are pending the owner (docs/TODO.md).

2026-10-30: HTTP API: auth modes, CSRF header, error mapping, actor

Status: accepted Context: Task 16 (apps/server). Decision:

• Auth modes per route: none, session (cookie only: tokens, sources create/edit/delete, device auth, bundle import, admin), token:<scope>, any:<scope> (cookie or Bearer). A Bearer token valid for another scope gets 403, an unknown or revoked one 401. App tokens cannot mint tokens or handle source credentials.
• CSRF: cookie-authenticated non-GET requests must send X-Requested-With: sttracker (403 otherwise). Browsers cannot add it cross-site without a CORS preflight, which the server never grants. Bearer requests are exempt. Cookie is stt_session, HttpOnly, SameSite=Lax, Secure when PUBLIC_URL is https, 30 days.
• Error mapping: zod and ValidationError → 400 validation (field: reason); AuthError → 401 (invalid_credentials, unauthenticated), 403 (forbidden), 400 with its own code (invite_required, username_taken, invalid_username); NotFoundError → 404 not_found; ImportError → 400 with its code (409 for account_not_empty, conflict); unique violations → 409 conflict; body over the cap → 413 validation; rate limit → 429 rate_limited; anything else → 500 internal with a fixed message, detail logged only.
• Limits: JSON bodies 1 MB, multipart 50 MB (counted while streaming, not trusted from Content-Length). Login and register: 10 per minute per client IP, in process (resets on restart; not shared across instances).
• Actor: every API write uses actor "user", including token-authenticated ones, because the change engine only protects fields whose field_sources is "user". A token:<name> actor would let the next sync overwrite a script's edit.
• Daily backup: queued for the next 03:00 local with lockKey "backup" at boot, 5 s after each run (the finishing job still holds the lock while it runs), and hourly as a safety net. All three are the same idempotent enqueue.
• Deleted connections (paused with no credentials) are hidden from GET sources and 404 on every sources/:id route.
• File imports travel in the job payload as base64 ({ connectionId, file: { name, base64 } }); syncHandler decodes them into runSync's { file } mode, recorded as run mode file.
• OpenAPI 3.1 is generated from the route table (z.toJSONSchema, input side, unrepresentable → any): every route with method, path, summary, security and request schema. Responses are not typed yet.

2026-10-30: Job failure semantics and plugin reload limits

Status: accepted Context: Task 9 fix round 1. Decision:

• Failures before runSync never retry. A missing/paused connection drops the job (DropJobError: done, message in last_error). A missing plugin or undecryptable credentials (CryptoError) insert a failed source_runs row, set the connection to error, advance last_run_at and throw NonRetryableError (job failed, no backoff). The scheduler still picks error connections, one interval later, so fixing the cause heals them.
• Stale locks. Every worker tick re-queues running jobs whose locked_at is over an hour old (attempts unchanged), so a crash cannot hold a lockKey forever. Job bookkeeping errors are logged ([jobs]), never unhandled; stop() never rejects and aborts the signal passed to handlers (handler(payload, { signal })).
• Reload evicts the entry file from require.cache and imports with ?v=<Date.now()> (Bun ignores the query alone for cached .ts). Only entry files reload: changes in a plugin's other modules need a server restart.
• credentials.save and updateConnection read-merge-write in one transaction (row lock on Postgres). ctx.log is buffered and appended to the run's source_runs.log when the run ends.

2026-10-30: Jobs, plugin registry, connections: rate limit in the manifest, secret split, per-connection lock

Status: accepted Context: Task 9 (packages/core/src/{jobs,registry,connections}.ts). Decision:

• Rate limit lives in the manifest (PluginManifest.rateLimit?: { perMinute, burst? }), not in plugin code. The host builds ctx.http from it (default 60/min, burst 5, UA STTracker/0.1 (+self-hosted)), so every plugin is throttled the same way and the per-plugin RATE_PER_MINUTE constants are gone.
• Secrets are split off config. Config fields typed "secret" in the manifest are stored only in the encrypted credentials_enc blob (with OAuth tokens a plugin saves), never in config. makePluginContext merges them back into ctx.config; listConnections returns plain config and hasCredentials only. updateConnection keeps secrets that are omitted and replaces plain config wholesale. credentials.save merges and persists immediately, so a token refreshed mid-pull survives a crash.
• Constructor deviation. createConnection/updateConnection take the plugin manifest as a trailing argument (needed for the split), and makePluginContext is async (it decrypts) and takes the manifest before an optional http.
• One sync per connection. Sync jobs carry lockKey = "sync:<connectionId>"; enqueue returns null while a queued|running job holds the key. On Postgres the check and insert run under pg_advisory_xact_lock(hashtext(key)). Claiming is update … where status = 'queued' checked on affected rows. Retries back off 2^attempts minutes, 5 attempts, then failed.
• Mode choice. The sync handler runs full when asked, when the plugin lacks pullChanges, or when no ok full run started within 7 days; else changes. A failed run is rethrown so the worker retries it. Consequences: Reload re-imports plugin modules with ?v=<Date.now()>; old module instances are not freed by Bun.

2026-10-30: Sync engine: hash composition, anti-flip-flop, no revive, root ancestor, dedupe

Status: accepted Context: Task 8 built the change engine (packages/core/src/sync/). Spec §8.3 fixes the outcomes; these are the mechanics a maintainer would otherwise reverse-engineer. Decision:

• Hash composition. sha256 of the stable-key-ordered JSON of {kind, work, entry|engagement|listen|list, deleted, mergedIntoKey}. key and raw are excluded; deleted is always a boolean and mergedIntoKey always present (null), so an omitted flag and an explicit false hash alike. A version row stores base64(Bun.gzipSync) of {normalized, raw}: the normalised record is what the next change is diffed against (field-level diff = previous version vs. incoming), the raw payload is kept for the record.
• Per-field decision. Only fields that differ from the previous version are considered, and of those only values that differ from the current STTracker value. field_sources[f] unset or this connection's actor → applied; "user" → pending changed {field, from: current, to: incoming}; another source → pending conflict {field, current, currentSource, incoming, incomingSource}. A new record landing on an entry that already exists goes through the same per-field decision, so a first sync never overwrites what the user or another source wrote. Unchanged values are never sent, so a plugin never takes ownership of a field it did not change.
• Anti-flip-flop and superseding. A pending changed/conflict for the same (record, field, incoming value) is never filed twice. A different newer value for the same (record, field) resolves the older pending event as ignored with payload.supersededBy = <new event id>, so at most one is pending per field. If the source comes round to the value STTracker already holds, its pending events for that field are resolved ignored with payload.reason = "source_agrees". keep_mine re-stamps field_sources[f] = "user", so later values from any source become a user-decided changed, never another source-vs-source conflict.
• No revive. If the mapped entry/engagement is soft-deleted, the record is still snapshotted but nothing is applied and no event is filed; a new record on a work whose entry the user deleted adds nothing. Only the user revives.
• Root ancestor. Creating an entry on a child work also creates a status "none" entry (plugin actor, empty field_sources) on the root ancestor work, unless the user has one, live or deleted.
• Mapped records are never re-matched. Only new records go through matchWork. Its result is committed at once as the record's row (current_hash "", target_entity "work"), so a failed apply is retried against the same work and an ID-less hint never leaks a second draft. A changed work hint refreshes the mapped work via updateWorkFromHint (shared works only). moveEntryToWork re-points source_records from a merged-away entry to the survivor in the same transaction, so a user's merge never orphans a source.
• Engagement dedupe. Among live engagements on the entry not already sourced by this connection: finish dates match when both are datetimes ≤ 24 h apart as instants, or when both are day precision and their dates (first 10 chars, computed with Date.UTC, no timezone shift) are ≤ 1 day apart; or start datetimes are the same instant. A match attaches an engagement_sources row and fills fields that are empty and that no source has set. Every other differing field goes through the per-field decision (so a disagreeing rating is a conflict), except the date that identified the play, whose difference is within the tolerance by definition.
• Removal. Absence counts only after a full pull whose iteration finished and whose records all applied without error; any record error fails the run and skips removal. A removed record that comes back clears deleted_at and retires its still-pending removed event as ignored. A merged-away record (merged_into_key set, or a pending merged event) is expected to vanish: no removal, ever. Accepting removed on an engagement other sources still report only deletes this connection's engagement_sources row; the engagement is soft-deleted when no source remains. Run counts include only records whose transaction committed.
• Listens skip snapshots. Listen records are not stored as source_records (they are high-volume and append-only). listens insert-or-ignore is the idempotence; they are never removed. Per batch of 500: one lookup of existing keys, each distinct recording resolved once, one multi-row insert.
• Lists. A new list record creates a user list (kind list, description Imported from <plugin id>), items matched outside the transaction (an ID-less item first reuses the user's own private work with the same media type and title). Lists have no field_sources; ownership is derived from user_history: name/description belong to whoever last changed them, items to the user once any user edit exists. A user-owned field becomes a pending changed (for items, the whole incoming item set); otherwise the fields are applied and items replaced (replaceListItems, one history row).
• Merge accept. The target work is the merge target record's mapped entry's work, else matchWork on its latest hint. Admin-merge events (toWorkId, no connection) resolve through the same resolveChange.
• Gone targets. Accept/keep_mine on an event whose target is deleted (or merged away with no live survivor, found through the user's work alias) resolves it ignored with payload.reason = "target_gone".

Alternatives:

• Diffing against the live entry instead of the previous version would re-file decided conflicts each run.
• Hashing raw would turn every cosmetic API change into a new version.
• Keeping one run "ok" despite failing records risks marking those records removed.

Consequences: Concurrent runs of one connection are not locked here (Task 9's worker; docs/TODO.md).

2026-10-30: Export format sttracker-export/v1: per-user zip, camelCase JSONL, deleted rows included, no secrets

Status: accepted Context: Task 15 (packages/core/src/{export,backup}.ts) fixed the on-disk bundle format that backups and restore share. Decision:

• Files. A zip (fflate) holding manifest.json ({ format, exportedAt, user: { username, displayName } }), works.jsonl, entries.jsonl, engagements.jsonl, listens.jsonl, lists.jsonl (items nested), copies.jsonl, custom_fields.jsonl (field definitions), history.jsonl, attachments.jsonl (links nested) and attachments/<sha256> bytes. One JSON object per line, \n separated.
• Keys. camelCase; JSON columns are parsed into real JSON; favorite is a boolean. user_id columns are omitted (the importing user owns everything). works.jsonl adds private and externalIds.
• Deleted rows are included with their deletedAt, and fieldSources/timestamps are kept, so a restore is lossless.
• No secrets. No users row beyond username/display name, no sessions, tokens, connections, credentials or source snapshots.
• Merges and aliases. works.jsonl rows carry mergedIntoId (the survivor and ancestors are exported too) and work_aliases.jsonl holds the user's { fromWorkId, toWorkId } rows. Both are additive to v1. Restore re-creates tombstones (for works it creates) and aliases, so a later write to the old work still resolves to the moved entry.
• Export is one snapshot: all reads run in a single transaction (repeatable read on Postgres).
• Restore (importUserBundle) needs an account with no entries (account_not_empty) and format === "sttracker-export/v1" (unsupported_format). Works with external IDs are re-matched (else created shared); works without become private. Ids are remapped (UUIDs inside history before/after and custom-field value keys too). Rows are inserted directly, so no new history is written. The import validates first (attachment mime/size/sha256, rating 0-100, status, visibility, 2 GB uncompressed cap) and writes all rows, attachments included, in one transaction; errors are invalid_attachment, corrupt_bundle, bundle_too_large, conflict. Works merged away at export time are exported as the tombstone row with its mergedIntoId.
• Backups write <dataDir>/backups/<ISO with ':'->'-'>/<username>.zip per user plus sttracker.db (VACUUM INTO, SQLite only), are built in <stamp>.tmp then renamed (modes 0700/0600), and prune to the newest keep (default 14). Consequences: Any change to these files after release needs v2 and a migration path for old bundles.

2026-10-30: User layer: per-user moves merge entries, keep aliases, and roll the library up by hierarchy

Status: accepted Context: Task 6 built the user layer (packages/core/src/{entries,engagements,history,lists,copies}.ts). Several behaviours are not obvious from the spec and would otherwise have to be reverse-engineered. Decision:

• Merge on move. moveEntryToWork onto a work where the user already has an entry merges the two:
  • Live target: the target's values win. Empty target fields are filled from the moved entry, carrying the moved entry's field_sources. Tags are unioned and custom fields and overrides merged; their field_sources is "user" if either side's was, otherwise the target's.
  • Differing non-empty review/private_notes are concatenated (target, \n\n---\n\n, moved).
  • A conflicting rating keeps the target's; the moved rating is recorded in the history row's before.
  • Engagements are re-parented, each with an entryId history row (restorable), and the moved entry is soft-deleted.
  • Soft-deleted target: revived with the moved entry's values and field_sources as-is.
• Alias bookkeeping. A move writes work_aliases (user, old → target), re-points aliases that led to the old work, drops any alias leaving the target (no cycles), tombstones the user's own private draft, and re-points this user's copies (with history) and list items. Writes to an aliased or merged work (upsertEntry, copies, list items) resolve to the target. Undoing a move via restoreVersion deletes the forward alias and writes no reverse alias, so later syncs onto the target land on the target.
• Library cursor. listEntries cursors are base64 JSON { sort, sortValue, id }. The sort is carried so a cursor cannot be replayed under another sort. For the title sort, sortValue is the SQL lower(w.title) of the last row, never a JS lowercase, so SQL and cursor agree on non-ASCII titles.
• Custom fields patches replace the whole map (customFields in an entry, engagement or copy patch). This makes restore exact. field_sources tracks the key customFields as a whole. Restore does not re-check values against today's definitions, because it is the user's own past data.
• List items are hard-deleted. list_items has no deleted_at. The list's user_history snapshot holds the ordered items, so restoreVersion brings removed items back.
• Hierarchy visibility. childSummary counts the user's live entries up to three levels below the entry's work (child, grandchild, great-grandchild). The default library view (parentId: null) lists entries with no ancestor, up to three levels, on which the user has a live entry. An episode whose show is untracked therefore stays visible, and is rolled up once the show gets an entry. Drilling in (parentId: X) applies the same rule relative to X: descendants up to three levels with no live-entry work strictly between them and X. childSummary still counts every descendant entry.
• Concurrent creates. upsertEntry inserts with ON CONFLICT (user_id, work_id) DO NOTHING, then patches through the update path, so a concurrent creator never aborts the caller's transaction.

Alternatives:

• Refusing a move onto an existing entry pushes the merge onto the user.
• Recursive CTEs for hierarchy are portable but heavier; three levels cover show → season → episode.
• A JS-lowercased cursor breaks on SQLite, whose lower() is ASCII-only.

Consequences: Undoing a move does not move copies or list items back, and hierarchy deeper than three levels is not rolled up (both in docs/TODO.md).

2026-10-30: Catalogue matching is race-safe through unique keys plus whole-transaction retry

Status: accepted Context: Two syncs matching the same new work at once both miss the lookup and both insert. On Postgres the loser hit the external_ids (namespace, value) unique key and crashed the sync; seasons and episodes had no key at all and were duplicated. Decision:

• Migration 0002_works_position_unique adds a unique index on works (parent_id, position). NULLs are distinct on both dialects, so top-level works and position-less children are unaffected.
• matchWork re-runs the whole resolve in a fresh transaction on a unique violation (SQLite "UNIQUE constraint failed", Postgres SQLSTATE 23505), up to 3 attempts. Postgres aborts the transaction on the error, so a statement-level retry is not possible.
• Adding missing external IDs to an existing work uses ON CONFLICT DO NOTHING.
• An ID match only counts if the work is shared or owned by the matching user, and private works can never carry external IDs (createWork refuses; global merges accept only shared works).
• A global merge repoints children of from to to unless to already has a child at that position.

Alternatives:

• Advisory or table locks: dialect-specific, and SQLite already serialises writes.
• Upserts throughout createWork: would hide genuine conflicts and need per-dialect RETURNING handling.

Consequences: Parents resolved without IDs or position (title/year reuse) still have no unique key and can be duplicated under concurrency; tracked in docs/TODO.md.

2026-10-30: A single Kysely dialect over Bun.SQL for SQLite and Postgres

Status: accepted Context: Kysely ships no driver for Bun. Bun.SQL speaks both sqlite:// and postgres:// behind one API. Decision:

• packages/db/src/dialect.ts is one Kysely Dialect whose driver wraps new SQL(url), choosing Kysely's SQLite or Postgres adapter, compiler and introspector by URL scheme.
• Postgres connections come from sql.reserve(). Bun's SQLite adapter does not support reserve(), so SQLite uses the single SQL instance as its only connection, serialised by a mutex so transactions never interleave.
• Transactions are plain begin/commit/rollback on the held connection.

Alternatives:

• Separate dialects over bun:sqlite and Bun.sql: two drivers to maintain for the same job.
• Community Bun dialect packages: an extra dependency, and none covers both databases.

Consequences: SQLite is effectively single-connection (fine for a self-hosted app). Savepoints and real streaming are not implemented; streamQuery returns the whole result as one chunk.

2026-10-30: JSON columns are stored as text on both dialects

Status: accepted Context: Spec §11 calls for jsonb. SQLite has no JSON type, so jsonb on Postgres would need per-dialect read/write code in every service that touches a JSON column. Decision: JSON columns are text everywhere, written with toJson() and read with fromJson() from @sttracker/db/json. fromJson returns a caller-supplied fallback for null or invalid text. Alternatives: jsonb on Postgres with per-dialect read/write code. Consequences: No JSON operators or indexes in SQL; filtering on JSON content happens in app code. Can be superseded later by a migration to jsonb if a query needs it.

2026-10-30: UUIDv7 instead of ULID for IDs

Status: accepted Context: Spec and AGENTS.md say IDs are ULIDs generated in app code. Bun has Bun.randomUUIDv7() built in. Decision: newId() in @sttracker/db/ids returns Bun.randomUUIDv7(). IDs are stored as text. Alternatives: ULID, which needs a dependency or hand-written encoder. Consequences: IDs are equally time-sortable as strings with no extra dependency. They are 36 characters instead of 26. Where AGENTS.md or the spec say "ULID", read "UUIDv7".

2026-10-30: Product design decisions recorded in spec 001

Status: accepted Context: Brainstorming with the owner produced the v1 design. Decision: See docs/specs/001-sttracker-v1-design.md §13 for the designer's calls. Owner decisions:

• Phase 1 is read-only.
• Write-back is opt-in per connection.
• Conflicts between sources go to the Changes inbox.
• Backloggd waits for its official export.
• The app UI is Hardcover-style; the public page is scrapbook + river. Alternatives: Recorded in the spec. Consequences: Phase 1 has no push implementations.

2026-10-30: Bun workspaces monorepo with Kysely for both databases

Status: accepted Context: The owner requires Bun with SQLite and Postgres options. Decision:

• Bun workspaces (apps/*, packages/*, plugins/*).
• Kysely is the single query layer, with portable migrations.
• SQLite runs through bun:sqlite and Postgres through Bun.sql.

Alternatives:

• Drizzle needs a separate schema per dialect.
• Prisma does not fit Bun-native drivers.
• Raw SQL would duplicate every query per dialect.

Consequences: We may need a small custom Kysely dialect for Bun.sql or bun:sqlite. Every DB test runs on both databases.

2026-10-30: Adopt AGENTS.md as single source of truth

Status: accepted Context: Agents had no durable project context and no consistent workflow between sessions. Decision:

• AGENTS.md is canonical; CLAUDE.md and GEMINI.md are pointers.
• Specs are permanent in docs/specs/.
• Plans live in docs/plans/ only while work is in progress and are removed before the merge request.
• Both locations override the superpowers default paths.

Alternatives:

• Per-tool rule files: they drift and create synchronisation debt.
• No rules file: agents re-explore the repo every session, which is measurably slower and costs more tokens.

Consequences: Rule changes happen in one file, and the pointer files must stay thin. Plans are working artefacts: kept after merge, they go stale and contradict the code, so the spec is the record.