STTracker/docs/plans/001-phase1-sdd/reports/task-6-report.md

Task 6 report: user layer (entries, engagements, history, lists, copies, custom fields)

Status: DONE_WITH_CONCERNS Branch: task/6 (worktree /home/user/STTracker/.worktrees/t6)

Takeover note

I took this task over from a previous implementer who was stopped by a usage-limit pause.

• Already committed by them: 56b86e2 feat(core): custom field definitions and value validation. I kept it unchanged.
• Uncommitted WIP from them: history.ts, entries.test.ts, history.test.ts. I reviewed all three and kept them, with these changes:
  • history.ts: restoreVersion now dispatches to engagement, copy and list restores as well as entries.
  • entries.test.ts: added ! to two array indexes to satisfy typecheck, and added one quickAdd media-type assertion.
  • history.test.ts: added a test for restoring a move.
• entries.ts did not exist yet. I wrote it, along with engagements, lists, copies and their tests.

Commits

• 13b81b1 feat(core): entries, user history and match suggestion accept/reject
• c7471f7 feat(core): engagements, lists and copies with history restore

Each commit was verified green on its own, on both databases, with typecheck clean. I split the work into two commits instead of the brief's single Step 5 message, because the controller asked for logical steps.

What I built

• history.ts
  • Actor type and recordHistory(tx, …), which stores only the changed keys and writes nothing if nothing changed.
  • getHistory, returning newest first.
  • restoreVersion, which re-applies before as actor "user" for entry, engagement, copy and list rows.
  • inTx helper: use the caller's tx if given, otherwise open a new transaction.
• entries.ts
  • Writes: upsertEntry, softDeleteEntry, moveEntryToWork, quickAdd. quickAdd uses a local insertPrivateWork and does not depend on Task 5.
  • Reads: getEntry and listEntries, both returning EntryView.
  • Suggestions: acceptSuggestion and rejectSuggestion.
  • Shared validators used by the other modules: rating, text, keys, partial date, default visibility, resolveVisibleWork, requireEntry, MEDIA_TYPES.
  • Data rules:
    • field_sources is set per patched key.
    • draft and childSummary are computed with correlated subqueries, not N+1 queries.
    • overrides (title, coverUrl, year only) are overlaid onto EntryView.work.
  • listEntries behaviour:
    • Keyset cursor is base64 JSON {sort, sortValue, id}. Default limit 50, capped at 200.
    • Sorts: updated desc, title (case-insensitive asc), rating (desc, nulls last).
    • Filters: mediaType (override-aware), status, tag (case-insensitive), draft, q (title or original title, LIKE-escaped), parentId (default null = top level).
• engagements.ts
  • addEngagement, updateEngagement, softDeleteEngagement, listEngagements.
  • Takes PartialDate. Value and precision are stored separately, and null is stored as null with precision "unknown".
  • A date's value must parse to exactly the stated precision; a full ISO datetime is accepted with precision day.
  • Custom fields are checked against engagement-level definitions.
  • listEngagements sorts by start date, with undated engagements last.
• lists.ts
  • createList, updateList, softDeleteList (an extra), addListItem (optional position), moveListItem (index is clamped), removeListItem, getList, listLists({kind?}).
  • Only a grouping may have rating or review.
  • Item positions stay dense, from 0 to n-1.
  • The list's history snapshot includes its items, so removing an item can be undone with restoreVersion.
• copies.ts
  • createCopy, updateCopy, softDeleteCopy, listCopies({workId?}). Actor defaults to "user", and there is an optional tx.
  • A copy needs either workId or titleHint.
  • priceCents must be a non-negative integer, and currency must be a three-letter ISO code.
  • acquiredAt is a partial date.
  • Custom fields are checked against copy-level definitions.
• Optional tx: every function the ruling lists takes tx? as its last parameter.
• Other users' rows: every function throws NotFoundError. Another user's private works also return NotFoundError.

Files

• packages/core/src/: history.ts, entries.ts, engagements.ts, lists.ts, copies.ts. custom-fields.ts came from the earlier commit.
• packages/core/test/: history.test.ts, entries.test.ts, engagements.test.ts, lists.test.ts, copies.test.ts, suggestions.test.ts. custom-fields.test.ts and user-layer-fixtures.ts came from the earlier commit.

All nine brief tests exist under their exact names. "custom select field rejects unknown option" is in custom-fields.test.ts, from the earlier commit.

TDD evidence

• RED (bun test packages/core/test, after writing all tests and before any implementation):

  error: Cannot find module '../src/entries.ts' …
  error: Cannot find module '../src/engagements.ts' …
  error: Cannot find module '../src/copies.ts' …
  error: Cannot find module '../src/lists.ts' …
   4 pass
   6 fail
  

• RED for the move-restore test (bun test packages/core/test/history.test.ts):

  (fail) history [sqlite] > restoreVersion undoes a move back to the draft work
  ValidationError: targetWorkId: same_work
  

  Fixed by deleting the user's alias before moving back.
• GREEN:
  • bun test packages/core (SQLite): 40 pass, 0 fail.
  • TEST_POSTGRES_URL=… bun test packages/core (both dialects): 79 pass, 0 fail.
  • Whole repo: bun test gives 73 pass; with Postgres, 118 pass, 0 fail.
  • Output is clean.
• Typecheck: bun run typecheck printed tsc --noEmit -p tsconfig.json and no errors.
• Not seen RED first: the quickAdd built-in media type check was added together with its assertion.

Decisions made (candidates for docs/DECISIONS.md)

• History snapshot keys:
  • Entry rows use the EntryPatch keys plus workId and deletedAt.
  • Engagement and copy rows use their patch keys plus deletedAt.
  • List rows use the list fields plus deletedAt and items.
  • The first history row of each entity is a creation row with an empty before; restoring it is a no-op.
• Works are resolved on every write: upsertEntry (and copy and list item writes) follow the user's work_aliases and global merged_into_id before writing, so writing to an old work id lands on the moved entry.
• Moving onto a work the user already has an entry for (the WIP test required this):
  • If the target entry is live, it is kept. Its empty fields are filled from the moved entry. Tags, custom fields and overrides are combined. Engagements are moved to the target entry, and the source entry is soft-deleted.
  • If the target entry is soft-deleted, the moved entry's values overwrite it and it is revived.
• Alias bookkeeping on a move:
  • Aliases that pointed at the old work are re-pointed to the target.
  • Any alias starting from the target is dropped, so no cycles form.
  • The user's own private draft work is tombstoned (merged_into_id).
• Restoring a move clears that tombstone and the alias, then moves the entry back.
• customFields patches replace the whole map, so restore is exact. field_sources key: customFields.
• Cursor format: the cursor also carries sort, so a cursor from one sort used with another sort throws ValidationError("cursor").
• quickAdd mediaType must be built-in (spec §6.1). User-defined types belong in mediaTypeOverride.
• acceptSuggestion throws ValidationError("status","not_pending") for a suggestion that is not pending. If the user has no live entry on the work, it only updates suggestion statuses.

Concerns

1. Hidden entries in the default list. The ruling's top-level filter is parent_id IS NULL and childSummary counts direct children only. As a result:

   • An episode entry whose show has no entry for this user does not appear in the default list.
   • A show → season → episode tree with no season entries rolls up 0 children on the show.

   Trakt imports create show and episode entries, so this will happen. Suggest a follow-up: either count grandchildren, or treat "parent has no live entry" as top level.

2. Docs duties not done. I did not edit docs/DECISIONS.md or docs/ARCHITECTURE.md, because the controller said to touch only my files while Tasks 5 and 7 run in parallel. The decisions above should go into DECISIONS.md at merge.

3. Search and title sort ignore overrides. q and the title sort use the work's title, not the user's override title, because overrides are a JSON text column. The tag filter matches the JSON text with LIKE.

4. Restore gaps after a merge-move. Engagements moved to the surviving entry are not written to engagement history. Restoring the source entry's soft-delete revives it on its old work, but the alias from that work stays in place.

5. Concurrent creates can race. Two simultaneous upsertEntry calls creating the same (user, work) entry fail on the unique constraint rather than retrying.

---

Fix round 1

Status: DONE Branch: task/6

Commits

• 4c65939 merge: feat/phase1 into task/6: no conflicts. Post-merge: 99 pass, typecheck clean.
• 512e6e9 fix(core): roll up descendants and surface orphaned entries; SQL-derived title cursor (items 1–3).
• 7154ca0 fix(core): race-safe entry upsert and lossless per-user moves (items 4–8 and the minors).
• 00fc018 docs: record user-layer decisions, core module map and known gaps (item 9).

Each commit was verified green on both dialects, with typecheck clean.

Findings addressed

1. childSummary now counts the user's live entries on works whose parent, grandparent or great-grandparent is the entry's work, through a portable 3-way OR over self-joined works. The rollup test gained the show → season (no entry) → episodes case.
2. The default list (parentId: null) now uses NOT EXISTS an ancestor, up to three levels, carrying a live entry of this user. New test "orphaned child entries are top level until an ancestor has an entry". It also checks that another user's entry and a soft-deleted entry on the show do not hide the episode.
3. The title cursor now carries the SQL lower(w.title) value, selected as title_key. New test "title pagination does not skip non-ASCII titles" pages one item at a time through seven mixed titles.
4. upsertEntry inserts defaults with onConflict(user_id, work_id).doNothing(), re-reads the row, and applies the patch through writeEntry. On a fresh create it writes a single history row with before = null. It never catches and retries.
   • New test "concurrent upserts…" uses Promise.all. It passed 5 of 5 repeated runs on Postgres.
5. Undoing a move (restoreEntryFromHistory) deletes the forward alias D→S, un-tombstones the draft, and calls moveEntry with { undo: true }. That mode skips all alias, copy and list bookkeeping.
   • The history test now asserts that no aliases remain, and that upsertEntry(S) afterwards creates a separate entry on S while the draft entry stays untouched.
6. field_sources on merge:
   • Combined fields (tags, customFields, overrides, concatenated text) get "user" if either side had "user", otherwise the destination's value.
   • A gap-filled field carries the source entry's value.
   • A revived soft-deleted destination gets the source's field_sources as-is; keys absent there are removed rather than stamped with the actor.
   • writeEntry now takes opts.sources, where undefined deletes the key. Two tests added.
7. Merge history:
   • Each re-parented engagement (live or deleted) gets updated_at bumped and a user_history row with {entryId} before and after.
   • The engagement snapshot now includes entryId, and engagement restore re-parents back after checking ownership.
   • Differing non-empty review/privateNotes are joined with \n\n---\n\n.
   • A conflicting rating keeps the destination's, and the destination's history row records before.rating = <source rating> and after.rating = <dst rating>. Restoring that row therefore applies the source rating.
   • One test covers all of this.
8. Copies and list items follow a move: repointCopiesAndListItems moves this user's copies (with copy history {workId}) and the list_items in this user's lists. New test in suggestions.test.ts via acceptSuggestion.
9. Docs: merged feat/phase1 first.
   • One entry added at the top of docs/DECISIONS.md covering every bullet asked for: merge-on-move, alias bookkeeping with undo, cursor format, whole-map customFields, list_items hard-delete, hierarchy visibility, plus the concurrent-create approach.
   • The packages/core row in docs/ARCHITECTURE.md now lists the existing modules and is marked "exists (partial)".
   • docs/TODO.md gained three known gaps: search ignores overrides, rollup stops at three levels, and undoing a move does not move copies or list items back.

Minors:

• rejectSuggestion throws ValidationError("status","not_pending") for a suggestion that is not pending.
• Engagement update and soft-delete throw NotFoundError when the parent entry is soft-deleted.
• Entry restore skips custom-field validation and only checks that the value is an object.

Each has a test.

TDD evidence

• RED: ran the new and updated tests against the pre-fix source by stashing packages/core/src.
  • bun test packages/core gave 10 failures: the move undo, restore custom fields, deleted-entry engagement, rejectSuggestion when not pending, copies and list items after accept, rollup grandchildren, orphans, non-ASCII pagination, merge into deleted, and merge text/rating/engagements.
  • With TEST_POSTGRES_URL=…, the concurrent test failed on Postgres.
  • The combined-sources test passed by coincidence at first, so I strengthened it (plugin source vs user destination) and re-ran it: 1 fail.
• GREEN (final):
  • bun test packages/core packages/db (SQLite): 108 pass, 0 fail.
  • The same with TEST_POSTGRES_URL=postgres://postgres:postgres@127.0.0.1:55432/postgres: 207 pass, 0 fail.
  • bun run typecheck: tsc --noEmit -p tsconfig.json with no errors.
  • git status is clean.

Remaining concerns

• Undoing a move is partial. Copies and list items are not moved back, and aliases that the move re-pointed (X→D became X→S) are not restored. Both are recorded in TODO.
• Restoring a merge's rating row applies the source rating, by design of the "record in before" ruling.
• Inconsistent DECISIONS.md order. The new entry is at the top, as instructed ("newest first"), but Task 5 appended its entry at the bottom, so the file's order is mixed.

---

Fix round 2

Status: DONE Commit: c6865b2 fix(core): drill-down lists descendants under untracked intermediates

What changed

• Drill-down rule. listEntries({ parentId: X }) now uses underParent(X), the top-level rule applied relative to X. It returns this user's live entries whose work is:

  • a direct child of X; or
  • a grandchild of X, when the parent between them (p1) carries no live entry of this user; or
  • a great-grandchild of X, when neither p1 nor p2 carries a live entry of this user.

  It is portable SQL using exists / not exists. The other filters (q, tag, and so on) still apply. The top-level ancestor filter is used only when parentId is null.

• Restore onto a deleted entry. Restoring an engagement's {entryId} history row onto a soft-deleted entry now throws ValidationError("entryId", "target entry is deleted").

• Docs. The hierarchy bullet in this branch's (still unmerged) DECISIONS entry now states the drill-down rule.

Tests

New assertions in entries.test.ts "listEntries rolls up children", using the show2 fixture:

• Drilling into the show lists both episodes.
• Drilling with a non-matching q returns nothing.
• After a season entry exists, drilling into the show lists only the season.
• Drilling into the season lists the episodes.

New test in engagements.test.ts: "restoring a re-parenting onto a soft-deleted entry is rejected".

TDD evidence

• RED (bun test packages/core, before the fix): 99 pass, 2 fail.
  • (fail) entries [sqlite] > listEntries rolls up children
  • (fail) engagements [sqlite] > restoring a re-parenting onto a soft-deleted entry is rejected
• GREEN:
  • bun test packages/core packages/db: 109 pass, 0 fail.
  • With TEST_POSTGRES_URL=…: 209 pass, 0 fail.
• Typecheck: bun run typecheck reports no errors.
• Worktree: git status is clean.

Note

childSummary counts all descendant entries up to three levels. Once the season has an entry, the show's summary is 3 (the season plus 2 episodes), while drilling into the show lists only the season. That matches the ruling: the rollup counts everything under the show, and the drill-down shows the next tracked level.