From fb4ff3660285d7399d29829943561cedccea7cc6 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Fri, 30 Oct 2026 15:24:48 +0100 Subject: [PATCH] docs: phase 1 implementation plan Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-phase1-own-your-data.md | 642 +++++++++++++++++++++++++ 1 file changed, 642 insertions(+) create mode 100644 docs/plans/001-phase1-own-your-data.md diff --git a/docs/plans/001-phase1-own-your-data.md b/docs/plans/001-phase1-own-your-data.md new file mode 100644 index 0000000..872d748 --- /dev/null +++ b/docs/plans/001-phase1-own-your-data.md @@ -0,0 +1,642 @@ +--- +status: in-progress +--- + +# Phase 1, "Own your data": implementation plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task by task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A self-hosted Bun app where users connect Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm and CSV/JSON sources. STTracker pulls and snapshots their data, detects platform-side changes into a Changes inbox, and never overwrites what the user wrote. Users can log anything without IDs and export or back up everything. SQLite or Postgres, shipped in Docker. + +**Architecture:** Bun workspaces. +- `packages/db`: Kysely over a single custom dialect wrapping `Bun.SQL`, which speaks both `sqlite://` and `postgres://`. +- `packages/plugin-sdk`: the plugin contract. +- `packages/core`: domain services and the sync/change engine. +- `plugins/*`: source integrations. They return `SourceRecord`s and never touch the DB. +- `apps/server`: `Bun.serve` API, job worker, and serving the web app. +- `apps/web`: React app via HTML imports. + +**Tech stack:** Bun 1.3.14, TypeScript 5.9, Kysely 0.28+, zod 4, fflate, React 19, wouter, @tanstack/react-query. + +**Spec:** `docs/specs/001-sttracker-v1-design.md` (read §6–§11 before any task). Design: `DESIGN.md`, `docs/design/mockups/round2-app-scrapbook-river.html`. + +## Global constraints + +- Follow `AGENTS.md` rules. In particular: Bun-native APIs only; plugins import only `@sttracker/plugin-sdk`; syncs never overwrite user-edited fields. +- **Runtime deps are approved for this plan only:** + - `kysely`, `zod`, `fflate` (root); + - `react`, `react-dom`, `wouter`, `@tanstack/react-query` (apps/web); + - `@types/react`, `@types/react-dom` (dev). + - Ask before adding anything else. +- **IDs:** `newId()` from `@sttracker/db/ids`. It returns `Bun.randomUUIDv7()`. (UUIDv7 replaces the spec's ULID: it is equally time-sortable and built in. This is recorded in DECISIONS by Task 1.) +- **Timestamps:** ISO-8601 UTC strings (`new Date().toISOString()`) stored as `text` on both dialects. +- **JSON columns:** `text` on both dialects. Use `toJson()`/`fromJson()` from `@sttracker/db/json`. (A simplification of spec §11's jsonb; recorded in DECISIONS by Task 1.) +- **Ratings:** integer 0–100 or null. +- **Partial dates:** `{ value: string; precision: "day" | "month" | "year" | "unknown" }`. `value` is `YYYY-MM-DD`, `YYYY-MM`, `YYYY` or `""`, or a full ISO datetime when the source gives a time. Partial dates are never converted through `Date`, which would shift them by timezone. +- **Status enum:** `"planned" | "in_progress" | "paused" | "completed" | "dropped" | "none"`. +- **MediaType enum:** `book comic manga game movie show season episode podcast podcast_episode audio_drama artist album track video other`. +- **Package exports:** each package's `package.json` has `"exports": { "./*": "./src/*.ts" }`. Import by path (e.g. `@sttracker/core/entries`). There are no barrel `index.ts` files, which avoids merge conflicts between parallel tasks. +- **DB tests** use `forEachDialect()` from `@sttracker/db/testing` and must pass on SQLite always, and on Postgres when `TEST_POSTGRES_URL` is set. Local value: `TEST_POSTGRES_URL=postgres://postgres:postgres@127.0.0.1:55432/postgres` (container `sttracker-pg-test`). +- **Tests never hit the network.** Plugins are tested with fixtures in `plugins//test/fixtures/*.json`, written by hand from the documented API shapes, and played through `createFixtureHttp()`. +- **Attribution strings** are shown on the About page (exact text): + - "STTracker uses Trakt but is not endorsed or certified by Trakt." + - "Manga data from MangaBaka (CC BY-NC-SA 4.0)." + - "Music data from ListenBrainz and Last.fm." + - "Book data from Hardcover." +- **Commits:** Conventional Commits, ending with the line `Co-Authored-By: Claude Opus 5.5 `. + +## Review focus + +1. **A full pull that fails part-way** (network error, 429s exhausted, thrown parser error) must mark *nothing* as removed. Only a pull that finished emits `removed` events. Test lives in Task 8. +2. **Running the same sync twice with unchanged source data** creates zero new `source_record_versions`, `change_events`, `engagements` or `listens`. Test lives in Task 8. +3. **A user edits a field, then the source changes it:** the user's value stays and a pending change appears. The same goes for two sources disagreeing: a pending `conflict`, with no flip-flopping on the next run. Test lives in Task 8 on both dialects. +4. **Messy CSV input** must round-trip without data loss: UTF-8 BOM, `;` delimiter, quoted newlines and quotes, empty rating, "2019" as a date, emoji titles. Test lives in Task 13. +5. **Rotating single-use refresh tokens (Trakt):** refreshed credentials are saved before any further request, and two syncs of the same connection never run at once. Tests live in Task 10 (save-before-continue) and Task 9 (connection lock). + +--- + +## Dependency graph and waves + +``` +Wave A (parallel): T1 db-core T3 plugin-sdk +Wave B: T2 schema (needs T1) +Wave C (parallel): T4 auth T5 catalogue+matching T6 user-layer T7 attachments+crypto + T10 trakt T11 hardcover T12 mangabaka T13 csv/json T14 listenbrainz+lastfm (plugins need only T3) +Wave D (parallel): T8 sync engine (T3,T5,T6) T9 jobs+registry (T2,T3,T7) T15 export/backup (T5,T6,T7) +Wave E: T16 server API (all core) +Wave F (parallel): T17 web shell+library+detail T18 web changes/sources/lists/owned/settings/export T19 docker+CI+docs +``` + +Parallel tasks run in separate git worktrees branched from `feat/phase1` and are merged back into it after review. + +--- + +### Task 1: `packages/db`: dialect, ids, json, migrations runner, test harness + +**Files:** +- Create: `packages/db/package.json` (`@sttracker/db`, deps `kysely`), `packages/db/src/dialect.ts`, `packages/db/src/db.ts`, `packages/db/src/ids.ts`, `packages/db/src/json.ts`, `packages/db/src/migrate.ts`, `packages/db/src/testing.ts` +- Create: workspace `package.json` stubs with `exports` for `packages/core` (`@sttracker/core`), `packages/plugin-sdk` (`@sttracker/plugin-sdk`), `apps/server` (`@sttracker/server`), `apps/web` (`@sttracker/web`). Add all approved deps to the stubs now (`bun add` in each), so later parallel tasks never touch `bun.lock`. +- Modify: `docs/DECISIONS.md` (UUIDv7, JSON as text, single Bun.SQL dialect) +- Test: `packages/db/test/db.test.ts` + +**Interfaces (produces):** +- `type Dialect = "sqlite" | "postgres"` +- `createDb(url: string): Promise`, where `interface Db { k: Kysely; dialect: Dialect; close(): Promise }`. `url` is `sqlite://`, `sqlite://:memory:` or `postgres://…`. `Database` is imported from `./schema` (Task 2). Until Task 2 lands, use a placeholder `export interface Database {}` in `src/schema.ts`. +- `newId(): string`, `nowIso(): string` +- `toJson(v: unknown): string`, `fromJson(s: string | null | undefined, fallback: T): T` +- `migrateToLatest(db: Db): Promise` uses Kysely `Migrator` with migrations imported statically from `src/migrations/index.ts` (`export const migrations: Record`). +- `forEachDialect(name: string, fn: (getDb: () => Db) => void): void` registers a `describe` per dialect. + - Before each test it builds a fresh migrated DB: SQLite `:memory:`, or on Postgres a fresh `create database st_` derived from `TEST_POSTGRES_URL`. + - It closes the DB after each test, and drops the Postgres DB in `afterAll`. + +- [ ] **Step 1: Write failing tests** in `db.test.ts` using `forEachDialect("db basics", …)`: + - `"round-trips a row"`: create a temp table via `sql`, insert `{id: newId(), j: toJson({a: 1})}`, select, `expect(fromJson(row.j, {})).toEqual({a: 1})`. + - `"transactions roll back"`: `await expect(db.k.transaction().execute(async tx => { insert…; throw new Error("x") })).rejects.toThrow("x")`, then count is 0. + - `"newId is time-sortable"`: 1000 ids are already sorted. +- [ ] **Step 2:** Run `bun test packages/db` → FAIL (module not found). +- [ ] **Step 3: Implement.** + - `dialect.ts`: a Kysely `Dialect` whose `Driver` wraps `new SQL(url)`. `acquireConnection` returns a connection from `sql.reserve()`. `executeQuery` calls `conn.unsafe(compiled.sql, compiled.parameters)` and maps it to `{ rows, numAffectedRows }`. Transactions use `BEGIN`/`COMMIT`/`ROLLBACK` on the reserved connection. + - Adapter/compiler/introspector: `SqliteAdapter`/`SqliteQueryCompiler` for sqlite, and `PostgresAdapter`/`PostgresQueryCompiler` for postgres. + - For sqlite, run `PRAGMA foreign_keys = ON; PRAGMA journal_mode = WAL;` on create (skip WAL for `:memory:`). +- [ ] **Step 4:** `bun test packages/db` → PASS. Also run `TEST_POSTGRES_URL=… bun test packages/db` → PASS on both. +- [ ] **Step 5:** Commit `feat(db): Bun.SQL Kysely dialect for sqlite and postgres`. + +### Task 2: Schema migration and `Database` types + +**Files:** Create `packages/db/src/schema.ts` (Kysely table interfaces) and `packages/db/src/migrations/0001_initial.ts`; modify `packages/db/src/migrations/index.ts`. Test: `packages/db/test/schema.test.ts`. + +**Interfaces (produces):** the `Database` interface with these tables. Column names are snake_case. Every table has `id text pk` unless noted. `ts` means ISO text. JSON is text. +- `users`(username unique, display_name, password_hash, role `admin|member`, preferences json, created_at, updated_at) +- `sessions`(user_id fk, token_hash unique, expires_at, created_at) +- `invites`(code_hash unique, created_by fk, used_by nullable, expires_at, created_at) +- `app_tokens`(user_id, name, token_hash unique, scopes json `("read"|"write"|"ingest")[]`, last_used_at, revoked_at, created_at) +- `works`(media_type, title, original_title, year int null, release_date null, creators json, description, cover_url, cover_attachment_id null, metadata json, parent_id null fk works, position json null, owner_user_id null fk users, merged_into_id null, created_at, updated_at) +- `external_ids`(work_id fk, namespace, value). Unique (namespace, value). +- `work_aliases`(user_id, from_work_id, to_work_id). Unique (user_id, from_work_id). +- `work_history`(work_id, before json, after json, actor, created_at) +- `entries`(user_id, work_id, status, rating int null, review null, private_notes null, favorite int 0/1, tags json, custom_fields json, visibility `private|friends|public`, overrides json, media_type_override null, field_sources json, created_at, updated_at, deleted_at null). Unique (user_id, work_id). +- `engagements`(entry_id, started_at null, started_precision, finished_at null, finished_precision, progress json, platform null, rating null, notes null, custom_fields json, field_sources json, created_at, updated_at, deleted_at) +- `engagement_sources`(engagement_id, connection_id, record_key). Unique (connection_id, record_key). +- `listens`(user_id, listened_at, artist, track, album null, duration_ms null, mbids json, work_id null, source, source_key, created_at). Unique (user_id, source, source_key). Index (user_id, listened_at). +- `lists`(user_id, name, description, kind `list|grouping|thread`, visibility, rating null, review null, created_at, updated_at, deleted_at) +- `list_items`(list_id, work_id, position int, note null) +- `custom_field_definitions`(user_id, name, type, options json, applies_to json `MediaType[] | "all"`, level `entry|engagement|copy`, created_at) +- `copies`(user_id, work_id null, title_hint null, format null, acquired_from null, acquired_at null, acquired_precision, price_cents int null, currency null, notes null, custom_fields json, source_ref null, created_at, updated_at, deleted_at) +- `attachments`(user_id, sha256, mime, size int, filename, caption null, taken_at null, visibility null, created_at) +- `attachment_links`(attachment_id, entity `entry|engagement|copy|list|work`, entity_id) +- `user_history`(user_id, entity, entity_id, before json, after json, actor, created_at) +- `match_suggestions`(user_id, work_id, candidate_work_id, score real, status `pending|accepted|rejected`, created_at) +- `source_connections`(user_id, plugin_id, name, config json, credentials_enc null, schedule_minutes int null, cursor null, status `active|paused|error`, last_run_at null, created_at) +- `source_runs`(connection_id, mode `full|changes|file`, started_at, finished_at null, status `running|ok|failed`, counts json, log text, error null) +- `source_records`(connection_id, key, kind, current_hash, first_seen, last_seen, deleted_at null, merged_into_key null, target_entity null, target_id null). Unique (connection_id, key). +- `source_record_versions`(record_id, hash, raw_gz text (base64 of gzip), seen_at) +- `change_events`(user_id, connection_id null, run_id null, record_id null, type `added|changed|removed|merged|conflict`, target_entity, target_id null, payload json, status `pending|auto_applied|accepted|ignored|kept_mine`, created_at, resolved_at null) +- `jobs`(type, payload json, run_at, status `queued|running|done|failed`, attempts int, last_error null, locked_at null, lock_key null, created_at). Index (status, run_at). Unique partial behaviour for lock_key is enforced in code (Task 9). + +- [ ] **Step 1: Failing test** `forEachDialect("schema")`, `"all tables exist and unique constraints hold"`: + - insert a user, a work, and two `external_ids` rows with the same (namespace, value); + - expect the second insert to reject; + - insert duplicate `entries` (user, work) and expect a reject. +- [ ] **Step 2:** `bun test packages/db/test/schema.test.ts` → FAIL. +- [ ] **Step 3:** Write the migration using only the Kysely schema builder with portable types (`text`, `integer`, `real`), plus indexes for every fk column. +- [ ] **Step 4:** PASS on both dialects. +- [ ] **Step 5:** Commit `feat(db): initial schema`. + +### Task 3: `packages/plugin-sdk`: contract, http client, fixture harness + +**Files:** Create `packages/plugin-sdk/src/{types.ts,http.ts,testing.ts,partial-date.ts}`. Tests: `packages/plugin-sdk/test/{http.test.ts,partial-date.test.ts}`. + +**Interfaces (produces):** `types.ts`, exactly these names: +```ts +export type MediaType = /* enum from Global constraints */; +export type Status = "planned" | "in_progress" | "paused" | "completed" | "dropped" | "none"; +export type Precision = "day" | "month" | "year" | "unknown"; +export interface PartialDate { value: string; precision: Precision } +export type Capability = "pull" | "pullChanges" | "push" | "ownedLibrary" | "ingest" | "lookup" | "importFile" | "exportFile"; +export interface ConfigField { key: string; label: string; type: "string" | "secret" | "number" | "boolean" | "select"; required?: boolean; options?: string[]; help?: string } +export interface PluginManifest { id: string; name: string; version: string; sdkVersion: 1; mediaTypes: MediaType[]; capabilities: Capability[]; config: ConfigField[]; attribution?: string } +export interface WorkHint { title: string; mediaType: MediaType; year?: number; creators?: { name: string; role?: string }[]; externalIds: Record; coverUrl?: string; description?: string; parent?: WorkHint; position?: { season?: number; episode?: number; index?: number } } +export interface EntryFields { status?: Status; rating?: number | null; review?: string | null; notes?: string | null; favorite?: boolean; tags?: string[] } +export interface EngagementFields { startedAt?: PartialDate | null; finishedAt?: PartialDate | null; progress?: Record; platform?: string | null; rating?: number | null; notes?: string | null } +export interface ListenFields { listenedAt: string; artist: string; track: string; album?: string; durationMs?: number; mbids?: { recording?: string; release?: string; artist?: string } } +interface Base { key: string; raw: unknown; deleted?: boolean; mergedIntoKey?: string } +export type SourceRecord = + | (Base & { kind: "entry"; work: WorkHint; entry: EntryFields }) + | (Base & { kind: "engagement"; work: WorkHint; engagement: EngagementFields }) + | (Base & { kind: "listen"; listen: ListenFields }) + | (Base & { kind: "list"; list: { name: string; description?: string; items: { work: WorkHint; note?: string }[] } }); +export interface HttpClient { json(url: string, init?: RequestInit & { query?: Record }): Promise; raw(url: string, init?: RequestInit): Promise } +export interface PluginContext { + http: HttpClient; log: (msg: string) => void; config: Record; + credentials: { get(): Promise | null>; save(c: Record): Promise }; + cursor: { get(): Promise; set(c: string): Promise }; + now(): Date; signal: AbortSignal; +} +export interface DeviceAuth { start(ctx: PluginContext): Promise<{ userCode: string; verificationUrl: string; deviceCode: string; intervalSec: number; expiresInSec: number }>; poll(ctx: PluginContext, deviceCode: string): Promise<"pending" | Record> } +export interface SourcePlugin { + manifest: PluginManifest; + deviceAuth?: DeviceAuth; + pull?(ctx: PluginContext): AsyncIterable; + pullChanges?(ctx: PluginContext): AsyncIterable; + lookup?(ctx: PluginContext, q: { title: string; mediaType?: MediaType; year?: number }): Promise; + importFile?(ctx: PluginContext, file: { name: string; bytes: Uint8Array }): AsyncIterable; +} +export function definePlugin(p: SourcePlugin): SourcePlugin; +``` +- `http.ts`: `createHttpClient(opts: { ratePerMinute?: number; burst?: number; maxRetries?: number; userAgent: string; fetchImpl?: typeof fetch; sleep?: (ms: number) => Promise }): HttpClient`. + - Token-bucket rate limiting. + - Retries on 429/5xx with exponential backoff, honouring `Retry-After` (seconds or HTTP date) and `X-RateLimit-Reset-In`. + - Throws `HttpError { status, body }` after the retries run out. + - `json` sends `Accept: application/json` and appends `query`. +- `testing.ts`: `createFixtureHttp(routes: Record unknown)>): { fetch: typeof fetch; calls: Request[] }`. The key is `"METHOD url-without-query"` or `"METHOD full-url"`. An unknown route throws. `createTestContext(partial?: Partial & { fetch?: typeof fetch }): PluginContext & { saved: { credentials: Record[]; cursor: string[] } }` provides in-memory credentials and cursor. `collect(it: AsyncIterable): Promise`. +- `partial-date.ts`: `parsePartialDate(input: string | null | undefined): PartialDate | null` accepts `2019`, `2019-03`, `2019-03-04`, full ISO datetime, `03/04/2019`→null (ambiguous), and `""`→null. `comparePartialDates(a, b): number`. + +- [ ] **Step 1: Failing tests.** + - `http.test.ts`: + - `"retries 429 honouring Retry-After"`: fetch returns 429 `Retry-After: 2` then 200; `sleep` was called with 2000; the result parsed. + - `"gives up after maxRetries with HttpError"`. + - `"rate limits to ratePerMinute"`: with a fake clock, 3 calls at `ratePerMinute: 60, burst: 1` sleep ~1000ms between calls. + - `partial-date.test.ts`: a table of the inputs above to the expected `{value, precision}`. +- [ ] **Step 2:** FAIL. +- [ ] **Step 3:** Implement. +- [ ] **Step 4:** `bun test packages/plugin-sdk` → PASS. +- [ ] **Step 5:** Commit `feat(sdk): plugin contract, http client, fixture harness`. + +### Task 4: Auth: users, sessions, invites, app tokens + +**Files:** Create `packages/core/src/auth.ts`. Test: `packages/core/test/auth.test.ts`. + +**Interfaces (produces):** +- `registerUser(db: Db, input: { username: string; password: string; displayName?: string; inviteCode?: string }): Promise`. The first user becomes `admin` and needs no invite. Later users need a valid, unused, unexpired invite or it throws `AuthError("invite_required")`. Usernames match `/^[a-z0-9_]{2,32}$/`, lowercased, unique. +- `login(db, username, password): Promise<{ user: User; sessionToken: string }>`. Sessions last 30 days. The token is 32 random bytes in base64url; only its SHA-256 is stored. +- `getSessionUser(db, token): Promise`, `logout(db, token)`. +- `createInvite(db, adminId, ttlDays = 7): Promise` (admins only, else `AuthError("forbidden")`). +- `createAppToken(db, userId, name, scopes): Promise<{ id: string; token: string }>`. The token is formatted `stt_` and only shown once. `verifyAppToken(db, token, scope): Promise` updates `last_used_at`. `revokeAppToken(db, userId, id)`. +- `type User = { id: string; username: string; displayName: string; role: "admin" | "member" }`. Passwords use `Bun.password.hash` (argon2id). + +- [ ] **Step 1: Failing tests** (`forEachDialect`): + - first user is admin; + - second registration without invite → `invite_required`; + - with invite → member, and the invite cannot be reused; + - wrong password → `AuthError("invalid_credentials")`; + - session round-trip and logout; + - app token: `verifyAppToken` with a missing scope → null; revoked → null. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects. +- [ ] **Step 5:** Commit `feat(core): users, sessions, invites, app tokens`. + +### Task 5: Catalogue: works, external IDs, matching + +**Files:** Create `packages/core/src/works.ts` and `packages/core/src/matching.ts`. Tests: `packages/core/test/{works,matching}.test.ts`. + +**Interfaces (produces):** +- `createWork(db, input: { hint: WorkHint; ownerUserId?: string | null; parentId?: string | null }, actor: string): Promise` inserts the work and its external_ids. +- `getWork(db, id): Promise` follows `merged_into_id` chains. +- `updateWorkFromHint(db, workId, hint, actor)` fills or updates shared-work display fields, writes `work_history`, adds missing external IDs, and never touches private works. +- `findWorkByExternalIds(db, ids: Record): Promise`. +- `searchWorks(db, userId, q: string, opts?: { mediaType?: MediaType; limit?: number }): Promise` covers shared works plus the user's private ones, with a case-insensitive `LIKE` on title. +- `matchWork(db, userId, hint: WorkHint, actor: string): Promise<{ workId: string; created: boolean; draft: boolean }>` follows spec §8.4: + 1. By external ID: link, and update display fields. + 2. If `hint.parent` + `hint.position`: resolve the parent recursively, then find the child by (parent_id, position). If not found, create a shared child. + 3. Hint has external IDs: create a shared work. + 4. Otherwise: create a private draft (`owner_user_id = userId`) and store up to 5 `match_suggestions`, where `fuzzyScore ≥ 0.6`. + - Shows, seasons and episodes create the whole chain. +- `fuzzyScore(a: {title, year?, creators?}, b: same): number` returns 0..1. Normalise case, punctuation, leading "the/a/an" and diacritics. Score is title token-set Jaccard × 0.7, + year match 0.2 (±1 year counts as 0.1), + any creator match 0.1. +- `resolveAlias(db, userId, workId): Promise`. +- `acceptSuggestion(db, userId, suggestionId)` moves the user's entry from the draft work to the candidate (via `moveEntryToWork` from T6, imported lazily to avoid a cycle). `rejectSuggestion(db, userId, suggestionId)`. +- `adminMergeWorks(db, adminUserId, fromId, toId)` (admin only, else `AuthError("forbidden")`) sets `merged_into_id`, moves the external_ids, writes `work_history`, and repoints every user's entries unless the user already has an entry on `toId`. In that case the user's `from` entry is kept and they get a `pending` `merged` change event. +- `type Work` mirrors the row with JSON fields parsed. + +- [ ] **Step 1: Failing tests:** + - `"match by isbn13 links existing"`; + - `"episode hint creates show→season→episode chain once"` (calling it twice → same ids, 3 works total); + - `"hint without ids creates private draft with suggestions"`; + - `"private works invisible to other users in search"`; + - `fuzzyScore("The Expanse: Leviathan Wakes", "Leviathan Wakes")` > 0.6; + - `fuzzyScore("Dune","Dune Messiah")` < 0.6; + - `"Pokémon"` vs `"Pokemon"` = 1 when year and creators are equal or absent. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects. +- [ ] **Step 5:** Commit `feat(core): works catalogue and matching`. + +### Task 6: User layer: entries, engagements, history, lists, copies, custom fields + +**Files:** Create `packages/core/src/{history.ts,entries.ts,engagements.ts,lists.ts,copies.ts,custom-fields.ts}`. Tests: one test file per source file in `packages/core/test/`. + +**Interfaces (produces):** +- `recordHistory(tx, { userId, entity, entityId, before, after, actor })` stores only the changed keys. +- `getHistory(db, userId, entity, entityId)` and `restoreVersion(db, userId, historyId)` (re-applies `before` as actor `user`). +- `type Actor = "user" | \`plugin:${string}:${string}\` | \`token:${string}\``. Plugin actors are `plugin::`. +- `upsertEntry(db, userId, workId, patch: Partial, actor: Actor): Promise`. `EntryPatch` keys: `status rating review privateNotes favorite tags customFields visibility overrides mediaTypeOverride`. + - It sets `field_sources[key] = actor` for each patched key. + - It writes history. + - It validates rating 0–100 and custom fields against definitions. + - Defaults: `status "none"`, `visibility` from user prefs (`"private"`). +- `getEntry(db, userId, entryId)`. +- `listEntries(db, userId, filter: { mediaType?; status?; tag?; draft?: boolean; q?; parentId?: string | null; sort?: "updated" | "title" | "rating"; limit?; cursor? }): Promise<{ items: EntryView[]; nextCursor: string | null }>`. + - `EntryView` = entry + work (with overrides applied) + `draft` (the work has no external IDs) + `childSummary: { total: number; completed: number } | null`. + - The default `parentId: null` lists top-level only. +- `softDeleteEntry`, `moveEntryToWork(db, userId, entryId, targetWorkId, actor)` (writes a `work_aliases` row). +- `quickAdd(db, userId, { title, mediaType? = "other" }): Promise` creates a private draft work, then the entry. +- `addEngagement` / `updateEngagement` / `softDeleteEngagement` (the same field_sources and history rules), plus `listEngagements(db, userId, entryId)`. +- Lists: + - `createList`, `updateList`, `addListItem`, `moveListItem`, `removeListItem`, `getList`, `listLists`; + - `kind` is `list|grouping|thread`, and a grouping may carry `rating`/`review`. +- Copies: `createCopy`, `updateCopy`, `softDeleteCopy`, `listCopies(db, userId, { workId? })`. Price is stored in cents. +- Custom fields: `defineCustomField`, `listCustomFields(db, userId, mediaType?)`, `validateCustomValues(defs, values)`, which throws `ValidationError(field, reason)`. + +- [ ] **Step 1: Failing tests:** + - `"upsertEntry records field_sources and history"`; + - `"restoreVersion brings back previous review"`; + - `"rating 101 rejected"`; + - `"quickAdd creates draft entry"` (`draft === true`); + - `"listEntries rolls up children"`: a show with 3 episode entries, 2 completed → `childSummary {total:3, completed:2}`, and episodes are not listed at the top level; + - `"engagement with year precision keeps '2019'"`; + - `"moveEntryToWork leaves alias"`; + - `"custom select field rejects unknown option"`; + - `"grouping list can be rated"`. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects. +- [ ] **Step 5:** Commit `feat(core): entries, engagements, history, lists, copies, custom fields`. + +### Task 7: Attachments and secret encryption + +**Files:** Create `packages/core/src/attachments.ts` and `packages/core/src/crypto.ts`. Tests: `packages/core/test/{attachments,crypto}.test.ts`. + +**Interfaces (produces):** +- `encryptJson(secret: string, value: unknown): Promise` and `decryptJson(secret: string, blob: string): Promise`. Use AES-256-GCM via WebCrypto, with the key = HKDF-SHA256 of `secret` and info `"sttracker-credentials"`. The blob is `v1..`. A wrong secret throws `CryptoError`. +- `storeAttachment(db, dataDir, userId, file: { bytes: Uint8Array; mime: string; filename: string; caption?: string; takenAt?: string }, link: { entity; entityId }): Promise`. Content is addressed by sha256 at `${dataDir}/attachments/ab/cd/` and deduplicated. Allowed mimes are `image/*`, `application/pdf` and `text/plain`, with a max of 25 MB (else `ValidationError`). +- `readAttachment(db, dataDir, userId, id): Promise<{ file: BunFile; mime }>` checks the owner. +- `listAttachments(db, userId, entity, entityId)`, `unlinkAttachment(...)`. +- Thumbnails are out of Phase 1. The UI sizes images with CSS (recorded in TODO). + +- [ ] **Step 1: Failing tests:** encrypt/decrypt round-trip; wrong secret throws; the same bytes stored twice produce one file and two links; `.exe` mime rejected; another user's read → null. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(core): attachments and credential encryption`. + +### Task 8: Sync engine: snapshots, diff, apply, Changes inbox + +**Files:** Create `packages/core/src/sync/{snapshot.ts,apply.ts,run.ts,changes.ts,listens.ts}`. Tests: `packages/core/test/sync/*.test.ts`. + +**Interfaces:** +- **Consumes:** `SourcePlugin`, `SourceRecord`, `PluginContext` (T3); `matchWork`, `updateWorkFromHint` (T5); `upsertEntry`, `addEngagement`, `moveEntryToWork`, `softDeleteEntry`, `softDeleteEngagement` (T6). +- **Produces:** + - `runSync(deps: { db: Db; makeContext: (conn: Connection, runId: string) => PluginContext }, plugin: SourcePlugin, connectionId: string, mode: "full" | "changes" | { file: { name: string; bytes: Uint8Array } }): Promise`. + - `RunSummary = { runId; status: "ok" | "failed"; counts: { seen; added; changed; removed; merged; conflicts; autoApplied; pending; listens }; error?: string }`. + - `listChanges(db, userId, { status?: "pending" | "all"; connectionId?; limit?; cursor? })`. + - `resolveChange(db, userId, changeId, action: "accept" | "ignore" | "keep_mine"): Promise`. + - `ingestListens(db, userId, source: string, records: SourceRecord[]): Promise<{ inserted: number }>`. + +**Rules (spec §8.3; implement exactly):** +1. **Hash** = sha256 of a stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list, deleted, mergedIntoKey}`. The raw payload is excluded from the hash, but stored in a version gzipped with `Bun.gzipSync` and base64-encoded. A new `source_record_versions` row is written only when the hash differs from `current_hash`. +2. **New record (`added`):** match the work. For `entry`, `upsertEntry` with only the fields present, under the plugin actor. For `engagement`, ensure an entry exists, then de-duplicate: if any non-deleted engagement on that entry has `finished_at` within 24 h of the incoming one (day-precision values compare as dates) or an equal `started_at` datetime, attach an `engagement_sources` row instead of inserting. Write a `change_events` row with status `auto_applied`. +3. **Changed record:** for each field that differs from the previous normalized version: + - `field_sources[f]` is unset or equals this connection's actor → apply, `auto_applied`; + - it is `"user"` → `pending` `changed` event with `{field, from, to}`, and the current value is kept; + - it is another connection's actor and the value differs → `pending` `conflict` event with `{field, current, currentSource, incoming, incomingSource}`. + - A pending event for the same (record, field, incoming value) is never duplicated. That is the anti-flip-flop rule. +4. **Removed:** only when the mode is `"full"`, the plugin iteration completed without throwing, and the record was not seen in this run, or when `deleted: true` arrives in any mode. Set `source_records.deleted_at` and write a `pending` `removed` event. Accepting it soft-deletes the target entry or engagement. +5. **Merged:** when `mergedIntoKey` is set, write a `pending` `merged` event. Accepting it calls `moveEntryToWork` to the work matched for the target record's hint. +6. **Listens:** go to `ingestListens`. Insert-or-ignore on (user, source, source_key), and match a work only via `mbids.recording` (→ an existing work, or a shared track work created with a `musicbrainz_recording` id). There is no change event per listen; counts go in the run summary. +7. **`resolveChange`:** `accept` applies the incoming value with actor `plugin:…` and sets `field_sources` to that actor. `keep_mine` sets `field_sources[f] = "user"` for conflicts. `ignore` does nothing to data. All three set `status` and `resolved_at`. +8. **Run bookkeeping:** set `source_runs.status`, `counts`, `log` and `error`; `source_connections.last_run_at`; and set `status = "error"` on failure. + +- [ ] **Step 1: Failing tests** using a fake in-test plugin whose `pull` yields records from an array: + - `"first full pull creates entries and auto_applied events"`; + - `"second identical pull creates nothing new"` (Review focus 2: counts versions, events, engagements and listens before/after); + - `"failed pull marks nothing removed"` (Review focus 1: the plugin yields 1 of 2 records, then throws); + - `"record missing from completed full pull → pending removed; accept soft-deletes entry"`; + - `"user-edited rating not overwritten; pending changed event"` (Review focus 3); + - `"two connections disagree → one pending conflict, not duplicated on rerun; keep_mine stops further conflicts for that value"`; + - `"same play from two sources dedupes into one engagement with two engagement_sources"`; + - `"merged record → pending merged; accept moves entry"`; + - `"listens insert-or-ignore and mbid matching"`. + - All of these run on both dialects. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects. +- [ ] **Step 5:** Commit `feat(core): sync engine and changes inbox`. + +### Task 9: Jobs, scheduler, plugin registry, connections + +**Files:** Create `packages/core/src/{jobs.ts,registry.ts,connections.ts}`. Tests: `packages/core/test/{jobs,registry,connections}.test.ts`, plus fixture plugin `packages/core/test/fixtures/plugins/echo/index.ts`. + +**Interfaces (produces):** +- `enqueue(db, type: string, payload: unknown, opts?: { runAt?: string; lockKey?: string }): Promise` returns null when a `queued|running` job with the same `lockKey` exists. Sync jobs use `lockKey = "sync:"` (Review focus 5). +- `startWorker(db, handlers: Record Promise>, opts?: { pollMs?: number; concurrency?: number }): { stop(): Promise }`. + - A job is claimed atomically (`update … where id = ? and status = 'queued'` and check affected rows). + - Retry with backoff `2^attempts` minutes up to 5 attempts, then `failed`. +- `startScheduler(db, opts?: { tickMs?: number }): { stop() }` enqueues `sync` for each active connection whose `last_run_at + schedule_minutes` has passed. +- `loadPlugins(dirs: string[]): Promise`, where `PluginRegistry = { get(id): SourcePlugin | undefined; list(): PluginManifest[]; reload(): Promise }`. + - Each subdirectory with `index.ts` whose default export passes `definePlugin` shape checks is registered (`sdkVersion === 1`, unique id). + - `reload` re-imports with `?v=`. + - A bad plugin is logged and skipped, never fatal. +- Connections: + - `createConnection(db, secret, userId, { pluginId, name, config, credentials?, scheduleMinutes? })`, `updateConnection`, `deleteConnection` (soft: `status "paused"` + credentials cleared), `listConnections(db, userId)`; + - `makePluginContext(db, secret, conn, runId, http: HttpClient): PluginContext`. Its credentials `save` encrypts and persists *immediately* (Review focus 5), and its cursor is persisted on `set`. + - `syncHandler(deps)` is the `"sync"` job handler: it loads the connection and plugin, calls `runSync` with `mode` from the payload (default `pullChanges` if the plugin supports it, otherwise `full`), and runs a weekly `full` regardless. + +- [ ] **Step 1: Failing tests:** + - `"enqueue with same lockKey returns null"`; + - `"worker retries then fails"`; + - `"scheduler enqueues due connections only"`; + - `"registry loads echo plugin and skips broken one"`; + - `"credentials.save persists encrypted before pull continues"` (the echo plugin saves creds, then throws; the DB holds the new encrypted creds). +- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects. +- [ ] **Step 5:** Commit `feat(core): jobs, scheduler, plugin registry, connections`. + +### Task 10: Trakt plugin + +**Files:** `plugins/trakt/{package.json,index.ts,src/*.ts,test/trakt.test.ts,test/fixtures/*.json}`. + +**Interfaces:** `definePlugin` with: +- manifest `id "trakt"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `movie show season episode`; +- config: `clientId` (string, required), `clientSecret` (secret, required); +- `attribution` = the exact Trakt line from Global constraints; +- `deviceAuth` via `POST /oauth/device/code` and `/oauth/device/token`. + +Requests use base `https://api.trakt.tv` with headers `trakt-api-version: 2` and `trakt-api-key: clientId`, at `ratePerMinute: 100`. Before each run, refresh the access token if it expires within 24 h (`POST /oauth/token`, `grant_type=refresh_token`), then **`ctx.credentials.save` before any other request**. + +**pull** yields: +- `GET /sync/history?page=&limit=250` (paged by `X-Pagination-Page-Count`): each history item → an `engagement` record with key `history:`, and `finishedAt` = `watched_at` at day precision but keeping the full datetime in `value`. Movies: hint `movie` with ids `{trakt, tmdb_movie, imdb}`. Episodes: hint `episode` with `parent` season → `parent` show (`tmdb_show`, `tvdb`, `imdb`, `trakt`) and `position {season, episode}`. +- `GET /sync/ratings`: `entry` records with key `rating::`, `rating = rating × 10`. +- `GET /sync/watchlist`: `entry` with status `planned`, key `watchlist::`. +- `GET /users/me/lists` + `/users/me/lists/:id/items`: `list` records. + +**pullChanges:** `GET /sync/last_activities`, compared with the cursor (JSON of the last activity timestamps). For the categories that changed, re-fetch: +- history with `start_at = previous cursor time`; +- ratings and watchlist in full. +- Set the new cursor at the end. + +**lookup:** `GET /search/movie,show?query=`. + +- [ ] **Step 1: Failing tests with fixtures:** history paging over 2 pages; episode hint chain and position; rating 8 → 80; watchlist → planned; a token near expiry is refreshed and saved before the first `/sync` call (assert the order of `fixture.calls` and `ctx.saved.credentials`); `pullChanges` with an unchanged `last_activities` yields nothing. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS (`bun test plugins/trakt`). +- [ ] **Step 5:** Commit `feat(plugins): trakt`. + +### Task 11: Hardcover plugin + +**Files:** `plugins/hardcover/...` (same layout as Task 10). + +**Interfaces:** manifest `id "hardcover"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `book`, config `token` (secret, required: personal access token, sent as `Authorization: Bearer`), and attribution "Book data from Hardcover." Requests are `POST https://api.hardcover.app/v1/graphql` at `ratePerMinute: 50`. GraphQL errors throw `HttpError`. + +**pull** pages `me { user_books(limit: 100, offset) { id status_id rating review_raw private_notes starred updated_at merged_at original_book_id book { id title release_year contributions { author { name } } book_series { position series { id name } } } edition { id isbn_13 isbn_10 asin } user_book_reads { id started_at finished_at progress_pages } } }` and yields: +- `entry` with key `ub:`: + - status map: 1→planned, 2→in_progress, 3→completed, 4→paused, 5→dropped, 6→none; + - `rating` ×20 (Hardcover is 0–5 with halves); + - `review_raw` → review, `private_notes` → notes, starred → favorite; + - hint ids `{hardcover_book, isbn13, isbn10, asin, hardcover_edition}`; if there is a series, `parent` = series hint (`hardcover_series`) with `position {index}`. +- `engagement` for each read with key `read:`, day-precision dates. +- When `merged_at` is set and `original_book_id ≠ book.id`, also set `mergedIntoKey` on a synthetic record with key `ub:@book:`, so the engine sees the old identity merged into the new one. +- Lists via `me { lists { id name description list_books { book {…} } } }`. + +**pullChanges:** the same query filtered by `updated_at > cursor`. Deletions are not reported, so the weekly full pull detects them. + +- [ ] **Step 1: Failing tests:** status and rating mapping (4.5 → 90); series parent hint; re-reads → 2 engagements; merge produces `mergedIntoKey`; GraphQL `errors` → throws; offset paging stops on a short page. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(plugins): hardcover`. + +### Task 12: MangaBaka plugin + +**Files:** `plugins/mangabaka/...`. + +**Interfaces:** manifest `id "mangabaka"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `manga comic`, config `token` (secret, required, header `x-api-key`), and attribution "Manga data from MangaBaka (CC BY-NC-SA 4.0)." Base `https://api.mangabaka.org`, `ratePerMinute: 150`. + +**pull** pages `GET /v1/my/library?page=` and yields `entry` records keyed `series:`: +- state map: considering|plan_to_read→planned, reading|rereading→in_progress, paused→paused, dropped→dropped, completed→completed; +- rating is already 0–100; +- `note` → notes; +- progress → an `engagement` record keyed `progress:` with `progress {chapter, volume}` and start/finish dates; +- hint ids `{mangabaka, anilist, mal, mangaupdates, kitsu}` from `source.*`. +- If the series `state === "merged"`, set `mergedIntoKey = "series:"`. + +**pullChanges:** `GET /v2/my/library/changes?cursor=`. Upserts yield records, deletes yield `deleted: true`, and the new cursor is saved. + +- [ ] **Step 1: Failing tests:** state mapping table; a merged series sets `mergedIntoKey`; the changes feed yields a deletion; cursor saved. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(plugins): mangabaka`. + +### Task 13: CSV/JSON import plugin and CSV codec + +**Files:** `packages/plugin-sdk/src/csv.ts` (the codec lives in the SDK so core can reuse it), `plugins/csv/{index.ts,src/mapping.ts,test/*.test.ts,test/fixtures/*}`. + +**Interfaces (produces):** +- `parseCsv(text: string, opts?: { delimiter?: "," | ";" | "\t" }): string[][]` (RFC 4180). It strips a BOM and auto-detects the delimiter from the header line. +- `toCsv(rows: string[][]): string` quotes as needed and uses `\r\n`. This is reused by Task 15. +- The manifest is `id "csv"`, capabilities `["importFile"]`, and config `mapping` (string: JSON `{ column: field }`, where the fields are `title mediaType status rating ratingScale(5|10|100) review finishedAt startedAt tags isbn13 imdb tmdb_movie igdb notes`). +- `importFile`: + - **`.csv`:** maps rows into `entry` records (plus an `engagement` when dates are present), keyed `row:`. + - **`.json`:** accepts an array of objects with the same field names. + - **Without a mapping**, it guesses from header names, case-insensitively, including the Goodreads export columns (`Title`, `Author`, `My Rating`, `Date Read`, `ISBN13`, `Exclusive Shelf`, `My Review`). + +- [ ] **Step 1: Failing tests (Review focus 4):** a fixture `messy.csv` with BOM, `;`, quoted newline and `""` quotes, an empty rating, `Date Read` = `2019`, and an emoji title; `parseCsv(toCsv(x))` deep-equals `x`; Goodreads fixture headers are auto-mapped; `Exclusive Shelf` `to-read` → planned and `read` → completed; `My Rating` 4 at scale 5 → 80. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(plugins): csv and json import`. + +### Task 14: ListenBrainz and Last.fm plugins + +**Files:** `plugins/listenbrainz/...` and `plugins/lastfm/...`. + +**ListenBrainz:** manifest `id "listenbrainz"`, capabilities `["pull","pullChanges"]`, mediaTypes `track album artist`, config `username` (string) + `token` (secret, `Authorization: Token`). Base `https://api.listenbrainz.org`. Respect `X-RateLimit-Reset-In` (already handled in T3). +- **pull (full backfill):** + 1. `POST /1/export/` (if the response is 400 "already pending", fall through to listing). + 2. Poll `GET /1/export/list` every 30 s via `ctx.now`/an injected sleep until the newest export's status is `complete`. + 3. Download `GET /1/export/{id}/download`. + 4. Unzip with `fflate.unzipSync`. + 5. Stream `listens/**/*.jsonl` → `listen` records keyed `:`, with `mbids` from `track_metadata.mbid_mapping`. + 6. `feedback.jsonl` with score 1 → an `entry` with `favorite: true`, track hint ids `{musicbrainz_recording}`. +- **pullChanges:** `GET /1/user/{u}/listens?min_ts=&count=1000` loops until empty, and the cursor becomes the max `listened_at`. + +**Last.fm:** manifest `id "lastfm"`, capabilities `["pull","pullChanges"]`, config `username` + `apiKey` (secret). Base `https://ws.audioscrobbler.com/2.0/`, `format=json`, `ratePerMinute: 240`. +- **pull:** pages `user.getRecentTracks&limit=200&page=`, skipping the `@attr.nowplaying` track, yielding `listen` records keyed `::` with MBIDs when non-empty. Also `user.getLovedTracks` → favorite `entry` records. +- **pullChanges:** `from=`. +- Error code 29 → throw `HttpError` with status 429 so the client retries. + +- [ ] **Step 1: Failing tests:** + - LB: export flow over fixtures (a pending then complete list, plus a zip built in-test with `fflate.zipSync`); a feedback love → favorite entry; the incremental cursor advances. + - Last.fm: the nowplaying track is skipped; empty MBIDs are omitted; error 29 is retried (via the fixture fetch returning 29 once). +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(plugins): listenbrainz and lastfm`. + +### Task 15: Export bundle, restore, scheduled backups + +**Files:** Create `packages/core/src/export.ts` and `packages/core/src/backup.ts`. Tests: `packages/core/test/{export,backup}.test.ts`. + +**Interfaces (produces):** +- `exportUserBundle(db, dataDir, userId): Promise` returns a zip (fflate) containing: + - `manifest.json` `{ format: "sttracker-export/v1", exportedAt, user: { username, displayName } }`; + - `works.jsonl` (every referenced work, plus ancestors, with `externalIds`); + - `entries.jsonl`, `engagements.jsonl`, `listens.jsonl`, `lists.jsonl` (with items), `copies.jsonl`, `custom_fields.jsonl`, `history.jsonl`, `attachments.jsonl`; + - `attachments/` file bytes. + - It excludes credentials, sessions and tokens. + - In all files, camelCase keys mirror the `Entry`/`Work` view types. +- `exportUserCsv(db, userId): Promise>` returns `{ "entries.csv": …, "engagements.csv": …, "listens.csv": … }` using `toCsv` from `@sttracker/plugin-sdk/csv` (Task 13). +- `importUserBundle(db, dataDir, userId, zip: Uint8Array): Promise<{ works: number; entries: number }>` refuses unless the user has no entries (`ImportError("account_not_empty")`). It re-matches works by external IDs; unmatched works become private works; old IDs are remapped to new ones. +- `runBackup(db, dataDir, opts: { keep: number }): Promise` writes `${dataDir}/backups//` containing one bundle per user, plus on SQLite `VACUUM INTO 'sttracker.db'`. It prunes to the newest `keep` (default 14). +- `"backup"` job handler: `backupHandler(deps)`. + +- [ ] **Step 1: Failing tests:** + - `"export then import into fresh user reproduces entries, ratings, engagements with precision, lists and attachments"`; + - `"bundle has no credential fields"` (grep the zip text for `password`, `token`, `credentials`); + - `"import into non-empty account refused"`; + - `"backup prunes to keep"`. + - All on both dialects (VACUUM is SQLite only). +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Commit `feat(core): export bundle, restore, backups`. + +### Task 16: `apps/server`: API, auth middleware, ingest mount, OpenAPI, boot + +**Files:** Create `apps/server/src/{main.ts,config.ts,http.ts,routes/*.ts,openapi.ts}` and `apps/server/test/api.test.ts`. + +**Interfaces:** +- **Config** (`config.ts`) reads env: + - `DATABASE_URL` (default `sqlite://${DATA_DIR}/sttracker.db`); + - `DATA_DIR` (default `./data`); + - `STTRACKER_SECRET` (required; refuse to boot without it, with the message "Set STTRACKER_SECRET to a long random string. It encrypts source credentials."); + - `PUBLIC_URL`, `PORT` (default 3000); + - `PLUGIN_DIRS` (default `plugins,${DATA_DIR}/plugins`). +- `createApp(deps: { db; registry; secret; dataDir }): { fetch(req: Request): Promise; routes }` is testable without listening. `main.ts` migrates, loads plugins, starts the worker and scheduler, schedules a daily `backup` job, and calls `Bun.serve({ routes: { "/*": index.html }, fetch: app.fetch })`. +- `http.ts`: `route({ method, path, auth: "none" | "session" | "token:read" | "token:write" | "any:read" | "any:write", body?: ZodType, query?: ZodType, handler })`. + - JSON errors have the shape `{ error: { code, message } }`, with codes `400 validation`, `401 unauthenticated`, `403 forbidden`, `404 not_found`, `409 conflict`. + - Session cookie: `stt_session`, HttpOnly, SameSite=Lax, Secure when `PUBLIC_URL` is https. + - `any:*` accepts a session or a Bearer app token. +- **Routes under `/api/v1`** (bodies validated with zod; each calls the core function named in Tasks 4–15): + - auth: `POST auth/register`, `POST auth/login`, `POST auth/logout`, `GET me`, `GET/POST/DELETE me/tokens`, `POST admin/invites`; + - entries: `GET entries` (the `listEntries` filter as query), `POST entries/quick`, `GET/PATCH/DELETE entries/:id`, `GET entries/:id/history`, `POST history/:id/restore`, `POST entries/:id/move`; + - engagements: `GET/POST entries/:id/engagements`, `PATCH/DELETE engagements/:id`; + - works: `GET works/search`, `GET works/:id`, `GET works/:id/suggestions`, `POST suggestions/:id/accept|reject`; + - lists: `GET/POST lists`, `GET/PATCH/DELETE lists/:id`, `POST lists/:id/items`, `PATCH/DELETE list-items/:id`; + - copies: `GET/POST copies`, `PATCH/DELETE copies/:id`; + - custom fields: `GET/POST custom-fields`; + - attachments: `POST attachments` (multipart: `file`, `entity`, `entityId`, `caption`), `GET attachments/:id`, `DELETE attachments/:id/links/:linkId`; + - plugins and sources: `GET plugins` (manifests), `GET/POST sources`, `PATCH/DELETE sources/:id`, `POST sources/:id/run` (enqueue, 409 if locked), `POST sources/:id/import` (multipart file → enqueue a file-mode run), `POST sources/:id/device/start`, `POST sources/:id/device/poll`, `GET sources/:id/runs`; + - changes: `GET changes`, `POST changes/:id/resolve` `{action}`; + - export: `GET export/bundle` (zip download), `GET export/csv?file=entries.csv`, `POST import/bundle`; + - `GET openapi.json` (generated from the route table with `z.toJSONSchema`); + - `GET about` (attributions from plugin manifests). +- `/ingest//*` → 404 in Phase 1 (the mount point exists, and no plugin has `ingest`). +- `POST admin/plugins/reload` (admin) → `registry.reload()`. + +- [ ] **Step 1: Failing tests** (`forEachDialect`, calling `app.fetch` directly): + - register → login cookie → `GET me`; + - unauthenticated `GET entries` → 401 JSON; + - quick add, then list shows a draft; + - PATCH rating 150 → 400 `validation`; + - an app token with `read` cannot PATCH (403); + - another user's entry → 404; + - `POST sources` with the echo plugin, then `POST sources/:id/run` twice → the second returns 409; + - export bundle content-type is `application/zip`; + - `openapi.json` lists `/api/v1/entries`; + - booting without `STTRACKER_SECRET` throws the exact message. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] Update `AGENTS.md` Commands with Dev (`bun --hot apps/server/src/main.ts`), verified by booting and hitting `/api/v1/openapi.json`. Update `docs/PERMISSIONS.md` and the tool files accordingly. +- [ ] **Step 5:** Commit `feat(server): REST API, auth, sources, changes, export`. + +### Task 17: Web app: shell, auth, library, quick add, entry detail + +**Files:** Create `apps/web/{index.html,src/main.tsx,src/api.ts,src/theme.css,src/routes/*.tsx,src/components/*.tsx}` and `apps/web/test/*.test.tsx` (logic only: `api.ts` and formatters). + +Follow `DESIGN.md` tokens exactly and use impeccable `craft`. Routes (wouter): +- `/login`, `/register` (the first-run screen when no users exist: "Create the admin account"); +- `/` (library); +- `/entries/:id`. + +Library: +- The three-pane layout from the round 2 mockup: sidebar nav (Library, Log, Lists, Owned, Changes with a pending-count badge in `--warn`, a Sources list, Settings), the list pane, and the detail pane. +- Search, filter chips by media type and status, and a "Draft" filter. +- The Quick add box, with the placeholder text "Quick add: type a title, sort it out later". Enter creates the entry and selects it. +- Rows show cover (or the striped draft cover), title, `media type · meta`, and stars or the amber `Draft` badge. Children roll up as "season 3 of 5"-style meta from `childSummary`. + +Detail: +- Cover, title, editable rating (stars in the user's scale; the setting defaults to 5 with halves), status select, tags, and custom fields. +- The orange change alert when pending changes target this entry, with "See what changed" → `/changes?entry=:id`. +- Boxes for Reads/Plays (engagements with partial dates rendered by precision: "Mar 2025 (no exact date)"), Your copy, Attachments (upload, thumbnails via CSS), Linked sources, and Review (Gelasio, Markdown textarea). +- Children with their own ratings, a History drawer with restore, and "Link to catalogue" for drafts (suggestions plus search). + +Formatters: `formatRating(value0to100, scale)` and `formatPartialDate(pd)`. + +- [ ] **Step 1: Failing tests:** `formatRating(90, "5half")` = `"★★★★½"`, `formatRating(null, …)` = `""`, `formatPartialDate({value:"2025-03",precision:"month"})` = `"Mar 2025"`, `{value:"2019",precision:"year"}` = `"2019"`; `api.ts` throws an `ApiError` carrying `code` on a non-2xx JSON error. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Manual check: run the server, register, quick add, and open the detail pane. Take a screenshot and compare it with the mockup. Fix glaring deviations. +- [ ] **Step 6:** Commit `feat(web): app shell, library, entry detail`. + +### Task 18: Web app: Changes, Sources, Log, Lists, Owned, Settings, Export, About + +**Files:** `apps/web/src/routes/{changes,sources,log,lists,owned,settings,about}.tsx` plus components. This task depends on Task 17's `api.ts`, `theme.css` and layout components; create new components only. + +- **Changes:** + - Grouped by run. + - Each card shows `type`, the entry title, a field-level before/after (or both sources' values for conflicts), and buttons "Use the new value" (accept), "Keep mine" (keep_mine) and "Ignore". + - Removed events say " no longer has this. Your copy here is safe." with buttons "Delete here too" (accept) and "Keep it" (ignore). + - Merged events say " merged this into ." with "Follow the merge". +- **Sources:** + - Plugin cards from `GET plugins`, and a connect form rendered from the `config` fields (secrets as password inputs). + - The device-code flow modal shows the `userCode` and the verification link, and polls. + - Per connection: status, last run, "Sync now", schedule select (off/hourly/6h/daily), runs log, and file upload for `importFile` plugins. +- **Log:** a chronological feed of engagements and listens (paged). +- **Lists:** create lists of kind list, grouping or thread; add works via search; drag-to-reorder (up/down buttons are acceptable); a grouping can be rated. +- **Owned:** a copies table with add/edit (bought where, when, price, format, notes). +- **Settings:** profile, rating scale, default visibility, custom fields editor, app tokens (create shows the token once with "Copy"; revoke), invites (admin), "Reload plugins" (admin), and Export (download bundle and CSVs, import a bundle into an empty account). +- **About:** the attributions. + +- [ ] **Step 1: Failing tests** for pure helpers: `describeChange(event)` returns the exact strings above for the removed/merged/conflict fixtures; `scheduleLabel(minutes)`. +- [ ] **Steps 2–4:** FAIL, then implement, then PASS. +- [ ] **Step 5:** Manual check of the Changes flow against a fake connection (echo plugin) in the running app. +- [ ] **Step 6:** Commit `feat(web): changes inbox, sources, log, lists, owned, settings`. + +### Task 19: Docker, compose, CI, docs + +**Files:** Create `Dockerfile`, `docker-compose.yml`, `.dockerignore`, `.env.example` and `.gitea/workflows/ci.yml`. Modify `AGENTS.md` (Commands), `docs/ARCHITECTURE.md` (replace "planned" with reality), `docs/PERMISSIONS.md` and the tool files, `docs/TODO.md` and `README.md` (Quick start). + +- **Dockerfile:** `oven/bun:1.3` base; `bun install --frozen-lockfile --production`; copy the source; `VOLUME /data`; `ENV DATA_DIR=/data`; `EXPOSE 3000`; `CMD ["bun","apps/server/src/main.ts"]`; run as a non-root `bun` user. +- **Compose:** the `sttracker` service with `./data:/data`, env from `.env`; plus a `postgres:17-alpine` service under `profiles: ["postgres"]`, with a comment showing the `DATABASE_URL` for it. +- **CI:** on push/PR, a postgres:17-alpine service, then `bun install --frozen-lockfile`, `bun run typecheck`, `bun test` with `TEST_POSTGRES_URL` set, and `test $(wc -l < AGENTS.md) -le 150`. + +- [ ] **Step 1:** `docker build -t sttracker:dev .` → succeeds. +- [ ] **Step 2:** `docker run --rm -e STTRACKER_SECRET=test -p 3999:3000 sttracker:dev` → `curl -s localhost:3999/api/v1/openapi.json` returns JSON. +- [ ] **Step 3:** Repeat with the compose postgres profile and `DATABASE_URL` → boots and migrates. +- [ ] **Step 4:** Update the docs. Every command in `AGENTS.md` must be verified in this session (typecheck now passes; `bun test` passes on both DBs). +- [ ] **Step 5:** Commit `chore: docker, compose, CI, docs`. + +--- + +## Completion + +- [ ] `bun run typecheck` clean. `bun test` passes with and without `TEST_POSTGRES_URL`. +- [ ] Whole-branch review (superpowers:requesting-code-review). Fix the findings. +- [ ] `git rm docs/plans/001-phase1-own-your-data.md`. Set the spec front matter to `status: implemented`. Merge `feat/phase1` → `main` via superpowers:finishing-a-development-branch.