docs: phase 1 implementation plan
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
560c27c04d
commit
fb4ff36602
@@ -0,0 +1,642 @@
|
||||
---
|
||||
status: in-progress
|
||||
---
|
||||
|
||||
# Phase 1, "Own your data": implementation plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task by task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** A self-hosted Bun app where users connect Hardcover, Trakt, MangaBaka, ListenBrainz, Last.fm and CSV/JSON sources. STTracker pulls and snapshots their data, detects platform-side changes into a Changes inbox, and never overwrites what the user wrote. Users can log anything without IDs and export or back up everything. SQLite or Postgres, shipped in Docker.
|
||||
|
||||
**Architecture:** Bun workspaces.
|
||||
- `packages/db`: Kysely over a single custom dialect wrapping `Bun.SQL`, which speaks both `sqlite://` and `postgres://`.
|
||||
- `packages/plugin-sdk`: the plugin contract.
|
||||
- `packages/core`: domain services and the sync/change engine.
|
||||
- `plugins/*`: source integrations. They return `SourceRecord`s and never touch the DB.
|
||||
- `apps/server`: `Bun.serve` API, job worker, and serving the web app.
|
||||
- `apps/web`: React app via HTML imports.
|
||||
|
||||
**Tech stack:** Bun 1.3.14, TypeScript 5.9, Kysely 0.28+, zod 4, fflate, React 19, wouter, @tanstack/react-query.
|
||||
|
||||
**Spec:** `docs/specs/001-sttracker-v1-design.md` (read §6–§11 before any task). Design: `DESIGN.md`, `docs/design/mockups/round2-app-scrapbook-river.html`.
|
||||
|
||||
## Global constraints
|
||||
|
||||
- Follow `AGENTS.md` rules. In particular: Bun-native APIs only; plugins import only `@sttracker/plugin-sdk`; syncs never overwrite user-edited fields.
|
||||
- **Runtime deps are approved for this plan only:**
|
||||
- `kysely`, `zod`, `fflate` (root);
|
||||
- `react`, `react-dom`, `wouter`, `@tanstack/react-query` (apps/web);
|
||||
- `@types/react`, `@types/react-dom` (dev).
|
||||
- Ask before adding anything else.
|
||||
- **IDs:** `newId()` from `@sttracker/db/ids`. It returns `Bun.randomUUIDv7()`. (UUIDv7 replaces the spec's ULID: it is equally time-sortable and built in. This is recorded in DECISIONS by Task 1.)
|
||||
- **Timestamps:** ISO-8601 UTC strings (`new Date().toISOString()`) stored as `text` on both dialects.
|
||||
- **JSON columns:** `text` on both dialects. Use `toJson()`/`fromJson()` from `@sttracker/db/json`. (A simplification of spec §11's jsonb; recorded in DECISIONS by Task 1.)
|
||||
- **Ratings:** integer 0–100 or null.
|
||||
- **Partial dates:** `{ value: string; precision: "day" | "month" | "year" | "unknown" }`. `value` is `YYYY-MM-DD`, `YYYY-MM`, `YYYY` or `""`, or a full ISO datetime when the source gives a time. Partial dates are never converted through `Date`, which would shift them by timezone.
|
||||
- **Status enum:** `"planned" | "in_progress" | "paused" | "completed" | "dropped" | "none"`.
|
||||
- **MediaType enum:** `book comic manga game movie show season episode podcast podcast_episode audio_drama artist album track video other`.
|
||||
- **Package exports:** each package's `package.json` has `"exports": { "./*": "./src/*.ts" }`. Import by path (e.g. `@sttracker/core/entries`). There are no barrel `index.ts` files, which avoids merge conflicts between parallel tasks.
|
||||
- **DB tests** use `forEachDialect()` from `@sttracker/db/testing` and must pass on SQLite always, and on Postgres when `TEST_POSTGRES_URL` is set. Local value: `TEST_POSTGRES_URL=postgres://postgres:[email protected]:55432/postgres` (container `sttracker-pg-test`).
|
||||
- **Tests never hit the network.** Plugins are tested with fixtures in `plugins/<id>/test/fixtures/*.json`, written by hand from the documented API shapes, and played through `createFixtureHttp()`.
|
||||
- **Attribution strings** are shown on the About page (exact text):
|
||||
- "STTracker uses Trakt but is not endorsed or certified by Trakt."
|
||||
- "Manga data from MangaBaka (CC BY-NC-SA 4.0)."
|
||||
- "Music data from ListenBrainz and Last.fm."
|
||||
- "Book data from Hardcover."
|
||||
- **Commits:** Conventional Commits, ending with the line `Co-Authored-By: Claude Opus 5.5 <[email protected]>`.
|
||||
|
||||
## Review focus
|
||||
|
||||
1. **A full pull that fails part-way** (network error, 429s exhausted, thrown parser error) must mark *nothing* as removed. Only a pull that finished emits `removed` events. Test lives in Task 8.
|
||||
2. **Running the same sync twice with unchanged source data** creates zero new `source_record_versions`, `change_events`, `engagements` or `listens`. Test lives in Task 8.
|
||||
3. **A user edits a field, then the source changes it:** the user's value stays and a pending change appears. The same goes for two sources disagreeing: a pending `conflict`, with no flip-flopping on the next run. Test lives in Task 8 on both dialects.
|
||||
4. **Messy CSV input** must round-trip without data loss: UTF-8 BOM, `;` delimiter, quoted newlines and quotes, empty rating, "2019" as a date, emoji titles. Test lives in Task 13.
|
||||
5. **Rotating single-use refresh tokens (Trakt):** refreshed credentials are saved before any further request, and two syncs of the same connection never run at once. Tests live in Task 10 (save-before-continue) and Task 9 (connection lock).
|
||||
|
||||
---
|
||||
|
||||
## Dependency graph and waves
|
||||
|
||||
```
|
||||
Wave A (parallel): T1 db-core T3 plugin-sdk
|
||||
Wave B: T2 schema (needs T1)
|
||||
Wave C (parallel): T4 auth T5 catalogue+matching T6 user-layer T7 attachments+crypto
|
||||
T10 trakt T11 hardcover T12 mangabaka T13 csv/json T14 listenbrainz+lastfm (plugins need only T3)
|
||||
Wave D (parallel): T8 sync engine (T3,T5,T6) T9 jobs+registry (T2,T3,T7) T15 export/backup (T5,T6,T7)
|
||||
Wave E: T16 server API (all core)
|
||||
Wave F (parallel): T17 web shell+library+detail T18 web changes/sources/lists/owned/settings/export T19 docker+CI+docs
|
||||
```
|
||||
|
||||
Parallel tasks run in separate git worktrees branched from `feat/phase1` and are merged back into it after review.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: `packages/db`: dialect, ids, json, migrations runner, test harness
|
||||
|
||||
**Files:**
|
||||
- Create: `packages/db/package.json` (`@sttracker/db`, deps `kysely`), `packages/db/src/dialect.ts`, `packages/db/src/db.ts`, `packages/db/src/ids.ts`, `packages/db/src/json.ts`, `packages/db/src/migrate.ts`, `packages/db/src/testing.ts`
|
||||
- Create: workspace `package.json` stubs with `exports` for `packages/core` (`@sttracker/core`), `packages/plugin-sdk` (`@sttracker/plugin-sdk`), `apps/server` (`@sttracker/server`), `apps/web` (`@sttracker/web`). Add all approved deps to the stubs now (`bun add` in each), so later parallel tasks never touch `bun.lock`.
|
||||
- Modify: `docs/DECISIONS.md` (UUIDv7, JSON as text, single Bun.SQL dialect)
|
||||
- Test: `packages/db/test/db.test.ts`
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `type Dialect = "sqlite" | "postgres"`
|
||||
- `createDb(url: string): Promise<Db>`, where `interface Db { k: Kysely<Database>; dialect: Dialect; close(): Promise<void> }`. `url` is `sqlite://<path>`, `sqlite://:memory:` or `postgres://…`. `Database` is imported from `./schema` (Task 2). Until Task 2 lands, use a placeholder `export interface Database {}` in `src/schema.ts`.
|
||||
- `newId(): string`, `nowIso(): string`
|
||||
- `toJson(v: unknown): string`, `fromJson<T>(s: string | null | undefined, fallback: T): T`
|
||||
- `migrateToLatest(db: Db): Promise<void>` uses Kysely `Migrator` with migrations imported statically from `src/migrations/index.ts` (`export const migrations: Record<string, Migration>`).
|
||||
- `forEachDialect(name: string, fn: (getDb: () => Db) => void): void` registers a `describe` per dialect.
|
||||
- Before each test it builds a fresh migrated DB: SQLite `:memory:`, or on Postgres a fresh `create database st_<random>` derived from `TEST_POSTGRES_URL`.
|
||||
- It closes the DB after each test, and drops the Postgres DB in `afterAll`.
|
||||
|
||||
- [ ] **Step 1: Write failing tests** in `db.test.ts` using `forEachDialect("db basics", …)`:
|
||||
- `"round-trips a row"`: create a temp table via `sql`, insert `{id: newId(), j: toJson({a: 1})}`, select, `expect(fromJson(row.j, {})).toEqual({a: 1})`.
|
||||
- `"transactions roll back"`: `await expect(db.k.transaction().execute(async tx => { insert…; throw new Error("x") })).rejects.toThrow("x")`, then count is 0.
|
||||
- `"newId is time-sortable"`: 1000 ids are already sorted.
|
||||
- [ ] **Step 2:** Run `bun test packages/db` → FAIL (module not found).
|
||||
- [ ] **Step 3: Implement.**
|
||||
- `dialect.ts`: a Kysely `Dialect` whose `Driver` wraps `new SQL(url)`. `acquireConnection` returns a connection from `sql.reserve()`. `executeQuery` calls `conn.unsafe(compiled.sql, compiled.parameters)` and maps it to `{ rows, numAffectedRows }`. Transactions use `BEGIN`/`COMMIT`/`ROLLBACK` on the reserved connection.
|
||||
- Adapter/compiler/introspector: `SqliteAdapter`/`SqliteQueryCompiler` for sqlite, and `PostgresAdapter`/`PostgresQueryCompiler` for postgres.
|
||||
- For sqlite, run `PRAGMA foreign_keys = ON; PRAGMA journal_mode = WAL;` on create (skip WAL for `:memory:`).
|
||||
- [ ] **Step 4:** `bun test packages/db` → PASS. Also run `TEST_POSTGRES_URL=… bun test packages/db` → PASS on both.
|
||||
- [ ] **Step 5:** Commit `feat(db): Bun.SQL Kysely dialect for sqlite and postgres`.
|
||||
|
||||
### Task 2: Schema migration and `Database` types
|
||||
|
||||
**Files:** Create `packages/db/src/schema.ts` (Kysely table interfaces) and `packages/db/src/migrations/0001_initial.ts`; modify `packages/db/src/migrations/index.ts`. Test: `packages/db/test/schema.test.ts`.
|
||||
|
||||
**Interfaces (produces):** the `Database` interface with these tables. Column names are snake_case. Every table has `id text pk` unless noted. `ts` means ISO text. JSON is text.
|
||||
- `users`(username unique, display_name, password_hash, role `admin|member`, preferences json, created_at, updated_at)
|
||||
- `sessions`(user_id fk, token_hash unique, expires_at, created_at)
|
||||
- `invites`(code_hash unique, created_by fk, used_by nullable, expires_at, created_at)
|
||||
- `app_tokens`(user_id, name, token_hash unique, scopes json `("read"|"write"|"ingest")[]`, last_used_at, revoked_at, created_at)
|
||||
- `works`(media_type, title, original_title, year int null, release_date null, creators json, description, cover_url, cover_attachment_id null, metadata json, parent_id null fk works, position json null, owner_user_id null fk users, merged_into_id null, created_at, updated_at)
|
||||
- `external_ids`(work_id fk, namespace, value). Unique (namespace, value).
|
||||
- `work_aliases`(user_id, from_work_id, to_work_id). Unique (user_id, from_work_id).
|
||||
- `work_history`(work_id, before json, after json, actor, created_at)
|
||||
- `entries`(user_id, work_id, status, rating int null, review null, private_notes null, favorite int 0/1, tags json, custom_fields json, visibility `private|friends|public`, overrides json, media_type_override null, field_sources json, created_at, updated_at, deleted_at null). Unique (user_id, work_id).
|
||||
- `engagements`(entry_id, started_at null, started_precision, finished_at null, finished_precision, progress json, platform null, rating null, notes null, custom_fields json, field_sources json, created_at, updated_at, deleted_at)
|
||||
- `engagement_sources`(engagement_id, connection_id, record_key). Unique (connection_id, record_key).
|
||||
- `listens`(user_id, listened_at, artist, track, album null, duration_ms null, mbids json, work_id null, source, source_key, created_at). Unique (user_id, source, source_key). Index (user_id, listened_at).
|
||||
- `lists`(user_id, name, description, kind `list|grouping|thread`, visibility, rating null, review null, created_at, updated_at, deleted_at)
|
||||
- `list_items`(list_id, work_id, position int, note null)
|
||||
- `custom_field_definitions`(user_id, name, type, options json, applies_to json `MediaType[] | "all"`, level `entry|engagement|copy`, created_at)
|
||||
- `copies`(user_id, work_id null, title_hint null, format null, acquired_from null, acquired_at null, acquired_precision, price_cents int null, currency null, notes null, custom_fields json, source_ref null, created_at, updated_at, deleted_at)
|
||||
- `attachments`(user_id, sha256, mime, size int, filename, caption null, taken_at null, visibility null, created_at)
|
||||
- `attachment_links`(attachment_id, entity `entry|engagement|copy|list|work`, entity_id)
|
||||
- `user_history`(user_id, entity, entity_id, before json, after json, actor, created_at)
|
||||
- `match_suggestions`(user_id, work_id, candidate_work_id, score real, status `pending|accepted|rejected`, created_at)
|
||||
- `source_connections`(user_id, plugin_id, name, config json, credentials_enc null, schedule_minutes int null, cursor null, status `active|paused|error`, last_run_at null, created_at)
|
||||
- `source_runs`(connection_id, mode `full|changes|file`, started_at, finished_at null, status `running|ok|failed`, counts json, log text, error null)
|
||||
- `source_records`(connection_id, key, kind, current_hash, first_seen, last_seen, deleted_at null, merged_into_key null, target_entity null, target_id null). Unique (connection_id, key).
|
||||
- `source_record_versions`(record_id, hash, raw_gz text (base64 of gzip), seen_at)
|
||||
- `change_events`(user_id, connection_id null, run_id null, record_id null, type `added|changed|removed|merged|conflict`, target_entity, target_id null, payload json, status `pending|auto_applied|accepted|ignored|kept_mine`, created_at, resolved_at null)
|
||||
- `jobs`(type, payload json, run_at, status `queued|running|done|failed`, attempts int, last_error null, locked_at null, lock_key null, created_at). Index (status, run_at). Unique partial behaviour for lock_key is enforced in code (Task 9).
|
||||
|
||||
- [ ] **Step 1: Failing test** `forEachDialect("schema")`, `"all tables exist and unique constraints hold"`:
|
||||
- insert a user, a work, and two `external_ids` rows with the same (namespace, value);
|
||||
- expect the second insert to reject;
|
||||
- insert duplicate `entries` (user, work) and expect a reject.
|
||||
- [ ] **Step 2:** `bun test packages/db/test/schema.test.ts` → FAIL.
|
||||
- [ ] **Step 3:** Write the migration using only the Kysely schema builder with portable types (`text`, `integer`, `real`), plus indexes for every fk column.
|
||||
- [ ] **Step 4:** PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(db): initial schema`.
|
||||
|
||||
### Task 3: `packages/plugin-sdk`: contract, http client, fixture harness
|
||||
|
||||
**Files:** Create `packages/plugin-sdk/src/{types.ts,http.ts,testing.ts,partial-date.ts}`. Tests: `packages/plugin-sdk/test/{http.test.ts,partial-date.test.ts}`.
|
||||
|
||||
**Interfaces (produces):** `types.ts`, exactly these names:
|
||||
```ts
|
||||
export type MediaType = /* enum from Global constraints */;
|
||||
export type Status = "planned" | "in_progress" | "paused" | "completed" | "dropped" | "none";
|
||||
export type Precision = "day" | "month" | "year" | "unknown";
|
||||
export interface PartialDate { value: string; precision: Precision }
|
||||
export type Capability = "pull" | "pullChanges" | "push" | "ownedLibrary" | "ingest" | "lookup" | "importFile" | "exportFile";
|
||||
export interface ConfigField { key: string; label: string; type: "string" | "secret" | "number" | "boolean" | "select"; required?: boolean; options?: string[]; help?: string }
|
||||
export interface PluginManifest { id: string; name: string; version: string; sdkVersion: 1; mediaTypes: MediaType[]; capabilities: Capability[]; config: ConfigField[]; attribution?: string }
|
||||
export interface WorkHint { title: string; mediaType: MediaType; year?: number; creators?: { name: string; role?: string }[]; externalIds: Record<string, string>; coverUrl?: string; description?: string; parent?: WorkHint; position?: { season?: number; episode?: number; index?: number } }
|
||||
export interface EntryFields { status?: Status; rating?: number | null; review?: string | null; notes?: string | null; favorite?: boolean; tags?: string[] }
|
||||
export interface EngagementFields { startedAt?: PartialDate | null; finishedAt?: PartialDate | null; progress?: Record<string, number | string>; platform?: string | null; rating?: number | null; notes?: string | null }
|
||||
export interface ListenFields { listenedAt: string; artist: string; track: string; album?: string; durationMs?: number; mbids?: { recording?: string; release?: string; artist?: string } }
|
||||
interface Base { key: string; raw: unknown; deleted?: boolean; mergedIntoKey?: string }
|
||||
export type SourceRecord =
|
||||
| (Base & { kind: "entry"; work: WorkHint; entry: EntryFields })
|
||||
| (Base & { kind: "engagement"; work: WorkHint; engagement: EngagementFields })
|
||||
| (Base & { kind: "listen"; listen: ListenFields })
|
||||
| (Base & { kind: "list"; list: { name: string; description?: string; items: { work: WorkHint; note?: string }[] } });
|
||||
export interface HttpClient { json<T>(url: string, init?: RequestInit & { query?: Record<string, string | number | undefined> }): Promise<T>; raw(url: string, init?: RequestInit): Promise<Response> }
|
||||
export interface PluginContext {
|
||||
http: HttpClient; log: (msg: string) => void; config: Record<string, unknown>;
|
||||
credentials: { get(): Promise<Record<string, string> | null>; save(c: Record<string, string>): Promise<void> };
|
||||
cursor: { get(): Promise<string | null>; set(c: string): Promise<void> };
|
||||
now(): Date; signal: AbortSignal;
|
||||
}
|
||||
export interface DeviceAuth { start(ctx: PluginContext): Promise<{ userCode: string; verificationUrl: string; deviceCode: string; intervalSec: number; expiresInSec: number }>; poll(ctx: PluginContext, deviceCode: string): Promise<"pending" | Record<string, string>> }
|
||||
export interface SourcePlugin {
|
||||
manifest: PluginManifest;
|
||||
deviceAuth?: DeviceAuth;
|
||||
pull?(ctx: PluginContext): AsyncIterable<SourceRecord>;
|
||||
pullChanges?(ctx: PluginContext): AsyncIterable<SourceRecord>;
|
||||
lookup?(ctx: PluginContext, q: { title: string; mediaType?: MediaType; year?: number }): Promise<WorkHint[]>;
|
||||
importFile?(ctx: PluginContext, file: { name: string; bytes: Uint8Array }): AsyncIterable<SourceRecord>;
|
||||
}
|
||||
export function definePlugin(p: SourcePlugin): SourcePlugin;
|
||||
```
|
||||
- `http.ts`: `createHttpClient(opts: { ratePerMinute?: number; burst?: number; maxRetries?: number; userAgent: string; fetchImpl?: typeof fetch; sleep?: (ms: number) => Promise<void> }): HttpClient`.
|
||||
- Token-bucket rate limiting.
|
||||
- Retries on 429/5xx with exponential backoff, honouring `Retry-After` (seconds or HTTP date) and `X-RateLimit-Reset-In`.
|
||||
- Throws `HttpError { status, body }` after the retries run out.
|
||||
- `json` sends `Accept: application/json` and appends `query`.
|
||||
- `testing.ts`: `createFixtureHttp(routes: Record<string, unknown | ((req: Request) => unknown)>): { fetch: typeof fetch; calls: Request[] }`. The key is `"METHOD url-without-query"` or `"METHOD full-url"`. An unknown route throws. `createTestContext(partial?: Partial<PluginContext> & { fetch?: typeof fetch }): PluginContext & { saved: { credentials: Record<string,string>[]; cursor: string[] } }` provides in-memory credentials and cursor. `collect<T>(it: AsyncIterable<T>): Promise<T[]>`.
|
||||
- `partial-date.ts`: `parsePartialDate(input: string | null | undefined): PartialDate | null` accepts `2019`, `2019-03`, `2019-03-04`, full ISO datetime, `03/04/2019`→null (ambiguous), and `""`→null. `comparePartialDates(a, b): number`.
|
||||
|
||||
- [ ] **Step 1: Failing tests.**
|
||||
- `http.test.ts`:
|
||||
- `"retries 429 honouring Retry-After"`: fetch returns 429 `Retry-After: 2` then 200; `sleep` was called with 2000; the result parsed.
|
||||
- `"gives up after maxRetries with HttpError"`.
|
||||
- `"rate limits to ratePerMinute"`: with a fake clock, 3 calls at `ratePerMinute: 60, burst: 1` sleep ~1000ms between calls.
|
||||
- `partial-date.test.ts`: a table of the inputs above to the expected `{value, precision}`.
|
||||
- [ ] **Step 2:** FAIL.
|
||||
- [ ] **Step 3:** Implement.
|
||||
- [ ] **Step 4:** `bun test packages/plugin-sdk` → PASS.
|
||||
- [ ] **Step 5:** Commit `feat(sdk): plugin contract, http client, fixture harness`.
|
||||
|
||||
### Task 4: Auth: users, sessions, invites, app tokens
|
||||
|
||||
**Files:** Create `packages/core/src/auth.ts`. Test: `packages/core/test/auth.test.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `registerUser(db: Db, input: { username: string; password: string; displayName?: string; inviteCode?: string }): Promise<User>`. The first user becomes `admin` and needs no invite. Later users need a valid, unused, unexpired invite or it throws `AuthError("invite_required")`. Usernames match `/^[a-z0-9_]{2,32}$/`, lowercased, unique.
|
||||
- `login(db, username, password): Promise<{ user: User; sessionToken: string }>`. Sessions last 30 days. The token is 32 random bytes in base64url; only its SHA-256 is stored.
|
||||
- `getSessionUser(db, token): Promise<User | null>`, `logout(db, token)`.
|
||||
- `createInvite(db, adminId, ttlDays = 7): Promise<string>` (admins only, else `AuthError("forbidden")`).
|
||||
- `createAppToken(db, userId, name, scopes): Promise<{ id: string; token: string }>`. The token is formatted `stt_<base64url>` and only shown once. `verifyAppToken(db, token, scope): Promise<User | null>` updates `last_used_at`. `revokeAppToken(db, userId, id)`.
|
||||
- `type User = { id: string; username: string; displayName: string; role: "admin" | "member" }`. Passwords use `Bun.password.hash` (argon2id).
|
||||
|
||||
- [ ] **Step 1: Failing tests** (`forEachDialect`):
|
||||
- first user is admin;
|
||||
- second registration without invite → `invite_required`;
|
||||
- with invite → member, and the invite cannot be reused;
|
||||
- wrong password → `AuthError("invalid_credentials")`;
|
||||
- session round-trip and logout;
|
||||
- app token: `verifyAppToken` with a missing scope → null; revoked → null.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(core): users, sessions, invites, app tokens`.
|
||||
|
||||
### Task 5: Catalogue: works, external IDs, matching
|
||||
|
||||
**Files:** Create `packages/core/src/works.ts` and `packages/core/src/matching.ts`. Tests: `packages/core/test/{works,matching}.test.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `createWork(db, input: { hint: WorkHint; ownerUserId?: string | null; parentId?: string | null }, actor: string): Promise<string>` inserts the work and its external_ids.
|
||||
- `getWork(db, id): Promise<Work | null>` follows `merged_into_id` chains.
|
||||
- `updateWorkFromHint(db, workId, hint, actor)` fills or updates shared-work display fields, writes `work_history`, adds missing external IDs, and never touches private works.
|
||||
- `findWorkByExternalIds(db, ids: Record<string,string>): Promise<string | null>`.
|
||||
- `searchWorks(db, userId, q: string, opts?: { mediaType?: MediaType; limit?: number }): Promise<Work[]>` covers shared works plus the user's private ones, with a case-insensitive `LIKE` on title.
|
||||
- `matchWork(db, userId, hint: WorkHint, actor: string): Promise<{ workId: string; created: boolean; draft: boolean }>` follows spec §8.4:
|
||||
1. By external ID: link, and update display fields.
|
||||
2. If `hint.parent` + `hint.position`: resolve the parent recursively, then find the child by (parent_id, position). If not found, create a shared child.
|
||||
3. Hint has external IDs: create a shared work.
|
||||
4. Otherwise: create a private draft (`owner_user_id = userId`) and store up to 5 `match_suggestions`, where `fuzzyScore ≥ 0.6`.
|
||||
- Shows, seasons and episodes create the whole chain.
|
||||
- `fuzzyScore(a: {title, year?, creators?}, b: same): number` returns 0..1. Normalise case, punctuation, leading "the/a/an" and diacritics. Score is title token-set Jaccard × 0.7, + year match 0.2 (±1 year counts as 0.1), + any creator match 0.1.
|
||||
- `resolveAlias(db, userId, workId): Promise<string>`.
|
||||
- `acceptSuggestion(db, userId, suggestionId)` moves the user's entry from the draft work to the candidate (via `moveEntryToWork` from T6, imported lazily to avoid a cycle). `rejectSuggestion(db, userId, suggestionId)`.
|
||||
- `adminMergeWorks(db, adminUserId, fromId, toId)` (admin only, else `AuthError("forbidden")`) sets `merged_into_id`, moves the external_ids, writes `work_history`, and repoints every user's entries unless the user already has an entry on `toId`. In that case the user's `from` entry is kept and they get a `pending` `merged` change event.
|
||||
- `type Work` mirrors the row with JSON fields parsed.
|
||||
|
||||
- [ ] **Step 1: Failing tests:**
|
||||
- `"match by isbn13 links existing"`;
|
||||
- `"episode hint creates show→season→episode chain once"` (calling it twice → same ids, 3 works total);
|
||||
- `"hint without ids creates private draft with suggestions"`;
|
||||
- `"private works invisible to other users in search"`;
|
||||
- `fuzzyScore("The Expanse: Leviathan Wakes", "Leviathan Wakes")` > 0.6;
|
||||
- `fuzzyScore("Dune","Dune Messiah")` < 0.6;
|
||||
- `"Pokémon"` vs `"Pokemon"` = 1 when year and creators are equal or absent.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(core): works catalogue and matching`.
|
||||
|
||||
### Task 6: User layer: entries, engagements, history, lists, copies, custom fields
|
||||
|
||||
**Files:** Create `packages/core/src/{history.ts,entries.ts,engagements.ts,lists.ts,copies.ts,custom-fields.ts}`. Tests: one test file per source file in `packages/core/test/`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `recordHistory(tx, { userId, entity, entityId, before, after, actor })` stores only the changed keys.
|
||||
- `getHistory(db, userId, entity, entityId)` and `restoreVersion(db, userId, historyId)` (re-applies `before` as actor `user`).
|
||||
- `type Actor = "user" | \`plugin:${string}:${string}\` | \`token:${string}\``. Plugin actors are `plugin:<pluginId>:<connectionId>`.
|
||||
- `upsertEntry(db, userId, workId, patch: Partial<EntryPatch>, actor: Actor): Promise<Entry>`. `EntryPatch` keys: `status rating review privateNotes favorite tags customFields visibility overrides mediaTypeOverride`.
|
||||
- It sets `field_sources[key] = actor` for each patched key.
|
||||
- It writes history.
|
||||
- It validates rating 0–100 and custom fields against definitions.
|
||||
- Defaults: `status "none"`, `visibility` from user prefs (`"private"`).
|
||||
- `getEntry(db, userId, entryId)`.
|
||||
- `listEntries(db, userId, filter: { mediaType?; status?; tag?; draft?: boolean; q?; parentId?: string | null; sort?: "updated" | "title" | "rating"; limit?; cursor? }): Promise<{ items: EntryView[]; nextCursor: string | null }>`.
|
||||
- `EntryView` = entry + work (with overrides applied) + `draft` (the work has no external IDs) + `childSummary: { total: number; completed: number } | null`.
|
||||
- The default `parentId: null` lists top-level only.
|
||||
- `softDeleteEntry`, `moveEntryToWork(db, userId, entryId, targetWorkId, actor)` (writes a `work_aliases` row).
|
||||
- `quickAdd(db, userId, { title, mediaType? = "other" }): Promise<EntryView>` creates a private draft work, then the entry.
|
||||
- `addEngagement` / `updateEngagement` / `softDeleteEngagement` (the same field_sources and history rules), plus `listEngagements(db, userId, entryId)`.
|
||||
- Lists:
|
||||
- `createList`, `updateList`, `addListItem`, `moveListItem`, `removeListItem`, `getList`, `listLists`;
|
||||
- `kind` is `list|grouping|thread`, and a grouping may carry `rating`/`review`.
|
||||
- Copies: `createCopy`, `updateCopy`, `softDeleteCopy`, `listCopies(db, userId, { workId? })`. Price is stored in cents.
|
||||
- Custom fields: `defineCustomField`, `listCustomFields(db, userId, mediaType?)`, `validateCustomValues(defs, values)`, which throws `ValidationError(field, reason)`.
|
||||
|
||||
- [ ] **Step 1: Failing tests:**
|
||||
- `"upsertEntry records field_sources and history"`;
|
||||
- `"restoreVersion brings back previous review"`;
|
||||
- `"rating 101 rejected"`;
|
||||
- `"quickAdd creates draft entry"` (`draft === true`);
|
||||
- `"listEntries rolls up children"`: a show with 3 episode entries, 2 completed → `childSummary {total:3, completed:2}`, and episodes are not listed at the top level;
|
||||
- `"engagement with year precision keeps '2019'"`;
|
||||
- `"moveEntryToWork leaves alias"`;
|
||||
- `"custom select field rejects unknown option"`;
|
||||
- `"grouping list can be rated"`.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(core): entries, engagements, history, lists, copies, custom fields`.
|
||||
|
||||
### Task 7: Attachments and secret encryption
|
||||
|
||||
**Files:** Create `packages/core/src/attachments.ts` and `packages/core/src/crypto.ts`. Tests: `packages/core/test/{attachments,crypto}.test.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `encryptJson(secret: string, value: unknown): Promise<string>` and `decryptJson<T>(secret: string, blob: string): Promise<T>`. Use AES-256-GCM via WebCrypto, with the key = HKDF-SHA256 of `secret` and info `"sttracker-credentials"`. The blob is `v1.<b64 iv>.<b64 ciphertext>`. A wrong secret throws `CryptoError`.
|
||||
- `storeAttachment(db, dataDir, userId, file: { bytes: Uint8Array; mime: string; filename: string; caption?: string; takenAt?: string }, link: { entity; entityId }): Promise<Attachment>`. Content is addressed by sha256 at `${dataDir}/attachments/ab/cd/<sha256>` and deduplicated. Allowed mimes are `image/*`, `application/pdf` and `text/plain`, with a max of 25 MB (else `ValidationError`).
|
||||
- `readAttachment(db, dataDir, userId, id): Promise<{ file: BunFile; mime }>` checks the owner.
|
||||
- `listAttachments(db, userId, entity, entityId)`, `unlinkAttachment(...)`.
|
||||
- Thumbnails are out of Phase 1. The UI sizes images with CSS (recorded in TODO).
|
||||
|
||||
- [ ] **Step 1: Failing tests:** encrypt/decrypt round-trip; wrong secret throws; the same bytes stored twice produce one file and two links; `.exe` mime rejected; another user's read → null.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(core): attachments and credential encryption`.
|
||||
|
||||
### Task 8: Sync engine: snapshots, diff, apply, Changes inbox
|
||||
|
||||
**Files:** Create `packages/core/src/sync/{snapshot.ts,apply.ts,run.ts,changes.ts,listens.ts}`. Tests: `packages/core/test/sync/*.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
- **Consumes:** `SourcePlugin`, `SourceRecord`, `PluginContext` (T3); `matchWork`, `updateWorkFromHint` (T5); `upsertEntry`, `addEngagement`, `moveEntryToWork`, `softDeleteEntry`, `softDeleteEngagement` (T6).
|
||||
- **Produces:**
|
||||
- `runSync(deps: { db: Db; makeContext: (conn: Connection, runId: string) => PluginContext }, plugin: SourcePlugin, connectionId: string, mode: "full" | "changes" | { file: { name: string; bytes: Uint8Array } }): Promise<RunSummary>`.
|
||||
- `RunSummary = { runId; status: "ok" | "failed"; counts: { seen; added; changed; removed; merged; conflicts; autoApplied; pending; listens }; error?: string }`.
|
||||
- `listChanges(db, userId, { status?: "pending" | "all"; connectionId?; limit?; cursor? })`.
|
||||
- `resolveChange(db, userId, changeId, action: "accept" | "ignore" | "keep_mine"): Promise<void>`.
|
||||
- `ingestListens(db, userId, source: string, records: SourceRecord[]): Promise<{ inserted: number }>`.
|
||||
|
||||
**Rules (spec §8.3; implement exactly):**
|
||||
1. **Hash** = sha256 of a stable-key-ordered JSON of `{kind, work, entry|engagement|listen|list, deleted, mergedIntoKey}`. The raw payload is excluded from the hash, but stored in a version gzipped with `Bun.gzipSync` and base64-encoded. A new `source_record_versions` row is written only when the hash differs from `current_hash`.
|
||||
2. **New record (`added`):** match the work. For `entry`, `upsertEntry` with only the fields present, under the plugin actor. For `engagement`, ensure an entry exists, then de-duplicate: if any non-deleted engagement on that entry has `finished_at` within 24 h of the incoming one (day-precision values compare as dates) or an equal `started_at` datetime, attach an `engagement_sources` row instead of inserting. Write a `change_events` row with status `auto_applied`.
|
||||
3. **Changed record:** for each field that differs from the previous normalized version:
|
||||
- `field_sources[f]` is unset or equals this connection's actor → apply, `auto_applied`;
|
||||
- it is `"user"` → `pending` `changed` event with `{field, from, to}`, and the current value is kept;
|
||||
- it is another connection's actor and the value differs → `pending` `conflict` event with `{field, current, currentSource, incoming, incomingSource}`.
|
||||
- A pending event for the same (record, field, incoming value) is never duplicated. That is the anti-flip-flop rule.
|
||||
4. **Removed:** only when the mode is `"full"`, the plugin iteration completed without throwing, and the record was not seen in this run, or when `deleted: true` arrives in any mode. Set `source_records.deleted_at` and write a `pending` `removed` event. Accepting it soft-deletes the target entry or engagement.
|
||||
5. **Merged:** when `mergedIntoKey` is set, write a `pending` `merged` event. Accepting it calls `moveEntryToWork` to the work matched for the target record's hint.
|
||||
6. **Listens:** go to `ingestListens`. Insert-or-ignore on (user, source, source_key), and match a work only via `mbids.recording` (→ an existing work, or a shared track work created with a `musicbrainz_recording` id). There is no change event per listen; counts go in the run summary.
|
||||
7. **`resolveChange`:** `accept` applies the incoming value with actor `plugin:…` and sets `field_sources` to that actor. `keep_mine` sets `field_sources[f] = "user"` for conflicts. `ignore` does nothing to data. All three set `status` and `resolved_at`.
|
||||
8. **Run bookkeeping:** set `source_runs.status`, `counts`, `log` and `error`; `source_connections.last_run_at`; and set `status = "error"` on failure.
|
||||
|
||||
- [ ] **Step 1: Failing tests** using a fake in-test plugin whose `pull` yields records from an array:
|
||||
- `"first full pull creates entries and auto_applied events"`;
|
||||
- `"second identical pull creates nothing new"` (Review focus 2: counts versions, events, engagements and listens before/after);
|
||||
- `"failed pull marks nothing removed"` (Review focus 1: the plugin yields 1 of 2 records, then throws);
|
||||
- `"record missing from completed full pull → pending removed; accept soft-deletes entry"`;
|
||||
- `"user-edited rating not overwritten; pending changed event"` (Review focus 3);
|
||||
- `"two connections disagree → one pending conflict, not duplicated on rerun; keep_mine stops further conflicts for that value"`;
|
||||
- `"same play from two sources dedupes into one engagement with two engagement_sources"`;
|
||||
- `"merged record → pending merged; accept moves entry"`;
|
||||
- `"listens insert-or-ignore and mbid matching"`.
|
||||
- All of these run on both dialects.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(core): sync engine and changes inbox`.
|
||||
|
||||
### Task 9: Jobs, scheduler, plugin registry, connections
|
||||
|
||||
**Files:** Create `packages/core/src/{jobs.ts,registry.ts,connections.ts}`. Tests: `packages/core/test/{jobs,registry,connections}.test.ts`, plus fixture plugin `packages/core/test/fixtures/plugins/echo/index.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `enqueue(db, type: string, payload: unknown, opts?: { runAt?: string; lockKey?: string }): Promise<string | null>` returns null when a `queued|running` job with the same `lockKey` exists. Sync jobs use `lockKey = "sync:<connectionId>"` (Review focus 5).
|
||||
- `startWorker(db, handlers: Record<string, (payload: any) => Promise<void>>, opts?: { pollMs?: number; concurrency?: number }): { stop(): Promise<void> }`.
|
||||
- A job is claimed atomically (`update … where id = ? and status = 'queued'` and check affected rows).
|
||||
- Retry with backoff `2^attempts` minutes up to 5 attempts, then `failed`.
|
||||
- `startScheduler(db, opts?: { tickMs?: number }): { stop() }` enqueues `sync` for each active connection whose `last_run_at + schedule_minutes` has passed.
|
||||
- `loadPlugins(dirs: string[]): Promise<PluginRegistry>`, where `PluginRegistry = { get(id): SourcePlugin | undefined; list(): PluginManifest[]; reload(): Promise<void> }`.
|
||||
- Each subdirectory with `index.ts` whose default export passes `definePlugin` shape checks is registered (`sdkVersion === 1`, unique id).
|
||||
- `reload` re-imports with `?v=<Date.now()>`.
|
||||
- A bad plugin is logged and skipped, never fatal.
|
||||
- Connections:
|
||||
- `createConnection(db, secret, userId, { pluginId, name, config, credentials?, scheduleMinutes? })`, `updateConnection`, `deleteConnection` (soft: `status "paused"` + credentials cleared), `listConnections(db, userId)`;
|
||||
- `makePluginContext(db, secret, conn, runId, http: HttpClient): PluginContext`. Its credentials `save` encrypts and persists *immediately* (Review focus 5), and its cursor is persisted on `set`.
|
||||
- `syncHandler(deps)` is the `"sync"` job handler: it loads the connection and plugin, calls `runSync` with `mode` from the payload (default `pullChanges` if the plugin supports it, otherwise `full`), and runs a weekly `full` regardless.
|
||||
|
||||
- [ ] **Step 1: Failing tests:**
|
||||
- `"enqueue with same lockKey returns null"`;
|
||||
- `"worker retries then fails"`;
|
||||
- `"scheduler enqueues due connections only"`;
|
||||
- `"registry loads echo plugin and skips broken one"`;
|
||||
- `"credentials.save persists encrypted before pull continues"` (the echo plugin saves creds, then throws; the DB holds the new encrypted creds).
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS on both dialects.
|
||||
- [ ] **Step 5:** Commit `feat(core): jobs, scheduler, plugin registry, connections`.
|
||||
|
||||
### Task 10: Trakt plugin
|
||||
|
||||
**Files:** `plugins/trakt/{package.json,index.ts,src/*.ts,test/trakt.test.ts,test/fixtures/*.json}`.
|
||||
|
||||
**Interfaces:** `definePlugin` with:
|
||||
- manifest `id "trakt"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `movie show season episode`;
|
||||
- config: `clientId` (string, required), `clientSecret` (secret, required);
|
||||
- `attribution` = the exact Trakt line from Global constraints;
|
||||
- `deviceAuth` via `POST /oauth/device/code` and `/oauth/device/token`.
|
||||
|
||||
Requests use base `https://api.trakt.tv` with headers `trakt-api-version: 2` and `trakt-api-key: clientId`, at `ratePerMinute: 100`. Before each run, refresh the access token if it expires within 24 h (`POST /oauth/token`, `grant_type=refresh_token`), then **`ctx.credentials.save` before any other request**.
|
||||
|
||||
**pull** yields:
|
||||
- `GET /sync/history?page=&limit=250` (paged by `X-Pagination-Page-Count`): each history item → an `engagement` record with key `history:<id>`, and `finishedAt` = `watched_at` at day precision but keeping the full datetime in `value`. Movies: hint `movie` with ids `{trakt, tmdb_movie, imdb}`. Episodes: hint `episode` with `parent` season → `parent` show (`tmdb_show`, `tvdb`, `imdb`, `trakt`) and `position {season, episode}`.
|
||||
- `GET /sync/ratings`: `entry` records with key `rating:<type>:<traktId>`, `rating = rating × 10`.
|
||||
- `GET /sync/watchlist`: `entry` with status `planned`, key `watchlist:<type>:<traktId>`.
|
||||
- `GET /users/me/lists` + `/users/me/lists/:id/items`: `list` records.
|
||||
|
||||
**pullChanges:** `GET /sync/last_activities`, compared with the cursor (JSON of the last activity timestamps). For the categories that changed, re-fetch:
|
||||
- history with `start_at = previous cursor time`;
|
||||
- ratings and watchlist in full.
|
||||
- Set the new cursor at the end.
|
||||
|
||||
**lookup:** `GET /search/movie,show?query=`.
|
||||
|
||||
- [ ] **Step 1: Failing tests with fixtures:** history paging over 2 pages; episode hint chain and position; rating 8 → 80; watchlist → planned; a token near expiry is refreshed and saved before the first `/sync` call (assert the order of `fixture.calls` and `ctx.saved.credentials`); `pullChanges` with an unchanged `last_activities` yields nothing.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS (`bun test plugins/trakt`).
|
||||
- [ ] **Step 5:** Commit `feat(plugins): trakt`.
|
||||
|
||||
### Task 11: Hardcover plugin
|
||||
|
||||
**Files:** `plugins/hardcover/...` (same layout as Task 10).
|
||||
|
||||
**Interfaces:** manifest `id "hardcover"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `book`, config `token` (secret, required: personal access token, sent as `Authorization: Bearer`), and attribution "Book data from Hardcover." Requests are `POST https://api.hardcover.app/v1/graphql` at `ratePerMinute: 50`. GraphQL errors throw `HttpError`.
|
||||
|
||||
**pull** pages `me { user_books(limit: 100, offset) { id status_id rating review_raw private_notes starred updated_at merged_at original_book_id book { id title release_year contributions { author { name } } book_series { position series { id name } } } edition { id isbn_13 isbn_10 asin } user_book_reads { id started_at finished_at progress_pages } } }` and yields:
|
||||
- `entry` with key `ub:<id>`:
|
||||
- status map: 1→planned, 2→in_progress, 3→completed, 4→paused, 5→dropped, 6→none;
|
||||
- `rating` ×20 (Hardcover is 0–5 with halves);
|
||||
- `review_raw` → review, `private_notes` → notes, starred → favorite;
|
||||
- hint ids `{hardcover_book, isbn13, isbn10, asin, hardcover_edition}`; if there is a series, `parent` = series hint (`hardcover_series`) with `position {index}`.
|
||||
- `engagement` for each read with key `read:<id>`, day-precision dates.
|
||||
- When `merged_at` is set and `original_book_id ≠ book.id`, also set `mergedIntoKey` on a synthetic record with key `ub:<id>@book:<original_book_id>`, so the engine sees the old identity merged into the new one.
|
||||
- Lists via `me { lists { id name description list_books { book {…} } } }`.
|
||||
|
||||
**pullChanges:** the same query filtered by `updated_at > cursor`. Deletions are not reported, so the weekly full pull detects them.
|
||||
|
||||
- [ ] **Step 1: Failing tests:** status and rating mapping (4.5 → 90); series parent hint; re-reads → 2 engagements; merge produces `mergedIntoKey`; GraphQL `errors` → throws; offset paging stops on a short page.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(plugins): hardcover`.
|
||||
|
||||
### Task 12: MangaBaka plugin
|
||||
|
||||
**Files:** `plugins/mangabaka/...`.
|
||||
|
||||
**Interfaces:** manifest `id "mangabaka"`, capabilities `["pull","pullChanges","lookup"]`, mediaTypes `manga comic`, config `token` (secret, required, header `x-api-key`), and attribution "Manga data from MangaBaka (CC BY-NC-SA 4.0)." Base `https://api.mangabaka.org`, `ratePerMinute: 150`.
|
||||
|
||||
**pull** pages `GET /v1/my/library?page=` and yields `entry` records keyed `series:<series_id>`:
|
||||
- state map: considering|plan_to_read→planned, reading|rereading→in_progress, paused→paused, dropped→dropped, completed→completed;
|
||||
- rating is already 0–100;
|
||||
- `note` → notes;
|
||||
- progress → an `engagement` record keyed `progress:<series_id>` with `progress {chapter, volume}` and start/finish dates;
|
||||
- hint ids `{mangabaka, anilist, mal, mangaupdates, kitsu}` from `source.*`.
|
||||
- If the series `state === "merged"`, set `mergedIntoKey = "series:<merged_with>"`.
|
||||
|
||||
**pullChanges:** `GET /v2/my/library/changes?cursor=`. Upserts yield records, deletes yield `deleted: true`, and the new cursor is saved.
|
||||
|
||||
- [ ] **Step 1: Failing tests:** state mapping table; a merged series sets `mergedIntoKey`; the changes feed yields a deletion; cursor saved.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(plugins): mangabaka`.
|
||||
|
||||
### Task 13: CSV/JSON import plugin and CSV codec
|
||||
|
||||
**Files:** `packages/plugin-sdk/src/csv.ts` (the codec lives in the SDK so core can reuse it), `plugins/csv/{index.ts,src/mapping.ts,test/*.test.ts,test/fixtures/*}`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `parseCsv(text: string, opts?: { delimiter?: "," | ";" | "\t" }): string[][]` (RFC 4180). It strips a BOM and auto-detects the delimiter from the header line.
|
||||
- `toCsv(rows: string[][]): string` quotes as needed and uses `\r\n`. This is reused by Task 15.
|
||||
- The manifest is `id "csv"`, capabilities `["importFile"]`, and config `mapping` (string: JSON `{ column: field }`, where the fields are `title mediaType status rating ratingScale(5|10|100) review finishedAt startedAt tags isbn13 imdb tmdb_movie igdb notes`).
|
||||
- `importFile`:
|
||||
- **`.csv`:** maps rows into `entry` records (plus an `engagement` when dates are present), keyed `row:<sha1 of title+mediaType+finishedAt>`.
|
||||
- **`.json`:** accepts an array of objects with the same field names.
|
||||
- **Without a mapping**, it guesses from header names, case-insensitively, including the Goodreads export columns (`Title`, `Author`, `My Rating`, `Date Read`, `ISBN13`, `Exclusive Shelf`, `My Review`).
|
||||
|
||||
- [ ] **Step 1: Failing tests (Review focus 4):** a fixture `messy.csv` with BOM, `;`, quoted newline and `""` quotes, an empty rating, `Date Read` = `2019`, and an emoji title; `parseCsv(toCsv(x))` deep-equals `x`; Goodreads fixture headers are auto-mapped; `Exclusive Shelf` `to-read` → planned and `read` → completed; `My Rating` 4 at scale 5 → 80.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(plugins): csv and json import`.
|
||||
|
||||
### Task 14: ListenBrainz and Last.fm plugins
|
||||
|
||||
**Files:** `plugins/listenbrainz/...` and `plugins/lastfm/...`.
|
||||
|
||||
**ListenBrainz:** manifest `id "listenbrainz"`, capabilities `["pull","pullChanges"]`, mediaTypes `track album artist`, config `username` (string) + `token` (secret, `Authorization: Token`). Base `https://api.listenbrainz.org`. Respect `X-RateLimit-Reset-In` (already handled in T3).
|
||||
- **pull (full backfill):**
|
||||
1. `POST /1/export/` (if the response is 400 "already pending", fall through to listing).
|
||||
2. Poll `GET /1/export/list` every 30 s via `ctx.now`/an injected sleep until the newest export's status is `complete`.
|
||||
3. Download `GET /1/export/{id}/download`.
|
||||
4. Unzip with `fflate.unzipSync`.
|
||||
5. Stream `listens/**/*.jsonl` → `listen` records keyed `<listened_at>:<recording_msid>`, with `mbids` from `track_metadata.mbid_mapping`.
|
||||
6. `feedback.jsonl` with score 1 → an `entry` with `favorite: true`, track hint ids `{musicbrainz_recording}`.
|
||||
- **pullChanges:** `GET /1/user/{u}/listens?min_ts=<cursor>&count=1000` loops until empty, and the cursor becomes the max `listened_at`.
|
||||
|
||||
**Last.fm:** manifest `id "lastfm"`, capabilities `["pull","pullChanges"]`, config `username` + `apiKey` (secret). Base `https://ws.audioscrobbler.com/2.0/`, `format=json`, `ratePerMinute: 240`.
|
||||
- **pull:** pages `user.getRecentTracks&limit=200&page=`, skipping the `@attr.nowplaying` track, yielding `listen` records keyed `<uts>:<artist>:<track>` with MBIDs when non-empty. Also `user.getLovedTracks` → favorite `entry` records.
|
||||
- **pullChanges:** `from=<cursor>`.
|
||||
- Error code 29 → throw `HttpError` with status 429 so the client retries.
|
||||
|
||||
- [ ] **Step 1: Failing tests:**
|
||||
- LB: export flow over fixtures (a pending then complete list, plus a zip built in-test with `fflate.zipSync`); a feedback love → favorite entry; the incremental cursor advances.
|
||||
- Last.fm: the nowplaying track is skipped; empty MBIDs are omitted; error 29 is retried (via the fixture fetch returning 29 once).
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(plugins): listenbrainz and lastfm`.
|
||||
|
||||
### Task 15: Export bundle, restore, scheduled backups
|
||||
|
||||
**Files:** Create `packages/core/src/export.ts` and `packages/core/src/backup.ts`. Tests: `packages/core/test/{export,backup}.test.ts`.
|
||||
|
||||
**Interfaces (produces):**
|
||||
- `exportUserBundle(db, dataDir, userId): Promise<Uint8Array>` returns a zip (fflate) containing:
|
||||
- `manifest.json` `{ format: "sttracker-export/v1", exportedAt, user: { username, displayName } }`;
|
||||
- `works.jsonl` (every referenced work, plus ancestors, with `externalIds`);
|
||||
- `entries.jsonl`, `engagements.jsonl`, `listens.jsonl`, `lists.jsonl` (with items), `copies.jsonl`, `custom_fields.jsonl`, `history.jsonl`, `attachments.jsonl`;
|
||||
- `attachments/<sha256>` file bytes.
|
||||
- It excludes credentials, sessions and tokens.
|
||||
- In all files, camelCase keys mirror the `Entry`/`Work` view types.
|
||||
- `exportUserCsv(db, userId): Promise<Record<string, string>>` returns `{ "entries.csv": …, "engagements.csv": …, "listens.csv": … }` using `toCsv` from `@sttracker/plugin-sdk/csv` (Task 13).
|
||||
- `importUserBundle(db, dataDir, userId, zip: Uint8Array): Promise<{ works: number; entries: number }>` refuses unless the user has no entries (`ImportError("account_not_empty")`). It re-matches works by external IDs; unmatched works become private works; old IDs are remapped to new ones.
|
||||
- `runBackup(db, dataDir, opts: { keep: number }): Promise<string>` writes `${dataDir}/backups/<ISO>/` containing one bundle per user, plus on SQLite `VACUUM INTO 'sttracker.db'`. It prunes to the newest `keep` (default 14).
|
||||
- `"backup"` job handler: `backupHandler(deps)`.
|
||||
|
||||
- [ ] **Step 1: Failing tests:**
|
||||
- `"export then import into fresh user reproduces entries, ratings, engagements with precision, lists and attachments"`;
|
||||
- `"bundle has no credential fields"` (grep the zip text for `password`, `token`, `credentials`);
|
||||
- `"import into non-empty account refused"`;
|
||||
- `"backup prunes to keep"`.
|
||||
- All on both dialects (VACUUM is SQLite only).
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Commit `feat(core): export bundle, restore, backups`.
|
||||
|
||||
### Task 16: `apps/server`: API, auth middleware, ingest mount, OpenAPI, boot
|
||||
|
||||
**Files:** Create `apps/server/src/{main.ts,config.ts,http.ts,routes/*.ts,openapi.ts}` and `apps/server/test/api.test.ts`.
|
||||
|
||||
**Interfaces:**
|
||||
- **Config** (`config.ts`) reads env:
|
||||
- `DATABASE_URL` (default `sqlite://${DATA_DIR}/sttracker.db`);
|
||||
- `DATA_DIR` (default `./data`);
|
||||
- `STTRACKER_SECRET` (required; refuse to boot without it, with the message "Set STTRACKER_SECRET to a long random string. It encrypts source credentials.");
|
||||
- `PUBLIC_URL`, `PORT` (default 3000);
|
||||
- `PLUGIN_DIRS` (default `plugins,${DATA_DIR}/plugins`).
|
||||
- `createApp(deps: { db; registry; secret; dataDir }): { fetch(req: Request): Promise<Response>; routes }` is testable without listening. `main.ts` migrates, loads plugins, starts the worker and scheduler, schedules a daily `backup` job, and calls `Bun.serve({ routes: { "/*": index.html }, fetch: app.fetch })`.
|
||||
- `http.ts`: `route({ method, path, auth: "none" | "session" | "token:read" | "token:write" | "any:read" | "any:write", body?: ZodType, query?: ZodType, handler })`.
|
||||
- JSON errors have the shape `{ error: { code, message } }`, with codes `400 validation`, `401 unauthenticated`, `403 forbidden`, `404 not_found`, `409 conflict`.
|
||||
- Session cookie: `stt_session`, HttpOnly, SameSite=Lax, Secure when `PUBLIC_URL` is https.
|
||||
- `any:*` accepts a session or a Bearer app token.
|
||||
- **Routes under `/api/v1`** (bodies validated with zod; each calls the core function named in Tasks 4–15):
|
||||
- auth: `POST auth/register`, `POST auth/login`, `POST auth/logout`, `GET me`, `GET/POST/DELETE me/tokens`, `POST admin/invites`;
|
||||
- entries: `GET entries` (the `listEntries` filter as query), `POST entries/quick`, `GET/PATCH/DELETE entries/:id`, `GET entries/:id/history`, `POST history/:id/restore`, `POST entries/:id/move`;
|
||||
- engagements: `GET/POST entries/:id/engagements`, `PATCH/DELETE engagements/:id`;
|
||||
- works: `GET works/search`, `GET works/:id`, `GET works/:id/suggestions`, `POST suggestions/:id/accept|reject`;
|
||||
- lists: `GET/POST lists`, `GET/PATCH/DELETE lists/:id`, `POST lists/:id/items`, `PATCH/DELETE list-items/:id`;
|
||||
- copies: `GET/POST copies`, `PATCH/DELETE copies/:id`;
|
||||
- custom fields: `GET/POST custom-fields`;
|
||||
- attachments: `POST attachments` (multipart: `file`, `entity`, `entityId`, `caption`), `GET attachments/:id`, `DELETE attachments/:id/links/:linkId`;
|
||||
- plugins and sources: `GET plugins` (manifests), `GET/POST sources`, `PATCH/DELETE sources/:id`, `POST sources/:id/run` (enqueue, 409 if locked), `POST sources/:id/import` (multipart file → enqueue a file-mode run), `POST sources/:id/device/start`, `POST sources/:id/device/poll`, `GET sources/:id/runs`;
|
||||
- changes: `GET changes`, `POST changes/:id/resolve` `{action}`;
|
||||
- export: `GET export/bundle` (zip download), `GET export/csv?file=entries.csv`, `POST import/bundle`;
|
||||
- `GET openapi.json` (generated from the route table with `z.toJSONSchema`);
|
||||
- `GET about` (attributions from plugin manifests).
|
||||
- `/ingest/<pluginId>/*` → 404 in Phase 1 (the mount point exists, and no plugin has `ingest`).
|
||||
- `POST admin/plugins/reload` (admin) → `registry.reload()`.
|
||||
|
||||
- [ ] **Step 1: Failing tests** (`forEachDialect`, calling `app.fetch` directly):
|
||||
- register → login cookie → `GET me`;
|
||||
- unauthenticated `GET entries` → 401 JSON;
|
||||
- quick add, then list shows a draft;
|
||||
- PATCH rating 150 → 400 `validation`;
|
||||
- an app token with `read` cannot PATCH (403);
|
||||
- another user's entry → 404;
|
||||
- `POST sources` with the echo plugin, then `POST sources/:id/run` twice → the second returns 409;
|
||||
- export bundle content-type is `application/zip`;
|
||||
- `openapi.json` lists `/api/v1/entries`;
|
||||
- booting without `STTRACKER_SECRET` throws the exact message.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] Update `AGENTS.md` Commands with Dev (`bun --hot apps/server/src/main.ts`), verified by booting and hitting `/api/v1/openapi.json`. Update `docs/PERMISSIONS.md` and the tool files accordingly.
|
||||
- [ ] **Step 5:** Commit `feat(server): REST API, auth, sources, changes, export`.
|
||||
|
||||
### Task 17: Web app: shell, auth, library, quick add, entry detail
|
||||
|
||||
**Files:** Create `apps/web/{index.html,src/main.tsx,src/api.ts,src/theme.css,src/routes/*.tsx,src/components/*.tsx}` and `apps/web/test/*.test.tsx` (logic only: `api.ts` and formatters).
|
||||
|
||||
Follow `DESIGN.md` tokens exactly and use impeccable `craft`. Routes (wouter):
|
||||
- `/login`, `/register` (the first-run screen when no users exist: "Create the admin account");
|
||||
- `/` (library);
|
||||
- `/entries/:id`.
|
||||
|
||||
Library:
|
||||
- The three-pane layout from the round 2 mockup: sidebar nav (Library, Log, Lists, Owned, Changes with a pending-count badge in `--warn`, a Sources list, Settings), the list pane, and the detail pane.
|
||||
- Search, filter chips by media type and status, and a "Draft" filter.
|
||||
- The Quick add box, with the placeholder text "Quick add: type a title, sort it out later". Enter creates the entry and selects it.
|
||||
- Rows show cover (or the striped draft cover), title, `media type · meta`, and stars or the amber `Draft` badge. Children roll up as "season 3 of 5"-style meta from `childSummary`.
|
||||
|
||||
Detail:
|
||||
- Cover, title, editable rating (stars in the user's scale; the setting defaults to 5 with halves), status select, tags, and custom fields.
|
||||
- The orange change alert when pending changes target this entry, with "See what changed" → `/changes?entry=:id`.
|
||||
- Boxes for Reads/Plays (engagements with partial dates rendered by precision: "Mar 2025 (no exact date)"), Your copy, Attachments (upload, thumbnails via CSS), Linked sources, and Review (Gelasio, Markdown textarea).
|
||||
- Children with their own ratings, a History drawer with restore, and "Link to catalogue" for drafts (suggestions plus search).
|
||||
|
||||
Formatters: `formatRating(value0to100, scale)` and `formatPartialDate(pd)`.
|
||||
|
||||
- [ ] **Step 1: Failing tests:** `formatRating(90, "5half")` = `"★★★★½"`, `formatRating(null, …)` = `""`, `formatPartialDate({value:"2025-03",precision:"month"})` = `"Mar 2025"`, `{value:"2019",precision:"year"}` = `"2019"`; `api.ts` throws an `ApiError` carrying `code` on a non-2xx JSON error.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Manual check: run the server, register, quick add, and open the detail pane. Take a screenshot and compare it with the mockup. Fix glaring deviations.
|
||||
- [ ] **Step 6:** Commit `feat(web): app shell, library, entry detail`.
|
||||
|
||||
### Task 18: Web app: Changes, Sources, Log, Lists, Owned, Settings, Export, About
|
||||
|
||||
**Files:** `apps/web/src/routes/{changes,sources,log,lists,owned,settings,about}.tsx` plus components. This task depends on Task 17's `api.ts`, `theme.css` and layout components; create new components only.
|
||||
|
||||
- **Changes:**
|
||||
- Grouped by run.
|
||||
- Each card shows `type`, the entry title, a field-level before/after (or both sources' values for conflicts), and buttons "Use the new value" (accept), "Keep mine" (keep_mine) and "Ignore".
|
||||
- Removed events say "<Source> no longer has this. Your copy here is safe." with buttons "Delete here too" (accept) and "Keep it" (ignore).
|
||||
- Merged events say "<Source> merged this into <target title>." with "Follow the merge".
|
||||
- **Sources:**
|
||||
- Plugin cards from `GET plugins`, and a connect form rendered from the `config` fields (secrets as password inputs).
|
||||
- The device-code flow modal shows the `userCode` and the verification link, and polls.
|
||||
- Per connection: status, last run, "Sync now", schedule select (off/hourly/6h/daily), runs log, and file upload for `importFile` plugins.
|
||||
- **Log:** a chronological feed of engagements and listens (paged).
|
||||
- **Lists:** create lists of kind list, grouping or thread; add works via search; drag-to-reorder (up/down buttons are acceptable); a grouping can be rated.
|
||||
- **Owned:** a copies table with add/edit (bought where, when, price, format, notes).
|
||||
- **Settings:** profile, rating scale, default visibility, custom fields editor, app tokens (create shows the token once with "Copy"; revoke), invites (admin), "Reload plugins" (admin), and Export (download bundle and CSVs, import a bundle into an empty account).
|
||||
- **About:** the attributions.
|
||||
|
||||
- [ ] **Step 1: Failing tests** for pure helpers: `describeChange(event)` returns the exact strings above for the removed/merged/conflict fixtures; `scheduleLabel(minutes)`.
|
||||
- [ ] **Steps 2–4:** FAIL, then implement, then PASS.
|
||||
- [ ] **Step 5:** Manual check of the Changes flow against a fake connection (echo plugin) in the running app.
|
||||
- [ ] **Step 6:** Commit `feat(web): changes inbox, sources, log, lists, owned, settings`.
|
||||
|
||||
### Task 19: Docker, compose, CI, docs
|
||||
|
||||
**Files:** Create `Dockerfile`, `docker-compose.yml`, `.dockerignore`, `.env.example` and `.gitea/workflows/ci.yml`. Modify `AGENTS.md` (Commands), `docs/ARCHITECTURE.md` (replace "planned" with reality), `docs/PERMISSIONS.md` and the tool files, `docs/TODO.md` and `README.md` (Quick start).
|
||||
|
||||
- **Dockerfile:** `oven/bun:1.3` base; `bun install --frozen-lockfile --production`; copy the source; `VOLUME /data`; `ENV DATA_DIR=/data`; `EXPOSE 3000`; `CMD ["bun","apps/server/src/main.ts"]`; run as a non-root `bun` user.
|
||||
- **Compose:** the `sttracker` service with `./data:/data`, env from `.env`; plus a `postgres:17-alpine` service under `profiles: ["postgres"]`, with a comment showing the `DATABASE_URL` for it.
|
||||
- **CI:** on push/PR, a postgres:17-alpine service, then `bun install --frozen-lockfile`, `bun run typecheck`, `bun test` with `TEST_POSTGRES_URL` set, and `test $(wc -l < AGENTS.md) -le 150`.
|
||||
|
||||
- [ ] **Step 1:** `docker build -t sttracker:dev .` → succeeds.
|
||||
- [ ] **Step 2:** `docker run --rm -e STTRACKER_SECRET=test -p 3999:3000 sttracker:dev` → `curl -s localhost:3999/api/v1/openapi.json` returns JSON.
|
||||
- [ ] **Step 3:** Repeat with the compose postgres profile and `DATABASE_URL` → boots and migrates.
|
||||
- [ ] **Step 4:** Update the docs. Every command in `AGENTS.md` must be verified in this session (typecheck now passes; `bun test` passes on both DBs).
|
||||
- [ ] **Step 5:** Commit `chore: docker, compose, CI, docs`.
|
||||
|
||||
---
|
||||
|
||||
## Completion
|
||||
|
||||
- [ ] `bun run typecheck` clean. `bun test` passes with and without `TEST_POSTGRES_URL`.
|
||||
- [ ] Whole-branch review (superpowers:requesting-code-review). Fix the findings.
|
||||
- [ ] `git rm docs/plans/001-phase1-own-your-data.md`. Set the spec front matter to `status: implemented`. Merge `feat/phase1` → `main` via superpowers:finishing-a-development-branch.
|
||||
Reference in New Issue
Block a user