--- status: approved --- # STTracker: design Date: 2026-10-30 Status: approved 2026-10-30 ## 1. Intent People track what they read, watch, play and listen to across many platforms (Hardcover, Trakt, Backloggd, MangaBaka, Last.fm/ListenBrainz…), plus spreadsheets and notes apps. Each platform: - can silently delete, merge or alter entries; - may have weak or missing export/backup; - has no bridge to the others, so moving between services is manual; - demands a "real" catalogue link (TMDB id, ISBN…) before you can log anything; - only covers one kind of media, with its own arbitrary limits. STTracker is a self-hosted **meta-tracker** that sits above all of them. It is the user's own, permanent copy of everything they've consumed and what they thought of it. Platforms become *sources* it pulls from (and sometimes pushes to); none of them is trusted as the record. **The one thing it must do well: put the user in control.** The user should never have to conform to the data. Anything a spreadsheet or an Obsidian note could hold, STTracker can hold, and no platform sync may overwrite or drop something the user wrote. ### Audience 1. The owner (single user, self-hosted). This is who v1 is for. 2. A few friends on the same instance. Multi-user accounts exist from day one. 3. Later, possibly a small public instance. Not designed for now, but nothing in v1 should block it. ## 2. Goals - **One model for all media.** Books, comics/manga, games, films, shows (seasons, episodes), podcasts/audio dramas, music (artists, albums, tracks) and "other" share one entry model and one set of features: status, rating, review, dates, re-reads/re-plays, lists, tags, custom fields, attachments. A media type is a label, not a separate product. - **Hierarchy where every level is first-class.** A series contains books; a show has seasons and episodes; an album has tracks; a game has DLC. Every level can carry its own rating, review, tags and attachments. A parent's rating is the user's own, not an average (an optional computed average may be *shown* alongside). - **Quick logging without IDs.** A title alone is a valid entry ("draft"). It can be linked to catalogue IDs later, or never. - **Watchdog.** Each source is snapshotted on a schedule. Additions, edits, deletions and merges made *on the platform* are detected and shown in a **Changes** inbox with a clear before/after. Nothing destructive is applied to the user's data automatically. - **Backup and export.** Scheduled full backups. Exports in open formats (JSON, CSV, Markdown) that are easy for people and AI tools to work with. - **Portability.** Move data between services, via write-back where a platform allows it, or via generated import files. - **Private owned library.** What you *own or have access to* (Steam games, a Jellyfin server, physical books) is tracked separately from what you've *consumed*. It is private, never shown on profiles, and used to match plays/reads and to browse for something to start next. Each copy can record where/when it was acquired, price, format and notes. - **Sources as plugins.** Each integration is a self-contained plugin against a small, stable contract, so plugins can be fixed, added or updated without changing the core. - **Usable by other apps.** STTracker has its own API with per-user, per-device tokens / app passwords, so other software (Jellyfin, Navidrome, KOReader, scripts, AI assistants) can read from and report to it. - **A personal public page.** A shareable page of the things that matter to the user (see §7). ## 3. Non-goals (for now) - Social features (following, feeds, comments). Nice-to-have, later. - A public catalogue that competes with TMDB/IGDB/OpenLibrary. STTracker caches metadata it needs; it is not a database for others. - A heavy recommendation engine. Cross-media suggestions start from the user's own links, tags and threads (and possibly AI help), later. - OAuth *provider* (third-party apps signing in with STTracker). App tokens cover v1; OAuth later with the public instance. - Scraping Backloggd. It has no API, is behind bot protection, and its ToS/robots discourage it. Backloggd support waits for its official CSV export. - Writing history *into* Last.fm (the API silently drops anything older than ~14 days). ## 4. Anti-goals (failure even if it "works") - A platform sync silently overwrites or deletes something the user entered. - The user can't log something because there's no catalogue match. - The user's data is stuck: export is lossy, proprietary or incomplete. - Media types feel like separate apps with different features. - The public page looks like an admin stats dashboard instead of something about stories and things the user loves. - Adding or fixing one integration requires touching the core. ## 5. Constraints - **Runtime:** Bun (1.3+). Follow Bun-native conventions: `Bun.serve()` with HTML imports for the frontend (no Vite/Express), `bun:sqlite` / `Bun.sql`, `bun test`, `Bun.password`, `Bun.file`, Bun's automatic `.env` loading. - **Databases:** SQLite (default, zero-config) and PostgreSQL (option). One code path for the domain logic; both dialects tested in CI. - **Frontend:** React. - **Deployment:** Docker image + compose file. All state under one data volume (`/data`: SQLite db if used, attachments, backups, raw snapshots, external plugins). - **Source terms that shape the design:** - Each self-hoster registers their own API credentials for Trakt, Last.fm, IGDB (Twitch), and optionally OAuth apps on Hardcover/MangaBaka. No shared keys are shipped. - Trakt requires the credit line "STTracker uses Trakt but is not endorsed or certified by Trakt." MangaBaka data is CC BY-NC-SA (attribution, non-commercial). IGDB/Last.fm require attribution, non-commercial. Show attributions in the app footer/about page. - Respect rate limits (Hardcover 60/min; MangaBaka 180/min; Trakt 500 GET/5 min, 1 write/s; IGDB 4/s; ListenBrainz ≈1/s with `X-RateLimit-*` headers; Last.fm ≈5/s). - Hardcover's API is beta and may break; its plugin must fail loudly and safely. - **Repo:** Gitea remote `https://gitea.lucaswinther.info/lucasw89/STTracker.git`. ## 6. Domain model All IDs are ULIDs (sortable, generated in app code so both dialects behave the same). All user-facing rows have `created_at`, `updated_at`, and soft-delete `deleted_at`. Ratings are stored as integers 0–100. Display scale (5 stars with halves, 10-point, 100-point) is a user preference. ### 6.1 Catalogue layer: `works` A **work** is a thing that can be consumed: a book, an edition-agnostic series, a game, a show, a season, an episode, an album, a track, a podcast, a manga series, etc. - `media_type` comes from a built-in set (book, comic, manga, game, movie, show, season, episode, podcast, podcast_episode, audio_drama, artist, album, track, video, other). Users can add their own types. A user-defined type behaves like `other` but has its own name, icon and default custom fields. - `parent_id` → `works`, forming the hierarchy (series → book; show → season → episode; album → track; game → DLC). A work may also belong to user groupings via lists (§6.5), which are separate from this structural parent. - Display fields: title, original title, year/date, creators (JSON list of `{name, role}`), cover (attachment or remote URL cached locally), description, extra metadata (JSON). - `owner_user_id` is null for shared catalogue works (created from sources) and set for private works a user created by hand (quick add). Private works are visible only to their owner. - `external_ids` table: (`work_id`, `namespace`, `value`), unique on (`namespace`, `value`). Namespaces include `isbn13`, `isbn10`, `asin`, `hardcover_book`, `hardcover_edition`, `tmdb_movie`, `tmdb_show`, `tvdb`, `imdb`, `trakt`, `igdb`, `steam`, `mangabaka`, `anilist`, `mal`, `musicbrainz_recording/release/artist`, etc. Plugins may register new namespaces. - **Work merges** (user-initiated, or confirmed from a suggestion) keep a `merged_into_id` tombstone, so old references and external IDs still resolve. #### Who controls the shared catalogue Shared works are a **metadata cache**, not user data. - Source plugins may update a shared work's display fields directly. Every change is recorded in `work_history`, so it can be inspected and reverted by an admin. - What a user sees can always be pinned with their own **overrides** (§6.2). Overrides are per user and never touched by syncs. - A platform merge (e.g. Hardcover dedupes a book) never merges shared works on its own. It produces a change event in each affected user's inbox. "Follow the merge" moves *that user's* entry to the target work. - A user merging two works is also per user: their entry moves to the target, and an alias is kept so their old references still resolve. Only an admin can merge shared works globally, which leaves a `merged_into_id` tombstone. - A private draft work stays private. When the user links it to a shared work, their entry moves to the shared work, and the draft is tombstoned and points to it. - User-defined media types are a per-user label stored on the entry (`media_type_override`). Shared works keep a built-in type, defaulting to `other`. - Search shows shared works plus the user's own private works. ### 6.2 User layer: `entries` An **entry** is one user's relationship with one work. It is unique per (user, work). - `status`: a small fixed set mapped across all media: planned, in_progress, paused, completed, dropped, plus "none" for owned-but-untouched or rated-only. Each media type can show its own wording ("Want to read", "Watching", "Played"). Users can rename the labels. - `rating` (0–100, nullable), `review` (Markdown), `private_notes` (Markdown), `favorite` flag, `tags`. - `custom_fields` (JSON): values for user-defined fields (§6.6). - `visibility`: private | friends | public. The default is a user preference, initially private. - **Overrides:** a user can override display fields (title, cover, year…) for themselves without changing the shared work. - **Draft:** an entry whose work has no external IDs and no confirmed match is shown as a "Draft" so the user can fix it later. This is a derived state, not a separate table. ### 6.3 `engagements`: each read, watch or play One row per time through: `entry_id`, `started_at`, `finished_at`, `date_precision` (day | month | year | unknown, so "I read this in 2019" is valid), `progress` (JSON: page/percent/chapter/episode/hours), `platform` (free text, e.g. "Switch", "Kindle"), `rating` and `notes` for that particular pass (optional), `source_ref` (which source reported it, if any). #### How deep imports go - Imports create shared works at every level the source reports, since works are cheap catalogue rows. For example, Trakt episode plays create show, season and episode works. - **Entries** are created only at the level where the source reports user data. A Trakt episode watch creates an episode entry plus an engagement on it. The show gets an entry too, holding the status and rating Trakt reports for the show. - The library list shows top-level entries by default, with children rolled up. For example: "season 3 of 5, 41 episodes watched". - Parent status is never derived automatically. The rollup is shown, and the user decides. - Tracks are the exception: plays stay in `listens` (§6.4), and track or album entries exist only when a source reports a rating/love or the user adds one. ### 6.4 `listens` Music scrobbles are high-volume (hundreds of thousands) and get their own append-only table: `user_id`, `listened_at`, `artist`, `track`, `album`, `duration`, optional MBIDs, `work_id` (nullable, filled in by background matching), `source`, `source_key` (ListenBrainz `listened_at`+`recording_msid`, or Last.fm `ts`+artist+track). Unique on (user, source, source_key). Albums and tracks can still have normal entries for ratings and reviews. **Matching listens** only links to existing works, or creates shared artist/album/track works when the listen carries MusicBrainz IDs. Listens with no IDs and no match stay unmatched (`work_id` null). Listens never create draft works. They remain fully visible in Log and in exports either way. ### 6.5 Lists User-created ordered collections of works, mixed media allowed. Each item has an optional note. Fields: name, description, visibility, `kind`: - `list`: a plain list. - `grouping`: the user's own "box set" / reading order, e.g. "Silent Hill, everything" or "my Cosmere order". It can be rated and reviewed like an entry (it has its own entry-like fields). - `thread`: a cross-media chain of related things, e.g. "cosmic dread: Leitner → Magnus Archives → Voices of the Void → Twilight Zone". Threads feed the public page's river view and future recommendations. ### 6.6 Custom fields `custom_field_definitions`: per user, `name`, `type` (text, long text, number, date, boolean, single select, multi select, url, rating), `applies_to` (all media types or a chosen subset), `level` (entry | engagement | copy). Values live in the JSON column of the matching row. Examples: "Recommended by", "Mood", "Translator". ### 6.7 Owned library: `copies` Private to the user, never public. Fields: `user_id`, `work_id` (nullable until matched), `title_hint` (raw title from the source when unmatched), `format` (hardback, ebook, Steam, disc, file on Jellyfin…), `acquired_from`, `acquired_at` (+ precision), `price` + `currency`, `notes`, `custom_fields`, `source_ref`. Copies are linked to entries through the shared work. Library-only sources (Steam, Jellyfin, Kavita…) create copies, not entries. ### 6.8 Attachments Content-addressed files (sha256) stored on disk under `/data/attachments`. The storage backend is behind an interface so S3-compatible storage can be added later. Attachments can be linked to an entry, engagement, copy, list or work (covers). Each has a caption, `taken_at`, and visibility (inherits from its parent by default). Thumbnails are generated for images. Attachments are included in backups. ### 6.9 History of the user's own data Every change to a user-layer row (entries, engagements, copies, lists, custom field values) writes a row to `user_history`: entity, id, before/after JSON diff, actor (user | plugin: | api token:), timestamp. Users can view an entry's history and restore an earlier version. Deletes are soft deletes. This, together with snapshots, is what makes "nothing gets lost" true. ### 6.10 Users and auth - `users`: username, display name, password hash (`Bun.password`, argon2id), role (admin | member), preferences JSON. The first registered user becomes admin. After that, registration is invite-only (admin creates invite links). - Sessions: secure HTTP-only cookie, server-side session table. - **App tokens** (also called "app passwords"): per user, named, scoped (read, write, ingest), revocable, last-used tracking, hashed at rest. Used by the API, plugins' inbound endpoints, scripts and AI tools. ## 7. Surfaces (UI) The mockups from brainstorming (round 2) are the reference. They are in `docs/design/mockups/` (`round2-app-scrapbook-river.html` is the reference; `round1-seven-directions.html` holds the parked themes). ### 7.1 The app (everyday use), "Hardcover-style" Dark, calm, dense. Indigo/near-violet surfaces, amber for ratings, orange reserved for change warnings. Three panes on desktop: navigation sidebar, library list, entry detail. Collapses to a stack on mobile. - **Navigation:** Library, Log (chronological engagements and listens), Lists, Owned, Changes (with an unread count), Sources (one item per connection), Your page, Settings. - **Library list:** search, filters (media type, status, rating, tag, source, draft), sort. "Quick add" box at the top: type a title, optionally pick a media type, done. The entry is created as a draft. - **Entry detail:** cover, title, rating, status, tags, custom fields; a change warning when a source changed something; engagements (reads/plays) with dates and precision; your copy (owned library); attachments; linked sources with sync status; review; hierarchy (parent and children, each with its own rating); history. - **Changes inbox:** grouped by source run. Each change shows before/after and actions: accept (apply to my data), ignore, or "keep mine". Merge events show "X was merged into Y on " with an option to follow the merge. - **Matching:** for drafts and unmatched copies/listens, show candidate works (by external ID, then fuzzy title+year+creator) and let the user confirm, pick another, or keep as-is. - **Sources:** connect/disconnect, credentials, schedule, last run, run now, per-run log, capability toggles (import / watch / write-back). - **Export & backup:** download exports, configure scheduled backups, and restore. ### 7.2 The public page, "scrapbook + river" A shareable URL per user (`/u/`), showing only what the user marks public. - **Scrapbook layout (default):** a grid-based collage of the user's chosen highlights. Polaroid-style cards for entries with screenshots/covers and handwritten-style captions, note cards for reviews, "ticket" cards for copies/provenance (only if the user explicitly chooses to show it), thread cards, and a "right now" strip of in-progress entries. Must read well on phones. - **River view:** a timeline where each media type is a stream whose width is time spent (from engagement dates/durations, listens and progress). Covers sit where things were finished, and threads draw links across streams. Tapping/hovering a cover shows its rating, date and a review excerpt. - **The user curates it:** pick highlights, order them, choose captions and which threads to show. - **Themes later:** the "shelf" and "card catalogue" directions from round 1 are parked as optional alternative themes. ## 8. Sources: the plugin system ### 8.1 Shape A plugin is a folder with a `manifest` and a TypeScript entry module exporting an object that implements the plugin contract from the `@sttracker/plugin-sdk` package. Plugins are loaded at startup from: - `plugins/` in the repo (built-ins, shipped in the image), and - `/data/plugins/` (external plugins dropped in by the instance admin). The admin can reload plugins from the UI without restarting the server, by re-importing with a cache-busting query. If hot reload proves unreliable in Bun, a restart is acceptable; it must not need a rebuild of the core. Plugins are **trusted code** running in-process. Only the instance admin installs them. Sandboxing is out of scope. ### 8.2 Contract (capabilities) The manifest declares `id`, `name`, `version`, `sdkVersion`, the media types covered, a config/credential schema (rendered as a form in Sources), and which capabilities it implements: | Capability | What it does | |---|---| | `pull` | Full fetch of the user's data from the source (backfill and periodic full snapshot). | | `pullChanges` | Incremental fetch using a cursor or "last activity" signal, if the source supports it. | | `push` | Write changes back to the source (status, rating, engagement, list…). | | `ownedLibrary` | Fetch owned/available items, which become `copies`. | | `ingest` | Expose HTTP endpoints under `/ingest//…` for other apps to push events (webhooks, ListenBrainz-compatible API, kosync). Authenticated with app tokens. | | `lookup` | Search/resolve catalogue metadata (used by matching). | | `importFile` / `exportFile` | Parse an uploaded file, or generate a file in the source's import format. | **Plugins never touch the database.** They receive a context (`http` client with rate limiting, retries and `Retry-After` handling; `log`; `credentials` get/refresh/save; `cursor` get/set; `now`) and return **normalised source records**: ``` SourceRecord { key: string // stable id of this record on the source kind: "entry" | "engagement" | "list" | "listItem" | "copy" | "listen" | "work" raw: unknown // exact payload, stored in the snapshot normalized: { work hints (title, media type, year, creators, external ids, parent hint), entry/engagement/copy/listen fields mapped to the STTracker model } mergedInto?: string // the platform says this record was merged into another deleted?: boolean // the platform reports deletion (else inferred from absence in a full pull) } ``` The core owns everything else: snapshot storage, diffing, change events, matching, applying changes to the user layer, history, scheduling and backups. ### 8.3 Snapshots and change detection - `source_connections`: user, plugin id, config, encrypted credentials (AES-GCM with a key from the `STTRACKER_SECRET` env var), schedule, cursor, status. - `source_records`: (connection, key) → current hash, `first_seen`, `last_seen`, `deleted_at`, `merged_into`, and the link to the STTracker row it maps to. - `source_record_versions`: (record, hash, raw payload compressed, `seen_at`). A new version is stored only when the hash changes. - `source_runs`: one per pull, with counts and a log. - After each run the core diffs against the previous state and emits `change_events`: added, changed (field-level diff), removed (missing from a full pull, or reported deleted), merged. - **Applying changes:** - New records are applied automatically. They create entries, engagements, copies or listens, matched to works. - A change to a field the user has never edited in STTracker is applied automatically and logged. - A change to a field the user has edited, every removal and every merge goes to the Changes inbox as a pending decision. Nothing the user wrote is overwritten or deleted without their say. - Per-field provenance (`field_sources` JSON on user rows: field → "user" | "plugin:@