Merge branch 'feat/phase1' into task/8
# Conflicts: # docs/ARCHITECTURE.md # docs/DECISIONS.md # docs/TODO.md
This commit is contained in:
@@ -16,7 +16,7 @@ user layer or files a Changes-inbox event.
|
||||
|---|---|---|
|
||||
| `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
|
||||
| `apps/web` | React app (HTML imports) | planned |
|
||||
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`). Export planned | exists (partial) |
|
||||
| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`) | exists (partial) |
|
||||
| `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
|
||||
| `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
|
||||
| `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |
|
||||
|
||||
@@ -52,6 +52,32 @@ are the mechanics a maintainer would otherwise reverse-engineer.
|
||||
**Consequences:** `list` records are snapshotted but not applied, and removal of a de-duplicated engagement
|
||||
deletes it even if another source still reports it (both in `docs/TODO.md`).
|
||||
|
||||
## 2026-10-30: Export format `sttracker-export/v1`: per-user zip, camelCase JSONL, deleted rows included, no secrets
|
||||
**Status:** accepted
|
||||
**Context:** Task 15 (`packages/core/src/{export,backup}.ts`) fixed the on-disk bundle format that backups and restore share.
|
||||
**Decision:**
|
||||
- **Files.** A zip (`fflate`) holding `manifest.json` (`{ format, exportedAt, user: { username, displayName } }`),
|
||||
`works.jsonl`, `entries.jsonl`, `engagements.jsonl`, `listens.jsonl`, `lists.jsonl` (items nested),
|
||||
`copies.jsonl`, `custom_fields.jsonl` (field definitions), `history.jsonl`, `attachments.jsonl` (links nested)
|
||||
and `attachments/<sha256>` bytes. One JSON object per line, `\n` separated.
|
||||
- **Keys.** camelCase; JSON columns are parsed into real JSON; `favorite` is a boolean. `user_id` columns are omitted
|
||||
(the importing user owns everything). `works.jsonl` adds `private` and `externalIds`.
|
||||
- **Deleted rows are included** with their `deletedAt`, and `fieldSources`/timestamps are kept, so a restore is lossless.
|
||||
- **No secrets.** No users row beyond username/display name, no sessions, tokens, connections, credentials or source snapshots.
|
||||
- **Merges and aliases.** `works.jsonl` rows carry `mergedIntoId` (the survivor and ancestors are exported too) and
|
||||
`work_aliases.jsonl` holds the user's `{ fromWorkId, toWorkId }` rows. Both are additive to v1. Restore re-creates
|
||||
tombstones (for works it creates) and aliases, so a later write to the old work still resolves to the moved entry.
|
||||
- **Export is one snapshot:** all reads run in a single transaction (repeatable read on Postgres).
|
||||
- **Restore** (`importUserBundle`) needs an account with no entries (`account_not_empty`) and `format === "sttracker-export/v1"`
|
||||
(`unsupported_format`). Works with external IDs are re-matched (else created shared); works without become private.
|
||||
Ids are remapped (UUIDs inside history `before`/`after` and custom-field value keys too). Rows are inserted directly, so no
|
||||
new history is written. The import validates first (attachment mime/size/sha256, rating 0-100, status, visibility, 2 GB
|
||||
uncompressed cap) and writes all rows, attachments included, in one transaction; errors are
|
||||
`invalid_attachment`, `corrupt_bundle`, `bundle_too_large`, `conflict`. Works merged away at export time are exported as the tombstone row without `mergedIntoId`.
|
||||
- **Backups** write `<dataDir>/backups/<ISO with ':'->'-'>/<username>.zip` per user plus `sttracker.db` (`VACUUM INTO`, SQLite only),
|
||||
are built in `<stamp>.tmp` then renamed (modes 0700/0600), and prune to the newest `keep` (default 14).
|
||||
**Consequences:** Any change to these files after release needs `v2` and a migration path for old bundles.
|
||||
|
||||
## 2026-10-30: User layer: per-user moves merge entries, keep aliases, and roll the library up by hierarchy
|
||||
**Status:** accepted
|
||||
**Context:** Task 6 built the user layer (`packages/core/src/{entries,engagements,history,lists,copies}.ts`).
|
||||
|
||||
+5
-7
@@ -24,15 +24,13 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
file, and compare steps and token use.
|
||||
|
||||
## Known issues
|
||||
- [ ] Sync: `list` records are snapshotted but not applied to `lists` (Trakt/Hardcover emit them). Decide
|
||||
the mapping (one list per record, items by matched work) and apply it through the change engine.
|
||||
- [ ] Sync: accepting a `removed` event on an engagement de-duplicated across sources soft-deletes it even if
|
||||
another connection still reports it; consider detaching that connection's `engagement_sources` row instead.
|
||||
- [ ] Sync: pending `changed`/`conflict` events for a value the source has since moved past stay pending;
|
||||
consider superseding them when a newer value for the same field arrives.
|
||||
- [ ] Sync: two concurrent runs of the same connection can both insert a new `source_records` row for one key
|
||||
(one fails on the unique key and is logged as a record error). The job worker should lock per connection.
|
||||
- [ ] Sync: `ingestListens` does one lookup and one insert per listen; batch it for large backfills.
|
||||
- [ ] Restore: a new child work whose parent+position collides with an existing child of a re-matched parent fails
|
||||
with `ImportError("conflict")` instead of being merged or repositioned.
|
||||
- [ ] Restore: an unlink of identical attachment bytes between the pre-write and the commit could remove the file;
|
||||
mitigated by re-writing after commit, not locked.
|
||||
- [ ] Export bundles hold attachment bytes in memory; very large libraries need a streaming zip.
|
||||
- [ ] Admin merge does not merge colliding children recursively; colliding children stay under the tombstoned parent.
|
||||
- [ ] Matching: a shared parent resolved without external IDs or position (title/year reuse in
|
||||
`packages/core/src/matching.ts`) has no unique key, so concurrent matches can create it twice.
|
||||
|
||||
Reference in New Issue
Block a user