feat(core): jobs, scheduler, plugin registry, connections

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

docs/ARCHITECTURE.md

@@ -16,7 +16,7 @@ user layer or files a Changes-inbox event.
 |---|---|---|
 | `apps/server` | `Bun.serve`: `/api/v1`, `/ingest/<plugin>`, job worker, static web | planned |
 | `apps/web` | React app (HTML imports) | planned |
-| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`) | exists (partial) |
+| `packages/core` | domain services, one module per area (import by path, no barrel): `auth`, `crypto`, `works`, `matching`, `entries` (entries, library listing, quick add, per-user moves, match suggestions), `engagements`, `history` (`user_history`, restore), `lists`, `copies`, `custom-fields`, `attachments`, `export` (per-user bundle, CSV, restore), `backup` (scheduled bundles + SQLite `VACUUM INTO`), and `sync/` (the change engine: `snapshot` hashing and versions, `apply` per-record diff and apply, `run` = `runSync`, `changes` = Changes inbox `listChanges`/`resolveChange`, `listens` = `ingestListens`), `jobs` (queue, worker, scheduler), `registry` (`loadPlugins`), `connections` (CRUD, plugin context, `syncHandler`) | exists (partial) |
 | `packages/db` | Kysely over one Bun.SQL dialect (SQLite + Postgres), ids, JSON helpers, migrations, `forEachDialect` test harness | exists (schema pending) |
 | `packages/plugin-sdk` | plugin contract, `SourceRecord`, http/rate-limit helpers, fixture test harness | planned |
 | `plugins/<id>` | one source integration each (hardcover, trakt, mangabaka, listenbrainz, lastfm, csv) | planned |

docs/DECISIONS.md

@@ -4,6 +4,27 @@ Append-only. Never edit or delete an entry. Supersede it with a new one that lin
 One entry per decision a future maintainer would otherwise have to reverse-engineer.
 No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, line number).
 
+## 2026-10-30: Jobs, plugin registry, connections: rate limit in the manifest, secret split, per-connection lock
+**Status:** accepted
+**Context:** Task 9 (`packages/core/src/{jobs,registry,connections}.ts`).
+**Decision:**
+- **Rate limit lives in the manifest** (`PluginManifest.rateLimit?: { perMinute, burst? }`), not in plugin code. The host builds
+  `ctx.http` from it (default 60/min, burst 5, UA `STTracker/0.1 (+self-hosted)`), so every plugin is throttled the same way
+  and the per-plugin `RATE_PER_MINUTE` constants are gone.
+- **Secrets are split off config.** Config fields typed `"secret"` in the manifest are stored only in the encrypted
+  `credentials_enc` blob (with OAuth tokens a plugin saves), never in `config`. `makePluginContext` merges them back into
+  `ctx.config`; `listConnections` returns plain config and `hasCredentials` only. `updateConnection` keeps secrets that are
+  omitted and replaces plain config wholesale. `credentials.save` merges and persists immediately, so a token refreshed
+  mid-pull survives a crash.
+- **Constructor deviation.** `createConnection`/`updateConnection` take the plugin `manifest` as a trailing argument (needed
+  for the split), and `makePluginContext` is async (it decrypts) and takes the manifest before an optional `http`.
+- **One sync per connection.** Sync jobs carry `lockKey = "sync:<connectionId>"`; `enqueue` returns null while a
+  `queued|running` job holds the key. On Postgres the check and insert run under `pg_advisory_xact_lock(hashtext(key))`.
+  Claiming is `update … where status = 'queued'` checked on affected rows. Retries back off `2^attempts` minutes, 5 attempts, then `failed`.
+- **Mode choice.** The sync handler runs `full` when asked, when the plugin lacks `pullChanges`, or when no `ok` full run started
+  within 7 days; else `changes`. A failed run is rethrown so the worker retries it.
+**Consequences:** Reload re-imports plugin modules with `?v=<Date.now()>`; old module instances are not freed by Bun.
+
 ## 2026-10-30: Export format `sttracker-export/v1`: per-user zip, camelCase JSONL, deleted rows included, no secrets
 **Status:** accepted
 **Context:** Task 15 (`packages/core/src/{export,backup}.ts`) fixed the on-disk bundle format that backups and restore share.

docs/TODO.md

@@ -24,6 +24,8 @@ Small, specific, independently actionable. Remove only when actually done.
     file, and compare steps and token use.
 
 ## Known issues
+- [ ] Jobs: a job left `running` by a crash is never re-queued; add a stale-lock reaper using `locked_at`.
+- [ ] Plugins: `ctx.log` is discarded; route it into `source_runs.log`.
 - [ ] Restore: a new child work whose parent+position collides with an existing child of a re-matched parent fails
   with `ImportError("conflict")` instead of being merged or repositioned.
 - [ ] Restore: an unlink of identical attachment bytes between the pre-write and the commit could remove the file;
@@ -35,8 +37,8 @@ Small, specific, independently actionable. Remove only when actually done.
   another connection still reports it; consider detaching that connection's `engagement_sources` row instead.
 - [ ] Sync: pending `changed`/`conflict` events for a value the source has since moved past stay pending;
   consider superseding them when a newer value for the same field arrives.
-- [ ] Sync: two concurrent runs of the same connection can both insert a new `source_records` row for one key
-  (one fails on the unique key and is logged as a record error). The job worker should lock per connection.
+- [ ] Sync: two concurrent runs of the same connection (outside the job queue, e.g. a manual run) can both insert a new
+  `source_records` row for one key. Queued syncs are serialised by `lockKey`.
 - [ ] Sync: `ingestListens` does one lookup and one insert per listen; batch it for large backfills.
 - [ ] Admin merge does not merge colliding children recursively; colliding children stay under the tombstoned parent.
 - [ ] Matching: a shared parent resolved without external IDs or position (title/year reuse in