fix(core): job failure semantics, stale-lock reaper, run-scoped context, reload

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
2026-10-30 20:52:51 +01:00
co-authored by Claude Sonnet 5.5
parent 166b8ecb5e
commit 6f628c4693
11 changed files with 356 additions and 83 deletions
+3 -3
View File
@@ -23,9 +23,9 @@ user layer or files a Changes-inbox event.
| `docs/design/mockups` | approved UI mockups (reference only) | exists |
## Data flow
Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are Task 9's job worker, planned):
1. The scheduler enqueues a job.
2. The worker loads the connection and plugin and calls `runSync(deps, plugin, connectionId, mode)`, injecting
Sync lifecycle (`packages/core/src/sync/`; steps 1–2 are `jobs.ts` and `connections.ts`):
1. The scheduler enqueues a `sync` job (lockKey `sync:<connectionId>`) for each non-paused connection that is due.
2. The worker claims the job; `syncHandler` loads the connection and plugin and calls `runSync(deps, plugin, connectionId, mode)`, injecting
`makeContext` (http client, credentials, cursor). `mode` picks `pull`, `pullChanges` or `importFile`.
3. `runSync` opens a `source_runs` row and iterates the plugin's `SourceRecord`s one at a time.
- `listen` records are batched into `ingestListens` (insert-or-ignore into `listens`; no snapshot, no events).
+16
View File
@@ -4,6 +4,22 @@ Append-only. Never edit or delete an entry. Supersede it with a new one that lin
One entry per decision a future maintainer would otherwise have to reverse-engineer.
No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, line number).
## 2026-10-30: Job failure semantics and plugin reload limits
**Status:** accepted
**Context:** Task 9 fix round 1.
**Decision:**
- **Failures before `runSync` never retry.** A missing/paused connection drops the job (`DropJobError`: `done`, message in
`last_error`). A missing plugin or undecryptable credentials (`CryptoError`) insert a failed `source_runs` row, set the
connection to `error`, advance `last_run_at` and throw `NonRetryableError` (job `failed`, no backoff). The scheduler still
picks `error` connections, one interval later, so fixing the cause heals them.
- **Stale locks.** Every worker tick re-queues `running` jobs whose `locked_at` is over an hour old (attempts unchanged), so a
crash cannot hold a lockKey forever. Job bookkeeping errors are logged (`[jobs]`), never unhandled; `stop()` never rejects
and aborts the `signal` passed to handlers (`handler(payload, { signal })`).
- **Reload** evicts the entry file from `require.cache` and imports with `?v=<Date.now()>` (Bun ignores the query alone for
cached `.ts`). Only entry files reload: changes in a plugin's other modules need a server restart.
- `credentials.save` and `updateConnection` read-merge-write in one transaction (row lock on Postgres). `ctx.log` is buffered and
appended to the run's `source_runs.log` when the run ends.
## 2026-10-30: Jobs, plugin registry, connections: rate limit in the manifest, secret split, per-connection lock
**Status:** accepted
**Context:** Task 9 (`packages/core/src/{jobs,registry,connections}.ts`).
+3 -2
View File
@@ -24,8 +24,9 @@ Small, specific, independently actionable. Remove only when actually done.
file, and compare steps and token use.
## Known issues
- [ ] Jobs: a job left `running` by a crash is never re-queued; add a stale-lock reaper using `locked_at`.
- [ ] Plugins: `ctx.log` is discarded; route it into `source_runs.log`.
- [ ] Plugins: reload re-imports entry files only; edits to a plugin's other modules need a server restart.
- [ ] Plugins: rate limiting is per run (each run builds its own client), not per plugin (spec §11); share a limiter per plugin in Phase 2.
- [ ] Plugins: `ctx.log` reaches `source_runs.log` only after the run ends, so a crash mid-run loses it.
- [ ] Restore: a new child work whose parent+position collides with an existing child of a re-matched parent fails
with `ImportError("conflict")` instead of being merged or repositioned.
- [ ] Restore: an unlink of identical attachment bytes between the pre-write and the commit could remove the file;