fix(core): job failure semantics, stale-lock reaper, run-scoped context, reload

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas Winther
2026-10-30 20:52:51 +01:00
co-authored by Claude Sonnet 5.5
parent 166b8ecb5e
commit 6f628c4693
11 changed files with 356 additions and 83 deletions
+16
View File
@@ -4,6 +4,22 @@ Append-only. Never edit or delete an entry. Supersede it with a new one that lin
One entry per decision a future maintainer would otherwise have to reverse-engineer.
No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry (date, title, line number).
## 2026-10-30: Job failure semantics and plugin reload limits
**Status:** accepted
**Context:** Task 9 fix round 1.
**Decision:**
- **Failures before `runSync` never retry.** A missing/paused connection drops the job (`DropJobError`: `done`, message in
`last_error`). A missing plugin or undecryptable credentials (`CryptoError`) insert a failed `source_runs` row, set the
connection to `error`, advance `last_run_at` and throw `NonRetryableError` (job `failed`, no backoff). The scheduler still
picks `error` connections, one interval later, so fixing the cause heals them.
- **Stale locks.** Every worker tick re-queues `running` jobs whose `locked_at` is over an hour old (attempts unchanged), so a
crash cannot hold a lockKey forever. Job bookkeeping errors are logged (`[jobs]`), never unhandled; `stop()` never rejects
and aborts the `signal` passed to handlers (`handler(payload, { signal })`).
- **Reload** evicts the entry file from `require.cache` and imports with `?v=<Date.now()>` (Bun ignores the query alone for
cached `.ts`). Only entry files reload: changes in a plugin's other modules need a server restart.
- `credentials.save` and `updateConnection` read-merge-write in one transaction (row lock on Postgres). `ctx.log` is buffered and
appended to the run's `source_runs.log` when the run ends.
## 2026-10-30: Jobs, plugin registry, connections: rate limit in the manifest, secret split, per-connection lock
**Status:** accepted
**Context:** Task 9 (`packages/core/src/{jobs,registry,connections}.ts`).