fix(refresh): make a source that stopped answering turn the run red

The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.

`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.

A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.

Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-17 21:56:01 +02:00
co-authored by Claude Opus 5
parent d2615c606c
commit 2a0ea1a796
5 changed files with 564 additions and 15 deletions
+12 -1
View File
@@ -177,10 +177,21 @@ section but must never claim the event title.
permission we have. A 404 means no restrictions.
- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
one per source per cycle. A failed source waits for the next cycle instead.
- **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it
states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates,
so a source we then skip costs nothing.
- Store raw bytes in `snapshots/<source-id>.html`, with hash/ETag/Last-Modified alongside it.
On failure: increment the failure streak, leave published events untouched, end as `failed`. A
source being down never mutates the feed.
source being down never mutates the feed. A non-`ok` status also records what turned us away — the
`Server` header, whether a `CF-Ray` was present, any `Retry-After` — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner is a bot farm.
**The failure streak is read, not just written.** `consecutiveFailures` reaching
`BROKEN_AFTER_FAILURES` (3, so ~36h at two cycles a day) promotes a source from "down" to `broken`:
annotated on the run page, listed in the job summary with its status code, and counted in the
`broken` step output that `refresh.yml` fails on *after* committing. See CLAUDE.md § Scraping
conduct for why that ordering is load-bearing.
**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
`.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by