fix(refresh): make a source that stopped answering turn the run red
The only snapshot the scheduled refresh has ever committed is endfield-wikigg-events. Seven sources existed at that run; the six game8.co ones yielded nothing, and no cycle since has committed anything. All of them fetch fine from a laptop, so whatever is happening happens on the runner — meanwhile eight of ten games were served from checked-in fixtures for three days behind a green tick. Nine failures out of ten was exit 0 with warnings buried in a log nobody opens. `consecutiveFailures` was already tracked and never read. A source that has failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported as `broken`: a GitHub annotation, a job-summary row carrying its status code, and a `broken` step output. The runner still exits 0 on it and `refresh.yml` fails on that output in a final step, after the commit and the CI dispatch — exiting non-zero from the runner would skip the commit and throw away the pages that did arrive, which is the opposite of what "one wiki down never blanks a calendar" is for. The streak is read from the store rather than from this cycle's outcome, so a source dead for days that happens to be inside its six-hour window has not recovered. A non-ok response now records what turned us away — the Server header, whether a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads identically whether the page moved behind a login or a CDN decided the runner is a bot farm, and that is the open question here. Values are trimmed and capped: the note lands in a workflow command and a markdown cell, and it came from a host we do not control. Also space requests to a host already asked this cycle, honouring its Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the per-source floor alone still permitted one cycle to arrive as eight back-to-back requests to a single site — which is what a burst looks like from the far end regardless of our intent, and is plausibly self-inflicted here. The wait is taken after the interval and robots gates, so a source we then skip costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
d2615c606c
commit
2a0ea1a796
+12
-1
@@ -177,10 +177,21 @@ section but must never claim the event title.
|
||||
permission we have. A 404 means no restrictions.
|
||||
- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
|
||||
one per source per cycle. A failed source waits for the next cycle instead.
|
||||
- **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it
|
||||
states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates,
|
||||
so a source we then skip costs nothing.
|
||||
- Store raw bytes in `snapshots/<source-id>.html`, with hash/ETag/Last-Modified alongside it.
|
||||
|
||||
On failure: increment the failure streak, leave published events untouched, end as `failed`. A
|
||||
source being down never mutates the feed.
|
||||
source being down never mutates the feed. A non-`ok` status also records what turned us away — the
|
||||
`Server` header, whether a `CF-Ray` was present, any `Retry-After` — because a bare `HTTP 403` reads
|
||||
identically whether the page moved behind a login or a CDN decided the runner is a bot farm.
|
||||
|
||||
**The failure streak is read, not just written.** `consecutiveFailures` reaching
|
||||
`BROKEN_AFTER_FAILURES` (3, so ~36h at two cycles a day) promotes a source from "down" to `broken`:
|
||||
annotated on the run page, listed in the job summary with its status code, and counted in the
|
||||
`broken` step output that `refresh.yml` fails on *after* committing. See CLAUDE.md § Scraping
|
||||
conduct for why that ordering is load-bearing.
|
||||
|
||||
**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
|
||||
`.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by
|
||||
|
||||
Reference in New Issue
Block a user