fix(refresh): make a source that stopped answering turn the run red

The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.

`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.

A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.

Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-17 21:56:01 +02:00
co-authored by Claude Opus 5
parent d2615c606c
commit 2a0ea1a796
5 changed files with 564 additions and 15 deletions
+17 -2
View File
@@ -180,6 +180,10 @@ Sources are community wikis. Treat them as a guest would:
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
- One request per source per refresh cycle, minimum 6 hours apart.
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
what a burst looks like from the far end regardless of our intent.
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
- Record `sourceUrl` on every event and surface attribution in the UI.
@@ -207,8 +211,19 @@ load-bearing here and not only a cost decision. Note also that Reverse: 1999, Bl
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything
that would make it fetch more often is a change to this section first.
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
read). Anything that would make it fetch more often is a change to this section first.
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
never blank a calendar or stop the sources that did answer from being committed — so a failure is
exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
runner instead would skip the commit and throw away the pages that did arrive. This tier exists
because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
opens the log to read is not a signal.
## Untrusted input