fix(refresh): make a source that stopped answering turn the run red
The only snapshot the scheduled refresh has ever committed is endfield-wikigg-events. Seven sources existed at that run; the six game8.co ones yielded nothing, and no cycle since has committed anything. All of them fetch fine from a laptop, so whatever is happening happens on the runner — meanwhile eight of ten games were served from checked-in fixtures for three days behind a green tick. Nine failures out of ten was exit 0 with warnings buried in a log nobody opens. `consecutiveFailures` was already tracked and never read. A source that has failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported as `broken`: a GitHub annotation, a job-summary row carrying its status code, and a `broken` step output. The runner still exits 0 on it and `refresh.yml` fails on that output in a final step, after the commit and the CI dispatch — exiting non-zero from the runner would skip the commit and throw away the pages that did arrive, which is the opposite of what "one wiki down never blanks a calendar" is for. The streak is read from the store rather than from this cycle's outcome, so a source dead for days that happens to be inside its six-hour window has not recovered. A non-ok response now records what turned us away — the Server header, whether a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads identically whether the page moved behind a login or a CDN decided the runner is a bot farm, and that is the open question here. Values are trimmed and capped: the note lands in a workflow command and a markdown cell, and it came from a host we do not control. Also space requests to a host already asked this cycle, honouring its Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the per-source floor alone still permitted one cycle to arrive as eight back-to-back requests to a single site — which is what a burst looks like from the far end regardless of our intent, and is plausibly self-inflicted here. The wait is taken after the interval and robots gates, so a source we then skip costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
d2615c606c
commit
2a0ea1a796
@@ -180,6 +180,10 @@ Sources are community wikis. Treat them as a guest would:
|
||||
|
||||
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
||||
- One request per source per refresh cycle, minimum 6 hours apart.
|
||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
|
||||
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
|
||||
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||
what a burst looks like from the far end regardless of our intent.
|
||||
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
||||
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
||||
- Record `sourceUrl` on every event and surface attribution in the UI.
|
||||
@@ -207,8 +211,19 @@ load-bearing here and not only a cost decision. Note also that Reverse: 1999, Bl
|
||||
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
||||
|
||||
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
|
||||
retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything
|
||||
that would make it fetch more often is a change to this section first.
|
||||
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
|
||||
read). Anything that would make it fetch more often is a change to this section first.
|
||||
|
||||
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
|
||||
never blank a calendar or stop the sources that did answer from being committed — so a failure is
|
||||
exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
|
||||
cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
|
||||
checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
|
||||
a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
|
||||
the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
|
||||
runner instead would skip the commit and throw away the pages that did arrive. This tier exists
|
||||
because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
|
||||
opens the log to read is not a signal.
|
||||
|
||||
## Untrusted input
|
||||
|
||||
|
||||
Reference in New Issue
Block a user