fix(refresh): only 200 is a page, so a 202 challenge names itself
The health check earned its keep on the first run: six sources broken, four cycles each, "last confirmed never" — and a last status of **202**. That is game8.co's edge answering a GitHub runner with a bot-management body instead of the wiki. `response.ok` admitted it. A 202 therefore passed the status gate as a document, reached the parser, yielded nothing, and was reported as "kept previous snapshot; new body yielded 0 events" — an accurate description of the symptom that never mentioned the status explaining it, which is why this looked like a parser problem for days. Require 200: 202 means the request was accepted for processing, 204 has no body, 206 is a fragment, and none of them is a page. The rejection note already names the Server header, so the next cycle reads `HTTP 202 (AkamaiGHost)` instead. Recorded in CLAUDE.md § Scraping conduct, including that this must not be worked around. The per-host spacing added alongside does not help and was never going to — a 202 on a cycle's first request is address reputation, not rate. Browser -shaped headers or a residential egress would be defeating an access control, which is the ground uma.moe was declined on, and game8's robots.txt is permissive, which makes doing it there worse rather than better. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
988f72a56f
commit
54052d50c4
@@ -193,6 +193,19 @@ AI-training crawlers. Our use is a low-rate personal aggregator with attribution
|
||||
training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch
|
||||
rate, and do not add an LLM that consumes page content.
|
||||
|
||||
**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns
|
||||
`202 Accepted` with a bot-management body to every one of the eight game8 sources, from the first
|
||||
scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse
|
||||
cleanly from a normal address. So `robots.txt` permits us and the network does not, and those eight
|
||||
games have only ever been built from checked-in fixtures in CI.
|
||||
|
||||
The per-host spacing above does not fix this and was not meant to: a 202 on the very first request
|
||||
of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a
|
||||
proxy, or a residential egress would each be defeating a deliberate access control, which is the
|
||||
same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose
|
||||
`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the
|
||||
refresh from an address game8 will serve, or to find those games another source.
|
||||
|
||||
A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
|
||||
|
||||
**Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass:
|
||||
|
||||
Reference in New Issue
Block a user