fix(refresh): only 200 is a page, so a 202 challenge names itself

The health check earned its keep on the first run: six sources broken, four
cycles each, "last confirmed never" — and a last status of **202**. That is
game8.co's edge answering a GitHub runner with a bot-management body instead of
the wiki.

`response.ok` admitted it. A 202 therefore passed the status gate as a
document, reached the parser, yielded nothing, and was reported as "kept
previous snapshot; new body yielded 0 events" — an accurate description of the
symptom that never mentioned the status explaining it, which is why this looked
like a parser problem for days. Require 200: 202 means the request was accepted
for processing, 204 has no body, 206 is a fragment, and none of them is a page.
The rejection note already names the Server header, so the next cycle reads
`HTTP 202 (AkamaiGHost)` instead.

Recorded in CLAUDE.md § Scraping conduct, including that this must not be worked
around. The per-host spacing added alongside does not help and was never going
to — a 202 on a cycle's first request is address reputation, not rate. Browser
-shaped headers or a residential egress would be defeating an access control,
which is the ground uma.moe was declined on, and game8's robots.txt is
permissive, which makes doing it there worse rather than better.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-17 22:05:05 +02:00
co-authored by Claude Opus 5
parent 988f72a56f
commit 54052d50c4
4 changed files with 51 additions and 1 deletions
+13
View File
@@ -193,6 +193,19 @@ AI-training crawlers. Our use is a low-rate personal aggregator with attribution
training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch
rate, and do not add an LLM that consumes page content. rate, and do not add an LLM that consumes page content.
**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns
`202 Accepted` with a bot-management body to every one of the eight game8 sources, from the first
scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse
cleanly from a normal address. So `robots.txt` permits us and the network does not, and those eight
games have only ever been built from checked-in fixtures in CI.
The per-host spacing above does not fix this and was not meant to: a 202 on the very first request
of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a
proxy, or a residential egress would each be defeating a deliberate access control, which is the
same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose
`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the
refresh from an address game8 will serve, or to find those games another source.
A source whose ToS forbids automated access does not get an adapter. Flag it and ask. A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
**Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass: **Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass:
+5
View File
@@ -177,6 +177,11 @@ section but must never claim the event title.
permission we have. A 404 means no restrictions. permission we have. A 404 means no restrictions.
- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says - 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
one per source per cycle. A failed source waits for the next cycle instead. one per source per cycle. A failed source waits for the next cycle instead.
- **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole
2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge
instead of the wiki. Admitting it fed that challenge page to the parser, which reported "yielded 0
events" — the symptom, with the status that explained it unmentioned. `204` has no body and `206`
is a fragment; none of them is a document.
- **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it - **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it
states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates, states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates,
so a source we then skip costs nothing. so a source we then skip costs nothing.
+9 -1
View File
@@ -346,7 +346,15 @@ async function refreshOne(
}; };
} }
if (!response.ok) { // 200 is the only status that means "here is the page". `response.ok` also
// admits the rest of the 2xx range, and that cost us the diagnosis: game8.co's
// edge answers a GitHub runner with **202 Accepted** and a bot-management
// body, which sailed through this gate as a document, reached the parser,
// yielded no events and was reported as "kept previous snapshot" — describing
// the symptom while the status that explained it went unmentioned. 202 means
// the request was accepted for processing; 204 has no body and 206 is a
// fragment. None of them is a wiki page.
if (response.status !== 200) {
await store.recordCheck(adapter.id, { await store.recordCheck(adapter.id, {
at: nowIso, at: nowIso,
status: response.status, status: response.status,
+24
View File
@@ -214,6 +214,30 @@ describe("one request per source per six hours", () => {
expect(calls).toHaveLength(1); expect(calls).toHaveLength(1);
}); });
test("a 2xx that is not 200 is not a page", async () => {
// game8.co's edge answers a GitHub runner with 202 and a bot-management
// body. `response.ok` admitted it, so it reached the parser and was reported
// as "yielded 0 events" — the symptom, while the status that explained it
// was never named. Six sources read that way for days.
await seed("<html><event></event></html>", "2026-08-01T00:00:00.000Z", 1);
const { opts } = options({
responder: () =>
new Response("<html>checking your browser</html>", {
status: 202,
headers: { Server: "AkamaiGHost" },
}),
});
const summary = await runRefresh(opts);
expect(summary.outcomes[0]?.result).toBe("failed");
expect(summary.outcomes[0]?.note).toContain("HTTP 202");
expect(summary.outcomes[0]?.note).toContain("AkamaiGHost");
// The page we already hold is still the page.
expect((await store.read("genshin-game8-events"))?.html).toBe(
"<html><event></event></html>",
);
});
test("never retries a failure inside the same cycle", async () => { test("never retries a failure inside the same cycle", async () => {
const { opts, calls } = options({ const { opts, calls } = options({
responder: () => new Response("nope", { status: 500 }), responder: () => new Response("nope", { status: 500 }),