diff --git a/CLAUDE.md b/CLAUDE.md index 0773a79..261f456 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -193,6 +193,19 @@ AI-training crawlers. Our use is a low-rate personal aggregator with attribution training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch rate, and do not add an LLM that consumes page content. +**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns +`202 Accepted` with a bot-management body to every one of the eight game8 sources, from the first +scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse +cleanly from a normal address. So `robots.txt` permits us and the network does not, and those eight +games have only ever been built from checked-in fixtures in CI. + +The per-host spacing above does not fix this and was not meant to: a 202 on the very first request +of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a +proxy, or a residential egress would each be defeating a deliberate access control, which is the +same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose +`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the +refresh from an address game8 will serve, or to find those games another source. + A source whose ToS forbids automated access does not get an adapter. Flag it and ask. **Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass: diff --git a/docs/INGESTION.md b/docs/INGESTION.md index e9db31a..2ad2654 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -177,6 +177,11 @@ section but must never claim the event title. permission we have. A 404 means no restrictions. - 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says one per source per cycle. A failed source waits for the next cycle instead. +- **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole + 2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge + instead of the wiki. Admitting it fed that challenge page to the parser, which reported "yielded 0 + events" — the symptom, with the status that explained it unmentioned. `204` has no body and `206` + is a fragment; none of them is a document. - **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates, so a source we then skip costs nothing. diff --git a/scripts/refresh-sources.ts b/scripts/refresh-sources.ts index 8f1b1a0..8eba7d3 100644 --- a/scripts/refresh-sources.ts +++ b/scripts/refresh-sources.ts @@ -346,7 +346,15 @@ async function refreshOne( }; } - if (!response.ok) { + // 200 is the only status that means "here is the page". `response.ok` also + // admits the rest of the 2xx range, and that cost us the diagnosis: game8.co's + // edge answers a GitHub runner with **202 Accepted** and a bot-management + // body, which sailed through this gate as a document, reached the parser, + // yielded no events and was reported as "kept previous snapshot" — describing + // the symptom while the status that explained it went unmentioned. 202 means + // the request was accepted for processing; 204 has no body and 206 is a + // fragment. None of them is a wiki page. + if (response.status !== 200) { await store.recordCheck(adapter.id, { at: nowIso, status: response.status, diff --git a/test/refresh.test.ts b/test/refresh.test.ts index 457ccc9..7537514 100644 --- a/test/refresh.test.ts +++ b/test/refresh.test.ts @@ -214,6 +214,30 @@ describe("one request per source per six hours", () => { expect(calls).toHaveLength(1); }); + test("a 2xx that is not 200 is not a page", async () => { + // game8.co's edge answers a GitHub runner with 202 and a bot-management + // body. `response.ok` admitted it, so it reached the parser and was reported + // as "yielded 0 events" — the symptom, while the status that explained it + // was never named. Six sources read that way for days. + await seed("", "2026-08-01T00:00:00.000Z", 1); + const { opts } = options({ + responder: () => + new Response("checking your browser", { + status: 202, + headers: { Server: "AkamaiGHost" }, + }), + }); + const summary = await runRefresh(opts); + + expect(summary.outcomes[0]?.result).toBe("failed"); + expect(summary.outcomes[0]?.note).toContain("HTTP 202"); + expect(summary.outcomes[0]?.note).toContain("AkamaiGHost"); + // The page we already hold is still the page. + expect((await store.read("genshin-game8-events"))?.html).toBe( + "", + ); + }); + test("never retries a failure inside the same cycle", async () => { const { opts, calls } = options({ responder: () => new Response("nope", { status: 500 }),