diff --git a/AGENTS.md b/AGENTS.md index da5fb11..99dd1a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest theme.ts — dark or light, and what a game hue reads as on each scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 732 tests +test/ 772 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -395,14 +395,25 @@ fails closed and skips. The permission is therefore a thing a human records once is a thing that needs an address Fandom serves. `--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing -that helps: `bun run refresh --assume-robots-on-403` treats a `403` **on `/robots.txt` itself** as -the permission recorded above rather than failing closed. It is not a workaround for a host that -turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still -says no, and it does nothing at all for game8.co, whose robots.txt reads fine and welcomes us while -its edge refuses the pages. It is refused under CI, because what it stands in for is a person having -read a file in a browser, and there is no person on a runner. Every host it applied to is named in -the run's warnings, so it stays a thing somebody decided this morning rather than a default. Nothing -else relaxes: one request per source, six hours apart, spaced per host, no retries. +that helps: `bun run refresh --assume-robots-on-403` treats **an interstitial challenge** on +`/robots.txt` itself as the permission recorded above rather than failing closed. It is not a +workaround for a host that turned us away — it never overrides a `robots.txt` we could read, so a +file that disallows us still says no, and it does nothing at all for game8.co, whose robots.txt reads +fine and welcomes us while its edge refuses the pages. It is refused under CI, because what it stands +in for is a person having read a file in a browser, and there is no person on a runner. Every host it +applied to is named in the run's warnings, so it stays a thing somebody decided this morning rather +than a default. Nothing else relaxes: one request per source, six hours apart, spaced per host, no +retries. + +**A `403` is two answers wearing one status code, and only one of them is covered.** A managed +challenge means "we cannot tell what you are" — the question a human answers by reading the file in a +browser, which is the whole basis of the concession. A bare `403` means "you are forbidden", and that +is a host declining us, which no permission recorded on our side may talk over. The flag claimed this +distinction from the day it was written and could not actually draw it, so it excused both; +`isInterstitialChallenge` (`src/ingest/robots.ts`) now decides, on Cloudflare's own +`cf-mitigated: challenge` header with the challenge page's markers as a fallback. A `403` whose body +cannot be read is unclassifiable, and unclassifiable is not challenged. This only ever *narrows* what +the flag opens — nothing that passed the gate before stops passing it. One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the robots gate **fails closed there and the source is skipped**. That is a warning line rather than a @@ -579,7 +590,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403` -described in § Fandom). Both overrides are interactive-only, refused under CI, and reported by name +described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are interactive-only, refused under CI, and reported by name in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first. **A source down is a warning; a source down for days is a broken build.** One wiki failing must diff --git a/docs/INGESTION.md b/docs/INGESTION.md index a2f72c4..29b9315 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -277,19 +277,33 @@ section but must never claim the event title. cycle is a shorter interval with extra steps. Every source asked early is listed in `summary.forced` and warned about; a run that was due anyway is never reported as forced, because a summary that cried "forced" on an ordinary run would train the reader to ignore the word. -- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre - address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with - a `200`, so the gate fails closed and four sources can never refresh — even though their rules are - known, because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct. - The flag makes the run proceed on that recorded permission. Three things bound it: it applies to - `403` **only** (a 401, a 5xx or a soft 404 are still "we do not know"); it never overrides a - `robots.txt` we *could* read, so a file that disallows us still says no; and it is refused under - CI, because it stands in for a human and there is none on a runner. Every host it applied to is - named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is - one nobody withdraws. It changes no other obligation: still one request per source, still six - hours apart, still spaced per host. +- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** For a host that will not serve + us `/robots.txt` at all, while the surface we actually read answers our own User-Agent with a + `200`, the gate fails closed and the source can never refresh — even though its rules are known, + because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct. The flag + makes the run proceed on that recorded permission. Four things bound it: + - It applies to `403` **only** — a 401, a 5xx or a soft 404 are still "we do not know". + - **The `403` must be an interstitial challenge, not a refusal** (`isInterstitialChallenge`). A + managed challenge is an edge saying "prove you are a browser", a question our fetcher cannot + answer and the site's operators never asked, which is what makes a human reading the rules a fair + substitute. A bare `403` is the site itself refusing us, and this flag has never been permitted + to talk over that — it claimed as much from the start and could not tell the two apart, so it + did. Detection is Cloudflare's own `cf-mitigated: challenge` header, falling back to the + challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not + excused. + - It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no. + - It is refused under CI, because it stands in for a human and there is none on a runner. + + Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an + override that reports nothing is one nobody withdraws. It changes no other obligation: still one + request per source, still six hours apart, still spaced per host. - 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says one per source per cycle. A failed source waits for the next cycle instead. +- **A `403` on `robots.txt` records which kind it was** — `an interstitial challenge`, `a refusal`, + or `unclassifiable` — and names the flag that does or does not cover it. Both fail closed and skip; + the difference is what the reader should do about it, and `robots.txt returned 403` said neither. A + challenge is a cue to refresh by hand on the recorded permission or move the run to a served + address; a refusal is a source to stop fetching. - **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole 2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge instead of the wiki. Admitting it fed that challenge page to the parser, which reported "yielded 0 diff --git a/src/ingest/robots.ts b/src/ingest/robots.ts index 9bf70cd..40b2526 100644 --- a/src/ingest/robots.ts +++ b/src/ingest/robots.ts @@ -401,20 +401,64 @@ export class RobotsCache { }; } - // A host that will not serve us the file at all, only when an operator has - // asked for this. `usable: true` with no rules is not a guess about what - // the site permits — it is standing in for rules a human read in a browser - // and wrote into AGENTS.md. Everything else about being a guest still - // applies: one request per source, six hours apart, spaced per host. - if (response.status === 403 && this.assumeAllowedWhenForbidden) { + // A `403` is two different answers wearing one status code, and the run has + // to be able to tell them apart — so classify every one, whether or not an + // override is in play. A managed challenge means "we cannot tell what you + // are": a question a non-browser cannot answer, and the only situation a + // permission recorded by hand stands in for. A plain `403` means "you are + // forbidden", which is the host itself declining us. + // + // Reporting it is not a nicety. INGESTION.md already requires a non-ok + // status to record what turned us away, because a bare `HTTP 403` reads + // identically whether a CDN decided we are a bot farm or the site said no — + // and those two want opposite responses from whoever reads the summary. One + // of them is what `--assume-robots-on-403` is for; the other is a source to + // stop fetching. + if (response.status === 403) { + let challenged: boolean | null; + try { + challenged = await isInterstitialChallenge(response); + } catch { + // Unreadable body: we cannot say which of the two this was, and + // unclassifiable is not challenged. + challenged = null; + } + + // `usable: true` with no rules is not a guess about what the site + // permits — it is standing in for rules a human read in a browser and + // wrote into AGENTS.md. Everything else about being a guest still + // applies: one request per source, six hours apart, spaced per host. + if (challenged === true && this.assumeAllowedWhenForbidden) { + return { + robots: ALLOW_ALL, + usable: true, + reason: + `robots.txt returned 403 behind an interstitial challenge; ` + + `proceeding on a permission recorded by hand ` + + `(--assume-robots-on-403)`, + at, + assumedOnForbidden: true, + }; + } + + const kind = + challenged === null + ? "unclassifiable" + : challenged + ? "an interstitial challenge" + : "a refusal"; + const advice = + challenged === true && !this.assumeAllowedWhenForbidden + ? "; --assume-robots-on-403 covers this on an interactive run" + : challenged === false + ? "; --assume-robots-on-403 does not cover a host that turned us away" + : ""; + return { robots: ALLOW_ALL, - usable: true, - reason: - `robots.txt returned 403; proceeding on a permission recorded by ` + - `hand (--assume-robots-on-403)`, + usable: false, + reason: `robots.txt returned 403 (${kind})${advice}`, at, - assumedOnForbidden: true, }; } @@ -461,6 +505,48 @@ export class RobotsCache { } } +/** + * Markers of an edge challenge page, as opposed to a page that says no. + * + * All Cloudflare's, because Cloudflare is what actually sits in front of the + * wikis here. The header is the reliable one — Cloudflare labels its own + * mitigations — and the body markers are the fallback for a challenge served + * without it. + */ +const CHALLENGE_BODY_MARKERS = [ + "_cf_chl_opt", + "/cdn-cgi/challenge-platform/", + "cf-browser-verification", + "Just a moment...", + "Enable JavaScript and cookies to continue", +] as const; + +/** + * Is this response an interstitial challenge rather than a refusal? + * + * The distinction `--assume-robots-on-403` rests on, and it is not cosmetic. A + * challenge is an edge saying "prove you are a browser" — a question our + * fetcher cannot answer and was never asked by the site's operators, which is + * why a human reading the rules in a browser is a fair substitute for reading + * them here. A bare `403` is the site itself refusing, and no recorded + * permission may talk over that. + * + * Consumes the body, so call it once and only on a response being classified. + */ +export async function isInterstitialChallenge( + response: Response, +): Promise { + // `cf-mitigated: challenge` is Cloudflare naming what it just did, so it + // settles the question without reading the body at all. + const mitigated = response.headers.get("cf-mitigated"); + if (mitigated !== null && mitigated.toLowerCase().includes("challenge")) { + return true; + } + + const head = (await response.text()).slice(0, 4096); + return CHALLENGE_BODY_MARKERS.some((marker) => head.includes(marker)); +} + /** * Is this body markup rather than robots.txt? * diff --git a/test/robots.test.ts b/test/robots.test.ts index 7081a9f..a0c27fa 100644 --- a/test/robots.test.ts +++ b/test/robots.test.ts @@ -4,6 +4,7 @@ import { crawlDelayMs, groupFor, isAllowed, + isInterstitialChallenge, parseRobots, patternMatches, requestTarget, @@ -353,9 +354,27 @@ describe("--assume-robots-on-403", () => { * This option is that recorded permission, and nothing wider. The tests below * are mostly about what it must NOT do. */ + /** + * A Cloudflare managed challenge, trimmed from a real one served by + * `nikke-goddess-of-victory-international.fandom.com/robots.txt` on + * 2026-08-19. The markers are the part under test. + */ + const CHALLENGE_BODY = + `Just a moment...` + + `` + + `` + + ``; + + const challenged = () => + new Response(CHALLENGE_BODY, { + status: 403, + headers: { "cf-mitigated": "challenge" }, + }); const forbidden = () => new Response("denied", { status: 403 }); - function cache(assume: boolean, responder = forbidden) { + function cache(assume: boolean, responder = challenged) { return new RobotsCache({ userAgent: UA, fetchImpl: async () => responder(), @@ -369,6 +388,25 @@ describe("--assume-robots-on-403", () => { expect(d.reason).toContain("403"); }); + /** + * The run has to be able to tell the two 403s apart from the summary alone. + * A challenge is somebody's cue to refresh by hand on the recorded + * permission; a refusal is a source to stop fetching. `robots.txt returned + * 403` said neither, and the ambiguity cost a day of wrong conclusions about + * Fandom. + */ + test("says which kind of 403 it was, with no override in play", async () => { + const chal = await cache(false).allows("https://x.fandom.com/api.php"); + expect(chal.allowed).toBe(false); + expect(chal.reason).toContain("interstitial challenge"); + expect(chal.reason).toContain("--assume-robots-on-403 covers this"); + + const deny = await cache(false, forbidden).allows("https://x.test/wiki/E"); + expect(deny.allowed).toBe(false); + expect(deny.reason).toContain("refusal"); + expect(deny.reason).toContain("does not cover"); + }); + test("with it, the host is fetched and the run is told why", async () => { const d = await cache(true).allows("https://x.fandom.com/api.php?action=parse"); expect(d.allowed).toBe(true); @@ -403,7 +441,65 @@ describe("--assume-robots-on-403", () => { }); test("is off unless asked for", async () => { - const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => forbidden() }); + const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => challenged() }); expect((await plain.allows("https://x.test/a")).allowed).toBe(false); }); + + /** + * The narrowing this option always claimed and did not have. A 403 is two + * different answers wearing one status code: an edge challenge is "we cannot + * tell what you are", which a human reading the rules in a browser genuinely + * answers, while a bare 403 is the site refusing us. Before this, both looked + * like Fandom's and the flag talked over the second one too. + */ + test("covers a challenge, never a plain refusal", async () => { + const d = await cache(true, forbidden).allows("https://x.test/wiki/Event"); + expect(d.allowed).toBe(false); + expect(d.assumedOnForbidden).toBeUndefined(); + expect(d.reason).toContain("refusal"); + }); + + test("a 403 whose body cannot be read is not excused", async () => { + // Unclassifiable is not the same as challenged, and only one of them is + // covered by a permission somebody recorded. + const torn = () => + new Response( + new ReadableStream({ + start(c) { + c.error(new Error("connection reset")); + }, + }), + { status: 403 }, + ); + const d = await cache(true, torn).allows("https://x.test/wiki/Event"); + expect(d.allowed).toBe(false); + expect(d.assumedOnForbidden).toBeUndefined(); + expect(d.reason).toContain("unclassifiable"); + }); + + describe("isInterstitialChallenge", () => { + test("trusts cf-mitigated, without reading the body", async () => { + const r = new Response("", { + status: 403, + headers: { "cf-mitigated": "challenge" }, + }); + expect(await isInterstitialChallenge(r)).toBe(true); + // The header settled it, so the body is still there to be read. + expect(r.bodyUsed).toBe(false); + }); + + test("falls back to the challenge page's own markers", async () => { + const r = new Response(CHALLENGE_BODY, { status: 403 }); + expect(await isInterstitialChallenge(r)).toBe(true); + }); + + test("a refusal is not a challenge", async () => { + for (const body of ["denied", "Forbidden", ""]) { + const r = new Response(body, { status: 403 }); + expect(`${body}: ${await isInterstitialChallenge(r)}`).toBe( + `${body}: false`, + ); + } + }); + }); });