robots: tell a challenged 403 from a refusal

`--assume-robots-on-403` stands in for rules a person read in a browser, for a
host that will not serve us the file. It always claimed it would not override a
host that had turned us away — but it could not tell the two 403s apart, so it
excused both. A managed challenge ("we cannot tell what you are") is a question
our fetcher cannot answer and the operators never asked, which is what makes a
human reading the rules a fair substitute. A bare 403 is the site itself saying
no, and nothing recorded on our side may talk over that.

`isInterstitialChallenge` now decides, on Cloudflare's own `cf-mitigated` header
with the challenge page's markers as a fallback; an unreadable body counts as
unclassifiable rather than challenged. This only ever narrows what the flag
opens, so nothing that passed the gate before stops passing it.

Every 403 is classified whether or not the flag is set, and the kind goes into
the reason string the run reports. `robots.txt returned 403` read identically
whether the answer was to refresh by hand on the recorded permission or to stop
fetching the source, and the summary is where somebody has to decide that.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

AGENTS.md

@@ -119,7 +119,7 @@ src/client/       React app, service worker, manifest
                   theme.ts — dark or light, and what a game hue reads as on each
 scripts/          build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
 serve.ts          static server + /api/health
-test/             732 tests
+test/             772 tests
 fixtures/<game>/  raw HTML + .expected.json per source — pinned, kept forever
 snapshots/        current page per source, rewritten by refresh — see its README
 ```
@@ -395,14 +395,25 @@ fails closed and skips. The permission is therefore a thing a human records once
 is a thing that needs an address Fandom serves.
 
 `--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing
-that helps: `bun run refresh --assume-robots-on-403` treats a `403` **on `/robots.txt` itself** as
-the permission recorded above rather than failing closed. It is not a workaround for a host that
-turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still
-says no, and it does nothing at all for game8.co, whose robots.txt reads fine and welcomes us while
-its edge refuses the pages. It is refused under CI, because what it stands in for is a person having
-read a file in a browser, and there is no person on a runner. Every host it applied to is named in
-the run's warnings, so it stays a thing somebody decided this morning rather than a default. Nothing
-else relaxes: one request per source, six hours apart, spaced per host, no retries.
+that helps: `bun run refresh --assume-robots-on-403` treats **an interstitial challenge** on
+`/robots.txt` itself as the permission recorded above rather than failing closed. It is not a
+workaround for a host that turned us away — it never overrides a `robots.txt` we could read, so a
+file that disallows us still says no, and it does nothing at all for game8.co, whose robots.txt reads
+fine and welcomes us while its edge refuses the pages. It is refused under CI, because what it stands
+in for is a person having read a file in a browser, and there is no person on a runner. Every host it
+applied to is named in the run's warnings, so it stays a thing somebody decided this morning rather
+than a default. Nothing else relaxes: one request per source, six hours apart, spaced per host, no
+retries.
+
+**A `403` is two answers wearing one status code, and only one of them is covered.** A managed
+challenge means "we cannot tell what you are" — the question a human answers by reading the file in a
+browser, which is the whole basis of the concession. A bare `403` means "you are forbidden", and that
+is a host declining us, which no permission recorded on our side may talk over. The flag claimed this
+distinction from the day it was written and could not actually draw it, so it excused both;
+`isInterstitialChallenge` (`src/ingest/robots.ts`) now decides, on Cloudflare's own
+`cf-mitigated: challenge` header with the challenge page's markers as a fallback. A `403` whose body
+cannot be read is unclassifiable, and unclassifiable is not challenged. This only ever *narrows* what
+the flag opens — nothing that passed the gate before stops passing it.
 
 One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
 robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
@@ -579,7 +590,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t
 `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
 opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
 (failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
-described in § Fandom). Both overrides are interactive-only, refused under CI, and reported by name
+described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are interactive-only, refused under CI, and reported by name
 in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
 
 **A source down is a warning; a source down for days is a broken build.** One wiki failing must

docs/INGESTION.md

@@ -277,19 +277,33 @@ section but must never claim the event title.
   cycle is a shorter interval with extra steps. Every source asked early is listed in
   `summary.forced` and warned about; a run that was due anyway is never reported as forced, because a
   summary that cried "forced" on an ordinary run would train the reader to ignore the word.
-- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre
-  address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with
-  a `200`, so the gate fails closed and four sources can never refresh — even though their rules are
-  known, because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct.
-  The flag makes the run proceed on that recorded permission. Three things bound it: it applies to
-  `403` **only** (a 401, a 5xx or a soft 404 are still "we do not know"); it never overrides a
-  `robots.txt` we *could* read, so a file that disallows us still says no; and it is refused under
-  CI, because it stands in for a human and there is none on a runner. Every host it applied to is
-  named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is
-  one nobody withdraws. It changes no other obligation: still one request per source, still six
-  hours apart, still spaced per host.
+- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** For a host that will not serve
+  us `/robots.txt` at all, while the surface we actually read answers our own User-Agent with a
+  `200`, the gate fails closed and the source can never refresh — even though its rules are known,
+  because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct. The flag
+  makes the run proceed on that recorded permission. Four things bound it:
+  - It applies to `403` **only** — a 401, a 5xx or a soft 404 are still "we do not know".
+  - **The `403` must be an interstitial challenge, not a refusal** (`isInterstitialChallenge`). A
+    managed challenge is an edge saying "prove you are a browser", a question our fetcher cannot
+    answer and the site's operators never asked, which is what makes a human reading the rules a fair
+    substitute. A bare `403` is the site itself refusing us, and this flag has never been permitted
+    to talk over that — it claimed as much from the start and could not tell the two apart, so it
+    did. Detection is Cloudflare's own `cf-mitigated: challenge` header, falling back to the
+    challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not
+    excused.
+  - It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no.
+  - It is refused under CI, because it stands in for a human and there is none on a runner.
+
+  Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
+  override that reports nothing is one nobody withdraws. It changes no other obligation: still one
+  request per source, still six hours apart, still spaced per host.
 - 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says
   one per source per cycle. A failed source waits for the next cycle instead.
+- **A `403` on `robots.txt` records which kind it was** — `an interstitial challenge`, `a refusal`,
+  or `unclassifiable` — and names the flag that does or does not cover it. Both fail closed and skip;
+  the difference is what the reader should do about it, and `robots.txt returned 403` said neither. A
+  challenge is a cue to refresh by hand on the recorded permission or move the run to a served
+  address; a refusal is a source to stop fetching.
 - **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole
   2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge
   instead of the wiki. Admitting it fed that challenge page to the parser, which reported "yielded 0

src/ingest/robots.ts

@@ -401,20 +401,64 @@ export class RobotsCache {
       };
     }
 
-    // A host that will not serve us the file at all, only when an operator has
-    // asked for this. `usable: true` with no rules is not a guess about what
-    // the site permits — it is standing in for rules a human read in a browser
-    // and wrote into AGENTS.md. Everything else about being a guest still
-    // applies: one request per source, six hours apart, spaced per host.
-    if (response.status === 403 && this.assumeAllowedWhenForbidden) {
+    // A `403` is two different answers wearing one status code, and the run has
+    // to be able to tell them apart — so classify every one, whether or not an
+    // override is in play. A managed challenge means "we cannot tell what you
+    // are": a question a non-browser cannot answer, and the only situation a
+    // permission recorded by hand stands in for. A plain `403` means "you are
+    // forbidden", which is the host itself declining us.
+    //
+    // Reporting it is not a nicety. INGESTION.md already requires a non-ok
+    // status to record what turned us away, because a bare `HTTP 403` reads
+    // identically whether a CDN decided we are a bot farm or the site said no —
+    // and those two want opposite responses from whoever reads the summary. One
+    // of them is what `--assume-robots-on-403` is for; the other is a source to
+    // stop fetching.
+    if (response.status === 403) {
+      let challenged: boolean | null;
+      try {
+        challenged = await isInterstitialChallenge(response);
+      } catch {
+        // Unreadable body: we cannot say which of the two this was, and
+        // unclassifiable is not challenged.
+        challenged = null;
+      }
+
+      // `usable: true` with no rules is not a guess about what the site
+      // permits — it is standing in for rules a human read in a browser and
+      // wrote into AGENTS.md. Everything else about being a guest still
+      // applies: one request per source, six hours apart, spaced per host.
+      if (challenged === true && this.assumeAllowedWhenForbidden) {
+        return {
+          robots: ALLOW_ALL,
+          usable: true,
+          reason:
+            `robots.txt returned 403 behind an interstitial challenge; ` +
+            `proceeding on a permission recorded by hand ` +
+            `(--assume-robots-on-403)`,
+          at,
+          assumedOnForbidden: true,
+        };
+      }
+
+      const kind =
+        challenged === null
+          ? "unclassifiable"
+          : challenged
+            ? "an interstitial challenge"
+            : "a refusal";
+      const advice =
+        challenged === true && !this.assumeAllowedWhenForbidden
+          ? "; --assume-robots-on-403 covers this on an interactive run"
+          : challenged === false
+            ? "; --assume-robots-on-403 does not cover a host that turned us away"
+            : "";
+
       return {
         robots: ALLOW_ALL,
-        usable: true,
-        reason:
-          `robots.txt returned 403; proceeding on a permission recorded by ` +
-          `hand (--assume-robots-on-403)`,
+        usable: false,
+        reason: `robots.txt returned 403 (${kind})${advice}`,
         at,
-        assumedOnForbidden: true,
       };
     }
 
@@ -461,6 +505,48 @@ export class RobotsCache {
   }
 }
 
+/**
+ * Markers of an edge challenge page, as opposed to a page that says no.
+ *
+ * All Cloudflare's, because Cloudflare is what actually sits in front of the
+ * wikis here. The header is the reliable one — Cloudflare labels its own
+ * mitigations — and the body markers are the fallback for a challenge served
+ * without it.
+ */
+const CHALLENGE_BODY_MARKERS = [
+  "_cf_chl_opt",
+  "/cdn-cgi/challenge-platform/",
+  "cf-browser-verification",
+  "Just a moment...",
+  "Enable JavaScript and cookies to continue",
+] as const;
+
+/**
+ * Is this response an interstitial challenge rather than a refusal?
+ *
+ * The distinction `--assume-robots-on-403` rests on, and it is not cosmetic. A
+ * challenge is an edge saying "prove you are a browser" — a question our
+ * fetcher cannot answer and was never asked by the site's operators, which is
+ * why a human reading the rules in a browser is a fair substitute for reading
+ * them here. A bare `403` is the site itself refusing, and no recorded
+ * permission may talk over that.
+ *
+ * Consumes the body, so call it once and only on a response being classified.
+ */
+export async function isInterstitialChallenge(
+  response: Response,
+): Promise<boolean> {
+  // `cf-mitigated: challenge` is Cloudflare naming what it just did, so it
+  // settles the question without reading the body at all.
+  const mitigated = response.headers.get("cf-mitigated");
+  if (mitigated !== null && mitigated.toLowerCase().includes("challenge")) {
+    return true;
+  }
+
+  const head = (await response.text()).slice(0, 4096);
+  return CHALLENGE_BODY_MARKERS.some((marker) => head.includes(marker));
+}
+
 /**
  * Is this body markup rather than robots.txt?
  *

test/robots.test.ts

@@ -4,6 +4,7 @@ import {
   crawlDelayMs,
   groupFor,
   isAllowed,
+  isInterstitialChallenge,
   parseRobots,
   patternMatches,
   requestTarget,
@@ -353,9 +354,27 @@ describe("--assume-robots-on-403", () => {
    * This option is that recorded permission, and nothing wider. The tests below
    * are mostly about what it must NOT do.
    */
+  /**
+   * A Cloudflare managed challenge, trimmed from a real one served by
+   * `nikke-goddess-of-victory-international.fandom.com/robots.txt` on
+   * 2026-08-19. The markers are the part under test.
+   */
+  const CHALLENGE_BODY =
+    `<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>` +
+    `<meta name="robots" content="noindex,nofollow"></head><body>` +
+    `<noscript>Enable JavaScript and cookies to continue</noscript>` +
+    `<script>window._cf_chl_opt = {cRay: 'a2dc2c7a38c95a94'};` +
+    `a.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1';` +
+    `</script></body></html>`;
+
+  const challenged = () =>
+    new Response(CHALLENGE_BODY, {
+      status: 403,
+      headers: { "cf-mitigated": "challenge" },
+    });
   const forbidden = () => new Response("denied", { status: 403 });
 
-  function cache(assume: boolean, responder = forbidden) {
+  function cache(assume: boolean, responder = challenged) {
     return new RobotsCache({
       userAgent: UA,
       fetchImpl: async () => responder(),
@@ -369,6 +388,25 @@ describe("--assume-robots-on-403", () => {
     expect(d.reason).toContain("403");
   });
 
+  /**
+   * The run has to be able to tell the two 403s apart from the summary alone.
+   * A challenge is somebody's cue to refresh by hand on the recorded
+   * permission; a refusal is a source to stop fetching. `robots.txt returned
+   * 403` said neither, and the ambiguity cost a day of wrong conclusions about
+   * Fandom.
+   */
+  test("says which kind of 403 it was, with no override in play", async () => {
+    const chal = await cache(false).allows("https://x.fandom.com/api.php");
+    expect(chal.allowed).toBe(false);
+    expect(chal.reason).toContain("interstitial challenge");
+    expect(chal.reason).toContain("--assume-robots-on-403 covers this");
+
+    const deny = await cache(false, forbidden).allows("https://x.test/wiki/E");
+    expect(deny.allowed).toBe(false);
+    expect(deny.reason).toContain("refusal");
+    expect(deny.reason).toContain("does not cover");
+  });
+
   test("with it, the host is fetched and the run is told why", async () => {
     const d = await cache(true).allows("https://x.fandom.com/api.php?action=parse");
     expect(d.allowed).toBe(true);
@@ -403,7 +441,65 @@ describe("--assume-robots-on-403", () => {
   });
 
   test("is off unless asked for", async () => {
-    const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => forbidden() });
+    const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => challenged() });
     expect((await plain.allows("https://x.test/a")).allowed).toBe(false);
   });
+
+  /**
+   * The narrowing this option always claimed and did not have. A 403 is two
+   * different answers wearing one status code: an edge challenge is "we cannot
+   * tell what you are", which a human reading the rules in a browser genuinely
+   * answers, while a bare 403 is the site refusing us. Before this, both looked
+   * like Fandom's and the flag talked over the second one too.
+   */
+  test("covers a challenge, never a plain refusal", async () => {
+    const d = await cache(true, forbidden).allows("https://x.test/wiki/Event");
+    expect(d.allowed).toBe(false);
+    expect(d.assumedOnForbidden).toBeUndefined();
+    expect(d.reason).toContain("refusal");
+  });
+
+  test("a 403 whose body cannot be read is not excused", async () => {
+    // Unclassifiable is not the same as challenged, and only one of them is
+    // covered by a permission somebody recorded.
+    const torn = () =>
+      new Response(
+        new ReadableStream({
+          start(c) {
+            c.error(new Error("connection reset"));
+          },
+        }),
+        { status: 403 },
+      );
+    const d = await cache(true, torn).allows("https://x.test/wiki/Event");
+    expect(d.allowed).toBe(false);
+    expect(d.assumedOnForbidden).toBeUndefined();
+    expect(d.reason).toContain("unclassifiable");
+  });
+
+  describe("isInterstitialChallenge", () => {
+    test("trusts cf-mitigated, without reading the body", async () => {
+      const r = new Response("", {
+        status: 403,
+        headers: { "cf-mitigated": "challenge" },
+      });
+      expect(await isInterstitialChallenge(r)).toBe(true);
+      // The header settled it, so the body is still there to be read.
+      expect(r.bodyUsed).toBe(false);
+    });
+
+    test("falls back to the challenge page's own markers", async () => {
+      const r = new Response(CHALLENGE_BODY, { status: 403 });
+      expect(await isInterstitialChallenge(r)).toBe(true);
+    });
+
+    test("a refusal is not a challenge", async () => {
+      for (const body of ["denied", "Forbidden", ""]) {
+        const r = new Response(body, { status: 403 });
+        expect(`${body}: ${await isInterstitialChallenge(r)}`).toBe(
+          `${body}: false`,
+        );
+      }
+    });
+  });
 });