feat(refresh): --assume-robots-on-403, for a host that hides only its robots.txt

Fandom answers our fetcher 403 on /robots.txt itself, from every address we
have, while api.php?action=parse answers the same User-Agent with a 200. The
gate fails closed on the unreadable file, so r1999, fgo, nikke and nikki can
never refresh — even though their rules are not unknown: a person read them in
a browser and wrote them into AGENTS.md, verbatim, which is how three of those
four were cleared in the first place.

This flag is that recorded permission, and nothing wider. What it deliberately
does not do is most of the design:

  - 403 only. A 401, a 5xx, a timeout or a soft 404 still mean we do not know
    what the site permits, and unknown is still not permission.
  - It never overrides a robots.txt we could read. A file that answers and
    disallows us is an answer, and it still wins. So this is no use for game8.co
    — whose robots.txt reads fine and welcomes us while its edge refuses the
    pages — and it must not become one.
  - Refused under CI. It stands in for a human having read a file this morning,
    and there is no human on a runner. A scheduled job asserting a permission
    nobody re-checked is exactly how "temporary" stops being temporary.
  - Loud. Every host it applied to is warned about by name, in the run log and
    in the summary, with a line saying to go re-read the file. An override that
    reports nothing is one nobody withdraws.

Nothing else about being a guest relaxes: one request per source, six hours
apart, spaced per host, conditional headers, no retries.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

AGENTS.md

@@ -119,7 +119,7 @@ src/client/       React app, service worker, manifest
                   theme.ts — dark or light, and what a game hue reads as on each
 scripts/          build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
 serve.ts          static server + /api/health
-test/             716 tests
+test/             724 tests
 fixtures/<game>/  raw HTML + .expected.json per source — pinned, kept forever
 snapshots/        current page per source, rewritten by refresh — see its README
 ```
@@ -385,6 +385,16 @@ fixture-backed* from any address; what it cannot do is pass the robots gate at r
 fails closed and skips. The permission is therefore a thing a human records once, and the freshness
 is a thing that needs an address Fandom serves.
 
+`--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing
+that helps: `bun run refresh --assume-robots-on-403` treats a `403` **on `/robots.txt` itself** as
+the permission recorded above rather than failing closed. It is not a workaround for a host that
+turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still
+says no, and it does nothing at all for game8.co, whose robots.txt reads fine and welcomes us while
+its edge refuses the pages. It is refused under CI, because what it stands in for is a person having
+read a file in a browser, and there is no person on a runner. Every host it applied to is named in
+the run's warnings, so it stays a thing somebody decided this morning rather than a default. Nothing
+else relaxes: one request per source, six hours apart, spaced per host, no retries.
+
 One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
 robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
 broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if
@@ -559,7 +569,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t
 
 `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
 retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
-read). Anything that would make it fetch more often is a change to this section first.
+read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first.
 
 **A source down is a warning; a source down for days is a broken build.** One wiki failing must
 never blank a calendar or stop the sources that did answer from being committed — so a failure is

docs/INGESTION.md

@@ -270,6 +270,17 @@ section but must never claim the event title.
 - Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
   5xxs or times out means "do not fetch", because a permission we could not read is not a
   permission we have. A 404 means no restrictions.
+- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre
+  address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with
+  a `200`, so the gate fails closed and four sources can never refresh — even though their rules are
+  known, because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct.
+  The flag makes the run proceed on that recorded permission. Three things bound it: it applies to
+  `403` **only** (a 401, a 5xx or a soft 404 are still "we do not know"); it never overrides a
+  `robots.txt` we *could* read, so a file that disallows us still says no; and it is refused under
+  CI, because it stands in for a human and there is none on a runner. Every host it applied to is
+  named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is
+  one nobody withdraws. It changes no other obligation: still one request per source, still six
+  hours apart, still spaced per host.
 - 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says
   one per source per cycle. A failed source waits for the next cycle instead.
 - **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole

scripts/refresh-sources.ts

@@ -4,12 +4,18 @@
  *   bun run refresh                          # the real thing
  *   bun run refresh --dry-run                # plan only, no requests, no writes
  *   bun run refresh --only genshin-game8-events
+ *   bun run refresh --assume-robots-on-403   # see § the flag, below
  *
  * This is the scheduled half of the pipeline (docs/INGESTION.md stages 1-2).
  * The rules it enforces are etiquette obligations, not preferences:
  *
  *   - robots.txt is read once per host per run and obeyed; unreadable means
- *     "do not fetch", never "assume yes".
+ *     "do not fetch", never "assume yes". `--assume-robots-on-403` is the one
+ *     opt-in exception, for a host that will not serve us the file at all while
+ *     serving us the page: it stands in for rules a person read in a browser
+ *     and wrote into AGENTS.md. It covers 403 only, never overrides a
+ *     robots.txt we could read, is refused under CI, and names every host it
+ *     touched. Nothing else about being a guest relaxes with it.
  *   - at most ONE request per source per cycle, and never sooner than six hours
  *     after the last attempt. There is deliberately no retry: a retry is a
  *     second request, and the next cycle is minutes-cheap compared to being a
@@ -95,6 +101,11 @@ export interface RefreshSummary {
   broken: BrokenSource[];
   /** Set when the run should exit non-zero. */
   hardFailure: string | null;
+  /**
+   * Hosts fetched under `--assume-robots-on-403`. Empty on every normal run,
+   * and on every CI run — the flag is refused there.
+   */
+  assumedRobots: string[];
 }
 
 export interface RobotsGate {
@@ -103,6 +114,8 @@ export interface RobotsGate {
     reason: string;
     /** From the host's `Crawl-delay`, when it states one. */
     crawlDelayMs?: number | null;
+    /** True when `--assume-robots-on-403` is what opened this host. */
+    assumedOnForbidden?: boolean;
   }>;
 }
 
@@ -148,6 +161,8 @@ export const DEFAULT_HOST_GAP_MS = 2_000;
 /** Per-cycle state shared across sources: which hosts we have already asked. */
 interface Cycle {
   requestedHosts: Set<string>;
+  /** Hosts fetched on `--assume-robots-on-403` rather than on a file we read. */
+  assumedRobots: Set<string>;
 }
 
 export async function runRefresh(
@@ -161,9 +176,10 @@ export async function runRefresh(
     warnings: [],
     broken: [],
     hardFailure: null,
+    assumedRobots: [],
   };
 
-  const cycle: Cycle = { requestedHosts: new Set() };
+  const cycle: Cycle = { requestedHosts: new Set(), assumedRobots: new Set() };
 
   const selected =
     options.only === null
@@ -230,6 +246,15 @@ export async function runRefresh(
     }
   }
 
+  // Named in the summary rather than only in the per-source log, so it survives
+  // into the job summary and cannot be scrolled past.
+  summary.assumedRobots = [...cycle.assumedRobots].sort();
+  for (const host of summary.assumedRobots) {
+    summary.warnings.push(
+      `${host}: fetched on --assume-robots-on-403; its robots.txt was not read this run`,
+    );
+  }
+
   // Every source failing is not "a wiki is down", it is us: no network, a bad
   // User-Agent, a proxy. That should stop the pipeline rather than look green.
   if (summary.attempted > 0 && summary.confirmed === 0) {
@@ -303,6 +328,12 @@ async function refreshOne(
       eventCount: meta?.eventCount ?? null,
     };
   }
+  // Fetching on a permission nobody can re-read is a thing the run has to say
+  // out loud, every time and per source. An override that reports nothing is an
+  // override that quietly becomes the default.
+  if (decision.assumedOnForbidden === true) {
+    cycle.assumedRobots.add(new URL(adapter.url).host);
+  }
 
   // Space requests to a host we have already asked this cycle. This sits after
   // the interval and robots gates on purpose: waiting on behalf of a source we
@@ -608,6 +639,21 @@ export async function rebuildFeedViaScript(): Promise<void> {
   if (code !== 0) throw new Error(`build-feed exited ${code}`);
 }
 
+/**
+ * Are we on a runner rather than at somebody's keyboard?
+ *
+ * Both variables, because `CI` is the convention every runner sets and
+ * `GITHUB_ACTIONS` is the one this repo's workflow guarantees. Erring towards
+ * "yes" is the safe direction: the only thing it costs is refusing an
+ * interactive-only flag to a human whose shell exports `CI`.
+ */
+function isCi(): boolean {
+  return (
+    process.env["CI"] !== undefined && process.env["CI"] !== "" ||
+    process.env["GITHUB_ACTIONS"] === "true"
+  );
+}
+
 interface Args {
   dryRun: boolean;
   only: string | null;
@@ -615,6 +661,8 @@ interface Args {
   userAgent: string;
   rebuild: boolean;
   help: boolean;
+  /** See `--assume-robots-on-403` in USAGE, and § Scraping conduct. */
+  assumeRobotsOn403: boolean;
 }
 
 export function parseArgs(argv: readonly string[]): Args {
@@ -625,6 +673,7 @@ export function parseArgs(argv: readonly string[]): Args {
     userAgent: process.env["REFRESH_USER_AGENT"] ?? DEFAULT_USER_AGENT,
     rebuild: true,
     help: false,
+    assumeRobotsOn403: false,
   };
 
   // A flag whose value is missing is a mistake, never a default. `--only` with
@@ -659,6 +708,9 @@ export function parseArgs(argv: readonly string[]): Args {
       case "--no-feed":
         args.rebuild = false;
         break;
+      case "--assume-robots-on-403":
+        args.assumeRobotsOn403 = true;
+        break;
       case "--help":
       case "-h":
         args.help = true;
@@ -675,13 +727,19 @@ export function parseArgs(argv: readonly string[]): Args {
 }
 
 const USAGE = `usage: bun run refresh [--dry-run] [--only <sourceId>] [--snapshots <dir>]
-                       [--user-agent <ua>] [--no-feed]
+                       [--user-agent <ua>] [--no-feed] [--assume-robots-on-403]
 
   --dry-run       report what each source would do; no requests, no writes
   --only <id>     refresh a single source (${ADAPTERS.map((a) => a.id).join(", ")})
   --snapshots     snapshot cache directory (default: snapshots, env SNAPSHOT_DIR)
   --user-agent    override the User-Agent (env REFRESH_USER_AGENT)
-  --no-feed       skip regenerating public/data/events.v1.json`;
+  --no-feed       skip regenerating public/data/events.v1.json
+  --assume-robots-on-403
+                  temporary, interactive-only. When a host answers 403 to
+                  /robots.txt itself, proceed on the permission recorded in
+                  AGENTS.md instead of failing closed. Refused under CI.
+                  Does NOT override a robots.txt we could read: a file that
+                  disallows us still says no.`;
 
 async function main(): Promise<number> {
   let args: Args;
@@ -704,12 +762,35 @@ async function main(): Promise<number> {
     return 2;
   }
 
+  // The flag stands in for a human having read a robots.txt in a browser and
+  // written it down. There is no human on a runner, and a scheduled job quietly
+  // asserting a permission nobody re-checked is how "temporary" becomes
+  // permanent — so CI is refused the option outright rather than trusted not to
+  // pass it. AGENTS.md § Scraping conduct is the argument.
+  if (args.assumeRobotsOn403 && isCi()) {
+    console.error(
+      "--assume-robots-on-403 is interactive-only and refused under CI.\n" +
+        "It asserts a permission a person read by hand; run the refresh from a " +
+        "machine the host serves instead.",
+    );
+    return 2;
+  }
+
   const store = new SnapshotStore(args.root);
   const robots = new RobotsCache({
     userAgent: args.userAgent,
     fetchImpl: (input, init) => fetch(input, init),
+    assumeAllowedWhenForbidden: args.assumeRobotsOn403,
   });
 
+  if (args.assumeRobotsOn403) {
+    console.warn(
+      "  ! --assume-robots-on-403: a host answering 403 to /robots.txt will be\n" +
+        "    fetched anyway, on the permission recorded in AGENTS.md. Temporary,\n" +
+        "    and every host it applies to is named at the end of this run.",
+    );
+  }
+
   console.log(
     `refresh: ${args.only ?? `${ADAPTERS.length} sources`}${args.dryRun ? " (dry run)" : ""}`,
   );
@@ -737,6 +818,12 @@ async function main(): Promise<number> {
       `${summary.warnings.length} warnings, ${summary.broken.length} broken`,
   );
   for (const warning of summary.warnings) console.warn(`  ! ${warning}`);
+  for (const host of summary.assumedRobots) {
+    console.warn(
+      `  ! ${host}: robots.txt was NOT read this run. Re-read it in a browser ` +
+        `and confirm AGENTS.md § Scraping conduct still describes it.`,
+    );
+  }
   for (const b of summary.broken) {
     console.error(
       `  !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` +

src/ingest/robots.ts

@@ -267,6 +267,12 @@ export interface RobotsDecision {
   /** Human-readable why, for the run log. */
   readonly reason: string;
   readonly crawlDelayMs: number | null;
+  /**
+   * True when this host was opened by `assumeAllowedWhenForbidden` rather than
+   * by a robots.txt we actually read. The caller is expected to say so out
+   * loud — an override nobody sees is an override nobody withdraws.
+   */
+  readonly assumedOnForbidden?: boolean;
 }
 
 export interface RobotsCacheOptions {
@@ -276,6 +282,20 @@ export interface RobotsCacheOptions {
   ttlMs?: number;
   now?: () => number;
   timeoutMs?: number;
+  /**
+   * Treat a `403` on **robots.txt itself** as permission, instead of failing
+   * closed. Off by default and never set from CI — see `--assume-robots-on-403`
+   * in `scripts/refresh-sources.ts` for the whole argument.
+   *
+   * The narrowness is the point. This covers exactly one situation: a host that
+   * will not serve us `/robots.txt` from this address, whose rules a human has
+   * therefore read in a browser and written down (AGENTS.md § Scraping conduct
+   * records Fandom's, verbatim). It does **not** touch a robots.txt we did read
+   * and that disallows us — `isAllowed` still decides that, and still says no.
+   * A file we can read and that refuses us is an answer; this is the case where
+   * there is no answer and one has been obtained by hand.
+   */
+  assumeAllowedWhenForbidden?: boolean;
 }
 
 interface CacheEntry {
@@ -284,6 +304,8 @@ interface CacheEntry {
   usable: boolean;
   reason: string;
   at: number;
+  /** True when `assumeAllowedWhenForbidden` is what made this entry usable. */
+  assumedOnForbidden?: boolean;
 }
 
 const DAY_MS = 24 * 60 * 60 * 1000;
@@ -304,6 +326,7 @@ export class RobotsCache {
   private readonly ttlMs: number;
   private readonly nowMs: () => number;
   private readonly timeoutMs: number;
+  private readonly assumeAllowedWhenForbidden: boolean;
 
   constructor(options: RobotsCacheOptions) {
     this.userAgent = options.userAgent;
@@ -311,6 +334,8 @@ export class RobotsCache {
     this.ttlMs = options.ttlMs ?? DAY_MS;
     this.nowMs = options.now ?? (() => Date.now());
     this.timeoutMs = options.timeoutMs ?? 20_000;
+    this.assumeAllowedWhenForbidden =
+      options.assumeAllowedWhenForbidden ?? false;
   }
 
   /** Number of robots.txt requests made, for tests and the run log. */
@@ -329,6 +354,9 @@ export class RobotsCache {
       allowed,
       reason: allowed ? entry.reason : `disallowed by ${origin}/robots.txt`,
       crawlDelayMs: crawlDelayMs(entry.robots, this.userAgent),
+      ...(entry.assumedOnForbidden === true
+        ? { assumedOnForbidden: true }
+        : {}),
     };
   }
 
@@ -373,6 +401,23 @@ export class RobotsCache {
       };
     }
 
+    // A host that will not serve us the file at all, only when an operator has
+    // asked for this. `usable: true` with no rules is not a guess about what
+    // the site permits — it is standing in for rules a human read in a browser
+    // and wrote into AGENTS.md. Everything else about being a guest still
+    // applies: one request per source, six hours apart, spaced per host.
+    if (response.status === 403 && this.assumeAllowedWhenForbidden) {
+      return {
+        robots: ALLOW_ALL,
+        usable: true,
+        reason:
+          `robots.txt returned 403; proceeding on a permission recorded by ` +
+          `hand (--assume-robots-on-403)`,
+        at,
+        assumedOnForbidden: true,
+      };
+    }
+
     if (response.status >= 400) {
       return {
         robots: ALLOW_ALL,

test/refresh.test.ts

@@ -263,6 +263,34 @@ describe("robots", () => {
     expect(summary.warnings).toHaveLength(1);
   });
 
+  test("fetching on an assumed robots permission is named in the summary", async () => {
+    // The override cannot be silent: a permission nobody can re-read is one
+    // nobody withdraws, so every host it applied to is reported by name and
+    // warned about, on a run that otherwise looks completely ordinary.
+    const { opts } = options({
+      robots: {
+        allows: async () => ({
+          allowed: true,
+          reason: "robots.txt returned 403; proceeding on a recorded permission",
+          assumedOnForbidden: true,
+        }),
+      },
+    });
+    const summary = await runRefresh(opts);
+
+    expect(summary.outcomes[0]?.result).toBe("fetched");
+    expect(summary.assumedRobots).toEqual(["game8.co"]);
+    expect(summary.warnings.some((w) => w.includes("--assume-robots-on-403"))).toBe(
+      true,
+    );
+  });
+
+  test("an ordinary run reports no assumed hosts at all", async () => {
+    const { opts } = options({});
+    const summary = await runRefresh(opts);
+    expect(summary.assumedRobots).toEqual([]);
+  });
+
   test("one source blocked is a warning; all of them is a failure", async () => {
     const blocked = {
       allows: async (url: string) => ({
@@ -769,6 +797,7 @@ describe("what the runner reports to the runner", () => {
       },
     ],
     hardFailure: null,
+    assumedRobots: [],
   };
 
   test("a broken source becomes an annotation on the run page", () => {
@@ -928,6 +957,11 @@ describe("flags", () => {
     });
   });
 
+  test("parseArgs reads --assume-robots-on-403, and it is off by default", () => {
+    expect(parseArgs([]).assumeRobotsOn403).toBe(false);
+    expect(parseArgs(["--assume-robots-on-403"]).assumeRobotsOn403).toBe(true);
+  });
+
   test("parseArgs rejects an unknown flag rather than ignoring it", () => {
     expect(() => parseArgs(["--force"])).toThrow("unknown flag");
   });

test/robots.test.ts

@@ -341,3 +341,69 @@ describe("RobotsCache", () => {
     expect(calls).toHaveLength(2);
   });
 });
+
+describe("--assume-robots-on-403", () => {
+  /**
+   * Fandom answers a datacentre address 403 on `/robots.txt` itself, while
+   * `api.php?action=parse` answers our own User-Agent with a 200. The gate
+   * fails closed on the unreadable file, so four sources can never refresh —
+   * even though their rules are known: a person read them in a browser and
+   * wrote them into AGENTS.md § Scraping conduct.
+   *
+   * This option is that recorded permission, and nothing wider. The tests below
+   * are mostly about what it must NOT do.
+   */
+  const forbidden = () => new Response("denied", { status: 403 });
+
+  function cache(assume: boolean, responder = forbidden) {
+    return new RobotsCache({
+      userAgent: UA,
+      fetchImpl: async () => responder(),
+      assumeAllowedWhenForbidden: assume,
+    });
+  }
+
+  test("without it, a 403 on robots.txt still fails closed", async () => {
+    const d = await cache(false).allows("https://x.fandom.com/api.php?action=parse");
+    expect(d.allowed).toBe(false);
+    expect(d.reason).toContain("403");
+  });
+
+  test("with it, the host is fetched and the run is told why", async () => {
+    const d = await cache(true).allows("https://x.fandom.com/api.php?action=parse");
+    expect(d.allowed).toBe(true);
+    expect(d.assumedOnForbidden).toBe(true);
+    expect(d.reason).toContain("--assume-robots-on-403");
+  });
+
+  test("does not override a robots.txt we could read", async () => {
+    // The distinction the whole option rests on. A file that answers and
+    // refuses us is an answer, and it still wins with the flag on.
+    const refuses = () =>
+      new Response("User-agent: *\nDisallow: /\n", { status: 200 });
+    const d = await cache(true, refuses).allows("https://x.fandom.com/api.php");
+    expect(d.allowed).toBe(false);
+    expect(d.assumedOnForbidden).toBeUndefined();
+  });
+
+  test("covers 403 only, not every way robots.txt can fail", async () => {
+    // A 500, a soft 404 and an unreachable host are "we do not know", and
+    // unknown is still not permission. Only 403 is "we know, and were told by
+    // hand" — see game8.co, whose robots.txt reads fine and welcomes us while
+    // its edge refuses the pages; this flag is no use there and must not be.
+    const cases: Array<[string, () => Response]> = [
+      ["500", () => new Response("oops", { status: 500 })],
+      ["401", () => new Response("nope", { status: 401 })],
+      ["soft 404", () => new Response("<!doctype html><html>", { status: 200 })],
+    ];
+    for (const [label, responder] of cases) {
+      const d = await cache(true, responder).allows("https://x.test/wiki/Event");
+      expect(`${label}: ${d.allowed}`).toBe(`${label}: false`);
+    }
+  });
+
+  test("is off unless asked for", async () => {
+    const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => forbidden() });
+    expect((await plain.allows("https://x.test/a")).allowed).toBe(false);
+  });
+});