diff --git a/AGENTS.md b/AGENTS.md index a58753d..26b6b53 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest theme.ts — dark or light, and what a game hue reads as on each scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 716 tests +test/ 724 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -385,6 +385,16 @@ fixture-backed* from any address; what it cannot do is pass the robots gate at r fails closed and skips. The permission is therefore a thing a human records once, and the freshness is a thing that needs an address Fandom serves. +`--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing +that helps: `bun run refresh --assume-robots-on-403` treats a `403` **on `/robots.txt` itself** as +the permission recorded above rather than failing closed. It is not a workaround for a host that +turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still +says no, and it does nothing at all for game8.co, whose robots.txt reads fine and welcomes us while +its edge refuses the pages. It is refused under CI, because what it stands in for is a person having +read a file in a browser, and there is no person on a runner. Every host it applied to is named in +the run's warnings, so it stays a thing somebody decided this morning rather than a default. Nothing +else relaxes: one request per source, six hours apart, spaced per host, no retries. + One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the robots gate **fails closed there and the source is skipped**. That is a warning line rather than a broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if @@ -559,7 +569,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be -read). Anything that would make it fetch more often is a change to this section first. +read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first. **A source down is a warning; a source down for days is a broken build.** One wiki failing must never blank a calendar or stop the sources that did answer from being committed — so a failure is diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 69eee28..906ff56 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -270,6 +270,17 @@ section but must never claim the event title. - Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that 5xxs or times out means "do not fetch", because a permission we could not read is not a permission we have. A 404 means no restrictions. +- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre + address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with + a `200`, so the gate fails closed and four sources can never refresh — even though their rules are + known, because a person read them in a browser and wrote them into AGENTS.md § Scraping conduct. + The flag makes the run proceed on that recorded permission. Three things bound it: it applies to + `403` **only** (a 401, a 5xx or a soft 404 are still "we do not know"); it never overrides a + `robots.txt` we *could* read, so a file that disallows us still says no; and it is refused under + CI, because it stands in for a human and there is none on a runner. Every host it applied to is + named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is + one nobody withdraws. It changes no other obligation: still one request per source, still six + hours apart, still spaced per host. - 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says one per source per cycle. A failed source waits for the next cycle instead. - **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole diff --git a/scripts/refresh-sources.ts b/scripts/refresh-sources.ts index d8e2e73..1c43d43 100644 --- a/scripts/refresh-sources.ts +++ b/scripts/refresh-sources.ts @@ -4,12 +4,18 @@ * bun run refresh # the real thing * bun run refresh --dry-run # plan only, no requests, no writes * bun run refresh --only genshin-game8-events + * bun run refresh --assume-robots-on-403 # see § the flag, below * * This is the scheduled half of the pipeline (docs/INGESTION.md stages 1-2). * The rules it enforces are etiquette obligations, not preferences: * * - robots.txt is read once per host per run and obeyed; unreadable means - * "do not fetch", never "assume yes". + * "do not fetch", never "assume yes". `--assume-robots-on-403` is the one + * opt-in exception, for a host that will not serve us the file at all while + * serving us the page: it stands in for rules a person read in a browser + * and wrote into AGENTS.md. It covers 403 only, never overrides a + * robots.txt we could read, is refused under CI, and names every host it + * touched. Nothing else about being a guest relaxes with it. * - at most ONE request per source per cycle, and never sooner than six hours * after the last attempt. There is deliberately no retry: a retry is a * second request, and the next cycle is minutes-cheap compared to being a @@ -95,6 +101,11 @@ export interface RefreshSummary { broken: BrokenSource[]; /** Set when the run should exit non-zero. */ hardFailure: string | null; + /** + * Hosts fetched under `--assume-robots-on-403`. Empty on every normal run, + * and on every CI run — the flag is refused there. + */ + assumedRobots: string[]; } export interface RobotsGate { @@ -103,6 +114,8 @@ export interface RobotsGate { reason: string; /** From the host's `Crawl-delay`, when it states one. */ crawlDelayMs?: number | null; + /** True when `--assume-robots-on-403` is what opened this host. */ + assumedOnForbidden?: boolean; }>; } @@ -148,6 +161,8 @@ export const DEFAULT_HOST_GAP_MS = 2_000; /** Per-cycle state shared across sources: which hosts we have already asked. */ interface Cycle { requestedHosts: Set; + /** Hosts fetched on `--assume-robots-on-403` rather than on a file we read. */ + assumedRobots: Set; } export async function runRefresh( @@ -161,9 +176,10 @@ export async function runRefresh( warnings: [], broken: [], hardFailure: null, + assumedRobots: [], }; - const cycle: Cycle = { requestedHosts: new Set() }; + const cycle: Cycle = { requestedHosts: new Set(), assumedRobots: new Set() }; const selected = options.only === null @@ -230,6 +246,15 @@ export async function runRefresh( } } + // Named in the summary rather than only in the per-source log, so it survives + // into the job summary and cannot be scrolled past. + summary.assumedRobots = [...cycle.assumedRobots].sort(); + for (const host of summary.assumedRobots) { + summary.warnings.push( + `${host}: fetched on --assume-robots-on-403; its robots.txt was not read this run`, + ); + } + // Every source failing is not "a wiki is down", it is us: no network, a bad // User-Agent, a proxy. That should stop the pipeline rather than look green. if (summary.attempted > 0 && summary.confirmed === 0) { @@ -303,6 +328,12 @@ async function refreshOne( eventCount: meta?.eventCount ?? null, }; } + // Fetching on a permission nobody can re-read is a thing the run has to say + // out loud, every time and per source. An override that reports nothing is an + // override that quietly becomes the default. + if (decision.assumedOnForbidden === true) { + cycle.assumedRobots.add(new URL(adapter.url).host); + } // Space requests to a host we have already asked this cycle. This sits after // the interval and robots gates on purpose: waiting on behalf of a source we @@ -608,6 +639,21 @@ export async function rebuildFeedViaScript(): Promise { if (code !== 0) throw new Error(`build-feed exited ${code}`); } +/** + * Are we on a runner rather than at somebody's keyboard? + * + * Both variables, because `CI` is the convention every runner sets and + * `GITHUB_ACTIONS` is the one this repo's workflow guarantees. Erring towards + * "yes" is the safe direction: the only thing it costs is refusing an + * interactive-only flag to a human whose shell exports `CI`. + */ +function isCi(): boolean { + return ( + process.env["CI"] !== undefined && process.env["CI"] !== "" || + process.env["GITHUB_ACTIONS"] === "true" + ); +} + interface Args { dryRun: boolean; only: string | null; @@ -615,6 +661,8 @@ interface Args { userAgent: string; rebuild: boolean; help: boolean; + /** See `--assume-robots-on-403` in USAGE, and § Scraping conduct. */ + assumeRobotsOn403: boolean; } export function parseArgs(argv: readonly string[]): Args { @@ -625,6 +673,7 @@ export function parseArgs(argv: readonly string[]): Args { userAgent: process.env["REFRESH_USER_AGENT"] ?? DEFAULT_USER_AGENT, rebuild: true, help: false, + assumeRobotsOn403: false, }; // A flag whose value is missing is a mistake, never a default. `--only` with @@ -659,6 +708,9 @@ export function parseArgs(argv: readonly string[]): Args { case "--no-feed": args.rebuild = false; break; + case "--assume-robots-on-403": + args.assumeRobotsOn403 = true; + break; case "--help": case "-h": args.help = true; @@ -675,13 +727,19 @@ export function parseArgs(argv: readonly string[]): Args { } const USAGE = `usage: bun run refresh [--dry-run] [--only ] [--snapshots ] - [--user-agent ] [--no-feed] + [--user-agent ] [--no-feed] [--assume-robots-on-403] --dry-run report what each source would do; no requests, no writes --only refresh a single source (${ADAPTERS.map((a) => a.id).join(", ")}) --snapshots snapshot cache directory (default: snapshots, env SNAPSHOT_DIR) --user-agent override the User-Agent (env REFRESH_USER_AGENT) - --no-feed skip regenerating public/data/events.v1.json`; + --no-feed skip regenerating public/data/events.v1.json + --assume-robots-on-403 + temporary, interactive-only. When a host answers 403 to + /robots.txt itself, proceed on the permission recorded in + AGENTS.md instead of failing closed. Refused under CI. + Does NOT override a robots.txt we could read: a file that + disallows us still says no.`; async function main(): Promise { let args: Args; @@ -704,12 +762,35 @@ async function main(): Promise { return 2; } + // The flag stands in for a human having read a robots.txt in a browser and + // written it down. There is no human on a runner, and a scheduled job quietly + // asserting a permission nobody re-checked is how "temporary" becomes + // permanent — so CI is refused the option outright rather than trusted not to + // pass it. AGENTS.md § Scraping conduct is the argument. + if (args.assumeRobotsOn403 && isCi()) { + console.error( + "--assume-robots-on-403 is interactive-only and refused under CI.\n" + + "It asserts a permission a person read by hand; run the refresh from a " + + "machine the host serves instead.", + ); + return 2; + } + const store = new SnapshotStore(args.root); const robots = new RobotsCache({ userAgent: args.userAgent, fetchImpl: (input, init) => fetch(input, init), + assumeAllowedWhenForbidden: args.assumeRobotsOn403, }); + if (args.assumeRobotsOn403) { + console.warn( + " ! --assume-robots-on-403: a host answering 403 to /robots.txt will be\n" + + " fetched anyway, on the permission recorded in AGENTS.md. Temporary,\n" + + " and every host it applies to is named at the end of this run.", + ); + } + console.log( `refresh: ${args.only ?? `${ADAPTERS.length} sources`}${args.dryRun ? " (dry run)" : ""}`, ); @@ -737,6 +818,12 @@ async function main(): Promise { `${summary.warnings.length} warnings, ${summary.broken.length} broken`, ); for (const warning of summary.warnings) console.warn(` ! ${warning}`); + for (const host of summary.assumedRobots) { + console.warn( + ` ! ${host}: robots.txt was NOT read this run. Re-read it in a browser ` + + `and confirm AGENTS.md § Scraping conduct still describes it.`, + ); + } for (const b of summary.broken) { console.error( ` !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` + diff --git a/src/ingest/robots.ts b/src/ingest/robots.ts index 03641be..9bf70cd 100644 --- a/src/ingest/robots.ts +++ b/src/ingest/robots.ts @@ -267,6 +267,12 @@ export interface RobotsDecision { /** Human-readable why, for the run log. */ readonly reason: string; readonly crawlDelayMs: number | null; + /** + * True when this host was opened by `assumeAllowedWhenForbidden` rather than + * by a robots.txt we actually read. The caller is expected to say so out + * loud — an override nobody sees is an override nobody withdraws. + */ + readonly assumedOnForbidden?: boolean; } export interface RobotsCacheOptions { @@ -276,6 +282,20 @@ export interface RobotsCacheOptions { ttlMs?: number; now?: () => number; timeoutMs?: number; + /** + * Treat a `403` on **robots.txt itself** as permission, instead of failing + * closed. Off by default and never set from CI — see `--assume-robots-on-403` + * in `scripts/refresh-sources.ts` for the whole argument. + * + * The narrowness is the point. This covers exactly one situation: a host that + * will not serve us `/robots.txt` from this address, whose rules a human has + * therefore read in a browser and written down (AGENTS.md § Scraping conduct + * records Fandom's, verbatim). It does **not** touch a robots.txt we did read + * and that disallows us — `isAllowed` still decides that, and still says no. + * A file we can read and that refuses us is an answer; this is the case where + * there is no answer and one has been obtained by hand. + */ + assumeAllowedWhenForbidden?: boolean; } interface CacheEntry { @@ -284,6 +304,8 @@ interface CacheEntry { usable: boolean; reason: string; at: number; + /** True when `assumeAllowedWhenForbidden` is what made this entry usable. */ + assumedOnForbidden?: boolean; } const DAY_MS = 24 * 60 * 60 * 1000; @@ -304,6 +326,7 @@ export class RobotsCache { private readonly ttlMs: number; private readonly nowMs: () => number; private readonly timeoutMs: number; + private readonly assumeAllowedWhenForbidden: boolean; constructor(options: RobotsCacheOptions) { this.userAgent = options.userAgent; @@ -311,6 +334,8 @@ export class RobotsCache { this.ttlMs = options.ttlMs ?? DAY_MS; this.nowMs = options.now ?? (() => Date.now()); this.timeoutMs = options.timeoutMs ?? 20_000; + this.assumeAllowedWhenForbidden = + options.assumeAllowedWhenForbidden ?? false; } /** Number of robots.txt requests made, for tests and the run log. */ @@ -329,6 +354,9 @@ export class RobotsCache { allowed, reason: allowed ? entry.reason : `disallowed by ${origin}/robots.txt`, crawlDelayMs: crawlDelayMs(entry.robots, this.userAgent), + ...(entry.assumedOnForbidden === true + ? { assumedOnForbidden: true } + : {}), }; } @@ -373,6 +401,23 @@ export class RobotsCache { }; } + // A host that will not serve us the file at all, only when an operator has + // asked for this. `usable: true` with no rules is not a guess about what + // the site permits — it is standing in for rules a human read in a browser + // and wrote into AGENTS.md. Everything else about being a guest still + // applies: one request per source, six hours apart, spaced per host. + if (response.status === 403 && this.assumeAllowedWhenForbidden) { + return { + robots: ALLOW_ALL, + usable: true, + reason: + `robots.txt returned 403; proceeding on a permission recorded by ` + + `hand (--assume-robots-on-403)`, + at, + assumedOnForbidden: true, + }; + } + if (response.status >= 400) { return { robots: ALLOW_ALL, diff --git a/test/refresh.test.ts b/test/refresh.test.ts index 46e6160..e4902e6 100644 --- a/test/refresh.test.ts +++ b/test/refresh.test.ts @@ -263,6 +263,34 @@ describe("robots", () => { expect(summary.warnings).toHaveLength(1); }); + test("fetching on an assumed robots permission is named in the summary", async () => { + // The override cannot be silent: a permission nobody can re-read is one + // nobody withdraws, so every host it applied to is reported by name and + // warned about, on a run that otherwise looks completely ordinary. + const { opts } = options({ + robots: { + allows: async () => ({ + allowed: true, + reason: "robots.txt returned 403; proceeding on a recorded permission", + assumedOnForbidden: true, + }), + }, + }); + const summary = await runRefresh(opts); + + expect(summary.outcomes[0]?.result).toBe("fetched"); + expect(summary.assumedRobots).toEqual(["game8.co"]); + expect(summary.warnings.some((w) => w.includes("--assume-robots-on-403"))).toBe( + true, + ); + }); + + test("an ordinary run reports no assumed hosts at all", async () => { + const { opts } = options({}); + const summary = await runRefresh(opts); + expect(summary.assumedRobots).toEqual([]); + }); + test("one source blocked is a warning; all of them is a failure", async () => { const blocked = { allows: async (url: string) => ({ @@ -769,6 +797,7 @@ describe("what the runner reports to the runner", () => { }, ], hardFailure: null, + assumedRobots: [], }; test("a broken source becomes an annotation on the run page", () => { @@ -928,6 +957,11 @@ describe("flags", () => { }); }); + test("parseArgs reads --assume-robots-on-403, and it is off by default", () => { + expect(parseArgs([]).assumeRobotsOn403).toBe(false); + expect(parseArgs(["--assume-robots-on-403"]).assumeRobotsOn403).toBe(true); + }); + test("parseArgs rejects an unknown flag rather than ignoring it", () => { expect(() => parseArgs(["--force"])).toThrow("unknown flag"); }); diff --git a/test/robots.test.ts b/test/robots.test.ts index e886235..7081a9f 100644 --- a/test/robots.test.ts +++ b/test/robots.test.ts @@ -341,3 +341,69 @@ describe("RobotsCache", () => { expect(calls).toHaveLength(2); }); }); + +describe("--assume-robots-on-403", () => { + /** + * Fandom answers a datacentre address 403 on `/robots.txt` itself, while + * `api.php?action=parse` answers our own User-Agent with a 200. The gate + * fails closed on the unreadable file, so four sources can never refresh — + * even though their rules are known: a person read them in a browser and + * wrote them into AGENTS.md § Scraping conduct. + * + * This option is that recorded permission, and nothing wider. The tests below + * are mostly about what it must NOT do. + */ + const forbidden = () => new Response("denied", { status: 403 }); + + function cache(assume: boolean, responder = forbidden) { + return new RobotsCache({ + userAgent: UA, + fetchImpl: async () => responder(), + assumeAllowedWhenForbidden: assume, + }); + } + + test("without it, a 403 on robots.txt still fails closed", async () => { + const d = await cache(false).allows("https://x.fandom.com/api.php?action=parse"); + expect(d.allowed).toBe(false); + expect(d.reason).toContain("403"); + }); + + test("with it, the host is fetched and the run is told why", async () => { + const d = await cache(true).allows("https://x.fandom.com/api.php?action=parse"); + expect(d.allowed).toBe(true); + expect(d.assumedOnForbidden).toBe(true); + expect(d.reason).toContain("--assume-robots-on-403"); + }); + + test("does not override a robots.txt we could read", async () => { + // The distinction the whole option rests on. A file that answers and + // refuses us is an answer, and it still wins with the flag on. + const refuses = () => + new Response("User-agent: *\nDisallow: /\n", { status: 200 }); + const d = await cache(true, refuses).allows("https://x.fandom.com/api.php"); + expect(d.allowed).toBe(false); + expect(d.assumedOnForbidden).toBeUndefined(); + }); + + test("covers 403 only, not every way robots.txt can fail", async () => { + // A 500, a soft 404 and an unreachable host are "we do not know", and + // unknown is still not permission. Only 403 is "we know, and were told by + // hand" — see game8.co, whose robots.txt reads fine and welcomes us while + // its edge refuses the pages; this flag is no use there and must not be. + const cases: Array<[string, () => Response]> = [ + ["500", () => new Response("oops", { status: 500 })], + ["401", () => new Response("nope", { status: 401 })], + ["soft 404", () => new Response("", { status: 200 })], + ]; + for (const [label, responder] of cases) { + const d = await cache(true, responder).allows("https://x.test/wiki/Event"); + expect(`${label}: ${d.allowed}`).toBe(`${label}: false`); + } + }); + + test("is off unless asked for", async () => { + const plain = new RobotsCache({ userAgent: UA, fetchImpl: async () => forbidden() }); + expect((await plain.allows("https://x.test/a")).allowed).toBe(false); + }); +});