diff --git a/AGENTS.md b/AGENTS.md index dbe0150..1f9c26a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -262,7 +262,7 @@ Sources are community wikis. Treat them as a guest would: - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL. - One request per source per refresh cycle, minimum 6 hours apart. **`--force` sets that floor - aside for one run, and only a person at a keyboard may pass it** — it is refused under CI, because + aside for one run, and only a person may pass it** — it is refused on an unattended run, because a schedule that forces every cycle is just a shorter interval with extra steps, and the interval is the obligation. What makes it defensible is what it does *not* change: conditional headers still go out, so a page that has not moved costs the host a `304` rather than a re-serve, and per-host @@ -470,11 +470,27 @@ that helps: `bun run refresh --assume-robots-on-403` treats **an interstitial ch `/robots.txt` itself as the permission recorded above rather than failing closed. It is not a workaround for a host that turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still says no, and it does nothing at all for game8.co, whose robots.txt reads -fine and welcomes us while its edge refuses the pages. It is refused under CI, because what it stands -in for is a person having read a file in a browser, and there is no person on a runner. Every host it -applied to is named in the run's warnings, so it stays a thing somebody decided this morning rather -than a default. Nothing else relaxes: one request per source, six hours apart, spaced per host, no -retries. +fine and welcomes us while its edge refuses the pages. Every host it applied to is named in the run's +warnings, so it stays a thing somebody decided this morning rather than a default. Nothing else +relaxes: one request per source, six hours apart, spaced per host, no retries. + +**Who may pass it is a question about the person, not the machine.** Both overrides are refused on an +*unattended* run and available to a person, and `runAttendance` (`scripts/refresh-sources.ts`) draws +that line: a local shell is a person, a `workflow_dispatch` is a person and GitHub records which one +in `GITHUB_ACTOR`, and a `schedule` — or any other runner event — is not. This replaced a blanket +`isCi()` check, which asked the coarser question and got the case that matters wrong: a dispatch sets +`CI=true`, so somebody clicking "Run workflow" was refused exactly as the cron was, and the overrides +were unreachable from the workflow at all. The run prints which override was used and who authorised +it. + +**The cron stays refused, and for the robots override that is not ceremony.** From an address that +gets a challenge we never receive `robots.txt` at all — so a schedule standing on the recorded +permission has no way to notice the host withdrawing it. The recorded permission has no expiry, and a +person re-reading the file in a browser is the only thing that ever re-validates it. A twice-daily job +asserting it forever would be fetching on a snapshot of consent taken on 2026-08-19. Note what the +challenge-vs-refusal narrowing does and does not cover here: a plain `403` still stops us, but a +`robots.txt` *edited* to disallow us would be invisible, because we get a challenge instead of a file. +That gap is the whole reason a person has to be the one asking. **A `403` is two answers wearing one status code, and only one of them is covered.** A managed challenge means "we cannot tell what you are" — the question a human answers by reading the file in a @@ -661,7 +677,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403` -described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are interactive-only, refused under CI, and reported by name +described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are refused on an unattended run — a `schedule` gets neither, a person or a `workflow_dispatch` may pass both (`runAttendance`) — and reported by name in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first. **A source down is a warning; a source down for days is a broken build.** One wiki failing must diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 154e7b9..b780482 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -292,7 +292,12 @@ section but must never claim the event title. challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not excused. - It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no. - - It is refused under CI, because it stands in for a human and there is none on a runner. + - **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a + local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub + names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is + load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a + cron standing on the recorded permission could never see the host withdraw it, and the recorded + permission has no expiry. Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is one nobody withdraws. It changes no other obligation: still one diff --git a/scripts/refresh-sources.ts b/scripts/refresh-sources.ts index 4656fb7..7676344 100644 --- a/scripts/refresh-sources.ts +++ b/scripts/refresh-sources.ts @@ -676,19 +676,51 @@ export async function rebuildFeedViaScript(): Promise { if (code !== 0) throw new Error(`build-feed exited ${code}`); } +/** Whether a person authorised this run, and who, for the override gates. */ +export type Attendance = + | { readonly attended: true; readonly by: string } + | { readonly attended: false; readonly why: string }; + /** - * Are we on a runner rather than at somebody's keyboard? + * Did a person ask for this run, or did a schedule? * - * Both variables, because `CI` is the convention every runner sets and - * `GITHUB_ACTIONS` is the one this repo's workflow guarantees. Erring towards - * "yes" is the safe direction: the only thing it costs is refusing an - * interactive-only flag to a human whose shell exports `CI`. + * The two overrides below stand in for somebody deciding something — that a + * page has moved and they want it now, or that they have read a robots.txt in a + * browser. What must never happen is a *schedule* asserting either one every + * cycle, because an override on every run is not an override, it is the new + * default with extra steps (AGENTS.md § Scraping conduct). + * + * This used to be `isCi()`, which asked the coarser question and got the wrong + * answer for the case that matters: a `workflow_dispatch` sets `CI=true`, so a + * human clicking "Run workflow" was refused exactly like the cron was. A + * dispatch is a person, and a better-evidenced one than a local shell — GitHub + * records who in `GITHUB_ACTOR`, and the run log keeps it. + * + * Erring towards "unattended" stays the safe direction: any runner event that is + * not a dispatch is treated as a schedule, so a `push`- or `repository_dispatch`- + * triggered run gets no overrides either. */ -function isCi(): boolean { - return ( - process.env["CI"] !== undefined && process.env["CI"] !== "" || - process.env["GITHUB_ACTIONS"] === "true" - ); +export function runAttendance( + env: Record = process.env, +): Attendance { + const onRunner = + (env["CI"] !== undefined && env["CI"] !== "") || + env["GITHUB_ACTIONS"] === "true"; + if (!onRunner) return { attended: true, by: "a local shell" }; + + const event = env["GITHUB_EVENT_NAME"]; + if (event === "workflow_dispatch") { + const actor = env["GITHUB_ACTOR"]; + return { + attended: true, + by: + actor === undefined || actor === "" + ? "a manual dispatch" + : `a manual dispatch by ${actor}`, + }; + } + + return { attended: false, why: event === undefined || event === "" ? "CI" : event }; } interface Args { @@ -811,33 +843,47 @@ async function main(): Promise { return 2; } - // The flag stands in for a human having read a robots.txt in a browser and - // written it down. There is no human on a runner, and a scheduled job quietly - // asserting a permission nobody re-checked is how "temporary" becomes - // permanent — so CI is refused the option outright rather than trusted not to - // pass it. AGENTS.md § Scraping conduct is the argument. - // Same reasoning as the robots override: this is a person deciding, once, - // that a page has moved and they want it now. A schedule deciding that every - // run is just a shorter interval with extra steps, and the interval is the - // obligation. - if (args.force && isCi()) { + // Both overrides are a person deciding something — that a page has moved and + // they want it now, or that they have read a robots.txt in a browser and + // written it down. Neither may be asserted by a *schedule*: an override on + // every cycle is not an override, it is the new default with extra steps, and + // for the robots one it would be worse than that. From an address that gets a + // challenge we never receive the file at all, so a cron standing on a recorded + // permission has no way to notice the host withdrawing it — the recorded + // permission has no expiry, and a person re-reading it is the only thing that + // ever re-validates it. AGENTS.md § Scraping conduct is the argument. + const attendance = runAttendance(); + + if (args.force && !attendance.attended) { console.error( - "--force is interactive-only and refused under CI.\n" + + `--force needs a person to ask for it; this run came from ` + + `${attendance.why}.\n` + "The 6h floor is what the scheduled runner is for; change the schedule, " + - "not the floor.", + "not the floor. Dispatch the workflow by hand to pass it.", ); return 2; } - if (args.assumeRobotsOn403 && isCi()) { + if (args.assumeRobotsOn403 && !attendance.attended) { console.error( - "--assume-robots-on-403 is interactive-only and refused under CI.\n" + - "It asserts a permission a person read by hand; run the refresh from a " + - "machine the host serves instead.", + `--assume-robots-on-403 needs a person to ask for it; this run came from ` + + `${attendance.why}.\n` + + "It asserts a permission a person read by hand, and a schedule cannot " + + "re-read it. Dispatch the workflow by hand to pass it.", ); return 2; } + // Who authorised the override is part of the record, not a detail. A local + // shell leaves no trace anybody else can read; a dispatch names the actor. + if (attendance.attended && (args.force || args.assumeRobotsOn403)) { + const used = [ + args.force ? "--force" : null, + args.assumeRobotsOn403 ? "--assume-robots-on-403" : null, + ].filter((f): f is string => f !== null); + console.log(` override: ${used.join(", ")} — authorised by ${attendance.by}`); + } + const store = new SnapshotStore(args.root); const robots = new RobotsCache({ userAgent: args.userAgent, diff --git a/test/refresh.test.ts b/test/refresh.test.ts index 090753e..00813ae 100644 --- a/test/refresh.test.ts +++ b/test/refresh.test.ts @@ -8,6 +8,7 @@ import { DEFAULT_HOST_GAP_MS, outputs, parseArgs, + runAttendance, runRefresh, stepSummary, type RefreshOptions, @@ -1076,3 +1077,65 @@ describe("flags", () => { ); }); }); + +/** + * The override gates ask who authorised the run, not whether a runner is + * present. That distinction is the whole reason the workflow can offer these + * toggles at all: a `workflow_dispatch` sets `CI=true`, so the old `isCi()` + * check refused a human clicking "Run workflow" exactly as it refused the cron. + * + * What must not regress is the other half — a *schedule* must never be able to + * assert either override, because an override on every cycle is the new default + * with extra steps, and for the robots one a cron cannot re-read the permission + * it would be standing on. + */ +describe("runAttendance", () => { + test("a local shell is a person", () => { + const a = runAttendance({}); + expect(a.attended).toBe(true); + expect(a.attended && a.by).toBe("a local shell"); + }); + + test("a scheduled run is not, and says so", () => { + const a = runAttendance({ CI: "true", GITHUB_EVENT_NAME: "schedule" }); + expect(a.attended).toBe(false); + expect(!a.attended && a.why).toBe("schedule"); + }); + + test("a manual dispatch is a person, and names the actor", () => { + const a = runAttendance({ + CI: "true", + GITHUB_ACTIONS: "true", + GITHUB_EVENT_NAME: "workflow_dispatch", + GITHUB_ACTOR: "StereotypicalCat", + }); + expect(a.attended).toBe(true); + expect(a.attended && a.by).toBe("a manual dispatch by StereotypicalCat"); + }); + + test("a dispatch with no actor recorded is still a person", () => { + const a = runAttendance({ CI: "true", GITHUB_EVENT_NAME: "workflow_dispatch" }); + expect(a.attended).toBe(true); + expect(a.attended && a.by).toBe("a manual dispatch"); + }); + + test("any other runner event counts as unattended", () => { + // Erring towards "no person" — a push- or repository_dispatch-triggered run + // gets no overrides either, so a schedule cannot be dressed up as one. + for (const event of ["push", "repository_dispatch", "pull_request", ""]) { + const a = runAttendance({ CI: "true", GITHUB_EVENT_NAME: event }); + expect(`${event}: ${a.attended}`).toBe(`${event}: false`); + } + }); + + test("GITHUB_ACTIONS alone is enough to count as a runner", () => { + const a = runAttendance({ GITHUB_ACTIONS: "true", GITHUB_EVENT_NAME: "schedule" }); + expect(a.attended).toBe(false); + }); + + test("an empty CI variable is not a runner", () => { + // `CI=` is how a shell unsets it in practice; treating it as a runner would + // refuse a person their own flags. + expect(runAttendance({ CI: "" }).attended).toBe(true); + }); +});