diff --git a/.github/workflows/refresh.yml b/.github/workflows/refresh.yml index f0c281c..8607f31 100644 --- a/.github/workflows/refresh.yml +++ b/.github/workflows/refresh.yml @@ -21,18 +21,21 @@ on: description: "Refresh a single source id (blank = all)" type: string default: "" - # Both of these are refused on the scheduled run and only work here, on a - # dispatch, because a dispatch is a person and GitHub records which one. - # See AGENTS.md § Scraping conduct: an override on every cycle is just a - # new default. Prefer pairing either with `only`. + # `force` is refused on the scheduled run and only works here, on a + # dispatch, because a dispatch is a person and GitHub records which one — + # a schedule that forces every cycle is just a shorter interval with extra + # steps. Prefer pairing it with `only`. force: description: "Ask sources before their 6h floor is up (one run only)" type: boolean default: false + # The scheduled run passes this one anyway (see the Refresh step), so this + # input exists to turn it OFF for a dispatch — which is how you see which + # hosts genuinely cannot be read, rather than which ones we assumed. assume_robots_on_403: description: "Treat a challenged robots.txt 403 as the recorded permission" type: boolean - default: false + default: true # Never two refreshes at once: they would both fetch, and the second would race # the first's commit. Queue instead of cancelling — a half-finished refresh that @@ -91,9 +94,13 @@ jobs: # run script, so a dispatch input cannot become shell. ONLY: ${{ inputs.only }} DRY_RUN: ${{ inputs.dry_run }} - # Empty on a scheduled run, so neither flag is ever passed by the cron. + # Empty on a scheduled run, so --force is never passed by the cron. FORCE: ${{ inputs.force }} + # Also empty on a schedule, which is why EVENT_NAME decides there. The + # cron stands on the permission recorded in AGENTS.md § Scraping + # conduct; a dispatch can set the input to false to see the real state. ASSUME_ROBOTS_ON_403: ${{ inputs.assume_robots_on_403 }} + EVENT_NAME: ${{ github.event_name }} run: | args=() if [ "$DRY_RUN" = "true" ]; then @@ -105,7 +112,7 @@ jobs: if [ "$FORCE" = "true" ]; then args+=(--force) fi - if [ "$ASSUME_ROBOTS_ON_403" = "true" ]; then + if [ "$ASSUME_ROBOTS_ON_403" = "true" ] || [ "$EVENT_NAME" = "schedule" ]; then args+=(--assume-robots-on-403) fi bun run refresh "${args[@]}" diff --git a/AGENTS.md b/AGENTS.md index 1f9c26a..2181b46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -483,14 +483,30 @@ in `GITHUB_ACTOR`, and a `schedule` — or any other runner event — is not. Th were unreachable from the workflow at all. The run prints which override was used and who authorised it. -**The cron stays refused, and for the robots override that is not ceremony.** From an address that -gets a challenge we never receive `robots.txt` at all — so a schedule standing on the recorded -permission has no way to notice the host withdrawing it. The recorded permission has no expiry, and a -person re-reading the file in a browser is the only thing that ever re-validates it. A twice-daily job -asserting it forever would be fetching on a snapshot of consent taken on 2026-08-19. Note what the -challenge-vs-refusal narrowing does and does not cover here: a plain `403` still stops us, but a -`robots.txt` *edited* to disallow us would be invisible, because we get a challenge instead of a file. -That gap is the whole reason a person has to be the one asking. +**The two overrides are not symmetrical, and `--force` is the stricter one.** A schedule may never +force: asking every cycle before the interval is up is a shorter interval with extra steps, and the +interval is the obligation. That one stays reachable only from a person or a dispatch. + +**The cron does pass `--assume-robots-on-403`, and that is a decision with a named cost** (taken by +the repository owner, 2026-08-20). Four Fandom sources skip on a challenged `robots.txt` every cycle, +and the alternative was four calendars sitting as stale as the last manual run — which for a product +whose whole promise is a trustworthy end date is the worse failure. So the schedule stands on the +permission recorded in this section, and **the owner re-reads those files by hand over time** rather +than the code pretending to. + +Be exact about the risk that buys, because it is not "we might be crawling against robots.txt": + +- A plain `403` still fails closed. The challenge-or-refusal split is what guarantees that much, so a + host that actually turns us away still stops the run. +- What is invisible is a `robots.txt` **edited** to disallow us. From a challenged address the file + never arrives, so a withdrawal would look exactly like the challenge we already expect. Nothing in + the pipeline can catch that; only the manual re-read can. +- **The per-cycle warning is therefore the compensating control, not a nicety.** Every run names each + host it stood on a hand-recorded permission for, says the file was not read, and tells the reader to + re-read it — as a `::warning` annotation on the run page, so it survives a completely green cycle + where nothing else draws the eye. `test/refresh.test.ts` pins both the one-warning-per-host rule and + that it reaches the run page. **Do not demote it to a log line**, and do not let a future change + make it conditional: it is the only thing that ever prompts the re-read this arrangement depends on. **A `403` is two answers wearing one status code, and only one of them is covered.** A managed challenge means "we cannot tell what you are" — the question a human answers by reading the file in a @@ -677,7 +693,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403` -described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are refused on an unattended run — a `schedule` gets neither, a person or a `workflow_dispatch` may pass both (`runAttendance`) — and reported by name +described in § Fandom, which covers a challenged `403` and never a plain refusal). The two overrides differ on who may pass them: `--force` is refused on an unattended run (`runAttendance` — a person or a `workflow_dispatch`, never a `schedule`), while `--assume-robots-on-403` is passed by the cron by standing decision and is reported by name in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first. **A source down is a warning; a source down for days is a broken build.** One wiki failing must diff --git a/docs/INGESTION.md b/docs/INGESTION.md index f60d92d..107870d 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -292,13 +292,14 @@ section but must never claim the event title. challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not excused. - It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no. - - **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a - local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub - names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is - load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a - cron standing on the recorded permission could never see the host withdraw it, and the recorded - permission has no expiry. `refresh.yml` therefore exposes both overrides as - `workflow_dispatch` inputs, which are empty on the cron and so can never be passed by it. + - **The cron passes it, unlike `--force`.** A standing decision (2026-08-20): four Fandom sources + skip on a challenged `robots.txt` every cycle, and four stale calendars are the worse failure for + a product that sells trustworthy end dates. `--force` remains gated on `runAttendance` — a person + or a `workflow_dispatch`, never a `schedule` — because forcing every cycle is just a shorter + interval. The accepted risk is narrow and named: a plain `403` still fails closed, but a + `robots.txt` *edited* to disallow us is invisible from a challenged address, so the owner re-reads + those files by hand. The per-cycle `::warning` naming every assumed host is the compensating + control and is pinned by tests; the dispatch input can be set to `false` to see the real state. Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an override that reports nothing is one nobody withdraws. It changes no other obligation: still one diff --git a/scripts/refresh-sources.ts b/scripts/refresh-sources.ts index 7676344..2dfe16a 100644 --- a/scripts/refresh-sources.ts +++ b/scripts/refresh-sources.ts @@ -864,15 +864,21 @@ async function main(): Promise { return 2; } - if (args.assumeRobotsOn403 && !attendance.attended) { - console.error( - `--assume-robots-on-403 needs a person to ask for it; this run came from ` + - `${attendance.why}.\n` + - "It asserts a permission a person read by hand, and a schedule cannot " + - "re-read it. Dispatch the workflow by hand to pass it.", - ); - return 2; - } + // `--assume-robots-on-403` is deliberately NOT gated on attendance, unlike + // `--force` above. The four Fandom sources skip on a challenged robots.txt + // every cycle, and the repository owner has taken on re-reading those files by + // hand over time rather than have four calendars go stale between manual runs + // (decided 2026-08-20). That is a standing decision about this project's own + // conduct, so the code records it instead of re-litigating it per run. + // + // What keeps it honest is the disclosure, which is why that is not optional: + // every cycle names each host it stood on a hand-recorded permission for and + // says the file was not read this run. The risk being accepted is specific — + // from a challenged address robots.txt never arrives, so a file *edited* to + // disallow us would be invisible (a plain 403 still fails closed, and the + // challenge-or-refusal split is what guarantees that much). The per-cycle + // warning is the only thing that ever prompts the re-read, so do not demote it + // to a log line. // Who authorised the override is part of the record, not a detail. A local // shell leaves no trace anybody else can read; a dispatch names the actor. diff --git a/test/refresh.test.ts b/test/refresh.test.ts index ca4b4e2..02002d5 100644 --- a/test/refresh.test.ts +++ b/test/refresh.test.ts @@ -901,6 +901,31 @@ describe("what the runner reports to the runner", () => { for (const line of lines) expect(line).not.toInclude("\n"); }); + test("an assumed robots permission annotates an otherwise green run", () => { + // This is now the scheduled run's standing state, not an exception: the cron + // passes --assume-robots-on-403, so every cycle fetches four Fandom sources + // on a permission it could not re-read. That warning is the whole + // compensating control for the risk being accepted (AGENTS.md § Scraping + // conduct), and a summary line can be scrolled past where a run-page + // annotation cannot — so it has to survive as one even when nothing is + // broken and there is no error to draw the eye. + const green = { + ...broken, + broken: [], + warnings: [ + "reverse1999.fandom.com: fetched on --assume-robots-on-403 — its " + + "robots.txt was NOT read this run. Re-read it in a browser.", + ], + assumedRobots: ["reverse1999.fandom.com"], + }; + const lines = annotations(green); + expect(lines).toHaveLength(1); + expect(lines[0]).toStartWith("::warning title=refresh::"); + expect(lines[0]).toContain("--assume-robots-on-403"); + expect(lines[0]).toContain("reverse1999.fandom.com"); + expect(lines[0]).not.toInclude("\n"); + }); + test("nothing to say means no annotations", () => { expect(annotations({ ...broken, warnings: [], broken: [] })).toEqual([]); }); @@ -978,12 +1003,14 @@ describe("the workflows that drive the refresh", () => { expect(refresh).not.toContain("push --force"); }); - test("refresh.yml offers both overrides to a dispatch and to no cron", async () => { - // The scheduled run must never be able to pass either one: an override on - // every cycle is the new default with extra steps, and for the robots one a - // cron cannot re-read the permission it would be standing on. Both flags are - // therefore reachable only through `workflow_dispatch` inputs, which are - // empty on a schedule, and each is guarded on the literal string "true". + test("refresh.yml lets the cron assume robots, and never lets it force", async () => { + // The two overrides are deliberately not symmetrical any more. --force stays + // reachable only through a dispatch input: a schedule that forces every cycle + // is a shorter interval with extra steps, and the interval is the obligation. + // --assume-robots-on-403 is passed by the cron on purpose (owner's decision, + // 2026-08-20) so four Fandom calendars do not sit stale between manual runs; + // the compensating control is the per-cycle warning naming every host, which + // is what prompts the manual re-read. const refresh = await read("refresh.yml"); const dispatch = refresh.slice( @@ -993,15 +1020,16 @@ describe("the workflows that drive the refresh", () => { expect(dispatch).toContain("force:"); expect(dispatch).toContain("assume_robots_on_403:"); - // Every occurrence of either flag is inside a test on its input variable. + // --force: guarded on its input alone, so a schedule (where it is empty) + // can never reach it. expect(refresh).toMatch(/if \[ "\$FORCE" = "true" \]; then\n\s*args\+=\(--force\)/); + expect(refresh).not.toMatch(/FORCE" = "true" \] \|\| \[ "\$EVENT_NAME"/); + + // --assume-robots-on-403: its input OR the run being a schedule. expect(refresh).toMatch( - /if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \]; then\n\s*args\+=\(--assume-robots-on-403\)/, + /if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \] \|\| \[ "\$EVENT_NAME" = "schedule" \]; then\n\s*args\+=\(--assume-robots-on-403\)/, ); - // The cron block itself carries no flags. - const cron = refresh.slice(refresh.indexOf("schedule:"), refresh.indexOf("workflow_dispatch:")); - expect(cron).not.toContain("force"); - expect(cron).not.toContain("assume"); + expect(refresh).toContain("EVENT_NAME: ${{ github.event_name }}"); }); test("refresh.yml turns red on a broken source only after committing", async () => {