refresh: let the cron stand on the recorded robots permission

Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.

Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.

The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.

Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.github/workflows/refresh.yml

@@ -21,18 +21,21 @@ on:
         description: "Refresh a single source id (blank = all)"
         type: string
         default: ""
-      # Both of these are refused on the scheduled run and only work here, on a
+      # `force` is refused on the scheduled run and only works here, on a
-      # dispatch, because a dispatch is a person and GitHub records which one.
+      # dispatch, because a dispatch is a person and GitHub records which one —
-      # See AGENTS.md § Scraping conduct: an override on every cycle is just a
+      # a schedule that forces every cycle is just a shorter interval with extra
-      # new default. Prefer pairing either with `only`.
+      # steps. Prefer pairing it with `only`.
       force:
         description: "Ask sources before their 6h floor is up (one run only)"
         type: boolean
         default: false
+      # The scheduled run passes this one anyway (see the Refresh step), so this
+      # input exists to turn it OFF for a dispatch — which is how you see which
+      # hosts genuinely cannot be read, rather than which ones we assumed.
       assume_robots_on_403:
         description: "Treat a challenged robots.txt 403 as the recorded permission"
         type: boolean
-        default: false
+        default: true
 
 # Never two refreshes at once: they would both fetch, and the second would race
 # the first's commit. Queue instead of cancelling — a half-finished refresh that
@@ -91,9 +94,13 @@ jobs:
           # run script, so a dispatch input cannot become shell.
           ONLY: ${{ inputs.only }}
           DRY_RUN: ${{ inputs.dry_run }}
-          # Empty on a scheduled run, so neither flag is ever passed by the cron.
+          # Empty on a scheduled run, so --force is never passed by the cron.
           FORCE: ${{ inputs.force }}
+          # Also empty on a schedule, which is why EVENT_NAME decides there. The
+          # cron stands on the permission recorded in AGENTS.md § Scraping
+          # conduct; a dispatch can set the input to false to see the real state.
           ASSUME_ROBOTS_ON_403: ${{ inputs.assume_robots_on_403 }}
+          EVENT_NAME: ${{ github.event_name }}
         run: |
           args=()
           if [ "$DRY_RUN" = "true" ]; then
@@ -105,7 +112,7 @@ jobs:
           if [ "$FORCE" = "true" ]; then
             args+=(--force)
           fi
-          if [ "$ASSUME_ROBOTS_ON_403" = "true" ]; then
+          if [ "$ASSUME_ROBOTS_ON_403" = "true" ] || [ "$EVENT_NAME" = "schedule" ]; then
             args+=(--assume-robots-on-403)
           fi
           bun run refresh "${args[@]}"

AGENTS.md

@@ -483,14 +483,30 @@ in `GITHUB_ACTOR`, and a `schedule` — or any other runner event — is not. Th
 were unreachable from the workflow at all. The run prints which override was used and who authorised
 it.
 
-**The cron stays refused, and for the robots override that is not ceremony.** From an address that
+**The two overrides are not symmetrical, and `--force` is the stricter one.** A schedule may never
-gets a challenge we never receive `robots.txt` at all — so a schedule standing on the recorded
+force: asking every cycle before the interval is up is a shorter interval with extra steps, and the
-permission has no way to notice the host withdrawing it. The recorded permission has no expiry, and a
+interval is the obligation. That one stays reachable only from a person or a dispatch.
-person re-reading the file in a browser is the only thing that ever re-validates it. A twice-daily job
+
-asserting it forever would be fetching on a snapshot of consent taken on 2026-08-19. Note what the
+**The cron does pass `--assume-robots-on-403`, and that is a decision with a named cost** (taken by
-challenge-vs-refusal narrowing does and does not cover here: a plain `403` still stops us, but a
+the repository owner, 2026-08-20). Four Fandom sources skip on a challenged `robots.txt` every cycle,
-`robots.txt` *edited* to disallow us would be invisible, because we get a challenge instead of a file.
+and the alternative was four calendars sitting as stale as the last manual run — which for a product
-That gap is the whole reason a person has to be the one asking.
+whose whole promise is a trustworthy end date is the worse failure. So the schedule stands on the
+permission recorded in this section, and **the owner re-reads those files by hand over time** rather
+than the code pretending to.
+
+Be exact about the risk that buys, because it is not "we might be crawling against robots.txt":
+
+- A plain `403` still fails closed. The challenge-or-refusal split is what guarantees that much, so a
+  host that actually turns us away still stops the run.
+- What is invisible is a `robots.txt` **edited** to disallow us. From a challenged address the file
+  never arrives, so a withdrawal would look exactly like the challenge we already expect. Nothing in
+  the pipeline can catch that; only the manual re-read can.
+- **The per-cycle warning is therefore the compensating control, not a nicety.** Every run names each
+  host it stood on a hand-recorded permission for, says the file was not read, and tells the reader to
+  re-read it — as a `::warning` annotation on the run page, so it survives a completely green cycle
+  where nothing else draws the eye. `test/refresh.test.ts` pins both the one-warning-per-host rule and
+  that it reaches the run page. **Do not demote it to a log line**, and do not let a future change
+  make it conditional: it is the only thing that ever prompts the re-read this arrangement depends on.
 
 **A `403` is two answers wearing one status code, and only one of them is covered.** A managed
 challenge means "we cannot tell what you are" — the question a human answers by reading the file in a
@@ -677,7 +693,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t
 `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
 opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
 (failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
-described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are refused on an unattended run — a `schedule` gets neither, a person or a `workflow_dispatch` may pass both (`runAttendance`) — and reported by name
+described in § Fandom, which covers a challenged `403` and never a plain refusal). The two overrides differ on who may pass them: `--force` is refused on an unattended run (`runAttendance` — a person or a `workflow_dispatch`, never a `schedule`), while `--assume-robots-on-403` is passed by the cron by standing decision and is reported by name
 in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
 
 **A source down is a warning; a source down for days is a broken build.** One wiki failing must

docs/INGESTION.md

@@ -292,13 +292,14 @@ section but must never claim the event title.
     challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not
     excused.
   - It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no.
-  - **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a
+  - **The cron passes it, unlike `--force`.** A standing decision (2026-08-20): four Fandom sources
-    local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub
+    skip on a challenged `robots.txt` every cycle, and four stale calendars are the worse failure for
-    names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is
+    a product that sells trustworthy end dates. `--force` remains gated on `runAttendance` — a person
-    load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a
+    or a `workflow_dispatch`, never a `schedule` — because forcing every cycle is just a shorter
-    cron standing on the recorded permission could never see the host withdraw it, and the recorded
+    interval. The accepted risk is narrow and named: a plain `403` still fails closed, but a
-    permission has no expiry. `refresh.yml` therefore exposes both overrides as
+    `robots.txt` *edited* to disallow us is invisible from a challenged address, so the owner re-reads
-    `workflow_dispatch` inputs, which are empty on the cron and so can never be passed by it.
+    those files by hand. The per-cycle `::warning` naming every assumed host is the compensating
+    control and is pinned by tests; the dispatch input can be set to `false` to see the real state.
 
   Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
   override that reports nothing is one nobody withdraws. It changes no other obligation: still one

scripts/refresh-sources.ts

@@ -864,15 +864,21 @@ async function main(): Promise<number> {
     return 2;
   }
 
-  if (args.assumeRobotsOn403 && !attendance.attended) {
+  // `--assume-robots-on-403` is deliberately NOT gated on attendance, unlike
-    console.error(
+  // `--force` above. The four Fandom sources skip on a challenged robots.txt
-      `--assume-robots-on-403 needs a person to ask for it; this run came from ` +
+  // every cycle, and the repository owner has taken on re-reading those files by
-        `${attendance.why}.\n` +
+  // hand over time rather than have four calendars go stale between manual runs
-        "It asserts a permission a person read by hand, and a schedule cannot " +
+  // (decided 2026-08-20). That is a standing decision about this project's own
-        "re-read it. Dispatch the workflow by hand to pass it.",
+  // conduct, so the code records it instead of re-litigating it per run.
-    );
+  //
-    return 2;
+  // What keeps it honest is the disclosure, which is why that is not optional:
-  }
+  // every cycle names each host it stood on a hand-recorded permission for and
+  // says the file was not read this run. The risk being accepted is specific —
+  // from a challenged address robots.txt never arrives, so a file *edited* to
+  // disallow us would be invisible (a plain 403 still fails closed, and the
+  // challenge-or-refusal split is what guarantees that much). The per-cycle
+  // warning is the only thing that ever prompts the re-read, so do not demote it
+  // to a log line.
 
   // Who authorised the override is part of the record, not a detail. A local
   // shell leaves no trace anybody else can read; a dispatch names the actor.

test/refresh.test.ts

@@ -901,6 +901,31 @@ describe("what the runner reports to the runner", () => {
     for (const line of lines) expect(line).not.toInclude("\n");
   });
 
+  test("an assumed robots permission annotates an otherwise green run", () => {
+    // This is now the scheduled run's standing state, not an exception: the cron
+    // passes --assume-robots-on-403, so every cycle fetches four Fandom sources
+    // on a permission it could not re-read. That warning is the whole
+    // compensating control for the risk being accepted (AGENTS.md § Scraping
+    // conduct), and a summary line can be scrolled past where a run-page
+    // annotation cannot — so it has to survive as one even when nothing is
+    // broken and there is no error to draw the eye.
+    const green = {
+      ...broken,
+      broken: [],
+      warnings: [
+        "reverse1999.fandom.com: fetched on --assume-robots-on-403 — its " +
+          "robots.txt was NOT read this run. Re-read it in a browser.",
+      ],
+      assumedRobots: ["reverse1999.fandom.com"],
+    };
+    const lines = annotations(green);
+    expect(lines).toHaveLength(1);
+    expect(lines[0]).toStartWith("::warning title=refresh::");
+    expect(lines[0]).toContain("--assume-robots-on-403");
+    expect(lines[0]).toContain("reverse1999.fandom.com");
+    expect(lines[0]).not.toInclude("\n");
+  });
+
   test("nothing to say means no annotations", () => {
     expect(annotations({ ...broken, warnings: [], broken: [] })).toEqual([]);
   });
@@ -978,12 +1003,14 @@ describe("the workflows that drive the refresh", () => {
     expect(refresh).not.toContain("push --force");
   });
 
-  test("refresh.yml offers both overrides to a dispatch and to no cron", async () => {
+  test("refresh.yml lets the cron assume robots, and never lets it force", async () => {
-    // The scheduled run must never be able to pass either one: an override on
+    // The two overrides are deliberately not symmetrical any more. --force stays
-    // every cycle is the new default with extra steps, and for the robots one a
+    // reachable only through a dispatch input: a schedule that forces every cycle
-    // cron cannot re-read the permission it would be standing on. Both flags are
+    // is a shorter interval with extra steps, and the interval is the obligation.
-    // therefore reachable only through `workflow_dispatch` inputs, which are
+    // --assume-robots-on-403 is passed by the cron on purpose (owner's decision,
-    // empty on a schedule, and each is guarded on the literal string "true".
+    // 2026-08-20) so four Fandom calendars do not sit stale between manual runs;
+    // the compensating control is the per-cycle warning naming every host, which
+    // is what prompts the manual re-read.
     const refresh = await read("refresh.yml");
 
     const dispatch = refresh.slice(
@@ -993,15 +1020,16 @@ describe("the workflows that drive the refresh", () => {
     expect(dispatch).toContain("force:");
     expect(dispatch).toContain("assume_robots_on_403:");
 
-    // Every occurrence of either flag is inside a test on its input variable.
+    // --force: guarded on its input alone, so a schedule (where it is empty)
+    // can never reach it.
     expect(refresh).toMatch(/if \[ "\$FORCE" = "true" \]; then\n\s*args\+=\(--force\)/);
+    expect(refresh).not.toMatch(/FORCE" = "true" \] \|\| \[ "\$EVENT_NAME"/);
+
+    // --assume-robots-on-403: its input OR the run being a schedule.
     expect(refresh).toMatch(
-      /if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \]; then\n\s*args\+=\(--assume-robots-on-403\)/,
+      /if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \] \|\| \[ "\$EVENT_NAME" = "schedule" \]; then\n\s*args\+=\(--assume-robots-on-403\)/,
     );
-    // The cron block itself carries no flags.
+    expect(refresh).toContain("EVENT_NAME: ${{ github.event_name }}");
-    const cron = refresh.slice(refresh.indexOf("schedule:"), refresh.indexOf("workflow_dispatch:"));
-    expect(cron).not.toContain("force");
-    expect(cron).not.toContain("assume");
   });
 
   test("refresh.yml turns red on a broken source only after committing", async () => {