refresh: let the cron stand on the recorded robots permission

Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.

Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.

The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.

Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-20 00:12:28 +02:00
co-authored by Claude Opus 5
parent f12398a29b
commit 3e463adf47
5 changed files with 102 additions and 44 deletions
+8 -7
View File
@@ -292,13 +292,14 @@ section but must never claim the event title.
challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not
excused.
- It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no.
- **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a
local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub
names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is
load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a
cron standing on the recorded permission could never see the host withdraw it, and the recorded
permission has no expiry. `refresh.yml` therefore exposes both overrides as
`workflow_dispatch` inputs, which are empty on the cron and so can never be passed by it.
- **The cron passes it, unlike `--force`.** A standing decision (2026-08-20): four Fandom sources
skip on a challenged `robots.txt` every cycle, and four stale calendars are the worse failure for
a product that sells trustworthy end dates. `--force` remains gated on `runAttendance` — a person
or a `workflow_dispatch`, never a `schedule` — because forcing every cycle is just a shorter
interval. The accepted risk is narrow and named: a plain `403` still fails closed, but a
`robots.txt` *edited* to disallow us is invisible from a challenged address, so the owner re-reads
those files by hand. The per-cycle `::warning` naming every assumed host is the compensating
control and is pinned by tests; the dispatch input can be set to `false` to see the real state.
Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
override that reports nothing is one nobody withdraws. It changes no other obligation: still one