refresh: let the cron stand on the recorded robots permission
Four Fandom sources have skipped on a challenged robots.txt every cycle, so their calendars are only ever as fresh as the last manual run. For a product whose promise is a trustworthy end date, four stale lanes are the worse failure — so the scheduled run now passes --assume-robots-on-403 and the owner re-reads those files by hand over time. Kept asymmetric on purpose: --force is still refused on an unattended run. Forcing every cycle really is just a shorter interval with extra steps, and nothing about this decision touches that. The accepted risk is narrower than "crawling against robots.txt", and worth stating precisely. A plain 403 still fails closed, so a host that turns us away still stops the run — that is what the challenge-or-refusal split buys. What is invisible is a robots.txt edited to disallow us, because from a challenged address a withdrawal looks exactly like the challenge we already expect. No code can catch that; only the re-read can. Which makes the per-cycle warning the compensating control rather than a courtesy, so it is now pinned as one: every host named, and surviving as a run-page annotation on a completely green cycle where nothing else draws the eye. A dispatch can set the input to false to see the real state. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f12398a29b
commit
3e463adf47
+8
-7
@@ -292,13 +292,14 @@ section but must never claim the event title.
|
||||
challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not
|
||||
excused.
|
||||
- It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no.
|
||||
- **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a
|
||||
local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub
|
||||
names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is
|
||||
load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a
|
||||
cron standing on the recorded permission could never see the host withdraw it, and the recorded
|
||||
permission has no expiry. `refresh.yml` therefore exposes both overrides as
|
||||
`workflow_dispatch` inputs, which are empty on the cron and so can never be passed by it.
|
||||
- **The cron passes it, unlike `--force`.** A standing decision (2026-08-20): four Fandom sources
|
||||
skip on a challenged `robots.txt` every cycle, and four stale calendars are the worse failure for
|
||||
a product that sells trustworthy end dates. `--force` remains gated on `runAttendance` — a person
|
||||
or a `workflow_dispatch`, never a `schedule` — because forcing every cycle is just a shorter
|
||||
interval. The accepted risk is narrow and named: a plain `403` still fails closed, but a
|
||||
`robots.txt` *edited* to disallow us is invisible from a challenged address, so the owner re-reads
|
||||
those files by hand. The per-cycle `::warning` naming every assumed host is the compensating
|
||||
control and is pinned by tests; the dispatch input can be set to `false` to see the real state.
|
||||
|
||||
Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
|
||||
override that reports nothing is one nobody withdraws. It changes no other obligation: still one
|
||||
|
||||
Reference in New Issue
Block a user