refresh: let the cron stand on the recorded robots permission

Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.

Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.

The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.

Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-20 00:12:28 +02:00
co-authored by Claude Opus 5
parent f12398a29b
commit 3e463adf47
5 changed files with 102 additions and 44 deletions
+15 -9
View File
@@ -864,15 +864,21 @@ async function main(): Promise<number> {
return 2;
}
if (args.assumeRobotsOn403 && !attendance.attended) {
console.error(
`--assume-robots-on-403 needs a person to ask for it; this run came from ` +
`${attendance.why}.\n` +
"It asserts a permission a person read by hand, and a schedule cannot " +
"re-read it. Dispatch the workflow by hand to pass it.",
);
return 2;
}
// `--assume-robots-on-403` is deliberately NOT gated on attendance, unlike
// `--force` above. The four Fandom sources skip on a challenged robots.txt
// every cycle, and the repository owner has taken on re-reading those files by
// hand over time rather than have four calendars go stale between manual runs
// (decided 2026-08-20). That is a standing decision about this project's own
// conduct, so the code records it instead of re-litigating it per run.
//
// What keeps it honest is the disclosure, which is why that is not optional:
// every cycle names each host it stood on a hand-recorded permission for and
// says the file was not read this run. The risk being accepted is specific —
// from a challenged address robots.txt never arrives, so a file *edited* to
// disallow us would be invisible (a plain 403 still fails closed, and the
// challenge-or-refusal split is what guarantees that much). The per-cycle
// warning is the only thing that ever prompts the re-read, so do not demote it
// to a log line.
// Who authorised the override is part of the record, not a detail. A local
// shell leaves no trace anybody else can read; a dispatch names the actor.