refresh: let the cron stand on the recorded robots permission
Four Fandom sources have skipped on a challenged robots.txt every cycle, so their calendars are only ever as fresh as the last manual run. For a product whose promise is a trustworthy end date, four stale lanes are the worse failure — so the scheduled run now passes --assume-robots-on-403 and the owner re-reads those files by hand over time. Kept asymmetric on purpose: --force is still refused on an unattended run. Forcing every cycle really is just a shorter interval with extra steps, and nothing about this decision touches that. The accepted risk is narrower than "crawling against robots.txt", and worth stating precisely. A plain 403 still fails closed, so a host that turns us away still stops the run — that is what the challenge-or-refusal split buys. What is invisible is a robots.txt edited to disallow us, because from a challenged address a withdrawal looks exactly like the challenge we already expect. No code can catch that; only the re-read can. Which makes the per-cycle warning the compensating control rather than a courtesy, so it is now pinned as one: every host named, and surviving as a run-page annotation on a completely green cycle where nothing else draws the eye. A dispatch can set the input to false to see the real state. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f12398a29b
commit
3e463adf47
@@ -864,15 +864,21 @@ async function main(): Promise<number> {
|
||||
return 2;
|
||||
}
|
||||
|
||||
if (args.assumeRobotsOn403 && !attendance.attended) {
|
||||
console.error(
|
||||
`--assume-robots-on-403 needs a person to ask for it; this run came from ` +
|
||||
`${attendance.why}.\n` +
|
||||
"It asserts a permission a person read by hand, and a schedule cannot " +
|
||||
"re-read it. Dispatch the workflow by hand to pass it.",
|
||||
);
|
||||
return 2;
|
||||
}
|
||||
// `--assume-robots-on-403` is deliberately NOT gated on attendance, unlike
|
||||
// `--force` above. The four Fandom sources skip on a challenged robots.txt
|
||||
// every cycle, and the repository owner has taken on re-reading those files by
|
||||
// hand over time rather than have four calendars go stale between manual runs
|
||||
// (decided 2026-08-20). That is a standing decision about this project's own
|
||||
// conduct, so the code records it instead of re-litigating it per run.
|
||||
//
|
||||
// What keeps it honest is the disclosure, which is why that is not optional:
|
||||
// every cycle names each host it stood on a hand-recorded permission for and
|
||||
// says the file was not read this run. The risk being accepted is specific —
|
||||
// from a challenged address robots.txt never arrives, so a file *edited* to
|
||||
// disallow us would be invisible (a plain 403 still fails closed, and the
|
||||
// challenge-or-refusal split is what guarantees that much). The per-cycle
|
||||
// warning is the only thing that ever prompts the re-read, so do not demote it
|
||||
// to a log line.
|
||||
|
||||
// Who authorised the override is part of the record, not a detail. A local
|
||||
// shell leaves no trace anybody else can read; a dispatch names the actor.
|
||||
|
||||
Reference in New Issue
Block a user