fix(refresh): report an assumed-robots host once, not twice
The run just made reported each of the four Fandom hosts twice — once from the warning pushed in runRefresh and once from a second loop in main() that printed summary.assumedRobots again. Eight lines for four facts, and the header above them said "5 warnings" while nine printed under it, so the count a reader uses to judge whether to read the list was wrong in the direction that matters. Mine, from the commit that added the flag: I put the reminder in main() to make it loud, having already made it loud in the summary. The instruction folds into the single warning instead, so it still says to go re-read the file in a browser, and it now also reaches the job summary and annotations, which only ever read summary.warnings. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a46b7a8214
commit
f38fe49d30
@@ -281,7 +281,9 @@ export async function runRefresh(
|
||||
summary.assumedRobots = [...cycle.assumedRobots].sort();
|
||||
for (const host of summary.assumedRobots) {
|
||||
summary.warnings.push(
|
||||
`${host}: fetched on --assume-robots-on-403; its robots.txt was not read this run`,
|
||||
`${host}: fetched on --assume-robots-on-403 — its robots.txt was NOT read ` +
|
||||
`this run. Re-read it in a browser and confirm AGENTS.md § Scraping ` +
|
||||
`conduct still describes it.`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -886,13 +888,10 @@ async function main(): Promise<number> {
|
||||
`\n${summary.changed} changed, ${summary.confirmed}/${summary.attempted} confirmed, ` +
|
||||
`${summary.warnings.length} warnings, ${summary.broken.length} broken`,
|
||||
);
|
||||
// One line per warning, and every warning is in `summary.warnings` — printing
|
||||
// a category separately here once double-reported the assumed-robots hosts
|
||||
// and left the "N warnings" count above disagreeing with the lines under it.
|
||||
for (const warning of summary.warnings) console.warn(` ! ${warning}`);
|
||||
for (const host of summary.assumedRobots) {
|
||||
console.warn(
|
||||
` ! ${host}: robots.txt was NOT read this run. Re-read it in a browser ` +
|
||||
`and confirm AGENTS.md § Scraping conduct still describes it.`,
|
||||
);
|
||||
}
|
||||
for (const b of summary.broken) {
|
||||
console.error(
|
||||
` !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` +
|
||||
|
||||
Reference in New Issue
Block a user