fix(refresh): report an assumed-robots host once, not twice

The run just made reported each of the four Fandom hosts twice — once from the
warning pushed in runRefresh and once from a second loop in main() that printed
summary.assumedRobots again. Eight lines for four facts, and the header above
them said "5 warnings" while nine printed under it, so the count a reader uses
to judge whether to read the list was wrong in the direction that matters.

Mine, from the commit that added the flag: I put the reminder in main() to make
it loud, having already made it loud in the summary.

The instruction folds into the single warning instead, so it still says to go
re-read the file in a browser, and it now also reaches the job summary and
annotations, which only ever read summary.warnings.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-19 05:07:49 +02:00
co-authored by Claude Opus 5
parent a46b7a8214
commit f38fe49d30
2 changed files with 14 additions and 9 deletions
+6 -7
View File
@@ -281,7 +281,9 @@ export async function runRefresh(
summary.assumedRobots = [...cycle.assumedRobots].sort();
for (const host of summary.assumedRobots) {
summary.warnings.push(
`${host}: fetched on --assume-robots-on-403; its robots.txt was not read this run`,
`${host}: fetched on --assume-robots-on-403 its robots.txt was NOT read ` +
`this run. Re-read it in a browser and confirm AGENTS.md § Scraping ` +
`conduct still describes it.`,
);
}
@@ -886,13 +888,10 @@ async function main(): Promise<number> {
`\n${summary.changed} changed, ${summary.confirmed}/${summary.attempted} confirmed, ` +
`${summary.warnings.length} warnings, ${summary.broken.length} broken`,
);
// One line per warning, and every warning is in `summary.warnings` — printing
// a category separately here once double-reported the assumed-robots hosts
// and left the "N warnings" count above disagreeing with the lines under it.
for (const warning of summary.warnings) console.warn(` ! ${warning}`);
for (const host of summary.assumedRobots) {
console.warn(
` ! ${host}: robots.txt was NOT read this run. Re-read it in a browser ` +
`and confirm AGENTS.md § Scraping conduct still describes it.`,
);
}
for (const b of summary.broken) {
console.error(
` !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` +
+8 -2
View File
@@ -363,9 +363,15 @@ describe("robots", () => {
expect(summary.outcomes[0]?.result).toBe("fetched");
expect(summary.assumedRobots).toEqual(["game8.co"]);
expect(summary.warnings.some((w) => w.includes("--assume-robots-on-403"))).toBe(
true,
// Exactly one warning per host, and it carries the instruction. The host
// was reported twice for a while — once from here and once from a second
// loop in main() — which also left the run's "N warnings" count disagreeing
// with the number of lines printed under it.
const assumed = summary.warnings.filter((w) =>
w.includes("--assume-robots-on-403"),
);
expect(assumed).toHaveLength(1);
expect(assumed[0]).toContain("Re-read it in a browser");
});
test("an ordinary run reports no assumed hosts at all", async () => {