fix(refresh): report an assumed-robots host once, not twice
The run just made reported each of the four Fandom hosts twice — once from the warning pushed in runRefresh and once from a second loop in main() that printed summary.assumedRobots again. Eight lines for four facts, and the header above them said "5 warnings" while nine printed under it, so the count a reader uses to judge whether to read the list was wrong in the direction that matters. Mine, from the commit that added the flag: I put the reminder in main() to make it loud, having already made it loud in the summary. The instruction folds into the single warning instead, so it still says to go re-read the file in a browser, and it now also reaches the job summary and annotations, which only ever read summary.warnings. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
a46b7a8214
commit
f38fe49d30
@@ -281,7 +281,9 @@ export async function runRefresh(
|
||||
summary.assumedRobots = [...cycle.assumedRobots].sort();
|
||||
for (const host of summary.assumedRobots) {
|
||||
summary.warnings.push(
|
||||
`${host}: fetched on --assume-robots-on-403; its robots.txt was not read this run`,
|
||||
`${host}: fetched on --assume-robots-on-403 — its robots.txt was NOT read ` +
|
||||
`this run. Re-read it in a browser and confirm AGENTS.md § Scraping ` +
|
||||
`conduct still describes it.`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -886,13 +888,10 @@ async function main(): Promise<number> {
|
||||
`\n${summary.changed} changed, ${summary.confirmed}/${summary.attempted} confirmed, ` +
|
||||
`${summary.warnings.length} warnings, ${summary.broken.length} broken`,
|
||||
);
|
||||
// One line per warning, and every warning is in `summary.warnings` — printing
|
||||
// a category separately here once double-reported the assumed-robots hosts
|
||||
// and left the "N warnings" count above disagreeing with the lines under it.
|
||||
for (const warning of summary.warnings) console.warn(` ! ${warning}`);
|
||||
for (const host of summary.assumedRobots) {
|
||||
console.warn(
|
||||
` ! ${host}: robots.txt was NOT read this run. Re-read it in a browser ` +
|
||||
`and confirm AGENTS.md § Scraping conduct still describes it.`,
|
||||
);
|
||||
}
|
||||
for (const b of summary.broken) {
|
||||
console.error(
|
||||
` !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` +
|
||||
|
||||
@@ -363,9 +363,15 @@ describe("robots", () => {
|
||||
|
||||
expect(summary.outcomes[0]?.result).toBe("fetched");
|
||||
expect(summary.assumedRobots).toEqual(["game8.co"]);
|
||||
expect(summary.warnings.some((w) => w.includes("--assume-robots-on-403"))).toBe(
|
||||
true,
|
||||
// Exactly one warning per host, and it carries the instruction. The host
|
||||
// was reported twice for a while — once from here and once from a second
|
||||
// loop in main() — which also left the run's "N warnings" count disagreeing
|
||||
// with the number of lines printed under it.
|
||||
const assumed = summary.warnings.filter((w) =>
|
||||
w.includes("--assume-robots-on-403"),
|
||||
);
|
||||
expect(assumed).toHaveLength(1);
|
||||
expect(assumed[0]).toContain("Re-read it in a browser");
|
||||
});
|
||||
|
||||
test("an ordinary run reports no assumed hosts at all", async () => {
|
||||
|
||||
Reference in New Issue
Block a user