docs: the Fandom 403 was curl's, and it is per-address
This file recorded a 403 as Fandom's posture, holding across every wiki and every address, which made those four sources permanently unschedulable. It was measured with curl, and curl is not what fetches: on one address, in the same minute, curl takes a 403 with a bare, a Chrome and our own User-Agent alike, while Bun's fetch is served the real robots.txt on all five hosts and the gate passes with no override. Two variables were folded into one there, and they are worth holding apart. The client mattered — but so does the address, and the client does not rescue it: the 17:46 UTC run on ubuntu-latest still skipped all four Fandom sources using that same Bun client. So a claim that these refresh on a schedule is a claim about the address the runner has, and on the new one it is untested rather than established. The next cycle settles it. Three traps recorded so the episode is not repeated: verify a fetch gate with the real RobotsCache from the address that will run it, never a shell client; the User-Agent is not a lever in either direction, so a 200 that appears when you drop it is the challenge's probabilistic half and not a header you tuned; and neither block is a rate limit, since every failure is first contact with that host, the spacing is already applied, and a challenge carries no Retry-After. More delay buys nothing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
47967a3136
commit
617b529dfd
@@ -297,6 +297,13 @@ section but must never claim the event title.
|
||||
Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
|
||||
override that reports nothing is one nobody withdraws. It changes no other obligation: still one
|
||||
request per source, still six hours apart, still spaced per host.
|
||||
|
||||
**Two variables decide whether it is needed at all, and they were once folded into one.** The
|
||||
client matters — a `curl` measurement said Fandom refused us while the runner's own Bun client was
|
||||
served the file — and so does the address, which the client does not rescue: `ubuntu-latest` still
|
||||
reported `skipped_robots` on all four Fandom sources using that same client. So verify a gate with
|
||||
`bun -e` and the real `RobotsCache` rather than a shell client, and verify it **from the address
|
||||
that will run it**. AGENTS.md § Fandom has both measurements.
|
||||
- 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says
|
||||
one per source per cycle. A failed source waits for the next cycle instead.
|
||||
- **A `403` on `robots.txt` records which kind it was** — `an interstitial challenge`, `a refusal`,
|
||||
|
||||
Reference in New Issue
Block a user