docs: the Fandom 403 was curl's, and it is per-address

This file recorded a 403 as Fandom's posture, holding across every wiki and
every address, which made those four sources permanently unschedulable. It was
measured with curl, and curl is not what fetches: on one address, in the same
minute, curl takes a 403 with a bare, a Chrome and our own User-Agent alike,
while Bun's fetch is served the real robots.txt on all five hosts and the gate
passes with no override.

Two variables were folded into one there, and they are worth holding apart. The
client mattered — but so does the address, and the client does not rescue it:
the 17:46 UTC run on ubuntu-latest still skipped all four Fandom sources using
that same Bun client. So a claim that these refresh on a schedule is a claim
about the address the runner has, and on the new one it is untested rather than
established. The next cycle settles it.

Three traps recorded so the episode is not repeated: verify a fetch gate with
the real RobotsCache from the address that will run it, never a shell client;
the User-Agent is not a lever in either direction, so a 200 that appears when
you drop it is the challenge's probabilistic half and not a header you tuned;
and neither block is a rate limit, since every failure is first contact with
that host, the spacing is already applied, and a challenge carries no
Retry-After. More delay buys nothing.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-19 23:38:11 +02:00
co-authored by Claude Opus 5
parent 47967a3136
commit 617b529dfd
3 changed files with 107 additions and 32 deletions
+7
View File
@@ -297,6 +297,13 @@ section but must never claim the event title.
Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
override that reports nothing is one nobody withdraws. It changes no other obligation: still one
request per source, still six hours apart, still spaced per host.
**Two variables decide whether it is needed at all, and they were once folded into one.** The
client matters — a `curl` measurement said Fandom refused us while the runner's own Bun client was
served the file — and so does the address, which the client does not rescue: `ubuntu-latest` still
reported `skipped_robots` on all four Fandom sources using that same client. So verify a gate with
`bun -e` and the real `RobotsCache` rather than a shell client, and verify it **from the address
that will run it**. AGENTS.md § Fandom has both measurements.
- 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says
one per source per cycle. A failed source waits for the next cycle instead.
- **A `403` on `robots.txt` records which kind it was** — `an interstitial challenge`, `a refusal`,