This file recorded a 403 as Fandom's posture, holding across every wiki and
every address, which made those four sources permanently unschedulable. It was
measured with curl, and curl is not what fetches: on one address, in the same
minute, curl takes a 403 with a bare, a Chrome and our own User-Agent alike,
while Bun's fetch is served the real robots.txt on all five hosts and the gate
passes with no override.
Two variables were folded into one there, and they are worth holding apart. The
client mattered — but so does the address, and the client does not rescue it:
the 17:46 UTC run on ubuntu-latest still skipped all four Fandom sources using
that same Bun client. So a claim that these refresh on a schedule is a claim
about the address the runner has, and on the new one it is untested rather than
established. The next cycle settles it.
Three traps recorded so the episode is not repeated: verify a fetch gate with
the real RobotsCache from the address that will run it, never a shell client;
the User-Agent is not a lever in either direction, so a 200 that appears when
you drop it is the challenge's probabilistic half and not a header you tuned;
and neither block is a rate limit, since every failure is first contact with
that host, the spacing is already applied, and a challenge carries no
Retry-After. More delay buys nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
