feat(refresh): --assume-robots-on-403, for a host that hides only its robots.txt

Fandom answers our fetcher 403 on /robots.txt itself, from every address we
have, while api.php?action=parse answers the same User-Agent with a 200. The
gate fails closed on the unreadable file, so r1999, fgo, nikke and nikki can
never refresh — even though their rules are not unknown: a person read them in
a browser and wrote them into AGENTS.md, verbatim, which is how three of those
four were cleared in the first place.

This flag is that recorded permission, and nothing wider. What it deliberately
does not do is most of the design:

  - 403 only. A 401, a 5xx, a timeout or a soft 404 still mean we do not know
    what the site permits, and unknown is still not permission.
  - It never overrides a robots.txt we could read. A file that answers and
    disallows us is an answer, and it still wins. So this is no use for game8.co
    — whose robots.txt reads fine and welcomes us while its edge refuses the
    pages — and it must not become one.
  - Refused under CI. It stands in for a human having read a file this morning,
    and there is no human on a runner. A scheduled job asserting a permission
    nobody re-checked is exactly how "temporary" stops being temporary.
  - Loud. Every host it applied to is warned about by name, in the run log and
    in the summary, with a line saying to go re-read the file. An override that
    reports nothing is one nobody withdraws.

Nothing else about being a guest relaxes: one request per source, six hours
apart, spaced per host, conditional headers, no retries.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

AGENTS.md

@@ -119,7 +119,7 @@ src/client/       React app, service worker, manifest
                   theme.ts — dark or light, and what a game hue reads as on each
 scripts/          build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
 serve.ts          static server + /api/health
-test/             716 tests
+test/             724 tests
 fixtures/<game>/  raw HTML + .expected.json per source — pinned, kept forever
 snapshots/        current page per source, rewritten by refresh — see its README
 ```
@@ -385,6 +385,16 @@ fixture-backed* from any address; what it cannot do is pass the robots gate at r
 fails closed and skips. The permission is therefore a thing a human records once, and the freshness
 is a thing that needs an address Fandom serves.
 
+`--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing
+that helps: `bun run refresh --assume-robots-on-403` treats a `403` **on `/robots.txt` itself** as
+the permission recorded above rather than failing closed. It is not a workaround for a host that
+turned us away — it never overrides a `robots.txt` we could read, so a file that disallows us still
+says no, and it does nothing at all for game8.co, whose robots.txt reads fine and welcomes us while
+its edge refuses the pages. It is refused under CI, because what it stands in for is a person having
+read a file in a browser, and there is no person on a runner. Every host it applied to is named in
+the run's warnings, so it stays a thing somebody decided this morning rather than a default. Nothing
+else relaxes: one request per source, six hours apart, spaced per host, no retries.
+
 One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
 robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
 broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if
@@ -559,7 +569,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t
 
 `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
 retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
-read). Anything that would make it fetch more often is a change to this section first.
+read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first.
 
 **A source down is a warning; a source down for days is a broken build.** One wiki failing must
 never blank a calendar or stop the sources that did answer from being committed — so a failure is