docs: the Fandom 403 was curl's, and it is per-address
This file recorded a 403 as Fandom's posture, holding across every wiki and every address, which made those four sources permanently unschedulable. It was measured with curl, and curl is not what fetches: on one address, in the same minute, curl takes a 403 with a bare, a Chrome and our own User-Agent alike, while Bun's fetch is served the real robots.txt on all five hosts and the gate passes with no override. Two variables were folded into one there, and they are worth holding apart. The client mattered — but so does the address, and the client does not rescue it: the 17:46 UTC run on ubuntu-latest still skipped all four Fandom sources using that same Bun client. So a claim that these refresh on a schedule is a claim about the address the runner has, and on the new one it is untested rather than established. The next cycle settles it. Three traps recorded so the episode is not repeated: verify a fetch gate with the real RobotsCache from the address that will run it, never a shell client; the User-Agent is not a lever in either direction, so a 200 that appears when you drop it is the challenge's probabilistic half and not a header you tuned; and neither block is a rate limit, since every failure is first contact with that host, the spacing is already applied, and a challenge carries no Retry-After. More delay buys nothing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
47967a3136
commit
617b529dfd
+7
-3
@@ -181,9 +181,11 @@ source immediately — full coverage, no template dependency.
|
||||
>
|
||||
> Three notes for whoever touches it next, all of them corrections or additions to what follows:
|
||||
>
|
||||
> - **The 403 is on `robots.txt` only.** `api.php?action=parse` answers our own User-Agent with a
|
||||
> `200` from a datacentre address; it is the robots file that is challenged. So the adapter could
|
||||
> be written and fixture-backed from here, and what it cannot do is pass the gate at refresh time.
|
||||
> - **The 403 is per-address and per-client — corrected 2026-08-19.** `api.php?action=parse` answers
|
||||
> our own User-Agent with a `200`, and on a served address so does `/robots.txt` when the runner's
|
||||
> own client asks: `curl` is challenged on that route and Bun's `fetch` is not. That does **not**
|
||||
> make it schedulable — the 17:46 UTC run on `ubuntu-latest` still reported `skipped_robots` for all
|
||||
> four Fandom sources with that same client. `AGENTS.md` § Fandom carries both measurements.
|
||||
> - **The zone is in the column header, not the cell** — `Start(UTC+9)` / `End(UTC+9)`. That makes
|
||||
> the header load-bearing: a table that stops naming its zone is refused rather than read as UTC.
|
||||
> - **Story-event starts carry no clock, only their ends do.** The start keeps the day the page
|
||||
@@ -747,6 +749,8 @@ Facts stated above that a future reader may want to re-check, with how they were
|
||||
| `fandom.ts`, `akwiki`, `wikigg`, `bawiki`, `holodori` cannot read the new pages | `canParse` returned `false` for every combination tried |
|
||||
| CI can fetch wiki.gg and Miraheze | `git log --name-only -- snapshots/` — `github-actions[bot]` commits carry `arknights-akwiki`, `endfield-wikigg`, `ba-bawiki` |
|
||||
| A Fandom `robots.txt` 403 means the source is skipped, not failed | `src/ingest/robots.ts` `RobotsCache.load` — `status >= 400` other than 404/410 → `usable: false` |
|
||||
| On a served address Fandom answers the runner's own client, and only `curl` was challenged | `bun -e` with `RobotsCache` + `DEFAULT_USER_AGENT` → `allowed`, `robots.txt ok`, all four sources; `curl` → `403` 8/8 with both a bare and a Chrome `User-Agent` |
|
||||
| A served address is still a precondition, so that is not schedulability | run `a60eb66` (17:46 UTC, `ubuntu-latest`) → `skipped_robots` on all four Fandom sources, same Bun client, alongside nine game8 `202`s |
|
||||
| All thirteen § 12 hubs exist | `GET https://game8.co/games/<Name>` → `200` for every one |
|
||||
| The `game8` parser already reads the MementoMori events page | ran `game8Parser.parse` against the fetched bytes: 7 events from the 7 data rows of `Current Events \| Duration` |
|
||||
| The FEH page is unreadable by shape, not by two header words | `columnLayout` finds no range column at all — `Availability` is a start and `End` is its own column, and `readStartEndTable` wants the rowspan `Start`/`End` pair |
|
||||
|
||||
Reference in New Issue
Block a user