Lucas WintherandClaude Opus 5
a46b7a8214
First run of --force and --assume-robots-on-403 together, and the first time every source has been confirmed in one cycle: 19/19, 0 broken. Nine game8 pages answered 200 from a normal address, which is the CI block behaving exactly as § Scraping conduct describes it, and all four Fandom sources fetched on the recorded permission. Nothing a reader sees changed. No source's event count moved, and diffing the parsed events of all eight changed pages against their previous snapshots finds no event added, removed, or re-dated. The byte deltas are between -286 and +32 on pages of a quarter-megabyte — page furniture, not schedule. That is worth stating rather than glossing: the value of this commit is the negative result. Nine of these games are fixture-backed in CI and only ever confirmed by hand, so "we looked, and the calendar is still right" is the thing the refresh exists to establish, and lastConfirmedAt is what the footer reads to tell a reader how old the answer is. fgo is the odd one: identical length, different hash. Every one of its 101 changed characters sits in MediaWiki's `NewPP limit report` comment at the foot of the body — cache timestamp and parser timings, fixed-width, outside the text we parse. So a Fandom source will report `fetched` and produce a diff on any forced refresh an hour apart, whether or not the wiki moved. Worth knowing before reading a future Fandom diff as news. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Snapshots
Raw pages, exactly as fetched. scripts/refresh-sources.ts writes them; nothing else fetches.
<source-id>.html the body verbatim — tracked
<source-id>.meta.json hash, size, charset, ETag, Last-Modified, when the bytes last changed — tracked
<source-id>.state.json when we last checked, and failure streak — gitignored
"Verbatim" means the bytes as served, not text we re-encoded. charset in the metadata records
what those bytes are in — the Content-Type header, else a <meta charset> in the page, else
UTF-8 — and bytes is the served length. A page in Shift_JIS or Latin-1 decoded as UTF-8 would be
stored as a field of U+FFFD with the original bytes gone; re-parsing could never recover it, and the
mojibake would reach slugify, moving every event ID for that source (AGENTS.md § Event IDs are
localStorage keys).
Every file is written to a sibling .tmp-* and renamed into place, body before metadata, so an
interrupted run leaves a stray temp file rather than a truncated snapshot or metadata describing
bytes that were never stored. Those temp files are gitignored: refresh.yml commits the whole
directory, so otherwise a run killed mid-write would pin a half-page in git forever.
Three reasons this is committed rather than cached:
- Re-parsing never re-fetches. Iterating on a parser reads these files, not the wikis (AGENTS.md § Scraping conduct).
- A refresh is reviewable. The commit diff is the page diff, so "an event vanished" is a question you can answer from git rather than from a wiki that has since changed again.
- The build stays offline.
bun run build:feedparses whichever of these exists and falls back tofixtures/otherwise, so a clean checkout and the container build work with no network.
The .state.json files are the exception: they change every cycle whether or not a page did, and
committing them would mean a commit per run saying nothing happened. CI keeps them in the actions
cache instead: refresh.yml saves that cache, and ci.yml restores it read-only before building
the feed. Both halves matter — lastConfirmedAt lives only there, and without the restore the feed
falls back to contentChangedAt and the UI calls every source stale two days after its bytes last
moved.
The metadata is rewritten on an unchanged page in one case: the server rotating an ETag or
Last-Modified while serving the same bytes. Keeping the old validator would mean sending a stale
If-None-Match forever and being served the whole page every cycle, so that diff is worth the
commit — contentChangedAt and the body stay put, so it is still visibly not a content change.
Fixtures are not the same thing. A fixture is pinned to a date and kept forever as the regression test for a page shape; a snapshot is the current page and is overwritten each time it changes.