Fetched by the runner rather than by hand, so the whole path is evidenced:
robots allowed at Crawl-delay 1, the 307 followed to the live version, canParse
satisfied, eventCount 16. The lane now builds from live bytes instead of
falling back to the checked-in fixture.
Kept out of the source commit because a snapshot is data this repository
rewrites on a schedule, and the fixture is the pinned copy that proves the
parser.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fetched under `--force`, authorised interactively — the held snapshot was the
1.4 page and predates the parser fixes, so the lane was serving one event while
the page listed a version's worth. It now reads 8, and the source's failure
count is back to zero.
`Sanity Supply` is not among them, and was in the previous snapshot. It has left
the upcoming table for the current-events card grid, which the parser still
cannot read, and wiki.gg does not list it — so an event starting 2026-08-26 is
unpublished until that shape is supported. Endfield's lane is 13 events rather
than the 14 it should be.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
First run of --force and --assume-robots-on-403 together, and the first time
every source has been confirmed in one cycle: 19/19, 0 broken. Nine game8 pages
answered 200 from a normal address, which is the CI block behaving exactly as
§ Scraping conduct describes it, and all four Fandom sources fetched on the
recorded permission.
Nothing a reader sees changed. No source's event count moved, and diffing the
parsed events of all eight changed pages against their previous snapshots finds
no event added, removed, or re-dated. The byte deltas are between -286 and +32
on pages of a quarter-megabyte — page furniture, not schedule.
That is worth stating rather than glossing: the value of this commit is the
negative result. Nine of these games are fixture-backed in CI and only ever
confirmed by hand, so "we looked, and the calendar is still right" is the thing
the refresh exists to establish, and lastConfirmedAt is what the footer reads to
tell a reader how old the answer is.
fgo is the odd one: identical length, different hash. Every one of its 101
changed characters sits in MediaWiki's `NewPP limit report` comment at the foot
of the body — cache timestamp and parser timings, fixed-width, outside the text
we parse. So a Fandom source will report `fetched` and produce a diff on any
forced refresh an hour apart, whether or not the wiki moved. Worth knowing
before reading a future Fandom diff as news.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both were built on 2026-08-19 and have been served from their fixtures ever
since, because CI cannot read Fandom's robots.txt and the gate fails closed.
Refreshed by hand from an address Fandom serves, which is the only way these
two ever move.
Each body is a real action=parse envelope and each parses to exactly what its
fixture does — three events for Nikke, seven for Infinity Nikki, same titles and
same dates. That is the expected result rather than a weak one: both fixtures
were captured from these pages today, so agreement is the check that the stored
bytes are the wiki and not a challenge page.
All four Fandom sources now have a snapshot. None of them will get another one
from the schedule.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Retiring nikki-game8-events left its snapshot behind. snapshots/ is defined as
the current page per source, and there is no such source — so these two files
are 200 KB of a wiki that stopped being updated in August 2025, tracked in git,
described by metadata naming a sourceId nothing resolves.
Harmless today: build-feed.ts iterates ADAPTERS, so an orphan is never read.
That is also what makes it worth removing rather than leaving. The one path
that would read it again is someone re-registering the id, and what they would
silently get back is the stale page the lane was moved off — the failure the
move was made to fix, arriving through a file nobody looked at.
The fixture stays. fixtures/nikki/game8-events-2026-08-17 is pinned and
permanent, it is the only page here carrying Game8's labelled "Start: ... End:
Permanent" shape, and test/adapters still drives it through the parser.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Eleven sources were inside the 6h floor and were left alone. Of the eight the
runner planned, five answered: Reverse: 1999, GFL2, Stella Sora, Chaos Zero
Nightmare and Umamusume — the last four getting their first snapshot, so those
games stop being built from checked-in fixtures.
Fate/Grand Order, Infinity Nikki and Nikke were skipped on a 403 from
robots.txt, which is the documented Fandom posture and fails closed as intended.
Reverse: 1999 is the one Fandom wiki that answered this time; its page bytes
moved but it parses to the same six events, so nothing on the calendar changed
for it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A normal `bun run refresh` from an address game8 serves, which is the only way
these nine pages move — the scheduled run in CI gets a 202 from game8's edge
and never lands them.
The two Fandom sources reported skipped_robots, as they now do from every
address available here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The guidance in here is not Claude-specific — it is what this project is,
what it refuses to do, and the rules that are invisible from the code. Any
agent working here needs it, and AGENTS.md is the name they all look for.
CLAUDE.md stays as a pointer so Claude Code still finds it, and holds no
guidance of its own; two copies would drift and the wrong one would be read.
Every cross-reference in the source comments, docs, workflow and skills now
names AGENTS.md, and the image ignores both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
One real cycle against game8.co while verifying the runner: 200, nine events,
stored. Genshin was the last game reading from a three-day-old fixture that had
a live source available.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every snapshot is written to a sibling `.tmp-*` and renamed into place, so a
run killed mid-write leaves one behind. `refresh.yml` commits the whole
snapshots/ directory, which would pin that truncated page in git forever — and
the point of the atomic write was that a crash leaves nothing a reader can
mistake for a page.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A review of code that has never made a live request. Each of these was
reproduced before it was fixed.
robots.txt matched a group by testing whether our whole User-Agent
contained the group's name. Our contact URL carries "StereotypicalCat",
so a wiki writing `User-agent: cat` matched us — and because a named
group *replaces* the wildcard group, that silently discarded every rule
under `User-agent: *`. Match the RFC 9309 product token instead. A 200
carrying an HTML "not found" page also parsed to zero rules and read as
permission; it now fails closed, while a genuinely empty body still
means "no restrictions".
The response body was read outside the try that guarded the request, so
one truncated body aborted the whole cycle: later sources were never
fetched, and the failed source never recorded its check, meaning a
re-dispatch would ask that wiki again minutes later.
Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A
page served in a legacy charset became replacement characters with the
original bytes gone — and mojibake in a title flows into slugify and
moves every localStorage key for that source. Decode by the declared
charset, keep the served bytes verbatim, and hash those.
Unchanged bytes skipped the metadata write, so once a server rotated its
ETag we sent a stale validator forever and it served full bodies instead
of 304s.
Also: snapshot writes go through a temp file and a rename, and `--only`
with no value is an argument error rather than "all sources".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.
bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.
Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.
A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.
Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
github-actions[bot]