The health check earned its keep on the first run: six sources broken, four
cycles each, "last confirmed never" — and a last status of **202**. That is
game8.co's edge answering a GitHub runner with a bot-management body instead of
the wiki.
`response.ok` admitted it. A 202 therefore passed the status gate as a
document, reached the parser, yielded nothing, and was reported as "kept
previous snapshot; new body yielded 0 events" — an accurate description of the
symptom that never mentioned the status explaining it, which is why this looked
like a parser problem for days. Require 200: 202 means the request was accepted
for processing, 204 has no body, 206 is a fragment, and none of them is a page.
The rejection note already names the Server header, so the next cycle reads
`HTTP 202 (AkamaiGHost)` instead.
Recorded in CLAUDE.md § Scraping conduct, including that this must not be worked
around. The per-host spacing added alongside does not help and was never going
to — a 202 on a cycle's first request is address reputation, not rate. Browser
-shaped headers or a residential egress would be defeating an access control,
which is the ground uma.moe was declined on, and game8's robots.txt is
permissive, which makes doing it there worse rather than better.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.
`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.
A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.
Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The shell is served cache-first, which is what makes the app work on a
train and also what makes a deploy invisible: a reader with the tab open
— the reader this app is built for — keeps running the bundle they first
loaded, so a new game or a corrected date reaches their device and sits
there with nothing saying why the page looks unchanged. An old app shown
as current is the same failure as old events shown as current.
So the worker now installs quietly and waits instead of calling
skipWaiting(), the page notices it waiting and says so, and the reader's
tap sends the skip-waiting message and reloads on controllerchange. The
app never reloads itself: someone may be mid-way through typing in one of
their own events, and the notice says what a reload costs (their place on
the page) and what it does not (marks and notes live in localStorage).
Detection is derived rather than remembered. build:static grew into a
script that stamps sw.js with a hash of the built shell, because the
browser only offers a worker whose bytes differ, and the predecessor —
a hand-bumped CACHE_VERSION — had already been forgotten once. The feed
is deliberately not part of that hash: it changes twice a day, needs no
reload, and announcing it would teach readers to dismiss the notice
unread. The cache name stays put for the same reason a per-build one
would be wrong — it holds the feed an offline reader is reading.
A first install is not an update and stays silent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A review of code that has never made a live request. Each of these was
reproduced before it was fixed.
robots.txt matched a group by testing whether our whole User-Agent
contained the group's name. Our contact URL carries "StereotypicalCat",
so a wiki writing `User-agent: cat` matched us — and because a named
group *replaces* the wildcard group, that silently discarded every rule
under `User-agent: *`. Match the RFC 9309 product token instead. A 200
carrying an HTML "not found" page also parsed to zero rules and read as
permission; it now fails closed, while a genuinely empty body still
means "no restrictions".
The response body was read outside the try that guarded the request, so
one truncated body aborted the whole cycle: later sources were never
fetched, and the failed source never recorded its check, meaning a
re-dispatch would ask that wiki again minutes later.
Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A
page served in a legacy charset became replacement characters with the
original bytes gone — and mojibake in a title flows into slugify and
moves every localStorage key for that source. Decode by the declared
charset, keep the served bytes verbatim, and hash those.
Unchanged bytes skipped the metadata write, so once a server rotated its
ETag we sent a stale validator forever and it served full bodies instead
of 304s.
Also: snapshot writes go through a temp file and a rename, and `--only`
with no value is an argument error rather than "all sources".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.
bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.
Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.
A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.
Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
First non-Game8 parser, and the first source that states region-scoped ends:
wiki.gg emits ISO timestamps with one timer per server region, so Endfield
events now carry exact times and separate Asia / Americas-Europe ends. That
gap is up to 13 hours, which is what regionEnds was built for.
Two sources for one game also exercised merge for real. It caught both
overlaps — including "Bedazzling Dawnstar" against Game8's "Bedazzling
Dawnstar Sign-In" — and flagged that the two disagree on the end date by 70
hours rather than averaging them into a date neither states.
Matching an appended qualifier needed a prefix test, not a subset test: a
subset would also fuse "Gold Clash" with "Gold Rush Clash Royale".
Also fixes build-feed picking fixtures by game rather than by source, which
handed the wiki.gg page to the Game8 parser once Endfield had two.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Emits exactly the shape GET /api/events.json will serve, so the client can
be developed against real parsed data before the server and database land.
Reads fixtures only, never the network.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Parsers are keyed by site template, not game: one Game8 parser serves
every Game8 page, and adapters bind a URL and game to a parser. Adding a
source for a known site is one SOURCES entry.
Game8 uses at least three templates. This handles label/value detail
tables (Genshin, 9 events) and column tables (NTE, 13 events). Its
image-grid schedule has no year and no end date, so Arknights: Endfield
is not supportable from that source and gets no adapter.
canParse is a structural tripwire: a site redesign fails the run loudly
instead of publishing an empty calendar that reads as "no events".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
