The build log prints a count per source, and CI now declines to fail on
a page that states its own emptiness — which leaves that source showing
an unexplained 0, reading as exactly the fault the gate just decided it
was not. The zero we are content with is the one that has to say why.
It is also the only line that would ever prompt anyone to ask whether a
lane quiet for a month is a game between patches or a wiki that reworded
the sentence a statesNoEvents check is still matching. Nothing else in
the pipeline can tell those apart.
The test count in the layout block was stale at 898 and moves with these
two commits, so it is corrected here.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Refusing every zero-event body is right when the alternative is silently
emptying a calendar, and wrong when the page has told us it is empty. A
gacha game goes quiet between versions, and read strictly that is a source
failing every cycle until the next patch ships: three of them reach the
broken tier and fail the workflow over a lane that is correctly empty,
while the snapshot being held ages out of date. Infinity Nikki sat in
exactly that state for four cycles.
So an empty parse is stored when the source's `statesNoEvents` vouches for
it, and the cycle counts as confirmed rather than failed. Everything else
is unchanged: a body that parses to nothing on its own still keeps the
previous snapshot, and a first fetch that yields nothing still stores
nothing.
The gate turns on the page's statement, never on the adapter merely being
able to make one — otherwise implementing `statesNoEvents` would quietly
switch the zero-events gate off for that source. A test pins that.
The run says which of the two empties it saw. "0 events — down from 6"
reads as the shape change it is not, and that note is what somebody
checking on a quiet lane actually sees.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CI failed on Infinity Nikki yielding no events, and was wrong to. Its
snapshot parses to six events; every one of them had ended by the morning
the build ran. The parser is fine — `--now 2026-08-14` gives six, today
gives none — and the page simply has nothing current left on it.
eventCount is counted after expired events are dropped, so "this parser has
stopped reading a redesigned page" and "this page's events have all
finished" arrived as the same zero. Only the first means our code is wrong,
and only the first should redden a build. So the feed now records what each
document yields parsed as of its own capture date, before expiry, and the
check fails on that instead.
Nikki cannot refresh itself out of this, either: docs/SOURCES.md records
that Fandom refuses the Actions runner. A lane with an empty calendar is a
real problem, but it is a refresh problem, so it is reported on the build
log and left visible rather than thrown.
parsedCount is nullable and defaulted, never required: the client validates
the whole feed with safeParse and the service worker serves the last feed it
downloaded, so a required field would have made every cached feed fail
validation and taken the offline promise with it. Null also covers a source
whose bytes were never confirmed live — there is no date to parse "as of",
and a build is not failed on missing information.
The rule moved to shared/feed.ts. A test did pin the old one, by grepping
the workflow for the string — which proved the check existed, never that it
was right.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.
Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.
The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.
Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Four Fandom sources have been skipping on a challenged robots.txt every cycle,
and the only thing that clears it is a person passing the recorded permission.
Until now there was no way to do that from the workflow, so it meant checking
out the repo and running the refresh by hand. Both overrides are dispatch
inputs now, guarded on the literal string "true" and empty on the cron, so the
schedule still cannot reach either one.
The force-push guard needed narrowing to allow this. It asserted the workflow
contains no "--force" anywhere, as a proxy for never rewriting the branch, and
the refresh runner's own --force flag trips that substring while having nothing
to do with pushing. It now matches force on a git push specifically, including
--force-with-lease, which is still one — a tighter assertion than the string it
replaces, checked against both a real force push and the flag it must ignore.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both overrides stand in for a person deciding something, and the gate that
enforced it asked the coarser question. `isCi()` got the case that matters
wrong: a `workflow_dispatch` sets CI=true, so somebody clicking "Run workflow"
was refused exactly as the cron was, which made both flags unreachable from the
workflow at all. A dispatch is a person, and a better-evidenced one than a local
shell — GitHub records which one, and the run log keeps it.
`runAttendance` draws the line where it belongs: a local shell is a person, a
dispatch is a person and names the actor, and a schedule — or any other runner
event, erring that way on purpose — is not. The run now prints which override
was used and who authorised it, because a local shell leaves no trace anybody
else can read.
The schedule stays refused, and for the robots override that is not ceremony.
From an address that gets a challenge we never receive robots.txt at all, so a
cron standing on the recorded permission has no way to notice the host
withdrawing it, and that permission has no expiry. The challenge-or-refusal
narrowing does not close this: a plain 403 still stops us, but a robots.txt
edited to disallow us would be invisible, because a challenge arrives instead of
a file. A person re-reading it is the only thing that ever re-validates it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The run just made reported each of the four Fandom hosts twice — once from the
warning pushed in runRefresh and once from a second loop in main() that printed
summary.assumedRobots again. Eight lines for four facts, and the header above
them said "5 warnings" while nine printed under it, so the count a reader uses
to judge whether to read the list was wrong in the direction that matters.
Mine, from the commit that added the flag: I put the reminder in main() to make
it loud, having already made it loud in the summary.
The instruction folds into the single warning instead, so it still says to go
re-read the file in a browser, and it now also reaches the job summary and
annotations, which only ever read summary.warnings.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The interval gate has no override, so a page you know has just changed cannot
be fetched until six hours after the last attempt. The only workaround was
deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and
lastConfirmedAt — resetting the broken-source streak and making the footer age
every source from when its bytes last changed rather than when we last
confirmed them. A flag that says what it means is better than a side effect
nobody documented.
It sets aside the interval and nothing else. Conditional headers still go out,
which is what makes forcing defensible at all: the host is asked, not
re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one
request per source and the no-retry rule all still apply — a source that was
not due and is also disallowed stays skipped, for the reason that matters.
Refused under CI, like --assume-robots-on-403 and for the same reason: a
schedule that forces every cycle is a shorter interval with extra steps, and
the interval is the obligation, not the default. So AGENTS.md § Scraping
conduct is amended rather than left to be quietly contradicted by a flag.
Every source asked early is named in summary.forced and warned about. A run
that was due anyway is never reported as forced — a summary that cried "forced"
on an ordinary run would train the reader to ignore the word.
Also repoints the "unknown flag" test, which used --force as its example and
stopped testing anything the moment --force existed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fandom answers our fetcher 403 on /robots.txt itself, from every address we
have, while api.php?action=parse answers the same User-Agent with a 200. The
gate fails closed on the unreadable file, so r1999, fgo, nikke and nikki can
never refresh — even though their rules are not unknown: a person read them in
a browser and wrote them into AGENTS.md, verbatim, which is how three of those
four were cleared in the first place.
This flag is that recorded permission, and nothing wider. What it deliberately
does not do is most of the design:
- 403 only. A 401, a 5xx, a timeout or a soft 404 still mean we do not know
what the site permits, and unknown is still not permission.
- It never overrides a robots.txt we could read. A file that answers and
disallows us is an answer, and it still wins. So this is no use for game8.co
— whose robots.txt reads fine and welcomes us while its edge refuses the
pages — and it must not become one.
- Refused under CI. It stands in for a human having read a file this morning,
and there is no human on a runner. A scheduled job asserting a permission
nobody re-checked is exactly how "temporary" stops being temporary.
- Loud. Every host it applied to is warned about by name, in the run log and
in the summary, with a line saying to go re-read the file. An override that
reports nothing is one nobody withdraws.
Nothing else about being a guest relaxes: one request per source, six hours
apart, spaced per host, conditional headers, no retries.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New bawiki parser plus the ba-bawiki-events source, giving the game its
two live and upcoming events at day precision.
It reads the rendered /wiki/Events page, which is the opposite call to the
Fandom source next door and for the same kind of reason: bluearchive.wiki
is Miraheze, whose robots.txt disallows /w/ and /*?action=, so there the
API is the closed route and the page is the surface * is allowed. It
serves our own User-Agent a 200, sets no Content-Signal and no Crawl-delay
for us. The Fandom wiki was declined earlier as a JP archive yielding
nothing live; this is the live source that assessment pointed at.
Three page facts shape the parser, each a way to publish a confidently
wrong date. The schedule is a JP/Global tabber and the Japanese version
runs four to nine months ahead, so only Global is published — the akwiki
hazard. The Global tab's nav button carries the id
tabber-Global_version-label and precedes both panels, so slicing from the
first matching id reads the Japanese schedule while believing it read
ours; the first version of this parser did exactly that and published a
JP-only event. And three tabs on the page are named Global, the schedule
plus Mini-Event and Joint Firing Drill, so the schedule is found by its
Name (EN) header rather than by position, with canParse asserting the same
lookup — a renamed tab or column fails the run instead of emptying the
lane.
The page states no time of day and no timezone anywhere, which is why the
dates are day precision and why ba gets no resetOffsets. It also settles
the five other schedule tables, which do carry a wall clock but name no
zone and mostly do not say which server: reading those as UTC would invent
the fact that matters most, and rounding to a day would not save it,
because a 04:00 local boundary falls either side of UTC midnight depending
on the offset assumed and the start's day is part of the event ID. They
are left unparsed deliberately.
The two-event count is asserted against an independent extraction off the
fixture, and both starts corroborate against the per-event infoboxes
elsewhere on the page — GL 2026-08-04 and GL 2026-09-01, where the JP
reruns those tabs would have given are 2026-04-01 and 2026-05-06.
Three comment counts were already stale before this and are corrected to
what the tree now holds rather than to what it held yesterday.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The health check earned its keep on the first run: six sources broken, four
cycles each, "last confirmed never" — and a last status of **202**. That is
game8.co's edge answering a GitHub runner with a bot-management body instead of
the wiki.
`response.ok` admitted it. A 202 therefore passed the status gate as a
document, reached the parser, yielded nothing, and was reported as "kept
previous snapshot; new body yielded 0 events" — an accurate description of the
symptom that never mentioned the status explaining it, which is why this looked
like a parser problem for days. Require 200: 202 means the request was accepted
for processing, 204 has no body, 206 is a fragment, and none of them is a page.
The rejection note already names the Server header, so the next cycle reads
`HTTP 202 (AkamaiGHost)` instead.
Recorded in CLAUDE.md § Scraping conduct, including that this must not be worked
around. The per-host spacing added alongside does not help and was never going
to — a 202 on a cycle's first request is address reputation, not rate. Browser
-shaped headers or a residential egress would be defeating an access control,
which is the ground uma.moe was declined on, and game8's robots.txt is
permissive, which makes doing it there worse rather than better.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.
`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.
A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.
Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A review of code that has never made a live request. Each of these was
reproduced before it was fixed.
robots.txt matched a group by testing whether our whole User-Agent
contained the group's name. Our contact URL carries "StereotypicalCat",
so a wiki writing `User-agent: cat` matched us — and because a named
group *replaces* the wildcard group, that silently discarded every rule
under `User-agent: *`. Match the RFC 9309 product token instead. A 200
carrying an HTML "not found" page also parsed to zero rules and read as
permission; it now fails closed, while a genuinely empty body still
means "no restrictions".
The response body was read outside the try that guarded the request, so
one truncated body aborted the whole cycle: later sources were never
fetched, and the failed source never recorded its check, meaning a
re-dispatch would ask that wiki again minutes later.
Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A
page served in a legacy charset became replacement characters with the
original bytes gone — and mojibake in a title flows into slugify and
moves every localStorage key for that source. Decode by the declared
charset, keep the served bytes verbatim, and hash those.
Unchanged bytes skipped the metadata write, so once a server rotated its
ETag we sent a stale validator forever and it served full bodies instead
of 304s.
Also: snapshot writes go through a temp file and a rename, and `--only`
with no value is an argument error rather than "all sources".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.
bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.
Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.
A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.
Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
