Both files stated the h4 rule as a flat prohibition — an unrecognised h4
must never claim the event title — which is no longer what the code does.
They now state the narrower claim: an h4 may fill an empty event title but
never take one, with the reason each page needs it. Genshin's labels arrive
after its h3 has claimed the slot; Wuthering Waves' block sits under a
section heading that leaves the slot empty.
INGESTION.md gains the block as shape 9, and the part that is easy to get
wrong on the way back: its heading has to be *in* the section vocabulary
rather than merely fall through, because an unrecognised heading is read as
an event name and would spend the title on the section.
AGENTS.md also gains a bullet for where a missing blurb hides. Two
templates now put unlock conditions in the cell where a description
belongs, and in both the real prose is in a per-event section further down
the page — under an h3 on one and an h4 on the other.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.
Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.
The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.
Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Four Fandom sources have been skipping on a challenged robots.txt every cycle,
and the only thing that clears it is a person passing the recorded permission.
Until now there was no way to do that from the workflow, so it meant checking
out the repo and running the refresh by hand. Both overrides are dispatch
inputs now, guarded on the literal string "true" and empty on the cron, so the
schedule still cannot reach either one.
The force-push guard needed narrowing to allow this. It asserted the workflow
contains no "--force" anywhere, as a proxy for never rewriting the branch, and
the refresh runner's own --force flag trips that substring while having nothing
to do with pushing. It now matches force on a git push specifically, including
--force-with-lease, which is still one — a tighter assertion than the string it
replaces, checked against both a real force push and the flag it must ignore.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both overrides stand in for a person deciding something, and the gate that
enforced it asked the coarser question. `isCi()` got the case that matters
wrong: a `workflow_dispatch` sets CI=true, so somebody clicking "Run workflow"
was refused exactly as the cron was, which made both flags unreachable from the
workflow at all. A dispatch is a person, and a better-evidenced one than a local
shell — GitHub records which one, and the run log keeps it.
`runAttendance` draws the line where it belongs: a local shell is a person, a
dispatch is a person and names the actor, and a schedule — or any other runner
event, erring that way on purpose — is not. The run now prints which override
was used and who authorised it, because a local shell leaves no trace anybody
else can read.
The schedule stays refused, and for the robots override that is not ceremony.
From an address that gets a challenge we never receive robots.txt at all, so a
cron standing on the recorded permission has no way to notice the host
withdrawing it, and that permission has no expiry. The challenge-or-refusal
narrowing does not close this: a plain 403 still stops us, but a robots.txt
edited to disallow us would be invisible, because a challenge arrives instead of
a file. A person re-reading it is the only thing that ever re-validates it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file recorded a 403 as Fandom's posture, holding across every wiki and
every address, which made those four sources permanently unschedulable. It was
measured with curl, and curl is not what fetches: on one address, in the same
minute, curl takes a 403 with a bare, a Chrome and our own User-Agent alike,
while Bun's fetch is served the real robots.txt on all five hosts and the gate
passes with no override.
Two variables were folded into one there, and they are worth holding apart. The
client mattered — but so does the address, and the client does not rescue it:
the 17:46 UTC run on ubuntu-latest still skipped all four Fandom sources using
that same Bun client. So a claim that these refresh on a schedule is a claim
about the address the runner has, and on the new one it is untested rather than
established. The next cycle settles it.
Three traps recorded so the episode is not repeated: verify a fetch gate with
the real RobotsCache from the address that will run it, never a shell client;
the User-Agent is not a lever in either direction, so a 200 that appears when
you drop it is the challenge's probabilistic half and not a header you tuned;
and neither block is a rate limit, since every failure is first contact with
that host, the spacing is already applied, and a challenge carries no
Retry-After. More delay buys nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`--assume-robots-on-403` stands in for rules a person read in a browser, for a
host that will not serve us the file. It always claimed it would not override a
host that had turned us away — but it could not tell the two 403s apart, so it
excused both. A managed challenge ("we cannot tell what you are") is a question
our fetcher cannot answer and the operators never asked, which is what makes a
human reading the rules a fair substitute. A bare 403 is the site itself saying
no, and nothing recorded on our side may talk over that.
`isInterstitialChallenge` now decides, on Cloudflare's own `cf-mitigated` header
with the challenge page's markers as a fallback; an unreadable body counts as
unclassifiable rather than challenged. This only ever narrows what the flag
opens, so nothing that passed the gate before stops passing it.
Every 403 is classified whether or not the flag is set, and the kind goes into
the reason string the run reports. `robots.txt returned 403` read identically
whether the answer was to refresh by hand on the recorded permission or to stop
fetching the source, and the summary is where somebody has to decide that.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The interval gate has no override, so a page you know has just changed cannot
be fetched until six hours after the last attempt. The only workaround was
deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and
lastConfirmedAt — resetting the broken-source streak and making the footer age
every source from when its bytes last changed rather than when we last
confirmed them. A flag that says what it means is better than a side effect
nobody documented.
It sets aside the interval and nothing else. Conditional headers still go out,
which is what makes forcing defensible at all: the host is asked, not
re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one
request per source and the no-retry rule all still apply — a source that was
not due and is also disallowed stays skipped, for the reason that matters.
Refused under CI, like --assume-robots-on-403 and for the same reason: a
schedule that forces every cycle is a shorter interval with extra steps, and
the interval is the obligation, not the default. So AGENTS.md § Scraping
conduct is amended rather than left to be quietly contradicted by a flag.
Every source asked early is named in summary.forced and warned about. A run
that was due anyway is never reported as forced — a summary that cried "forced"
on an ordinary run would train the reader to ignore the word.
Also repoints the "unknown flag" test, which used --force as its example and
stopped testing anything the moment --force existed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fandom answers our fetcher 403 on /robots.txt itself, from every address we
have, while api.php?action=parse answers the same User-Agent with a 200. The
gate fails closed on the unreadable file, so r1999, fgo, nikke and nikki can
never refresh — even though their rules are not unknown: a person read them in
a browser and wrote them into AGENTS.md, verbatim, which is how three of those
four were cleared in the first place.
This flag is that recorded permission, and nothing wider. What it deliberately
does not do is most of the design:
- 403 only. A 401, a 5xx, a timeout or a soft 404 still mean we do not know
what the site permits, and unknown is still not permission.
- It never overrides a robots.txt we could read. A file that answers and
disallows us is an answer, and it still wins. So this is no use for game8.co
— whose robots.txt reads fine and welcomes us while its edge refuses the
pages — and it must not become one.
- Refused under CI. It stands in for a human having read a file this morning,
and there is no human on a runner. A scheduled job asserting a permission
nobody re-checked is exactly how "temporary" stops being temporary.
- Loud. Every host it applied to is warned about by name, in the run log and
in the summary, with a line saying to go re-read the file. An override that
reports nothing is one nobody withdraws.
Nothing else about being a guest relaxes: one request per source, six hours
apart, spaced per host, conditional headers, no retries.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Three parsers decide currency against ctx.now, because their page has no
"ongoing" heading worth trusting: bawiki.ts, and the Fate/Grand Order and
Infinity Nikki branches of fandom.ts. All three compared Date.parse(endsAt) to
now, and all three publish day-precision ends — so the comparison was against
the 00:00Z placeholder, which § Domain rules says is not an instant and nothing
may read literally.
clockFor already knows that and resolves such a boundary to the reset opening
that game-day on the reader's server. The parsers did not, so the feed retired
a row up to nine hours before the app, the countdown and the game all agreed it
was over. On the pinned Infinity Nikki fixture, "Inspiration Burst" left the
feed at 2026-08-22T01:00Z while clockFor still called it live for an American
reader until 09:00Z, and the page said it ran to 03:59 server time. The reader
does not see a stale row; they see the deadline they were counting down to
disappear on its last day, which is the silent drop this codebase treats as the
dangerous failure.
latestBoundaryMs answers clockFor's question for the last region rather than
one reader's, so a row is history only once it is history everywhere. It follows
a game's own server map and reset hour for the same reason clockFor does —
Endfield and Reverse: 1999 both move. Nothing stored changes: this is one
comparison, not a resolved boundary written into the feed, and no expected.json
moves.
Being generous by nine hours costs an expired row at the bottom of a list.
Being strict costs a live one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Game8 page stopped being updated on 31 August 2025 and mentions 2026 zero
times, and it had been publishing five year-old events with endsAt: null — which
the app renders as live-with-unknown-end indefinitely. That is worse than an
empty lane, and nothing in the pipeline catches it, because a stale page is not
a broken one: it fetches, it parses, it passes every test.
The replacement is the game's Fandom wiki, permitted by the same standard
robots.txt that cleared Nikke, maintained to within two days, and shaped as
Current / Upcoming / Permanent / Past tables of Event | Duration | Description
| Type.
Its durations state a wall clock on both sides and no zone anywhere for that
column, so parseZonelessClockRange reads the clock and discards it, publishing
the printed date at day precision. That invents nothing and treats these cells
exactly as every Game8 date already is. Converting instead would mean picking an
offset, and the offset moves the day: July 16 20:00 read as UTC-7 is July 17,
and the start's day is half of every event ID this game will ever have.
The Game8 entry is removed rather than kept as a second opinion, since a source
whose every row is wrong is not corroboration. Its fixture stays: it is the only
page here carrying the labelled "Start: ... End: Permanent" shape, and the test
now drives it through the parser directly.
Titles come from a link's title attribute, which never passes through text(), so
they are entity-decoded by hand — otherwise "Alison's Travel Shop" becomes a
slug, and a slug is a localStorage key. The ingest sanitiser caught exactly that
during the build.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Its robots.txt was read in a browser and is the standard Fandom file: no
Disallow: / for *, /api.php?action= explicitly allowed, and only the namespaces
we never want refused. The AI crawlers it blocks by name are not us. That was
the one thing missing — the API itself has always answered our own User-Agent
with a 200, and it is only robots.txt that a datacentre address cannot read.
The zone lives in the column header, Start(UTC+9) / End(UTC+9), and no date in
any cell carries an offset. So the header is the safety property: a table that
stops naming its zone is refused rather than read as UTC, which is the Blue
Archive hazard arriving one column to the left.
Two shapes worth knowing. Every title is an image read from <a title>, and the
newest row is the one whose logo has not been uploaded yet — it renders as a
red link reading "File:Persona on Frontline logo.png", so a reader that only
understood the link title would silently drop today's live event. And story
events state a clock only on the end, so the start keeps the day the page
printed rather than being shifted nine hours into the previous one; the start's
day is half an event ID. That is the FGO rule applied to the opposite gap.
resetOffsets and resetHourLocal ship with the game because the page evidences
both: events end 04:59:59 and the banner replacing them starts 05:00:00, one
second later, on a stated UTC+9. Adding that later would re-label day keys
readers had already logged ticks under.
Seconds now survive the date helpers, since rounding 04:59:59 to the minute
would make an event overlap the banner that succeeds it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Near matching exists to reconcile two sources describing one event under
different titles. Within one source it has no such job: the page has already
told us these are two rows, the parser has already dropped repeats of the same
id, and fusing them overrules a distinction the publisher made on purpose.
The loose 0.80 threshold that makes the rule useful across sources makes it
actively wrong within one. Umamusume's banner list runs "3 Star Guaranteed 1.5
Anniversary Scout (Character)" and "(Support)" concurrently — titles differing
by a single parenthetical, starting the same day — and fusing them dropped a
live banner off the calendar with nothing anywhere reporting it. A silent drop
is the failure this codebase ranks as the dangerous one.
Same-id fusion is untouched, so nothing duplicates.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
uma.moe stays declined — its API sits behind a Cloudflare Turnstile proof
header, and an adapter would mean defeating a deliberate access control. This
is the surface that is simply open, and it needed the parser change in the
commit before this one to be readable at all.
The stable URL matters: Game8 also publishes monthly "August 2026 Release
Schedule" pages whose id changes every month, which a static registry cannot
follow. "List of All Banners" does not move.
Four current banners, both columns of the side-by-side table, and none of the
Previous Banners below them. The tenth game8 source, so it carries the same CI
blindness as the other nine and only a manual refresh moves it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Game8's banner pages lay two schedules side by side inside one <table> under a
spanning label row: "Standard Banners | Banner | Rating | Availability | Paid
Banners | ...". That row is not merely unhelpful, it is plausible — it contains
both column words, so it resolves and puts the range at index 3, which no
three-cell data row has. Every row then fails to date and the table yields
nothing at all, with no error anywhere. So readColumnTable now falls back to
row 1, and only when row 0 produced nothing, which is what keeps every page
that parses today parsing identically.
Those pages also schedule banners rather than events and head their sections
accordingly, so the vocabulary learns "List of All Banners", "All Current
Banners" and — the one that matters — "Previous Banners", whose rows are dated
identically to the live ones directly above them.
This parser serves nine sources, so the widening was measured rather than
trusted: every pinned fixture and every live snapshot was parsed before and
after, and no existing source's output changed by a single event.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No parser work at all — a GameId, a games entry, a SOURCES line, a fixture and
a test. The page lists six current events and the parser publishes four; the
two it skips print "Start Date: -", and no start means no event ID, so that is
the skip-never-guess rule working rather than a silent drop.
The cost is worth stating plainly: this is the ninth game8 source, and game8's
edge answers the Actions runner with a 202 and a bot-management body, so the
lane is built from a checked-in fixture in CI from day one and only a manual
`bun run refresh` moves it. freshness() discloses that in the footer, which is
what it is for.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This wiki publishes the same schedule twice and the fuller surface is the worse
one. /wiki/Banner_List has 55 clean rows with full wall clocks and states no
timezone anywhere on the page; the front page's Current Banners module emits
the same instants as real <time datetime="...-07:00"> elements. The two agree
exactly, which is strong evidence the list is UTC and is still only evidence —
so we read the surface that says what it means and pay four live banners
instead of a full history for it. If an editor ever states the zone on
Banner_List, that page becomes the better source immediately.
Two ways this could have failed silently. The template writes its BEM
underscores as __, so a selector written against the class name a
browser shows matches nothing at all; and banner names are red links to
?action=edit&redlink=1, which Miraheze's robots.txt disallows, so a href
carrying a query is refused and the page URL stands in.
No resetOffsets, and here the source did state an offset: -07:00, which is US
Pacific and therefore shifts twice a year. That is the Fate/Grand Order gap
arriving through a page that looks like it answered the question.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The best date material this project has after wiki.gg: every row states an
exact instant on both boundaries and names the zone, so nothing is converted
and nothing is assumed. `parseIsoClockRangeUtc` requires that `(UTC)` rather
than defaulting to it — a row that ever loses it should drop out instead of
landing hours off a boundary the reader is watching in-game.
The hazard here is the Server column. CN, EN and JP rows share one table and
the Chinese schedule runs about a year ahead, which is the akwiki CN-column
problem verbatim: publish EN, skip the rest. Betas are fenced off separately,
being dated exactly like everything else and playable by nobody, and the page
is an archive back to 2023, so currency is decided against ctx.now as in
bawiki. That leaves one live event today — thin, and true.
No resetOffsets: the EN boundaries land on three different clocks, which is a
patch window rather than a reset hour.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A source that prints "August 19, 2026" and no time is stored at 00:00Z,
which is a placeholder for "somewhere in this day" rather than a claim
about when the day opens. The countdown read it literally, which turns it
into exactly that claim — and the UTC day is nobody's. Wuthering Waves
events dated the 19th were still running three hours after we had retired
them, because a European player's day opens at 04:00 on a UTC+1 server;
Asia was four hours out the other way and America nine. Sixty-two of the
eighty-six published events were affected, across ten of thirteen games,
so the reader was reading a wrong number far more often than a right one.
A day-precision boundary now resolves to the reset that opens that
game-day on the reader's server — the same grid daily.ts keys every tick
by, and the only clock we hold for a game. That is a reading of the date
the source printed, not a time invented for it: nothing stored moves, no
event ID moves, and the parsers still refuse to guess.
Two boundaries are never re-anchored. A regionEnds value is already the
instant a source stated per server, so anchoring it would throw a fact
away. And an event the reader typed in was resolved by readerInstant in
their own timezone, which is a stated time too — theirs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The adapter shipped with no docs at all, which left the one thing worth
writing down unwritten: this wiki publishes two servers' schedules on
two cross-linked pages, and the Japanese one parses just as cleanly as
ours. That now has a row in the assessed-sources table, a section under
Fandom, and a line in the ingestion pitfalls table alongside the akwiki
CN column and the bawiki JP tab — three sources bitten the same way is
a pattern, not a coincidence.
Also records what resetOffsets cannot express, since fgo is the first
game to need it: a server clock that shifts with daylight saving.
DATA-MODEL's GameId listing was two games behind the schema, missing
holodori as well as fgo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New holodoriwiki parser plus the holodori-holodoriwiki-events source,
giving the game its two live and upcoming events at exact precision.
holodori.wiki is Miraheze, so this is the same call bawiki makes and for
the same reason: robots.txt disallows /w/ and /*?action=, which closes the
API route and leaves the rendered /wiki/Events page as the surface * is
allowed. It serves our own User-Agent a 200, sets no Content-Signal and no
Crawl-delay for us, and the wiki is CC BY-SA 4.0.
What makes this source unusually good is that it states its timezone on
every cell — 08/17/2026 8:00PM (JST) — so it is the only wiki source here
publishing exact precision on both boundaries without a per-region timer.
parseSlashClockZone requires that zone rather than defaulting to UTC: a row
that ever loses it drops out instead of silently landing nine hours off,
and an abbreviation absent from the table is refused rather than guessed,
which is why the table holds only JST and not the plausible-looking
candidates that are not fixed offsets. The 12-hour clock gets its own
handling and its own tests, because 12PM and 12AM are the two readings a
naive parser gets wrong while still producing a valid-looking instant.
Inclusion is fenced by the Current Events <h2> and bounded by the next one.
That matters more than it looks: Past Events sits directly below with
identical columns, so a reader taking every wikitable would put the back
catalogue on the calendar with nothing to mark it. Rows are still checked
against ctx.now on top of the heading, because Current is maintained by
hand and goes stale before anyone moves a row down.
Two rows are deliberately missing. Beginner Mission runs Game Launch →
Unknown, and no start means no event ID — a permanent tutorial chore is not
what a calendar of deadlines is for. An Unknown end is kept as endsAt null,
and unlike bawiki this parser does not drop a started-but-undated row: the
heading has already said the event is running, which is the fact bawiki
lacks when reading an archive.
Every event title is still a red link — the wiki has no article for any of
them yet — so each points at ?action=edit&redlink=1, a create-page form on
a path this robots.txt disallows. Refusing a href with a query keeps those
out and falls back to the events page; the articles get linked when they
exist, with no change here.
holodori gets resetOffsets of UTC+9 across all three regions, evidenced
rather than assumed the way Arknights' is: the game launched worldwide
simultaneously on one service, every boundary is stated in JST, and
Training Support Missions ends at 3:59AM JST, one minute before a 04:00
local reset. Only the offset is overridden; the hour is the default.
Setting it now is free, because no reader has a day key for a game the app
has never shipped — the same override a year from now would re-label ticks
already logged.
Both boundaries and the exclusions are asserted against an independent
extraction off the fixture, computed through a timezone library rather than
by the parser's own arithmetic.
Several counts in AGENTS.md and PRD.md were already stale before this — the
Fate/Grand Order source did not update them — and are corrected to what the
tree now holds rather than to what it held yesterday.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New bawiki parser plus the ba-bawiki-events source, giving the game its
two live and upcoming events at day precision.
It reads the rendered /wiki/Events page, which is the opposite call to the
Fandom source next door and for the same kind of reason: bluearchive.wiki
is Miraheze, whose robots.txt disallows /w/ and /*?action=, so there the
API is the closed route and the page is the surface * is allowed. It
serves our own User-Agent a 200, sets no Content-Signal and no Crawl-delay
for us. The Fandom wiki was declined earlier as a JP archive yielding
nothing live; this is the live source that assessment pointed at.
Three page facts shape the parser, each a way to publish a confidently
wrong date. The schedule is a JP/Global tabber and the Japanese version
runs four to nine months ahead, so only Global is published — the akwiki
hazard. The Global tab's nav button carries the id
tabber-Global_version-label and precedes both panels, so slicing from the
first matching id reads the Japanese schedule while believing it read
ours; the first version of this parser did exactly that and published a
JP-only event. And three tabs on the page are named Global, the schedule
plus Mini-Event and Joint Firing Drill, so the schedule is found by its
Name (EN) header rather than by position, with canParse asserting the same
lookup — a renamed tab or column fails the run instead of emptying the
lane.
The page states no time of day and no timezone anywhere, which is why the
dates are day precision and why ba gets no resetOffsets. It also settles
the five other schedule tables, which do carry a wall clock but name no
zone and mostly do not say which server: reading those as UTC would invent
the fact that matters most, and rounding to a day would not save it,
because a 04:00 local boundary falls either side of UTC midnight depending
on the offset assumed and the start's day is part of the event ID. They
are left unparsed deliberately.
The two-event count is asserted against an independent extraction off the
fixture, and both starts corroborate against the per-event infoboxes
elsewhere on the page — GL 2026-08-04 and GL 2026-09-01, where the JP
reruns those tabs would have given are 2026-04-01 and 2026-05-06.
Three comment counts were already stale before this and are corrected to
what the tree now holds rather than to what it held yesterday.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
bluearchive.wiki gives each boundary its own column and writes it as a
plain 2026-08-04, so unlike every reader here there is no range to split
and no field order to infer. Day precision, because the page states no
time of day.
Anchored at both ends, which matters more for this shape than for any
other in the module: it is the least distinctive one, and unanchored it
would happily find a date inside an article slug or a version string.
Also names the two readers the paragraph below the table was describing.
"The last two" stopped being true when rows were appended after them, and
the examples belong to parseAdjacentFullRange either way.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Each of these was true when written and stopped being true when a source
landed. dates.ts holds ten formats, not nine; INGESTION says Game8 uses
"at least four page templates" directly above a list of seven; its testing
note still credits "both current adapters" when there are eleven; PRD
lists the seven launch games against ten in GameId. AGENTS also missed
fandom from the parsers-are-keyed-by-site bullet and build-static.ts from
the tree, which matters because another section leans on the latter.
The PRD table stays as launch scope rather than becoming a roster to keep
in sync — GameId is named as the live answer instead, which is the same
move that keeps this drift from recurring. Its no-schema-change claim is
sharpened to name GachaEvent, because Reverse: 1999 did add a field:
resetHourLocal, to game metadata, which moves no stored key. The claim as
written read as false against a commit that plainly added one.
Also draws sanitize into the stage diagram as 2.5. It runs on every source
without the pipeline arranging it, which is exactly why it kept being
absent from the picture.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
bluearchive.fandom.com/wiki/Event/Event_List fetches and parses cleanly,
so the reason to decline it is not obvious from trying it: every one of its
88 rows is JP-server history, the newest ending 2026-02-18, so it yields
zero live or upcoming events. That reads as an empty lane plus a
permanently rejected snapshot, since the runner rejects a body parsing to
nothing — a failure mode worth naming in the assessment table, because
"clean table, no error" invites writing the adapter first.
bluearchive.wiki is the live alternative and stays unbuilt for now: it is
Miraheze, its robots closes the API route we used for Fandom, and it lists
JP and Global in separate tables — the akwiki hazard, which wants settling
before a line is written rather than after.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New `fandom` parser plus the r1999-fandom-events source, giving the game
six live and upcoming events at exact precision.
It fetches api.php?action=parse, not /wiki/Events. The rendered page
answers a non-browser client with a Cloudflare challenge, and getting past
that would be defeating an access control — the reason uma.moe was
declined. The wiki's robots.txt instead allows /api.php?action= for *, and
that endpoint serves our real User-Agent a 200, so this reads the
sanctioned surface with our own headers and no impersonation. The body is
therefore JSON, which is what canParse checks first: a challenge page or
an error payload must fail loudly, not parse to zero events.
Two page facts shape the parser. Titles come from each row's <b>, because
a missing banner image renders as a red link reading "File:<Event>
Banner.png" that a cell-text reader would publish as the event name. And
the page is an archive of 154 rows since v1.1 with no ongoing section to
gate on, so inclusion is decided against ctx.now — the six-event count is
asserted against an independent extraction off the fixture, per
docs/INGESTION.md § Testing.
Because robots.txt is unreadable from a challenged address, the gate fails
closed in CI and the source is skipped there — a warning, not a broken
build. Refreshing it means running `bun run refresh` from an address
Fandom serves.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds parseOrdinalDateTimeRange for "November 9th, 05:00 - December 4th,
2023, 04:59 (UTC-5)". Ordinal days anchor the pattern at both ends so it
cannot claim text the looser readers should get, and the year rolls back
across New Year as parseMonthDayRange does.
The offset is required rather than defaulted. parseSlashDateTimeRange has
to read its wall-clock times as UTC because its source states no zone;
here the zone is part of the format, so a cell without one returns null
instead of being silently read five hours early. Calendar validation runs
on the stated local fields, before the shift, or "February 30th, 23:00
(UTC-5)" would convert into a real instant in March.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The guidance in here is not Claude-specific — it is what this project is,
what it refuses to do, and the rules that are invisible from the code. Any
agent working here needs it, and AGENTS.md is the name they all look for.
CLAUDE.md stays as a pointer so Claude Code still finds it, and holds no
guidance of its own; two copies would drift and the wrong one would be read.
Every cross-reference in the source comments, docs, workflow and skills now
names AGENTS.md, and the image ignores both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The health check earned its keep on the first run: six sources broken, four
cycles each, "last confirmed never" — and a last status of **202**. That is
game8.co's edge answering a GitHub runner with a bot-management body instead of
the wiki.
`response.ok` admitted it. A 202 therefore passed the status gate as a
document, reached the parser, yielded nothing, and was reported as "kept
previous snapshot; new body yielded 0 events" — an accurate description of the
symptom that never mentioned the status explaining it, which is why this looked
like a parser problem for days. Require 200: 202 means the request was accepted
for processing, 204 has no body, 206 is a fragment, and none of them is a page.
The rejection note already names the Server header, so the next cycle reads
`HTTP 202 (AkamaiGHost)` instead.
Recorded in CLAUDE.md § Scraping conduct, including that this must not be worked
around. The per-host spacing added alongside does not help and was never going
to — a 202 on a cycle's first request is address reputation, not rate. Browser
-shaped headers or a residential egress would be defeating an access control,
which is the ground uma.moe was declined on, and game8's robots.txt is
permissive, which makes doing it there worse rather than better.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.
`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.
A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.
Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The most-named game in the release thread, and the cheapest one to add: the
GameId, hue and dailyTasks have been sitting in games.ts since launch with no
source behind them.
arknights.wiki.gg is a better source than Game8 for this game, but it is not the
mp-event shape wikigg.ts reads, so it gets its own parser. Two properties of the
page drive the code:
- Every row lists CN and Global, about five months apart. Only Global is
published; a row without one yields no event rather than borrowing the CN
date. A CN date on a Global calendar is a confidently wrong date.
- Only the next boundary carries a machine-readable timer — the end while an
event runs, the start while it is upcoming — so precision differs per side and
flips when the event goes live. An exact instant is accepted only when it
falls on the same UTC day as the date beside it, because startsAt.slice(0, 10)
is part of the event ID: a start that moved a day would orphan every
completion mark on the morning the event began.
Also sets resetOffsets for all three regions to UTC-7. Not a blanket per-game
offset — Arknights genuinely runs one Global server for every region we model,
and the page evidences the clock: every ending event ends at 10:59:59Z, which is
03:59:59 at UTC-7, one second before a 04:00 reset.
Six events, counted independently against the page before and after parsing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Nine sources across eight games now, two more Game8 page shapes, two more
date formats, and the h4 rule that neither game's page can be read
without.
Also notes what the two new games do *not* have: no `resetOffsets` entry,
so both take the regional default. Neither source states a server map, and
`resetOffsets` feeds `dayKey`, which is a localStorage key — so that is
recorded as an assumption to confirm rather than left to look like a
verified fact.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The status sections claimed the feed was generated from fixtures and the
scheduler unbuilt, which stopped being true. Also documents the
sanitisation stage and the two new key spaces — `dailies:<game>` and
game-day keys — beside the existing warning about event IDs, since they
carry the same "no server-side recovery" property.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The docs had drifted in ways that would mislead: DATA-MODEL documented a
localStorage shape the code stopped using (completedAt, no ignored store),
INGESTION claimed three Game8 templates when five are known, ARCHITECTURE
still listed the whole client and time.ts as unbuilt, and the review-quarantine
skill described a pipeline that does not exist yet without saying so.
Adds the parser roster and the six date formats as tables, documents the
subpath/base-href and offline behaviour, and records the new product surface
(first-run picker, ignore, offline, credit) as PRD features.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Endfield was previously written off as undatable. That was wrong: a first
pass only inspected its Duration rows (all "Permanently Available") and its
year-less version grid, and missed an "Event | Schedule & Summary" table
whose cells read "Period: 08/09/26 - 08/30/26 During the event...". Two real
events, with a year.
Adds MM/DD/YY range parsing for that shape. Month-first ordering is not
assumed lightly — Endfield's own version grid reads 01/22, 04/17, 07/16 for
versions 1.0, 1.2 and 1.4, chronological only if the month comes first.
Also stops presenting unlock conditions as descriptions. Several templates
put "Reach Union Level 8" where a blurb would go; filling the summary slot
with text that never says what the event is is worse than leaving it empty.
Wuthering Waves correctly drops to zero summaries as a result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Event data is parsed deterministically; there is no model call, API key or
per-run cost anywhere in the pipeline. A source that cannot be parsed
deterministically gets no adapter rather than an inference fallback.
Documents the parser/adapter/merge split, records that Game8 uses three
page templates, and adds Neverness to Everness.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
