`iso()` read back the month and the day to catch February 30 rolling over to
March 2, and took the year on trust. `Date.UTC` maps years 0–99 into the 1900s,
so a four-digit "0050" came back as 1950 with the month and day intact — past
both existing checks, because nothing about the resulting date is impossible.
Same class of failure as the rollover the guard was written for: a boundary the
source never stated, published as though it had. Reading the year back too is
the skip-rather-than-guess rule this module runs on, applied to the one field
that was not checked. It narrows only — `parseShortSlashRange`'s two-digit pivot
produces full years and is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`--assume-robots-on-403` stands in for rules a person read in a browser, for a
host that will not serve us the file. It always claimed it would not override a
host that had turned us away — but it could not tell the two 403s apart, so it
excused both. A managed challenge ("we cannot tell what you are") is a question
our fetcher cannot answer and the operators never asked, which is what makes a
human reading the rules a fair substitute. A bare 403 is the site itself saying
no, and nothing recorded on our side may talk over that.
`isInterstitialChallenge` now decides, on Cloudflare's own `cf-mitigated` header
with the challenge page's markers as a fallback; an unreadable body counts as
unclassifiable rather than challenged. This only ever narrows what the flag
opens, so nothing that passed the gate before stops passing it.
Every 403 is classified whether or not the flag is set, and the kind goes into
the reason string the run reports. `robots.txt returned 403` read identically
whether the answer was to refresh by hand on the recorded permission or to stop
fetching the source, and the summary is where somebody has to decide that.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fandom answers our fetcher 403 on /robots.txt itself, from every address we
have, while api.php?action=parse answers the same User-Agent with a 200. The
gate fails closed on the unreadable file, so r1999, fgo, nikke and nikki can
never refresh — even though their rules are not unknown: a person read them in
a browser and wrote them into AGENTS.md, verbatim, which is how three of those
four were cleared in the first place.
This flag is that recorded permission, and nothing wider. What it deliberately
does not do is most of the design:
- 403 only. A 401, a 5xx, a timeout or a soft 404 still mean we do not know
what the site permits, and unknown is still not permission.
- It never overrides a robots.txt we could read. A file that answers and
disallows us is an answer, and it still wins. So this is no use for game8.co
— whose robots.txt reads fine and welcomes us while its edge refuses the
pages — and it must not become one.
- Refused under CI. It stands in for a human having read a file this morning,
and there is no human on a runner. A scheduled job asserting a permission
nobody re-checked is exactly how "temporary" stops being temporary.
- Loud. Every host it applied to is warned about by name, in the run log and
in the summary, with a line saying to go re-read the file. An override that
reports nothing is one nobody withdraws.
Nothing else about being a guest relaxes: one request per source, six hours
apart, spaced per host, conditional headers, no retries.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Three parsers decide currency against ctx.now, because their page has no
"ongoing" heading worth trusting: bawiki.ts, and the Fate/Grand Order and
Infinity Nikki branches of fandom.ts. All three compared Date.parse(endsAt) to
now, and all three publish day-precision ends — so the comparison was against
the 00:00Z placeholder, which § Domain rules says is not an instant and nothing
may read literally.
clockFor already knows that and resolves such a boundary to the reset opening
that game-day on the reader's server. The parsers did not, so the feed retired
a row up to nine hours before the app, the countdown and the game all agreed it
was over. On the pinned Infinity Nikki fixture, "Inspiration Burst" left the
feed at 2026-08-22T01:00Z while clockFor still called it live for an American
reader until 09:00Z, and the page said it ran to 03:59 server time. The reader
does not see a stale row; they see the deadline they were counting down to
disappear on its last day, which is the silent drop this codebase treats as the
dangerous failure.
latestBoundaryMs answers clockFor's question for the last region rather than
one reader's, so a row is history only once it is history everywhere. It follows
a game's own server map and reset hour for the same reason clockFor does —
Endfield and Reverse: 1999 both move. Nothing stored changes: this is one
comparison, not a resolved boundary written into the feed, and no expected.json
moves.
Being generous by nine hours costs an expired row at the bottom of a list.
Being strict costs a live one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Game8 page stopped being updated on 31 August 2025 and mentions 2026 zero
times, and it had been publishing five year-old events with endsAt: null — which
the app renders as live-with-unknown-end indefinitely. That is worse than an
empty lane, and nothing in the pipeline catches it, because a stale page is not
a broken one: it fetches, it parses, it passes every test.
The replacement is the game's Fandom wiki, permitted by the same standard
robots.txt that cleared Nikke, maintained to within two days, and shaped as
Current / Upcoming / Permanent / Past tables of Event | Duration | Description
| Type.
Its durations state a wall clock on both sides and no zone anywhere for that
column, so parseZonelessClockRange reads the clock and discards it, publishing
the printed date at day precision. That invents nothing and treats these cells
exactly as every Game8 date already is. Converting instead would mean picking an
offset, and the offset moves the day: July 16 20:00 read as UTC-7 is July 17,
and the start's day is half of every event ID this game will ever have.
The Game8 entry is removed rather than kept as a second opinion, since a source
whose every row is wrong is not corroboration. Its fixture stays: it is the only
page here carrying the labelled "Start: ... End: Permanent" shape, and the test
now drives it through the parser directly.
Titles come from a link's title attribute, which never passes through text(), so
they are entity-decoded by hand — otherwise "Alison's Travel Shop" becomes a
slug, and a slug is a localStorage key. The ingest sanitiser caught exactly that
during the build.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Its robots.txt was read in a browser and is the standard Fandom file: no
Disallow: / for *, /api.php?action= explicitly allowed, and only the namespaces
we never want refused. The AI crawlers it blocks by name are not us. That was
the one thing missing — the API itself has always answered our own User-Agent
with a 200, and it is only robots.txt that a datacentre address cannot read.
The zone lives in the column header, Start(UTC+9) / End(UTC+9), and no date in
any cell carries an offset. So the header is the safety property: a table that
stops naming its zone is refused rather than read as UTC, which is the Blue
Archive hazard arriving one column to the left.
Two shapes worth knowing. Every title is an image read from <a title>, and the
newest row is the one whose logo has not been uploaded yet — it renders as a
red link reading "File:Persona on Frontline logo.png", so a reader that only
understood the link title would silently drop today's live event. And story
events state a clock only on the end, so the start keeps the day the page
printed rather than being shifted nine hours into the previous one; the start's
day is half an event ID. That is the FGO rule applied to the opposite gap.
resetOffsets and resetHourLocal ship with the game because the page evidences
both: events end 04:59:59 and the banner replacing them starts 05:00:00, one
second later, on a stated UTC+9. Adding that later would re-label day keys
readers had already logged ticks under.
Seconds now survive the date helpers, since rounding 04:59:59 to the minute
would make an event overlap the banner that succeeds it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Near matching exists to reconcile two sources describing one event under
different titles. Within one source it has no such job: the page has already
told us these are two rows, the parser has already dropped repeats of the same
id, and fusing them overrules a distinction the publisher made on purpose.
The loose 0.80 threshold that makes the rule useful across sources makes it
actively wrong within one. Umamusume's banner list runs "3 Star Guaranteed 1.5
Anniversary Scout (Character)" and "(Support)" concurrently — titles differing
by a single parenthetical, starting the same day — and fusing them dropped a
live banner off the calendar with nothing anywhere reporting it. A silent drop
is the failure this codebase ranks as the dangerous one.
Same-id fusion is untouched, so nothing duplicates.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
uma.moe stays declined — its API sits behind a Cloudflare Turnstile proof
header, and an adapter would mean defeating a deliberate access control. This
is the surface that is simply open, and it needed the parser change in the
commit before this one to be readable at all.
The stable URL matters: Game8 also publishes monthly "August 2026 Release
Schedule" pages whose id changes every month, which a static registry cannot
follow. "List of All Banners" does not move.
Four current banners, both columns of the side-by-side table, and none of the
Previous Banners below them. The tenth game8 source, so it carries the same CI
blindness as the other nine and only a manual refresh moves it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Game8's banner pages lay two schedules side by side inside one <table> under a
spanning label row: "Standard Banners | Banner | Rating | Availability | Paid
Banners | ...". That row is not merely unhelpful, it is plausible — it contains
both column words, so it resolves and puts the range at index 3, which no
three-cell data row has. Every row then fails to date and the table yields
nothing at all, with no error anywhere. So readColumnTable now falls back to
row 1, and only when row 0 produced nothing, which is what keeps every page
that parses today parsing identically.
Those pages also schedule banners rather than events and head their sections
accordingly, so the vocabulary learns "List of All Banners", "All Current
Banners" and — the one that matters — "Previous Banners", whose rows are dated
identically to the live ones directly above them.
This parser serves nine sources, so the widening was measured rather than
trusted: every pinned fixture and every live snapshot was parsed before and
after, and no existing source's output changed by a single event.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No parser work at all — a GameId, a games entry, a SOURCES line, a fixture and
a test. The page lists six current events and the parser publishes four; the
two it skips print "Start Date: -", and no start means no event ID, so that is
the skip-never-guess rule working rather than a silent drop.
The cost is worth stating plainly: this is the ninth game8 source, and game8's
edge answers the Actions runner with a 202 and a bot-management body, so the
lane is built from a checked-in fixture in CI from day one and only a manual
`bun run refresh` moves it. freshness() discloses that in the footer, which is
what it is for.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This wiki publishes the same schedule twice and the fuller surface is the worse
one. /wiki/Banner_List has 55 clean rows with full wall clocks and states no
timezone anywhere on the page; the front page's Current Banners module emits
the same instants as real <time datetime="...-07:00"> elements. The two agree
exactly, which is strong evidence the list is UTC and is still only evidence —
so we read the surface that says what it means and pay four live banners
instead of a full history for it. If an editor ever states the zone on
Banner_List, that page becomes the better source immediately.
Two ways this could have failed silently. The template writes its BEM
underscores as __, so a selector written against the class name a
browser shows matches nothing at all; and banner names are red links to
?action=edit&redlink=1, which Miraheze's robots.txt disallows, so a href
carrying a query is refused and the page URL stands in.
No resetOffsets, and here the source did state an offset: -07:00, which is US
Pacific and therefore shifts twice a year. That is the Fate/Grand Order gap
arriving through a page that looks like it answered the question.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The best date material this project has after wiki.gg: every row states an
exact instant on both boundaries and names the zone, so nothing is converted
and nothing is assumed. `parseIsoClockRangeUtc` requires that `(UTC)` rather
than defaulting to it — a row that ever loses it should drop out instead of
landing hours off a boundary the reader is watching in-game.
The hazard here is the Server column. CN, EN and JP rows share one table and
the Chinese schedule runs about a year ahead, which is the akwiki CN-column
problem verbatim: publish EN, skip the rest. Betas are fenced off separately,
being dated exactly like everything else and playable by nobody, and the page
is an archive back to 2023, so currency is decided against ctx.now as in
bawiki. That leaves one live event today — thin, and true.
No resetOffsets: the EN boundaries land on three different clocks, which is a
patch window rather than a reset hour.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every game names its gacha something else, and inferType only knew the
five words the game8 sources use. Fate/Grand Order calls its banners
"Summoning Campaign" — the whole of that page's banner vocabulary — so
without this every FGO pull lands as "other" and the type filter stops
meaning anything for that lane.
Nothing on the current English page carries the word, so no checked-in
expected output moves; the unit case is what pins the behaviour until
the next refresh brings a banner in.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The source landed pointed at Event_List, whose first sentence is "This
page lists all Events in Fate/Grand Order Japan". Every date it
published was a JP date on a calendar whose other twelve games are
global — the akwiki CN column again, and the failure this product
exists to prevent. Event_List_(US) is the English schedule and is what
the adapter reads now; a test asserts the URL so it cannot drift back.
The page is a different template from the Reverse: 1999 one, so the
parser branches, the way game8 branches over seven shapes. Its sections
are fenced by banner images with the label drawn in a positioned div —
no heading, no id — and only ONGOING EVENTS is read. FUTURE EVENTS
states a month with no day, and PAST EVENTS states no year at all, so
neither can be dated and neither is guessed at. canParse asserts both
dividers and the Duration label, which turns a redesign into a stalled
source rather than a lane that quietly empties.
The rewrite drops what the JP branch needed and this page does not: the
hand-rolled month/year rollover (parseFullRange already reads both
stated years), the any-typed locals, and a heading-to-duration regex
whose nested lazy quantifiers hung for minutes on a 620KB body the
moment one anchor stopped matching. Durations name a zone but no clock,
so the stated calendar day is kept rather than shifted into UTC —
there is nothing to anchor a conversion to, and the start's day is half
the event ID.
Also reverts two things done to quiet tests rather than answer them.
The parser no longer normalises NFKC: that was pre-empting the
sanitizer so "real fixtures need no repair at all" would pass, and the
seam is where repairs belong. And renderedHtml no longer reaches into
a legacy {"*": …} envelope, which every source here rules out by
pinning formatversion=2.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New holodoriwiki parser plus the holodori-holodoriwiki-events source,
giving the game its two live and upcoming events at exact precision.
holodori.wiki is Miraheze, so this is the same call bawiki makes and for
the same reason: robots.txt disallows /w/ and /*?action=, which closes the
API route and leaves the rendered /wiki/Events page as the surface * is
allowed. It serves our own User-Agent a 200, sets no Content-Signal and no
Crawl-delay for us, and the wiki is CC BY-SA 4.0.
What makes this source unusually good is that it states its timezone on
every cell — 08/17/2026 8:00PM (JST) — so it is the only wiki source here
publishing exact precision on both boundaries without a per-region timer.
parseSlashClockZone requires that zone rather than defaulting to UTC: a row
that ever loses it drops out instead of silently landing nine hours off,
and an abbreviation absent from the table is refused rather than guessed,
which is why the table holds only JST and not the plausible-looking
candidates that are not fixed offsets. The 12-hour clock gets its own
handling and its own tests, because 12PM and 12AM are the two readings a
naive parser gets wrong while still producing a valid-looking instant.
Inclusion is fenced by the Current Events <h2> and bounded by the next one.
That matters more than it looks: Past Events sits directly below with
identical columns, so a reader taking every wikitable would put the back
catalogue on the calendar with nothing to mark it. Rows are still checked
against ctx.now on top of the heading, because Current is maintained by
hand and goes stale before anyone moves a row down.
Two rows are deliberately missing. Beginner Mission runs Game Launch →
Unknown, and no start means no event ID — a permanent tutorial chore is not
what a calendar of deadlines is for. An Unknown end is kept as endsAt null,
and unlike bawiki this parser does not drop a started-but-undated row: the
heading has already said the event is running, which is the fact bawiki
lacks when reading an archive.
Every event title is still a red link — the wiki has no article for any of
them yet — so each points at ?action=edit&redlink=1, a create-page form on
a path this robots.txt disallows. Refusing a href with a query keeps those
out and falls back to the events page; the articles get linked when they
exist, with no change here.
holodori gets resetOffsets of UTC+9 across all three regions, evidenced
rather than assumed the way Arknights' is: the game launched worldwide
simultaneously on one service, every boundary is stated in JST, and
Training Support Missions ends at 3:59AM JST, one minute before a 04:00
local reset. Only the offset is overridden; the hour is the default.
Setting it now is free, because no reader has a day key for a game the app
has never shipped — the same override a year from now would re-label ticks
already logged.
Both boundaries and the exclusions are asserted against an independent
extraction off the fixture, computed through a timezone library rather than
by the parser's own arithmetic.
Several counts in AGENTS.md and PRD.md were already stale before this — the
Fate/Grand Order source did not update them — and are corrected to what the
tree now holds rather than to what it held yesterday.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New bawiki parser plus the ba-bawiki-events source, giving the game its
two live and upcoming events at day precision.
It reads the rendered /wiki/Events page, which is the opposite call to the
Fandom source next door and for the same kind of reason: bluearchive.wiki
is Miraheze, whose robots.txt disallows /w/ and /*?action=, so there the
API is the closed route and the page is the surface * is allowed. It
serves our own User-Agent a 200, sets no Content-Signal and no Crawl-delay
for us. The Fandom wiki was declined earlier as a JP archive yielding
nothing live; this is the live source that assessment pointed at.
Three page facts shape the parser, each a way to publish a confidently
wrong date. The schedule is a JP/Global tabber and the Japanese version
runs four to nine months ahead, so only Global is published — the akwiki
hazard. The Global tab's nav button carries the id
tabber-Global_version-label and precedes both panels, so slicing from the
first matching id reads the Japanese schedule while believing it read
ours; the first version of this parser did exactly that and published a
JP-only event. And three tabs on the page are named Global, the schedule
plus Mini-Event and Joint Firing Drill, so the schedule is found by its
Name (EN) header rather than by position, with canParse asserting the same
lookup — a renamed tab or column fails the run instead of emptying the
lane.
The page states no time of day and no timezone anywhere, which is why the
dates are day precision and why ba gets no resetOffsets. It also settles
the five other schedule tables, which do carry a wall clock but name no
zone and mostly do not say which server: reading those as UTC would invent
the fact that matters most, and rounding to a day would not save it,
because a 04:00 local boundary falls either side of UTC midnight depending
on the offset assumed and the start's day is part of the event ID. They
are left unparsed deliberately.
The two-event count is asserted against an independent extraction off the
fixture, and both starts corroborate against the per-event infoboxes
elsewhere on the page — GL 2026-08-04 and GL 2026-09-01, where the JP
reruns those tabs would have given are 2026-04-01 and 2026-05-06.
Three comment counts were already stale before this and are corrected to
what the tree now holds rather than to what it held yesterday.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
bluearchive.wiki gives each boundary its own column and writes it as a
plain 2026-08-04, so unlike every reader here there is no range to split
and no field order to infer. Day precision, because the page states no
time of day.
Anchored at both ends, which matters more for this shape than for any
other in the module: it is the least distinctive one, and unanchored it
would happily find a date inside an article slug or a version string.
Also names the two readers the paragraph below the table was describing.
"The last two" stopped being true when rows were appended after them, and
the examples belong to parseAdjacentFullRange either way.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New `fandom` parser plus the r1999-fandom-events source, giving the game
six live and upcoming events at exact precision.
It fetches api.php?action=parse, not /wiki/Events. The rendered page
answers a non-browser client with a Cloudflare challenge, and getting past
that would be defeating an access control — the reason uma.moe was
declined. The wiki's robots.txt instead allows /api.php?action= for *, and
that endpoint serves our real User-Agent a 200, so this reads the
sanctioned surface with our own headers and no impersonation. The body is
therefore JSON, which is what canParse checks first: a challenge page or
an error payload must fail loudly, not parse to zero events.
Two page facts shape the parser. Titles come from each row's <b>, because
a missing banner image renders as a red link reading "File:<Event>
Banner.png" that a cell-text reader would publish as the event name. And
the page is an archive of 154 rows since v1.1 with no ongoing section to
gate on, so inclusion is decided against ctx.now — the six-event count is
asserted against an independent extraction off the fixture, per
docs/INGESTION.md § Testing.
Because robots.txt is unreadable from a challenged address, the gate fails
closed in CI and the source is skipped there — a warning, not a broken
build. Refreshing it means running `bun run refresh` from an address
Fandom serves.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds parseOrdinalDateTimeRange for "November 9th, 05:00 - December 4th,
2023, 04:59 (UTC-5)". Ordinal days anchor the pattern at both ends so it
cannot claim text the looser readers should get, and the year rolls back
across New Year as parseMonthDayRange does.
The offset is required rather than defaulted. parseSlashDateTimeRange has
to read its wall-clock times as UTC because its source states no zone;
here the zone is part of the format, so a cell without one returns null
instead of being silently read five hours early. Calendar validation runs
on the stated local fields, before the shift, or "February 30th, 23:00
(UTC-5)" would convert into a real instant in March.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The guidance in here is not Claude-specific — it is what this project is,
what it refuses to do, and the rules that are invisible from the code. Any
agent working here needs it, and AGENTS.md is the name they all look for.
CLAUDE.md stays as a pointer so Claude Code still finds it, and holds no
guidance of its own; two copies would drift and the wrong one would be read.
Every cross-reference in the source comments, docs, workflow and skills now
names AGENTS.md, and the image ignores both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The most-named game in the release thread, and the cheapest one to add: the
GameId, hue and dailyTasks have been sitting in games.ts since launch with no
source behind them.
arknights.wiki.gg is a better source than Game8 for this game, but it is not the
mp-event shape wikigg.ts reads, so it gets its own parser. Two properties of the
page drive the code:
- Every row lists CN and Global, about five months apart. Only Global is
published; a row without one yields no event rather than borrowing the CN
date. A CN date on a Global calendar is a confidently wrong date.
- Only the next boundary carries a machine-readable timer — the end while an
event runs, the start while it is upcoming — so precision differs per side and
flips when the event goes live. An exact instant is accepted only when it
falls on the same UTC day as the date beside it, because startsAt.slice(0, 10)
is part of the event ID: a start that moved a day would orphan every
completion mark on the morning the event began.
Also sets resetOffsets for all three regions to UTC-7. Not a blanket per-game
offset — Arknights genuinely runs one Global server for every region we model,
and the page evidences the clock: every ending event ends at 10:59:59Z, which is
03:59:59 at UTC-7, one second before a 04:00 reset.
Six events, counted independently against the page before and after parsing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the game and its Game8 events page, the second of the games named in
the release thread. `parseAdjacentFullRange` reads the shape this page
uses: two full dates divided by an `<hr>` rather than a dash, so a
tag-stripping reader sees only whitespace between them. It is anchored at
both ends and needs a year on each half, because without that "August 12,
2026 Day 3 rewards" reads "Day 3" as an end.
Three of the four live rows publish. "Take Your Heart" ends 30 days after
each player makes an account, so it has no calendar date and no honest
place on a calendar. "Login Campaigns" names two candidate ends
("July 16/30, 2026") — it keeps its real start and takes no end, and the
leftover is not shown as a summary either, since a date the parser
refused to trust must not reappear dressed as information.
The page lists each live event twice: once in a bare Event|Duration table
and again under its own heading with Start Date / End Date rows and a
paragraph of prose. The second copy corroborates the dates, so deduping
now fills a missing summary from the copy it drops. Dates are still taken
wholesale from the better-dated copy and never blended.
Verified by re-extracting the page independently of the parser: 4 rows,
3 events, the fourth correctly skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the game and its Game8 events page. One `SOURCES` entry against the
existing game8 parser, plus one date shape that page needs:
`parseLabelledStartEnd` reads a duration cell holding `Start: <date>` and
`End: <date>` split by a `<br>`, which a tag-stripping reader flattens
into one run of text.
Four of the seven events say `End: Permanent`. That is the source telling
us there is no deadline, so they publish with `endsAt: null` and
`endPrecision: "unknown"` rather than a plausible date. The labelled cell
is also structure end to end, so it never becomes a summary — stripping
the leading half would leave "End: Permanent" standing where a
description belongs.
Verified by re-extracting the page with a throwaway script independent of
the parser: 7 rows in, 7 events out, every date matching.
Note the source itself is stale — its "Current Events" table still lists
January-May 2025 and "Upcoming Events" says there are none. The adapter is
right; the page looks abandoned, and Nikki coverage worth relying on needs
a second source at a higher priority.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Game8 uses h4 for two opposite jobs. Persona 5 hides its entire finished
back catalogue behind nothing but an `<h4>Finished Events</h4>` inside a
collapsed accordion, so a reader blind to h4 sees one uninterrupted run of
tables and would publish fifty dead events. Genshin uses h4 the other way
round, for sub-headings *within* a single event ("Availability Period",
"Characters & Rewards for this Test Run").
So h4 now gates sections but never names one: an unrecognised h4 leaves
the current event title alone, where an unrecognised h2 or h3 still sets
it. Letting one claim the title renames "Character Test Runs" to the
label sitting above its own date table, which the Genshin fixture catches.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A review of code that has never made a live request. Each of these was
reproduced before it was fixed.
robots.txt matched a group by testing whether our whole User-Agent
contained the group's name. Our contact URL carries "StereotypicalCat",
so a wiki writing `User-agent: cat` matched us — and because a named
group *replaces* the wildcard group, that silently discarded every rule
under `User-agent: *`. Match the RFC 9309 product token instead. A 200
carrying an HTML "not found" page also parsed to zero rules and read as
permission; it now fails closed, while a genuinely empty body still
means "no restrictions".
The response body was read outside the try that guarded the request, so
one truncated body aborted the whole cycle: later sources were never
fetched, and the failed source never recorded its check, meaning a
re-dispatch would ask that wiki again minutes later.
Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A
page served in a legacy charset became replacement characters with the
original bytes gone — and mojibake in a title flows into slugify and
moves every localStorage key for that source. Decode by the declared
charset, keep the served bytes verbatim, and hash those.
Unchanged bytes skipped the metadata write, so once a server rotated its
ETag we sent a stale validator forever and it served full bodies instead
of 304s.
Also: snapshot writes go through a temp file and a rename, and `--only`
with no value is an argument error rather than "all sources".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The cut was marked with U+2026, which NFKC decomposes into three dots —
so re-sanitising a truncated string grew it by two characters and re-cut
it at a different word boundary. A title would quietly rewrite itself
every time the event was re-ingested, and the module promises
idempotency in its own docstring.
Append what normalisation would produce instead. The existing corpus
missed this because its only over-length entry has no space in its last
40%, so it happened to re-truncate to the identical string; the new test
uses prose.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.
bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.
Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.
A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.
Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Scraped text becomes React content, JSON on disk and eventually SQLite
rows, so it gets cleaned at one seam: the parse wrapper in toAdapter().
Every source passes through it, a source added tomorrow is covered
without its author doing anything, and no parser can opt out — parsers
stay pure readers of one site's markup.
Removes script/style/comment content and residual tags, decodes entities
to a fixed point so an encoded tag cannot resurrect in a later decoder,
NFKC-normalises, strips control, zero-width and bidi-override characters
(an RTL override visually spoofs a title), bounds each field to the cap
the schema already declares, and requires sourceUrl to be absolute
http(s).
Three things it will not do: touch a date, drop an event it could clean
instead, or repair in silence — every repair and drop is logged by
default. Event IDs are localStorage keys, so an ID is recomputed only
when a sanitised title actually changed and the ID was minted the
standard way; all seven fixtures pass through unrepaired and
byte-identical, which is the regression guard.
Also stops decodeEntities throwing RangeError on an out-of-range
numeric reference, which would have taken a whole source's events down.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
First non-Game8 parser, and the first source that states region-scoped ends:
wiki.gg emits ISO timestamps with one timer per server region, so Endfield
events now carry exact times and separate Asia / Americas-Europe ends. That
gap is up to 13 hours, which is what regionEnds was built for.
Two sources for one game also exercised merge for real. It caught both
overlaps — including "Bedazzling Dawnstar" against Game8's "Bedazzling
Dawnstar Sign-In" — and flagged that the two disagree on the end date by 70
hours rather than averaging them into a date neither states.
Matching an appended qualifier needed a prefix test, not a subset test: a
subset would also fuse "Gold Clash" with "Gold Rush Clash Royale".
Also fixes build-feed picking fixtures by game rather than by source, which
handed the wiki.gg page to the Game8 parser once Endfield had two.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Endfield was previously written off as undatable. That was wrong: a first
pass only inspected its Duration rows (all "Permanently Available") and its
year-less version grid, and missed an "Event | Schedule & Summary" table
whose cells read "Period: 08/09/26 - 08/30/26 During the event...". Two real
events, with a year.
Adds MM/DD/YY range parsing for that shape. Month-first ordering is not
assumed lightly — Endfield's own version grid reads 01/22, 04/17, 07/16 for
versions 1.0, 1.2 and 1.4, chronological only if the month comes first.
Also stops presenting unlock conditions as descriptions. Several templates
put "Reach Union Level 8" where a blurb would go; filling the summary slot
with text that never says what the event is is worse than leaving it empty.
Wuthering Waves correctly drops to zero summaries as a result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Three more Game8 templates. ZZZ spans each event name across two rows with a
rowspan, so a flat cell reader sees [title, "Start", date] then ["End",
date]; losing that pairing would silently halve the calendar.
Section matching also had to widen — these pages head their sections
"Featured Events", "List of Events and Schedule" and "Ongoing Events".
Verified independently of the parser: HSR's year-less summary table lists
six ongoing events and all six are captured via their detail rows, so
nothing is silently dropped.
28 events added across the three games.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Honkai: Star Rail and Wuthering Waves state a year on each side of a range
("Aug. 14, 2026 - Aug. 24, 2026"), which the year-on-the-end parser cannot
read.
Both also publish genuinely open-ended events — "Jul. 24, 2026 - End of
4.6", "July 10, 2026 - Permanent". Those keep their real start and report no
end, rather than being dropped or given a guessed one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Column tables already carried a description column, but label/value detail
tables did not — Genshin events had no blurb at all. The page follows each
detail table with a sentence of prose, so take that, skipping the
call-to-action paragraphs, which are navigation rather than description.
Summary coverage goes from 13/22 events to 22/22.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Collapses the same event seen by two sources, matching on ID or on title
similarity plus start-date proximity. Proximity is the real guard against
false positives, since a rerun reuses its name months later.
Agreement from an independent source raises confidence; the same row seen
twice in one document does not. Disagreement on an end date is recorded as
a conflict for the review gate rather than averaged — splitting the
difference would publish a date neither source asserts.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Parsers are keyed by site template, not game: one Game8 parser serves
every Game8 page, and adapters bind a URL and game to a parser. Adding a
source for a known site is one SOURCES entry.
Game8 uses at least three templates. This handles label/value detail
tables (Genshin, 9 events) and column tables (NTE, 13 events). Its
image-grid schedule has no year and no end date, so Arknights: Endfield
is not supportable from that source and gets no adapter.
canParse is a structural tripwire: a site redesign fails the run loudly
instead of publishing an empty calendar that reads as "no events".
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every date function returns null rather than inferring a missing year,
month or end date. A source that does not state an end yields
endsAt: null, never a plausible guess — a confidently wrong end date is
the failure this product exists to prevent.
Handles the three formats seen on Game8: "August 12, 2026",
"August 12 - September 21, 2026" (year applies to both ends, rolling the
start back when the range crosses New Year), and a slash datetime range.
Abbreviated months parse too, so a source switching to "Apr. 29" does not
silently drop events.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
