Add karendar.com as the event source for Punishing: Gray Raven (PGR). Karendar publishes a dedicated Global event calendar with exact UTC timestamps in clean server-rendered HTML. - Register 'pgr' in GameId and GAMES registry (#CC292B, Kuro Games). - Add parseWeekdayDayMonthYearUtc in src/ingest/dates.ts with unit tests. - Implement karendarParser in src/ingest/parsers/karendar.ts reading 'week', 'ongoing', and 'upcoming' sections, mapping indefinite/permanent ends to endsAt: null, and categorizing tags to EventType. - Register pgr-karendar-events in src/ingest/adapters/index.ts. - Add pinned fixture, expected output, and comprehensive test suites. - Update documentation in AGENTS.md, README.md, docs/INGESTION.md, and docs/SOURCES.md.
1161 lines
93 KiB
Markdown
1161 lines
93 KiB
Markdown
# AGENTS.md
|
||
|
||
This file provides guidance to coding agents working in this repository. It is the working
|
||
agreement: what this project is, the constraints it holds to, and the rules that are not visible
|
||
from the code alone. Read it before changing anything.
|
||
|
||
`CLAUDE.md` points here, so Claude Code picks it up too — keep the guidance in this file and leave
|
||
that one a pointer.
|
||
|
||
## Read the docs before changing the thing they describe
|
||
|
||
This file is the working agreement, not the specification. `docs/` holds the reasoning, and it is
|
||
written for whoever touches that area next — reading the relevant one first is the difference
|
||
between repairing a rule and rediscovering it the expensive way.
|
||
|
||
| Doc | What it settles | Read it before |
|
||
|---|---|---|
|
||
| `docs/PRD.md` | What the product is for, feature by feature (F1–F14), and the quality bar for dates | Changing behaviour a reader can see, or arguing something is out of scope |
|
||
| `docs/DATA-MODEL.md` | `GachaEvent`, the SQLite tables, every `localStorage` key space, the export format | Touching `src/shared/schema.ts`, an ID scheme, a stored key, or the game/reset table |
|
||
| `docs/INGESTION.md` | The six pipeline stages, the parser/adapter/merge layering, date formats, the review gate | Adding a source, writing or repairing a parser, or changing the fetch runner |
|
||
| `docs/ARCHITECTURE.md` | Process shape, file layout, request paths, offline and update mechanics | Moving files, adding a route, or changing the service worker |
|
||
| `docs/FEEDBACK.md` | What readers actually said about the first release, and the work it argues for | Deciding what to build next |
|
||
| `docs/SOURCES.md` | Which sites publish a usable schedule for the games we still do not cover, and what is wrong with the ones that do not | Picking the next game to add, or assessing a source request |
|
||
|
||
Two rules that follow from that:
|
||
|
||
- **The docs are part of the change.** A change that makes a sentence in `docs/` false is not
|
||
finished until that sentence is fixed. They are the only record of *why*, so drift costs the next
|
||
agent the whole reasoning, not just a detail.
|
||
- **When this file and a doc disagree, that is a bug — say so.** Neither one silently wins. This
|
||
file summarises; the doc holds the argument, so fix whichever is actually wrong rather than
|
||
reconciling them in your head and moving on.
|
||
|
||
## What this is
|
||
|
||
A web app that aggregates live and upcoming events across popular gacha games, plots them on a
|
||
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
||
progress on events that repeat daily, and lets a user mark events completed.
|
||
|
||
**Status: working app, refreshing itself on a schedule.** Schema, ten parsers, twenty-two sources across
|
||
twenty games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
||
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
||
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
||
feed is a static JSON file built from snapshots, falling back to checked-in fixtures.
|
||
|
||
## Three constraints that shape everything
|
||
|
||
1. **No accounts, no logins, no user records.** Completion state lives in the browser's
|
||
`localStorage`, keyed by event ID. There is no user table and no session. Any request implying
|
||
"sync across devices" is solved with export/import JSON, not a server-side user.
|
||
2. **No LLM in the pipeline.** Event data is extracted by deterministic code-based parsers only.
|
||
There is no Anthropic dependency, no API key, and no per-run inference cost. A source that
|
||
cannot be parsed deterministically does not get an adapter — see `docs/INGESTION.md` § No LLM.
|
||
3. **A server is allowed** (Bun) and owns fetching, parsing, and SQLite. The client only ever calls
|
||
this app's own `/api/*`.
|
||
|
||
## Stack
|
||
|
||
| Layer | Choice |
|
||
|---|---|
|
||
| Runtime / server / bundler / test runner | Bun 1.3 (`Bun.serve`, `bun:sqlite`, `bun test`, `bun build`) |
|
||
| UI | React 19 + TypeScript (strict) + Tailwind |
|
||
| Storage | SQLite via `bun:sqlite` (gitignored — `*.sqlite`) |
|
||
| Validation | Zod — one schema module shared by server and client |
|
||
|
||
The only runtime dependency is `zod`. Do not add a bundler, test runner, HTTP client, or HTML
|
||
parsing library — Bun covers all four. `tsconfig.json` runs `strict` plus
|
||
`noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, `noUnusedLocals` and
|
||
`noUnusedParameters`.
|
||
|
||
The last two are there because this project has no linter and does not want one: a binding nothing
|
||
reads is the only kind of dead code a reader cannot tell from a deliberate seam, and `tsc` already
|
||
walks every file. Three had accumulated behind `strict` — an unread `type View` import, and a
|
||
`useGameMeta()` resolver in two components that each draw their colours from somewhere else. A
|
||
genuinely unused parameter is spelled with a leading underscore, which both flags already exempt.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
bun install
|
||
bun test # full suite, offline, no network, no build needed
|
||
bun run typecheck # tsc --noEmit
|
||
bun run dev # build then serve on :3000
|
||
bun run build # feed + css + js + static into public/
|
||
|
||
# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping
|
||
# conduct before running it, and prefer --dry-run.
|
||
bun run refresh --dry-run
|
||
bun run refresh --only genshin-game8-events
|
||
|
||
# Run one source against its fixture (offline, free)
|
||
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
|
||
bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
|
||
|
||
# Single test file / single test
|
||
bun test test/dates.test.ts
|
||
bun test --test-name-pattern "year-less"
|
||
|
||
# Hosting under a subpath (GitHub Pages)
|
||
BASE_PATH=/gacha-event-tracker/ bun run build
|
||
```
|
||
|
||
**`build:js` passes `--production`, and that flag is load-bearing.** Without it Bun bundles React's
|
||
*development* build: 566 KB rather than 344 KB (164 KB vs 103 KB gzipped), every element creation
|
||
validated at runtime, and `StrictMode` double-invoking effects — which in this app means
|
||
`fetchFeed` runs twice and every reader downloads the feed twice on every load. It shipped that way
|
||
until 2026-08-20; on a 4G/4×-CPU profile fixing it moved first contentful paint from 1480 ms to
|
||
972 ms.
|
||
|
||
**Do not reach for `--define process.env.NODE_ENV='"production"'` instead.** It looks equivalent and
|
||
produces a bundle that does not run. The define flips React to its production build while the JSX
|
||
transform still emits `jsxDEV` calls into `react/jsx-dev-runtime`, so the page dies on
|
||
`jsxDEV is not a function` — minified to `z is not a function`, which is what it looks like in a
|
||
built copy. `--production` switches the transform *and* the env together, which is why it is the
|
||
only one of the two that works. Setting `NODE_ENV=production` in the environment has the same defect
|
||
as the define. **A change here must be loaded in a browser, not merely built** — both broken variants
|
||
build clean, typecheck clean and pass all 843 tests, because nothing in the suite executes the
|
||
bundle.
|
||
|
||
**Tests must never need build output.** They run before `bun run build` in CI; anything reading
|
||
`public/` must create its own fixture tree instead.
|
||
|
||
`bun run parse ... --json` is also how `.expected.json` fixtures are regenerated after an
|
||
intentional parser change. Regenerating them makes the test self-consistent, not correct — always
|
||
re-verify a sample against the live page afterward.
|
||
|
||
## Current state of the code
|
||
|
||
```
|
||
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
||
custom.ts — reader-authored games and events, and their key spaces
|
||
src/ingest/ html.ts, dates.ts (sixteen formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||
health.ts — which of the three empties a source's zero was; pure
|
||
parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts, bawiki.ts, holodori.ts, iopwiki.ts,
|
||
stellasora.ts, arustats.ts, karendar.ts — keyed by SITE, not game
|
||
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
||
src/client/ React app, service worker, manifest
|
||
state/ progress, daily log, ignores, prefs, sort — all localStorage
|
||
gameOrder.ts — the reader's game order, and the A–Z fallback; pure
|
||
useCustom.ts — the reader's own games and events (PRD F13)
|
||
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
||
zoom.ts — the timeline's scale ladder; pure
|
||
lanes.ts — how the timeline stacks: a lane per game, or one deadline queue; pure
|
||
theme.ts — dark or light, and what a game hue reads as on each
|
||
scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||
serve.ts static server + /api/health
|
||
test/ 1,099 tests
|
||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||
snapshots/ current page per source, rewritten by refresh — see its README
|
||
```
|
||
|
||
Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on
|
||
disk.
|
||
|
||
## Domain rules that are not obvious from the code
|
||
|
||
These come from how gacha games actually schedule things, and they cause most bugs here:
|
||
|
||
- **Store every timestamp as UTC ISO 8601.** Sources publish in a mix of UTC+8, server-local, and
|
||
"after maintenance".
|
||
- **Banner ends are usually global and simultaneous; event ends are usually per-region.** Character
|
||
banners end at one instant worldwide; story/login events end at each region's daily reset (Asia /
|
||
America / Europe differ by hours). `regionScoped` and `regionEnds` exist for this — do not
|
||
collapse them into one timestamp.
|
||
- **`endsAt: null` is a correct, expected value.** An event whose end is genuinely unannounced gets
|
||
`endsAt: null` and `endPrecision: "unknown"`. **Never invent a plausible date to satisfy a
|
||
non-null type.** This is the worst failure mode this codebase has, because the user's entire
|
||
reason for visiting is trusting the end date.
|
||
- **A date with no time of day is stored as 00:00Z, and that is a placeholder, not an instant.**
|
||
Most sources print `August 19, 2026` and nothing else, so `dates.ts` returns `precision: "day"` at
|
||
UTC midnight because it has to return something. Counting down to it literally turns the
|
||
placeholder into a claim the source never made — that the day opens in UTC — and retires an event
|
||
up to nine hours before the game does, while the reader is standing in the game watching a longer
|
||
timer. So `clockFor` (`src/shared/time.ts`) resolves a day-precision boundary to the reset that
|
||
opens that game-day on the reader's server, via `dayStartMs`: the same clock `daily.ts` keys every
|
||
tick by, and the only fact we hold about a game's day. Two boundaries are never re-anchored — a
|
||
`regionEnds` value, which exists precisely because the source stated an instant per server, and an
|
||
event the reader typed in, which `readerInstant` already resolved in their own timezone. This is a
|
||
*reading* of the printed date, not an invented time, and it changes nothing stored: the feed,
|
||
every event ID and the parsers are untouched, so it is one resolution at the point where the
|
||
region is finally known.
|
||
The same clock governs the *other* end of the pipeline. A parser whose page has no trustworthy
|
||
"ongoing" heading decides currency against `ctx.now` itself, and comparing the 00:00Z placeholder
|
||
to `now` retires a row hours before `clockFor` calls it over for anybody — the reader watches a
|
||
deadline they were counting down to vanish on its last day. So `latestBoundaryMs` answers the same
|
||
question for the *last* region, and `bawiki.ts` and three branches of `fandom.ts` ask it. Nothing
|
||
stored changes: it is one comparison, not a resolved boundary written to the feed.
|
||
- **Patch cycles are ~6 weeks, and a span over a year is a parse error rather than a long event.**
|
||
`test/adapters/game8.test.ts` rejects one across every fixture. Two corrections to what this rule
|
||
used to say, both load-bearing:
|
||
- **The ceiling is 365 days, not 180** (raised 2026-09-12, repository owner). A real event finally
|
||
exceeded 180: Genshin's anniversary 5-star selection runs 336 days, correctly dated on its wiki,
|
||
and 180 would have dropped a deadline readers want. 365 keeps the guard aimed at what it was
|
||
built for, because a misread year puts the *end* twelve months out and so reads as span + ~365 —
|
||
405 days for a six-week event, still caught. What it newly admits is only the genuinely
|
||
year-long event, a shape this domain does have; the same widening clears Fire Emblem Heroes'
|
||
seven-month new-player banner (`docs/SOURCES.md` § 12b), which had been parked on this question.
|
||
- **There is no validator.** This file claimed "the validator and the tests both reject it" and
|
||
only the second half was ever true — the validator belongs to the quarantine gate described in
|
||
`docs/INGESTION.md`, which is specified and **not built**. So the rule is enforced per adapter,
|
||
by a test, against a pinned fixture: a source added without a fixture in that table is a source
|
||
the rule does not cover.
|
||
|
||
## Working on parsers
|
||
|
||
- **Parsers are pure.** No network, no `Date.now()`, no randomness — time arrives as `ctx.now`.
|
||
This is what makes fixture tests meaningful; a parser that reads the clock cannot be tested.
|
||
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
|
||
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
|
||
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
|
||
- **Parsers are keyed by site, not game.** One `game8` parser serves nine sources and `fandom` five;
|
||
`wikigg`, `akwiki`, `bawiki`, `holodoriwiki`, `iopwiki`, `stellasorawiki` and `arustats` serve one
|
||
each — `wikigg` and `akwiki` share a host family and have entirely different templates, and
|
||
`bawiki`, `holodoriwiki` and `stellasorawiki` are all Miraheze wikis whose page templates have
|
||
nothing in common. `arustats` is the odd one out and the only parser here that does not read
|
||
markup at all: that site server-renders its schedule into a Next.js `__NEXT_DATA__` blob, so the
|
||
parser reads JSON and the rendered grid is ignored. Adding a source for a known site is one
|
||
`SOURCES` entry; a new site is a parser module.
|
||
- **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on
|
||
every row, five months apart. Publish the one our readers are on and skip the row that lacks it —
|
||
a CN date on a Global calendar is a confidently wrong date, not a near miss.
|
||
- **Game8 has no single template.** Eight shapes are known and a page may mix them: label/value
|
||
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
|
||
cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, `<hr>`-separated date pairs, and
|
||
two schedules laid side by side in one `<table>` under a spanning label row.
|
||
Full table in `docs/INGESTION.md`. Before assuming a new
|
||
Game8 page will work, dump its structure and check **every** table — Endfield was written off as
|
||
undatable on a pass that only inspected its `Duration` rows, and its real events were further
|
||
down the page.
|
||
- **The header row is the row that dates rows, not the first one.** Game8's banner pages put the
|
||
Standard and Paid schedules side by side inside one `<table>` and label the pair
|
||
`Standard Banners | Banner | Rating | Availability | Paid Banners | …`. That row is not merely
|
||
unhelpful, it is *plausible* — it contains both column words, so it resolves and puts the range at
|
||
an index no data row has, and the whole table yields nothing with no error anywhere. So
|
||
`readColumnTable` falls back to row 1 **only when row 0 produced nothing**, which is what keeps
|
||
every page that parses today parsing identically. Verified rather than assumed: the change was
|
||
diffed across all pinned fixtures and every live snapshot, and moved no existing event.
|
||
- **Some Game8 wikis schedule banners, not events**, and head their sections accordingly —
|
||
`List of All Banners`, `All Current Banners`, and a `Previous Banners` back catalogue that
|
||
`previous events` does not match. All three are in the section vocabulary now. The finished rows
|
||
sit directly below the live ones and are dated identically, so that exclusion is the only thing
|
||
between the calendar and a year of expired banners.
|
||
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
|
||
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
|
||
accordion, while Genshin uses `h4` for sub-headings *inside* one event and Wuthering Waves titles
|
||
each event of its summary block with one. So `h4` gates sections, and names one only when nothing
|
||
else has: an unrecognised `h4` may **fill an empty event title but never take one**. Genshin's h3
|
||
has already claimed the slot before its labels arrive; Wuthering Waves' block sits under a
|
||
recognised section heading, so the slot is empty and the `h4` is the only name there is.
|
||
- **A blurb the listing table lacks is usually further down the page.** Two Game8 templates put
|
||
unlock conditions (`Reach Union Level 8`, `Unlocked by default`) in the cell where a description
|
||
belongs, and `isRequirementOnly` drops those rather than answer "what is this event?" with "how
|
||
do I qualify for it?". The real prose sits in a per-event section — under an `h3` on Persona 5's
|
||
page and an `h4` on Wuthering Waves'. `dedupe` blends it onto the better-dated copy and never
|
||
blends a date, so reading both shapes adds information and can never contradict any.
|
||
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
|
||
per server region, which is the only reason `regionEnds` carries real data anywhere.
|
||
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
|
||
events vanish with no error. Abbreviated months (`Apr. 29 - May 13, 2026`) and slash-dated open
|
||
ends (`09/02/2026 - TBA`) are supported for exactly this reason. When adding a source, compare the
|
||
parser's event count against an independent count of the page.
|
||
- **A source resting on one row is a source about to read zero.** `eventCount` in a snapshot's
|
||
`.meta.json` is the count the whole lane depends on, and a small one is a warning rather than a
|
||
fact about the game. Endfield's Game8 page read `1` for a week — one row of an *upcoming* table,
|
||
while four live events sat in a card grid the parser cannot see — and when Game8 re-cut that table
|
||
for the next version in a notation `parseOpenRange` did not know, the source read `0` and the
|
||
refresh reported a shape change. Neither number was ever a count of the page. When a source's
|
||
count is far below what the page shows, find out which shape is being skipped before the count
|
||
reaches zero and the gate has to guess for you.
|
||
- **A count of zero has two causes, and only the page can tell them apart.** A source that broke and
|
||
a game that is between patches both read `0`, so nothing downstream can separate them — which is
|
||
why `canParse` must identify a page by its *structure* (headings, ids, column names) and never by
|
||
finding a row, and why a page that prints something like `There are no Events in this category`
|
||
gets to say so through `statesNoEvents` rather than being counted as a failure. Infinity Nikki hit
|
||
both halves of this at once; see § Fandom.
|
||
|
||
## Event IDs are localStorage keys
|
||
|
||
```
|
||
`${game}:${slugify(title)}:${startsAt.slice(0, 10)}`
|
||
→ "genshin:mutual-aid-in-bloom-into-the-frostlands:2026-08-12"
|
||
```
|
||
|
||
Changing `slugify` or `eventId` in `src/shared/schema.ts` — including seemingly cosmetic changes to
|
||
the slug rules — **silently orphans every completion mark every user has, with no server-side
|
||
recovery**, because the server never had the data. If it must change, ship a client-side migration
|
||
that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such
|
||
change.
|
||
|
||
Two more key spaces have the same property, for the same reason:
|
||
|
||
- **`dailies:<game>`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore.
|
||
Two segments, so it cannot collide with an event ID.
|
||
- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at
|
||
04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the
|
||
format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day.
|
||
The clock those keys are cut on — `RESET_HOUR_LOCAL`, `serverOffsetUtc`, `resetHourFor`,
|
||
`resetShiftMs` — lives in `time.ts`, not `daily.ts`, because the countdown resolves day-precision
|
||
boundaries on the same grid (§ Domain rules). Ticks are no longer its only caller, so a change
|
||
there now moves a reader's streak **and** every undated end date at once.
|
||
A game whose server map differs lists the affected regions in `resetOffsets` (`games.ts`) —
|
||
Endfield serves Europe off the Americas machine, so `europe` is UTC-5 there and its reset is
|
||
09:00 UTC, not 03:00. Keep that override **per region**: a blanket per-game offset drags the
|
||
regions that do have their own server onto someone else's clock.
|
||
A game that rolls on a different *hour* says so in `resetHourLocal` instead — Reverse: 1999 resets
|
||
at 05:00, not 04:00, so its day rolls at 10:00 UTC on its single UTC-5 server. Do not encode that
|
||
as a bent `resetOffsets` value: shifting a game's stated server offset to land the right instant
|
||
would misreport the server clock to everything else that asks for it. Both fields are absent for
|
||
every game that takes the default, which is why adding the second one moved nobody's day keys.
|
||
Neither field can express a server whose offset *shifts*: Fate/Grand Order's English server runs
|
||
on US Pacific, which observes daylight saving, and one fixed number is wrong for half the year in
|
||
either direction — so `fgo` takes the default and `games.ts` says why. Reaching for a value anyway
|
||
would re-label day keys twice a year, which is the one thing this whole section exists to prevent.
|
||
Every day-key function takes an optional `game` — **anything reading or writing a tick must pass
|
||
it**, or it writes under one clock and reads under another. A day that drops out of `dailyDays`
|
||
renders no pip, so a tick on it becomes unreachable; check real fixture windows before changing an
|
||
offset.
|
||
|
||
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
|
||
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
|
||
fixtures, that is a data-loss bug, not a diff to regenerate.
|
||
|
||
## Scraping conduct
|
||
|
||
Sources are community wikis. Treat them as a guest would:
|
||
|
||
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
||
- One request per source per refresh cycle, minimum 6 hours apart. **`--force` sets that floor
|
||
aside for one run, and only a person may pass it** — it is refused on an unattended run, because
|
||
a schedule that forces every cycle is just a shorter interval with extra steps, and the interval is
|
||
the obligation. What makes it defensible is what it does *not* change: conditional headers still go
|
||
out, so a page that has not moved costs the host a `304` rather than a re-serve, and per-host
|
||
spacing, robots, the one-request-per-source rule and the no-retry rule all still apply. Prefer it
|
||
with `--only`: forcing twenty sources to re-ask a question they answered an hour ago is the
|
||
behaviour this bullet exists to prevent, whatever flag authorised it. The run names every source it
|
||
asked early, and a run that was due anyway is never reported as forced.
|
||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Nine of the
|
||
twenty sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive
|
||
as nine back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||
what a burst looks like from the far end regardless of our intent.
|
||
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
||
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
||
- Record `sourceUrl` on every event and surface attribution in the UI.
|
||
|
||
Note that game8.co disallows `GPTBot` and `Google-Extended` in `robots.txt` — it has opted out of
|
||
AI-training crawlers. Our use is a low-rate personal aggregator with attribution and no model
|
||
training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch
|
||
rate, and do not add an LLM that consumes page content.
|
||
|
||
**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns
|
||
`202 Accepted` with a bot-management body to every one of the nine game8 sources, from the first
|
||
scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse
|
||
cleanly from a normal address. So `robots.txt` permits us and the network does not, and those nine
|
||
games have only ever been built from checked-in fixtures in CI.
|
||
|
||
The per-host spacing above does not fix this and was not meant to: a 202 on the very first request
|
||
of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a
|
||
proxy, or a residential egress would each be defeating a deliberate access control, which is the
|
||
same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose
|
||
`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the
|
||
refresh from an address game8 will serve, or to find those games another source.
|
||
|
||
A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
|
||
|
||
**Sources assessed and declined** (2026-08-17, extended 2026-08-19), so these are not
|
||
re-litigated each pass:
|
||
|
||
| Source | Verdict |
|
||
|---|---|
|
||
| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source |
|
||
| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` |
|
||
| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below |
|
||
| `bluearchive.fandom.com` | **Declined.** Fetches and parses fine; the page is the problem. Its `Event/Event_List` is a JP-server archive whose newest entry ended 2026-02-18, so all 88 rows are history and it yields **zero** live or upcoming events. An adapter would put an empty lane on the calendar and, because the runner rejects a body that parses to nothing, report a broken source forever. Same failure as the Infinity Nikki Game8 page, further along |
|
||
| `bluearchive.wiki` | **Built** (2026-08-17), from the rendered `/wiki/Events` page — see § Blue Archive below |
|
||
| `fategrandorder.fandom.com` | **Built** (2026-08-18), via `api.php` like Reverse: 1999 — but off `Event_List_(US)`, **not** `Event_List`, which is the Japanese server. See § Fandom below |
|
||
| `holodori.wiki` | **Built** (2026-08-18), from the rendered `/wiki/Events` page. Miraheze again, so the same call as Blue Archive; CC BY-SA 4.0, no `Content-Signal`, no `Crawl-delay` for `*` |
|
||
| `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
||
| `iopwiki.com` | **Built** (2026-08-19), Girls' Frontline 2 — see § IOP Wiki below. `robots.txt` is two lines, `User-agent: *` and `Crawl-Delay: 20`, no `Disallow` anywhere |
|
||
| `stellasora.miraheze.org` | **Built** (2026-08-19), from the front page's `Current Banners` module and **not** `/wiki/Banner_List` — see § Stella Sora below |
|
||
| `game8.co/games/Chaos-Zero-Nightmare` | **Built** (2026-08-19). Zero parser work — the existing `game8` parser reads it. The ninth game8 source, so fixture-backed in CI from day one |
|
||
| `game8.co/games/Umamusume-Pretty-Derby` | **Built** (2026-08-19), off the stable `List of All Banners` page, not the monthly release-schedule pages whose URL changes every month. Cost a widening of `game8.ts`'s section and column vocabulary — see § Working on parsers |
|
||
| `nikke-…-international.fandom.com` | **Built** (2026-08-19), via `api.php` like Reverse: 1999 and FGO. Its `robots.txt` was read in a browser and is the standard Fandom file — see § Fandom below. Richest schedule of anything added in this pass: story events *and* dated pickup banners, with the reset clock evidenced on the page |
|
||
| `infinity-nikki.fandom.com` | **Built** (2026-08-19), replacing the Game8 page for Infinity Nikki, which had been stale since August 2025. Same standard Fandom `robots.txt`. Published at **day precision**: the page states a wall clock and no zone for it — see § Fandom |
|
||
| `genshin-impact.fandom.com` | **Built** (2026-09-12), via `api.php` like the other Fandom sources — see § Fandom below. Fifth Fandom template. Adds 11 events Game8 never listed; priority kept lower than Game8 to protect existing localStorage event IDs |
|
||
| `infinitynikki.miraheze.org` | **Declined.** Exists and serves `robots.txt`, but the wiki is abandoned — front page last edited 11 February 2025 and `/wiki/Events` returns a permission error. Checked as a replacement for the stale Infinity Nikki Game8 page |
|
||
| `prydwen.gg/infinity-nikki` | **Declined.** 404 — prydwen does not cover Infinity Nikki |
|
||
| `grayravens.com` (Punishing: Gray Raven) | **Declined.** Conduct is fine; the data is not. The whole 626 KB `/wiki/Events` page contains exactly one date range, written as prose, one event per six-week patch |
|
||
| `karendar.com` (Punishing: Gray Raven) | **Built** (2026-09-12). Fan-made PGR event calendar for Global. Full SSR HTML with 57 active events across week/ongoing/upcoming. Exact UTC timestamps, `robots.txt` allows `/`, no auth or API required |
|
||
| `guardian-tales.fandom.com` | **Declined.** Parses fine and contains no 2026 date at all — newest dated entry is 2025. The `bluearchive.fandom.com` failure again: parses cleanly to nothing live |
|
||
| `blhx.fandom.com`, `azurlane-archive.fandom.com` | **Declined.** The two Fandom alternatives to the declined koumakan wiki are dead archives — `Event_Calendar` stops in **2021**, and the archive wiki's headings have nothing under them. Azur Lane still has no source |
|
||
| Aether Gazer | **Do not build.** The developer confirmed no further content updates after 23 July 2026, with store listings removed 17 October 2026. The wiki dates nothing anyway — `Event_Guide_List` is an image gallery. A lane that will be empty by winter |
|
||
| `arustats.com` (Honkai Impact 3rd) | **Built** (2026-08-27), and the only source here that publishes **estimated** dates — see § Honkai Impact 3rd below before touching it. `robots.txt` is `Allow: /` for `*` with `/hi3/*` and `/en-us/*` named explicitly, no `Disallow`, no `Content-Signal`, `Crawl-delay: 1`. Solves the per-version URL problem `marisaimpact` failed: `/en-us/hi3/timeline` answers `307` to the live version |
|
||
| `marisaimpact.com` (Honkai Impact 3rd) | **Declined** (2026-08-19). Conduct is clear — its `robots.txt` is comments only, with no directive and no `Content-Signal`, and the page answers our own `User-Agent` with a `200`. The data is the problem: the schedule is a grid of week columns headed `Estimated date for Regional Servers` under a page that says `Based on CN server`, it states **no year anywhere**, and it lives at a per-version URL — `/calendar89` is v8.9 and expires on 20 August, with no stable route to the current one. `docs/SOURCES.md` § 13 |
|
||
| game8.co hubs for Black Beacon, Brawl Stars, Destiny: Rising, Diablo Immortal, Epic Seven, Fire Emblem Shadows, Gundam UC Engage, Mongil: Star Dive, Pokémon Champions, Pokémon UNITE, Tower of Fantasy | **Declined** (2026-08-19). All thirteen hubs in that sweep exist and answer `200`; these eleven have no usable schedule. Six are abandoned wikis whose newest page is 2021–2025 — the Infinity Nikki failure mode, a source that parses perfectly and publishes history. Gundam's calendar prints ends with no starts, so no event ID; Pokémon UNITE is fresh but its template fails `canParse`, which is the check working. Per-game evidence in `docs/SOURCES.md` § 12 |
|
||
| `game8.co/games/MementoMori`, `game8.co/games/fire-emblem-heroes` | **Assessed, not yet built** (2026-08-19) — the two live finds of that sweep, and proposals rather than decisions. MementoMori parses today with no parser change; FEH has the freshest page of any source here and needs a ninth Game8 column shape plus a ruling on the 180-day rule, which one real seven-month banner breaks. See `docs/SOURCES.md` §§ 12a–12b |
|
||
|
||
**The Infinity Nikki lane was rebuilt on a live source, and its Game8 source was retired.**
|
||
`game8.co/games/Infinity-Nikki/archives/487445` stopped being updated on 31 August 2025 — it
|
||
mentions the year 2026 zero times — and had been publishing five year-old events with
|
||
`endsAt: null`, which the app renders as live-with-unknown-end indefinitely. That is worse than an
|
||
empty lane and it is the failure this product exists to prevent, arriving through a source that
|
||
looks perfectly healthy to the runner, because **a stale page is not a broken one**: no failure
|
||
streak, no annotation, no `broken` tier. Nothing in the pipeline catches this. When adding a source,
|
||
check when the page was last updated, not only whether it parses.
|
||
|
||
The lane now comes from `infinity-nikki.fandom.com` at day precision (§ Fandom above), and the Game8
|
||
entry is gone from `SOURCES` rather than kept as a second opinion — a source whose every row is
|
||
wrong is not corroboration. Its fixture stays in `fixtures/nikki/`, because it is the only page here
|
||
carrying Game8's labelled `Start: … End: Permanent` shape and `test/adapters` still drives it
|
||
through the parser directly as a regression test.
|
||
|
||
`.github/ISSUE_TEMPLATE/feature_request.yml` points readers at that table by heading, so a source
|
||
request can be checked against it before anyone writes it up — the loudest feedback on the first
|
||
release was "not enough games" (`docs/FEEDBACK.md`), which makes this the request that arrives most.
|
||
Keep the heading if the section moves.
|
||
|
||
wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference`
|
||
with `Allow: /`, and disallow `ClaudeBot` and other AI crawlers by name. Our fetcher is neither: it
|
||
trains nothing, and no LLM reads the page content — constraint 2 is what keeps that true, so it is
|
||
load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive,
|
||
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
||
|
||
**Fandom: read the API, never the page.** `reverse1999.fandom.com/wiki/Events` answers some
|
||
non-browser clients with a Cloudflare managed challenge — HTTP 403, `Just a moment…`, "Enable
|
||
JavaScript" — and `/robots.txt` is on the same challenged route, though our own client is served both
|
||
(see the correction below). Browser-shaped headers or a JS-executing client would get past a
|
||
challenge and **must not be used**: that is defeating a deliberate access control, the same reason
|
||
`uma.moe` was declined above.
|
||
|
||
What makes this source legitimate anyway is that the wiki publishes a second, sanctioned surface. Its
|
||
`robots.txt` — read in a browser, where it serves fine — has no `Disallow: /` for `*` and explicitly
|
||
**allows** `/api.php?action=`, and that endpoint answers our real `User-Agent` with a `200` and a JSON
|
||
body. So the adapter fetches `api.php?action=parse&page=Events`, with no impersonation anywhere: our
|
||
own headers, on a path the site put in writing. The only namespaces `*` is refused are `Special:`,
|
||
`User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links
|
||
for that reason.
|
||
|
||
**The Fandom `403` is per-address *and* per-client, and one of those was measured wrong — 2026-08-19.**
|
||
For one day this section recorded that `403` as Fandom's posture, holding across every wiki and every
|
||
address, which made those four sources permanently unschedulable. Two separate variables were folded
|
||
into that one claim, and they have to be held apart:
|
||
|
||
- **The client.** The original measurement was `curl`, and `curl` is not what fetches. On one address,
|
||
in the same minute:
|
||
|
||
| Client | `GET /robots.txt` |
|
||
|---|---|
|
||
| `curl`, no `User-Agent` override | `403`, 8 tries out of 8 |
|
||
| `curl`, a Chrome `User-Agent` | `403`, 8 tries out of 8 |
|
||
| `curl`, our real `User-Agent` | `403` |
|
||
| **Bun `fetch`, our real `User-Agent`** | **`200`, and the real file, on all five hosts** |
|
||
|
||
There, `RobotsCache` returns `allowed` / `robots.txt ok` for all four sources with no override, and
|
||
the file it reads is byte-for-byte the standard Fandom one recorded above.
|
||
|
||
- **The address, which the client does not rescue.** The scheduled run `a60eb66` (17:46 UTC, on
|
||
`ubuntu-latest`) reported `skipped_robots` for all four Fandom sources using that same Bun client,
|
||
in the same cycle that nine game8 sources took the usual `202 CloudFront`. So a served address is a
|
||
precondition, not a detail: **a claim that these four now refresh on a schedule is a claim about the
|
||
address the runner has**, and it must be re-measured there rather than inherited from here.
|
||
|
||
**`[self-hosted, safe-ip]` was then measured, and it fixed neither edge.** The first cycle after that
|
||
switch reported `403 (an interstitial challenge)` for all four Fandom sources; the next one, once the
|
||
6h floor was up, reported `202 CloudFront` for all nine game8 sources. So that address is challenged
|
||
by Cloudflare and refused by CloudFront alike, and the runner move bought nothing on its own — worth
|
||
recording plainly, because "use a different runner" is the first idea anybody has here and it has now
|
||
been tried.
|
||
|
||
What did fix the Fandom half was the cron passing `--assume-robots-on-403` (below): the following
|
||
cycle put all four through at `200`, two of them with fresh bytes. **game8 has no equivalent and
|
||
cannot get one** — its `202` is the page itself being withheld, not a permission we hold and cannot
|
||
re-read, so there is nothing for a recorded decision to stand on. Those nine remain fixture-backed in
|
||
CI until they are fetched from an address CloudFront serves or replaced with another source.
|
||
|
||
Three things worth keeping from the episode, because each one is a trap:
|
||
|
||
- **`curl` is the wrong instrument for a question about this pipeline.** The 403 is a Cloudflare
|
||
*managed challenge* (`cf-mitigated: challenge`, `cType: 'managed'`), and a managed challenge is
|
||
scored largely on the TLS and HTTP/2 fingerprint of the client. Bun's stack passes where curl's
|
||
does not. **Verify a fetch gate with `bun -e` and the real `RobotsCache`**, never with a shell
|
||
client, or you will record a conclusion about curl and file it under Fandom.
|
||
- **The `User-Agent` is not the lever, in either direction.** Adding a browser `User-Agent` to curl
|
||
changes nothing — 403 either way — so a 200 that appears when you drop the header is the
|
||
challenge's probabilistic half, not a header you tuned. `fategrandorder` answered `403` once and
|
||
then `200` seven times running to an identical request. This matters because the obvious reading of
|
||
such a result is "shape the headers until it works", which is precisely the impersonation the top
|
||
of this section forbids. It also does not work.
|
||
- **The challenge is route-scoped.** `api.php` is exempt at Fandom's edge and answers everything;
|
||
`/robots.txt`, `/wiki/*` and `/` are the challenged routes. So a challenge on the HTML page tells
|
||
you nothing about the surface the adapter actually reads.
|
||
|
||
A **new** Fandom source still needs that wiki's own `robots.txt` read and recorded here before it is
|
||
built — read it with the runner's client, which can. That is what Nikke's clearance on 2026-08-19
|
||
rests on, and the file is the standard Fandom one: no `Disallow: /` for `*`, `/api.php?action=`
|
||
explicitly allowed, only `Special:`, `User:`, `User_talk:`, `Template:`, `Template_talk:`, `Help:`
|
||
and `UserProfile:` refused, and the named AI crawlers it blocks (`GPTBot`, `CCBot`, `OAI-SearchBot`,
|
||
`ImagesiftBot`) are not us.
|
||
|
||
**Should a challenge ever return, it is still not a licence to work around it.** An adapter can be
|
||
written and fixture-backed from any address, because `api.php?action=parse` answers our own
|
||
User-Agent with a `200`; what a challenged address cannot do is pass the robots gate, which fails
|
||
closed and skips. The answer then is the one below — a person refreshing from an address Fandom
|
||
serves, on a permission recorded by hand — and never browser-shaped headers.
|
||
|
||
**And no, a managed challenge cannot be waited out or solved on our side.** Asked directly on
|
||
2026-08-19, so it is not re-tested each pass:
|
||
|
||
- **Delays do nothing, because neither block is a rate limit.** Three independent reasons, and the
|
||
run of 17:46 UTC on 2026-08-19 supplies the first two:
|
||
- **Every one of these failures was a *first* request to that host in the cycle.** `robots.txt` is
|
||
fetched once per host per run, so each of the four Fandom `403`s was first contact with a
|
||
different host. `genshin-game8-events` is the first game8 source in the order and it took a `202`
|
||
before any other game8 request existed. A mitigation on request one cannot have been caused by
|
||
the pace of requests two through nine.
|
||
- **The spacing is already there.** game8's `robots.txt` states no `Crawl-delay`, so the runner
|
||
applies `DEFAULT_HOST_GAP_MS` (2s) between all nine game8 requests — and all nine still `202`.
|
||
Cycles are ≥6h apart and game8's `lastConfirmedAt` is still `never`, so months of 12-hour
|
||
spacing has not moved it either.
|
||
- **Neither status is a rate response.** Rate limiting is `429` or `503` with `Retry-After`. What
|
||
arrives is `202` with a CloudFront bot-management body, and `403` with
|
||
`cf-mitigated: challenge` and no `Retry-After`. Those are identity verdicts, not pace verdicts.
|
||
|
||
Spacing requests is an obligation we owe the host regardless, and it stays. It is simply not a
|
||
lever on this, and adding more of it would only slow the run down while changing nothing.
|
||
- **Carrying cookies does nothing.** curl with a cookie jar across three requests is `403` every
|
||
time. The edge issues `__cf_bm` (bot management) and never `cf_clearance`, which is the only cookie
|
||
that marks a challenge as passed.
|
||
- **Earning `cf_clearance` is the access control.** It is issued only after the
|
||
`/cdn-cgi/challenge-platform/` script runs and returns a valid proof from a browser environment.
|
||
Executing that to obtain the cookie is defeating a deliberate access control — the exact ground
|
||
`uma.moe` was declined on, where the gate was a Turnstile proof header. A gate not written in
|
||
`robots.txt` is still a gate.
|
||
- **It does not even work.** A real headless browser pointed at these hosts gets a managed challenge
|
||
that never resolves, so the option being refused on conduct is also the option that fails.
|
||
|
||
The legitimate lever is the address — but the one alternative tried here was not a better address.
|
||
`refresh.yml` ran on `[self-hosted, safe-ip]` from 2026-08-19 and was switched back to `ubuntu-latest`
|
||
on 2026-08-20, because that runner was challenged by Cloudflare and refused by CloudFront alike (§ the
|
||
per-address measurement above). So this remains the route for game8 and it is still unwalked: an
|
||
address CloudFront serves, or another source. Do not read the revert as the idea being wrong — read it
|
||
as that particular address not being the one.
|
||
|
||
`--assume-robots-on-403` is the one concession to that, and it is deliberately the narrowest thing
|
||
that helps: `bun run refresh --assume-robots-on-403` treats **an interstitial challenge** on
|
||
`/robots.txt` itself as the permission recorded above rather than failing closed. It is not a
|
||
workaround for a host that turned us away — it never overrides a `robots.txt` we could read, so a
|
||
file that disallows us still says no, and it does nothing at all for game8.co, whose robots.txt reads
|
||
fine and welcomes us while its edge refuses the pages. Every host it applied to is named in the run's
|
||
warnings, so it stays a thing somebody decided this morning rather than a default. Nothing else
|
||
relaxes: one request per source, six hours apart, spaced per host, no retries.
|
||
|
||
**Who may pass it is a question about the person, not the machine.** Both overrides are refused on an
|
||
*unattended* run and available to a person, and `runAttendance` (`scripts/refresh-sources.ts`) draws
|
||
that line: a local shell is a person, a `workflow_dispatch` is a person and GitHub records which one
|
||
in `GITHUB_ACTOR`, and a `schedule` — or any other runner event — is not. This replaced a blanket
|
||
`isCi()` check, which asked the coarser question and got the case that matters wrong: a dispatch sets
|
||
`CI=true`, so somebody clicking "Run workflow" was refused exactly as the cron was, and the overrides
|
||
were unreachable from the workflow at all. The run prints which override was used and who authorised
|
||
it.
|
||
|
||
**The two overrides are not symmetrical, and `--force` is the stricter one.** A schedule may never
|
||
force: asking every cycle before the interval is up is a shorter interval with extra steps, and the
|
||
interval is the obligation. That one stays reachable only from a person or a dispatch.
|
||
|
||
**The cron does pass `--assume-robots-on-403`, and that is a decision with a named cost** (taken by
|
||
the repository owner, 2026-08-20). Four Fandom sources skip on a challenged `robots.txt` every cycle,
|
||
and the alternative was four calendars sitting as stale as the last manual run — which for a product
|
||
whose whole promise is a trustworthy end date is the worse failure. So the schedule stands on the
|
||
permission recorded in this section, and **the owner re-reads those files by hand over time** rather
|
||
than the code pretending to.
|
||
|
||
Be exact about the risk that buys, because it is not "we might be crawling against robots.txt":
|
||
|
||
- A plain `403` still fails closed. The challenge-or-refusal split is what guarantees that much, so a
|
||
host that actually turns us away still stops the run.
|
||
- What is invisible is a `robots.txt` **edited** to disallow us. From a challenged address the file
|
||
never arrives, so a withdrawal would look exactly like the challenge we already expect. Nothing in
|
||
the pipeline can catch that; only the manual re-read can.
|
||
- **The per-cycle warning is therefore the compensating control, not a nicety.** Every run names each
|
||
host it stood on a hand-recorded permission for, says the file was not read, and tells the reader to
|
||
re-read it — as a `::warning` annotation on the run page, so it survives a completely green cycle
|
||
where nothing else draws the eye. `test/refresh.test.ts` pins both the one-warning-per-host rule and
|
||
that it reaches the run page. **Do not demote it to a log line**, and do not let a future change
|
||
make it conditional: it is the only thing that ever prompts the re-read this arrangement depends on.
|
||
|
||
**A `403` is two answers wearing one status code, and only one of them is covered.** A managed
|
||
challenge means "we cannot tell what you are" — the question a human answers by reading the file in a
|
||
browser, which is the whole basis of the concession. A bare `403` means "you are forbidden", and that
|
||
is a host declining us, which no permission recorded on our side may talk over. The flag claimed this
|
||
distinction from the day it was written and could not actually draw it, so it excused both;
|
||
`isInterstitialChallenge` (`src/ingest/robots.ts`) now decides, on Cloudflare's own
|
||
`cf-mitigated: challenge` header with the challenge page's markers as a fallback. A `403` whose body
|
||
cannot be read is unclassifiable, and unclassifiable is not challenged. This only ever *narrows* what
|
||
the flag opens — nothing that passed the gate before stops passing it.
|
||
|
||
One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
|
||
robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
|
||
broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if
|
||
*every* source is blocked — so the scheduled refresh simply never updates this game, and the feed
|
||
falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address
|
||
Fandom serves, which is how its first snapshot was taken.
|
||
|
||
**Five Fandom templates now, and the third states its zone in a column header.** The Nikke wiki's
|
||
`Event` page is `Event | Start(UTC+9) | End(UTC+9) | Archived(?)` for story events and
|
||
`Nikke | Start(UTC+9) | End(UTC+9)` for pickup banners. That header is the safety property, not a
|
||
convenience: no date in any cell carries an offset, so a table whose Start/End columns stop naming a
|
||
zone must be **refused** rather than read as UTC — the Blue Archive hazard, arriving one column to
|
||
the left, and `canParse` asserts the lookup. Two more things about it:
|
||
|
||
- **Every title is an image, and the newest row is the one without one.** Names come from the
|
||
wrapping `<a title="Project Matis">`, but an event whose logo has not been uploaded yet renders as
|
||
a red link reading `File:Persona on Frontline logo.png` — so a reader that only understood
|
||
`<a title>` would silently drop *today's live event* and publish a calendar missing what is on
|
||
now. The file name is the fallback, and a test pins that exact row.
|
||
- **A start with no clock keeps the day the page printed.** Story events state a bare date on the
|
||
start and a clock on the end; converting the bare one from UTC+9 would move it to the previous
|
||
calendar day, and the start's day is half an event ID. That is the Fate/Grand Order rule below,
|
||
applied to the opposite gap — there, a zone with no clock; here, a clock on only one side.
|
||
|
||
**The fourth is Infinity Nikki, and it is published at day precision on purpose.**
|
||
`infinity-nikki.fandom.com` (the unhyphenated name 301s to it) heads its `Current Events` and
|
||
`Upcoming Events` tables `Event | Duration | Description | Type`, and every duration reads
|
||
`July 20, 2026 04:00 – August 10, 2026 03:49` — a full date and a wall clock on both sides, and **no
|
||
zone anywhere on the page** for that column. The only zone statements are prose elsewhere dating
|
||
version launches `(UTC-7)` and a note that rewards reset at `04:00 (Server Time)`; the durations do
|
||
run `04:00 → 03:59`, which only lands on a reset boundary if the column is server-local. Strong, and
|
||
circumstantial.
|
||
|
||
So `parseZonelessClockRange` reads the clock and throws it away, publishing the printed date at day
|
||
precision. That invents nothing and treats these cells exactly as every Game8 date is already
|
||
treated. Converting instead would mean picking an offset, and the offset moves the *day*:
|
||
`July 16, 2026 20:00` read as UTC-7 is `2026-07-17T03:00Z`, and the start's day is half of every
|
||
event ID this game will ever have. If the wiki's editors ever state the zone on that column, this
|
||
source can carry exact instants and should.
|
||
|
||
Two shapes to know: `Permanent Events` and `Past Events` share the page and are fenced off by
|
||
heading, and titles come from a link's `title` attribute — which means they must be **entity-decoded
|
||
by hand**, because an attribute never passes through `text()` and `Alison's Travel Shop` would
|
||
otherwise become a slug, and a slug is a localStorage key. The sanitiser catches exactly that, and a
|
||
parser needing repair on its own fixture is a parser with a bug.
|
||
|
||
**This page goes empty between versions, and that is a third thing a source can be.** On 2026-09-03,
|
||
with 2.7's events ended and 2.8 not yet listed, the wiki replaced both tables with `There are no
|
||
Events in this category` — the markup otherwise untouched, `Past Events` still carrying twenty
|
||
tables of the same shape. `isInfinityNikkiEventPage` identified the page by finding a *populated*
|
||
table, so it read that as a redesign, and the source spent four cycles reporting
|
||
`the source has likely been redesigned` at a page nobody had redesigned. Two rules came out of it,
|
||
and both generalise past this wiki:
|
||
|
||
- **A `canParse` that reads data cannot tell a rewrite from a quiet week.** It now reads the section
|
||
headings, which an empty table does not take with it — *either* `Current Events` or
|
||
`Upcoming Events`, not both, because requiring the pair would fail the source over a renamed
|
||
heading it does not even read. `docs/INGESTION.md` already said to keep these checks structural;
|
||
this one had drifted into content and nothing caught it until a game went quiet.
|
||
- **A page that states its own emptiness is answering, not failing.** `statesNoEvents` (optional on
|
||
`SourceParser`) lets the runner store that as a real answer, so the snapshot advances and the
|
||
streak stays clean instead of reaching the `broken` tier over a correctly empty lane. It must rest
|
||
on the page's own words — a redesign yields zero rows too, and storing *that* is the silent
|
||
emptying the zero-events gate exists to prevent. Only this template implements it; the other three
|
||
say nothing either way when empty and keep the strict gate.
|
||
- **Two things judge a zero, and telling only one of them is half a fix.** The runner learned this on
|
||
2026-09-03; `scripts/build-feed.ts` did not, and CI fails the build on `brokenSources`
|
||
(`parsedCount === 0`). So for four days every green refresh was followed by a red CI run over a lane
|
||
that was correctly empty — the same rule, contradicting itself across two scripts. The runner's
|
||
verdict cannot travel on its own: only a parser has seen the page, and by the time `brokenSources`
|
||
runs there is nothing left but the feed. `SourceHealth.statesNoEvents` carries it,
|
||
`src/ingest/health.ts` sets it, and both ends now ask it the same way — of an empty parse only, from
|
||
the page's own words only.
|
||
It lives in a module because `build-feed.ts` writes `public/`, so importing it from a test runs a
|
||
build and the rule sat where no test could reach it. That is how the two ends drifted, and it is
|
||
the lesson `brokenSources` itself already carried: it was inline in `ci.yml` and pinned by grepping
|
||
that file for a string, which proved the check existed and never that it was right.
|
||
|
||
An empty Nikki lane is therefore now the truth rather than a gap, exactly as GFL2's thin weeks are.
|
||
The lane refills on its own when 2.8 is listed, with no parser change. CI says so out loud rather
|
||
than silently — `quietSources` prints a note beside the count, because an unexplained `0` in the log
|
||
reads as exactly the fault the gate just declined to call it, and it is the only line that would ever
|
||
prompt somebody to ask whether a month-long quiet lane is a quiet game or a wording change under a
|
||
`statesNoEvents` that still matches.
|
||
|
||
**The fifth is Genshin Impact, adding a second source for a lane CI could not fetch.**
|
||
`genshin-impact.fandom.com` publishes `Current Events` and `Upcoming Events` wikitables under `h3`
|
||
headings with columns `Event | Duration | Type(s)`. Genshin's Game8 page has never been fetchable from
|
||
CI (CloudFront returns a `202` on all runners), so until this source existed the game's lane was built
|
||
from a checked-in fixture and could only age. Fandom answers via `api.php` and gives Genshin a live
|
||
source CI can refresh. Four things to know about it:
|
||
|
||
- **The title is the caption link's display text, and neither attribute.** The banner cell contains an
|
||
image link and a text caption below it. The link `title` names the parent article
|
||
(`Miliastra Pass/2026-08-12` for `Phantasmagoric Chronicle`), and the `img alt` carries whatever file
|
||
name was uploaded (`Stygian Onslaught 2025-10-29` on an event starting 2026-08-19). Reading attributes
|
||
here would corrupt event titles — the opposite of the Nikke rule, where cell text was missing and
|
||
attributes were required. A row with no caption is skipped rather than guessing from misleading
|
||
attributes.
|
||
- **Run-date suffixes are stripped.** Recurring events often carry their run date in the subpage title
|
||
(`Overflowing Abundance 2026-09-14`). The date names the run, the start date is already half the
|
||
event ID, and keeping it would duplicate the date on the calendar.
|
||
- **`latestBoundaryMs` gates currency.** Like FGO and Infinity Nikki, this source prints day-precision
|
||
dates (`Aug. 14, 2026 – Aug. 24, 2026`), so `parseFullRange` stores UTC midnight as a placeholder.
|
||
Currency is checked against `latestBoundaryMs` so an event is not retired prematurely on its final day
|
||
before the last region has rolled.
|
||
- **Game8 priority protects localStorage keys.** The incumbent `genshin-game8-events` has
|
||
`priority: 10` while Fandom takes default priority. Two live events are titled slightly differently
|
||
(`To Temper Thyself and Journey Far` vs `… Cycle 5`, and `Stygian Onslaught` vs
|
||
`…: Battle of the Starburst`). In `merge.ts`, higher priority breaks ties when confidence is equal
|
||
(0.85). If Fandom won those near-matches, the event IDs would change and readers' localStorage
|
||
completion marks would silently orphan. With Game8 prioritized: 0 IDs lost, 11 net-new events added,
|
||
0 conflicts, and 5 corroborated events earning confidence bonuses.
|
||
- **The 336-day event raised the duration ceiling to 365 days.** Genshin's anniversary 5-star selection
|
||
runs 336 days (`2025-10-22` → `2026-09-23`), correctly dated on this wiki. This motivated raising the
|
||
sanity ceiling in `test/adapters/game8.test.ts` and `docs/INGESTION.md` from 180 to 365 days.
|
||
|
||
**The second Fandom source's page is chosen, not obvious.**
|
||
`fategrandorder.fandom.com` publishes two schedules: `Event_List` opens "This page lists all Events
|
||
in Fate/Grand Order Japan", and `Event_List_(US)` is the English server. They run months apart, each
|
||
links the other, and reading the Japanese one on an English calendar is the `akwiki` CN column again
|
||
— it was how this source first landed, and every date it published was a JP date. The adapter is
|
||
pointed at `page=Event_List_(US)` and a test asserts it; `parsers/fandom.ts` carries the reasoning.
|
||
|
||
Three more things about that page, all of them ways to publish or lose a date:
|
||
|
||
- **Its sections are fenced by pictures.** `ONGOING EVENTS`, `FUTURE EVENTS` and `PAST EVENTS` are
|
||
banner images with the label drawn in a positioned `<div>` over them — no heading, no id. Only the
|
||
ongoing section is parsed, and `canParse` asserts both of the dividers that bound it, so a
|
||
redesign fails the source rather than emptying the lane.
|
||
- **The other two sections cannot be dated, and that is the whole reason they are skipped.**
|
||
`FUTURE EVENTS` gives an ETA of `August 2026` — a month with no day, and a day is half an event
|
||
ID. `PAST EVENTS` is 111 monthly tables that state no year anywhere; the *Japanese* page's
|
||
equivalents carry it in a `MMYYYY` table id, which is a difference easily assumed away.
|
||
- **Every duration names a zone and no clock** — `August 12, 2026 ~ August 26, 2026 PDT`. So the
|
||
boundaries stay on the day the page states rather than being shifted into UTC: there is no time of
|
||
day to anchor a conversion to, and the start's day is part of the event ID. That `PDT` is also the
|
||
evidence that the English server is one machine on US Pacific — see `games.ts`, where it does
|
||
*not* become a `resetOffsets` entry, because Pacific observes daylight saving and that field holds
|
||
one fixed number.
|
||
|
||
**Blue Archive: the page, never the API — the opposite call to Fandom.** `bluearchive.wiki` is a
|
||
Miraheze wiki, and Miraheze's `robots.txt` **disallows** `/w/` and `/*?action=`. So the route
|
||
`parsers/fandom.ts` takes is the one that is closed here, and the rendered `/wiki/Events` page is the
|
||
surface `*` is allowed — it answers our own `User-Agent` with a `200`, no `Content-Signal`, and no
|
||
`Crawl-delay` for us. `Special:` is disallowed too, which is why `parsers/bawiki.ts` skips those links
|
||
exactly as the Fandom one does.
|
||
|
||
Three things about that page are worth knowing before touching it, all of them ways to publish a
|
||
confidently wrong date:
|
||
|
||
- **It states JP and Global in separate tabs, and the Japanese one runs four to nine months ahead.**
|
||
Same hazard as the CN column on `akwiki`, same answer: publish Global only. The tab's *nav button*
|
||
carries the id `tabber-Global_version-label` and sits above **both** panels, so a reader that slices
|
||
from the first id match reads the Japanese schedule while believing it read ours.
|
||
- **There are three Global tabs, not one** — the schedule, plus Mini-Event and Joint Firing Drill
|
||
further down, whose ids are the same name with `_2` and `_3`. The parser finds the schedule by its
|
||
`Name (EN)` header rather than by position, and `canParse` asserts that lookup, so a renamed tab or
|
||
column fails the run instead of quietly emptying the lane.
|
||
- **The page states no time of day and no timezone anywhere.** The schedule's dates are bare
|
||
`YYYY-MM-DD`, which is honest day precision. Its five other tables (Mini-Event, Reward campaigns,
|
||
Attendance bonuses, Guide missions, Joint Firing Drill) *do* carry a wall clock — `08/12/2026 11:00`
|
||
— but name no zone for it, and **three of the five do not say which server they describe**. Those
|
||
are deliberately unparsed, and that second clause is the whole reason: a clock whose server is
|
||
unknown cannot even be labelled with a day, because you do not know whose day it is. Attendance
|
||
bonuses would be a real dailies source if a zone is ever stated. For the same reason `ba` has no
|
||
`resetOffsets`: Blue Archive Global does run one worldwide server, but nothing in this source says
|
||
on what clock.
|
||
|
||
**This rule governs a clock with no known server, not a clock with no stated zone** — a
|
||
distinction worth drawing precisely, because it was drawn the wrong way once. As first written it
|
||
said rounding such a cell to a day "does not save it", which would also have condemned every Game8
|
||
date in this repository: those state no zone either, and are published at day precision without
|
||
anyone minding. The Infinity Nikki wiki (§ Fandom) is the case that forced the correction — one
|
||
worldwide service, a clock, no zone — and it is read at day precision on the printed date, which
|
||
invents nothing. What is still forbidden is *converting* an unzoned clock by picking an offset,
|
||
because the offset moves the day and the start's day is half an event ID.
|
||
|
||
**hololive Dreams: the same Miraheze call as Blue Archive, and the opposite data.** `holodori.wiki`
|
||
is Miraheze too, so `/wiki/Events` is the surface `*` is allowed and `/w/` and `?action=` are closed
|
||
— `parsers/holodori.ts` takes the route `bawiki.ts` takes, for the reason it takes it. What differs
|
||
is the quality of what is there, and three things are worth knowing:
|
||
|
||
- **It states its timezone on every cell.** Every boundary is `08/17/2026 8:00PM (JST)`, which makes
|
||
this the only wiki source here publishing `exact` precision on both sides without a per-region
|
||
timer. `parseSlashClockZone` **requires** the zone rather than defaulting to UTC, so a row that
|
||
ever loses it drops out instead of landing nine hours off. That is also where `holodori`'s
|
||
`resetOffsets` of UTC+9 comes from — evidenced, not assumed; see docs/DATA-MODEL.md.
|
||
- **Inclusion is fenced by an `<h2>`, and the two tables are identical.** `Current Events` and
|
||
`Past Events` have the same columns, so a reader that took every `wikitable` would put the back
|
||
catalogue on the calendar with nothing to mark it. Rows are checked against `ctx.now` on top of
|
||
the heading, because "Current" is maintained by hand and goes stale before anyone moves a row.
|
||
- **Every event title is still a red link.** The wiki has no article for any of them yet, so each
|
||
links to `?action=edit&redlink=1` — a create-page form, and a `?action=` URL this wiki's
|
||
robots.txt disallows. The parser refuses a href with a query and falls back to the events page;
|
||
when the articles exist, they get linked with no change.
|
||
|
||
Two rows on the page are not events and are meant to be missing. `Beginner Mission` runs
|
||
`Game Launch` → `Unknown`: no start means no event ID, and a permanent tutorial chore is not what a
|
||
calendar of deadlines is for. An `Unknown` **end** is kept, though — that is `endsAt: null`, and
|
||
unlike `bawiki.ts` this parser does not drop a started-but-undated row, because the heading has
|
||
already said the event is running.
|
||
|
||
**IOP Wiki: the Server column is the whole safety story.** `iopwiki.com/wiki/GFL2_Events` is the
|
||
best date material here after wiki.gg — every row states an exact instant on both boundaries *and*
|
||
names the zone (`2026-08-06 13:00 - 2026-08-26 22:59 (UTC)`), so `parseIsoClockRangeUtc` converts
|
||
nothing and both sides are `exact`. Three things about it:
|
||
|
||
- **CN, EN and JP rows share one table**, and the Chinese schedule runs about a year ahead. This is
|
||
the `akwiki` CN-column hazard verbatim and gets the same answer: publish `EN`, skip the rest. It
|
||
would be wrong by *months* on a row that otherwise looks perfect.
|
||
- **`Betas` is a section, not an event type.** Closed beta rows are dated exactly like everything
|
||
else and would parse cleanly onto a calendar of things nobody can play. Fenced on the `<h2>`.
|
||
- **The page is an archive**, 145 rows back to 2023, so inclusion is decided against `ctx.now` as in
|
||
`bawiki.ts`. The lane is therefore thin by design — one live event on a quiet week is the truth,
|
||
not a gap.
|
||
|
||
The zone requirement is deliberate: `parseIsoClockRangeUtc` refuses a row that loses its `(UTC)`
|
||
rather than assuming it, exactly as `parseSlashClockZone` does. GFL2 takes no `resetOffsets`: its EN
|
||
boundaries land on three different clocks (22:59, 08:59 and 02:59 UTC), which is a patch window
|
||
rather than a reset hour — Arknights and Reverse: 1999 each earned an override from a single
|
||
boundary their whole page agreed on.
|
||
|
||
**Stella Sora: the front page, not the article — the opposite call to Blue Archive.** Miraheze
|
||
again, so `/wiki/` is open and `/w/` and `?action=` are closed. But this wiki publishes its schedule
|
||
twice, and the fuller surface is the worse one:
|
||
|
||
- `/wiki/Banner_List` has 55 clean rows with full wall clocks and states **no timezone anywhere**.
|
||
- The front page's `Current Banners` module emits the same instants as real
|
||
`<time datetime="2026-08-17T20:00-07:00">` elements.
|
||
|
||
The two agree exactly, which is strong evidence the table is UTC and is still only evidence — so we
|
||
read the surface that says what it means and pay for it in coverage: four live banners instead of a
|
||
full history. If an editor ever states the zone on `Banner_List`, that page becomes the better
|
||
source immediately. Two traps in the markup: the template writes its BEM underscores as `__`,
|
||
so a selector written against the name a browser shows finds **nothing at all**; and banner names are
|
||
red links to `?action=edit&redlink=1`, which robots.txt disallows, so a href with a query is refused
|
||
and the page URL stands in — the `holodori.ts` rule.
|
||
|
||
Stella Sora takes no `resetOffsets` either, and for the opposite reason to most: it states an offset
|
||
outright, and the offset is `-07:00` — US Pacific, which shifts by an hour twice a year. That is the
|
||
Fate/Grand Order problem arriving through a source that looks like it answered the question.
|
||
|
||
**Honkai Impact 3rd: the one source here whose dates are estimates, and the only one whose URL
|
||
tracks itself.** `arustats.com` publishes a per-version timeline, and both halves of that sentence
|
||
matter.
|
||
|
||
- **The URL is version-less on purpose, and must stay that way.**
|
||
`/en-us/hi3/timeline` answers `307` to the live version — `/en-us/hi3/timeline/9.0` today — so the
|
||
site names its own current version server-side and the runner's `redirect: "follow"` lands on it.
|
||
That is the stable route `docs/SOURCES.md` § 13 recorded `marisaimpact.com` as lacking, and it is
|
||
why this game needs no scheduled URL edit. **Pinning `/9.0` in `SOURCES` would publish a finished
|
||
schedule as current the day 9.1 ships** — § 11's stale-source failure on a six-week clock. A test
|
||
asserts the registered URL carries no version.
|
||
- **Every boundary it publishes is an estimate, which nothing else here does.** The page schedules by
|
||
*week bucket*: an event is a bar spanning whole week columns, no event states a date of its own,
|
||
and the header over the grid reads `GLB/SEA` / `ESTIMATED WEEK`. So a start or end from this source
|
||
is the edge of a bucket the site estimated, not a date anybody announced. That is a weaker claim
|
||
than the day-precision reading Game8 and Infinity Nikki get, where the page did print a date per
|
||
event and we only declined to invent a time of day for it.
|
||
- **It was built with that cost named, not overlooked.** `docs/SOURCES.md` § 13 declined
|
||
`marisaimpact.com` on this exact ground; § 14 records the decision to take the trade here anyway
|
||
(repository owner, 2026-08-27) and what would retire it. `ESTIMATE_CONFIDENCE` (0.4, in
|
||
`parsers/arustats.ts`) is what carries the fact into the data, well below the 0.85–0.95 a
|
||
date-stating source earns — so a real Honkai Impact 3rd source outranks this one on `mergeEvents`
|
||
automatically, with nobody having to remember to remove it. **Do not raise that number** without
|
||
re-opening the decision.
|
||
- **Nothing on screen says "estimated" yet.** `confidence` is not read anywhere in `src/client/`, so
|
||
the reader currently sees this lane exactly as they see a wiki-sourced one. That is a known gap
|
||
rather than an oversight: saying so in the UI needs a schema field or a client change, which
|
||
§ Adding a game source calls a design question rather than adapter work. If this source stays,
|
||
that gap is the thing to close next.
|
||
- **Two shapes to know.** `startsAt` is half of every event ID, so an estimated start the site
|
||
revises moves IDs and orphans completion marks — a hazard the wiki sources do not carry. And
|
||
`endWeek` is *exclusive* and runs one past the grid to mean "to the end of the version"; an index
|
||
further out than that is skipped rather than clamped, because pinning an unreadable bar to the
|
||
version's edge would be inventing the boundary.
|
||
|
||
Also unlike everything else here, this parser reads no markup: the site is Next.js and
|
||
server-renders the schedule into `__NEXT_DATA__`, so the parser reads that JSON and ignores the
|
||
grid. `canParse` asserts the blob, the week array and its dates, so a redesign fails the source
|
||
rather than emptying the lane. The host sends neither `ETag` nor `Last-Modified`, so conditional
|
||
requests buy nothing and the 6h floor is the only thing sparing it — do not lower it here.
|
||
|
||
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
|
||
opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
|
||
(failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
|
||
described in § Fandom, which covers a challenged `403` and never a plain refusal). The two overrides differ on who may pass them: `--force` is refused on an unattended run (`runAttendance` — a person or a `workflow_dispatch`, never a `schedule`), while `--assume-robots-on-403` is passed by the cron by standing decision and is reported by name
|
||
in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
|
||
|
||
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
|
||
never blank a calendar or stop the sources that did answer from being committed — so a failure is
|
||
exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
|
||
cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
|
||
checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
|
||
a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
|
||
the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
|
||
runner instead would skip the commit and throw away the pages that did arrive. This tier exists
|
||
because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
|
||
opens the log to read is not a signal.
|
||
|
||
## Untrusted input
|
||
|
||
Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust
|
||
boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single
|
||
seam every source passes through — **do not sanitize inside a parser**, and do not add a code path
|
||
that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup.
|
||
|
||
The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is
|
||
the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5.
|
||
|
||
## Events that repeat daily
|
||
|
||
Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed
|
||
day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published —
|
||
`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an
|
||
event's length. It adds **no schema field**, so the feed contract is untouched.
|
||
|
||
- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks
|
||
the wrong box for four hours every night.
|
||
- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null`
|
||
rule applies here exactly as it does to a countdown.
|
||
- **A tick is never removed except by the reader**, including ticks outside the window the feed now
|
||
claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere
|
||
else.
|
||
- **A repeating event the reader marked done leaves the strip.** They have said there is nothing
|
||
left to do; keeping a tickable chip for it is the app arguing with them. Their logged days are
|
||
untouched, so unmarking it brings the chip and the streak straight back.
|
||
- **Detection is a guess, not a verdict — and it ships off.** `prefs.detectDaily` defaults to
|
||
`false` and the control is labelled experimental: wording is a weak signal and gets it wrong in
|
||
both directions, so a new reader opts in rather than out. The default moves nothing for an
|
||
existing reader, whose stored `prefs` wins. The reader can mark any event as repeating, or unmark
|
||
one detection got wrong (`progress.daily`, resolved by `resolveDaily`), whether the guessing is
|
||
on or off. Store an override only when it *disagrees* with detection —
|
||
recording agreement would freeze today's guess and stop a better parser from ever reaching that
|
||
event. Neither control ever deletes a mark or a logged day, so both are reversible.
|
||
|
||
## Events the reader entered themselves
|
||
|
||
No adapter list covers a ten-game player, so a reader can define a game and type in events
|
||
(PRD F13, `src/shared/custom.ts`, `src/client/state/useCustom.ts`). They join the same lists,
|
||
timeline, sort, filters, progress, ignore and daily stores as scraped events. Four rules:
|
||
|
||
- **Their ids live in their own spaces**: `mygame:<slug>` and `myevent:<random>`. Never
|
||
`${game}:${slug}:${date}` — a reader can type a scraped event's exact title and date, and that
|
||
collision would silently share one completion mark and one streak between two events. Random also
|
||
means renaming their own event never moves its id. `dailies`, `mygame` and `myevent` are reserved
|
||
first segments and **none may ever become a `GameId`**; a test pins this.
|
||
- **Nothing they type enters the ingest pipeline.** `sanitize.ts` and `merge.ts` are for pages we do
|
||
not control. Their events are not fetched, parsed, merged, scored or quarantined.
|
||
- **A hand-entered date is never attributed to a source.** No `sourceUrl`, no source link, and the
|
||
row and detail sheet both say it is theirs. `"I don't know when it ends"` is an offered answer, for
|
||
the same reason the parsers are forbidden from guessing one.
|
||
- **They are in the export.** These exist in one browser and nowhere else, so an export without them
|
||
is a lossy backup. Import merges by id and never removes.
|
||
|
||
A lane may now be a game the reader invented, so `gameMeta` is a context resolver (`metaFor`, pure
|
||
and total) rather than a direct lookup — a lane can outlive its game when an import carries an event
|
||
whose game did not come with it.
|
||
|
||
**Retiring a game, a source or a page must never cost the reader a row they typed.** We retire
|
||
things routinely — a source moves, a page goes stale, a game shuts down — and their events are the
|
||
only copy in existence. Nothing in the client deletes: no store prunes against the feed, `knownGames`
|
||
only ever appends, and `metaFor` renders a lane whose game is gone rather than dropping it. The one
|
||
place this could break is the load path. `useCustom` reads through `validRecords`, which **drops a
|
||
record that fails its schema**, and the survivors are what the next write persists — so a record that
|
||
stops parsing is not hidden pending a fix, it is deleted from the device by the act of opening the
|
||
app. That is why `CustomEvent.game` is `z.string()` and not `GameId`: narrowing it to the enum reads
|
||
like a tightening and would arm every future game removal to erase reader data on next launch.
|
||
`test/custom.test.ts` § retiring a game, a source or a page pins it.
|
||
|
||
## Shipping a new version
|
||
|
||
The shell is cached cache-first, so a reader with the tab open keeps the bundle they first loaded.
|
||
An old app presented as current is the same failure as old events presented as current, so a waiting
|
||
version is disclosed and reloaded on a tap (PRD F14, `docs/ARCHITECTURE.md` § Shipping a new version
|
||
to an open page). Four things hold it up:
|
||
|
||
- **`sw.js` must not `skipWaiting()` on install.** It activates only on the `skip-waiting` message
|
||
the reader's tap sends. Claiming an open page unasked runs the old bundle against the new cache and
|
||
says nothing.
|
||
- **`__BUILD__` must stay in `sw.js`.** `scripts/build-static.ts` substitutes a hash of the built
|
||
shell for it, which is what makes a deploy's worker bytes differ and therefore detectable. It
|
||
throws if the placeholder is gone — do not "fix" that by dropping the substitution. There is no
|
||
`CACHE_VERSION` bump ritual any more; the cache name is a namespace, and per-build names would
|
||
discard the stored feed an offline reader is reading.
|
||
- **The feed is not part of the build id.** It changes twice a day and needs no reload; announcing it
|
||
as a new version teaches readers to dismiss the notice unread.
|
||
- **The app never reloads itself.** Someone may be mid-way through typing an event in.
|
||
|
||
## Conventions
|
||
|
||
- **Commit straight to `main`.** This is a solo repo and its history is a single line; do not open a
|
||
branch for a change unless asked for one. Committing still waits to be asked.
|
||
- **Only ever commit your own work.** Stage the files you changed, by path, and nothing else. The
|
||
working tree may already hold edits, untracked fixtures or a half-finished experiment that someone
|
||
else — the user, or another agent — put there and has not decided about yet; `git add -A`, `git
|
||
commit -a` and `git stash` all sweep those into your change or out of sight. Authorship in the log
|
||
then says you wrote something you never read, and the commit stops being the one coherent change
|
||
the bullet below asks for. If unrelated changes are in the way, say what you see and leave them
|
||
alone.
|
||
|
||
**Another agent may be working in this tree right now, not merely before you.** The index is
|
||
shared and it moves under you: a `git status` that was clean when you started can hold four staged
|
||
files by the time you commit, and none of them yours. So read `git status --porcelain` immediately
|
||
before every commit and treat anything you did not touch as a stop sign, and when the index already
|
||
holds someone else's staged work, commit with an explicit pathspec — `git commit -- <your paths>`,
|
||
which takes the working-tree content of exactly those paths and leaves the rest of the index where
|
||
its owner left it. This is not hypothetical: on 2026-08-19 a session ran a sweeping commit that
|
||
swallowed a 64-line source assessment another session had just finished, and published it under a
|
||
one-line message about snapshot freshness. Both changes were fine; the log stopped being true.
|
||
|
||
**If you find your work inside someone else's commit, say so and ask before rewriting it.** The
|
||
fix is a `reset --soft` and two pathspec commits, and it is quick — but the commit you would be
|
||
rewriting is theirs, the session that wrote it may still be running, and racing it for `HEAD` costs
|
||
more than the mixed message does. Ask, then split.
|
||
- **Commits are self-contained and succinct.** One coherent change per commit, typechecking and
|
||
passing tests on its own — a feature spanning layers splits as model → store → UI → docs, each
|
||
step green by itself, even when that means widening a type in the model commit that only the UI
|
||
commit uses. `docs/FEEDBACK.md` makes the same argument for adapters specifically: do not batch
|
||
six games into one commit, because each one is a fixture and a test that has to prove itself.
|
||
Succinct is about the message, not the change: a one-line subject in plain English, and a body
|
||
that says *why* — the reasoning a diff cannot show — rather than listing the files it touched.
|
||
Never reformat code the change did not touch; a formatter the project does not run buries a
|
||
100-line change in a 550-line diff.
|
||
- **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never
|
||
hand-write an interface that duplicates a schema.
|
||
- Every adapter ships a fixture in `fixtures/<game>/` and a test asserting parsed output. This is
|
||
how a source silently changing shape gets caught.
|
||
- Keep old fixtures when a source changes shape — the old one is the regression test proving the
|
||
parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the
|
||
current page and gets overwritten. Do not conflate them.
|
||
- **Colour is a token, and every token has two answers.** The app ships dark and offers light
|
||
(PRD F15), and the whole difference is a set of custom properties re-struck under
|
||
`:root[data-theme="light"]` in `styles.css`. So a component names `ink`, `hairline` or `soon` and
|
||
never a colour: a literal — `bg-white`, a hex, a hardcoded scrim — is a component that looks
|
||
right in one theme and wrong in the other, and nothing will tell you which. The two things that
|
||
genuinely cannot be tokens are the game hues, which are *data* (`games.ts`, and the reader's own),
|
||
and the pre-paint script in `index.html`; both are handled in `src/client/state/theme.ts` and
|
||
pinned by `test/theme.test.ts`.
|
||
Two rules follow. **The heat ramp must stay readable on whichever ground it is on** — it carries
|
||
meaning, and the dark theme's amber is 1.9:1 on paper, so each step is re-struck rather than
|
||
reused. And **the dark theme does not move**: `readableHue` returns dark untouched by
|
||
construction, because adding a theme is not a licence to redraw the one that shipped.
|
||
- **The page is two columns past `lg`, and the split is the one below.** What the page *tells* the
|
||
reader to do — the next deadlines, tonight's dailies — pins to a rail on the left and stays put
|
||
while the lists it *shows* them scroll beside it. Below that breakpoint it is one column in the
|
||
same order. The focus bar goes at the top of that rail rather than full-width above both columns:
|
||
it narrows what the rail holds, and a wide row of chips above everything pushes the headline
|
||
deadline down the page. On a phone, and on the timeline, which has no rail, it is back at the top
|
||
of the page — one render site per view, never both at once. The rail's rule belongs to the panel, not the column: the panel is short and the list
|
||
is long, so a full-height divider would spend most of its length walling off a gap.
|
||
- **Truncating a list is not re-sorting it.** Each section shows `LIST_CAP` rows and offers "show
|
||
all N". The rows below the cut keep their place in the order, stay counted in the header, and stay
|
||
on the timeline — so the *sorting groups, it never reorders* rule below holds for what is hidden
|
||
exactly as it does for what is shown. Expanding is per-visit state, not a stored preference: it is
|
||
something a reader does while reading one list, not a statement about how the app should work.
|
||
- **A game we add arrives switched off, and `knownGames` absent means *unrecorded*.** Adding a
|
||
source is our decision; a reader who plays two games did not ask for the other twelve. So a lane
|
||
missing from `prefs.knownGames` is new *to them* and is hidden on sight (`adoptNewLanes`, PRD F8).
|
||
The trap is the other reading: every install from before this existed has no `knownGames` at all,
|
||
and treating that as "has been offered nothing" switches off every game they already read. Seeding
|
||
records what is on their screen and changes nothing else. Lanes they invented (`mygame:`) are
|
||
recorded but never hidden.
|
||
- **The order games appear in is the reader's, and absent means unplaced.** Nothing used to decide
|
||
it: the focus bar, the settings list, the timeline's lanes and the dailies strip all rendered
|
||
`App`'s `games`, which is whichever game happened to hold the first event row. `orderGames`
|
||
(`src/client/state/gameOrder.ts`) is now the one rule, and every surface goes through it — sorted
|
||
on the **name** and never the `LaneId`, because the id is not what is printed, and through
|
||
`localeCompare`, because `<` files hololive Dreams after every capitalised game. An absent
|
||
`prefs.gameOrder` means *the reader has never placed a game*, not an empty order, so they get the
|
||
alphabetical rule — the `knownGames` distinction again, and the same trap. Two properties are
|
||
load-bearing rather than incidental. The result is **always a permutation of the lanes it was
|
||
given**: a game dropped there is indistinguishable on screen from one the reader switched off, and
|
||
switching it back on would not bring it back. And a lane it does not name **trails** the ones it
|
||
does, which is what keeps a game we add later out of the middle of a hand-made order, and what
|
||
keeps a retired source's slot warm rather than pruning it. `games` itself is left in feed order on
|
||
purpose: `adoptNewLanes` diffs it and `knownGames` is seeded from it, so ordering it at source
|
||
would let a display preference reach the code that hides a reader's games.
|
||
- **Reordering is a settings-only affordance.** The focus bar and the dailies strip are the fastest
|
||
tap targets in the app, and a drag target on top of a tick target costs somebody a streak the first
|
||
time it misfires — so the live surfaces are never draggable. Both a handle and ↑↓ buttons ship,
|
||
because touch fires no drag events at all: the arrows are the mechanism and the drag is the pointer
|
||
fast path, and being ordinary buttons is what makes the whole thing reachable by keyboard and
|
||
screen reader without a second implementation of the same interaction.
|
||
- **The settings panel is groups that state their own answer, and they ship closed.** Six
|
||
`<details>` groups — games, server region, appearance, what you see, your own, your progress — each
|
||
with its current state on the summary line (`Europe`, `17 of 18 on · A–Z`, `plus finished, not
|
||
started`). It was one open block in two columns, which is readable at four games and not at
|
||
eighteen, where the game list is eighteen rows of four controls sitting above the checkbox somebody
|
||
came to tick. The state line is the half that makes collapsing honest: without it every question
|
||
about how the app is set up costs a click, so the closed panel has to be a report and not a menu.
|
||
Three rules follow. **Do not default a group open** to make something inside it findable — the empty
|
||
states name the group as well as the switch, which is what makes that unnecessary. The summaries are
|
||
**derived from `prefs` at render**, never stored, so they cannot drift from the controls they
|
||
describe. And **one group answers one question**, which is what the sixth group is: the region and
|
||
the theme were a group called `Reading`, grouped on "both are how do I read this?" and summarising
|
||
as `Europe · Dark` — two unrelated answers joined by a dot, under a name for neither of them, in a
|
||
panel whose whole premise is that a closed line answers *its* group. The region is not a reading
|
||
preference: region-scoped ends and every daily reset are cut on that server's clock (§ Domain
|
||
rules), so it is the one control here that can make a countdown wrong, and it is now a line of its
|
||
own rather than filed behind a word for the theme.
|
||
Native `<details>` for the reason the reorder arrows are ordinary buttons: keyboard and screen
|
||
reader reach it without a second implementation. `summary` is not an `a`, `button` or `[tabindex]`,
|
||
so it needs its own `:focus-visible` rule in `styles.css` — the shared one does not reach it.
|
||
- **The panel is one column of groups, and it uses the page's width.** Those are two claims and only
|
||
the first survived a desktop read: the rows were held to `max-w-3xl` inside a shell a third wider,
|
||
so every rule inside the panel stopped short of the panel's own, and it was the one block on a wide
|
||
page not using the width — between a two-column checklist and a three-column footer that both do.
|
||
A name at one end of a row and its state at the other is what "running now" already does with a
|
||
title and its countdown, so full width is the house pattern rather than a stretch. Prose inside a
|
||
group keeps its own measure, and the game list — the one thing tall enough to need it — takes **two
|
||
columns of its own past `lg` once there are nine or more games**, flowing down then across so the
|
||
numbers still read in a straight line and an arrow still swaps a row with its neighbour. Eighteen
|
||
rows in one column was a 1,000px ribbon with two thirds of the width empty beside it.
|
||
- **A game's dailies stay together, and grouping never re-sorts inside a group.** `dailyGroups`
|
||
(`Dailies.tsx`) emits a game's standing chore followed by that game's repeating events, in the
|
||
order those arrived; it replaced `[...chores, ...repeating]`, which put a game's chore and its own
|
||
login event at opposite ends of the strip. Collapsed, the grouping is **adjacency only** — no
|
||
per-game headings, because that strip is the part of the page answerable in ten seconds and a
|
||
heading per game makes it the tallest block on it, pushing "next to expire" down the page.
|
||
- **A day already gone is still tickable, and the window bounds display and never data.** People play
|
||
at midnight and tick at breakfast, so `catchUpDays` (`src/shared/daily.ts`) offers the last
|
||
`CATCH_UP_DAYS` game-days for a standing chore and for an event whose end was never announced —
|
||
the two cases `dailyDays` cannot answer. It never returns a day past today: a tick claims you did
|
||
it, and a future pip is a control for a claim that cannot be true, so it is absent rather than
|
||
disabled. A tick older than the window stays logged and keeps counting toward the streak — nothing
|
||
prunes the log against a window, here or anywhere else.
|
||
- **Which view opens is the reader's answer.** `prefs.view` is asked once on the first run (PRD F8)
|
||
and written by the tabs from then on. It was component state, which meant a reader who preferred
|
||
the timeline was put back on the list by every reload, with nothing to blame but the app
|
||
forgetting. The stored answer is theirs; do not add a heuristic that overrides it.
|
||
- **A list row is one target.** The event row opens the event and does nothing else — status,
|
||
effort, notes and the daily checklist all live in the detail sheet. A second control inside a
|
||
full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens"
|
||
without adding a second stop for keyboard and screen-reader users.
|
||
- **Sorting groups, it never reorders within a group.** Every mode falls back to
|
||
`endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product
|
||
exists for.
|
||
- **Telling the reader to do something is not the same as showing it to them.** `showCompleted` and
|
||
`showIgnored` decide what they can *look at*; the "next to expire" headline and the dailies strip
|
||
are *instructions*, so both drop anything done or ignored regardless (`outstanding` in
|
||
`src/client/state/lens.ts`). Being pointed at a job you already finished is the bug either way.
|
||
For the same reason "next to expire" reads the minimum end date rather than the head of the list,
|
||
which under "doing first" is a different event entirely.
|
||
- **The page states its own age unprompted, and reads it off the data.** The footer says when event
|
||
data last refreshed on every load, not only past the two-day threshold — a page silent about its age
|
||
reads as current, and "how old is this?" has to be answerable before a countdown is worth trusting
|
||
(PRD F7). `freshness()` in `src/shared/feed.ts` is the one definition: it takes the newest
|
||
`lastSuccessAt` and **never `generatedAt`**, which is a build stamp that would call a
|
||
fixture-backed calendar minutes old, and it treats a game as only as fresh as its *oldest* source,
|
||
so one live wiki cannot vouch for a stalled sibling. Given that thirteen sources cannot be fetched
|
||
from CI at all (§ Scraping conduct), this disclosure is the only thing standing between a reader and
|
||
a confidently stale calendar — do not let a future change source it from the build clock.
|
||
**The notice under it is scoped to the reader's lanes; the line itself is not.** The age is a fact
|
||
about the feed, but the list beneath it is an instruction whose only remedy is "check that game's
|
||
source page" — unactionable for a game they switched off, and with nineteen lanes and two on it
|
||
buries the one they can do something about. So `Colophon` takes `prefs.hiddenGames` and filters
|
||
both the named list *and* the count the summarising branch measures against, or that branch would
|
||
never fire for a reader with most of the calendar off. Two things must not follow from it: the
|
||
credits are attribution and stay whole, and the notice **says** it is scoped — narrowing what the
|
||
footer measured while still printing "nothing has refreshed" would turn a claim about four games
|
||
into one about eighteen, in the one paragraph on the page whose whole job is being trusted about
|
||
age.
|