docs: record the two new sources and the shapes they added

Nine sources across eight games now, two more Game8 page shapes, two more
date formats, and the h4 rule that neither game's page can be read
without.

Also notes what the two new games do *not* have: no `resetOffsets` entry,
so both take the regional default. Neither source states a server map, and
`resetOffsets` feeds `dayKey`, which is a localStorage key — so that is
recorded as an assumption to confirm rather than left to look like a
verified fact.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-17 17:21:19 +02:00
co-authored by Claude Opus 5
parent d8b539bce1
commit 8c2ec19c22
4 changed files with 43 additions and 11 deletions
+1 -1
View File
@@ -25,7 +25,7 @@ Fetch the page exactly once.
**3. Decide whether an existing parser covers it.** **3. Decide whether an existing parser covers it.**
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game. Two exist: `game8` Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game. Two exist: `game8`
(six sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first: (eight sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first:
- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new - **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
parsing code. This is the common case and should be the first thing you try. parsing code. This is the common case and should be the first thing you try.
+12 -7
View File
@@ -8,8 +8,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
progress on events that repeat daily, and lets a user mark events completed. progress on events that repeat daily, and lets a user mark events completed.
**Status: working app, refreshing itself on a schedule.** Schema, two parsers, seven sources across **Status: working app, refreshing itself on a schedule.** Schema, two parsers, nine sources across
six games, the full interface, offline support, a static server, a Docker image and CI all exist and eight games, the full interface, offline support, a static server, a Docker image and CI all exist and
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
@@ -76,7 +76,7 @@ re-verify a sample against the live page afterward.
``` ```
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
src/ingest/ html.ts, dates.ts (six formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts src/ingest/ html.ts, dates.ts (eight formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
parsers/ game8.ts, wikigg.ts — keyed by SITE, not game parsers/ game8.ts, wikigg.ts — keyed by SITE, not game
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
src/client/ React app, service worker, manifest src/client/ React app, service worker, manifest
@@ -84,7 +84,7 @@ src/client/ React app, service worker, manifest
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
serve.ts static server + /api/health serve.ts static server + /api/health
test/ 318 tests test/ 348 tests
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
snapshots/ current page per source, rewritten by refresh — see its README snapshots/ current page per source, rewritten by refresh — see its README
``` ```
@@ -116,14 +116,19 @@ These come from how gacha games actually schedule things, and they cause most bu
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing - **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent. recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
- **Parsers are keyed by site, not game.** One `game8` parser serves six sources; `wikigg` serves - **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg` serves
one. Adding a source for a known site is one `SOURCES` entry; a new site is a parser module. one. Adding a source for a known site is one `SOURCES` entry; a new site is a parser module.
- **Game8 has no single template.** Five shapes are known and a page may mix them: label/value - **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
cells, and rowspan Start/End pairs. Full table in `docs/INGESTION.md`. Before assuming a new cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `<hr>`-separated date pairs.
Full table in `docs/INGESTION.md`. Before assuming a new
Game8 page will work, dump its structure and check **every** table — Endfield was written off as Game8 page will work, dump its structure and check **every** table — Endfield was written off as
undatable on a pass that only inspected its `Duration` rows, and its real events were further undatable on a pass that only inspected its `Duration` rows, and its real events were further
down the page. down the page.
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but
never names one — an unrecognised `h4` must leave the current event title alone.
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer - **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
per server region, which is the only reason `regionEnds` carries real data anywhere. per server region, which is the only reason `regionEnds` carries real data anywhere.
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes - **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
+7 -1
View File
@@ -9,7 +9,7 @@
import { z } from "zod"; import { z } from "zod";
export const GameId = z.enum([ export const GameId = z.enum([
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "nikki", "p5x",
]); ]);
export const EventType = z.enum([ export const EventType = z.enum([
@@ -254,9 +254,15 @@ reader cannot see.
| Game | Reset (server local) | Server offset | Reset (UTC) | Copenhagen, summer / winter | | Game | Reset (server local) | Server offset | Reset (UTC) | Copenhagen, summer / winter |
|---|---|---|---|---| |---|---|---|---|---|
| Genshin, Star Rail, ZZZ, Wuwa, NTE | 04:00 | region (EU = UTC+1) | 03:00 | 05:00 / 04:00 | | Genshin, Star Rail, ZZZ, Wuwa, NTE | 04:00 | region (EU = UTC+1) | 03:00 | 05:00 / 04:00 |
| Infinity Nikki, P5X | 04:00 | region (assumed) | 03:00 | 05:00 / 04:00 |
| Endfield, Europe | 04:00 | UTC-5 (on the Americas server) | 09:00 | 11:00 / 10:00 | | Endfield, Europe | 04:00 | UTC-5 (on the Americas server) | 09:00 | 11:00 / 10:00 |
| Endfield, Asia / Americas | 04:00 | regional default | 20:00 / 09:00 | — | | Endfield, Asia / Americas | 04:00 | regional default | 20:00 / 09:00 | — |
Infinity Nikki and P5X carry **no `resetOffsets` entry**, so they take the regional default. That is
an assumption, not a verified server map — neither source states one. Confirm it against the games
before relying on it, and note that adding an override later re-labels the game-day of ticks readers
have already logged, which is the change this table warns about below.
These server offsets are **fixed and do not observe DST**, so the reader's local reset time moves by These server offsets are **fixed and do not observe DST**, so the reader's local reset time moves by
an hour across the European clock change while the UTC instant stays put. an hour across the European clock change while the UTC instant stays put.
+23 -2
View File
@@ -45,7 +45,7 @@ Consequences worth internalising:
| Parser | Site | Sources using it | | Parser | Site | Sources using it |
|---|---|---| |---|---|---|
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE | | `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X |
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | | `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
`wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server `wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server
@@ -63,11 +63,19 @@ All live in `src/ingest/dates.ts`, each returning null rather than inferring any
| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves | | `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves |
| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield | | `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield |
| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events | | `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events |
| `parseLabelledStartEnd` | `Start: January 24, 2025 End: Permanent` | Infinity Nikki |
| `parseAdjacentFullRange` | `July 30, 2026 August 13, 2026` (halves split by an `<hr>`) | Persona 5: The Phantom X |
| `parseOpenRange` | `Jul. 24, 2026 - End of 4.6`, `July 10, 2026 - Permanent` | Star Rail, Wuthering Waves | | `parseOpenRange` | `Jul. 24, 2026 - End of 4.6`, `July 10, 2026 - Permanent` | Star Rail, Wuthering Waves |
`parseOpenRange` is tried last because it is the most permissive — it accepts any leading full date `parseOpenRange` is tried last because it is the most permissive — it accepts any leading full date
and reports no end. and reports no end.
The last two are anchored at both ends and require a year on each half, which is what keeps them from
eating prose. `August 12, 2026 Day 3 rewards are doubled` would otherwise read "Day 3" as an end, and
`June 25, 2026 July 16/30, 2026` names *two* candidate ends — so it takes neither, and the leftover is
not shown as a summary either (a date the parser refused to trust must not reappear dressed as
information).
### The parser interface ### The parser interface
```ts ```ts
@@ -125,12 +133,25 @@ Game8 uses at least four page templates and a game's page may use any of them:
(`Period: 08/09/26 - 08/30/26 During the event...`). *(Arknights: Endfield)* (`Period: 08/09/26 - 08/30/26 During the event...`). *(Arknights: Endfield)*
5. **Rowspan Start/End pairs** — the event name spans two rows, so a flat cell reader sees 5. **Rowspan Start/End pairs** — the event name spans two rows, so a flat cell reader sees
`[title, "Start", date]` then `["End", date]`. *(Zenless Zone Zero)* `[title, "Start", date]` then `["End", date]`. *(Zenless Zone Zero)*
6. **Labelled cells** — one cell holding `Start: <date>` and `End: <date>` split by a `<br>`, where
the end half is often the word `Permanent`. *(Infinity Nikki)*
7. **`<hr>`-separated pairs** — a `Event | Duration` table whose two dates are divided by a rule
rather than a dash, so a tag-stripping reader sees only whitespace between them. The same page
repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of
prose; those corroborate the dates and supply the blurb the flat table lacks.
*(Persona 5: The Phantom X)*
Shapes 1, 2, 4 and 5 are handled. Before assuming a new Game8 page will work, dump its heading/table Shapes 1, 2, 4, 5, 6 and 7 are handled. Before assuming a new Game8 page will work, dump its heading/table
structure and check which shape it uses — and check **every** table, not just the obvious one. structure and check which shape it uses — and check **every** table, not just the obvious one.
Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its
two real events were in a table further down. two real events were in a table further down.
**Check what ends a section, too.** Headings decide inclusion and the level is not consistent: Persona
5 puts its whole finished back catalogue behind an `<h4>Finished Events</h4>` inside a collapsed
accordion, so a reader that ignores `h4` publishes fifty dead events. Genshin uses `h4` the opposite
way — for sub-headings *within* one event ("Availability Period") — so an unrecognised `h4` gates the
section but must never claim the event title.
## Stage 1 — fetch ## Stage 1 — fetch
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends - Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends