docs: record the two new sources and the shapes they added
Nine sources across eight games now, two more Game8 page shapes, two more date formats, and the h4 rule that neither game's page can be read without. Also notes what the two new games do *not* have: no `resetOffsets` entry, so both take the regional default. Neither source states a server map, and `resetOffsets` feeds `dayKey`, which is a localStorage key — so that is recorded as an assumption to confirm rather than left to look like a verified fact. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5
Claude Opus 5
parent
d8b539bce1
commit
8c2ec19c22
@@ -25,7 +25,7 @@ Fetch the page exactly once.
|
||||
**3. Decide whether an existing parser covers it.**
|
||||
|
||||
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game. Two exist: `game8`
|
||||
(six sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first:
|
||||
(eight sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first:
|
||||
|
||||
- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
|
||||
parsing code. This is the common case and should be the first thing you try.
|
||||
|
||||
@@ -8,8 +8,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p
|
||||
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
||||
progress on events that repeat daily, and lets a user mark events completed.
|
||||
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, two parsers, seven sources across
|
||||
six games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, two parsers, nine sources across
|
||||
eight games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
||||
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
||||
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
||||
@@ -76,7 +76,7 @@ re-verify a sample against the live page afterward.
|
||||
|
||||
```
|
||||
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
||||
src/ingest/ html.ts, dates.ts (six formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||||
src/ingest/ html.ts, dates.ts (eight formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||||
parsers/ game8.ts, wikigg.ts — keyed by SITE, not game
|
||||
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
||||
src/client/ React app, service worker, manifest
|
||||
@@ -84,7 +84,7 @@ src/client/ React app, service worker, manifest
|
||||
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
||||
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||||
serve.ts static server + /api/health
|
||||
test/ 318 tests
|
||||
test/ 348 tests
|
||||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||||
snapshots/ current page per source, rewritten by refresh — see its README
|
||||
```
|
||||
@@ -116,14 +116,19 @@ These come from how gacha games actually schedule things, and they cause most bu
|
||||
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
|
||||
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
|
||||
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
|
||||
- **Parsers are keyed by site, not game.** One `game8` parser serves six sources; `wikigg` serves
|
||||
- **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg` serves
|
||||
one. Adding a source for a known site is one `SOURCES` entry; a new site is a parser module.
|
||||
- **Game8 has no single template.** Five shapes are known and a page may mix them: label/value
|
||||
- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value
|
||||
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
|
||||
cells, and rowspan Start/End pairs. Full table in `docs/INGESTION.md`. Before assuming a new
|
||||
cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `<hr>`-separated date pairs.
|
||||
Full table in `docs/INGESTION.md`. Before assuming a new
|
||||
Game8 page will work, dump its structure and check **every** table — Endfield was written off as
|
||||
undatable on a pass that only inspected its `Duration` rows, and its real events were further
|
||||
down the page.
|
||||
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
|
||||
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
|
||||
accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but
|
||||
never names one — an unrecognised `h4` must leave the current event title alone.
|
||||
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
|
||||
per server region, which is the only reason `regionEnds` carries real data anywhere.
|
||||
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
|
||||
|
||||
+7
-1
@@ -9,7 +9,7 @@
|
||||
import { z } from "zod";
|
||||
|
||||
export const GameId = z.enum([
|
||||
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte",
|
||||
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "nikki", "p5x",
|
||||
]);
|
||||
|
||||
export const EventType = z.enum([
|
||||
@@ -254,9 +254,15 @@ reader cannot see.
|
||||
| Game | Reset (server local) | Server offset | Reset (UTC) | Copenhagen, summer / winter |
|
||||
|---|---|---|---|---|
|
||||
| Genshin, Star Rail, ZZZ, Wuwa, NTE | 04:00 | region (EU = UTC+1) | 03:00 | 05:00 / 04:00 |
|
||||
| Infinity Nikki, P5X | 04:00 | region (assumed) | 03:00 | 05:00 / 04:00 |
|
||||
| Endfield, Europe | 04:00 | UTC-5 (on the Americas server) | 09:00 | 11:00 / 10:00 |
|
||||
| Endfield, Asia / Americas | 04:00 | regional default | 20:00 / 09:00 | — |
|
||||
|
||||
Infinity Nikki and P5X carry **no `resetOffsets` entry**, so they take the regional default. That is
|
||||
an assumption, not a verified server map — neither source states one. Confirm it against the games
|
||||
before relying on it, and note that adding an override later re-labels the game-day of ticks readers
|
||||
have already logged, which is the change this table warns about below.
|
||||
|
||||
These server offsets are **fixed and do not observe DST**, so the reader's local reset time moves by
|
||||
an hour across the European clock change while the UTC instant stays put.
|
||||
|
||||
|
||||
+23
-2
@@ -45,7 +45,7 @@ Consequences worth internalising:
|
||||
|
||||
| Parser | Site | Sources using it |
|
||||
|---|---|---|
|
||||
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE |
|
||||
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X |
|
||||
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
|
||||
|
||||
`wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server
|
||||
@@ -63,11 +63,19 @@ All live in `src/ingest/dates.ts`, each returning null rather than inferring any
|
||||
| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves |
|
||||
| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield |
|
||||
| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events |
|
||||
| `parseLabelledStartEnd` | `Start: January 24, 2025 End: Permanent` | Infinity Nikki |
|
||||
| `parseAdjacentFullRange` | `July 30, 2026 August 13, 2026` (halves split by an `<hr>`) | Persona 5: The Phantom X |
|
||||
| `parseOpenRange` | `Jul. 24, 2026 - End of 4.6`, `July 10, 2026 - Permanent` | Star Rail, Wuthering Waves |
|
||||
|
||||
`parseOpenRange` is tried last because it is the most permissive — it accepts any leading full date
|
||||
and reports no end.
|
||||
|
||||
The last two are anchored at both ends and require a year on each half, which is what keeps them from
|
||||
eating prose. `August 12, 2026 Day 3 rewards are doubled` would otherwise read "Day 3" as an end, and
|
||||
`June 25, 2026 July 16/30, 2026` names *two* candidate ends — so it takes neither, and the leftover is
|
||||
not shown as a summary either (a date the parser refused to trust must not reappear dressed as
|
||||
information).
|
||||
|
||||
### The parser interface
|
||||
|
||||
```ts
|
||||
@@ -125,12 +133,25 @@ Game8 uses at least four page templates and a game's page may use any of them:
|
||||
(`Period: 08/09/26 - 08/30/26 During the event...`). *(Arknights: Endfield)*
|
||||
5. **Rowspan Start/End pairs** — the event name spans two rows, so a flat cell reader sees
|
||||
`[title, "Start", date]` then `["End", date]`. *(Zenless Zone Zero)*
|
||||
6. **Labelled cells** — one cell holding `Start: <date>` and `End: <date>` split by a `<br>`, where
|
||||
the end half is often the word `Permanent`. *(Infinity Nikki)*
|
||||
7. **`<hr>`-separated pairs** — a `Event | Duration` table whose two dates are divided by a rule
|
||||
rather than a dash, so a tag-stripping reader sees only whitespace between them. The same page
|
||||
repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of
|
||||
prose; those corroborate the dates and supply the blurb the flat table lacks.
|
||||
*(Persona 5: The Phantom X)*
|
||||
|
||||
Shapes 1, 2, 4 and 5 are handled. Before assuming a new Game8 page will work, dump its heading/table
|
||||
Shapes 1, 2, 4, 5, 6 and 7 are handled. Before assuming a new Game8 page will work, dump its heading/table
|
||||
structure and check which shape it uses — and check **every** table, not just the obvious one.
|
||||
Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its
|
||||
two real events were in a table further down.
|
||||
|
||||
**Check what ends a section, too.** Headings decide inclusion and the level is not consistent: Persona
|
||||
5 puts its whole finished back catalogue behind an `<h4>Finished Events</h4>` inside a collapsed
|
||||
accordion, so a reader that ignores `h4` publishes fifty dead events. Genshin uses `h4` the opposite
|
||||
way — for sub-headings *within* one event ("Availability Period") — so an unrecognised `h4` gates the
|
||||
section but must never claim the event title.
|
||||
|
||||
## Stage 1 — fetch
|
||||
|
||||
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
|
||||
|
||||
Reference in New Issue
Block a user