diff --git a/AGENTS.md b/AGENTS.md index cf89dbc..c409145 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -105,14 +105,14 @@ re-verify a sample against the live page afterward. ``` src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts custom.ts — reader-authored games and events, and their key spaces -src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts +src/ingest/ html.ts, dates.ts (ten formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts — keyed by SITE, not game adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam src/client/ React app, service worker, manifest state/ progress, daily log, ignores, prefs, sort — all localStorage useCustom.ts — the reader's own games and events (PRD F13) lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure -scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) +scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health test/ 466 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever @@ -146,9 +146,10 @@ These come from how gacha games actually schedule things, and they cause most bu - **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a recoverable disappointment; a confidently wrong date is the failure this product exists to prevent. -- **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg` and - `akwiki` serve one each — same host family, entirely different templates. Adding a source for a - known site is one `SOURCES` entry; a new site is a parser module. +- **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg`, + `akwiki` and `fandom` serve one each — the first two share a host family and have entirely + different templates. Adding a source for a known site is one `SOURCES` entry; a new site is a + parser module. - **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on every row, five months apart. Publish the one our readers are on and skip the row that lacks it — a CN date on a Global calendar is a confidently wrong date, not a near miss. diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 49c06e9..4659856 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -4,11 +4,14 @@ Six stages, run per source. Every stage writes its outcome to `ingest_runs` so a ago can be diagnosed without re-running. ``` -fetch → parse → merge → validate → reconcile → gate → publish - │ - └──► quarantine +fetch → parse → merge → validate → reconcile → gate and publish + │ │ + (sanitize) └──► quarantine ``` +Sanitizing is stage 2.5 rather than a stage of its own: it is wired into the adapter seam, so it +runs on every source without the pipeline arranging it. + ## No LLM Event data is extracted by deterministic code only. There is no model call anywhere in this @@ -156,7 +159,8 @@ produced it. | Free-form prose with no table structure | Find a different source | | A clean table whose newest row is months old | **Not a source, an archive.** Check the *latest* date before writing anything: `bluearchive.fandom.com` parses perfectly and yields zero live events, which shows up as an empty lane and a permanently rejected snapshot rather than as an error | -Game8 uses at least four page templates and a game's page may use any of them: +Game8 uses at least seven page templates, a game's page may use any of them, and one page may mix +several: 1. **Label/value detail tables** — `Event Start` / `Event End` rows under a per-event `h3`, full dates with year. *(Genshin Impact)* @@ -393,7 +397,8 @@ Every adapter ships: **Regenerating `.expected.json` from the parser makes the test self-consistent, not correct.** After an intentional change, re-verify a sample against the live page — and ideally extract the same data a second way (a throwaway script over the fixture) to confirm counts and dates independently. That -independent check is what caught the exact event counts for both current adapters. +independent check is what pinned the exact event counts on every adapter here, and it is what caught +Endfield's real events sitting in a table the first pass never read. When a source changes shape, capture a new fixture **alongside** the old one and keep both — the old fixture is the regression test proving the parser still handles the previous format. diff --git a/docs/PRD.md b/docs/PRD.md index 002fa54..a3756dd 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -45,9 +45,16 @@ event, because a missing event sends them to a wiki while a wrong one makes them | Arknights: Endfield | `endfield` | | Neverness to Everness | `nte` | -Adding a game must require no schema change — only a `GameId` entry and a source registration. -That is the test of whether the data model is right. A game may have several sources; see -`docs/INGESTION.md` § Three layers. +Added since launch, on the strength of the release thread (`docs/FEEDBACK.md` § P1): Infinity Nikki +(`nikki`), Persona 5: The Phantom X (`p5x`), Reverse: 1999 (`r1999`). **`GameId` in +`src/shared/schema.ts` is the live answer** and `SOURCES` says which of them actually have a source — +this table is the launch scope, not a roster to keep in sync. + +Adding a game must require no change to `GachaEvent` — only a `GameId` entry, its `games.ts` +metadata, and a source registration. That is the test of whether the data model is right, and it has +held: the ten games here have cost the event schema nothing. Per-game *metadata* does occasionally +grow (Reverse: 1999 needed `resetHourLocal` for a 05:00 reset), which is a different file and moves +no stored key. A game may have several sources; see `docs/INGESTION.md` § Three layers. ### Features