From efae87ae59e1e2889773c5678442ec4745daab33 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Tue, 18 Aug 2026 21:30:38 +0200 Subject: [PATCH] docs: record the FGO source, and the trap it fell into first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The adapter shipped with no docs at all, which left the one thing worth writing down unwritten: this wiki publishes two servers' schedules on two cross-linked pages, and the Japanese one parses just as cleanly as ours. That now has a row in the assessed-sources table, a section under Fandom, and a line in the ingestion pitfalls table alongside the akwiki CN column and the bawiki JP tab — three sources bitten the same way is a pattern, not a coincidence. Also records what resetOffsets cannot express, since fgo is the first game to need it: a server clock that shifts with daylight saving. DATA-MODEL's GameId listing was two games behind the schema, missing holodori as well as fgo. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 40 +++++++++++++++++++++++++++++++++++----- docs/DATA-MODEL.md | 2 +- docs/INGESTION.md | 25 +++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index da35fdd..4be5090 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -115,7 +115,7 @@ src/client/ React app, service worker, manifest zoom.ts — the timeline's scale ladder; pure scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 579 tests +test/ 591 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -201,10 +201,15 @@ Two more key spaces have the same property, for the same reason: at 05:00, not 04:00, so its day rolls at 10:00 UTC on its single UTC-5 server. Do not encode that as a bent `resetOffsets` value: shifting a game's stated server offset to land the right instant would misreport the server clock to everything else that asks for it. Both fields are absent for - every game that takes the default, which is why adding the second one moved nobody's day keys. Every day-key function takes an - optional `game` — **anything reading or writing a tick must pass it**, or it writes under one - clock and reads under another. A day that drops out of `dailyDays` renders no pip, so a tick on it - becomes unreachable; check real fixture windows before changing an offset. + every game that takes the default, which is why adding the second one moved nobody's day keys. + Neither field can express a server whose offset *shifts*: Fate/Grand Order's English server runs + on US Pacific, which observes daylight saving, and one fixed number is wrong for half the year in + either direction — so `fgo` takes the default and `games.ts` says why. Reaching for a value anyway + would re-label day keys twice a year, which is the one thing this whole section exists to prevent. + Every day-key function takes an optional `game` — **anything reading or writing a tick must pass + it**, or it writes under one clock and reads under another. A day that drops out of `dailyDays` + renders no pip, so a tick on it becomes unreachable; check real fixture windows before changing an + offset. The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real @@ -253,6 +258,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and | `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below | | `bluearchive.fandom.com` | **Declined.** Fetches and parses fine; the page is the problem. Its `Event/Event_List` is a JP-server archive whose newest entry ended 2026-02-18, so all 88 rows are history and it yields **zero** live or upcoming events. An adapter would put an empty lane on the calendar and, because the runner rejects a body that parses to nothing, report a broken source forever. Same failure as the Infinity Nikki Game8 page, further along | | `bluearchive.wiki` | **Built** (2026-08-17), from the rendered `/wiki/Events` page — see § Blue Archive below | +| `fategrandorder.fandom.com` | **Built** (2026-08-18), via `api.php` like Reverse: 1999 — but off `Event_List_(US)`, **not** `Event_List`, which is the Japanese server. See § Fandom below | | `holodori.wiki` | **Built** (2026-08-18), from the rendered `/wiki/Events` page. Miraheze again, so the same call as Blue Archive; CC BY-SA 4.0, no `Content-Signal`, no `Crawl-delay` for `*` | | `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h | @@ -288,6 +294,30 @@ broken build — `skipped_robots` does not touch the failure streak, and the run falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address Fandom serves, which is how its first snapshot was taken. +**Two Fandom sources now, and the second one's page is chosen, not obvious.** +`fategrandorder.fandom.com` publishes two schedules: `Event_List` opens "This page lists all Events +in Fate/Grand Order Japan", and `Event_List_(US)` is the English server. They run months apart, each +links the other, and reading the Japanese one on an English calendar is the `akwiki` CN column again +— it was how this source first landed, and every date it published was a JP date. The adapter is +pointed at `page=Event_List_(US)` and a test asserts it; `parsers/fandom.ts` carries the reasoning. + +Three more things about that page, all of them ways to publish or lose a date: + +- **Its sections are fenced by pictures.** `ONGOING EVENTS`, `FUTURE EVENTS` and `PAST EVENTS` are + banner images with the label drawn in a positioned `
` over them — no heading, no id. Only the + ongoing section is parsed, and `canParse` asserts both of the dividers that bound it, so a + redesign fails the source rather than emptying the lane. +- **The other two sections cannot be dated, and that is the whole reason they are skipped.** + `FUTURE EVENTS` gives an ETA of `August 2026` — a month with no day, and a day is half an event + ID. `PAST EVENTS` is 111 monthly tables that state no year anywhere; the *Japanese* page's + equivalents carry it in a `MMYYYY` table id, which is a difference easily assumed away. +- **Every duration names a zone and no clock** — `August 12, 2026 ~ August 26, 2026 PDT`. So the + boundaries stay on the day the page states rather than being shifted into UTC: there is no time of + day to anchor a conversion to, and the start's day is part of the event ID. That `PDT` is also the + evidence that the English server is one machine on US Pacific — see `games.ts`, where it does + *not* become a `resetOffsets` entry, because Pacific observes daylight saving and that field holds + one fixed number. + **Blue Archive: the page, never the API — the opposite call to Fandom.** `bluearchive.wiki` is a Miraheze wiki, and Miraheze's `robots.txt` **disallows** `/w/` and `/*?action=`. So the route `parsers/fandom.ts` takes is the one that is closed here, and the rendered `/wiki/Events` page is the diff --git a/docs/DATA-MODEL.md b/docs/DATA-MODEL.md index ee52bac..0ef2875 100644 --- a/docs/DATA-MODEL.md +++ b/docs/DATA-MODEL.md @@ -10,7 +10,7 @@ import { z } from "zod"; export const GameId = z.enum([ "genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "nikki", "p5x", "r1999", - "ba", + "ba", "fgo", "holodori", ]); export const EventType = z.enum([ diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 9a7584a..8e6be25 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -51,7 +51,7 @@ Consequences worth internalising: | `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X | | `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | | `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights | -| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — `Event \| Time Period \| Version` wikitables | Reverse: 1999, Fate/Grand Order | +| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — two page templates: `Event \| Time Period \| Version` wikitables, and FGO's picture-fenced `ONGOING EVENTS` blocks | Reverse: 1999, Fate/Grand Order | | `bawiki` | bluearchive.wiki's rendered `/wiki/Events` — a JP/Global tabber over `Name (EN) \| Start date \| End date \| Notes` wikitables | Blue Archive | | `holodoriwiki` | holodori.wiki's rendered `/wiki/Events` — `Current Events` and `Past Events` wikitables over `Event \| Type \| Start Date \| End Date` | hololive Dreams | @@ -75,6 +75,26 @@ rather than parse to zero events. Two page facts drive the rest of it: anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so; `akwiki` and `game8` can gate on a heading instead, and should where one exists. +The second Fandom source, Fate/Grand Order, shares the envelope and nothing else, so `fandom` is one +parser over two templates the way `game8` is one over seven. `canParse` and the parse branch both +route on the same check, and the differences are worth knowing before touching either: + +- **The page is a choice.** `fategrandorder.fandom.com` publishes `Event_List` (Japanese server) and + `Event_List_(US)` (English), months apart and cross-linked. The adapter reads the `(US)` one; a + test asserts the URL, because this source shipped once off the Japanese page and every date it + published was wrong by a server. AGENTS.md § Fandom has the rest. +- **Sections are fenced by pictures, not headings.** `ONGOING EVENTS`, `FUTURE EVENTS` and + `PAST EVENTS` are banner images with the label in a positioned `
`. Only the ongoing section + is read, and `canParse` asserts both dividers bounding it. +- **The other two sections are undatable, which is why they are skipped rather than filtered.** + Upcoming rows give a month and no day; the 111 past tables state no year at all — unlike the + Japanese page's, which carry it in a `MMYYYY` table id. +- **Durations name a zone but no clock** (`August 12, 2026 ~ August 26, 2026 PDT`), so the stated + calendar day is kept as-is: there is no time of day for a UTC conversion to anchor to, and the + start's day is half the event ID. +- **`(US)` is stripped from the title and kept in the URL.** It disambiguates the English article + from the Japanese one, so it belongs to the article's name and not to the event's. + `bawiki` is the mirror image of `fandom`: same MediaWiki software, opposite conclusion about which surface to read. bluearchive.wiki is Miraheze, whose `robots.txt` disallows `/w/` and `/*?action=`, so the API is closed and the rendered `/wiki/Events` page is what `*` is allowed — and it serves our @@ -111,7 +131,7 @@ All live in `src/ingest/dates.ts`, each returning null rather than inferring any |---|---|---| | `parseMonthDayYear` | `August 12, 2026` | Genshin detail rows | | `parseMonthDayRange` | `August 12 - September 21, 2026` (year on the end only) | Genshin, NTE | -| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves | +| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves, Fate/Grand Order | | `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield | | `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events | | `parseLabelledStartEnd` | `Start: January 24, 2025 End: Permanent` | Infinity Nikki | @@ -177,6 +197,7 @@ produced it. | Label/value or column tables with full dates including a year | Good — an existing parser may already handle it | | Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing | | Free-form prose with no table structure | Find a different source | +| One wiki, two servers' schedules | **Read which one before writing anything.** Three sources here publish both: `akwiki` in a CN column, `bawiki` in a JP tab, `fategrandorder.fandom.com` on a whole separate page that says so in its first sentence. Every one of them parses cleanly and every one of them is months wrong for our readers | | A clean table whose newest row is months old | **Not a source, an archive.** Check the *latest* date before writing anything: `bluearchive.fandom.com` parses perfectly and yields zero live events, which shows up as an empty lane and a permanently rejected snapshot rather than as an error | Game8 uses at least seven page templates, a game's page may use any of them, and one page may mix