docs: record the FGO source, and the trap it fell into first
The adapter shipped with no docs at all, which left the one thing worth writing down unwritten: this wiki publishes two servers' schedules on two cross-linked pages, and the Japanese one parses just as cleanly as ours. That now has a row in the assessed-sources table, a section under Fandom, and a line in the ingestion pitfalls table alongside the akwiki CN column and the bawiki JP tab — three sources bitten the same way is a pattern, not a coincidence. Also records what resetOffsets cannot express, since fgo is the first game to need it: a server clock that shifts with daylight saving. DATA-MODEL's GameId listing was two games behind the schema, missing holodori as well as fgo. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9f0bb62c98
commit
efae87ae59
@@ -115,7 +115,7 @@ src/client/ React app, service worker, manifest
|
|||||||
zoom.ts — the timeline's scale ladder; pure
|
zoom.ts — the timeline's scale ladder; pure
|
||||||
scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||||||
serve.ts static server + /api/health
|
serve.ts static server + /api/health
|
||||||
test/ 579 tests
|
test/ 591 tests
|
||||||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||||||
snapshots/ current page per source, rewritten by refresh — see its README
|
snapshots/ current page per source, rewritten by refresh — see its README
|
||||||
```
|
```
|
||||||
@@ -201,10 +201,15 @@ Two more key spaces have the same property, for the same reason:
|
|||||||
at 05:00, not 04:00, so its day rolls at 10:00 UTC on its single UTC-5 server. Do not encode that
|
at 05:00, not 04:00, so its day rolls at 10:00 UTC on its single UTC-5 server. Do not encode that
|
||||||
as a bent `resetOffsets` value: shifting a game's stated server offset to land the right instant
|
as a bent `resetOffsets` value: shifting a game's stated server offset to land the right instant
|
||||||
would misreport the server clock to everything else that asks for it. Both fields are absent for
|
would misreport the server clock to everything else that asks for it. Both fields are absent for
|
||||||
every game that takes the default, which is why adding the second one moved nobody's day keys. Every day-key function takes an
|
every game that takes the default, which is why adding the second one moved nobody's day keys.
|
||||||
optional `game` — **anything reading or writing a tick must pass it**, or it writes under one
|
Neither field can express a server whose offset *shifts*: Fate/Grand Order's English server runs
|
||||||
clock and reads under another. A day that drops out of `dailyDays` renders no pip, so a tick on it
|
on US Pacific, which observes daylight saving, and one fixed number is wrong for half the year in
|
||||||
becomes unreachable; check real fixture windows before changing an offset.
|
either direction — so `fgo` takes the default and `games.ts` says why. Reaching for a value anyway
|
||||||
|
would re-label day keys twice a year, which is the one thing this whole section exists to prevent.
|
||||||
|
Every day-key function takes an optional `game` — **anything reading or writing a tick must pass
|
||||||
|
it**, or it writes under one clock and reads under another. A day that drops out of `dailyDays`
|
||||||
|
renders no pip, so a tick on it becomes unreachable; check real fixture windows before changing an
|
||||||
|
offset.
|
||||||
|
|
||||||
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
|
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
|
||||||
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
|
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
|
||||||
@@ -253,6 +258,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and
|
|||||||
| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below |
|
| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below |
|
||||||
| `bluearchive.fandom.com` | **Declined.** Fetches and parses fine; the page is the problem. Its `Event/Event_List` is a JP-server archive whose newest entry ended 2026-02-18, so all 88 rows are history and it yields **zero** live or upcoming events. An adapter would put an empty lane on the calendar and, because the runner rejects a body that parses to nothing, report a broken source forever. Same failure as the Infinity Nikki Game8 page, further along |
|
| `bluearchive.fandom.com` | **Declined.** Fetches and parses fine; the page is the problem. Its `Event/Event_List` is a JP-server archive whose newest entry ended 2026-02-18, so all 88 rows are history and it yields **zero** live or upcoming events. An adapter would put an empty lane on the calendar and, because the runner rejects a body that parses to nothing, report a broken source forever. Same failure as the Infinity Nikki Game8 page, further along |
|
||||||
| `bluearchive.wiki` | **Built** (2026-08-17), from the rendered `/wiki/Events` page — see § Blue Archive below |
|
| `bluearchive.wiki` | **Built** (2026-08-17), from the rendered `/wiki/Events` page — see § Blue Archive below |
|
||||||
|
| `fategrandorder.fandom.com` | **Built** (2026-08-18), via `api.php` like Reverse: 1999 — but off `Event_List_(US)`, **not** `Event_List`, which is the Japanese server. See § Fandom below |
|
||||||
| `holodori.wiki` | **Built** (2026-08-18), from the rendered `/wiki/Events` page. Miraheze again, so the same call as Blue Archive; CC BY-SA 4.0, no `Content-Signal`, no `Crawl-delay` for `*` |
|
| `holodori.wiki` | **Built** (2026-08-18), from the rendered `/wiki/Events` page. Miraheze again, so the same call as Blue Archive; CC BY-SA 4.0, no `Content-Signal`, no `Crawl-delay` for `*` |
|
||||||
| `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
| `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
||||||
|
|
||||||
@@ -288,6 +294,30 @@ broken build — `skipped_robots` does not touch the failure streak, and the run
|
|||||||
falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address
|
falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address
|
||||||
Fandom serves, which is how its first snapshot was taken.
|
Fandom serves, which is how its first snapshot was taken.
|
||||||
|
|
||||||
|
**Two Fandom sources now, and the second one's page is chosen, not obvious.**
|
||||||
|
`fategrandorder.fandom.com` publishes two schedules: `Event_List` opens "This page lists all Events
|
||||||
|
in Fate/Grand Order Japan", and `Event_List_(US)` is the English server. They run months apart, each
|
||||||
|
links the other, and reading the Japanese one on an English calendar is the `akwiki` CN column again
|
||||||
|
— it was how this source first landed, and every date it published was a JP date. The adapter is
|
||||||
|
pointed at `page=Event_List_(US)` and a test asserts it; `parsers/fandom.ts` carries the reasoning.
|
||||||
|
|
||||||
|
Three more things about that page, all of them ways to publish or lose a date:
|
||||||
|
|
||||||
|
- **Its sections are fenced by pictures.** `ONGOING EVENTS`, `FUTURE EVENTS` and `PAST EVENTS` are
|
||||||
|
banner images with the label drawn in a positioned `<div>` over them — no heading, no id. Only the
|
||||||
|
ongoing section is parsed, and `canParse` asserts both of the dividers that bound it, so a
|
||||||
|
redesign fails the source rather than emptying the lane.
|
||||||
|
- **The other two sections cannot be dated, and that is the whole reason they are skipped.**
|
||||||
|
`FUTURE EVENTS` gives an ETA of `August 2026` — a month with no day, and a day is half an event
|
||||||
|
ID. `PAST EVENTS` is 111 monthly tables that state no year anywhere; the *Japanese* page's
|
||||||
|
equivalents carry it in a `MMYYYY` table id, which is a difference easily assumed away.
|
||||||
|
- **Every duration names a zone and no clock** — `August 12, 2026 ~ August 26, 2026 PDT`. So the
|
||||||
|
boundaries stay on the day the page states rather than being shifted into UTC: there is no time of
|
||||||
|
day to anchor a conversion to, and the start's day is part of the event ID. That `PDT` is also the
|
||||||
|
evidence that the English server is one machine on US Pacific — see `games.ts`, where it does
|
||||||
|
*not* become a `resetOffsets` entry, because Pacific observes daylight saving and that field holds
|
||||||
|
one fixed number.
|
||||||
|
|
||||||
**Blue Archive: the page, never the API — the opposite call to Fandom.** `bluearchive.wiki` is a
|
**Blue Archive: the page, never the API — the opposite call to Fandom.** `bluearchive.wiki` is a
|
||||||
Miraheze wiki, and Miraheze's `robots.txt` **disallows** `/w/` and `/*?action=`. So the route
|
Miraheze wiki, and Miraheze's `robots.txt` **disallows** `/w/` and `/*?action=`. So the route
|
||||||
`parsers/fandom.ts` takes is the one that is closed here, and the rendered `/wiki/Events` page is the
|
`parsers/fandom.ts` takes is the one that is closed here, and the rendered `/wiki/Events` page is the
|
||||||
|
|||||||
+1
-1
@@ -10,7 +10,7 @@ import { z } from "zod";
|
|||||||
|
|
||||||
export const GameId = z.enum([
|
export const GameId = z.enum([
|
||||||
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "nikki", "p5x", "r1999",
|
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield", "nte", "nikki", "p5x", "r1999",
|
||||||
"ba",
|
"ba", "fgo", "holodori",
|
||||||
]);
|
]);
|
||||||
|
|
||||||
export const EventType = z.enum([
|
export const EventType = z.enum([
|
||||||
|
|||||||
+23
-2
@@ -51,7 +51,7 @@ Consequences worth internalising:
|
|||||||
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X |
|
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X |
|
||||||
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
|
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
|
||||||
| `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights |
|
| `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights |
|
||||||
| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — `Event \| Time Period \| Version` wikitables | Reverse: 1999, Fate/Grand Order |
|
| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — two page templates: `Event \| Time Period \| Version` wikitables, and FGO's picture-fenced `ONGOING EVENTS` blocks | Reverse: 1999, Fate/Grand Order |
|
||||||
| `bawiki` | bluearchive.wiki's rendered `/wiki/Events` — a JP/Global tabber over `Name (EN) \| Start date \| End date \| Notes` wikitables | Blue Archive |
|
| `bawiki` | bluearchive.wiki's rendered `/wiki/Events` — a JP/Global tabber over `Name (EN) \| Start date \| End date \| Notes` wikitables | Blue Archive |
|
||||||
| `holodoriwiki` | holodori.wiki's rendered `/wiki/Events` — `Current Events` and `Past Events` wikitables over `Event \| Type \| Start Date \| End Date` | hololive Dreams |
|
| `holodoriwiki` | holodori.wiki's rendered `/wiki/Events` — `Current Events` and `Past Events` wikitables over `Event \| Type \| Start Date \| End Date` | hololive Dreams |
|
||||||
|
|
||||||
@@ -75,6 +75,26 @@ rather than parse to zero events. Two page facts drive the rest of it:
|
|||||||
anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so;
|
anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so;
|
||||||
`akwiki` and `game8` can gate on a heading instead, and should where one exists.
|
`akwiki` and `game8` can gate on a heading instead, and should where one exists.
|
||||||
|
|
||||||
|
The second Fandom source, Fate/Grand Order, shares the envelope and nothing else, so `fandom` is one
|
||||||
|
parser over two templates the way `game8` is one over seven. `canParse` and the parse branch both
|
||||||
|
route on the same check, and the differences are worth knowing before touching either:
|
||||||
|
|
||||||
|
- **The page is a choice.** `fategrandorder.fandom.com` publishes `Event_List` (Japanese server) and
|
||||||
|
`Event_List_(US)` (English), months apart and cross-linked. The adapter reads the `(US)` one; a
|
||||||
|
test asserts the URL, because this source shipped once off the Japanese page and every date it
|
||||||
|
published was wrong by a server. AGENTS.md § Fandom has the rest.
|
||||||
|
- **Sections are fenced by pictures, not headings.** `ONGOING EVENTS`, `FUTURE EVENTS` and
|
||||||
|
`PAST EVENTS` are banner images with the label in a positioned `<div>`. Only the ongoing section
|
||||||
|
is read, and `canParse` asserts both dividers bounding it.
|
||||||
|
- **The other two sections are undatable, which is why they are skipped rather than filtered.**
|
||||||
|
Upcoming rows give a month and no day; the 111 past tables state no year at all — unlike the
|
||||||
|
Japanese page's, which carry it in a `MMYYYY` table id.
|
||||||
|
- **Durations name a zone but no clock** (`August 12, 2026 ~ August 26, 2026 PDT`), so the stated
|
||||||
|
calendar day is kept as-is: there is no time of day for a UTC conversion to anchor to, and the
|
||||||
|
start's day is half the event ID.
|
||||||
|
- **`(US)` is stripped from the title and kept in the URL.** It disambiguates the English article
|
||||||
|
from the Japanese one, so it belongs to the article's name and not to the event's.
|
||||||
|
|
||||||
`bawiki` is the mirror image of `fandom`: same MediaWiki software, opposite conclusion about which
|
`bawiki` is the mirror image of `fandom`: same MediaWiki software, opposite conclusion about which
|
||||||
surface to read. bluearchive.wiki is Miraheze, whose `robots.txt` disallows `/w/` and `/*?action=`,
|
surface to read. bluearchive.wiki is Miraheze, whose `robots.txt` disallows `/w/` and `/*?action=`,
|
||||||
so the API is closed and the rendered `/wiki/Events` page is what `*` is allowed — and it serves our
|
so the API is closed and the rendered `/wiki/Events` page is what `*` is allowed — and it serves our
|
||||||
@@ -111,7 +131,7 @@ All live in `src/ingest/dates.ts`, each returning null rather than inferring any
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `parseMonthDayYear` | `August 12, 2026` | Genshin detail rows |
|
| `parseMonthDayYear` | `August 12, 2026` | Genshin detail rows |
|
||||||
| `parseMonthDayRange` | `August 12 - September 21, 2026` (year on the end only) | Genshin, NTE |
|
| `parseMonthDayRange` | `August 12 - September 21, 2026` (year on the end only) | Genshin, NTE |
|
||||||
| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves |
|
| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves, Fate/Grand Order |
|
||||||
| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield |
|
| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield |
|
||||||
| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events |
|
| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events |
|
||||||
| `parseLabelledStartEnd` | `Start: January 24, 2025 End: Permanent` | Infinity Nikki |
|
| `parseLabelledStartEnd` | `Start: January 24, 2025 End: Permanent` | Infinity Nikki |
|
||||||
@@ -177,6 +197,7 @@ produced it.
|
|||||||
| Label/value or column tables with full dates including a year | Good — an existing parser may already handle it |
|
| Label/value or column tables with full dates including a year | Good — an existing parser may already handle it |
|
||||||
| Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing |
|
| Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing |
|
||||||
| Free-form prose with no table structure | Find a different source |
|
| Free-form prose with no table structure | Find a different source |
|
||||||
|
| One wiki, two servers' schedules | **Read which one before writing anything.** Three sources here publish both: `akwiki` in a CN column, `bawiki` in a JP tab, `fategrandorder.fandom.com` on a whole separate page that says so in its first sentence. Every one of them parses cleanly and every one of them is months wrong for our readers |
|
||||||
| A clean table whose newest row is months old | **Not a source, an archive.** Check the *latest* date before writing anything: `bluearchive.fandom.com` parses perfectly and yields zero live events, which shows up as an empty lane and a permanently rejected snapshot rather than as an error |
|
| A clean table whose newest row is months old | **Not a source, an archive.** Check the *latest* date before writing anything: `bluearchive.fandom.com` parses perfectly and yields zero live events, which shows up as an empty lane and a permanently rejected snapshot rather than as an error |
|
||||||
|
|
||||||
Game8 uses at least seven page templates, a game's page may use any of them, and one page may mix
|
Game8 uses at least seven page templates, a game's page may use any of them, and one page may mix
|
||||||
|
|||||||
Reference in New Issue
Block a user