From 3377fcb2112da9dca3d59286a6a02267d25dfd11 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Mon, 17 Aug 2026 22:46:47 +0200 Subject: [PATCH] feat(ingest): add the Reverse: 1999 event source via Fandom's API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New `fandom` parser plus the r1999-fandom-events source, giving the game six live and upcoming events at exact precision. It fetches api.php?action=parse, not /wiki/Events. The rendered page answers a non-browser client with a Cloudflare challenge, and getting past that would be defeating an access control — the reason uma.moe was declined. The wiki's robots.txt instead allows /api.php?action= for *, and that endpoint serves our real User-Agent a 200, so this reads the sanctioned surface with our own headers and no impersonation. The body is therefore JSON, which is what canParse checks first: a challenge page or an error payload must fail loudly, not parse to zero events. Two page facts shape the parser. Titles come from each row's , because a missing banner image renders as a red link reading "File: Banner.png" that a cell-text reader would publish as the event name. And the page is an archive of 154 rows since v1.1 with no ongoing section to gate on, so inclusion is decided against ctx.now — the six-event count is asserted against an independent extraction off the fixture, per docs/INGESTION.md § Testing. Because robots.txt is unreadable from a challenged address, the gate fails closed in CI and the source is skipped there — a warning, not a broken build. Refreshing it means running `bun run refresh` from an address Fandom serves. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 37 +++- docs/INGESTION.md | 19 +- .../fandom-events-2026-08-17.expected.json | 128 ++++++++++++ fixtures/r1999/fandom-events-2026-08-17.html | 1 + src/ingest/adapters/index.ts | 12 ++ src/ingest/parsers/fandom.ts | 183 ++++++++++++++++++ src/ingest/parsers/index.ts | 3 + test/adapters/game8.test.ts | 95 +++++++++ 8 files changed, 469 insertions(+), 9 deletions(-) create mode 100644 fixtures/r1999/fandom-events-2026-08-17.expected.json create mode 100644 fixtures/r1999/fandom-events-2026-08-17.html create mode 100644 src/ingest/parsers/fandom.ts diff --git a/AGENTS.md b/AGENTS.md index 003965a..e452e24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,8 +13,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day progress on events that repeat daily, and lets a user mark events completed. -**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across -nine games, the full interface, offline support, a static server, a Docker image and CI all exist and +**Status: working app, refreshing itself on a schedule.** Schema, four parsers, eleven sources across +ten games, the full interface, offline support, a static server, a Docker image and CI all exist and are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the @@ -83,7 +83,7 @@ re-verify a sample against the live page afterward. src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts custom.ts — reader-authored games and events, and their key spaces src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts - parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game + parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts — keyed by SITE, not game adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam src/client/ React app, service worker, manifest state/ progress, daily log, ignores, prefs, sort — all localStorage @@ -91,7 +91,7 @@ src/client/ React app, service worker, manifest lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 401 tests +test/ 466 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -190,9 +190,9 @@ Sources are community wikis. Treat them as a guest would: - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL. - One request per source per refresh cycle, minimum 6 hours apart. -- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten - sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as - eight back-to-back requests to a single site — which is the shape an edge network throttles, and +- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the + eleven sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive + as eight back-to-back requests to a single site — which is the shape an edge network throttles, and what a burst looks like from the far end regardless of our intent. - Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged". - Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.** @@ -224,7 +224,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and |---|---| | `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source | | `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` | -| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have | +| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below | | `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h | wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference` @@ -233,6 +233,27 @@ trains nothing, and no LLM reads the page content — constraint 2 is what keeps load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive, Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401. +**Fandom: read the API, never the page.** `reverse1999.fandom.com/wiki/Events` answers a non-browser +client with a Cloudflare managed challenge — HTTP 403, `Just a moment…`, "Enable JavaScript" — and so +does `/robots.txt` itself, from a datacenter address. Browser-shaped headers or a JS-executing client +would get past both and **must not be used**: that is defeating a deliberate access control, the same +reason `uma.moe` was declined above. + +What makes this source legitimate anyway is that the wiki publishes a second, sanctioned surface. Its +`robots.txt` — read in a browser, where it serves fine — has no `Disallow: /` for `*` and explicitly +**allows** `/api.php?action=`, and that endpoint answers our real `User-Agent` with a `200` and a JSON +body. So the adapter fetches `api.php?action=parse&page=Events`, with no impersonation anywhere: our +own headers, on a path the site put in writing. The only namespaces `*` is refused are `Special:`, +`User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links +for that reason. + +One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the +robots gate **fails closed there and the source is skipped**. That is a warning line rather than a +broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if +*every* source is blocked — so the scheduled refresh simply never updates this game, and the feed +falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address +Fandom serves, which is how its first snapshot was taken. + `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be read). Anything that would make it fetch more often is a change to this section first. diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 57355f7..193f1b6 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -48,11 +48,28 @@ Consequences worth internalising: | `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X | | `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | | `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights | +| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — `Event \| Time Period \| Version` wikitables | Reverse: 1999 | `wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server region, so its events carry exact precision and real `regionEnds`. Prefer a source like that over a prose wiki when both exist, and give it a higher `priority`. +`fandom` is the only parser whose body is not HTML: it reads an `action=parse` JSON envelope and +takes the rendered wikitext out of `parse.text`. That is not a preference — the rendered page is +behind a Cloudflare challenge and the API is the surface the wiki's `robots.txt` allows, so the +adapter's URL is an API call and the stored snapshot is JSON despite its `.html` name (every body +`snapshots/` keeps is named `.html`, whatever its content type). Its `canParse` therefore checks +the envelope as well as the table, because an error payload or a challenge page must fail loudly +rather than parse to zero events. Two page facts drive the rest of it: + +- **The title is the row's ``, never the cell text.** The cell leads with a banner image, and a + missing image renders as a red link reading `File: Banner.png` — which a cell-text reader + publishes as the event's name. +- **These pages are archives, not schedules.** All five tables list every event since version 1.1 + (154 rows, six of them unfinished when the fixture was captured), with no "ongoing" section to + anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so; + `akwiki` and `game8` can gate on a heading instead, and should where one exists. + `akwiki` shares a host family with `wikigg` and nothing else — arknights.wiki.gg has no `mp-event` cards, so the two are separate modules rather than one parser with a branch. Two things about that page shape are worth knowing before touching it: @@ -247,7 +264,7 @@ Three constraints it holds: repair and drop emits a note whose default sink is `console.warn` — silence is not something a future caller gets for free (§ Silent drops). - **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the - title *and* the incoming ID was minted the standard way. All seven fixtures pass through with + title *and* the incoming ID was minted the standard way. All eleven fixtures pass through with zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys and moving one orphans a reader's marks with no server-side recovery. diff --git a/fixtures/r1999/fandom-events-2026-08-17.expected.json b/fixtures/r1999/fandom-events-2026-08-17.expected.json new file mode 100644 index 0000000..c77e139 --- /dev/null +++ b/fixtures/r1999/fandom-events-2026-08-17.expected.json @@ -0,0 +1,128 @@ +[ + { + "id": "r1999:8-bit-arcade-remix:2026-08-13", + "game": "r1999", + "title": "8-Bit Arcade Remix", + "type": "other", + "summary": "Time-Limited Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:on-anothers-sorrow:2026-08-13", + "game": "r1999", + "title": "On Another's Sorrow", + "type": "other", + "summary": "Version Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-21T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/wiki/On_Another%27s_Sorrow", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:the-you-thats-meant-to-be:2026-08-13", + "game": "r1999", + "title": "The You That's Meant to Be", + "type": "story", + "summary": "Character Story Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:truth-a-posteriori:2026-08-22", + "game": "r1999", + "title": "Truth a Posteriori", + "type": "other", + "summary": "Side Events", + "startsAt": "2026-08-22T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:a-stranger-to-memory-lane:2026-09-03", + "game": "r1999", + "title": "A Stranger to Memory Lane", + "type": "story", + "summary": "Character Story Events", + "startsAt": "2026-09-03T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:what-does-a-brick-want:2026-09-17", + "game": "r1999", + "title": "What Does a Brick Want", + "type": "other", + "summary": "Warm Up Events", + "startsAt": "2026-09-17T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + } +] diff --git a/fixtures/r1999/fandom-events-2026-08-17.html b/fixtures/r1999/fandom-events-2026-08-17.html new file mode 100644 index 0000000..9c0c180 --- /dev/null +++ b/fixtures/r1999/fandom-events-2026-08-17.html @@ -0,0 +1 @@ +{"parse":{"title":"Events","pageid":3479,"text":"

A list of all events available in the Global Version of Reverse: 1999.\n

\n\n\n

Version Events[]

\n

Version Events are the main events of each version, as they come with new stories and gameplay featuring the newly introduced characters.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"The
The Theft of the Rimet Cup\n		November 9th, 05:00 - December 4th, 2023, 04:59 (UTC-5)\n1.1\n
\"A
A Nightmare at Green Lake\n		December 7th, 2023, 05:00 - January 15th, 2024, 04:59 (UTC-5)\n1.2\n
\"Journey
Journey to Mor Pankh\n		January 18th, 05:00 - February 26th, 2024, 04:59 (UTC-5)\n1.3\n
\"The
The Prisoner in the Cave\n		February 29th, 05:00 - April 15th, 2024, 04:59 (UTC-5)\n1.4\n
\"Revival!
Revival! The Uluru Games\n		April 18th, 05:00 - May 27th, 2024, 04:59 (UTC-5)\n1.5\n
\"Notes
Notes on Shuori\n		May 30th, 05:00 - July 8th, 2024, 04:59 (UTC-5)\n1.6\n
\"E
E lucevan le stelle\n		July 11th, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Farewell,
Farewell, Rayashki\n		August 15th, 05:00 - September 16th, 2024, 04:59 (UTC-5)\n1.8\n
\"Vereinsamt
Vereinsamt\n		September 19th, 05:00 - October 28th, 2024, 04:59 (UTC-5)\n1.9\n
\"Floor
Floor It! To the Golden City\n		October 31st, 05:00 - December 2nd, 2024, 04:59 (UTC-5)\n2.0\n
\"Route
Route 77: The Haunted Highway\n		December 5th, 2024, 05:00 - January 6th, 2025, 04:59 (UTC-5)\n2.1\n
\"Tristes
Tristes Tropiques\n		January 9th, 05:00 - February 17th, 2025, 04:59 (UTC-5)\n2.2\n
\"Chronicles
Chronicles of Uluru: London Dawning\n		February 20th, 05:00 - March 24th, 2025, 04:59 (UTC-5)\n2.3\n
\"Last
Last Evenings on Earth\n		March 27th, 05:00 - April 23rd, 2025, 04:59 (UTC-5)\n2.4\n
\"Showdown
Showdown in Chinatown\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"Folie
Folie et Déraison\n		May 29th, 05:00 - June 30th, 2025, 04:59 (UTC-5)\n2.6\n
\"1987
1987 Cosmic Overture\n		July 3rd, 05:00 - August 4th, 2025, 04:59 (UTC-5)\n2.7\n
\"La
La Primavera di Firenze\n		August 7th, 05:00 - September 15th, 2025, 04:59 (UTC-5)\nS01\n
\"Drama
Drama in Athens\n		August 28th, 05:00 - September 19th, 2025, 04:59 (UTC-5)\n
\"Paradise
Paradise Regained\n		September 19th, 05:00 - October 27th, 2025, 04:59 (UTC-5)\n2.8\n
\"A
A Long Long Way\n		October 30th, 05:00 - December 8th, 2025, 04:59 (UTC-5)\n3.0\n
\"The
The Midnight Whistle\n		December 11th, 2025, 05:00 - January 19th, 2026, 04:59 (UTC-5)\n3.1\n
\"A
A Flowing Feast\n		January 20th, 05:00 - March 2nd, 2026, 04:59 (UTC-5)\n3.2\n
\"The
The Campaign's Tale\n		March 5th, 05:00 - April 13th, 2026, 04:59 (UTC-5)\n3.3\n
\"Spring
Spring Unending\n		April 16th, 05:00 - May 25th, 2026, 04:59 (UTC-5)\n3.4\n
\"The
The Turquoise Serpent Club\n		May 28th, 05:00 - June 29th, 2026, 04:59 (UTC-5)\n3.5\n
\"Last
Last and First Principles\n		July 2nd, 05:00 - July 22nd, 2026, 04:59 (UTC-5)\n3.6\n
\"Polymerized
Polymerized Dreams\n		July 23rd, 05:00 - August 12th, 2026, 04:59 (UTC-5)\nS02\n
\"On
On Another's Sorrow\n		August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)\n3.7\n
\n

Character Story Events[]

\n

Character story events are focused on the new 6-star characters each version and acts as their solo story to give the player more information regarding them.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Thief
Thief of the Thieves\n		November 9th, 05:00 - November 23rd, 2023, 04:59 (UTC-5)\n1.1\n
\"The
The Puppy and The Hippie\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n
\"The
The Old Teeth and The Worn Marks\n		December 7th, 05:00 - December 28th, 2023, 04:59 (UTC-5)\n1.2\n
\"One
One Flew Over the Old House\n		December 28th, 2023, 05:00 - January 18th, 2024, 04:59 (UTC-5)\n
\"Between
Between Dusts and Stars\n		January 18th, 05:00 - February 8th, 2024, 04:59 (UTC-5)\n1.3\n
\"Bells
Bells Echoing in the Valley\n		February 8th, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n
\"As
As the Sand Remembered\n		February 29th, 05:00 - March 28th, 2024, 04:59 (UTC-5)\n1.4\n
\"Trudge
Trudge in the Long Night\n		March 28th, 05:00 - April 18th, 2024, 04:59 (UTC-5)\n
\"Reunion
Reunion of Fires\n		April 18th, 05:00 - May 9th, 2024, 04:59 (UTC-5)\n1.5\n
\"As
As the Driven Snow Falls\n		May 9th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n
\"Light
Light to the Tavern\n		May 30th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Flowing
Flowing in the Wind\n		June 20th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n
\"The
The Small Room\n		July 11th, 05:00 - August 1st, 2024, 04:59 (UTC-5)\n1.7\n
\"The
The Tall Lighthouse\n		August 1st, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n
\"Dawn
Dawn Arrives as Usual\n		August 15th, 05:00 - August 29th, 2024, 04:59 (UTC-5)\n1.8\n
\"Silver
Silver Knot\n		August 29th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n
\"Flying
Flying to the Past\n		September 19th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n1.9\n
\"Through
Through the Glass\n		October 10th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n
\"On
On the Thousandth Night\n		October 31st, 05:00 - November 21st, 2024, 04:59 (UTC-5)\n2.0\n
\"The
The San Francisco Kids\n		November 21st, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n
\"Bedtime
Bedtime Story\n		December 5th, 05:00 - December 26th, 2024, 04:59 (UTC-5)\n2.1\n
\"The
The Red Wall\n		December 26th, 2024, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n
\"The
The Overcast October\n		January 9th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Into
Into the Rainforest\n		January 30th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n
\"Shadow
Shadow of the Sun\n		February 20th, 05:00 - March 6th, 2025, 04:59 (UTC-5)\n2.3\n
\"Night
Night Escapades on Cross Street\n		March 6th, 05:00 - March 27th, 2025, 04:59 (UTC-5)\n
\"Beyond
Beyond the Coastline\n		March 27th, 05:00 - April 10th, 2025, 04:59 (UTC-5)\n2.4\n
\"Nukutaeao
Nukutaeao: Building Paradise!\n		April 10th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n
\"Sow
Sow and Reap\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"The
The Great Trickster\n		May 8th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n
\"Ruinas
Ruinas Gloriosas y Directrices de Metáforas\n		May 29th, 05:00 - June 19th, 2025, 04:59 (UTC-5)\n2.6\n
\"The
The Answering Machine, the Butterfly, and the Literary Critic\n		June 19th, 05:00 - July 3rd, 2025, 04:59 (UTC-5)\n
\"Gap
Gap Month\n		July 3rd, 05:00 - July 24th, 2025, 04:59 (UTC-5)\n2.7\n
\"Doomsday
Doomsday & The Days After\n		July 24th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n
\"In
In Her Own Reflection\n		September 19th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"A
A Moment on the White Expanse\n		October 9th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n
\"Dawn
Dawn Rises Over the Kill Zone\n		October 30th, 05:00 - November 20th, 2025, 04:59 (UTC-5)\n3.0\n
\"Fuga
Fuga a 3 Soggetti\n		November 20th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n
\"A
A Shadow's Weight\n		December 11th, 2025, 05:00 - January 1st, 2026, 04:59 (UTC-5)\n3.1\n
\"Cold
Cold Steel\n		January 1st, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n
\"The
The Long Rain of Fate\n		January 20th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\"An
An Unheated Gem\n		February 12th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n
\"In
In the Name of Chivalry\n		March 5th, 05:00 - March 26th, 2026, 04:59 (UTC-5)\n3.3\n
\"Into
Into the Maelstrom\n		March 26th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n
\"Spring
Spring Comes Slowly\n		April 16th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"To
To Slay a Tiger\n		May 7th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n
\"Counterpoint
Counterpoint of Life and Death\n		May 28th, 05:00 - June 18th, 2026, 04:59 (UTC-5)\n3.5\n
\"Whirling
Whirling Blooms\n		June 18th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n
\"Complete
Complete Induction\n		July 2nd, 05:00 - July 23rd, 2026, 04:59 (UTC-5)\n3.6\n
File:The You That's Meant to Be Banner.png
The You That's Meant to Be\n		August 13th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
File:A Stranger to Memory Lane Banner.png
A Stranger to Memory Lane\n		September 3rd, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n
\n

Side Events[]

\n

Side events usually went side by side with the new 6-star characters as they served as a mini mission gameplay. \n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Bio
Bio of the Great Thief\n		November 9th, 05:00 - November 23rd, 2023, 04:59 (UTC-5)\n1.1\n
\"Pickles's
Pickles's Free Translation\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n
\"Camping
Camping Safety Guide\n		December 7th, 05:00 - December 21st, 2023, 04:59 (UTC-5)\n1.2\n
\"A
A Tooth for a Tooth\n		December 28th, 2023, 05:00 - January 11th, 2024, 04:59 (UTC-5)\n
\"Home
Home Practice\n		January 18th, 05:00 - February 1st, 2024, 04:59 (UTC-5)\n1.3\n
\"Shamane's
Shamane's Left Arm\n		February 8th, 05:00 - February 22nd, 2024, 04:59 (UTC-5)\n
\"Set
Set Sail Again\n		February 29th, 05:00 - March 14th, 2024, 04:59 (UTC-5)\n1.4\n
\"Sealed
Sealed Records\n		March 28th, 05:00 - April 11th, 2024, 04:59 (UTC-5)\n
\"The
The Game News\n		April 18th, 05:00 - May 2nd, 2024, 04:59 (UTC-5)\n1.5\n
\"Blazing
Blazing RPS Arena\n		May 9th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n
\"Echoes
Echoes in the Mountain\n		June 8th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Puppet
Puppet Arena\n		February 20th, 05:00 - March 27th, 04:59 (UTC-5)\n2.3\n
\"From
From the Depths of Space\n		March 27th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
\"When
When the Alarm Sounds\n		May 1st, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"When
When the Alarm Sounds\n		July 17th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"Pre-Storm
Pre-Storm Protocol\n		September 19th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"When
When the Alarm Sounds\n		October 9th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n
\"Pre-Storm
Pre-Storm Protocol\n		December 20th, 2025, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"Pre-Storm
Pre-Storm Protocol\n		May 1st, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Pre-Storm
Pre-Storm Protocol\n		May 28th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
File:Truth a Posteriori Banner.png
Truth a Posteriori\n		August 22nd, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n

Time-Limited Events[]

\n

UTTU events are time-limited UTTU gameplays meant to challenge the player and in turn give them great rewards.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Trophy
UTTU Flash Gathering: Trophy of the Past Glory\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n1.1\n
\"A
UTTU Flash Gathering: A Night Visit to the Lake\n		December 28th, 05:00 - January 18th, 2024, 04:59 (UTC-5)\n1.2\n
\"The
UTTU Flash Gathering: The First Epic\n		February 8th, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n1.3\n
\"Wandering
UTTU Flash Gathering: Wandering on the Shore\n		March 28th, 05:00 - April 11th, 2024, 04:59 (UTC-5)\n1.4\n
\"The
UTTU Flash Gathering: The Epic of Runners\n		May 9th, 05:00 - May 23th, 2024, 04:59 (UTC-5)\n1.5\n
\"The
UTTU Flash Gathering: The Story of a Besieged Town\n		June 20th, 05:00 - July 4th, 2024, 04:59 (UTC-5)\n1.6\n
\"Beyond
UTTU Spotlight Edition: Beyond the Ring\n		August 1st, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Polar
UTTU Spotlight Edition: Polar Town\n		August 29th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n1.8\n
\"Fire
UTTU Spotlight Edition: Fire Thieves\n		October 10th, 05:00 - October 24th, 2024, 04:59 (UTC-5)\n1.9\n
\"Midnight
UTTU Spotlight Edition: Midnight Mayhem\n		November 21st, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n2.0\n
\"Stories
UTTU Spotlight Edition: Stories on the Route\n		December 26th, 2024, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n2.1\n
\"Phototaxis-
Mask On\n		January 30th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Tourney
UTTU Spotlight Edition: Tourney in the Fog\n		March 6th, 05:00 - March 20th, 2025, 04:59 (UTC-5)\n2.3\n
\"Phototaxis-
Phototaxis In Study: Besides the Eclipse\n		April 10th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
\"Critter
Critter Crash\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"Phototaxis-
Phototaxis In Study: Beyond the Boundaries\n		June 18th, 05:00 - July 2nd, 2025, 04:59 (UTC-5)\n2.6\n
\"Friendly
Puppet Arena - Friendly Match\n		July 5th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"Critter
Critter Crash\n		September 25th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"Ruins
UTTU Flash Gathering: Ruins and Renewal\n		November 20th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n3.0\n
\"Artefact
Puppet Arena - Artefact Masters\n		January 1st, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"Critter
Critter Crash\n		February 12th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\"8-Bit
8-Bit Arcade Remix\n		March 26th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n3.3\n
\"Musae's
Musae's Arcade\n		April 16th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Critter
Critter Crash\n		May 28th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
\"Odes
Odes to the Spirits\n		June 18th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n
\"Back
Back to Chelomey\n		July 23rd, 05:00 - August 13th, 2026, 04:59 (UTC-5)\nS02\n
File:Beach Battle Banner.png
8-Bit Arcade Remix\n		August 13th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n

Warm Up Events[]

\n

Warm up events are sneak peeks of the upcoming version of the game.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"The
The Satellite\n		November 2nd, 05:00 - November 9th, 2023, 04:59 (UTC-5)\n1.0\n
\"Mysterious
Mysterious Waves\n		February 22nd, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n1.3\n
\"Letter
Letter From the Attic\n		May 25th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n1.5\n
\"Andante
Andante of Music and Sound\n		July 6th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Go
Go North, Friend!\n		August 10th, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Just
Just Reports\n		September 14th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n1.8\n
\"Flea
Flea Market & Gold Rush\n		October 26th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n1.9\n
\"PEEK
Peek through the Peephole\n		November 30th, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n2.0\n
\"Gifts
Gifts From Across the Equator\n		January 4th, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n2.1\n
\"Memo-
Memo: Prepare for Victory!\n		February 15th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Anchors
Anchors Aweigh, \"Free Breeze\"!\n		March 22th, 05:00 - March 27th, 2025, 04:59 (UTC-5)\n2.3\n
\"Caller
Caller ID Unknown\n		April 17th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
N/A
From the End of the Earth\n		May 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
N/A
Lab Snapshots\n		June 28th, 05:00 - July 3rd, 2025, 04:59 (UTC-5)\n2.6\n
\"In
In the Margins of History\n		July 31st, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"A
A Message on the Roaring Wind\n		September 12th, 05:00 - September 19th, 2025, 04:59 (UTC-5)\nS01\n
\"A
A Letter From No Man's Land\n		October 23rd, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"Steam
Steam Whistle Prelude\n		December 4th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n3.0\n
\"Salut,
Salut, Paname\n		January 13th, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"The
The Steppe Before the Flames\n		February 26th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\""Special
\"Special Issue on the East: Huanzhou Spring\"\n		April 9th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n3.3\n
\"Carvings
Carvings on the Sun Stone\n		May 21st, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Still
Still Stands the Silver Tower\n		June 25th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
\"Welcome
Welcome to the Future\n		July 16th, 05:00 - July 23rd, 2026, 04:59 (UTC-5)\n3.6\n
\"Call
Call from the Alma Mater\n		August 6th, 05:00 - August 13th, 2026, 04:59 (UTC-5)\nS02\n
File:What Does a Brick Want Banner.png
What Does a Brick Want\n		September 17th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n\n\n\n\n\n
"}} \ No newline at end of file diff --git a/src/ingest/adapters/index.ts b/src/ingest/adapters/index.ts index 3df8aa0..1cc7836 100644 --- a/src/ingest/adapters/index.ts +++ b/src/ingest/adapters/index.ts @@ -84,6 +84,18 @@ const SOURCES: SourceSpec[] = [ url: "https://game8.co/games/Persona-5-Phantom-X/archives/532244", parserId: "game8", }, + { + id: "r1999-fandom-events", + game: "r1999", + // The MediaWiki API, not `/wiki/Events`. The rendered page answers a + // non-browser client with a Cloudflare interstitial, while this wiki's + // robots.txt allows `/api.php?action=` for `User-agent: *` and the endpoint + // serves our real User-Agent a 200. See `parsers/fandom.ts` for the full + // reasoning; the short version is that this is the surface the site put in + // writing, reached without pretending to be anything we are not. + url: "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + parserId: "fandom", + }, ]; function toAdapter(spec: SourceSpec): Adapter { diff --git a/src/ingest/parsers/fandom.ts b/src/ingest/parsers/fandom.ts new file mode 100644 index 0000000..22b14f8 --- /dev/null +++ b/src/ingest/parsers/fandom.ts @@ -0,0 +1,183 @@ +import { eventId, type GachaEvent } from "../../shared/schema.ts"; +import { parseOrdinalDateTimeRange } from "../dates.ts"; +import { text } from "../html.ts"; +import type { ParseContext } from "../adapters/types.ts"; +import { inferType } from "./game8.ts"; +import type { SourceParser } from "./types.ts"; + +/** + * Fandom wikis, read through the MediaWiki action API. + * + * **Why the API and not the page.** `reverse1999.fandom.com/wiki/Events` + * answers a non-browser client with a Cloudflare interstitial ("Just a + * moment…", HTTP 403), and dressing our fetcher up as a browser to get past it + * would be defeating a deliberate access control — the reason `uma.moe` was + * declined in AGENTS.md § Scraping conduct. The wiki's own `robots.txt` instead + * *allows* `/api.php?action=` for `User-agent: *`, and that endpoint answers our + * real User-Agent with a 200. So this parser reads the sanctioned surface with + * no impersonation anywhere: same headers, a path the site put in writing. + * + * The body is therefore JSON rather than HTML: + * + * {"parse":{"title":"Events","pageid":3479,"text":"
"}} + * + * `parse.text` is the rendered wikitext, and the shape below is what this family + * of pages puts in it — `wikitable`s under one `h2` per section: + * + *

Version Events

+ * + * + * + * + * + * + * Two details decide most of the code. + * + * **The title is the ``, never the cell text.** The cell leads with a banner + * image, and when that image is missing MediaWiki renders a red link whose + * visible text is `File:A Stranger to Memory Lane Banner.png`. A cell-text + * reader publishes that as the event's name. The `` holds the title in both + * cases. + * + * **These pages are archives, not schedules.** All five tables list every event + * since 1.1 — 154 rows, of which six had not yet ended when the fixture was + * captured. Publishing the rest would put three years of finished events on the + * calendar and hand the validator a hundred rows whose start predates its + * two-year sanity window. So inclusion is decided against `ctx.now`, which is + * injected precisely so a parser can be time-aware without reading the clock. + */ + +const SECTION = /]*>([\s\S]*?)<\/h2>|]*>([\s\S]*?)<\/table>/gi; +const ROW = /]*>([\s\S]*?)<\/tr>/gi; +const CELL = /]*>([\s\S]*?)<\/t\1>/gi; +const BOLD = /]*>([\s\S]*?)<\/b>/i; +const EDIT_SECTION = /]*class="[^"]*mw-editsection[^"]*"[^>]*>[\s\S]*?<\/span>/gi; +/** + * A link to the event's own article. `Special:` is excluded deliberately: a + * missing banner image renders as a `Special:Upload` link, which is both the + * wrong page to send a reader to and a path this wiki's robots.txt disallows. + */ +const ARTICLE_LINK = /]*href="(\/wiki\/(?!Special:)[^"#?]+)"/i; + +/** The rendered HTML inside an `action=parse` response, or null. */ +export function renderedHtml(body: string): string | null { + let payload: unknown; + try { + payload = JSON.parse(body); + } catch { + return null; + } + + const html = (payload as { parse?: { text?: unknown } } | null)?.parse?.text; + return typeof html === "string" ? html : null; +} + +export function parseFandomEventsPage( + body: string, + ctx: ParseContext, +): GachaEvent[] { + const rendered = renderedHtml(body); + if (rendered === null) return []; + + const flat = rendered.replace(EDIT_SECTION, "").replace(/\s+/g, " "); + const nowMs = Date.parse(ctx.now); + const out: GachaEvent[] = []; + let section = ""; + + for (const node of flat.matchAll(SECTION)) { + const heading = node[1]; + if (heading !== undefined) { + section = text(heading); + continue; + } + + const table = node[2] ?? ""; + // Every section table states the same three columns. Checking them keeps a + // navbox or an infobox elsewhere in the page from being read as a schedule. + const headers = [...table.matchAll(/]*>([\s\S]*?)<\/th>/gi)].map( + (h) => text(h[1] ?? "").toLowerCase(), + ); + if (!headers.includes("event") || !headers.includes("time period")) continue; + + for (const row of table.matchAll(ROW)) { + const cells = [...(row[1] ?? "").matchAll(CELL)].map((c) => ({ + tag: c[1] ?? "", + html: c[2] ?? "", + })); + if (cells.length < 2 || cells.some((c) => c.tag === "h")) continue; + + const titleCell = cells[0]?.html ?? ""; + const title = text(BOLD.exec(titleCell)?.[1] ?? ""); + if (title.length === 0) continue; + + const range = parseOrdinalDateTimeRange(text(cells[1]?.html ?? "")); + // A row stating no year on either half is unresolvable — the fixture has + // exactly one — and a row this reader cannot date yields no event rather + // than a guessed one. + if (range === null) continue; + + const { start, end } = range; + if (end.iso <= start.iso) continue; + + // Live and upcoming only. An event whose end has passed is history this + // page keeps and the calendar does not want. + if (Date.parse(end.iso) < nowMs) continue; + + const href = ARTICLE_LINK.exec(titleCell)?.[1]; + + out.push({ + id: eventId(ctx.game, title, start.iso), + game: ctx.game, + title, + // The section heading is the source's own classification, and a better + // signal than the title alone — "Character Story Events" names a story + // event whose title says nothing about it. + type: inferType(`${title} ${section}`), + summary: section.length > 0 ? section : null, + startsAt: start.iso, + startPrecision: start.precision, + endsAt: end.iso, + endPrecision: end.precision, + // One global server on a single stated offset: every row on the page + // reads (UTC-5), and the page draws no distinction between regions. + // `regionScoped` means the source separates them, and this one does not. + regionScoped: false, + regionEnds: null, + sourceUrl: + href === undefined + ? ctx.sourceUrl + : new URL(href, ctx.sourceUrl).toString(), + sourceId: ctx.sourceId, + status: "published", + // Both boundaries are exact instants converted from a stated offset, + // with nothing inferred — the same footing as the wiki.gg timers. + confidence: 0.95, + extractionMethod: "parser", + version: 1, + firstSeenAt: ctx.now, + updatedAt: ctx.now, + }); + } + } + + return out.sort((a, b) => + a.startsAt === b.startsAt + ? a.id.localeCompare(b.id) + : a.startsAt.localeCompare(b.startsAt), + ); +} + +export const fandomParser: SourceParser = { + id: "fandom", + label: "Fandom", + canParse(body: string): boolean { + // Structural, and deliberately about the envelope as much as the content: + // this source is an API, so a body that is not an `action=parse` response + // is the failure worth catching loudly — an error payload, a login wall, or + // the Cloudflare interstitial the plain page serves would all land here. + const rendered = renderedHtml(body); + if (rendered === null) return false; + return /class="[^"]*wikitable/.test(rendered) && /Time Period/i.test(rendered); + }, + parse: parseFandomEventsPage, +}; diff --git a/src/ingest/parsers/index.ts b/src/ingest/parsers/index.ts index 6659dcc..96c2d65 100644 --- a/src/ingest/parsers/index.ts +++ b/src/ingest/parsers/index.ts @@ -1,4 +1,5 @@ import { arknightsWikiParser } from "./akwiki.ts"; +import { fandomParser } from "./fandom.ts"; import { game8Parser } from "./game8.ts"; import { wikiGgParser } from "./wikigg.ts"; import type { SourceParser } from "./types.ts"; @@ -12,6 +13,7 @@ export const PARSERS: SourceParser[] = [ game8Parser, wikiGgParser, arknightsWikiParser, + fandomParser, ]; export function parserById(id: string): SourceParser | undefined { @@ -20,5 +22,6 @@ export function parserById(id: string): SourceParser | undefined { export type { SourceParser } from "./types.ts"; export { arknightsWikiParser } from "./akwiki.ts"; +export { fandomParser } from "./fandom.ts"; export { game8Parser } from "./game8.ts"; export { wikiGgParser } from "./wikigg.ts"; diff --git a/test/adapters/game8.test.ts b/test/adapters/game8.test.ts index 27b9b52..9b99ec9 100644 --- a/test/adapters/game8.test.ts +++ b/test/adapters/game8.test.ts @@ -1,7 +1,9 @@ import { describe, expect, test } from "bun:test"; import { adapterById } from "../../src/ingest/adapters/index.ts"; import type { Adapter } from "../../src/ingest/adapters/types.ts"; +import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts"; import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts"; +import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts"; import { inferType } from "../../src/ingest/parsers/game8.ts"; import { GachaEvent, type EventType } from "../../src/shared/schema.ts"; @@ -31,6 +33,10 @@ const CASES: Array<{ adapter: Adapter; fixture: string }> = [ { adapter: adapter("nikki-game8-events"), fixture: "fixtures/nikki/game8-events-2026-08-17" }, { adapter: adapter("p5x-game8-events"), fixture: "fixtures/p5x/game8-events-2026-08-17" }, { adapter: adapter("arknights-akwiki-events"), fixture: "fixtures/arknights/akwiki-events-2026-08-17" }, + // A `.html` fixture holding JSON, deliberately: this source is the MediaWiki + // action API, and `snapshots/` names every stored body `.html` whatever + // its content type. The fixture is the bytes the fetcher would store. + { adapter: adapter("r1999-fandom-events"), fixture: "fixtures/r1999/fandom-events-2026-08-17" }, ]; async function runAdapter(adapter: Adapter, fixture: string) { @@ -421,3 +427,92 @@ describe("wiki.gg parser", () => { } }); }); + +describe("fandom parser", () => { + const fixture = "fixtures/r1999/fandom-events-2026-08-17"; + const r1999 = adapter("r1999-fandom-events"); + + test("publishes only the events that have not ended", async () => { + // The page is an archive: 154 rows spanning every version since 1.1, of + // which six had not ended at the pinned clock. Counted independently off + // the fixture before this expectation was written — the count is the guard + // against a shape change making events vanish silently. + const events = await runAdapter(r1999, fixture); + expect(events).toHaveLength(6); + for (const e of events) { + expect(e.endsAt).not.toBeNull(); + expect(Date.parse(e.endsAt!)).toBeGreaterThanOrEqual(Date.parse(NOW)); + } + }); + + test("converts the stated UTC-5 offset rather than reading it as UTC", async () => { + // "August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)". Reading the + // wall clock as UTC would put every boundary five hours early — and a start + // that crossed a UTC midnight would move the event's ID with it. + const events = await runAdapter(r1999, fixture); + const version = events.find((e) => e.title === "On Another's Sorrow"); + expect(version?.startsAt).toBe("2026-08-13T10:00:00.000Z"); + expect(version?.endsAt).toBe("2026-09-21T09:59:00.000Z"); + expect(version?.startPrecision).toBe("exact"); + expect(version?.endPrecision).toBe("exact"); + }); + + test("takes the title from the , not the cell text", async () => { + // A missing banner image renders as a red link whose visible text is + // "File:A Stranger to Memory Lane Banner.png". A cell-text reader publishes + // that filename as the event's name. + const events = await runAdapter(r1999, fixture); + const titles = events.map((e) => e.title); + expect(titles).toContain("A Stranger to Memory Lane"); + for (const t of titles) { + expect(t).not.toMatch(/^File:/); + expect(t).not.toMatch(/\.png/i); + } + }); + + test("carries the section heading as the summary and types from it", async () => { + const events = await runAdapter(r1999, fixture); + const story = events.find((e) => e.title === "The You That's Meant to Be"); + // The title alone says nothing about what kind of event it is; the section + // it sits under does. + expect(story?.summary).toBe("Character Story Events"); + expect(story?.type).toBe("story"); + // The [edit] link MediaWiki renders inside every heading is not part of it. + for (const e of events) expect(e.summary).not.toMatch(/edit/i); + }); + + test("states one global end, not per-region ends", async () => { + // Every row reads (UTC-5) and the page draws no regional distinction. + for (const e of await runAdapter(r1999, fixture)) { + expect(e.regionScoped).toBe(false); + expect(e.regionEnds).toBeNull(); + } + }); + + test("rejects a body that is not an action=parse response", async () => { + // The plain wiki page answers a non-browser client with a Cloudflare + // interstitial. Feeding that to the parser must fail loudly rather than + // report zero events, which reads downstream as "nothing is on". + const interstitial = "Just a moment..."; + expect(fandomParser.canParse(interstitial)).toBe(false); + expect(fandomParser.canParse('{"error":{"code":"missingtitle"}}')).toBe(false); + expect(() => + r1999.parse(interstitial, { + now: NOW, + sourceUrl: r1999.url, + sourceId: r1999.id, + game: r1999.game, + }), + ).toThrow(/redesigned/); + }); + + test("a row stating no year on either half yields no event", async () => { + // "February 20th, 05:00 - March 27th, 04:59 (UTC-5)" — the fixture has + // exactly one, and there is no honest year to give it. + const body = await Bun.file(`${fixture}.html`).text(); + const rendered = renderedHtml(body); + expect(rendered).not.toBeNull(); + expect(rendered!).toContain("February 20th, 05:00 - March 27th, 04:59"); + expect(parseOrdinalDateTimeRange("February 20th, 05:00 - March 27th, 04:59 (UTC-5)")).toBeNull(); + }); +});
EventTime PeriodVersion

TITLE		August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)3.7