diff --git a/AGENTS.md b/AGENTS.md index 003965a..e452e24 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,8 +13,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day progress on events that repeat daily, and lets a user mark events completed. -**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across -nine games, the full interface, offline support, a static server, a Docker image and CI all exist and +**Status: working app, refreshing itself on a schedule.** Schema, four parsers, eleven sources across +ten games, the full interface, offline support, a static server, a Docker image and CI all exist and are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the @@ -83,7 +83,7 @@ re-verify a sample against the live page afterward. src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts custom.ts — reader-authored games and events, and their key spaces src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts - parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game + parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts — keyed by SITE, not game adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam src/client/ React app, service worker, manifest state/ progress, daily log, ignores, prefs, sort — all localStorage @@ -91,7 +91,7 @@ src/client/ React app, service worker, manifest lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 401 tests +test/ 466 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -190,9 +190,9 @@ Sources are community wikis. Treat them as a guest would: - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL. - One request per source per refresh cycle, minimum 6 hours apart. -- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten - sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as - eight back-to-back requests to a single site — which is the shape an edge network throttles, and +- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the + eleven sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive + as eight back-to-back requests to a single site — which is the shape an edge network throttles, and what a burst looks like from the far end regardless of our intent. - Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged". - Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.** @@ -224,7 +224,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and |---|---| | `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source | | `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` | -| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have | +| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below | | `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h | wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference` @@ -233,6 +233,27 @@ trains nothing, and no LLM reads the page content — constraint 2 is what keeps load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive, Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401. +**Fandom: read the API, never the page.** `reverse1999.fandom.com/wiki/Events` answers a non-browser +client with a Cloudflare managed challenge — HTTP 403, `Just a moment…`, "Enable JavaScript" — and so +does `/robots.txt` itself, from a datacenter address. Browser-shaped headers or a JS-executing client +would get past both and **must not be used**: that is defeating a deliberate access control, the same +reason `uma.moe` was declined above. + +What makes this source legitimate anyway is that the wiki publishes a second, sanctioned surface. Its +`robots.txt` — read in a browser, where it serves fine — has no `Disallow: /` for `*` and explicitly +**allows** `/api.php?action=`, and that endpoint answers our real `User-Agent` with a `200` and a JSON +body. So the adapter fetches `api.php?action=parse&page=Events`, with no impersonation anywhere: our +own headers, on a path the site put in writing. The only namespaces `*` is refused are `Special:`, +`User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links +for that reason. + +One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the +robots gate **fails closed there and the source is skipped**. That is a warning line rather than a +broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if +*every* source is blocked — so the scheduled refresh simply never updates this game, and the feed +falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address +Fandom serves, which is how its first snapshot was taken. + `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be read). Anything that would make it fetch more often is a change to this section first. diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 57355f7..193f1b6 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -48,11 +48,28 @@ Consequences worth internalising: | `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X | | `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | | `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights | +| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — `Event \| Time Period \| Version` wikitables | Reverse: 1999 | `wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server region, so its events carry exact precision and real `regionEnds`. Prefer a source like that over a prose wiki when both exist, and give it a higher `priority`. +`fandom` is the only parser whose body is not HTML: it reads an `action=parse` JSON envelope and +takes the rendered wikitext out of `parse.text`. That is not a preference — the rendered page is +behind a Cloudflare challenge and the API is the surface the wiki's `robots.txt` allows, so the +adapter's URL is an API call and the stored snapshot is JSON despite its `.html` name (every body +`snapshots/` keeps is named `.html`, whatever its content type). Its `canParse` therefore checks +the envelope as well as the table, because an error payload or a challenge page must fail loudly +rather than parse to zero events. Two page facts drive the rest of it: + +- **The title is the row's ``, never the cell text.** The cell leads with a banner image, and a + missing image renders as a red link reading `File: Banner.png` — which a cell-text reader + publishes as the event's name. +- **These pages are archives, not schedules.** All five tables list every event since version 1.1 + (154 rows, six of them unfinished when the fixture was captured), with no "ongoing" section to + anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so; + `akwiki` and `game8` can gate on a heading instead, and should where one exists. + `akwiki` shares a host family with `wikigg` and nothing else — arknights.wiki.gg has no `mp-event` cards, so the two are separate modules rather than one parser with a branch. Two things about that page shape are worth knowing before touching it: @@ -247,7 +264,7 @@ Three constraints it holds: repair and drop emits a note whose default sink is `console.warn` — silence is not something a future caller gets for free (§ Silent drops). - **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the - title *and* the incoming ID was minted the standard way. All seven fixtures pass through with + title *and* the incoming ID was minted the standard way. All eleven fixtures pass through with zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys and moving one orphans a reader's marks with no server-side recovery. diff --git a/fixtures/r1999/fandom-events-2026-08-17.expected.json b/fixtures/r1999/fandom-events-2026-08-17.expected.json new file mode 100644 index 0000000..c77e139 --- /dev/null +++ b/fixtures/r1999/fandom-events-2026-08-17.expected.json @@ -0,0 +1,128 @@ +[ + { + "id": "r1999:8-bit-arcade-remix:2026-08-13", + "game": "r1999", + "title": "8-Bit Arcade Remix", + "type": "other", + "summary": "Time-Limited Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:on-anothers-sorrow:2026-08-13", + "game": "r1999", + "title": "On Another's Sorrow", + "type": "other", + "summary": "Version Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-21T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/wiki/On_Another%27s_Sorrow", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:the-you-thats-meant-to-be:2026-08-13", + "game": "r1999", + "title": "The You That's Meant to Be", + "type": "story", + "summary": "Character Story Events", + "startsAt": "2026-08-13T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:truth-a-posteriori:2026-08-22", + "game": "r1999", + "title": "Truth a Posteriori", + "type": "other", + "summary": "Side Events", + "startsAt": "2026-08-22T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:a-stranger-to-memory-lane:2026-09-03", + "game": "r1999", + "title": "A Stranger to Memory Lane", + "type": "story", + "summary": "Character Story Events", + "startsAt": "2026-09-03T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + }, + { + "id": "r1999:what-does-a-brick-want:2026-09-17", + "game": "r1999", + "title": "What Does a Brick Want", + "type": "other", + "summary": "Warm Up Events", + "startsAt": "2026-09-17T10:00:00.000Z", + "startPrecision": "exact", + "endsAt": "2026-09-24T09:59:00.000Z", + "endPrecision": "exact", + "regionScoped": false, + "regionEnds": null, + "sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + "sourceId": "r1999-fandom-events", + "status": "published", + "confidence": 0.95, + "extractionMethod": "parser", + "version": 1, + "firstSeenAt": "2026-08-14T00:00:00.000Z", + "updatedAt": "2026-08-14T00:00:00.000Z" + } +] diff --git a/fixtures/r1999/fandom-events-2026-08-17.html b/fixtures/r1999/fandom-events-2026-08-17.html new file mode 100644 index 0000000..9c0c180 --- /dev/null +++ b/fixtures/r1999/fandom-events-2026-08-17.html @@ -0,0 +1 @@ +{"parse":{"title":"Events","pageid":3479,"text":"

A list of all events available in the Global Version of Reverse: 1999.\n

\n\n\n

Version Events[]

\n

Version Events are the main events of each version, as they come with new stories and gameplay featuring the newly introduced characters.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"The
The Theft of the Rimet Cup\n		November 9th, 05:00 - December 4th, 2023, 04:59 (UTC-5)\n1.1\n
\"A
A Nightmare at Green Lake\n		December 7th, 2023, 05:00 - January 15th, 2024, 04:59 (UTC-5)\n1.2\n
\"Journey
Journey to Mor Pankh\n		January 18th, 05:00 - February 26th, 2024, 04:59 (UTC-5)\n1.3\n
\"The
The Prisoner in the Cave\n		February 29th, 05:00 - April 15th, 2024, 04:59 (UTC-5)\n1.4\n
\"Revival!
Revival! The Uluru Games\n		April 18th, 05:00 - May 27th, 2024, 04:59 (UTC-5)\n1.5\n
\"Notes
Notes on Shuori\n		May 30th, 05:00 - July 8th, 2024, 04:59 (UTC-5)\n1.6\n
\"E
E lucevan le stelle\n		July 11th, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Farewell,
Farewell, Rayashki\n		August 15th, 05:00 - September 16th, 2024, 04:59 (UTC-5)\n1.8\n
\"Vereinsamt
Vereinsamt\n		September 19th, 05:00 - October 28th, 2024, 04:59 (UTC-5)\n1.9\n
\"Floor
Floor It! To the Golden City\n		October 31st, 05:00 - December 2nd, 2024, 04:59 (UTC-5)\n2.0\n
\"Route
Route 77: The Haunted Highway\n		December 5th, 2024, 05:00 - January 6th, 2025, 04:59 (UTC-5)\n2.1\n
\"Tristes
Tristes Tropiques\n		January 9th, 05:00 - February 17th, 2025, 04:59 (UTC-5)\n2.2\n
\"Chronicles
Chronicles of Uluru: London Dawning\n		February 20th, 05:00 - March 24th, 2025, 04:59 (UTC-5)\n2.3\n
\"Last
Last Evenings on Earth\n		March 27th, 05:00 - April 23rd, 2025, 04:59 (UTC-5)\n2.4\n
\"Showdown
Showdown in Chinatown\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"Folie
Folie et Déraison\n		May 29th, 05:00 - June 30th, 2025, 04:59 (UTC-5)\n2.6\n
\"1987
1987 Cosmic Overture\n		July 3rd, 05:00 - August 4th, 2025, 04:59 (UTC-5)\n2.7\n
\"La
La Primavera di Firenze\n		August 7th, 05:00 - September 15th, 2025, 04:59 (UTC-5)\nS01\n
\"Drama
Drama in Athens\n		August 28th, 05:00 - September 19th, 2025, 04:59 (UTC-5)\n
\"Paradise
Paradise Regained\n		September 19th, 05:00 - October 27th, 2025, 04:59 (UTC-5)\n2.8\n
\"A
A Long Long Way\n		October 30th, 05:00 - December 8th, 2025, 04:59 (UTC-5)\n3.0\n
\"The
The Midnight Whistle\n		December 11th, 2025, 05:00 - January 19th, 2026, 04:59 (UTC-5)\n3.1\n
\"A
A Flowing Feast\n		January 20th, 05:00 - March 2nd, 2026, 04:59 (UTC-5)\n3.2\n
\"The
The Campaign's Tale\n		March 5th, 05:00 - April 13th, 2026, 04:59 (UTC-5)\n3.3\n
\"Spring
Spring Unending\n		April 16th, 05:00 - May 25th, 2026, 04:59 (UTC-5)\n3.4\n
\"The
The Turquoise Serpent Club\n		May 28th, 05:00 - June 29th, 2026, 04:59 (UTC-5)\n3.5\n
\"Last
Last and First Principles\n		July 2nd, 05:00 - July 22nd, 2026, 04:59 (UTC-5)\n3.6\n
\"Polymerized
Polymerized Dreams\n		July 23rd, 05:00 - August 12th, 2026, 04:59 (UTC-5)\nS02\n
\"On
On Another's Sorrow\n		August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)\n3.7\n
\n

Character Story Events[]

\n

Character story events are focused on the new 6-star characters each version and acts as their solo story to give the player more information regarding them.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Thief
Thief of the Thieves\n		November 9th, 05:00 - November 23rd, 2023, 04:59 (UTC-5)\n1.1\n
\"The
The Puppy and The Hippie\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n
\"The
The Old Teeth and The Worn Marks\n		December 7th, 05:00 - December 28th, 2023, 04:59 (UTC-5)\n1.2\n
\"One
One Flew Over the Old House\n		December 28th, 2023, 05:00 - January 18th, 2024, 04:59 (UTC-5)\n
\"Between
Between Dusts and Stars\n		January 18th, 05:00 - February 8th, 2024, 04:59 (UTC-5)\n1.3\n
\"Bells
Bells Echoing in the Valley\n		February 8th, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n
\"As
As the Sand Remembered\n		February 29th, 05:00 - March 28th, 2024, 04:59 (UTC-5)\n1.4\n
\"Trudge
Trudge in the Long Night\n		March 28th, 05:00 - April 18th, 2024, 04:59 (UTC-5)\n
\"Reunion
Reunion of Fires\n		April 18th, 05:00 - May 9th, 2024, 04:59 (UTC-5)\n1.5\n
\"As
As the Driven Snow Falls\n		May 9th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n
\"Light
Light to the Tavern\n		May 30th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Flowing
Flowing in the Wind\n		June 20th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n
\"The
The Small Room\n		July 11th, 05:00 - August 1st, 2024, 04:59 (UTC-5)\n1.7\n
\"The
The Tall Lighthouse\n		August 1st, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n
\"Dawn
Dawn Arrives as Usual\n		August 15th, 05:00 - August 29th, 2024, 04:59 (UTC-5)\n1.8\n
\"Silver
Silver Knot\n		August 29th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n
\"Flying
Flying to the Past\n		September 19th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n1.9\n
\"Through
Through the Glass\n		October 10th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n
\"On
On the Thousandth Night\n		October 31st, 05:00 - November 21st, 2024, 04:59 (UTC-5)\n2.0\n
\"The
The San Francisco Kids\n		November 21st, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n
\"Bedtime
Bedtime Story\n		December 5th, 05:00 - December 26th, 2024, 04:59 (UTC-5)\n2.1\n
\"The
The Red Wall\n		December 26th, 2024, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n
\"The
The Overcast October\n		January 9th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Into
Into the Rainforest\n		January 30th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n
\"Shadow
Shadow of the Sun\n		February 20th, 05:00 - March 6th, 2025, 04:59 (UTC-5)\n2.3\n
\"Night
Night Escapades on Cross Street\n		March 6th, 05:00 - March 27th, 2025, 04:59 (UTC-5)\n
\"Beyond
Beyond the Coastline\n		March 27th, 05:00 - April 10th, 2025, 04:59 (UTC-5)\n2.4\n
\"Nukutaeao
Nukutaeao: Building Paradise!\n		April 10th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n
\"Sow
Sow and Reap\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"The
The Great Trickster\n		May 8th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n
\"Ruinas
Ruinas Gloriosas y Directrices de Metáforas\n		May 29th, 05:00 - June 19th, 2025, 04:59 (UTC-5)\n2.6\n
\"The
The Answering Machine, the Butterfly, and the Literary Critic\n		June 19th, 05:00 - July 3rd, 2025, 04:59 (UTC-5)\n
\"Gap
Gap Month\n		July 3rd, 05:00 - July 24th, 2025, 04:59 (UTC-5)\n2.7\n
\"Doomsday
Doomsday & The Days After\n		July 24th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n
\"In
In Her Own Reflection\n		September 19th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"A
A Moment on the White Expanse\n		October 9th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n
\"Dawn
Dawn Rises Over the Kill Zone\n		October 30th, 05:00 - November 20th, 2025, 04:59 (UTC-5)\n3.0\n
\"Fuga
Fuga a 3 Soggetti\n		November 20th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n
\"A
A Shadow's Weight\n		December 11th, 2025, 05:00 - January 1st, 2026, 04:59 (UTC-5)\n3.1\n
\"Cold
Cold Steel\n		January 1st, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n
\"The
The Long Rain of Fate\n		January 20th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\"An
An Unheated Gem\n		February 12th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n
\"In
In the Name of Chivalry\n		March 5th, 05:00 - March 26th, 2026, 04:59 (UTC-5)\n3.3\n
\"Into
Into the Maelstrom\n		March 26th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n
\"Spring
Spring Comes Slowly\n		April 16th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"To
To Slay a Tiger\n		May 7th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n
\"Counterpoint
Counterpoint of Life and Death\n		May 28th, 05:00 - June 18th, 2026, 04:59 (UTC-5)\n3.5\n
\"Whirling
Whirling Blooms\n		June 18th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n
\"Complete
Complete Induction\n		July 2nd, 05:00 - July 23rd, 2026, 04:59 (UTC-5)\n3.6\n
File:The You That's Meant to Be Banner.png
The You That's Meant to Be\n		August 13th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
File:A Stranger to Memory Lane Banner.png
A Stranger to Memory Lane\n		September 3rd, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n
\n

Side Events[]

\n

Side events usually went side by side with the new 6-star characters as they served as a mini mission gameplay. \n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Bio
Bio of the Great Thief\n		November 9th, 05:00 - November 23rd, 2023, 04:59 (UTC-5)\n1.1\n
\"Pickles's
Pickles's Free Translation\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n
\"Camping
Camping Safety Guide\n		December 7th, 05:00 - December 21st, 2023, 04:59 (UTC-5)\n1.2\n
\"A
A Tooth for a Tooth\n		December 28th, 2023, 05:00 - January 11th, 2024, 04:59 (UTC-5)\n
\"Home
Home Practice\n		January 18th, 05:00 - February 1st, 2024, 04:59 (UTC-5)\n1.3\n
\"Shamane's
Shamane's Left Arm\n		February 8th, 05:00 - February 22nd, 2024, 04:59 (UTC-5)\n
\"Set
Set Sail Again\n		February 29th, 05:00 - March 14th, 2024, 04:59 (UTC-5)\n1.4\n
\"Sealed
Sealed Records\n		March 28th, 05:00 - April 11th, 2024, 04:59 (UTC-5)\n
\"The
The Game News\n		April 18th, 05:00 - May 2nd, 2024, 04:59 (UTC-5)\n1.5\n
\"Blazing
Blazing RPS Arena\n		May 9th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n
\"Echoes
Echoes in the Mountain\n		June 8th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Puppet
Puppet Arena\n		February 20th, 05:00 - March 27th, 04:59 (UTC-5)\n2.3\n
\"From
From the Depths of Space\n		March 27th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
\"When
When the Alarm Sounds\n		May 1st, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"When
When the Alarm Sounds\n		July 17th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"Pre-Storm
Pre-Storm Protocol\n		September 19th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"When
When the Alarm Sounds\n		October 9th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n
\"Pre-Storm
Pre-Storm Protocol\n		December 20th, 2025, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"Pre-Storm
Pre-Storm Protocol\n		May 1st, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Pre-Storm
Pre-Storm Protocol\n		May 28th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
File:Truth a Posteriori Banner.png
Truth a Posteriori\n		August 22nd, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n

Time-Limited Events[]

\n

UTTU events are time-limited UTTU gameplays meant to challenge the player and in turn give them great rewards.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"Trophy
UTTU Flash Gathering: Trophy of the Past Glory\n		November 23rd, 05:00 - December 7th, 2023, 04:59 (UTC-5)\n1.1\n
\"A
UTTU Flash Gathering: A Night Visit to the Lake\n		December 28th, 05:00 - January 18th, 2024, 04:59 (UTC-5)\n1.2\n
\"The
UTTU Flash Gathering: The First Epic\n		February 8th, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n1.3\n
\"Wandering
UTTU Flash Gathering: Wandering on the Shore\n		March 28th, 05:00 - April 11th, 2024, 04:59 (UTC-5)\n1.4\n
\"The
UTTU Flash Gathering: The Epic of Runners\n		May 9th, 05:00 - May 23th, 2024, 04:59 (UTC-5)\n1.5\n
\"The
UTTU Flash Gathering: The Story of a Besieged Town\n		June 20th, 05:00 - July 4th, 2024, 04:59 (UTC-5)\n1.6\n
\"Beyond
UTTU Spotlight Edition: Beyond the Ring\n		August 1st, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Polar
UTTU Spotlight Edition: Polar Town\n		August 29th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n1.8\n
\"Fire
UTTU Spotlight Edition: Fire Thieves\n		October 10th, 05:00 - October 24th, 2024, 04:59 (UTC-5)\n1.9\n
\"Midnight
UTTU Spotlight Edition: Midnight Mayhem\n		November 21st, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n2.0\n
\"Stories
UTTU Spotlight Edition: Stories on the Route\n		December 26th, 2024, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n2.1\n
\"Phototaxis-
Mask On\n		January 30th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Tourney
UTTU Spotlight Edition: Tourney in the Fog\n		March 6th, 05:00 - March 20th, 2025, 04:59 (UTC-5)\n2.3\n
\"Phototaxis-
Phototaxis In Study: Besides the Eclipse\n		April 10th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
\"Critter
Critter Crash\n		April 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
\"Phototaxis-
Phototaxis In Study: Beyond the Boundaries\n		June 18th, 05:00 - July 2nd, 2025, 04:59 (UTC-5)\n2.6\n
\"Friendly
Puppet Arena - Friendly Match\n		July 5th, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"Critter
Critter Crash\n		September 25th, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"Ruins
UTTU Flash Gathering: Ruins and Renewal\n		November 20th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n3.0\n
\"Artefact
Puppet Arena - Artefact Masters\n		January 1st, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"Critter
Critter Crash\n		February 12th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\"8-Bit
8-Bit Arcade Remix\n		March 26th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n3.3\n
\"Musae's
Musae's Arcade\n		April 16th, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Critter
Critter Crash\n		May 28th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
\"Odes
Odes to the Spirits\n		June 18th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n
\"Back
Back to Chelomey\n		July 23rd, 05:00 - August 13th, 2026, 04:59 (UTC-5)\nS02\n
File:Beach Battle Banner.png
8-Bit Arcade Remix\n		August 13th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n

Warm Up Events[]

\n

Warm up events are sneak peeks of the upcoming version of the game.\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Event\nTime Period\nVersion\n
\"The
The Satellite\n		November 2nd, 05:00 - November 9th, 2023, 04:59 (UTC-5)\n1.0\n
\"Mysterious
Mysterious Waves\n		February 22nd, 05:00 - February 29th, 2024, 04:59 (UTC-5)\n1.3\n
\"Letter
Letter From the Attic\n		May 25th, 05:00 - May 30th, 2024, 04:59 (UTC-5)\n1.5\n
\"Andante
Andante of Music and Sound\n		July 6th, 05:00 - July 11th, 2024, 04:59 (UTC-5)\n1.6\n
\"Go
Go North, Friend!\n		August 10th, 05:00 - August 15th, 2024, 04:59 (UTC-5)\n1.7\n
\"Just
Just Reports\n		September 14th, 05:00 - September 19th, 2024, 04:59 (UTC-5)\n1.8\n
\"Flea
Flea Market & Gold Rush\n		October 26th, 05:00 - October 31st, 2024, 04:59 (UTC-5)\n1.9\n
\"PEEK
Peek through the Peephole\n		November 30th, 05:00 - December 5th, 2024, 04:59 (UTC-5)\n2.0\n
\"Gifts
Gifts From Across the Equator\n		January 4th, 05:00 - January 9th, 2025, 04:59 (UTC-5)\n2.1\n
\"Memo-
Memo: Prepare for Victory!\n		February 15th, 05:00 - February 20th, 2025, 04:59 (UTC-5)\n2.2\n
\"Anchors
Anchors Aweigh, \"Free Breeze\"!\n		March 22th, 05:00 - March 27th, 2025, 04:59 (UTC-5)\n2.3\n
\"Caller
Caller ID Unknown\n		April 17th, 05:00 - April 24th, 2025, 04:59 (UTC-5)\n2.4\n
N/A
From the End of the Earth\n		May 24th, 05:00 - May 29th, 2025, 04:59 (UTC-5)\n2.5\n
N/A
Lab Snapshots\n		June 28th, 05:00 - July 3rd, 2025, 04:59 (UTC-5)\n2.6\n
\"In
In the Margins of History\n		July 31st, 05:00 - August 7th, 2025, 04:59 (UTC-5)\n2.7\n
\"A
A Message on the Roaring Wind\n		September 12th, 05:00 - September 19th, 2025, 04:59 (UTC-5)\nS01\n
\"A
A Letter From No Man's Land\n		October 23rd, 05:00 - October 30th, 2025, 04:59 (UTC-5)\n2.8\n
\"Steam
Steam Whistle Prelude\n		December 4th, 05:00 - December 11th, 2025, 04:59 (UTC-5)\n3.0\n
\"Salut,
Salut, Paname\n		January 13th, 05:00 - January 20th, 2026, 04:59 (UTC-5)\n3.1\n
\"The
The Steppe Before the Flames\n		February 26th, 05:00 - March 5th, 2026, 04:59 (UTC-5)\n3.2\n
\""Special
\"Special Issue on the East: Huanzhou Spring\"\n		April 9th, 05:00 - April 16th, 2026, 04:59 (UTC-5)\n3.3\n
\"Carvings
Carvings on the Sun Stone\n		May 21st, 05:00 - May 28th, 2026, 04:59 (UTC-5)\n3.4\n
\"Still
Still Stands the Silver Tower\n		June 25th, 05:00 - July 2nd, 2026, 04:59 (UTC-5)\n3.5\n
\"Welcome
Welcome to the Future\n		July 16th, 05:00 - July 23rd, 2026, 04:59 (UTC-5)\n3.6\n
\"Call
Call from the Alma Mater\n		August 6th, 05:00 - August 13th, 2026, 04:59 (UTC-5)\nS02\n
File:What Does a Brick Want Banner.png
What Does a Brick Want\n		September 17th, 05:00 - September 24th, 2026, 04:59 (UTC-5)\n3.7\n
\n\n\n\n\n\n
"}} \ No newline at end of file diff --git a/src/ingest/adapters/index.ts b/src/ingest/adapters/index.ts index 3df8aa0..1cc7836 100644 --- a/src/ingest/adapters/index.ts +++ b/src/ingest/adapters/index.ts @@ -84,6 +84,18 @@ const SOURCES: SourceSpec[] = [ url: "https://game8.co/games/Persona-5-Phantom-X/archives/532244", parserId: "game8", }, + { + id: "r1999-fandom-events", + game: "r1999", + // The MediaWiki API, not `/wiki/Events`. The rendered page answers a + // non-browser client with a Cloudflare interstitial, while this wiki's + // robots.txt allows `/api.php?action=` for `User-agent: *` and the endpoint + // serves our real User-Agent a 200. See `parsers/fandom.ts` for the full + // reasoning; the short version is that this is the surface the site put in + // writing, reached without pretending to be anything we are not. + url: "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json", + parserId: "fandom", + }, ]; function toAdapter(spec: SourceSpec): Adapter { diff --git a/src/ingest/parsers/fandom.ts b/src/ingest/parsers/fandom.ts new file mode 100644 index 0000000..22b14f8 --- /dev/null +++ b/src/ingest/parsers/fandom.ts @@ -0,0 +1,183 @@ +import { eventId, type GachaEvent } from "../../shared/schema.ts"; +import { parseOrdinalDateTimeRange } from "../dates.ts"; +import { text } from "../html.ts"; +import type { ParseContext } from "../adapters/types.ts"; +import { inferType } from "./game8.ts"; +import type { SourceParser } from "./types.ts"; + +/** + * Fandom wikis, read through the MediaWiki action API. + * + * **Why the API and not the page.** `reverse1999.fandom.com/wiki/Events` + * answers a non-browser client with a Cloudflare interstitial ("Just a + * moment…", HTTP 403), and dressing our fetcher up as a browser to get past it + * would be defeating a deliberate access control — the reason `uma.moe` was + * declined in AGENTS.md § Scraping conduct. The wiki's own `robots.txt` instead + * *allows* `/api.php?action=` for `User-agent: *`, and that endpoint answers our + * real User-Agent with a 200. So this parser reads the sanctioned surface with + * no impersonation anywhere: same headers, a path the site put in writing. + * + * The body is therefore JSON rather than HTML: + * + * {"parse":{"title":"Events","pageid":3479,"text":"
"}} + * + * `parse.text` is the rendered wikitext, and the shape below is what this family + * of pages puts in it — `wikitable`s under one `h2` per section: + * + *

Version Events

+ * + * + * + * + * + * + * Two details decide most of the code. + * + * **The title is the ``, never the cell text.** The cell leads with a banner + * image, and when that image is missing MediaWiki renders a red link whose + * visible text is `File:A Stranger to Memory Lane Banner.png`. A cell-text + * reader publishes that as the event's name. The `` holds the title in both + * cases. + * + * **These pages are archives, not schedules.** All five tables list every event + * since 1.1 — 154 rows, of which six had not yet ended when the fixture was + * captured. Publishing the rest would put three years of finished events on the + * calendar and hand the validator a hundred rows whose start predates its + * two-year sanity window. So inclusion is decided against `ctx.now`, which is + * injected precisely so a parser can be time-aware without reading the clock. + */ + +const SECTION = /]*>([\s\S]*?)<\/h2>|]*>([\s\S]*?)<\/table>/gi; +const ROW = /]*>([\s\S]*?)<\/tr>/gi; +const CELL = /]*>([\s\S]*?)<\/t\1>/gi; +const BOLD = /]*>([\s\S]*?)<\/b>/i; +const EDIT_SECTION = /]*class="[^"]*mw-editsection[^"]*"[^>]*>[\s\S]*?<\/span>/gi; +/** + * A link to the event's own article. `Special:` is excluded deliberately: a + * missing banner image renders as a `Special:Upload` link, which is both the + * wrong page to send a reader to and a path this wiki's robots.txt disallows. + */ +const ARTICLE_LINK = /]*href="(\/wiki\/(?!Special:)[^"#?]+)"/i; + +/** The rendered HTML inside an `action=parse` response, or null. */ +export function renderedHtml(body: string): string | null { + let payload: unknown; + try { + payload = JSON.parse(body); + } catch { + return null; + } + + const html = (payload as { parse?: { text?: unknown } } | null)?.parse?.text; + return typeof html === "string" ? html : null; +} + +export function parseFandomEventsPage( + body: string, + ctx: ParseContext, +): GachaEvent[] { + const rendered = renderedHtml(body); + if (rendered === null) return []; + + const flat = rendered.replace(EDIT_SECTION, "").replace(/\s+/g, " "); + const nowMs = Date.parse(ctx.now); + const out: GachaEvent[] = []; + let section = ""; + + for (const node of flat.matchAll(SECTION)) { + const heading = node[1]; + if (heading !== undefined) { + section = text(heading); + continue; + } + + const table = node[2] ?? ""; + // Every section table states the same three columns. Checking them keeps a + // navbox or an infobox elsewhere in the page from being read as a schedule. + const headers = [...table.matchAll(/]*>([\s\S]*?)<\/th>/gi)].map( + (h) => text(h[1] ?? "").toLowerCase(), + ); + if (!headers.includes("event") || !headers.includes("time period")) continue; + + for (const row of table.matchAll(ROW)) { + const cells = [...(row[1] ?? "").matchAll(CELL)].map((c) => ({ + tag: c[1] ?? "", + html: c[2] ?? "", + })); + if (cells.length < 2 || cells.some((c) => c.tag === "h")) continue; + + const titleCell = cells[0]?.html ?? ""; + const title = text(BOLD.exec(titleCell)?.[1] ?? ""); + if (title.length === 0) continue; + + const range = parseOrdinalDateTimeRange(text(cells[1]?.html ?? "")); + // A row stating no year on either half is unresolvable — the fixture has + // exactly one — and a row this reader cannot date yields no event rather + // than a guessed one. + if (range === null) continue; + + const { start, end } = range; + if (end.iso <= start.iso) continue; + + // Live and upcoming only. An event whose end has passed is history this + // page keeps and the calendar does not want. + if (Date.parse(end.iso) < nowMs) continue; + + const href = ARTICLE_LINK.exec(titleCell)?.[1]; + + out.push({ + id: eventId(ctx.game, title, start.iso), + game: ctx.game, + title, + // The section heading is the source's own classification, and a better + // signal than the title alone — "Character Story Events" names a story + // event whose title says nothing about it. + type: inferType(`${title} ${section}`), + summary: section.length > 0 ? section : null, + startsAt: start.iso, + startPrecision: start.precision, + endsAt: end.iso, + endPrecision: end.precision, + // One global server on a single stated offset: every row on the page + // reads (UTC-5), and the page draws no distinction between regions. + // `regionScoped` means the source separates them, and this one does not. + regionScoped: false, + regionEnds: null, + sourceUrl: + href === undefined + ? ctx.sourceUrl + : new URL(href, ctx.sourceUrl).toString(), + sourceId: ctx.sourceId, + status: "published", + // Both boundaries are exact instants converted from a stated offset, + // with nothing inferred — the same footing as the wiki.gg timers. + confidence: 0.95, + extractionMethod: "parser", + version: 1, + firstSeenAt: ctx.now, + updatedAt: ctx.now, + }); + } + } + + return out.sort((a, b) => + a.startsAt === b.startsAt + ? a.id.localeCompare(b.id) + : a.startsAt.localeCompare(b.startsAt), + ); +} + +export const fandomParser: SourceParser = { + id: "fandom", + label: "Fandom", + canParse(body: string): boolean { + // Structural, and deliberately about the envelope as much as the content: + // this source is an API, so a body that is not an `action=parse` response + // is the failure worth catching loudly — an error payload, a login wall, or + // the Cloudflare interstitial the plain page serves would all land here. + const rendered = renderedHtml(body); + if (rendered === null) return false; + return /class="[^"]*wikitable/.test(rendered) && /Time Period/i.test(rendered); + }, + parse: parseFandomEventsPage, +}; diff --git a/src/ingest/parsers/index.ts b/src/ingest/parsers/index.ts index 6659dcc..96c2d65 100644 --- a/src/ingest/parsers/index.ts +++ b/src/ingest/parsers/index.ts @@ -1,4 +1,5 @@ import { arknightsWikiParser } from "./akwiki.ts"; +import { fandomParser } from "./fandom.ts"; import { game8Parser } from "./game8.ts"; import { wikiGgParser } from "./wikigg.ts"; import type { SourceParser } from "./types.ts"; @@ -12,6 +13,7 @@ export const PARSERS: SourceParser[] = [ game8Parser, wikiGgParser, arknightsWikiParser, + fandomParser, ]; export function parserById(id: string): SourceParser | undefined { @@ -20,5 +22,6 @@ export function parserById(id: string): SourceParser | undefined { export type { SourceParser } from "./types.ts"; export { arknightsWikiParser } from "./akwiki.ts"; +export { fandomParser } from "./fandom.ts"; export { game8Parser } from "./game8.ts"; export { wikiGgParser } from "./wikigg.ts"; diff --git a/test/adapters/game8.test.ts b/test/adapters/game8.test.ts index 27b9b52..9b99ec9 100644 --- a/test/adapters/game8.test.ts +++ b/test/adapters/game8.test.ts @@ -1,7 +1,9 @@ import { describe, expect, test } from "bun:test"; import { adapterById } from "../../src/ingest/adapters/index.ts"; import type { Adapter } from "../../src/ingest/adapters/types.ts"; +import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts"; import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts"; +import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts"; import { inferType } from "../../src/ingest/parsers/game8.ts"; import { GachaEvent, type EventType } from "../../src/shared/schema.ts"; @@ -31,6 +33,10 @@ const CASES: Array<{ adapter: Adapter; fixture: string }> = [ { adapter: adapter("nikki-game8-events"), fixture: "fixtures/nikki/game8-events-2026-08-17" }, { adapter: adapter("p5x-game8-events"), fixture: "fixtures/p5x/game8-events-2026-08-17" }, { adapter: adapter("arknights-akwiki-events"), fixture: "fixtures/arknights/akwiki-events-2026-08-17" }, + // A `.html` fixture holding JSON, deliberately: this source is the MediaWiki + // action API, and `snapshots/` names every stored body `.html` whatever + // its content type. The fixture is the bytes the fetcher would store. + { adapter: adapter("r1999-fandom-events"), fixture: "fixtures/r1999/fandom-events-2026-08-17" }, ]; async function runAdapter(adapter: Adapter, fixture: string) { @@ -421,3 +427,92 @@ describe("wiki.gg parser", () => { } }); }); + +describe("fandom parser", () => { + const fixture = "fixtures/r1999/fandom-events-2026-08-17"; + const r1999 = adapter("r1999-fandom-events"); + + test("publishes only the events that have not ended", async () => { + // The page is an archive: 154 rows spanning every version since 1.1, of + // which six had not ended at the pinned clock. Counted independently off + // the fixture before this expectation was written — the count is the guard + // against a shape change making events vanish silently. + const events = await runAdapter(r1999, fixture); + expect(events).toHaveLength(6); + for (const e of events) { + expect(e.endsAt).not.toBeNull(); + expect(Date.parse(e.endsAt!)).toBeGreaterThanOrEqual(Date.parse(NOW)); + } + }); + + test("converts the stated UTC-5 offset rather than reading it as UTC", async () => { + // "August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)". Reading the + // wall clock as UTC would put every boundary five hours early — and a start + // that crossed a UTC midnight would move the event's ID with it. + const events = await runAdapter(r1999, fixture); + const version = events.find((e) => e.title === "On Another's Sorrow"); + expect(version?.startsAt).toBe("2026-08-13T10:00:00.000Z"); + expect(version?.endsAt).toBe("2026-09-21T09:59:00.000Z"); + expect(version?.startPrecision).toBe("exact"); + expect(version?.endPrecision).toBe("exact"); + }); + + test("takes the title from the , not the cell text", async () => { + // A missing banner image renders as a red link whose visible text is + // "File:A Stranger to Memory Lane Banner.png". A cell-text reader publishes + // that filename as the event's name. + const events = await runAdapter(r1999, fixture); + const titles = events.map((e) => e.title); + expect(titles).toContain("A Stranger to Memory Lane"); + for (const t of titles) { + expect(t).not.toMatch(/^File:/); + expect(t).not.toMatch(/\.png/i); + } + }); + + test("carries the section heading as the summary and types from it", async () => { + const events = await runAdapter(r1999, fixture); + const story = events.find((e) => e.title === "The You That's Meant to Be"); + // The title alone says nothing about what kind of event it is; the section + // it sits under does. + expect(story?.summary).toBe("Character Story Events"); + expect(story?.type).toBe("story"); + // The [edit] link MediaWiki renders inside every heading is not part of it. + for (const e of events) expect(e.summary).not.toMatch(/edit/i); + }); + + test("states one global end, not per-region ends", async () => { + // Every row reads (UTC-5) and the page draws no regional distinction. + for (const e of await runAdapter(r1999, fixture)) { + expect(e.regionScoped).toBe(false); + expect(e.regionEnds).toBeNull(); + } + }); + + test("rejects a body that is not an action=parse response", async () => { + // The plain wiki page answers a non-browser client with a Cloudflare + // interstitial. Feeding that to the parser must fail loudly rather than + // report zero events, which reads downstream as "nothing is on". + const interstitial = "Just a moment..."; + expect(fandomParser.canParse(interstitial)).toBe(false); + expect(fandomParser.canParse('{"error":{"code":"missingtitle"}}')).toBe(false); + expect(() => + r1999.parse(interstitial, { + now: NOW, + sourceUrl: r1999.url, + sourceId: r1999.id, + game: r1999.game, + }), + ).toThrow(/redesigned/); + }); + + test("a row stating no year on either half yields no event", async () => { + // "February 20th, 05:00 - March 27th, 04:59 (UTC-5)" — the fixture has + // exactly one, and there is no honest year to give it. + const body = await Bun.file(`${fixture}.html`).text(); + const rendered = renderedHtml(body); + expect(rendered).not.toBeNull(); + expect(rendered!).toContain("February 20th, 05:00 - March 27th, 04:59"); + expect(parseOrdinalDateTimeRange("February 20th, 05:00 - March 27th, 04:59 (UTC-5)")).toBeNull(); + }); +});
EventTime PeriodVersion

TITLE		August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)3.7