feat(ingest): add the Reverse: 1999 event source via Fandom's API
New `fandom` parser plus the r1999-fandom-events source, giving the game six live and upcoming events at exact precision. It fetches api.php?action=parse, not /wiki/Events. The rendered page answers a non-browser client with a Cloudflare challenge, and getting past that would be defeating an access control — the reason uma.moe was declined. The wiki's robots.txt instead allows /api.php?action= for *, and that endpoint serves our real User-Agent a 200, so this reads the sanctioned surface with our own headers and no impersonation. The body is therefore JSON, which is what canParse checks first: a challenge page or an error payload must fail loudly, not parse to zero events. Two page facts shape the parser. Titles come from each row's <b>, because a missing banner image renders as a red link reading "File:<Event> Banner.png" that a cell-text reader would publish as the event name. And the page is an archive of 154 rows since v1.1 with no ongoing section to gate on, so inclusion is decided against ctx.now — the six-event count is asserted against an independent extraction off the fixture, per docs/INGESTION.md § Testing. Because robots.txt is unreadable from a challenged address, the gate fails closed in CI and the source is skipped there — a warning, not a broken build. Refreshing it means running `bun run refresh` from an address Fandom serves. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1a65a1f06c
commit
3377fcb211
@@ -13,8 +13,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p
|
||||
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
||||
progress on events that repeat daily, and lets a user mark events completed.
|
||||
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across
|
||||
nine games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, four parsers, eleven sources across
|
||||
ten games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
||||
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
||||
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
||||
@@ -83,7 +83,7 @@ re-verify a sample against the live page afterward.
|
||||
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
||||
custom.ts — reader-authored games and events, and their key spaces
|
||||
src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||||
parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game
|
||||
parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts — keyed by SITE, not game
|
||||
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
||||
src/client/ React app, service worker, manifest
|
||||
state/ progress, daily log, ignores, prefs, sort — all localStorage
|
||||
@@ -91,7 +91,7 @@ src/client/ React app, service worker, manifest
|
||||
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
||||
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||||
serve.ts static server + /api/health
|
||||
test/ 401 tests
|
||||
test/ 466 tests
|
||||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||||
snapshots/ current page per source, rewritten by refresh — see its README
|
||||
```
|
||||
@@ -190,9 +190,9 @@ Sources are community wikis. Treat them as a guest would:
|
||||
|
||||
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
||||
- One request per source per refresh cycle, minimum 6 hours apart.
|
||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
|
||||
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
|
||||
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the
|
||||
eleven sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive
|
||||
as eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||
what a burst looks like from the far end regardless of our intent.
|
||||
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
||||
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
||||
@@ -224,7 +224,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and
|
||||
|---|---|
|
||||
| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source |
|
||||
| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` |
|
||||
| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have |
|
||||
| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below |
|
||||
| `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
||||
|
||||
wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference`
|
||||
@@ -233,6 +233,27 @@ trains nothing, and no LLM reads the page content — constraint 2 is what keeps
|
||||
load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive,
|
||||
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
||||
|
||||
**Fandom: read the API, never the page.** `reverse1999.fandom.com/wiki/Events` answers a non-browser
|
||||
client with a Cloudflare managed challenge — HTTP 403, `Just a moment…`, "Enable JavaScript" — and so
|
||||
does `/robots.txt` itself, from a datacenter address. Browser-shaped headers or a JS-executing client
|
||||
would get past both and **must not be used**: that is defeating a deliberate access control, the same
|
||||
reason `uma.moe` was declined above.
|
||||
|
||||
What makes this source legitimate anyway is that the wiki publishes a second, sanctioned surface. Its
|
||||
`robots.txt` — read in a browser, where it serves fine — has no `Disallow: /` for `*` and explicitly
|
||||
**allows** `/api.php?action=`, and that endpoint answers our real `User-Agent` with a `200` and a JSON
|
||||
body. So the adapter fetches `api.php?action=parse&page=Events`, with no impersonation anywhere: our
|
||||
own headers, on a path the site put in writing. The only namespaces `*` is refused are `Special:`,
|
||||
`User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links
|
||||
for that reason.
|
||||
|
||||
One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
|
||||
robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
|
||||
broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if
|
||||
*every* source is blocked — so the scheduled refresh simply never updates this game, and the feed
|
||||
falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address
|
||||
Fandom serves, which is how its first snapshot was taken.
|
||||
|
||||
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
|
||||
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
|
||||
read). Anything that would make it fetch more often is a change to this section first.
|
||||
|
||||
+18
-1
@@ -48,11 +48,28 @@ Consequences worth internalising:
|
||||
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X |
|
||||
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
|
||||
| `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights |
|
||||
| `fandom` | Fandom wikis via the MediaWiki `action=parse` API — `Event \| Time Period \| Version` wikitables | Reverse: 1999 |
|
||||
|
||||
`wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server
|
||||
region, so its events carry exact precision and real `regionEnds`. Prefer a source like that over a
|
||||
prose wiki when both exist, and give it a higher `priority`.
|
||||
|
||||
`fandom` is the only parser whose body is not HTML: it reads an `action=parse` JSON envelope and
|
||||
takes the rendered wikitext out of `parse.text`. That is not a preference — the rendered page is
|
||||
behind a Cloudflare challenge and the API is the surface the wiki's `robots.txt` allows, so the
|
||||
adapter's URL is an API call and the stored snapshot is JSON despite its `.html` name (every body
|
||||
`snapshots/` keeps is named `<id>.html`, whatever its content type). Its `canParse` therefore checks
|
||||
the envelope as well as the table, because an error payload or a challenge page must fail loudly
|
||||
rather than parse to zero events. Two page facts drive the rest of it:
|
||||
|
||||
- **The title is the row's `<b>`, never the cell text.** The cell leads with a banner image, and a
|
||||
missing image renders as a red link reading `File:<Event> Banner.png` — which a cell-text reader
|
||||
publishes as the event's name.
|
||||
- **These pages are archives, not schedules.** All five tables list every event since version 1.1
|
||||
(154 rows, six of them unfinished when the fixture was captured), with no "ongoing" section to
|
||||
anchor on. Inclusion is therefore decided against `ctx.now`, the one parser here that does so;
|
||||
`akwiki` and `game8` can gate on a heading instead, and should where one exists.
|
||||
|
||||
`akwiki` shares a host family with `wikigg` and nothing else — arknights.wiki.gg has no `mp-event`
|
||||
cards, so the two are separate modules rather than one parser with a branch. Two things about that
|
||||
page shape are worth knowing before touching it:
|
||||
@@ -247,7 +264,7 @@ Three constraints it holds:
|
||||
repair and drop emits a note whose default sink is `console.warn` — silence is not something a
|
||||
future caller gets for free (§ Silent drops).
|
||||
- **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the
|
||||
title *and* the incoming ID was minted the standard way. All seven fixtures pass through with
|
||||
title *and* the incoming ID was minted the standard way. All eleven fixtures pass through with
|
||||
zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys
|
||||
and moving one orphans a reader's marks with no server-side recovery.
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
[
|
||||
{
|
||||
"id": "r1999:8-bit-arcade-remix:2026-08-13",
|
||||
"game": "r1999",
|
||||
"title": "8-Bit Arcade Remix",
|
||||
"type": "other",
|
||||
"summary": "Time-Limited Events",
|
||||
"startsAt": "2026-08-13T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-24T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "r1999:on-anothers-sorrow:2026-08-13",
|
||||
"game": "r1999",
|
||||
"title": "On Another's Sorrow",
|
||||
"type": "other",
|
||||
"summary": "Version Events",
|
||||
"startsAt": "2026-08-13T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-21T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/wiki/On_Another%27s_Sorrow",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "r1999:the-you-thats-meant-to-be:2026-08-13",
|
||||
"game": "r1999",
|
||||
"title": "The You That's Meant to Be",
|
||||
"type": "story",
|
||||
"summary": "Character Story Events",
|
||||
"startsAt": "2026-08-13T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-24T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "r1999:truth-a-posteriori:2026-08-22",
|
||||
"game": "r1999",
|
||||
"title": "Truth a Posteriori",
|
||||
"type": "other",
|
||||
"summary": "Side Events",
|
||||
"startsAt": "2026-08-22T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-24T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "r1999:a-stranger-to-memory-lane:2026-09-03",
|
||||
"game": "r1999",
|
||||
"title": "A Stranger to Memory Lane",
|
||||
"type": "story",
|
||||
"summary": "Character Story Events",
|
||||
"startsAt": "2026-09-03T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-24T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "r1999:what-does-a-brick-want:2026-09-17",
|
||||
"game": "r1999",
|
||||
"title": "What Does a Brick Want",
|
||||
"type": "other",
|
||||
"summary": "Warm Up Events",
|
||||
"startsAt": "2026-09-17T10:00:00.000Z",
|
||||
"startPrecision": "exact",
|
||||
"endsAt": "2026-09-24T09:59:00.000Z",
|
||||
"endPrecision": "exact",
|
||||
"regionScoped": false,
|
||||
"regionEnds": null,
|
||||
"sourceUrl": "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
"sourceId": "r1999-fandom-events",
|
||||
"status": "published",
|
||||
"confidence": 0.95,
|
||||
"extractionMethod": "parser",
|
||||
"version": 1,
|
||||
"firstSeenAt": "2026-08-14T00:00:00.000Z",
|
||||
"updatedAt": "2026-08-14T00:00:00.000Z"
|
||||
}
|
||||
]
|
||||
File diff suppressed because one or more lines are too long
@@ -84,6 +84,18 @@ const SOURCES: SourceSpec[] = [
|
||||
url: "https://game8.co/games/Persona-5-Phantom-X/archives/532244",
|
||||
parserId: "game8",
|
||||
},
|
||||
{
|
||||
id: "r1999-fandom-events",
|
||||
game: "r1999",
|
||||
// The MediaWiki API, not `/wiki/Events`. The rendered page answers a
|
||||
// non-browser client with a Cloudflare interstitial, while this wiki's
|
||||
// robots.txt allows `/api.php?action=` for `User-agent: *` and the endpoint
|
||||
// serves our real User-Agent a 200. See `parsers/fandom.ts` for the full
|
||||
// reasoning; the short version is that this is the surface the site put in
|
||||
// writing, reached without pretending to be anything we are not.
|
||||
url: "https://reverse1999.fandom.com/api.php?action=parse&page=Events&prop=text&formatversion=2&format=json",
|
||||
parserId: "fandom",
|
||||
},
|
||||
];
|
||||
|
||||
function toAdapter(spec: SourceSpec): Adapter {
|
||||
|
||||
@@ -0,0 +1,183 @@
|
||||
import { eventId, type GachaEvent } from "../../shared/schema.ts";
|
||||
import { parseOrdinalDateTimeRange } from "../dates.ts";
|
||||
import { text } from "../html.ts";
|
||||
import type { ParseContext } from "../adapters/types.ts";
|
||||
import { inferType } from "./game8.ts";
|
||||
import type { SourceParser } from "./types.ts";
|
||||
|
||||
/**
|
||||
* Fandom wikis, read through the MediaWiki action API.
|
||||
*
|
||||
* **Why the API and not the page.** `reverse1999.fandom.com/wiki/Events`
|
||||
* answers a non-browser client with a Cloudflare interstitial ("Just a
|
||||
* moment…", HTTP 403), and dressing our fetcher up as a browser to get past it
|
||||
* would be defeating a deliberate access control — the reason `uma.moe` was
|
||||
* declined in AGENTS.md § Scraping conduct. The wiki's own `robots.txt` instead
|
||||
* *allows* `/api.php?action=` for `User-agent: *`, and that endpoint answers our
|
||||
* real User-Agent with a 200. So this parser reads the sanctioned surface with
|
||||
* no impersonation anywhere: same headers, a path the site put in writing.
|
||||
*
|
||||
* The body is therefore JSON rather than HTML:
|
||||
*
|
||||
* {"parse":{"title":"Events","pageid":3479,"text":"<div …>"}}
|
||||
*
|
||||
* `parse.text` is the rendered wikitext, and the shape below is what this family
|
||||
* of pages puts in it — `wikitable`s under one `h2` per section:
|
||||
*
|
||||
* <h2>Version Events</h2>
|
||||
* <table class="wikitable sortable">
|
||||
* <tr><th>Event</th><th>Time Period</th><th>Version</th></tr>
|
||||
* <tr><td><span …><img …></span><br><b>TITLE</b></td>
|
||||
* <td>August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)</td>
|
||||
* <td>3.7</td>
|
||||
*
|
||||
* Two details decide most of the code.
|
||||
*
|
||||
* **The title is the `<b>`, never the cell text.** The cell leads with a banner
|
||||
* image, and when that image is missing MediaWiki renders a red link whose
|
||||
* visible text is `File:A Stranger to Memory Lane Banner.png`. A cell-text
|
||||
* reader publishes that as the event's name. The `<b>` holds the title in both
|
||||
* cases.
|
||||
*
|
||||
* **These pages are archives, not schedules.** All five tables list every event
|
||||
* since 1.1 — 154 rows, of which six had not yet ended when the fixture was
|
||||
* captured. Publishing the rest would put three years of finished events on the
|
||||
* calendar and hand the validator a hundred rows whose start predates its
|
||||
* two-year sanity window. So inclusion is decided against `ctx.now`, which is
|
||||
* injected precisely so a parser can be time-aware without reading the clock.
|
||||
*/
|
||||
|
||||
const SECTION = /<h2\b[^>]*>([\s\S]*?)<\/h2>|<table\b[^>]*>([\s\S]*?)<\/table>/gi;
|
||||
const ROW = /<tr\b[^>]*>([\s\S]*?)<\/tr>/gi;
|
||||
const CELL = /<t([dh])\b[^>]*>([\s\S]*?)<\/t\1>/gi;
|
||||
const BOLD = /<b\b[^>]*>([\s\S]*?)<\/b>/i;
|
||||
const EDIT_SECTION = /<span\b[^>]*class="[^"]*mw-editsection[^"]*"[^>]*>[\s\S]*?<\/span>/gi;
|
||||
/**
|
||||
* A link to the event's own article. `Special:` is excluded deliberately: a
|
||||
* missing banner image renders as a `Special:Upload` link, which is both the
|
||||
* wrong page to send a reader to and a path this wiki's robots.txt disallows.
|
||||
*/
|
||||
const ARTICLE_LINK = /<a\b[^>]*href="(\/wiki\/(?!Special:)[^"#?]+)"/i;
|
||||
|
||||
/** The rendered HTML inside an `action=parse` response, or null. */
|
||||
export function renderedHtml(body: string): string | null {
|
||||
let payload: unknown;
|
||||
try {
|
||||
payload = JSON.parse(body);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
const html = (payload as { parse?: { text?: unknown } } | null)?.parse?.text;
|
||||
return typeof html === "string" ? html : null;
|
||||
}
|
||||
|
||||
export function parseFandomEventsPage(
|
||||
body: string,
|
||||
ctx: ParseContext,
|
||||
): GachaEvent[] {
|
||||
const rendered = renderedHtml(body);
|
||||
if (rendered === null) return [];
|
||||
|
||||
const flat = rendered.replace(EDIT_SECTION, "").replace(/\s+/g, " ");
|
||||
const nowMs = Date.parse(ctx.now);
|
||||
const out: GachaEvent[] = [];
|
||||
let section = "";
|
||||
|
||||
for (const node of flat.matchAll(SECTION)) {
|
||||
const heading = node[1];
|
||||
if (heading !== undefined) {
|
||||
section = text(heading);
|
||||
continue;
|
||||
}
|
||||
|
||||
const table = node[2] ?? "";
|
||||
// Every section table states the same three columns. Checking them keeps a
|
||||
// navbox or an infobox elsewhere in the page from being read as a schedule.
|
||||
const headers = [...table.matchAll(/<th\b[^>]*>([\s\S]*?)<\/th>/gi)].map(
|
||||
(h) => text(h[1] ?? "").toLowerCase(),
|
||||
);
|
||||
if (!headers.includes("event") || !headers.includes("time period")) continue;
|
||||
|
||||
for (const row of table.matchAll(ROW)) {
|
||||
const cells = [...(row[1] ?? "").matchAll(CELL)].map((c) => ({
|
||||
tag: c[1] ?? "",
|
||||
html: c[2] ?? "",
|
||||
}));
|
||||
if (cells.length < 2 || cells.some((c) => c.tag === "h")) continue;
|
||||
|
||||
const titleCell = cells[0]?.html ?? "";
|
||||
const title = text(BOLD.exec(titleCell)?.[1] ?? "");
|
||||
if (title.length === 0) continue;
|
||||
|
||||
const range = parseOrdinalDateTimeRange(text(cells[1]?.html ?? ""));
|
||||
// A row stating no year on either half is unresolvable — the fixture has
|
||||
// exactly one — and a row this reader cannot date yields no event rather
|
||||
// than a guessed one.
|
||||
if (range === null) continue;
|
||||
|
||||
const { start, end } = range;
|
||||
if (end.iso <= start.iso) continue;
|
||||
|
||||
// Live and upcoming only. An event whose end has passed is history this
|
||||
// page keeps and the calendar does not want.
|
||||
if (Date.parse(end.iso) < nowMs) continue;
|
||||
|
||||
const href = ARTICLE_LINK.exec(titleCell)?.[1];
|
||||
|
||||
out.push({
|
||||
id: eventId(ctx.game, title, start.iso),
|
||||
game: ctx.game,
|
||||
title,
|
||||
// The section heading is the source's own classification, and a better
|
||||
// signal than the title alone — "Character Story Events" names a story
|
||||
// event whose title says nothing about it.
|
||||
type: inferType(`${title} ${section}`),
|
||||
summary: section.length > 0 ? section : null,
|
||||
startsAt: start.iso,
|
||||
startPrecision: start.precision,
|
||||
endsAt: end.iso,
|
||||
endPrecision: end.precision,
|
||||
// One global server on a single stated offset: every row on the page
|
||||
// reads (UTC-5), and the page draws no distinction between regions.
|
||||
// `regionScoped` means the source separates them, and this one does not.
|
||||
regionScoped: false,
|
||||
regionEnds: null,
|
||||
sourceUrl:
|
||||
href === undefined
|
||||
? ctx.sourceUrl
|
||||
: new URL(href, ctx.sourceUrl).toString(),
|
||||
sourceId: ctx.sourceId,
|
||||
status: "published",
|
||||
// Both boundaries are exact instants converted from a stated offset,
|
||||
// with nothing inferred — the same footing as the wiki.gg timers.
|
||||
confidence: 0.95,
|
||||
extractionMethod: "parser",
|
||||
version: 1,
|
||||
firstSeenAt: ctx.now,
|
||||
updatedAt: ctx.now,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return out.sort((a, b) =>
|
||||
a.startsAt === b.startsAt
|
||||
? a.id.localeCompare(b.id)
|
||||
: a.startsAt.localeCompare(b.startsAt),
|
||||
);
|
||||
}
|
||||
|
||||
export const fandomParser: SourceParser = {
|
||||
id: "fandom",
|
||||
label: "Fandom",
|
||||
canParse(body: string): boolean {
|
||||
// Structural, and deliberately about the envelope as much as the content:
|
||||
// this source is an API, so a body that is not an `action=parse` response
|
||||
// is the failure worth catching loudly — an error payload, a login wall, or
|
||||
// the Cloudflare interstitial the plain page serves would all land here.
|
||||
const rendered = renderedHtml(body);
|
||||
if (rendered === null) return false;
|
||||
return /class="[^"]*wikitable/.test(rendered) && /Time Period/i.test(rendered);
|
||||
},
|
||||
parse: parseFandomEventsPage,
|
||||
};
|
||||
@@ -1,4 +1,5 @@
|
||||
import { arknightsWikiParser } from "./akwiki.ts";
|
||||
import { fandomParser } from "./fandom.ts";
|
||||
import { game8Parser } from "./game8.ts";
|
||||
import { wikiGgParser } from "./wikigg.ts";
|
||||
import type { SourceParser } from "./types.ts";
|
||||
@@ -12,6 +13,7 @@ export const PARSERS: SourceParser[] = [
|
||||
game8Parser,
|
||||
wikiGgParser,
|
||||
arknightsWikiParser,
|
||||
fandomParser,
|
||||
];
|
||||
|
||||
export function parserById(id: string): SourceParser | undefined {
|
||||
@@ -20,5 +22,6 @@ export function parserById(id: string): SourceParser | undefined {
|
||||
|
||||
export type { SourceParser } from "./types.ts";
|
||||
export { arknightsWikiParser } from "./akwiki.ts";
|
||||
export { fandomParser } from "./fandom.ts";
|
||||
export { game8Parser } from "./game8.ts";
|
||||
export { wikiGgParser } from "./wikigg.ts";
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { adapterById } from "../../src/ingest/adapters/index.ts";
|
||||
import type { Adapter } from "../../src/ingest/adapters/types.ts";
|
||||
import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts";
|
||||
import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts";
|
||||
import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts";
|
||||
import { inferType } from "../../src/ingest/parsers/game8.ts";
|
||||
import { GachaEvent, type EventType } from "../../src/shared/schema.ts";
|
||||
|
||||
@@ -31,6 +33,10 @@ const CASES: Array<{ adapter: Adapter; fixture: string }> = [
|
||||
{ adapter: adapter("nikki-game8-events"), fixture: "fixtures/nikki/game8-events-2026-08-17" },
|
||||
{ adapter: adapter("p5x-game8-events"), fixture: "fixtures/p5x/game8-events-2026-08-17" },
|
||||
{ adapter: adapter("arknights-akwiki-events"), fixture: "fixtures/arknights/akwiki-events-2026-08-17" },
|
||||
// A `.html` fixture holding JSON, deliberately: this source is the MediaWiki
|
||||
// action API, and `snapshots/` names every stored body `<id>.html` whatever
|
||||
// its content type. The fixture is the bytes the fetcher would store.
|
||||
{ adapter: adapter("r1999-fandom-events"), fixture: "fixtures/r1999/fandom-events-2026-08-17" },
|
||||
];
|
||||
|
||||
async function runAdapter(adapter: Adapter, fixture: string) {
|
||||
@@ -421,3 +427,92 @@ describe("wiki.gg parser", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("fandom parser", () => {
|
||||
const fixture = "fixtures/r1999/fandom-events-2026-08-17";
|
||||
const r1999 = adapter("r1999-fandom-events");
|
||||
|
||||
test("publishes only the events that have not ended", async () => {
|
||||
// The page is an archive: 154 rows spanning every version since 1.1, of
|
||||
// which six had not ended at the pinned clock. Counted independently off
|
||||
// the fixture before this expectation was written — the count is the guard
|
||||
// against a shape change making events vanish silently.
|
||||
const events = await runAdapter(r1999, fixture);
|
||||
expect(events).toHaveLength(6);
|
||||
for (const e of events) {
|
||||
expect(e.endsAt).not.toBeNull();
|
||||
expect(Date.parse(e.endsAt!)).toBeGreaterThanOrEqual(Date.parse(NOW));
|
||||
}
|
||||
});
|
||||
|
||||
test("converts the stated UTC-5 offset rather than reading it as UTC", async () => {
|
||||
// "August 13th, 05:00 - September 21st, 2026, 04:59 (UTC-5)". Reading the
|
||||
// wall clock as UTC would put every boundary five hours early — and a start
|
||||
// that crossed a UTC midnight would move the event's ID with it.
|
||||
const events = await runAdapter(r1999, fixture);
|
||||
const version = events.find((e) => e.title === "On Another's Sorrow");
|
||||
expect(version?.startsAt).toBe("2026-08-13T10:00:00.000Z");
|
||||
expect(version?.endsAt).toBe("2026-09-21T09:59:00.000Z");
|
||||
expect(version?.startPrecision).toBe("exact");
|
||||
expect(version?.endPrecision).toBe("exact");
|
||||
});
|
||||
|
||||
test("takes the title from the <b>, not the cell text", async () => {
|
||||
// A missing banner image renders as a red link whose visible text is
|
||||
// "File:A Stranger to Memory Lane Banner.png". A cell-text reader publishes
|
||||
// that filename as the event's name.
|
||||
const events = await runAdapter(r1999, fixture);
|
||||
const titles = events.map((e) => e.title);
|
||||
expect(titles).toContain("A Stranger to Memory Lane");
|
||||
for (const t of titles) {
|
||||
expect(t).not.toMatch(/^File:/);
|
||||
expect(t).not.toMatch(/\.png/i);
|
||||
}
|
||||
});
|
||||
|
||||
test("carries the section heading as the summary and types from it", async () => {
|
||||
const events = await runAdapter(r1999, fixture);
|
||||
const story = events.find((e) => e.title === "The You That's Meant to Be");
|
||||
// The title alone says nothing about what kind of event it is; the section
|
||||
// it sits under does.
|
||||
expect(story?.summary).toBe("Character Story Events");
|
||||
expect(story?.type).toBe("story");
|
||||
// The [edit] link MediaWiki renders inside every heading is not part of it.
|
||||
for (const e of events) expect(e.summary).not.toMatch(/edit/i);
|
||||
});
|
||||
|
||||
test("states one global end, not per-region ends", async () => {
|
||||
// Every row reads (UTC-5) and the page draws no regional distinction.
|
||||
for (const e of await runAdapter(r1999, fixture)) {
|
||||
expect(e.regionScoped).toBe(false);
|
||||
expect(e.regionEnds).toBeNull();
|
||||
}
|
||||
});
|
||||
|
||||
test("rejects a body that is not an action=parse response", async () => {
|
||||
// The plain wiki page answers a non-browser client with a Cloudflare
|
||||
// interstitial. Feeding that to the parser must fail loudly rather than
|
||||
// report zero events, which reads downstream as "nothing is on".
|
||||
const interstitial = "<html><head><title>Just a moment...</title></head></html>";
|
||||
expect(fandomParser.canParse(interstitial)).toBe(false);
|
||||
expect(fandomParser.canParse('{"error":{"code":"missingtitle"}}')).toBe(false);
|
||||
expect(() =>
|
||||
r1999.parse(interstitial, {
|
||||
now: NOW,
|
||||
sourceUrl: r1999.url,
|
||||
sourceId: r1999.id,
|
||||
game: r1999.game,
|
||||
}),
|
||||
).toThrow(/redesigned/);
|
||||
});
|
||||
|
||||
test("a row stating no year on either half yields no event", async () => {
|
||||
// "February 20th, 05:00 - March 27th, 04:59 (UTC-5)" — the fixture has
|
||||
// exactly one, and there is no honest year to give it.
|
||||
const body = await Bun.file(`${fixture}.html`).text();
|
||||
const rendered = renderedHtml(body);
|
||||
expect(rendered).not.toBeNull();
|
||||
expect(rendered!).toContain("February 20th, 05:00 - March 27th, 04:59");
|
||||
expect(parseOrdinalDateTimeRange("February 20th, 05:00 - March 27th, 04:59 (UTC-5)")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user