feat(ingest): add the Reverse: 1999 event source via Fandom's API
New `fandom` parser plus the r1999-fandom-events source, giving the game six live and upcoming events at exact precision. It fetches api.php?action=parse, not /wiki/Events. The rendered page answers a non-browser client with a Cloudflare challenge, and getting past that would be defeating an access control — the reason uma.moe was declined. The wiki's robots.txt instead allows /api.php?action= for *, and that endpoint serves our real User-Agent a 200, so this reads the sanctioned surface with our own headers and no impersonation. The body is therefore JSON, which is what canParse checks first: a challenge page or an error payload must fail loudly, not parse to zero events. Two page facts shape the parser. Titles come from each row's <b>, because a missing banner image renders as a red link reading "File:<Event> Banner.png" that a cell-text reader would publish as the event name. And the page is an archive of 154 rows since v1.1 with no ongoing section to gate on, so inclusion is decided against ctx.now — the six-event count is asserted against an independent extraction off the fixture, per docs/INGESTION.md § Testing. Because robots.txt is unreadable from a challenged address, the gate fails closed in CI and the source is skipped there — a warning, not a broken build. Refreshing it means running `bun run refresh` from an address Fandom serves. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
1a65a1f06c
commit
3377fcb211
@@ -13,8 +13,8 @@ A web app that aggregates live and upcoming events across popular gacha games, p
|
||||
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
||||
progress on events that repeat daily, and lets a user mark events completed.
|
||||
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across
|
||||
nine games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
**Status: working app, refreshing itself on a schedule.** Schema, four parsers, eleven sources across
|
||||
ten games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
||||
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
||||
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
||||
@@ -83,7 +83,7 @@ re-verify a sample against the live page afterward.
|
||||
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
||||
custom.ts — reader-authored games and events, and their key spaces
|
||||
src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||||
parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game
|
||||
parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts — keyed by SITE, not game
|
||||
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
||||
src/client/ React app, service worker, manifest
|
||||
state/ progress, daily log, ignores, prefs, sort — all localStorage
|
||||
@@ -91,7 +91,7 @@ src/client/ React app, service worker, manifest
|
||||
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
||||
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||||
serve.ts static server + /api/health
|
||||
test/ 401 tests
|
||||
test/ 466 tests
|
||||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||||
snapshots/ current page per source, rewritten by refresh — see its README
|
||||
```
|
||||
@@ -190,9 +190,9 @@ Sources are community wikis. Treat them as a guest would:
|
||||
|
||||
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
||||
- One request per source per refresh cycle, minimum 6 hours apart.
|
||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
|
||||
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
|
||||
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the
|
||||
eleven sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive
|
||||
as eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||
what a burst looks like from the far end regardless of our intent.
|
||||
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
||||
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
||||
@@ -224,7 +224,7 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and
|
||||
|---|---|
|
||||
| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source |
|
||||
| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` |
|
||||
| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have |
|
||||
| `reverse1999.fandom.com` | **Built** (2026-08-17), via `api.php`, not the wiki page — see § Fandom below |
|
||||
| `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
||||
|
||||
wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference`
|
||||
@@ -233,6 +233,27 @@ trains nothing, and no LLM reads the page content — constraint 2 is what keeps
|
||||
load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive,
|
||||
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
||||
|
||||
**Fandom: read the API, never the page.** `reverse1999.fandom.com/wiki/Events` answers a non-browser
|
||||
client with a Cloudflare managed challenge — HTTP 403, `Just a moment…`, "Enable JavaScript" — and so
|
||||
does `/robots.txt` itself, from a datacenter address. Browser-shaped headers or a JS-executing client
|
||||
would get past both and **must not be used**: that is defeating a deliberate access control, the same
|
||||
reason `uma.moe` was declined above.
|
||||
|
||||
What makes this source legitimate anyway is that the wiki publishes a second, sanctioned surface. Its
|
||||
`robots.txt` — read in a browser, where it serves fine — has no `Disallow: /` for `*` and explicitly
|
||||
**allows** `/api.php?action=`, and that endpoint answers our real `User-Agent` with a `200` and a JSON
|
||||
body. So the adapter fetches `api.php?action=parse&page=Events`, with no impersonation anywhere: our
|
||||
own headers, on a path the site put in writing. The only namespaces `*` is refused are `Special:`,
|
||||
`User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links
|
||||
for that reason.
|
||||
|
||||
One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the
|
||||
robots gate **fails closed there and the source is skipped**. That is a warning line rather than a
|
||||
broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if
|
||||
*every* source is blocked — so the scheduled refresh simply never updates this game, and the feed
|
||||
falls back to the checked-in fixture. Refreshing it means running `bun run refresh` from an address
|
||||
Fandom serves, which is how its first snapshot was taken.
|
||||
|
||||
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
|
||||
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
|
||||
read). Anything that would make it fetch more often is a change to this section first.
|
||||
|
||||
Reference in New Issue
Block a user