diff --git a/AGENTS.md b/AGENTS.md index 197e73f..1ebf20e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -301,6 +301,30 @@ re-litigated each pass: | `stellasora.miraheze.org` | **Built** (2026-08-19), from the front page's `Current Banners` module and **not** `/wiki/Banner_List` — see § Stella Sora below | | `game8.co/games/Chaos-Zero-Nightmare` | **Built** (2026-08-19). Zero parser work — the existing `game8` parser reads it. The ninth game8 source, so fixture-backed in CI from day one | | `game8.co/games/Umamusume-Pretty-Derby` | **Built** (2026-08-19), off the stable `List of All Banners` page, not the monthly release-schedule pages whose URL changes every month. Cost a widening of `game8.ts`'s section and column vocabulary — see § Working on parsers | +| `nikke-…-international.fandom.com` | **Blocked, not declined** (2026-08-19). The API answers our User-Agent with a 200 and the data is the best of any unbuilt source — story events *and* dated pickup banners, with an evidenced 05:00 UTC+9 reset. But this wiki's `robots.txt` cannot be read from here **at all**: 403 to our fetcher and an unresolving Cloudflare challenge to a real browser. The permission has to be in writing before the adapter exists, so this waits on someone reading that file from an address Fandom serves | +| `infinitynikki.miraheze.org` | **Declined.** Exists and serves `robots.txt`, but the wiki is abandoned — front page last edited 11 February 2025 and `/wiki/Events` returns a permission error. Checked as a replacement for the stale Infinity Nikki Game8 page | +| `prydwen.gg/infinity-nikki` | **Declined.** 404 — prydwen does not cover Infinity Nikki | +| `grayravens.com` (Punishing: Gray Raven) | **Declined.** Conduct is fine; the data is not. The whole 626 KB `/wiki/Events` page contains exactly one date range, written as prose, one event per six-week patch | +| `guardian-tales.fandom.com` | **Declined.** Parses fine and contains no 2026 date at all — newest dated entry is 2025. The `bluearchive.fandom.com` failure again: parses cleanly to nothing live | +| `blhx.fandom.com`, `azurlane-archive.fandom.com` | **Declined.** The two Fandom alternatives to the declined koumakan wiki are dead archives — `Event_Calendar` stops in **2021**, and the archive wiki's headings have nothing under them. Azur Lane still has no source | +| Aether Gazer | **Do not build.** The developer confirmed no further content updates after 23 July 2026, with store listings removed 17 October 2026. The wiki dates nothing anyway — `Event_Guide_List` is an image gallery. A lane that will be empty by winter | + +**The Infinity Nikki lane is knowingly stale, and there is nowhere to move it** (checked +2026-08-19). `game8.co/games/Infinity-Nikki/archives/487445` fetches, parses and passes every test — +and its page says `Last updated on: August 31, 2025`. It mentions the year 2026 zero times. Seven +events parse out of it, of which **five carry `endsAt: null`** and so read as live-with-unknown-end +forever, on a calendar whose whole purpose is telling a reader what is still on. That is worse than +an empty lane, and it is the failure this product exists to prevent — arriving through a source that +looks perfectly healthy to the runner, because a stale page is not a broken one. + +Every replacement was checked and none works: `infinitynikki.wiki.gg` 401s (no such wiki), +`prydwen.gg` does not cover the game, `infinitynikki.miraheze.org` is abandoned (front page last +edited February 2025, `/wiki/Events` a permission error), Game8's own `Banners` page for the game is +staler still (May 2025) and its whole Infinity Nikki wiki is unmaintained, and the Fandom wiki is +behind the blanket 403 above. **This is a decision for the repository owner, not a bug to fix in +code**: either the lane stays and the footer's freshness disclosure carries it, or the game is +retired. Retiring it is not free — `nikki` is the first segment of every completion key that game's +readers have, so dropping the `GameId` orphans them with no server-side recovery (§ Event IDs). `.github/ISSUE_TEMPLATE/feature_request.yml` points readers at that table by heading, so a source request can be checked against it before anyone writes it up — the loudest feedback on the first @@ -327,6 +351,19 @@ own headers, on a path the site put in writing. The only namespaces `*` is refus `User:`, `Template:` and `Help:`, none of which we want; `parsers/fandom.ts` skips `Special:` links for that reason. +**Fandom's posture tightened on 2026-08-19, and it now covers every wiki.** On 2026-08-18 the +standard `robots.txt` was still readable from a plain address — `blhx.fandom.com` served it `200`, +which is how the permission above was confirmed. As of 2026-08-19 **every** Fandom wiki tried +(`reverse1999`, `fategrandorder`, `nikke-…-international`, `infinitynikki`, `blhx`) answers `403` +to our fetcher, and a real headless browser gets a Cloudflare managed challenge that never resolves. +Two consequences, and neither is a licence to work around it: + +- The two built Fandom sources now report `skipped_robots` on **every** run, from any address we + have, so `r1999` and `fgo` are permanently fixture-backed until someone refreshes them from an + address Fandom serves. `fgo` has never had a snapshot at all. +- A **new** Fandom source cannot be added at all, because the permission cannot be read. That is + what blocks Nikke, whose data is otherwise the best of any unbuilt source. + One consequence to keep in mind: because `/robots.txt` is unreadable from a challenged address, the robots gate **fails closed there and the source is skipped**. That is a warning line rather than a broken build — `skipped_robots` does not touch the failure streak, and the run only hard-fails if diff --git a/docs/FEEDBACK.md b/docs/FEEDBACK.md index 3ce6426..eac0a95 100644 --- a/docs/FEEDBACK.md +++ b/docs/FEEDBACK.md @@ -305,8 +305,9 @@ diagnosis in each item still holds — what changed is whether it has been acted | P1b cap the long list | **Done** (2026-08-18). Both sections cap at six with "show all N" — truncation of the view, not a re-sort | | P1b persist `view` | **Done** (2026-08-18). `prefs.view`, and the first run now asks which one to open on (PRD F8) | | P1b Calendar → Timeline | **Done.** The tab reads "Timeline" | -| P1c more games | **Done, four of them.** Infinity Nikki, Persona 5: The Phantom X, Reverse: 1999, Blue Archive — one commit each. Of the games named in the thread, Azur Lane and Umamusume are **declined on conduct** rather than pending, and the declined and cleared-but-unbuilt candidates are recorded in `AGENTS.md` § Scraping conduct so they are not re-litigated | +| P1c more games | **Done, nine of them.** Infinity Nikki, Persona 5: The Phantom X, Reverse: 1999, Blue Archive, then Fate/Grand Order and hololive Dreams, then Girls' Frontline 2, Stella Sora, Chaos Zero Nightmare and Umamusume (2026-08-19). Of the games named in this thread, **Umamusume is now built** — via Game8, not the declined `uma.moe` — and Azur Lane and Aether Gazer stay declined. **Nikke is blocked, not declined**: its data is the best of anything unbuilt and its wiki's `robots.txt` can no longer be read from any client here, so the permission cannot be established. See `docs/SOURCES.md` § 3 | | P1c roadmap in the colophon | **Not done** | +| P1 games still unserved | Nikke (blocked on a readable `robots.txt`), Azur Lane, Punishing: Gray Raven, Guardian Tales, Aether Gazer (shutting down), Silver Palace (unreleased). Separately, **Infinity Nikki's source is stale** and has no reachable replacement — `docs/SOURCES.md` § 11 | | P2 colour and screenshot | **Layout done** (2026-08-18), colour untouched. The palette was left alone deliberately — the two-axis system was not what made the screenshot read as unfinished, a dense single column on a wide screen was. Past `lg` the page is now a pinned deadline rail beside the lists, the timeline is a board with its axis and names pinned, and the footer and settings are columns. The screenshot still wants reshooting | | P2 custom events | **Done.** PRD F13 first, then the code; `mygame:` / `myevent:` key spaces, in the export | diff --git a/docs/SOURCES.md b/docs/SOURCES.md index 044229d..f41ebda 100644 --- a/docs/SOURCES.md +++ b/docs/SOURCES.md @@ -1,9 +1,18 @@ # SOURCES.md — event sources for the games still missing A source assessment for the games named in `docs/FEEDBACK.md` § P1 that still have no adapter. -Written 2026-08-18. Nothing here has been built: no fixture was captured, no `SOURCES` entry added, -no parser written. This is the reconnaissance that decides what is worth one pass of -**add-game-source** and what is worth writing off. +Written 2026-08-18. **Acted on 2026-08-19**: four of the games below were built, one turned out to be +blocked rather than merely awkward, and the rest were declined. Each section keeps its original +reconnaissance and carries its outcome at the top, because the reasoning is what stops the next pass +re-deriving it. + +What changed on 2026-08-19, in one paragraph. **Built:** Girls' Frontline 2 (`iopwiki`, new parser), +Stella Sora (`stellasorawiki`, new parser), Chaos Zero Nightmare (`game8`, no parser work) and +Umamusume (`game8`, after widening its section and column vocabulary). **Blocked:** Nikke — its +`robots.txt` cannot be read from here at all any more, so the permission this file required before +building could not be obtained. **Declined:** Punishing: Gray Raven, Guardian Tales, Azur Lane and +Aether Gazer, all as proposed. One thing not on the original list also came out of the pass: the +**Infinity Nikki source is stale** and has no reachable replacement — see § 11. Verdicts here are proposals. Once one is acted on — built or declined — the *decision* belongs in `AGENTS.md` § Scraping conduct, whose table exists so a source is not re-litigated every pass and @@ -25,7 +34,11 @@ Two things this method cannot tell you, and both matter: conduct). The only hard evidence we hold is what CI has actually committed: `git log` on `snapshots/` shows `github-actions[bot]` landing **arknights.wiki.gg**, **endfield.wiki.gg** and **bluearchive.wiki** (Miraheze). Fandom sources have never refreshed in CI, because `robots.txt` - itself is challenged from a datacentre address and the gate fails closed. + itself is challenged from a datacentre address and the gate fails closed — and as of 2026-08-19 + that challenge covers every Fandom wiki from *any* address tried here, not just CI's. + Of the 2026-08-19 additions, **Stella Sora** is Miraheze and so is the one likely to refresh in CI; + **IOP Wiki** is permissive but unproven; **Chaos Zero Nightmare** and **Umamusume** are game8 and + therefore blind in CI by construction. - **Whether a page will still look like this in six weeks.** Each recommendation below names the assertion `canParse` should make, so a redesign fails the source loudly instead of emptying a lane. @@ -33,20 +46,22 @@ Two things this method cannot tell you, and both matter: | Game | Status | |---|---| -| Arknights, Reverse: 1999, Blue Archive, Persona 5X | **Built** | -| Azur Lane, Umamusume | Declined on conduct (`azurlane.koumakan.jp` `ai-input=no`; `uma.moe` Turnstile) | -| **Girls' Frontline 2, Nikke, Chaos Zero Nightmare, Stella Sora, Punishing: Gray Raven, Guardian Tales, Aether Gazer** | **No source — assessed below** | +| Arknights, Reverse: 1999, Blue Archive, Persona 5X | **Built** (2026-08-17/18) | +| **Girls' Frontline 2, Stella Sora, Chaos Zero Nightmare, Umamusume** | **Built** (2026-08-19) | +| Nikke | **Blocked.** Best unbuilt data here; `robots.txt` unreadable, so the permission cannot be established — see § 3 | +| Azur Lane | Declined on conduct (`azurlane.koumakan.jp` `ai-input=no`), and both Fandom alternatives are dead archives | +| Punishing: Gray Raven, Guardian Tales, Aether Gazer | Declined — see §§ 6, 7, 9 | | Silver Palace | Unreleased | ## Summary of findings | Game | Best source found | Parser | Refreshes in CI? | Verdict | |---|---|---|---|---| -| Girls' Frontline 2 | `iopwiki.com/wiki/GFL2_Events` | new (`iopwiki`) | likely (permissive robots, unproven) | **Build first** — best data quality of anything here | -| Stella Sora | `stellasora.miraheze.org/wiki/Main_Page` | new (or extend `bawiki`) | yes (Miraheze proven) | **Build** — banners only, and read the main page, not the banner list | -| Nikke | `nikke-…-international.fandom.com/api.php?…page=Event` | new shape | **no** — Fandom robots fails closed | **Build**, eyes open: fixture-backed in CI like r1999 | -| Chaos Zero Nightmare | `game8.co/games/Chaos-Zero-Nightmare/archives/559899` | **existing** `game8` | **no** — game8 202s the runner | **Build** if you want a cheap win; it is a fixture-only lane | -| Umamusume | `game8.co/games/Umamusume-Pretty-Derby/archives/536311` | `game8` + two header words | **no** | Possible, but it edits a parser eight sources share | +| Girls' Frontline 2 | `iopwiki.com/wiki/GFL2_Events` | new (`iopwiki`) | likely (permissive robots, unproven) | **BUILT** 2026-08-19 — best data quality of anything here | +| Stella Sora | `stellasora.miraheze.org/wiki/Main_Page` | new (`stellasorawiki`) | yes (Miraheze proven) | **BUILT** 2026-08-19 — banners only, from the main page | +| Nikke | `nikke-…-international.fandom.com/api.php?…page=Event` | new shape | **no** — Fandom robots fails closed | **BLOCKED** — the robots.txt precondition could not be met from any client here | +| Chaos Zero Nightmare | `game8.co/games/Chaos-Zero-Nightmare/archives/559899` | **existing** `game8` | **no** — game8 202s the runner | **BUILT** 2026-08-19 — no parser work, fixture-only lane | +| Umamusume | `game8.co/games/Umamusume-Pretty-Derby/archives/536311` | `game8` + section and column vocabulary | **no** | **BUILT** 2026-08-19 — cost more than "two header words"; see § 5 | | Punishing: Gray Raven | `grayravens.com/wiki/Events` | — | — | **Decline for now** — one dated string on the whole page | | Guardian Tales | `guardian-tales.fandom.com/wiki/Events` | — | — | **Decline** — wiki stopped dating events in 2025 | | Azur Lane | none | — | — | **Still declined** — the Fandom alternatives are 2021 archives | @@ -55,7 +70,7 @@ Two things this method cannot tell you, and both matter: --- -## 1. Girls' Frontline 2: Exilium — build this one first +## 1. Girls' Frontline 2: Exilium — BUILT 2026-08-19 **Source:** `https://iopwiki.com/wiki/GFL2_Events` (IOP Wiki, the Girls' Frontline universe wiki) @@ -114,7 +129,7 @@ zero unparsed, which is the count check `AGENTS.md` § Silent drops asks for. La --- -## 2. Stella Sora — build the main page, not the banner list +## 2. Stella Sora — BUILT 2026-08-19, from the main page and not the banner list **Source:** `https://stellasora.miraheze.org/` — a Miraheze wiki, so the same call as Blue Archive and hololive Dreams: `/wiki/` is the surface `*` is allowed, `/w/` and `/*?action=` are disallowed. CC @@ -150,7 +165,21 @@ source immediately — full coverage, no template dependency. --- -## 3. Nikke — the best data, on the surface CI cannot reach +## 3. Nikke — blocked on a permission that can no longer be read + +> **Outcome (2026-08-19): not built, and not declined either.** The precondition this section set — +> "read this wiki's `robots.txt` in a browser and confirm it is the same file" — could not be met. +> From this machine that file answers **403** to our fetcher, and a real headless browser gets a +> Cloudflare managed challenge that never resolves. Worse, the same is now true of *every* Fandom +> wiki tried, including `blhx.fandom.com`, which served it `200` on 2026-08-18 and is what this +> section's permission argument was leaning on. Nothing about the data changed — it is still the +> richest schedule of anything unbuilt — but the permission has to be in writing before the adapter +> exists, and it cannot currently be obtained. **Unblocking it is one action by someone on an +> address Fandom serves:** open that URL, confirm the standard file (allows `/api.php?action=`, +> disallows only `Special:`/`User:`/`Template:`/`Help:`), and paste it into this file. Everything +> below then applies unchanged, including the free `resetOffsets` win. + + **Source:** `https://nikke-goddess-of-victory-international.fandom.com/api.php?action=parse&page=Event&prop=text&formatversion=2&format=json` @@ -205,7 +234,7 @@ a moment's care in `games.ts`, `schema.ts` and the fixture directory name. --- -## 4. Chaos Zero Nightmare — the cheapest adapter available, and a ninth blind source +## 4. Chaos Zero Nightmare — BUILT 2026-08-19; the cheapest adapter available, and a ninth blind source **Source:** `https://game8.co/games/Chaos-Zero-Nightmare/archives/559899` ("List of All Events") @@ -246,7 +275,28 @@ is stale (its "latest events" are April–May 2026); `czn.gg` is a WordPress sit --- -## 5. Umamusume — reachable, but it edits a parser eight sources share +## 5. Umamusume — built, and the estimate below was wrong + +> **Outcome (2026-08-19): BUILT**, off the stable `List of All Banners` page as recommended. Two +> corrections to what follows, both worth having before the next Game8 page is assessed: +> +> - **"A few characters" was wrong.** Teaching `COL_TITLE` the word `Banner` and `COL_RANGE` the word +> `Availability` gets you **zero events**. The page's schedule sits under `List of All Banners` → +> `All Current Banners`, which the *section* vocabulary did not match either, and its back +> catalogue sits under `Previous Banners`, which `previous events` does not match — so without +> that exclusion the finished rows would have gone straight onto the calendar. Beyond that, the +> current-banners table lays the Standard and Paid schedules side by side inside one `` +> under a spanning label row, and that row is *plausible* enough to resolve both columns at +> indices no data row has. `readColumnTable` now decides the header by what it produces, trying +> row 1 only when row 0 produced nothing. +> - **The blast radius was measurable, and it was zero.** The worry below — that widening a parser +> eight sources share starts matching tables it currently ignores — is right in principle and was +> settled by measurement rather than by regenerating `.expected.json` until the tests agreed: every +> pinned fixture *and* every live snapshot was parsed before and after, and no existing source's +> output changed by a single event. Do that diff on any future change to `game8.ts`; it costs one +> script and it is the only thing that distinguishes a safe widening from a silent one. + + `uma.moe` stays declined; nothing about the Turnstile gate has changed. Two other surfaces exist. @@ -365,6 +415,32 @@ projections point at late 2026 / early 2027. Nothing to scrape, as `docs/FEEDBAC --- +## 11. Infinity Nikki — not on the original list, and the most urgent thing found + +This game already has an adapter, which is why it was not assessed on 2026-08-18. It should have +been. `game8.co/games/Infinity-Nikki/archives/487445` fetches, parses, and passes every test — and +the page itself says `Last updated on: August 31, 2025`, with **zero** occurrences of the string +`2026`. Seven events parse out of it, five of them with `endsAt: null`, which the app renders as +live-with-unknown-end indefinitely. A year-old event presented as current is precisely the failure +this product exists to prevent, and it arrives through a source that looks perfectly healthy to the +runner: a stale page is not a broken one, so no failure streak, no annotation, no `broken` tier. + +Every replacement was checked on 2026-08-19 and none works: + +| Candidate | Result | +|---|---| +| `infinitynikki.wiki.gg` | 401 — no such wiki (as with Nikke, Umamusume, Blue Archive, R1999) | +| `infinitynikki.miraheze.org` | Exists, serves robots.txt, **abandoned** — front page last edited 11 Feb 2025, `/wiki/Events` returns a permission error | +| `prydwen.gg/infinity-nikki` | 404 — not a game prydwen covers | +| `game8.co/…/archives/487444` (Banners) | Staler still: last updated 20 May 2025 | +| `infinitynikki.fandom.com` | Behind the blanket Fandom 403 (§ 3) | + +So there is nothing to move the lane to. **This is a call for the repository owner**, and both +options cost something: leaving it means a knowingly stale lane carried only by the footer's +freshness disclosure, and retiring it means dropping a `GameId` that is the first segment of every +completion key that game's readers hold, with no server-side recovery. Recorded in `AGENTS.md` +§ Scraping conduct so the next pass does not rediscover it. + ## What every one of these costs, beyond the source Adding a game is never only a `SOURCES` entry: @@ -381,19 +457,19 @@ Adding a game is never only a `SOURCES` entry: ## Recommended order -1. **Girls' Frontline 2** — best dates in the list (explicit UTC, both boundaries exact), permissive - host, live event today. Costs a parser module; buys the strongest lane since wiki.gg. -2. **Stella Sora** — Miraheze, which is the one family we know refreshes in CI. Banners only, and - read the main page's `