From 220e73ff7acb788dc3eb9da370659af0b294ba79 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Wed, 19 Aug 2026 03:51:48 +0200 Subject: [PATCH] fix(game8): decide a column table's header by what it produces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Game8's banner pages lay two schedules side by side inside one under a spanning label row: "Standard Banners | Banner | Rating | Availability | Paid Banners | ...". That row is not merely unhelpful, it is plausible — it contains both column words, so it resolves and puts the range at index 3, which no three-cell data row has. Every row then fails to date and the table yields nothing at all, with no error anywhere. So readColumnTable now falls back to row 1, and only when row 0 produced nothing, which is what keeps every page that parses today parsing identically. Those pages also schedule banners rather than events and head their sections accordingly, so the vocabulary learns "List of All Banners", "All Current Banners" and — the one that matters — "Previous Banners", whose rows are dated identically to the live ones directly above them. This parser serves nine sources, so the widening was measured rather than trusted: every pinned fixture and every live snapshot was parsed before and after, and no existing source's output changed by a single event. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 20 +++++++++-- docs/INGESTION.md | 8 ++++- src/ingest/parsers/game8.ts | 70 ++++++++++++++++++++++++++++++------ test/adapters/game8.test.ts | 72 ++++++++++++++++++++++++++++++++++++- 4 files changed, 154 insertions(+), 16 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 46d9cbb..47eabfb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest theme.ts — dark or light, and what a game hue reads as on each scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 670 tests +test/ 674 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -172,13 +172,27 @@ These come from how gacha games actually schedule things, and they cause most bu - **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on every row, five months apart. Publish the one our readers are on and skip the row that lacks it — a CN date on a Global calendar is a confidently wrong date, not a near miss. -- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value +- **Game8 has no single template.** Eight shapes are known and a page may mix them: label/value detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb - cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `
`-separated date pairs. + cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, `
`-separated date pairs, and + two schedules laid side by side in one `
` under a spanning label row. Full table in `docs/INGESTION.md`. Before assuming a new Game8 page will work, dump its structure and check **every** table — Endfield was written off as undatable on a pass that only inspected its `Duration` rows, and its real events were further down the page. +- **The header row is the row that dates rows, not the first one.** Game8's banner pages put the + Standard and Paid schedules side by side inside one `
` and label the pair + `Standard Banners | Banner | Rating | Availability | Paid Banners | …`. That row is not merely + unhelpful, it is *plausible* — it contains both column words, so it resolves and puts the range at + an index no data row has, and the whole table yields nothing with no error anywhere. So + `readColumnTable` falls back to row 1 **only when row 0 produced nothing**, which is what keeps + every page that parses today parsing identically. Verified rather than assumed: the change was + diffed across all pinned fixtures and every live snapshot, and moved no existing event. +- **Some Game8 wikis schedule banners, not events**, and head their sections accordingly — + `List of All Banners`, `All Current Banners`, and a `Previous Banners` back catalogue that + `previous events` does not match. All three are in the section vocabulary now. The finished rows + sit directly below the live ones and are dated identically, so that exclusion is the only thing + between the calendar and a year of expired banners. - **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona 5 hides fifty finished events behind nothing but an `

Finished Events

` in a collapsed accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but diff --git a/docs/INGESTION.md b/docs/INGESTION.md index f3787e3..cebf8fd 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -231,8 +231,14 @@ several: repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of prose; those corroborate the dates and supply the blurb the flat table lacks. *(Persona 5: The Phantom X)* +8. **Two schedules side by side in one `
`**, under a spanning label row — + `Standard Banners | Banner | Rating | Availability | Paid Banners | Banner | …`, with the real + header on the row below and three-cell data rows under that. The label row is *plausible*: it + contains both column words, resolves, and puts the range at an index no data row has, so the + table yields nothing at all with no error. `readColumnTable` therefore decides the header by what + it produces — row 1 is tried **only when row 0 produced nothing**. *(Umamusume)* -Shapes 1, 2, 4, 5, 6 and 7 are handled. Before assuming a new Game8 page will work, dump its heading/table +Shapes 1, 2, 4, 5, 6, 7 and 8 are handled. Before assuming a new Game8 page will work, dump its heading/table structure and check which shape it uses — and check **every** table, not just the obvious one. Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its two real events were in a table further down. diff --git a/src/ingest/parsers/game8.ts b/src/ingest/parsers/game8.ts index f5f0489..f874579 100644 --- a/src/ingest/parsers/game8.ts +++ b/src/ingest/parsers/game8.ts @@ -45,6 +45,13 @@ const INCLUDED_SECTIONS = [ /list of (all )?events/i, /all available events/i, /ongoing events/i, + // Some Game8 wikis schedule banners rather than events, and head their + // sections accordingly — Umamusume's page is `List of All Banners` → + // `All Current Banners`. Kept as separate patterns rather than making + // "events?" optional above, so "Banner Guides" (a nav table) still matches + // nothing. + /list of (all )?banners/i, + /current banners/i, ]; /** @@ -57,6 +64,12 @@ const EXCLUDED_SECTIONS = [ /previous events/i, /ended events/i, /finished events/i, + // The banner-scheduling pages need their own back-catalogue heading for the + // same reason: Umamusume's finished rows sit under `Previous Banners`, which + // `previous events` does not match, and they are dated exactly like the live + // ones directly above them. + /previous banners/i, + /past banners/i, ]; /** @@ -72,9 +85,9 @@ const END_LABEL = /^(event|test run|banner)?\s*end(\s+date)?$/i; const RANGE_LABEL = /^(availability period|event period|duration|period|dates)$/i; /** Column-table header matchers. */ -const COL_TITLE = /^(.*\b)?events?$/i; +const COL_TITLE = /^(.*\b)?(events?|banners?)$/i; const COL_RANGE = - /^(event |all )?(duration|dates?|event date|period|availability period|schedule)( ?& ?summary| and summary)?$/i; + /^(event |all )?(duration|dates?|event date|period|availability(?: period)?(?: \(utc\))?|schedule)( ?& ?summary| and summary)?$/i; const COL_START = /^start$/i; const COL_END = /^end$/i; const COL_SUMMARY = /^(event )?(details?|description|overview)$/i; @@ -205,18 +218,32 @@ function readLabelledDates( /** Shape 2: one row per event, with a title column and a range column. */ function readColumnTable(rows: string[][]): Candidate[] { - if (rows.length < 2) return []; - const header = rows[0]; - if (header === undefined) return []; + // Usually row 0 heads the table. Where it does not, row 1 does: Game8 lays + // two schedules side by side inside one `
` and gives the pair a + // spanning label row — Umamusume's current banners are headed + // `Standard Banners | Banner | Rating | Availability | Paid Banners | …`, + // with the real header on the row below and three-cell data rows under that. + // + // That label row is not merely unhelpful, it is *plausible*: it contains the + // word "Banners" and the word "Availability", so it resolves both columns and + // puts the range at index 3, which no data row has. Every row then fails to + // date and the table yields nothing at all. So the header is decided by what + // it produces rather than by where it sits — and row 0 still wins whenever it + // produces anything, which is what keeps every page that parses today parsing + // exactly as it did. + const fromFirst = readRowsUnder(rows, 0); + return fromFirst.length > 0 ? fromFirst : readRowsUnder(rows, 1); +} - const titleIdx = header.findIndex((h) => COL_TITLE.test(h)); - const rangeIdx = header.findIndex((h) => COL_RANGE.test(h)); - if (titleIdx === -1 || rangeIdx === -1) return []; - - const summaryIdx = header.findIndex((h) => COL_SUMMARY.test(h)); +/** Read `rows`, treating row `headerIdx` as the header and the rest as data. */ +function readRowsUnder(rows: string[][], headerIdx: number): Candidate[] { + if (rows.length < headerIdx + 2) return []; + const layout = columnLayout(rows[headerIdx]); + if (layout === null) return []; + const { titleIdx, rangeIdx, summaryIdx } = layout; const out: Candidate[] = []; - for (const row of rows.slice(1)) { + for (const row of rows.slice(headerIdx + 1)) { const title = row[titleIdx]?.trim(); const rangeCell = row[rangeIdx]; if (!title || rangeCell === undefined) continue; @@ -241,6 +268,27 @@ function readColumnTable(rows: string[][]): Candidate[] { return out; } +interface ColumnLayout { + titleIdx: number; + rangeIdx: number; + summaryIdx: number; +} + +/** Where the columns sit, if this row is a header row at all. */ +function columnLayout(header: string[] | undefined): ColumnLayout | null { + if (header === undefined) return null; + + const titleIdx = header.findIndex((h) => COL_TITLE.test(h)); + const rangeIdx = header.findIndex((h) => COL_RANGE.test(h)); + if (titleIdx === -1 || rangeIdx === -1) return null; + + return { + titleIdx, + rangeIdx, + summaryIdx: header.findIndex((h) => COL_SUMMARY.test(h)), + }; +} + /** * Try every known range shape, most specific first. `parseOpenRange` is last * because it is the most permissive — it accepts any leading full date and diff --git a/test/adapters/game8.test.ts b/test/adapters/game8.test.ts index a27e35e..a146a97 100644 --- a/test/adapters/game8.test.ts +++ b/test/adapters/game8.test.ts @@ -5,7 +5,7 @@ import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts"; import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts"; import { blueArchiveWikiParser } from "../../src/ingest/parsers/bawiki.ts"; import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts"; -import { inferType } from "../../src/ingest/parsers/game8.ts"; +import { game8Parser, inferType } from "../../src/ingest/parsers/game8.ts"; import { holodoriWikiParser } from "../../src/ingest/parsers/holodori.ts"; import { GachaEvent, type EventType } from "../../src/shared/schema.ts"; @@ -1351,3 +1351,73 @@ describe("Chaos Zero Nightmare (game8)", () => { } }); }); + +describe("game8 column vocabulary", () => { + // The parser directly rather than through an adapter: `canParse` guards the + // seam against a redesigned *page*, and these are hand-built table snippets. + const parse = (html: string) => + game8Parser.parse(html, { + now: NOW, + // Hand-built table snippets, so the context is a label rather than a + // claim about a game — these assert table shape, not any one page. + sourceUrl: "https://game8.co/games/Genshin-Impact/archives/301601", + sourceId: "genshin-game8-events", + game: "genshin", + }); + + test("falls back to the second row when the first is a spanning label", () => { + // Game8 lays two schedules side by side in one
and gives the pair a + // label row. Reading that row as the header finds the range column at an + // index no data row has, so every row fails to date and the table silently + // yields nothing. + const events = parse( + `

List of All Banners

+ + + + +
Standard BannersBannerRatingAvailabilityPaid BannersBannerRatingAvailability
BannerRatingAvailability
Seeking the Pearl★★★★☆8/12/2026 - 8/21/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["Seeking the Pearl"]); + }); + + test("row 0 still wins wherever it resolves both columns", () => { + // The fallback must never let a page that parses today start reading a + // different row. Here row 0 is a real header and row 1 is data. + const events = parse( + `

Current Events

+ + + +
EventDuration
First8/12/2026 - 8/21/2026
Second8/13/2026 - 8/22/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["First", "Second"]); + }); + + test("reads a banner-scheduling page's headings and columns", () => { + const events = parse( + `

All Current Banners

+ + +
BannerAvailability (UTC)
Live One8/12/2026 - 8/21/2026
+

Previous Banners

+ + +
BannerAvailability
Finished One8/1/2026 - 8/9/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["Live One"]); + }); + + test("`Banner Guides` is navigation, not a schedule", () => { + // The widened title column must not turn Game8's nav tables into events. + // This one has no range column at all, so it yields nothing either way — + // the assertion is that widening `Event` to `Banner` did not change that. + const events = parse( + `

Current Events

+ + +
Banner Guides
List of All BannersUpcoming Banners
`, + ); + expect(events).toEqual([]); + }); +});