fix(game8): decide a column table's header by what it produces

Game8's banner pages lay two schedules side by side inside one <table> under a
spanning label row: "Standard Banners | Banner | Rating | Availability | Paid
Banners | ...". That row is not merely unhelpful, it is plausible — it contains
both column words, so it resolves and puts the range at index 3, which no
three-cell data row has. Every row then fails to date and the table yields
nothing at all, with no error anywhere. So readColumnTable now falls back to
row 1, and only when row 0 produced nothing, which is what keeps every page
that parses today parsing identically.

Those pages also schedule banners rather than events and head their sections
accordingly, so the vocabulary learns "List of All Banners", "All Current
Banners" and — the one that matters — "Previous Banners", whose rows are dated
identically to the live ones directly above them.

This parser serves nine sources, so the widening was measured rather than
trusted: every pinned fixture and every live snapshot was parsed before and
after, and no existing source's output changed by a single event.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-19 03:51:48 +02:00
co-authored by Claude Opus 5
parent 6c7369db10
commit 220e73ff7a
4 changed files with 154 additions and 16 deletions
+17 -3
View File
@@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest
theme.ts — dark or light, and what a game hue reads as on each theme.ts — dark or light, and what a game hue reads as on each
scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
serve.ts static server + /api/health serve.ts static server + /api/health
test/ 670 tests test/ 674 tests
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
snapshots/ current page per source, rewritten by refresh — see its README snapshots/ current page per source, rewritten by refresh — see its README
``` ```
@@ -172,13 +172,27 @@ These come from how gacha games actually schedule things, and they cause most bu
- **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on - **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on
every row, five months apart. Publish the one our readers are on and skip the row that lacks it — every row, five months apart. Publish the one our readers are on and skip the row that lacks it —
a CN date on a Global calendar is a confidently wrong date, not a near miss. a CN date on a Global calendar is a confidently wrong date, not a near miss.
- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value - **Game8 has no single template.** Eight shapes are known and a page may mix them: label/value
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `<hr>`-separated date pairs. cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, `<hr>`-separated date pairs, and
two schedules laid side by side in one `<table>` under a spanning label row.
Full table in `docs/INGESTION.md`. Before assuming a new Full table in `docs/INGESTION.md`. Before assuming a new
Game8 page will work, dump its structure and check **every** table — Endfield was written off as Game8 page will work, dump its structure and check **every** table — Endfield was written off as
undatable on a pass that only inspected its `Duration` rows, and its real events were further undatable on a pass that only inspected its `Duration` rows, and its real events were further
down the page. down the page.
- **The header row is the row that dates rows, not the first one.** Game8's banner pages put the
Standard and Paid schedules side by side inside one `<table>` and label the pair
`Standard Banners | Banner | Rating | Availability | Paid Banners | …`. That row is not merely
unhelpful, it is *plausible* — it contains both column words, so it resolves and puts the range at
an index no data row has, and the whole table yields nothing with no error anywhere. So
`readColumnTable` falls back to row 1 **only when row 0 produced nothing**, which is what keeps
every page that parses today parsing identically. Verified rather than assumed: the change was
diffed across all pinned fixtures and every live snapshot, and moved no existing event.
- **Some Game8 wikis schedule banners, not events**, and head their sections accordingly —
`List of All Banners`, `All Current Banners`, and a `Previous Banners` back catalogue that
`previous events` does not match. All three are in the section vocabulary now. The finished rows
sit directly below the live ones and are dated identically, so that exclusion is the only thing
between the calendar and a year of expired banners.
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona - **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed 5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but
+7 -1
View File
@@ -231,8 +231,14 @@ several:
repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of
prose; those corroborate the dates and supply the blurb the flat table lacks. prose; those corroborate the dates and supply the blurb the flat table lacks.
*(Persona 5: The Phantom X)* *(Persona 5: The Phantom X)*
8. **Two schedules side by side in one `<table>`**, under a spanning label row —
`Standard Banners | Banner | Rating | Availability | Paid Banners | Banner | …`, with the real
header on the row below and three-cell data rows under that. The label row is *plausible*: it
contains both column words, resolves, and puts the range at an index no data row has, so the
table yields nothing at all with no error. `readColumnTable` therefore decides the header by what
it produces — row 1 is tried **only when row 0 produced nothing**. *(Umamusume)*
Shapes 1, 2, 4, 5, 6 and 7 are handled. Before assuming a new Game8 page will work, dump its heading/table Shapes 1, 2, 4, 5, 6, 7 and 8 are handled. Before assuming a new Game8 page will work, dump its heading/table
structure and check which shape it uses — and check **every** table, not just the obvious one. structure and check which shape it uses — and check **every** table, not just the obvious one.
Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its
two real events were in a table further down. two real events were in a table further down.
+59 -11
View File
@@ -45,6 +45,13 @@ const INCLUDED_SECTIONS = [
/list of (all )?events/i, /list of (all )?events/i,
/all available events/i, /all available events/i,
/ongoing events/i, /ongoing events/i,
// Some Game8 wikis schedule banners rather than events, and head their
// sections accordingly — Umamusume's page is `List of All Banners` →
// `All Current Banners`. Kept as separate patterns rather than making
// "events?" optional above, so "Banner Guides" (a nav table) still matches
// nothing.
/list of (all )?banners/i,
/current banners/i,
]; ];
/** /**
@@ -57,6 +64,12 @@ const EXCLUDED_SECTIONS = [
/previous events/i, /previous events/i,
/ended events/i, /ended events/i,
/finished events/i, /finished events/i,
// The banner-scheduling pages need their own back-catalogue heading for the
// same reason: Umamusume's finished rows sit under `Previous Banners`, which
// `previous events` does not match, and they are dated exactly like the live
// ones directly above them.
/previous banners/i,
/past banners/i,
]; ];
/** /**
@@ -72,9 +85,9 @@ const END_LABEL = /^(event|test run|banner)?\s*end(\s+date)?$/i;
const RANGE_LABEL = /^(availability period|event period|duration|period|dates)$/i; const RANGE_LABEL = /^(availability period|event period|duration|period|dates)$/i;
/** Column-table header matchers. */ /** Column-table header matchers. */
const COL_TITLE = /^(.*\b)?events?$/i; const COL_TITLE = /^(.*\b)?(events?|banners?)$/i;
const COL_RANGE = const COL_RANGE =
/^(event |all )?(duration|dates?|event date|period|availability period|schedule)( ?& ?summary| and summary)?$/i; /^(event |all )?(duration|dates?|event date|period|availability(?: period)?(?: \(utc\))?|schedule)( ?& ?summary| and summary)?$/i;
const COL_START = /^start$/i; const COL_START = /^start$/i;
const COL_END = /^end$/i; const COL_END = /^end$/i;
const COL_SUMMARY = /^(event )?(details?|description|overview)$/i; const COL_SUMMARY = /^(event )?(details?|description|overview)$/i;
@@ -205,18 +218,32 @@ function readLabelledDates(
/** Shape 2: one row per event, with a title column and a range column. */ /** Shape 2: one row per event, with a title column and a range column. */
function readColumnTable(rows: string[][]): Candidate[] { function readColumnTable(rows: string[][]): Candidate[] {
if (rows.length < 2) return []; // Usually row 0 heads the table. Where it does not, row 1 does: Game8 lays
const header = rows[0]; // two schedules side by side inside one `<table>` and gives the pair a
if (header === undefined) return []; // spanning label row — Umamusume's current banners are headed
// `Standard Banners | Banner | Rating | Availability | Paid Banners | …`,
// with the real header on the row below and three-cell data rows under that.
//
// That label row is not merely unhelpful, it is *plausible*: it contains the
// word "Banners" and the word "Availability", so it resolves both columns and
// puts the range at index 3, which no data row has. Every row then fails to
// date and the table yields nothing at all. So the header is decided by what
// it produces rather than by where it sits — and row 0 still wins whenever it
// produces anything, which is what keeps every page that parses today parsing
// exactly as it did.
const fromFirst = readRowsUnder(rows, 0);
return fromFirst.length > 0 ? fromFirst : readRowsUnder(rows, 1);
}
const titleIdx = header.findIndex((h) => COL_TITLE.test(h)); /** Read `rows`, treating row `headerIdx` as the header and the rest as data. */
const rangeIdx = header.findIndex((h) => COL_RANGE.test(h)); function readRowsUnder(rows: string[][], headerIdx: number): Candidate[] {
if (titleIdx === -1 || rangeIdx === -1) return []; if (rows.length < headerIdx + 2) return [];
const layout = columnLayout(rows[headerIdx]);
const summaryIdx = header.findIndex((h) => COL_SUMMARY.test(h)); if (layout === null) return [];
const { titleIdx, rangeIdx, summaryIdx } = layout;
const out: Candidate[] = []; const out: Candidate[] = [];
for (const row of rows.slice(1)) { for (const row of rows.slice(headerIdx + 1)) {
const title = row[titleIdx]?.trim(); const title = row[titleIdx]?.trim();
const rangeCell = row[rangeIdx]; const rangeCell = row[rangeIdx];
if (!title || rangeCell === undefined) continue; if (!title || rangeCell === undefined) continue;
@@ -241,6 +268,27 @@ function readColumnTable(rows: string[][]): Candidate[] {
return out; return out;
} }
interface ColumnLayout {
titleIdx: number;
rangeIdx: number;
summaryIdx: number;
}
/** Where the columns sit, if this row is a header row at all. */
function columnLayout(header: string[] | undefined): ColumnLayout | null {
if (header === undefined) return null;
const titleIdx = header.findIndex((h) => COL_TITLE.test(h));
const rangeIdx = header.findIndex((h) => COL_RANGE.test(h));
if (titleIdx === -1 || rangeIdx === -1) return null;
return {
titleIdx,
rangeIdx,
summaryIdx: header.findIndex((h) => COL_SUMMARY.test(h)),
};
}
/** /**
* Try every known range shape, most specific first. `parseOpenRange` is last * Try every known range shape, most specific first. `parseOpenRange` is last
* because it is the most permissive — it accepts any leading full date and * because it is the most permissive — it accepts any leading full date and
+71 -1
View File
@@ -5,7 +5,7 @@ import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts";
import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts"; import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts";
import { blueArchiveWikiParser } from "../../src/ingest/parsers/bawiki.ts"; import { blueArchiveWikiParser } from "../../src/ingest/parsers/bawiki.ts";
import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts"; import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts";
import { inferType } from "../../src/ingest/parsers/game8.ts"; import { game8Parser, inferType } from "../../src/ingest/parsers/game8.ts";
import { holodoriWikiParser } from "../../src/ingest/parsers/holodori.ts"; import { holodoriWikiParser } from "../../src/ingest/parsers/holodori.ts";
import { GachaEvent, type EventType } from "../../src/shared/schema.ts"; import { GachaEvent, type EventType } from "../../src/shared/schema.ts";
@@ -1351,3 +1351,73 @@ describe("Chaos Zero Nightmare (game8)", () => {
} }
}); });
}); });
describe("game8 column vocabulary", () => {
// The parser directly rather than through an adapter: `canParse` guards the
// seam against a redesigned *page*, and these are hand-built table snippets.
const parse = (html: string) =>
game8Parser.parse(html, {
now: NOW,
// Hand-built table snippets, so the context is a label rather than a
// claim about a game — these assert table shape, not any one page.
sourceUrl: "https://game8.co/games/Genshin-Impact/archives/301601",
sourceId: "genshin-game8-events",
game: "genshin",
});
test("falls back to the second row when the first is a spanning label", () => {
// Game8 lays two schedules side by side in one <table> and gives the pair a
// label row. Reading that row as the header finds the range column at an
// index no data row has, so every row fails to date and the table silently
// yields nothing.
const events = parse(
`<h2>List of All Banners</h2><table>
<tr><th>Standard Banners</th><th>Banner</th><th>Rating</th><th>Availability</th>
<th>Paid Banners</th><th>Banner</th><th>Rating</th><th>Availability</th></tr>
<tr><th>Banner</th><th>Rating</th><th>Availability</th></tr>
<tr><td>Seeking the Pearl</td><td>★★★★☆</td><td>8/12/2026 - 8/21/2026</td></tr>
</table>`,
);
expect(events.map((e) => e.title)).toEqual(["Seeking the Pearl"]);
});
test("row 0 still wins wherever it resolves both columns", () => {
// The fallback must never let a page that parses today start reading a
// different row. Here row 0 is a real header and row 1 is data.
const events = parse(
`<h2>Current Events</h2><table>
<tr><th>Event</th><th>Duration</th></tr>
<tr><td>First</td><td>8/12/2026 - 8/21/2026</td></tr>
<tr><td>Second</td><td>8/13/2026 - 8/22/2026</td></tr>
</table>`,
);
expect(events.map((e) => e.title)).toEqual(["First", "Second"]);
});
test("reads a banner-scheduling page's headings and columns", () => {
const events = parse(
`<h2>All Current Banners</h2><table>
<tr><th>Banner</th><th>Availability (UTC)</th></tr>
<tr><td>Live One</td><td>8/12/2026 - 8/21/2026</td></tr>
</table>
<h3>Previous Banners</h3><table>
<tr><th>Banner</th><th>Availability</th></tr>
<tr><td>Finished One</td><td>8/1/2026 - 8/9/2026</td></tr>
</table>`,
);
expect(events.map((e) => e.title)).toEqual(["Live One"]);
});
test("`Banner Guides` is navigation, not a schedule", () => {
// The widened title column must not turn Game8's nav tables into events.
// This one has no range column at all, so it yields nothing either way —
// the assertion is that widening `Event` to `Banner` did not change that.
const events = parse(
`<h2>Current Events</h2><table>
<tr><th>Banner Guides</th></tr>
<tr><td>List of All Banners</td><td>Upcoming Banners</td></tr>
</table>`,
);
expect(events).toEqual([]);
});
});