diff --git a/AGENTS.md b/AGENTS.md index 46d9cbb..47eabfb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest theme.ts — dark or light, and what a game hue reads as on each scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 670 tests +test/ 674 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -172,13 +172,27 @@ These come from how gacha games actually schedule things, and they cause most bu - **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on every row, five months apart. Publish the one our readers are on and skip the row that lacks it — a CN date on a Global calendar is a confidently wrong date, not a near miss. -- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value +- **Game8 has no single template.** Eight shapes are known and a page may mix them: label/value detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb - cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `
`-separated date pairs. + cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, `
`-separated date pairs, and + two schedules laid side by side in one `` under a spanning label row. Full table in `docs/INGESTION.md`. Before assuming a new Game8 page will work, dump its structure and check **every** table — Endfield was written off as undatable on a pass that only inspected its `Duration` rows, and its real events were further down the page. +- **The header row is the row that dates rows, not the first one.** Game8's banner pages put the + Standard and Paid schedules side by side inside one `
` and label the pair + `Standard Banners | Banner | Rating | Availability | Paid Banners | …`. That row is not merely + unhelpful, it is *plausible* — it contains both column words, so it resolves and puts the range at + an index no data row has, and the whole table yields nothing with no error anywhere. So + `readColumnTable` falls back to row 1 **only when row 0 produced nothing**, which is what keeps + every page that parses today parsing identically. Verified rather than assumed: the change was + diffed across all pinned fixtures and every live snapshot, and moved no existing event. +- **Some Game8 wikis schedule banners, not events**, and head their sections accordingly — + `List of All Banners`, `All Current Banners`, and a `Previous Banners` back catalogue that + `previous events` does not match. All three are in the section vocabulary now. The finished rows + sit directly below the live ones and are dated identically, so that exclusion is the only thing + between the calendar and a year of expired banners. - **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona 5 hides fifty finished events behind nothing but an `

Finished Events

` in a collapsed accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but diff --git a/docs/INGESTION.md b/docs/INGESTION.md index f3787e3..cebf8fd 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -231,8 +231,14 @@ several: repeats each live event under its own `h3` with `Start Date` / `End Date` rows and a paragraph of prose; those corroborate the dates and supply the blurb the flat table lacks. *(Persona 5: The Phantom X)* +8. **Two schedules side by side in one `
`**, under a spanning label row — + `Standard Banners | Banner | Rating | Availability | Paid Banners | Banner | …`, with the real + header on the row below and three-cell data rows under that. The label row is *plausible*: it + contains both column words, resolves, and puts the range at an index no data row has, so the + table yields nothing at all with no error. `readColumnTable` therefore decides the header by what + it produces — row 1 is tried **only when row 0 produced nothing**. *(Umamusume)* -Shapes 1, 2, 4, 5, 6 and 7 are handled. Before assuming a new Game8 page will work, dump its heading/table +Shapes 1, 2, 4, 5, 6, 7 and 8 are handled. Before assuming a new Game8 page will work, dump its heading/table structure and check which shape it uses — and check **every** table, not just the obvious one. Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its two real events were in a table further down. diff --git a/src/ingest/parsers/game8.ts b/src/ingest/parsers/game8.ts index f5f0489..f874579 100644 --- a/src/ingest/parsers/game8.ts +++ b/src/ingest/parsers/game8.ts @@ -45,6 +45,13 @@ const INCLUDED_SECTIONS = [ /list of (all )?events/i, /all available events/i, /ongoing events/i, + // Some Game8 wikis schedule banners rather than events, and head their + // sections accordingly — Umamusume's page is `List of All Banners` → + // `All Current Banners`. Kept as separate patterns rather than making + // "events?" optional above, so "Banner Guides" (a nav table) still matches + // nothing. + /list of (all )?banners/i, + /current banners/i, ]; /** @@ -57,6 +64,12 @@ const EXCLUDED_SECTIONS = [ /previous events/i, /ended events/i, /finished events/i, + // The banner-scheduling pages need their own back-catalogue heading for the + // same reason: Umamusume's finished rows sit under `Previous Banners`, which + // `previous events` does not match, and they are dated exactly like the live + // ones directly above them. + /previous banners/i, + /past banners/i, ]; /** @@ -72,9 +85,9 @@ const END_LABEL = /^(event|test run|banner)?\s*end(\s+date)?$/i; const RANGE_LABEL = /^(availability period|event period|duration|period|dates)$/i; /** Column-table header matchers. */ -const COL_TITLE = /^(.*\b)?events?$/i; +const COL_TITLE = /^(.*\b)?(events?|banners?)$/i; const COL_RANGE = - /^(event |all )?(duration|dates?|event date|period|availability period|schedule)( ?& ?summary| and summary)?$/i; + /^(event |all )?(duration|dates?|event date|period|availability(?: period)?(?: \(utc\))?|schedule)( ?& ?summary| and summary)?$/i; const COL_START = /^start$/i; const COL_END = /^end$/i; const COL_SUMMARY = /^(event )?(details?|description|overview)$/i; @@ -205,18 +218,32 @@ function readLabelledDates( /** Shape 2: one row per event, with a title column and a range column. */ function readColumnTable(rows: string[][]): Candidate[] { - if (rows.length < 2) return []; - const header = rows[0]; - if (header === undefined) return []; + // Usually row 0 heads the table. Where it does not, row 1 does: Game8 lays + // two schedules side by side inside one `
` and gives the pair a + // spanning label row — Umamusume's current banners are headed + // `Standard Banners | Banner | Rating | Availability | Paid Banners | …`, + // with the real header on the row below and three-cell data rows under that. + // + // That label row is not merely unhelpful, it is *plausible*: it contains the + // word "Banners" and the word "Availability", so it resolves both columns and + // puts the range at index 3, which no data row has. Every row then fails to + // date and the table yields nothing at all. So the header is decided by what + // it produces rather than by where it sits — and row 0 still wins whenever it + // produces anything, which is what keeps every page that parses today parsing + // exactly as it did. + const fromFirst = readRowsUnder(rows, 0); + return fromFirst.length > 0 ? fromFirst : readRowsUnder(rows, 1); +} - const titleIdx = header.findIndex((h) => COL_TITLE.test(h)); - const rangeIdx = header.findIndex((h) => COL_RANGE.test(h)); - if (titleIdx === -1 || rangeIdx === -1) return []; - - const summaryIdx = header.findIndex((h) => COL_SUMMARY.test(h)); +/** Read `rows`, treating row `headerIdx` as the header and the rest as data. */ +function readRowsUnder(rows: string[][], headerIdx: number): Candidate[] { + if (rows.length < headerIdx + 2) return []; + const layout = columnLayout(rows[headerIdx]); + if (layout === null) return []; + const { titleIdx, rangeIdx, summaryIdx } = layout; const out: Candidate[] = []; - for (const row of rows.slice(1)) { + for (const row of rows.slice(headerIdx + 1)) { const title = row[titleIdx]?.trim(); const rangeCell = row[rangeIdx]; if (!title || rangeCell === undefined) continue; @@ -241,6 +268,27 @@ function readColumnTable(rows: string[][]): Candidate[] { return out; } +interface ColumnLayout { + titleIdx: number; + rangeIdx: number; + summaryIdx: number; +} + +/** Where the columns sit, if this row is a header row at all. */ +function columnLayout(header: string[] | undefined): ColumnLayout | null { + if (header === undefined) return null; + + const titleIdx = header.findIndex((h) => COL_TITLE.test(h)); + const rangeIdx = header.findIndex((h) => COL_RANGE.test(h)); + if (titleIdx === -1 || rangeIdx === -1) return null; + + return { + titleIdx, + rangeIdx, + summaryIdx: header.findIndex((h) => COL_SUMMARY.test(h)), + }; +} + /** * Try every known range shape, most specific first. `parseOpenRange` is last * because it is the most permissive — it accepts any leading full date and diff --git a/test/adapters/game8.test.ts b/test/adapters/game8.test.ts index a27e35e..a146a97 100644 --- a/test/adapters/game8.test.ts +++ b/test/adapters/game8.test.ts @@ -5,7 +5,7 @@ import { parseOrdinalDateTimeRange } from "../../src/ingest/dates.ts"; import { arknightsWikiParser } from "../../src/ingest/parsers/akwiki.ts"; import { blueArchiveWikiParser } from "../../src/ingest/parsers/bawiki.ts"; import { fandomParser, renderedHtml } from "../../src/ingest/parsers/fandom.ts"; -import { inferType } from "../../src/ingest/parsers/game8.ts"; +import { game8Parser, inferType } from "../../src/ingest/parsers/game8.ts"; import { holodoriWikiParser } from "../../src/ingest/parsers/holodori.ts"; import { GachaEvent, type EventType } from "../../src/shared/schema.ts"; @@ -1351,3 +1351,73 @@ describe("Chaos Zero Nightmare (game8)", () => { } }); }); + +describe("game8 column vocabulary", () => { + // The parser directly rather than through an adapter: `canParse` guards the + // seam against a redesigned *page*, and these are hand-built table snippets. + const parse = (html: string) => + game8Parser.parse(html, { + now: NOW, + // Hand-built table snippets, so the context is a label rather than a + // claim about a game — these assert table shape, not any one page. + sourceUrl: "https://game8.co/games/Genshin-Impact/archives/301601", + sourceId: "genshin-game8-events", + game: "genshin", + }); + + test("falls back to the second row when the first is a spanning label", () => { + // Game8 lays two schedules side by side in one
and gives the pair a + // label row. Reading that row as the header finds the range column at an + // index no data row has, so every row fails to date and the table silently + // yields nothing. + const events = parse( + `

List of All Banners

+ + + + +
Standard BannersBannerRatingAvailabilityPaid BannersBannerRatingAvailability
BannerRatingAvailability
Seeking the Pearl★★★★☆8/12/2026 - 8/21/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["Seeking the Pearl"]); + }); + + test("row 0 still wins wherever it resolves both columns", () => { + // The fallback must never let a page that parses today start reading a + // different row. Here row 0 is a real header and row 1 is data. + const events = parse( + `

Current Events

+ + + +
EventDuration
First8/12/2026 - 8/21/2026
Second8/13/2026 - 8/22/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["First", "Second"]); + }); + + test("reads a banner-scheduling page's headings and columns", () => { + const events = parse( + `

All Current Banners

+ + +
BannerAvailability (UTC)
Live One8/12/2026 - 8/21/2026
+

Previous Banners

+ + +
BannerAvailability
Finished One8/1/2026 - 8/9/2026
`, + ); + expect(events.map((e) => e.title)).toEqual(["Live One"]); + }); + + test("`Banner Guides` is navigation, not a schedule", () => { + // The widened title column must not turn Game8's nav tables into events. + // This one has no range column at all, so it yields nothing either way — + // the assertion is that widening `Event` to `Banner` did not change that. + const events = parse( + `

Current Events

+ + +
Banner Guides
List of All BannersUpcoming Banners
`, + ); + expect(events).toEqual([]); + }); +});