feat: add wiki.gg as a second Endfield source

First non-Game8 parser, and the first source that states region-scoped ends:
wiki.gg emits ISO timestamps with one timer per server region, so Endfield
events now carry exact times and separate Asia / Americas-Europe ends. That
gap is up to 13 hours, which is what regionEnds was built for.

Two sources for one game also exercised merge for real. It caught both
overlaps — including "Bedazzling Dawnstar" against Game8's "Bedazzling
Dawnstar Sign-In" — and flagged that the two disagree on the end date by 70
hours rather than averaging them into a date neither states.

Matching an appended qualifier needed a prefix test, not a subset test: a
subset would also fuse "Gold Clash" with "Gold Rush Clash Royale".

Also fixes build-feed picking fixtures by game rather than by source, which
handed the wiki.gg page to the Game8 parser once Endfield had two.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-15 01:17:17 +02:00
co-authored by Claude Opus 5
parent f4b133274a
commit dd596b5a43
10 changed files with 1256 additions and 16 deletions
+7 -2
View File
@@ -82,13 +82,18 @@ date makes you miss content. Given the choice, this ships nothing rather than a
| Wuthering Waves | Game8 | 10 | | Wuthering Waves | Game8 | 10 |
| Zenless Zone Zero | Game8 | 12 | | Zenless Zone Zero | Game8 | 12 |
| Neverness to Everness | Game8 | 13 | | Neverness to Everness | Game8 | 13 |
| Arknights: Endfield | Game8 | 2 | | Arknights: Endfield | Game8 + wiki.gg | 6 |
Game8 uses a different page template for almost every game — label/value detail tables, column Game8 uses a different page template for almost every game — label/value detail tables, column
tables, rowspan Start/End pairs — and four different date formats between them. One parser handles tables, rowspan Start/End pairs — and four different date formats between them. One parser handles
all of it; each game costs a registry entry. all of it; each game costs a registry entry.
Endfield yields only two events because most of its page genuinely has no dates — every `Duration` Endfield is the first game with two sources. wiki.gg publishes machine-readable ISO timestamps with
one timer per server region, so its events carry exact times and per-region ends — it outranks Game8
and wins when they disagree. Merge caught two real disagreements between them, each 70 hours apart on
the end date; those are flagged rather than averaged.
Its Game8 page yields only two events because most of it genuinely has no dates — every `Duration`
row reads "Permanently Available" and its version grid shows `07/16` with no year. The two dated row reads "Permanently Available" and its version grid shows `07/16` with no year. The two dated
events hide in a combined cell (`Period: 08/09/26 - 08/30/26 During the event...`), which is where events hide in a combined cell (`Period: 08/09/26 - 08/30/26 During the event...`), which is where
the `MM/DD/YY` parser earns its place. the `MM/DD/YY` parser earns its place.
@@ -0,0 +1,132 @@
[
{
"id": "endfield:heat-rage-mega-arena:2026-07-30",
"game": "endfield",
"title": "HEAT RAGE! MEGA ARENA!",
"type": "challenge",
"summary": "Challenge Event",
"startsAt": "2026-07-30T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-08-12T20:00:00.000Z",
"endPrecision": "exact",
"regionScoped": true,
"regionEnds": {
"asia": "2026-08-12T20:00:00.000Z",
"america": "2026-08-13T09:00:00.000Z",
"europe": "2026-08-13T09:00:00.000Z"
},
"sourceUrl": "https://endfield.wiki.gg/wiki/HEAT_RAGE!_MEGA_ARENA!",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
},
{
"id": "endfield:monumental-etching-beastly-howl:2026-08-06",
"game": "endfield",
"title": "Monumental Etching: Beastly Howl",
"type": "challenge",
"summary": "Challenge Event",
"startsAt": "2026-08-06T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-08-19T22:00:00.000Z",
"endPrecision": "exact",
"regionScoped": false,
"regionEnds": null,
"sourceUrl": "https://endfield.wiki.gg/wiki/Monumental_Etching:_Beastly_Howl",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
},
{
"id": "endfield:bedazzling-dawnstar:2026-08-09",
"game": "endfield",
"title": "Bedazzling Dawnstar",
"type": "login",
"summary": "Sign-In Event",
"startsAt": "2026-08-09T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-09-01T22:00:00.000Z",
"endPrecision": "exact",
"regionScoped": false,
"regionEnds": null,
"sourceUrl": "https://endfield.wiki.gg/wiki/Bedazzling_Dawnstar",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
},
{
"id": "endfield:like-a-star-streaking-through-the-boundaries:2026-08-09",
"game": "endfield",
"title": "Like a Star Streaking Through the Boundaries",
"type": "other",
"summary": "Narrative Event",
"startsAt": "2026-08-09T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-09-01T22:00:00.000Z",
"endPrecision": "exact",
"regionScoped": false,
"regionEnds": null,
"sourceUrl": "https://endfield.wiki.gg/wiki/Like_a_Star_Streaking_Through_the_Boundaries",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
},
{
"id": "endfield:season-of-delirating:2026-08-09",
"game": "endfield",
"title": "Season of Delirating",
"type": "other",
"summary": "Echoes of War Event",
"startsAt": "2026-08-09T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-09-01T22:00:00.000Z",
"endPrecision": "exact",
"regionScoped": false,
"regionEnds": null,
"sourceUrl": "https://endfield.wiki.gg/wiki/Season_of_Delirating",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
},
{
"id": "endfield:the-rooted-realm:2026-08-09",
"game": "endfield",
"title": "The Rooted Realm",
"type": "other",
"summary": "Fun & Games Event",
"startsAt": "2026-08-09T04:00:00.000Z",
"startPrecision": "exact",
"endsAt": "2026-09-01T22:00:00.000Z",
"endPrecision": "exact",
"regionScoped": false,
"regionEnds": null,
"sourceUrl": "https://endfield.wiki.gg/wiki/The_Rooted_Realm",
"sourceId": "endfield-wikigg-events",
"status": "published",
"confidence": 0.95,
"extractionMethod": "parser",
"version": 1,
"firstSeenAt": "2026-08-14T00:00:00.000Z",
"updatedAt": "2026-08-14T00:00:00.000Z"
}
]
File diff suppressed because one or more lines are too long
+11 -4
View File
@@ -14,13 +14,20 @@ import type { GachaEvent, GameId } from "../src/shared/schema.ts";
const OUT = "public/data/events.v1.json"; const OUT = "public/data/events.v1.json";
/** Newest fixture per adapter. */ /**
* Newest fixture for one *source*, not one game.
*
* A game can have several sources, and fixtures are named `<site>-events-<date>`
* against adapter ids of `<game>-<site>-events`. Globbing by game alone hands
* one site's page to another site's parser.
*/
async function latestFixture(adapterId: string, game: GameId) { async function latestFixture(adapterId: string, game: GameId) {
const glob = new Bun.Glob(`fixtures/${game}/*.html`); const site = adapterId.replace(`${game}-`, "").replace(/-events$/, "");
const files = [...glob.scanSync(".")].sort(); const pattern = `fixtures/${game}/${site}-*.html`;
const files = [...new Bun.Glob(pattern).scanSync(".")].sort();
const file = files.at(-1); const file = files.at(-1);
if (file === undefined) { if (file === undefined) {
throw new Error(`no fixture found for ${adapterId} (fixtures/${game}/*.html)`); throw new Error(`no fixture found for ${adapterId} (${pattern})`);
} }
return { file, html: await Bun.file(file).text() }; return { file, html: await Bun.file(file).text() };
} }
+9
View File
@@ -50,6 +50,15 @@ const SOURCES: SourceSpec[] = [
url: "https://game8.co/games/Arknights-Endfield/archives/535443", url: "https://game8.co/games/Arknights-Endfield/archives/535443",
parserId: "game8", parserId: "game8",
}, },
{
id: "endfield-wikigg-events",
game: "endfield",
url: "https://endfield.wiki.gg/wiki/Event",
parserId: "wikigg",
// Exact timestamps and per-region ends beat Game8's day-precision prose,
// so this source wins when the two disagree.
priority: 10,
},
{ {
id: "nte-game8-events", id: "nte-game8-events",
game: "nte", game: "nte",
+32 -6
View File
@@ -123,12 +123,36 @@ function isSameEvent(
toleranceHours: number, toleranceHours: number,
): boolean { ): boolean {
if (a.id === b.id) return true; if (a.id === b.id) return true;
if (titleSimilarity(a.title, b.title) < titleThreshold) return false;
// Overlap alone misses a source that appends a qualifier: "Bedazzling
// Dawnstar" vs "Bedazzling Dawnstar Sign-In" scores 0.67, well under any
// safe threshold, yet is plainly one event.
const similar =
titleSimilarity(a.title, b.title) >= titleThreshold ||
titleExtends(a.title, b.title);
if (!similar) return false;
// Similar titles are not enough — reruns reuse names. Require the start dates // Similar titles are not enough — reruns reuse names. Require the start dates
// to be close before treating two entries as one event. // to be close before treating two entries as one event.
return hoursBetween(a.startsAt, b.startsAt) <= toleranceHours; return hoursBetween(a.startsAt, b.startsAt) <= toleranceHours;
} }
/**
* True when one title is the other with words appended — the shape a source
* qualifier actually takes ("Bedazzling Dawnstar" → "… Sign-In").
*
* Deliberately a prefix test, not a subset test. Subset matching would also
* fuse "Gold Clash" with "Gold Rush Clash Royale", which are different events.
* Two words is the floor: a one-word title would swallow half the calendar.
*/
export function titleExtends(a: string, b: string): boolean {
const ta = tokenList(a);
const tb = tokenList(b);
const [small, large] = ta.length <= tb.length ? [ta, tb] : [tb, ta];
if (small.length < 2 || small.length === large.length) return false;
return small.every((word, i) => large[i] === word);
}
function findConflict( function findConflict(
a: GachaEvent, a: GachaEvent,
b: GachaEvent, b: GachaEvent,
@@ -163,12 +187,14 @@ export function titleSimilarity(a: string, b: string): number {
return (2 * shared) / (ta.size + tb.size); return (2 * shared) / (ta.size + tb.size);
} }
function tokens(title: string): Set<string> { function tokenList(title: string): string[] {
return new Set( return title
title
.toLowerCase() .toLowerCase()
.replace(/[^a-z0-9\s]/g, " ") .replace(/[^a-z0-9\s]/g, " ")
.split(/\s+/) .split(/\s+/)
.filter((w) => w.length > 0), .filter((w) => w.length > 0);
); }
function tokens(title: string): Set<string> {
return new Set(tokenList(title));
} }
+3 -1
View File
@@ -1,4 +1,5 @@
import { game8Parser } from "./game8.ts"; import { game8Parser } from "./game8.ts";
import { wikiGgParser } from "./wikigg.ts";
import type { SourceParser } from "./types.ts"; import type { SourceParser } from "./types.ts";
/** /**
@@ -6,7 +7,7 @@ import type { SourceParser } from "./types.ts";
* an entry in `adapters/index.ts`; adding a new *site* means a parser module * an entry in `adapters/index.ts`; adding a new *site* means a parser module
* here and one line below. * here and one line below.
*/ */
export const PARSERS: SourceParser[] = [game8Parser]; export const PARSERS: SourceParser[] = [game8Parser, wikiGgParser];
export function parserById(id: string): SourceParser | undefined { export function parserById(id: string): SourceParser | undefined {
return PARSERS.find((p) => p.id === id); return PARSERS.find((p) => p.id === id);
@@ -14,3 +15,4 @@ export function parserById(id: string): SourceParser | undefined {
export type { SourceParser } from "./types.ts"; export type { SourceParser } from "./types.ts";
export { game8Parser } from "./game8.ts"; export { game8Parser } from "./game8.ts";
export { wikiGgParser } from "./wikigg.ts";
+142
View File
@@ -0,0 +1,142 @@
import {
eventId,
Region,
type GachaEvent,
type Precision,
} from "../../shared/schema.ts";
import { text } from "../html.ts";
import type { ParseContext } from "../adapters/types.ts";
import { inferType } from "./game8.ts";
import type { SourceParser } from "./types.ts";
/**
* wiki.gg event pages (MediaWiki with the `mp-event` template).
*
* A markedly better source than a prose wiki: each event carries machine
* readable ISO timestamps, one timer per server region:
*
* <div class="mp-event">
* <div class="mp-event-header">[<span class="mp-event-name">TITLE</span>]
* <span class="mp-event-type">Challenge Event</span></div>
* <div class="mp-event-image"><a href="/wiki/SLUG" …></div>
* <ul class="mp-event-timers">
* <li data-start="…Z" data-end="…Z"><span class="mp-server-name">Asia:</span>…
*
* This is the first source that states region-scoped ends, which is what
* `regionScoped` / `regionEnds` exist for — the difference is up to 13 hours.
*/
const EVENT_BLOCK = /<div class="mp-event">([\s\S]*?)<ul class="mp-event-timers">([\s\S]*?)<\/ul>/gi;
const NAME = /<span class="mp-event-name">([\s\S]*?)<\/span>/i;
const TYPE = /<span class="mp-event-type">([\s\S]*?)<\/span>/i;
const LINK = /<a href="(\/wiki\/[^"]+)"/i;
const TIMER =
/<li[^>]*class="[^"]*mp-event-timer[^"]*"[^>]*data-start="([^"]+)"[^>]*data-end="([^"]+)"[\s\S]*?<span class="[^"]*mp-server-name[^"]*">([\s\S]*?)<\/span>/gi;
/**
* Server labels vary in wording ("Americas / Europe:", "Asia:"), so match on
* the words present rather than expecting one label per region. A single timer
* can legitimately cover two regions.
*/
function regionsFor(label: string): Region[] {
const l = label.toLowerCase();
const out: Region[] = [];
if (/asia/.test(l)) out.push("asia");
if (/america/.test(l)) out.push("america");
if (/europe|eu\b/.test(l)) out.push("europe");
return out;
}
interface Timer {
regions: Region[];
start: number;
end: number;
}
export function parseWikiGgEventsPage(
html: string,
ctx: ParseContext,
): GachaEvent[] {
const flat = html.replace(/\s+/g, " ");
const out: GachaEvent[] = [];
for (const block of flat.matchAll(EVENT_BLOCK)) {
const head = block[1] ?? "";
const timersHtml = block[2] ?? "";
const title = text(NAME.exec(head)?.[1] ?? "");
if (title.length === 0) continue;
const timers: Timer[] = [];
for (const t of timersHtml.matchAll(TIMER)) {
const start = Date.parse(t[1] ?? "");
const end = Date.parse(t[2] ?? "");
if (Number.isNaN(start) || Number.isNaN(end) || end <= start) continue;
timers.push({ regions: regionsFor(text(t[3] ?? "")), start, end });
}
if (timers.length === 0) continue;
const regionEnds: Partial<Record<Region, string>> = {};
for (const timer of timers) {
for (const region of timer.regions) {
regionEnds[region] = new Date(timer.end).toISOString();
}
}
const covered = Object.keys(regionEnds).length;
// Region-scoped only when the source actually distinguishes regions and
// they genuinely differ. One timer for everyone is a global instant.
const distinctEnds = new Set(Object.values(regionEnds)).size;
const regionScoped = covered > 0 && distinctEnds > 1;
// Canonical start is the earliest any region sees it; the canonical end is
// the earliest any region loses it. `endsAt` is only a fallback for a
// region the source did not list, so the conservative choice is right.
const startsAt = new Date(
Math.min(...timers.map((t) => t.start)),
).toISOString();
const endsAt = new Date(Math.min(...timers.map((t) => t.end))).toISOString();
const href = LINK.exec(block[0] ?? "")?.[1];
const typeLabel = text(TYPE.exec(head)?.[1] ?? "");
const precision: Precision = "exact";
out.push({
id: eventId(ctx.game, title, startsAt),
game: ctx.game,
title,
type: inferType(`${title} ${typeLabel}`),
summary: typeLabel.length > 0 ? typeLabel : null,
startsAt,
startPrecision: precision,
endsAt,
endPrecision: precision,
regionScoped,
regionEnds: regionScoped ? (regionEnds as Record<Region, string>) : null,
sourceUrl: href === undefined ? ctx.sourceUrl : new URL(href, ctx.sourceUrl).toString(),
sourceId: ctx.sourceId,
status: "published",
// Exact timestamps straight from the source, no inference anywhere.
confidence: 0.95,
extractionMethod: "parser",
version: 1,
firstSeenAt: ctx.now,
updatedAt: ctx.now,
});
}
return out.sort((a, b) =>
a.startsAt === b.startsAt
? a.id.localeCompare(b.id)
: a.startsAt.localeCompare(b.startsAt),
);
}
export const wikiGgParser: SourceParser = {
id: "wikigg",
label: "wiki.gg",
canParse(html: string): boolean {
return /class="mp-event"/.test(html) && /mp-event-timer/.test(html);
},
parse: parseWikiGgEventsPage,
};
+32
View File
@@ -26,6 +26,7 @@ const CASES: Array<{ adapter: Adapter; fixture: string }> = [
{ adapter: adapter("wuwa-game8-events"), fixture: "fixtures/wuwa/game8-events-2026-08-14" }, { adapter: adapter("wuwa-game8-events"), fixture: "fixtures/wuwa/game8-events-2026-08-14" },
{ adapter: adapter("zzz-game8-events"), fixture: "fixtures/zzz/game8-events-2026-08-14" }, { adapter: adapter("zzz-game8-events"), fixture: "fixtures/zzz/game8-events-2026-08-14" },
{ adapter: adapter("endfield-game8-events"), fixture: "fixtures/endfield/game8-events-2026-08-14" }, { adapter: adapter("endfield-game8-events"), fixture: "fixtures/endfield/game8-events-2026-08-14" },
{ adapter: adapter("endfield-wikigg-events"), fixture: "fixtures/endfield/wikigg-events-2026-08-14" },
]; ];
async function runAdapter(adapter: Adapter, fixture: string) { async function runAdapter(adapter: Adapter, fixture: string) {
@@ -224,3 +225,34 @@ describe("endfield", () => {
expect(rooted?.summary).not.toMatch(/^Period:/); expect(rooted?.summary).not.toMatch(/^Period:/);
}); });
}); });
describe("wiki.gg parser", () => {
test("reads exact per-region timers", async () => {
// The first source that states region-scoped ends. Asia and the Americas
// differ by hours, which is precisely what regionEnds exists to carry.
const events = await runAdapter(
adapter("endfield-wikigg-events"),
"fixtures/endfield/wikigg-events-2026-08-14",
);
expect(events).toHaveLength(6);
const heat = events.find((e) => e.title === "HEAT RAGE! MEGA ARENA!");
expect(heat?.startPrecision).toBe("exact");
expect(heat?.regionScoped).toBe(true);
expect(heat?.regionEnds?.asia).toBe("2026-08-12T20:00:00.000Z");
expect(heat?.regionEnds?.america).toBe("2026-08-13T09:00:00.000Z");
// endsAt is the fallback for a region the source did not list, so it takes
// the earliest — never promise more time than some region actually gets.
expect(heat?.endsAt).toBe("2026-08-12T20:00:00.000Z");
});
test("links each event to its own wiki page", async () => {
const events = await runAdapter(
adapter("endfield-wikigg-events"),
"fixtures/endfield/wikigg-events-2026-08-14",
);
for (const e of events) {
expect(e.sourceUrl).toStartWith("https://endfield.wiki.gg/wiki/");
}
});
});
+58 -1
View File
@@ -1,5 +1,9 @@
import { describe, expect, test } from "bun:test"; import { describe, expect, test } from "bun:test";
import { mergeEvents, titleSimilarity } from "../src/ingest/merge.ts"; import {
mergeEvents,
titleExtends,
titleSimilarity,
} from "../src/ingest/merge.ts";
import type { GachaEvent } from "../src/shared/schema.ts"; import type { GachaEvent } from "../src/shared/schema.ts";
const NOW = "2026-08-14T00:00:00.000Z"; const NOW = "2026-08-14T00:00:00.000Z";
@@ -134,3 +138,56 @@ describe("titleSimilarity", () => {
expect(titleSimilarity("Gold Clash", "Fishing Frenzy")).toBeLessThan(0.3); expect(titleSimilarity("Gold Clash", "Fishing Frenzy")).toBeLessThan(0.3);
}); });
}); });
describe("titleExtends", () => {
test("matches a title with an appended qualifier", () => {
// Real case: Game8 says "Bedazzling Dawnstar Sign-In", wiki.gg says
// "Bedazzling Dawnstar". Token overlap scores 0.67 — under any safe
// threshold — but it is plainly one event.
expect(titleExtends("Bedazzling Dawnstar", "Bedazzling Dawnstar Sign-In")).toBe(true);
});
test("refuses a single shared word", () => {
// A one-word title would otherwise swallow everything containing it.
expect(titleExtends("Rerun", "Overflowing Abundance Rerun")).toBe(false);
});
test("refuses a shared-word subset that is not a prefix", () => {
// Subset matching would fuse these; they are different events.
expect(titleExtends("Gold Clash", "Gold Rush Clash Royale")).toBe(false);
});
test("refuses two titles of equal length", () => {
expect(titleExtends("Gold Clash", "Gold Rush")).toBe(false);
});
});
describe("merging two sources for one game", () => {
test("collapses an appended-qualifier duplicate and flags the date gap", () => {
const g8 = event({
id: "endfield:bedazzling-dawnstar-sign-in:2026-08-09",
game: "endfield",
title: "Bedazzling Dawnstar Sign-In",
startsAt: "2026-08-09T00:00:00.000Z",
endsAt: "2026-08-30T00:00:00.000Z",
sourceId: "endfield-game8-events",
confidence: 0.85,
});
const wiki = event({
id: "endfield:bedazzling-dawnstar:2026-08-09",
game: "endfield",
title: "Bedazzling Dawnstar",
startsAt: "2026-08-09T04:00:00.000Z",
endsAt: "2026-09-01T22:00:00.000Z",
sourceId: "endfield-wikigg-events",
confidence: 0.95,
});
const { events, conflicts } = mergeEvents([[g8], [wiki]]);
expect(events).toHaveLength(1);
// The better-sourced copy wins, and the disagreement is surfaced rather
// than averaged into a date neither source states.
expect(events[0]?.sourceId).toBe("endfield-wikigg-events");
expect(conflicts).toHaveLength(1);
expect(conflicts[0]?.field).toBe("endsAt");
});
});