gacha-event-tracker/docs/INGESTION.md

Ingestion Pipeline

Six stages, run per source. Every stage writes its outcome to ingest_runs so a failure two days ago can be diagnosed without re-running.

fetch → parse → merge → validate → reconcile → gate → publish
                                                  │
                                                  └──► quarantine

No LLM

Event data is extracted by deterministic code only. There is no model call anywhere in this pipeline, no API key, and no per-run cost.

This is a deliberate constraint, not an omission:

• A source that cannot be parsed deterministically does not get an adapter. Report it rather than reaching for inference.
• Parser output is reproducible — the same fixture always yields the same events, which is what makes the fixture tests meaningful.
• Iterating is free and offline: bun run parse <adapter-id> <fixture>.

If a source's markup is too unstable to parse, the answer is a different source, not a model.

Three layers: parsers, adapters, merge

The layering is what makes a second, third, or tenth source cheap.

Layer	Answers	Lives in	Scope
Parser	"How is this site laid out?"	src/ingest/parsers/	One site template, many games
Adapter	"Which URL, for which game, via which parser?"	src/ingest/adapters/index.ts	One page
Merge	"These sources disagree — now what?"	src/ingest/merge.ts	One game, many sources

Consequences worth internalising:

• Adding a source for a site already parsed = one entry in SOURCES. No new parsing code.
• Adding a new site = one parser module + its PARSERS entry, then adapters as above.
• A game may have any number of sources. parseGame(game, documents, now) runs them all and merges.

The parser interface

export interface SourceParser {
  id: string;                                   // "game8"
  label: string;                                // "Game8"
  canParse(html: string): boolean;              // structural sanity check
  parse(html: string, ctx: ParseContext): GachaEvent[];
}

canParse is the redesign tripwire. Without it, a site rewrite makes every selector miss and the parser returns zero events — which reads downstream as "this game has no events" rather than as a failure. The adapter throws when canParse is false, so the run fails loudly and the previously published events stay put.

Keep canParse structural, not content-based, and do not over-fit it. Game8's own pages differ in attribute quote style (class="a-table" on Genshin, class='a-table' on NTE), which is exactly the kind of variation a naive check gets wrong. Every regex in html.ts is attribute-agnostic for the same reason.

The adapter registry

const SOURCES: SourceSpec[] = [
  { id: "genshin-game8-events", game: "genshin",
    url: "https://game8.co/games/Genshin-Impact/archives/301601", parserId: "game8" },
  { id: "nte-game8-events", game: "nte",
    url: "https://game8.co/games/Neverness-to-Everness/archives/592073", parserId: "game8" },
];

priority (default 0) breaks ties when two sources disagree and neither is clearly better — give official feeds a higher number than community wikis. Adapter ids are "<game>-<site>-<page>" and are recorded on every event as sourceId, so any row in the feed traces back to the source that produced it.

Assessing a new source

Source shape	Verdict
JSON API, or an HTML table with consistent headers	Good — write the adapter
Label/value or column tables with full dates including a year	Good — an existing parser may already handle it
Dates without a year, or no end date at all	Unsupportable — yields nothing rather than guessing
Free-form prose with no table structure	Find a different source

Game8 uses at least three page templates and a game's page may use any of them:

1. Label/value detail tables — Event Start / Event End rows under a per-event h3, full dates with year. (Genshin Impact)
2. Column tables — Event | Duration | Event Details | Rewards, one row per event, under a section heading. (Neverness to Everness)
3. Image-grid schedules — a bare MM/DD, no year, no end date. Unsupportable. (Arknights: Endfield)

Shapes 1 and 2 are handled. Before assuming a new Game8 page will work, dump its heading/table structure and check which shape it uses.

Stage 1 — fetch

• Send If-None-Match / If-Modified-Since from sources.etag / last_modified. A 304 ends the run as skipped_unchanged.
• User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker).
• Honor robots.txt; cache parsed robots per host for 24h.
• 20s timeout; retry twice with backoff on 5xx and network errors; never retry 4xx.
• Store raw bytes in snapshots.

On failure: increment consecutive_failures, leave published events untouched, end as failed. A source being down never mutates the feed.

Stage 2 — parse

Hash the raw body (sha256) → content_hash. If it matches sources.content_hash, end as skipped_unchanged and do no further work.

Otherwise call adapter.parse(html, ctx), which runs canParse and then the parser. Because parsers are pure, this stage is fully reproducible offline against the stored snapshot:

bun run parse <adapter-id> fixtures/<game>/<source>-<date>.html

Watch the event count. A source that changes date format or table shape makes events vanish with no error — the parser simply matches nothing. Compare each run's events_seen against the previous run and flag a large drop. A source that went from 13 events to 2 has broken, not quieted down. This is the most likely real failure mode of a parser-only pipeline, and nothing else surfaces it.

Stage 3 — merge

Only meaningful when a game has more than one source; a single-source game passes straight through.

mergeEvents(groups) compares events across sources:

1. Same ID → same event; keep the higher-confidence copy.
2. Near match — same game, title similarity ≥ 0.80, starts within 24h — → same event under different titles; keep the higher-confidence copy.
3. Otherwise → distinct events; keep both.

Title similarity alone would merge a rerun with its original, since reruns reuse the name. The start-date proximity check is the actual guard; the title threshold is deliberately loose (0.80) so that "Stygian Onslaught" and "Stygian Onslaught Event" collapse into one row rather than showing the user a duplicate.

Agreement raises confidence (+0.10) only across different sourceIds. The same row seen twice in one document is not corroboration.

Disagreement is surfaced, never averaged. Two sources whose endsAt differ by more than 24 hours produce a conflicts entry; the pipeline routes those to quarantine. Splitting the difference between two dates would produce a value neither source asserts — the worst possible answer for a product whose promise is date accuracy.

Stage 4 — validate

Zod parse against GachaEvent, then calendar sanity rules. Anything failing a hard rule goes to quarantine with reason: 'sanity_failed' — never to the feed.

Hard rules (reject):

Rule	Rationale
endsAt after startsAt when both present	A backwards interval is always a parse error
Duration under 180 days	Patch cycles are ~6 weeks; longer means a misread year
startsAt within [now − 2y, now + 1y]	Catches century typos and relative-date misreads
endsAt null exactly when endPrecision is "unknown"	The two fields must agree
regionEnds non-null exactly when regionScoped	Same
All regionEnds values within 24h of each other	Region resets differ by hours, not days
title non-empty, ≤ 200 chars, not a placeholder	Catches header rows scraped as events

Rules 1, 4, and 5 are enforced by GachaEvent itself in src/shared/schema.ts, so they cannot be bypassed by constructing an event object directly.

Soft rules (reduce confidence, do not reject):

• Duration under 1 hour or over 60 days → −0.2
• Title very similar to another event in the same batch → −0.15 (likely a duplicate row)

Stage 5 — reconcile

Diff validated candidates against currently published events.

1. Exact ID match → compare fields. Unchanged: no-op. Changed: update.
2. Near match → update the existing event, keeping the existing ID. This is what survives a wiki renaming an event without orphaning every user's completion mark.
3. No match → new event.
4. Published event absent from this run → mark status = 'delisted'. Never delete.

Conflict detection. A candidate moving an already-published endsAt by more than 24 hours is a date_conflict. The user may have planned around the old date, so route it to quarantine regardless of confidence.

Scoring

Confidence records how firmly the sources pinned an event down, so the gate can hold back weak cases. The parser assigns a base score; merge and reconcile adjust it.

base                                          0.95
−0.05  a boundary is day-precision rather than exact
−0.15  the end date is unknown (endsAt null)
+0.10  an independent source corroborates
+0.15  identical event parsed in a previous run
−0.20  any soft rule fired
−0.30  a date_conflict against a published event

Clamp to [0, 1]. CONFIDENCE_THRESHOLD (default 0.8) is the gate. Under the current parser a day-precision event with a known end scores 0.85 and publishes, while one with an unknown end scores 0.75 and is held — the intended bias.

Stage 6 — gate and publish

Condition	Destination
Confidence at or above threshold, no conflict	publish
Confidence below threshold	quarantine, low_confidence
Cross-source or cross-run date disagreement	quarantine, date_conflict
Failed a hard rule	quarantine, sanity_failed
Shape the schema does not recognise	quarantine, novel_shape

A quarantined event does not block its siblings — if eight pass and two are held, the eight publish.

Publish upserts by ID in a transaction. Bump version and updatedAt only when a field actually changed, or the freshness badge (PRD F7) becomes meaningless. Update sources.content_hash, etag, last_success_at, and reset consecutive_failures.

The review gate

Quarantined events surface at GET /review on the admin listener (127.0.0.1:ADMIN_PORT). See docs/ARCHITECTURE.md § Why /review needs no auth.

• POST /api/review/:id/approve — writes to events with extraction_method: 'manual', confidence: 1.0. Approving with edits is supported; the corrected value publishes.
• POST /api/review/:id/reject — stamps resolution only. The candidate is held again next run if the source has not changed, which is intended.

Quarantine depth is the pipeline's health signal. A growing queue means a source changed shape. /api/health exposes the count.

Testing

Every adapter ships:

1. fixtures/<game>/<source>-<YYYY-MM-DD>.html — a real captured page.
2. fixtures/<game>/<source>-<YYYY-MM-DD>.expected.json — the exact GachaEvent[] it produces.
3. A test running parse against the fixture with a pinned ctx.now, asserting deep equality.

bun test must pass with no network access.

Regenerating .expected.json from the parser makes the test self-consistent, not correct. After an intentional change, re-verify a sample against the live page — and ideally extract the same data a second way (a throwaway script over the fixture) to confirm counts and dates independently. That independent check is what caught the exact event counts for both current adapters.

When a source changes shape, capture a new fixture alongside the old one and keep both — the old fixture is the regression test proving the parser still handles the previous format.

test/dates.test.ts covers the cases that matter most: a missing year returns null rather than guessing, impossible calendar dates are rejected, ranges crossing New Year roll the start year back, and abbreviated months parse. test/merge.test.ts covers cross-source agreement, disagreement, and rerun disambiguation. These are the last line of defense before a wrong date reaches a user.