Lucas WintherandClaude Opus 5
3ea7286f56
Event data is parsed deterministically; there is no model call, API key or per-run cost anywhere in the pipeline. A source that cannot be parsed deterministically gets no adapter rather than an inference fallback. Documents the parser/adapter/merge split, records that Game8 uses three page templates, and adds Neverness to Everness. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
264 lines
12 KiB
Markdown
264 lines
12 KiB
Markdown
# Ingestion Pipeline
|
||
|
||
Six stages, run per source. Every stage writes its outcome to `ingest_runs` so a failure two days
|
||
ago can be diagnosed without re-running.
|
||
|
||
```
|
||
fetch → parse → merge → validate → reconcile → gate → publish
|
||
│
|
||
└──► quarantine
|
||
```
|
||
|
||
## No LLM
|
||
|
||
Event data is extracted by deterministic code only. There is no model call anywhere in this
|
||
pipeline, no API key, and no per-run cost.
|
||
|
||
This is a deliberate constraint, not an omission:
|
||
|
||
- A source that cannot be parsed deterministically **does not get an adapter.** Report it rather
|
||
than reaching for inference.
|
||
- Parser output is reproducible — the same fixture always yields the same events, which is what
|
||
makes the fixture tests meaningful.
|
||
- Iterating is free and offline: `bun run parse <adapter-id> <fixture>`.
|
||
|
||
If a source's markup is too unstable to parse, the answer is a different source, not a model.
|
||
|
||
## Three layers: parsers, adapters, merge
|
||
|
||
The layering is what makes a second, third, or tenth source cheap.
|
||
|
||
| Layer | Answers | Lives in | Scope |
|
||
|---|---|---|---|
|
||
| **Parser** | "How is this *site* laid out?" | `src/ingest/parsers/` | One site template, many games |
|
||
| **Adapter** | "Which URL, for which game, via which parser?" | `src/ingest/adapters/index.ts` | One page |
|
||
| **Merge** | "These sources disagree — now what?" | `src/ingest/merge.ts` | One game, many sources |
|
||
|
||
Consequences worth internalising:
|
||
|
||
- Adding a source for a site already parsed = **one entry in `SOURCES`**. No new parsing code.
|
||
- Adding a new *site* = one parser module + its `PARSERS` entry, then adapters as above.
|
||
- A game may have any number of sources. `parseGame(game, documents, now)` runs them all and
|
||
merges.
|
||
|
||
### The parser interface
|
||
|
||
```ts
|
||
export interface SourceParser {
|
||
id: string; // "game8"
|
||
label: string; // "Game8"
|
||
canParse(html: string): boolean; // structural sanity check
|
||
parse(html: string, ctx: ParseContext): GachaEvent[];
|
||
}
|
||
```
|
||
|
||
`canParse` is the redesign tripwire. Without it, a site rewrite makes every selector miss and the
|
||
parser returns zero events — which reads downstream as "this game has no events" rather than as a
|
||
failure. The adapter throws when `canParse` is false, so the run fails loudly and the previously
|
||
published events stay put.
|
||
|
||
Keep `canParse` structural, not content-based, and **do not over-fit it**. Game8's own pages differ
|
||
in attribute quote style (`class="a-table"` on Genshin, `class='a-table'` on NTE), which is exactly
|
||
the kind of variation a naive check gets wrong. Every regex in `html.ts` is attribute-agnostic for
|
||
the same reason.
|
||
|
||
### The adapter registry
|
||
|
||
```ts
|
||
const SOURCES: SourceSpec[] = [
|
||
{ id: "genshin-game8-events", game: "genshin",
|
||
url: "https://game8.co/games/Genshin-Impact/archives/301601", parserId: "game8" },
|
||
{ id: "nte-game8-events", game: "nte",
|
||
url: "https://game8.co/games/Neverness-to-Everness/archives/592073", parserId: "game8" },
|
||
];
|
||
```
|
||
|
||
`priority` (default 0) breaks ties when two sources disagree and neither is clearly better — give
|
||
official feeds a higher number than community wikis. Adapter ids are `"<game>-<site>-<page>"` and
|
||
are recorded on every event as `sourceId`, so any row in the feed traces back to the source that
|
||
produced it.
|
||
|
||
### Assessing a new source
|
||
|
||
| Source shape | Verdict |
|
||
|---|---|
|
||
| JSON API, or an HTML table with consistent headers | Good — write the adapter |
|
||
| Label/value or column tables with full dates including a year | Good — an existing parser may already handle it |
|
||
| Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing |
|
||
| Free-form prose with no table structure | Find a different source |
|
||
|
||
Game8 uses at least three page templates and a game's page may use any of them:
|
||
|
||
1. **Label/value detail tables** — `Event Start` / `Event End` rows under a per-event `h3`, full
|
||
dates with year. *(Genshin Impact)*
|
||
2. **Column tables** — `Event | Duration | Event Details | Rewards`, one row per event, under a
|
||
section heading. *(Neverness to Everness)*
|
||
3. **Image-grid schedules** — a bare `MM/DD`, no year, no end date. **Unsupportable.**
|
||
*(Arknights: Endfield)*
|
||
|
||
Shapes 1 and 2 are handled. Before assuming a new Game8 page will work, dump its heading/table
|
||
structure and check which shape it uses.
|
||
|
||
## Stage 1 — fetch
|
||
|
||
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
|
||
the run as `skipped_unchanged`.
|
||
- `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
|
||
- Honor `robots.txt`; cache parsed robots per host for 24h.
|
||
- 20s timeout; retry twice with backoff on 5xx and network errors; never retry 4xx.
|
||
- Store raw bytes in `snapshots`.
|
||
|
||
On failure: increment `consecutive_failures`, leave published events untouched, end as `failed`. A
|
||
source being down never mutates the feed.
|
||
|
||
## Stage 2 — parse
|
||
|
||
Hash the raw body (sha256) → `content_hash`. **If it matches `sources.content_hash`, end as
|
||
`skipped_unchanged`** and do no further work.
|
||
|
||
Otherwise call `adapter.parse(html, ctx)`, which runs `canParse` and then the parser. Because
|
||
parsers are pure, this stage is fully reproducible offline against the stored snapshot:
|
||
|
||
```
|
||
bun run parse <adapter-id> fixtures/<game>/<source>-<date>.html
|
||
```
|
||
|
||
**Watch the event count.** A source that changes date format or table shape makes events vanish with
|
||
no error — the parser simply matches nothing. Compare each run's `events_seen` against the previous
|
||
run and flag a large drop. A source that went from 13 events to 2 has broken, not quieted down.
|
||
This is the most likely real failure mode of a parser-only pipeline, and nothing else surfaces it.
|
||
|
||
## Stage 3 — merge
|
||
|
||
Only meaningful when a game has more than one source; a single-source game passes straight through.
|
||
|
||
`mergeEvents(groups)` compares events across sources:
|
||
|
||
1. **Same ID** → same event; keep the higher-confidence copy.
|
||
2. **Near match** — same game, title similarity ≥ 0.80, starts within 24h — → same event under
|
||
different titles; keep the higher-confidence copy.
|
||
3. **Otherwise** → distinct events; keep both.
|
||
|
||
Title similarity alone would merge a rerun with its original, since reruns reuse the name. The
|
||
start-date proximity check is the actual guard; the title threshold is deliberately loose (0.80) so
|
||
that "Stygian Onslaught" and "Stygian Onslaught Event" collapse into one row rather than showing
|
||
the user a duplicate.
|
||
|
||
**Agreement raises confidence (+0.10) only across different `sourceId`s.** The same row seen twice
|
||
in one document is not corroboration.
|
||
|
||
**Disagreement is surfaced, never averaged.** Two sources whose `endsAt` differ by more than 24
|
||
hours produce a `conflicts` entry; the pipeline routes those to quarantine. Splitting the difference
|
||
between two dates would produce a value neither source asserts — the worst possible answer for a
|
||
product whose promise is date accuracy.
|
||
|
||
## Stage 4 — validate
|
||
|
||
Zod parse against `GachaEvent`, then calendar sanity rules. Anything failing a hard rule goes to
|
||
quarantine with `reason: 'sanity_failed'` — never to the feed.
|
||
|
||
**Hard rules (reject):**
|
||
|
||
| Rule | Rationale |
|
||
|---|---|
|
||
| `endsAt` after `startsAt` when both present | A backwards interval is always a parse error |
|
||
| Duration under 180 days | Patch cycles are ~6 weeks; longer means a misread year |
|
||
| `startsAt` within [now − 2y, now + 1y] | Catches century typos and relative-date misreads |
|
||
| `endsAt` null exactly when `endPrecision` is `"unknown"` | The two fields must agree |
|
||
| `regionEnds` non-null exactly when `regionScoped` | Same |
|
||
| All `regionEnds` values within 24h of each other | Region resets differ by hours, not days |
|
||
| `title` non-empty, ≤ 200 chars, not a placeholder | Catches header rows scraped as events |
|
||
|
||
Rules 1, 4, and 5 are enforced by `GachaEvent` itself in `src/shared/schema.ts`, so they cannot be
|
||
bypassed by constructing an event object directly.
|
||
|
||
**Soft rules (reduce confidence, do not reject):**
|
||
|
||
- Duration under 1 hour or over 60 days → −0.2
|
||
- Title very similar to another event in the same batch → −0.15 (likely a duplicate row)
|
||
|
||
## Stage 5 — reconcile
|
||
|
||
Diff validated candidates against currently published events.
|
||
|
||
1. **Exact ID match** → compare fields. Unchanged: no-op. Changed: update.
|
||
2. **Near match** → update the existing event, **keeping the existing ID**. This is what survives a
|
||
wiki renaming an event without orphaning every user's completion mark.
|
||
3. **No match** → new event.
|
||
4. **Published event absent from this run** → mark `status = 'delisted'`. Never delete.
|
||
|
||
**Conflict detection.** A candidate moving an already-published `endsAt` by more than 24 hours is a
|
||
`date_conflict`. The user may have planned around the old date, so route it to quarantine regardless
|
||
of confidence.
|
||
|
||
### Scoring
|
||
|
||
Confidence records how firmly the sources pinned an event down, so the gate can hold back weak
|
||
cases. The parser assigns a base score; merge and reconcile adjust it.
|
||
|
||
```
|
||
base 0.95
|
||
−0.05 a boundary is day-precision rather than exact
|
||
−0.15 the end date is unknown (endsAt null)
|
||
+0.10 an independent source corroborates
|
||
+0.15 identical event parsed in a previous run
|
||
−0.20 any soft rule fired
|
||
−0.30 a date_conflict against a published event
|
||
```
|
||
|
||
Clamp to [0, 1]. `CONFIDENCE_THRESHOLD` (default 0.8) is the gate. Under the current parser a
|
||
day-precision event with a known end scores 0.85 and publishes, while one with an unknown end
|
||
scores 0.75 and is held — the intended bias.
|
||
|
||
## Stage 6 — gate and publish
|
||
|
||
| Condition | Destination |
|
||
|---|---|
|
||
| Confidence at or above threshold, no conflict | publish |
|
||
| Confidence below threshold | quarantine, `low_confidence` |
|
||
| Cross-source or cross-run date disagreement | quarantine, `date_conflict` |
|
||
| Failed a hard rule | quarantine, `sanity_failed` |
|
||
| Shape the schema does not recognise | quarantine, `novel_shape` |
|
||
|
||
A quarantined event does not block its siblings — if eight pass and two are held, the eight publish.
|
||
|
||
Publish upserts by ID in a transaction. Bump `version` and `updatedAt` only when a field actually
|
||
changed, or the freshness badge (PRD F7) becomes meaningless. Update `sources.content_hash`,
|
||
`etag`, `last_success_at`, and reset `consecutive_failures`.
|
||
|
||
## The review gate
|
||
|
||
Quarantined events surface at `GET /review` on the admin listener (`127.0.0.1:ADMIN_PORT`). See
|
||
`docs/ARCHITECTURE.md` § Why `/review` needs no auth.
|
||
|
||
- `POST /api/review/:id/approve` — writes to `events` with `extraction_method: 'manual'`,
|
||
`confidence: 1.0`. Approving with edits is supported; the corrected value publishes.
|
||
- `POST /api/review/:id/reject` — stamps resolution only. The candidate is held again next run if
|
||
the source has not changed, which is intended.
|
||
|
||
**Quarantine depth is the pipeline's health signal.** A growing queue means a source changed shape.
|
||
`/api/health` exposes the count.
|
||
|
||
## Testing
|
||
|
||
Every adapter ships:
|
||
|
||
1. `fixtures/<game>/<source>-<YYYY-MM-DD>.html` — a real captured page.
|
||
2. `fixtures/<game>/<source>-<YYYY-MM-DD>.expected.json` — the exact `GachaEvent[]` it produces.
|
||
3. A test running `parse` against the fixture with a pinned `ctx.now`, asserting deep equality.
|
||
|
||
`bun test` must pass with no network access.
|
||
|
||
**Regenerating `.expected.json` from the parser makes the test self-consistent, not correct.** After
|
||
an intentional change, re-verify a sample against the live page — and ideally extract the same data
|
||
a second way (a throwaway script over the fixture) to confirm counts and dates independently. That
|
||
independent check is what caught the exact event counts for both current adapters.
|
||
|
||
When a source changes shape, capture a new fixture **alongside** the old one and keep both — the old
|
||
fixture is the regression test proving the parser still handles the previous format.
|
||
|
||
`test/dates.test.ts` covers the cases that matter most: a missing year returns null rather than
|
||
guessing, impossible calendar dates are rejected, ranges crossing New Year roll the start year back,
|
||
and abbreviated months parse. `test/merge.test.ts` covers cross-source agreement, disagreement, and
|
||
rerun disambiguation. These are the last line of defense before a wrong date reaches a user.
|