# Ingestion Pipeline Six stages, run per source. Every stage writes its outcome to `ingest_runs` so a failure two days ago can be diagnosed without re-running. ``` fetch → parse → merge → validate → reconcile → gate and publish │ │ (sanitize) └──► quarantine ``` Sanitizing is stage 2.5 rather than a stage of its own: it is wired into the adapter seam, so it runs on every source without the pipeline arranging it. ## No LLM Event data is extracted by deterministic code only. There is no model call anywhere in this pipeline, no API key, and no per-run cost. This is a deliberate constraint, not an omission: - A source that cannot be parsed deterministically **does not get an adapter.** Report it rather than reaching for inference. - Parser output is reproducible — the same fixture always yields the same events, which is what makes the fixture tests meaningful. - Iterating is free and offline: `bun run parse `. If a source's markup is too unstable to parse, the answer is a different source, not a model. ## Three layers: parsers, adapters, merge The layering is what makes a second, third, or tenth source cheap. | Layer | Answers | Lives in | Scope | |---|---|---|---| | **Parser** | "How is this *site* laid out?" | `src/ingest/parsers/` | One site template, many games | | **Adapter** | "Which URL, for which game, via which parser?" | `src/ingest/adapters/index.ts` | One page | | **Merge** | "These sources disagree — now what?" | `src/ingest/merge.ts` | One game, many sources | Consequences worth internalising: - Adding a source for a site already parsed = **one entry in `SOURCES`**. No new parsing code. - Adding a new *site* = one parser module + its `PARSERS` entry, then adapters as above. - A game may have any number of sources. `parseGame(game, documents, now)` runs them all and merges. ### Parsers in the tree | Parser | Site | Sources using it | |---|---|---| | `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE, Infinity Nikki, Persona 5: The Phantom X, Chaos Zero Nightmare | | `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | | `akwiki` | arknights.wiki.gg's `mrfz-wtable` "Ongoing/upcoming" table | Arknights | | `fandom` | Fandom wikis via the MediaWiki `action=parse` API — two page templates: `Event \| Time Period \| Version` wikitables, and FGO's picture-fenced `ONGOING EVENTS` blocks | Reverse: 1999, Fate/Grand Order | | `bawiki` | bluearchive.wiki's rendered `/wiki/Events` — a JP/Global tabber over `Name (EN) \| Start date \| End date \| Notes` wikitables | Blue Archive | | `holodoriwiki` | holodori.wiki's rendered `/wiki/Events` — `Current Events` and `Past Events` wikitables over `Event \| Type \| Start Date \| End Date` | hololive Dreams | | `iopwiki` | iopwiki.com's `gf-table event-period` tables — `Title \| Period (start/end) \| Server \| Type \| Comment`, one table per event and one row per server | Girls' Frontline 2 | | `stellasorawiki` | stellasora.miraheze.org's front-page `Current Banners` module — `