# Ingestion Pipeline

Seven stages, run per source. Every stage writes its outcome to `ingest_runs` so a failure two days
ago can be diagnosed without re-running or re-paying.

```
fetch → clean → parse|extract → validate → reconcile → gate → publish
                                                        │
                                                        └──► quarantine
```

## The adapter contract

An adapter is the only per-game code. Everything downstream of `parse` is shared.

```ts
export interface Adapter {
  id: string;                    // 'genshin-wiki-events'
  game: GameId;
  url: string;
  strategy: "parser" | "llm" | "parser_then_llm";
  minIntervalMs?: number;        // default 6h

  /** Narrow the cleaned document to just the region containing event data. */
  select?(cleaned: string): string;

  /**
   * Deterministic parse. Return null to fall through to LLM extraction
   * (only meaningful when strategy is 'parser_then_llm').
   * Pure over its input — no network, no clock, no randomness. This is what
   * makes fixture tests possible.
   */
  parse?(cleaned: string, ctx: ParseContext): RawEvent[] | null;

  /** Extra instructions appended to the shared extraction prompt. */
  extractionHints?: string;

  /** Game-specific normalization: reset times, region offsets, patch cadence. */
  normalize(raw: RawEvent, ctx: ParseContext): GachaEvent;
}

export interface ParseContext {
  now: string;                   // injected, never Date.now() — keeps parse pure and testable
  sourceUrl: string;
  sourceId: string;
  game: GameId;
}
```

**`parse` must not read the clock.** It takes `now` from `ctx`. This is what lets a fixture test
assert exact output for a page captured last March.

### Choosing a strategy

| Source shape | Strategy |
|---|---|
| JSON API, or a stable HTML table with consistent headers | `parser` |
| Free-form patch notes, announcement prose, inconsistent markup | `llm` |
| Mostly-stable markup that occasionally changes | `parser_then_llm` |

Prefer `parser`. It is free, deterministic, and instantly testable. The LLM exists for sources that
genuinely cannot be parsed reliably, not as the default. A source with a clean API that goes through
the model is a bug.

## Stage 1 — fetch

- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
  the run as `skipped_unchanged` with zero further cost.
- `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
- Honor `robots.txt`. Cache the parsed robots per host for 24h.
- 20s timeout; retry twice with exponential backoff on 5xx and network errors; never retry 4xx.
- Store the raw bytes in `snapshots`.

On failure: increment `consecutive_failures`, leave published events untouched, end the run as
`failed`. A source being down never mutates the feed.

## Stage 2 — clean

Reduce the document before it costs anything. This stage is the second-biggest cost lever after the
content-hash skip.

- Drop `<script>`, `<style>`, `<svg>`, `<noscript>`, comments, nav, header, footer, and known
  wiki chrome (edit links, category boxes, reference lists).
- Collapse whitespace; convert tables to pipe-delimited text; keep headings as `#` markers so
  section structure survives.
- Apply `adapter.select()` if present to isolate the event region.
- Hash the result (sha256) → `content_hash`.

**If `content_hash` matches `sources.content_hash`, end the run as `skipped_unchanged`.** This is
the check that keeps a 6-hourly schedule from costing anything on a quiet week — most runs should
end here.

A typical wiki page goes from ~15k tokens raw to ~5k cleaned. Verify with `messages.count_tokens`
when tuning, not by guessing.

## Stage 3 — parse or extract

Per strategy. `parse` produces `RawEvent[]` directly. `extract` sends the cleaned text to
`claude-opus-5` with a structured-output schema — see `docs/LLM-EXTRACTION.md` for the request
shape, prompt, and cost rules.

`parser_then_llm` calls `parse` first and falls through to `extract` only when it returns `null`.
When that fallthrough happens, log it loudly: it means the source changed shape and the parser needs
updating. A `parser_then_llm` source that is silently always falling through is paying LLM prices
for a parser that no longer works.

## Stage 4 — validate

Zod parse against `GachaEvent`, then calendar sanity rules. Anything failing a hard rule goes to
quarantine with `reason: 'sanity_failed'` — never to the feed.

**Hard rules (reject):**

| Rule | Rationale |
|---|---|
| `endsAt > startsAt` when both present | A backwards interval is always a parse error |
| Duration ≤ 180 days | Patch cycles are ~6 weeks; 180d means a year was misread as a range |
| `startsAt` within [now − 2y, now + 1y] | Catches century typos and relative-date misreads |
| `endsAt` null ⟺ `endPrecision === "unknown"` | The two fields must agree |
| `regionEnds` non-null ⟺ `regionScoped` | Same |
| All `regionEnds` values within 24h of each other | Region resets differ by hours, not days |
| `title` non-empty, ≤ 200 chars, not a placeholder ("TBD", "Event", "Unknown") | Catches header rows scraped as events |

**Soft rules (reduce confidence, do not reject):**

- Duration under 1 hour or over 60 days → −0.2
- `startPrecision` or `endPrecision` is `"day"` → −0.1
- Title very similar to another event in the same batch → −0.15 (likely a duplicate row)

## Stage 5 — reconcile

Diff the validated candidates against currently published events for this source.

1. **Exact ID match** → compare fields. Unchanged: no-op. Changed: candidate update.
2. **Near match** — same game, date windows overlap, title similarity ≥ 0.85 — → treat as an update
   to the existing event, **keeping the existing ID**. This is what survives a wiki renaming an
   event without orphaning every user's completion mark.
3. **No match** → new event.
4. **Published event absent from this run's candidates** → mark `status = 'delisted'`. Do not
   delete.

**Conflict detection.** A candidate that changes an already-published `endsAt` by more than 24 hours
is a `date_conflict`. This is the case worth being paranoid about: the user may have already planned
around the old date, and a silent shift is exactly the failure the product exists to prevent. Route
it to quarantine regardless of confidence.

### Scoring

Confidence is computed here, from evidence — **not** taken from the model's self-report. A model
saying "confidence: 0.95" is a token prediction, not a measurement.

```
base            parser → 0.95        llm → 0.70
+0.15  the same event was extracted identically from a previous run
+0.10  both timestamps have precision "exact"
+0.10  a second source for the same game corroborates within 1 hour
−0.20  any soft rule fired
−0.30  this is a date_conflict against a published event
```

Clamp to [0, 1]. `CONFIDENCE_THRESHOLD` (default 0.8) is the gate.

## Stage 6 — gate

| Condition | Destination |
|---|---|
| `confidence >= CONFIDENCE_THRESHOLD` and no conflict | publish |
| `confidence < CONFIDENCE_THRESHOLD` | quarantine — `low_confidence` |
| `date_conflict` | quarantine — `date_conflict`, with `conflicts_with` set |
| failed a hard rule | quarantine — `sanity_failed` |
| new event type or field the schema does not recognize | quarantine — `novel_shape` |

A quarantined event does not block its siblings. If eight events in a run pass and two are held, the
eight publish.

## Stage 7 — publish

Upsert by ID inside a transaction. Bump `version` and `updatedAt` only when a field actually
changed — an unchanged run must not churn `updatedAt`, or the freshness badge (PRD F7) becomes
meaningless. Update `sources.content_hash`, `etag`, `last_success_at`, and reset
`consecutive_failures`.

## The review gate

Quarantined events surface at `GET /review` on the admin listener (`127.0.0.1:ADMIN_PORT`). See
`docs/ARCHITECTURE.md` § Why `/review` needs no auth — the routes are simply not registered on the
public listener.

The review UI shows, per held event: the parsed fields, the reason and detail, the conflicting
published event side-by-side when applicable, a link to the source, and the exact cleaned text
excerpt the extraction came from. A reviewer needs to answer "is this date right?" without leaving
the page.

- `POST /api/review/:id/approve` — writes to `events` with `extraction_method: 'manual'` and
  `confidence: 1.0`, and stamps `resolved_at` / `resolution`.
- `POST /api/review/:id/reject` — stamps resolution only. The event is not published, and the same
  candidate will be re-held on the next run if the source has not changed.

Approving with edits is supported: the reviewer can correct a date before approving. That corrected
value is the one that publishes.

**Quarantine depth is the health signal for the whole pipeline.** A growing queue means a source
changed shape or the prompt regressed. `/api/health` exposes the count; watch it.

## Testing

Every adapter ships:

1. `fixtures/<game>/<source>-<YYYY-MM-DD>.html` — a real captured page.
2. `fixtures/<game>/<source>-<YYYY-MM-DD>.expected.json` — the exact `GachaEvent[]` it should
   produce.
3. A test running `parse` + `normalize` against the fixture with a pinned `ctx.now`, asserting
   deep equality.

`bun test` must pass with no network access. When a source changes shape, capture a new fixture
alongside the old one and keep both — the old fixture is the regression test proving the parser
still handles the previous format.

Validator rules get their own unit tests with deliberately broken inputs: backwards intervals,
1000-year durations, `endsAt` set with `endPrecision: "unknown"`. These rules are the last line of
defense before a wrong date reaches a user; test them like it.