Initial agent auxilliary files.
This commit is contained in:
Lucas Winther
@@ -0,0 +1,221 @@
|
||||
# Ingestion Pipeline
|
||||
|
||||
Seven stages, run per source. Every stage writes its outcome to `ingest_runs` so a failure two days
|
||||
ago can be diagnosed without re-running or re-paying.
|
||||
|
||||
```
|
||||
fetch → clean → parse|extract → validate → reconcile → gate → publish
|
||||
│
|
||||
└──► quarantine
|
||||
```
|
||||
|
||||
## The adapter contract
|
||||
|
||||
An adapter is the only per-game code. Everything downstream of `parse` is shared.
|
||||
|
||||
```ts
|
||||
export interface Adapter {
|
||||
id: string; // 'genshin-wiki-events'
|
||||
game: GameId;
|
||||
url: string;
|
||||
strategy: "parser" | "llm" | "parser_then_llm";
|
||||
minIntervalMs?: number; // default 6h
|
||||
|
||||
/** Narrow the cleaned document to just the region containing event data. */
|
||||
select?(cleaned: string): string;
|
||||
|
||||
/**
|
||||
* Deterministic parse. Return null to fall through to LLM extraction
|
||||
* (only meaningful when strategy is 'parser_then_llm').
|
||||
* Pure over its input — no network, no clock, no randomness. This is what
|
||||
* makes fixture tests possible.
|
||||
*/
|
||||
parse?(cleaned: string, ctx: ParseContext): RawEvent[] | null;
|
||||
|
||||
/** Extra instructions appended to the shared extraction prompt. */
|
||||
extractionHints?: string;
|
||||
|
||||
/** Game-specific normalization: reset times, region offsets, patch cadence. */
|
||||
normalize(raw: RawEvent, ctx: ParseContext): GachaEvent;
|
||||
}
|
||||
|
||||
export interface ParseContext {
|
||||
now: string; // injected, never Date.now() — keeps parse pure and testable
|
||||
sourceUrl: string;
|
||||
sourceId: string;
|
||||
game: GameId;
|
||||
}
|
||||
```
|
||||
|
||||
**`parse` must not read the clock.** It takes `now` from `ctx`. This is what lets a fixture test
|
||||
assert exact output for a page captured last March.
|
||||
|
||||
### Choosing a strategy
|
||||
|
||||
| Source shape | Strategy |
|
||||
|---|---|
|
||||
| JSON API, or a stable HTML table with consistent headers | `parser` |
|
||||
| Free-form patch notes, announcement prose, inconsistent markup | `llm` |
|
||||
| Mostly-stable markup that occasionally changes | `parser_then_llm` |
|
||||
|
||||
Prefer `parser`. It is free, deterministic, and instantly testable. The LLM exists for sources that
|
||||
genuinely cannot be parsed reliably, not as the default. A source with a clean API that goes through
|
||||
the model is a bug.
|
||||
|
||||
## Stage 1 — fetch
|
||||
|
||||
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
|
||||
the run as `skipped_unchanged` with zero further cost.
|
||||
- `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
|
||||
- Honor `robots.txt`. Cache the parsed robots per host for 24h.
|
||||
- 20s timeout; retry twice with exponential backoff on 5xx and network errors; never retry 4xx.
|
||||
- Store the raw bytes in `snapshots`.
|
||||
|
||||
On failure: increment `consecutive_failures`, leave published events untouched, end the run as
|
||||
`failed`. A source being down never mutates the feed.
|
||||
|
||||
## Stage 2 — clean
|
||||
|
||||
Reduce the document before it costs anything. This stage is the second-biggest cost lever after the
|
||||
content-hash skip.
|
||||
|
||||
- Drop `<script>`, `<style>`, `<svg>`, `<noscript>`, comments, nav, header, footer, and known
|
||||
wiki chrome (edit links, category boxes, reference lists).
|
||||
- Collapse whitespace; convert tables to pipe-delimited text; keep headings as `#` markers so
|
||||
section structure survives.
|
||||
- Apply `adapter.select()` if present to isolate the event region.
|
||||
- Hash the result (sha256) → `content_hash`.
|
||||
|
||||
**If `content_hash` matches `sources.content_hash`, end the run as `skipped_unchanged`.** This is
|
||||
the check that keeps a 6-hourly schedule from costing anything on a quiet week — most runs should
|
||||
end here.
|
||||
|
||||
A typical wiki page goes from ~15k tokens raw to ~5k cleaned. Verify with `messages.count_tokens`
|
||||
when tuning, not by guessing.
|
||||
|
||||
## Stage 3 — parse or extract
|
||||
|
||||
Per strategy. `parse` produces `RawEvent[]` directly. `extract` sends the cleaned text to
|
||||
`claude-opus-5` with a structured-output schema — see `docs/LLM-EXTRACTION.md` for the request
|
||||
shape, prompt, and cost rules.
|
||||
|
||||
`parser_then_llm` calls `parse` first and falls through to `extract` only when it returns `null`.
|
||||
When that fallthrough happens, log it loudly: it means the source changed shape and the parser needs
|
||||
updating. A `parser_then_llm` source that is silently always falling through is paying LLM prices
|
||||
for a parser that no longer works.
|
||||
|
||||
## Stage 4 — validate
|
||||
|
||||
Zod parse against `GachaEvent`, then calendar sanity rules. Anything failing a hard rule goes to
|
||||
quarantine with `reason: 'sanity_failed'` — never to the feed.
|
||||
|
||||
**Hard rules (reject):**
|
||||
|
||||
| Rule | Rationale |
|
||||
|---|---|
|
||||
| `endsAt > startsAt` when both present | A backwards interval is always a parse error |
|
||||
| Duration ≤ 180 days | Patch cycles are ~6 weeks; 180d means a year was misread as a range |
|
||||
| `startsAt` within [now − 2y, now + 1y] | Catches century typos and relative-date misreads |
|
||||
| `endsAt` null ⟺ `endPrecision === "unknown"` | The two fields must agree |
|
||||
| `regionEnds` non-null ⟺ `regionScoped` | Same |
|
||||
| All `regionEnds` values within 24h of each other | Region resets differ by hours, not days |
|
||||
| `title` non-empty, ≤ 200 chars, not a placeholder ("TBD", "Event", "Unknown") | Catches header rows scraped as events |
|
||||
|
||||
**Soft rules (reduce confidence, do not reject):**
|
||||
|
||||
- Duration under 1 hour or over 60 days → −0.2
|
||||
- `startPrecision` or `endPrecision` is `"day"` → −0.1
|
||||
- Title very similar to another event in the same batch → −0.15 (likely a duplicate row)
|
||||
|
||||
## Stage 5 — reconcile
|
||||
|
||||
Diff the validated candidates against currently published events for this source.
|
||||
|
||||
1. **Exact ID match** → compare fields. Unchanged: no-op. Changed: candidate update.
|
||||
2. **Near match** — same game, date windows overlap, title similarity ≥ 0.85 — → treat as an update
|
||||
to the existing event, **keeping the existing ID**. This is what survives a wiki renaming an
|
||||
event without orphaning every user's completion mark.
|
||||
3. **No match** → new event.
|
||||
4. **Published event absent from this run's candidates** → mark `status = 'delisted'`. Do not
|
||||
delete.
|
||||
|
||||
**Conflict detection.** A candidate that changes an already-published `endsAt` by more than 24 hours
|
||||
is a `date_conflict`. This is the case worth being paranoid about: the user may have already planned
|
||||
around the old date, and a silent shift is exactly the failure the product exists to prevent. Route
|
||||
it to quarantine regardless of confidence.
|
||||
|
||||
### Scoring
|
||||
|
||||
Confidence is computed here, from evidence — **not** taken from the model's self-report. A model
|
||||
saying "confidence: 0.95" is a token prediction, not a measurement.
|
||||
|
||||
```
|
||||
base parser → 0.95 llm → 0.70
|
||||
+0.15 the same event was extracted identically from a previous run
|
||||
+0.10 both timestamps have precision "exact"
|
||||
+0.10 a second source for the same game corroborates within 1 hour
|
||||
−0.20 any soft rule fired
|
||||
−0.30 this is a date_conflict against a published event
|
||||
```
|
||||
|
||||
Clamp to [0, 1]. `CONFIDENCE_THRESHOLD` (default 0.8) is the gate.
|
||||
|
||||
## Stage 6 — gate
|
||||
|
||||
| Condition | Destination |
|
||||
|---|---|
|
||||
| `confidence >= CONFIDENCE_THRESHOLD` and no conflict | publish |
|
||||
| `confidence < CONFIDENCE_THRESHOLD` | quarantine — `low_confidence` |
|
||||
| `date_conflict` | quarantine — `date_conflict`, with `conflicts_with` set |
|
||||
| failed a hard rule | quarantine — `sanity_failed` |
|
||||
| new event type or field the schema does not recognize | quarantine — `novel_shape` |
|
||||
|
||||
A quarantined event does not block its siblings. If eight events in a run pass and two are held, the
|
||||
eight publish.
|
||||
|
||||
## Stage 7 — publish
|
||||
|
||||
Upsert by ID inside a transaction. Bump `version` and `updatedAt` only when a field actually
|
||||
changed — an unchanged run must not churn `updatedAt`, or the freshness badge (PRD F7) becomes
|
||||
meaningless. Update `sources.content_hash`, `etag`, `last_success_at`, and reset
|
||||
`consecutive_failures`.
|
||||
|
||||
## The review gate
|
||||
|
||||
Quarantined events surface at `GET /review` on the admin listener (`127.0.0.1:ADMIN_PORT`). See
|
||||
`docs/ARCHITECTURE.md` § Why `/review` needs no auth — the routes are simply not registered on the
|
||||
public listener.
|
||||
|
||||
The review UI shows, per held event: the parsed fields, the reason and detail, the conflicting
|
||||
published event side-by-side when applicable, a link to the source, and the exact cleaned text
|
||||
excerpt the extraction came from. A reviewer needs to answer "is this date right?" without leaving
|
||||
the page.
|
||||
|
||||
- `POST /api/review/:id/approve` — writes to `events` with `extraction_method: 'manual'` and
|
||||
`confidence: 1.0`, and stamps `resolved_at` / `resolution`.
|
||||
- `POST /api/review/:id/reject` — stamps resolution only. The event is not published, and the same
|
||||
candidate will be re-held on the next run if the source has not changed.
|
||||
|
||||
Approving with edits is supported: the reviewer can correct a date before approving. That corrected
|
||||
value is the one that publishes.
|
||||
|
||||
**Quarantine depth is the health signal for the whole pipeline.** A growing queue means a source
|
||||
changed shape or the prompt regressed. `/api/health` exposes the count; watch it.
|
||||
|
||||
## Testing
|
||||
|
||||
Every adapter ships:
|
||||
|
||||
1. `fixtures/<game>/<source>-<YYYY-MM-DD>.html` — a real captured page.
|
||||
2. `fixtures/<game>/<source>-<YYYY-MM-DD>.expected.json` — the exact `GachaEvent[]` it should
|
||||
produce.
|
||||
3. A test running `parse` + `normalize` against the fixture with a pinned `ctx.now`, asserting
|
||||
deep equality.
|
||||
|
||||
`bun test` must pass with no network access. When a source changes shape, capture a new fixture
|
||||
alongside the old one and keep both — the old fixture is the regression test proving the parser
|
||||
still handles the previous format.
|
||||
|
||||
Validator rules get their own unit tests with deliberately broken inputs: backwards intervals,
|
||||
1000-year durations, `endsAt` set with `endPrecision: "unknown"`. These rules are the last line of
|
||||
defense before a wrong date reaches a user; test them like it.
|
||||
Reference in New Issue
Block a user