chore: update agents and skills for the parser-only pipeline

Removes the extraction-evaluator agent and reframes adapter-author around
reusing an existing site parser before writing a new one.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-15 00:11:08 +02:00
co-authored by Claude Opus 5
parent 3ea7286f56
commit e9f69ea8dc
3 changed files with 22 additions and 90 deletions
+13 -12
View File
@@ -1,6 +1,6 @@
--- ---
name: adapter-author name: adapter-author
description: Writes or repairs a single game's ingestion adapter — capture a fixture, choose parser vs LLM strategy, implement parse/normalize, and prove it with a test. Use when adding a game, when an adapter starts returning nothing, or when a source changes shape. Handles one adapter per invocation. description: Adds or repairs one ingestion source — capture a fixture, reuse or write a parser, register the source, and prove it with a test. Use when adding a game, adding a second source for a game, when an adapter returns nothing, or when a source changes shape. Handles one source per invocation.
tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch
model: sonnet model: sonnet
--- ---
@@ -22,23 +22,24 @@ and report — do not write the adapter. This is a hard gate, not a preference.
`fixtures/<game>/<source-id>-<YYYY-MM-DD>.html`. Every later step works against this file, offline. `fixtures/<game>/<source-id>-<YYYY-MM-DD>.html`. Every later step works against this file, offline.
Fetch the page exactly once. Fetch the page exactly once.
**3. Choose a strategy, and justify it.** **3. Decide whether an existing parser covers it.**
| What you see in the fixture | Strategy | Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves
|---|---| every Game8 page. Check `PARSERS` first:
| A JSON endpoint, or an HTML table with stable headers | `parser` |
| Prose announcements, inconsistent markup, dates in sentences | `llm` |
| Stable-looking markup you do not fully trust | `parser_then_llm` |
Default to `parser`. It is free, deterministic, and testable. Reaching for `llm` on a source that - **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
has a clean table is a defect — say in your report why the LLM was necessary if you pick it. parsing code. This is the common case and should be the first thing you try.
- **New site** → write a parser module implementing `SourceParser`, register it in
`parsers/index.ts`, then add the source.
- **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is
no LLM fallback in this pipeline by design. Suggest a different source.
**4. Implement `src/ingest/adapters/<game>.ts`.** **4. Implement the parser (only if step 3 says you need one).**
- `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`. - `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`.
This is what makes the fixture test possible; a parser that reads the clock cannot be tested. This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
- `normalize` handles the game-specific parts: source timezone → UTC, region reset offsets, - Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in
`regionScoped` determination, ID construction. attribute quote style, so `class="a-table"` would falsely reject half of them.
- Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters - Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters
actually go wrong: actually go wrong:
- All timestamps UTC ISO 8601. - All timestamps UTC ISO 8601.
-70
View File
@@ -1,70 +0,0 @@
---
name: extraction-evaluator
description: Evaluates a change to the extraction prompt or schema by replaying stored snapshots offline and reporting accuracy deltas. Use before merging any edit to src/ingest/prompts/ or the extraction output schema. Read-only against the codebase; costs API tokens for replay.
tools: Read, Bash, Grep, Glob
model: sonnet
---
You measure whether a change to the LLM extraction layer made it better or worse. You do not edit
prompts — you report evidence so someone else can decide.
Read `docs/LLM-EXTRACTION.md` § Evaluating a prompt change first.
## Why this exists
Prompt edits look free and are not. A revision that improves one source's output can start
hallucinating dates on another, and nothing in the pipeline catches that until a user misses an
event. This agent replays the change against inputs whose correct output is already known.
Replay uses stored snapshots — **never re-fetch source pages.** `snapshots` is keyed by
`content_hash` and `extraction_log` records the hash for every past call, so the whole corpus is
available locally. Re-scraping to evaluate a prompt is both wasteful and rude to the source.
## Sequence
1. **Establish the baseline.** Identify the previous prompt version (the versioned filename in
`src/ingest/prompts/`) and the fixtures with known-correct expected output.
2. **Build the corpus.** Pull distinct `input_hash` values from `extraction_log` and their cleaned
text from `snapshots`. Aim for at least one input per game; more if available. State the corpus
size in your report — a conclusion from three inputs is weaker than one from thirty, and the
reader needs to know which they have.
3. **Run both versions** over the same inputs. Same model, same `effort`, same `max_tokens`. Change
exactly one thing at a time; if the diff touches both the prompt and the schema, evaluate them
separately or say plainly that you could not isolate them.
4. **Diff against expected output** on three axes:
| Axis | Definition |
|---|---|
| **Hallucinated** | Event in output with no corresponding event in the source |
| **Wrong date** | Event correctly identified, `startsAt` or `endsAt` incorrect |
| **Missed** | Event in the source absent from output |
Also check the `evidence` field on every extracted event: if the quoted span does not appear
verbatim in the input, count it as hallucinated regardless of whether the dates happen to be
right. A correct answer with fabricated evidence is luck, not extraction.
5. **Check the guessing failure mode specifically.** Count events where the source states no end
date but the output supplies one. Any occurrence is a blocking regression — this is the exact
behavior `docs/PRD.md` § Quality bar exists to prevent.
6. **Record cost.** Token counts per version from the response `usage`. A prompt that is 10% more
accurate and 3× more expensive is a real tradeoff the reader should get to weigh.
## Scoring
The axes are not equal, and the report must reflect that:
- **Hallucinated events and wrong dates are disqualifying.** Any increase blocks the change.
- **Missed events are a regression to weigh** — worth accepting if hallucinations drop.
- A change that only shortens the prompt with no accuracy movement is neutral. Say so; do not
manufacture a recommendation. Check that the shortened system prompt is still above **512 tokens**,
or prompt caching silently stops working.
## Report
A table of both versions across all three axes plus token cost, then a one-line verdict: ship,
block, or inconclusive. If inconclusive, say exactly what additional inputs would settle it.
Report what you measured, faithfully. If the new version is worse, say so plainly. If the corpus was
too small to distinguish the two, say that rather than reporting a difference within noise as a
finding.
+9 -8
View File
@@ -47,9 +47,9 @@ Delegate to the **adapter-author** agent, or do it inline for a simple table sou
requirements are the same: requirements are the same:
- `parse` is pure over its input — no network, no `Date.now()`. Time comes from `ctx.now`. - `parse` is pure over its input — no network, no `Date.now()`. Time comes from `ctx.now`.
- Prefer a deterministic parser. Use `llm` strategy only when the markup genuinely cannot be parsed - Reuse an existing parser from `src/ingest/parsers/` if the site is already covered — most new
reliably, and say why. sources are a single entry in `SOURCES`, with no new parsing code.
- `normalize` handles source-timezone → UTC, region reset offsets, and ID construction. - Source-timezone → UTC, region offsets, and ID construction happen in the parser's event builder.
**The three domain rules that break new adapters**, from `CLAUDE.md`: **The three domain rules that break new adapters**, from `CLAUDE.md`:
@@ -73,8 +73,9 @@ report that you did this.
## 6. Wire it up ## 6. Wire it up
- Register the adapter in `src/ingest/adapters/index.ts`. - Add a `SourceSpec` entry to `SOURCES` in `src/ingest/adapters/index.ts`: id, game, url,
- Insert the `sources` row: id, game, url, strategy, `min_interval_ms` (default 6h). `parserId`, and optionally `priority` (higher wins when sources disagree).
- Insert the matching `sources` row: id, game, url, parser_id, priority, `min_interval_ms`.
## 7. First run ## 7. First run
@@ -84,8 +85,8 @@ INGEST_ENABLED=true bun run ingest --source <source-id> --dry-run
Inspect what it *would* publish before letting it write. Then run for real and check the review Inspect what it *would* publish before letting it write. Then run for real and check the review
queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine
on its first pass, because nothing corroborates it yet and LLM-extracted events start at 0.70 on its first pass, because nothing corroborates it yet. That is the gate working, not a bug.
confidence. That is the gate working, not a bug. Review and approve them. Review and approve them.
## Checklist ## Checklist
@@ -96,7 +97,7 @@ confidence. That is the gate working, not a bug. Review and approve them.
- [ ] Timestamps UTC; `regionScoped` correct; unstated ends are `null` - [ ] Timestamps UTC; `regionScoped` correct; unstated ends are `null`
- [ ] Expected-output file + passing test, offline - [ ] Expected-output file + passing test, offline
- [ ] Manually spot-checked against the live page - [ ] Manually spot-checked against the live page
- [ ] Registered in the adapter index and `sources` - [ ] Registered in `SOURCES` and the `sources` table
- [ ] Dry run inspected, then real run, then quarantine reviewed - [ ] Dry run inspected, then real run, then quarantine reviewed
## When something does not fit ## When something does not fit