chore: update agents and skills for the parser-only pipeline
Removes the extraction-evaluator agent and reframes adapter-author around reusing an existing site parser before writing a new one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
3ea7286f56
commit
e9f69ea8dc
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
name: adapter-author
|
name: adapter-author
|
||||||
description: Writes or repairs a single game's ingestion adapter — capture a fixture, choose parser vs LLM strategy, implement parse/normalize, and prove it with a test. Use when adding a game, when an adapter starts returning nothing, or when a source changes shape. Handles one adapter per invocation.
|
description: Adds or repairs one ingestion source — capture a fixture, reuse or write a parser, register the source, and prove it with a test. Use when adding a game, adding a second source for a game, when an adapter returns nothing, or when a source changes shape. Handles one source per invocation.
|
||||||
tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch
|
tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch
|
||||||
model: sonnet
|
model: sonnet
|
||||||
---
|
---
|
||||||
@@ -22,23 +22,24 @@ and report — do not write the adapter. This is a hard gate, not a preference.
|
|||||||
`fixtures/<game>/<source-id>-<YYYY-MM-DD>.html`. Every later step works against this file, offline.
|
`fixtures/<game>/<source-id>-<YYYY-MM-DD>.html`. Every later step works against this file, offline.
|
||||||
Fetch the page exactly once.
|
Fetch the page exactly once.
|
||||||
|
|
||||||
**3. Choose a strategy, and justify it.**
|
**3. Decide whether an existing parser covers it.**
|
||||||
|
|
||||||
| What you see in the fixture | Strategy |
|
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves
|
||||||
|---|---|
|
every Game8 page. Check `PARSERS` first:
|
||||||
| A JSON endpoint, or an HTML table with stable headers | `parser` |
|
|
||||||
| Prose announcements, inconsistent markup, dates in sentences | `llm` |
|
|
||||||
| Stable-looking markup you do not fully trust | `parser_then_llm` |
|
|
||||||
|
|
||||||
Default to `parser`. It is free, deterministic, and testable. Reaching for `llm` on a source that
|
- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
|
||||||
has a clean table is a defect — say in your report why the LLM was necessary if you pick it.
|
parsing code. This is the common case and should be the first thing you try.
|
||||||
|
- **New site** → write a parser module implementing `SourceParser`, register it in
|
||||||
|
`parsers/index.ts`, then add the source.
|
||||||
|
- **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is
|
||||||
|
no LLM fallback in this pipeline by design. Suggest a different source.
|
||||||
|
|
||||||
**4. Implement `src/ingest/adapters/<game>.ts`.**
|
**4. Implement the parser (only if step 3 says you need one).**
|
||||||
|
|
||||||
- `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`.
|
- `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`.
|
||||||
This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
|
This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
|
||||||
- `normalize` handles the game-specific parts: source timezone → UTC, region reset offsets,
|
- Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in
|
||||||
`regionScoped` determination, ID construction.
|
attribute quote style, so `class="a-table"` would falsely reject half of them.
|
||||||
- Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters
|
- Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters
|
||||||
actually go wrong:
|
actually go wrong:
|
||||||
- All timestamps UTC ISO 8601.
|
- All timestamps UTC ISO 8601.
|
||||||
|
|||||||
@@ -1,70 +0,0 @@
|
|||||||
---
|
|
||||||
name: extraction-evaluator
|
|
||||||
description: Evaluates a change to the extraction prompt or schema by replaying stored snapshots offline and reporting accuracy deltas. Use before merging any edit to src/ingest/prompts/ or the extraction output schema. Read-only against the codebase; costs API tokens for replay.
|
|
||||||
tools: Read, Bash, Grep, Glob
|
|
||||||
model: sonnet
|
|
||||||
---
|
|
||||||
|
|
||||||
You measure whether a change to the LLM extraction layer made it better or worse. You do not edit
|
|
||||||
prompts — you report evidence so someone else can decide.
|
|
||||||
|
|
||||||
Read `docs/LLM-EXTRACTION.md` § Evaluating a prompt change first.
|
|
||||||
|
|
||||||
## Why this exists
|
|
||||||
|
|
||||||
Prompt edits look free and are not. A revision that improves one source's output can start
|
|
||||||
hallucinating dates on another, and nothing in the pipeline catches that until a user misses an
|
|
||||||
event. This agent replays the change against inputs whose correct output is already known.
|
|
||||||
|
|
||||||
Replay uses stored snapshots — **never re-fetch source pages.** `snapshots` is keyed by
|
|
||||||
`content_hash` and `extraction_log` records the hash for every past call, so the whole corpus is
|
|
||||||
available locally. Re-scraping to evaluate a prompt is both wasteful and rude to the source.
|
|
||||||
|
|
||||||
## Sequence
|
|
||||||
|
|
||||||
1. **Establish the baseline.** Identify the previous prompt version (the versioned filename in
|
|
||||||
`src/ingest/prompts/`) and the fixtures with known-correct expected output.
|
|
||||||
2. **Build the corpus.** Pull distinct `input_hash` values from `extraction_log` and their cleaned
|
|
||||||
text from `snapshots`. Aim for at least one input per game; more if available. State the corpus
|
|
||||||
size in your report — a conclusion from three inputs is weaker than one from thirty, and the
|
|
||||||
reader needs to know which they have.
|
|
||||||
3. **Run both versions** over the same inputs. Same model, same `effort`, same `max_tokens`. Change
|
|
||||||
exactly one thing at a time; if the diff touches both the prompt and the schema, evaluate them
|
|
||||||
separately or say plainly that you could not isolate them.
|
|
||||||
4. **Diff against expected output** on three axes:
|
|
||||||
|
|
||||||
| Axis | Definition |
|
|
||||||
|---|---|
|
|
||||||
| **Hallucinated** | Event in output with no corresponding event in the source |
|
|
||||||
| **Wrong date** | Event correctly identified, `startsAt` or `endsAt` incorrect |
|
|
||||||
| **Missed** | Event in the source absent from output |
|
|
||||||
|
|
||||||
Also check the `evidence` field on every extracted event: if the quoted span does not appear
|
|
||||||
verbatim in the input, count it as hallucinated regardless of whether the dates happen to be
|
|
||||||
right. A correct answer with fabricated evidence is luck, not extraction.
|
|
||||||
|
|
||||||
5. **Check the guessing failure mode specifically.** Count events where the source states no end
|
|
||||||
date but the output supplies one. Any occurrence is a blocking regression — this is the exact
|
|
||||||
behavior `docs/PRD.md` § Quality bar exists to prevent.
|
|
||||||
|
|
||||||
6. **Record cost.** Token counts per version from the response `usage`. A prompt that is 10% more
|
|
||||||
accurate and 3× more expensive is a real tradeoff the reader should get to weigh.
|
|
||||||
|
|
||||||
## Scoring
|
|
||||||
|
|
||||||
The axes are not equal, and the report must reflect that:
|
|
||||||
|
|
||||||
- **Hallucinated events and wrong dates are disqualifying.** Any increase blocks the change.
|
|
||||||
- **Missed events are a regression to weigh** — worth accepting if hallucinations drop.
|
|
||||||
- A change that only shortens the prompt with no accuracy movement is neutral. Say so; do not
|
|
||||||
manufacture a recommendation. Check that the shortened system prompt is still above **512 tokens**,
|
|
||||||
or prompt caching silently stops working.
|
|
||||||
|
|
||||||
## Report
|
|
||||||
|
|
||||||
A table of both versions across all three axes plus token cost, then a one-line verdict: ship,
|
|
||||||
block, or inconclusive. If inconclusive, say exactly what additional inputs would settle it.
|
|
||||||
|
|
||||||
Report what you measured, faithfully. If the new version is worse, say so plainly. If the corpus was
|
|
||||||
too small to distinguish the two, say that rather than reporting a difference within noise as a
|
|
||||||
finding.
|
|
||||||
@@ -47,9 +47,9 @@ Delegate to the **adapter-author** agent, or do it inline for a simple table sou
|
|||||||
requirements are the same:
|
requirements are the same:
|
||||||
|
|
||||||
- `parse` is pure over its input — no network, no `Date.now()`. Time comes from `ctx.now`.
|
- `parse` is pure over its input — no network, no `Date.now()`. Time comes from `ctx.now`.
|
||||||
- Prefer a deterministic parser. Use `llm` strategy only when the markup genuinely cannot be parsed
|
- Reuse an existing parser from `src/ingest/parsers/` if the site is already covered — most new
|
||||||
reliably, and say why.
|
sources are a single entry in `SOURCES`, with no new parsing code.
|
||||||
- `normalize` handles source-timezone → UTC, region reset offsets, and ID construction.
|
- Source-timezone → UTC, region offsets, and ID construction happen in the parser's event builder.
|
||||||
|
|
||||||
**The three domain rules that break new adapters**, from `CLAUDE.md`:
|
**The three domain rules that break new adapters**, from `CLAUDE.md`:
|
||||||
|
|
||||||
@@ -73,8 +73,9 @@ report that you did this.
|
|||||||
|
|
||||||
## 6. Wire it up
|
## 6. Wire it up
|
||||||
|
|
||||||
- Register the adapter in `src/ingest/adapters/index.ts`.
|
- Add a `SourceSpec` entry to `SOURCES` in `src/ingest/adapters/index.ts`: id, game, url,
|
||||||
- Insert the `sources` row: id, game, url, strategy, `min_interval_ms` (default 6h).
|
`parserId`, and optionally `priority` (higher wins when sources disagree).
|
||||||
|
- Insert the matching `sources` row: id, game, url, parser_id, priority, `min_interval_ms`.
|
||||||
|
|
||||||
## 7. First run
|
## 7. First run
|
||||||
|
|
||||||
@@ -84,8 +85,8 @@ INGEST_ENABLED=true bun run ingest --source <source-id> --dry-run
|
|||||||
|
|
||||||
Inspect what it *would* publish before letting it write. Then run for real and check the review
|
Inspect what it *would* publish before letting it write. Then run for real and check the review
|
||||||
queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine
|
queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine
|
||||||
on its first pass, because nothing corroborates it yet and LLM-extracted events start at 0.70
|
on its first pass, because nothing corroborates it yet. That is the gate working, not a bug.
|
||||||
confidence. That is the gate working, not a bug. Review and approve them.
|
Review and approve them.
|
||||||
|
|
||||||
## Checklist
|
## Checklist
|
||||||
|
|
||||||
@@ -96,7 +97,7 @@ confidence. That is the gate working, not a bug. Review and approve them.
|
|||||||
- [ ] Timestamps UTC; `regionScoped` correct; unstated ends are `null`
|
- [ ] Timestamps UTC; `regionScoped` correct; unstated ends are `null`
|
||||||
- [ ] Expected-output file + passing test, offline
|
- [ ] Expected-output file + passing test, offline
|
||||||
- [ ] Manually spot-checked against the live page
|
- [ ] Manually spot-checked against the live page
|
||||||
- [ ] Registered in the adapter index and `sources`
|
- [ ] Registered in `SOURCES` and the `sources` table
|
||||||
- [ ] Dry run inspected, then real run, then quarantine reviewed
|
- [ ] Dry run inspected, then real run, then quarantine reviewed
|
||||||
|
|
||||||
## When something does not fit
|
## When something does not fit
|
||||||
|
|||||||
Reference in New Issue
Block a user