chore: update agents and skills for the parser-only pipeline
Removes the extraction-evaluator agent and reframes adapter-author around reusing an existing site parser before writing a new one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5
Claude Opus 5
parent
3ea7286f56
commit
e9f69ea8dc
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: adapter-author
|
||||
description: Writes or repairs a single game's ingestion adapter — capture a fixture, choose parser vs LLM strategy, implement parse/normalize, and prove it with a test. Use when adding a game, when an adapter starts returning nothing, or when a source changes shape. Handles one adapter per invocation.
|
||||
description: Adds or repairs one ingestion source — capture a fixture, reuse or write a parser, register the source, and prove it with a test. Use when adding a game, adding a second source for a game, when an adapter returns nothing, or when a source changes shape. Handles one source per invocation.
|
||||
tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch
|
||||
model: sonnet
|
||||
---
|
||||
@@ -22,23 +22,24 @@ and report — do not write the adapter. This is a hard gate, not a preference.
|
||||
`fixtures/<game>/<source-id>-<YYYY-MM-DD>.html`. Every later step works against this file, offline.
|
||||
Fetch the page exactly once.
|
||||
|
||||
**3. Choose a strategy, and justify it.**
|
||||
**3. Decide whether an existing parser covers it.**
|
||||
|
||||
| What you see in the fixture | Strategy |
|
||||
|---|---|
|
||||
| A JSON endpoint, or an HTML table with stable headers | `parser` |
|
||||
| Prose announcements, inconsistent markup, dates in sentences | `llm` |
|
||||
| Stable-looking markup you do not fully trust | `parser_then_llm` |
|
||||
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves
|
||||
every Game8 page. Check `PARSERS` first:
|
||||
|
||||
Default to `parser`. It is free, deterministic, and testable. Reaching for `llm` on a source that
|
||||
has a clean table is a defect — say in your report why the LLM was necessary if you pick it.
|
||||
- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
|
||||
parsing code. This is the common case and should be the first thing you try.
|
||||
- **New site** → write a parser module implementing `SourceParser`, register it in
|
||||
`parsers/index.ts`, then add the source.
|
||||
- **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is
|
||||
no LLM fallback in this pipeline by design. Suggest a different source.
|
||||
|
||||
**4. Implement `src/ingest/adapters/<game>.ts`.**
|
||||
**4. Implement the parser (only if step 3 says you need one).**
|
||||
|
||||
- `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`.
|
||||
This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
|
||||
- `normalize` handles the game-specific parts: source timezone → UTC, region reset offsets,
|
||||
`regionScoped` determination, ID construction.
|
||||
- Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in
|
||||
attribute quote style, so `class="a-table"` would falsely reject half of them.
|
||||
- Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters
|
||||
actually go wrong:
|
||||
- All timestamps UTC ISO 8601.
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
name: extraction-evaluator
|
||||
description: Evaluates a change to the extraction prompt or schema by replaying stored snapshots offline and reporting accuracy deltas. Use before merging any edit to src/ingest/prompts/ or the extraction output schema. Read-only against the codebase; costs API tokens for replay.
|
||||
tools: Read, Bash, Grep, Glob
|
||||
model: sonnet
|
||||
---
|
||||
|
||||
You measure whether a change to the LLM extraction layer made it better or worse. You do not edit
|
||||
prompts — you report evidence so someone else can decide.
|
||||
|
||||
Read `docs/LLM-EXTRACTION.md` § Evaluating a prompt change first.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Prompt edits look free and are not. A revision that improves one source's output can start
|
||||
hallucinating dates on another, and nothing in the pipeline catches that until a user misses an
|
||||
event. This agent replays the change against inputs whose correct output is already known.
|
||||
|
||||
Replay uses stored snapshots — **never re-fetch source pages.** `snapshots` is keyed by
|
||||
`content_hash` and `extraction_log` records the hash for every past call, so the whole corpus is
|
||||
available locally. Re-scraping to evaluate a prompt is both wasteful and rude to the source.
|
||||
|
||||
## Sequence
|
||||
|
||||
1. **Establish the baseline.** Identify the previous prompt version (the versioned filename in
|
||||
`src/ingest/prompts/`) and the fixtures with known-correct expected output.
|
||||
2. **Build the corpus.** Pull distinct `input_hash` values from `extraction_log` and their cleaned
|
||||
text from `snapshots`. Aim for at least one input per game; more if available. State the corpus
|
||||
size in your report — a conclusion from three inputs is weaker than one from thirty, and the
|
||||
reader needs to know which they have.
|
||||
3. **Run both versions** over the same inputs. Same model, same `effort`, same `max_tokens`. Change
|
||||
exactly one thing at a time; if the diff touches both the prompt and the schema, evaluate them
|
||||
separately or say plainly that you could not isolate them.
|
||||
4. **Diff against expected output** on three axes:
|
||||
|
||||
| Axis | Definition |
|
||||
|---|---|
|
||||
| **Hallucinated** | Event in output with no corresponding event in the source |
|
||||
| **Wrong date** | Event correctly identified, `startsAt` or `endsAt` incorrect |
|
||||
| **Missed** | Event in the source absent from output |
|
||||
|
||||
Also check the `evidence` field on every extracted event: if the quoted span does not appear
|
||||
verbatim in the input, count it as hallucinated regardless of whether the dates happen to be
|
||||
right. A correct answer with fabricated evidence is luck, not extraction.
|
||||
|
||||
5. **Check the guessing failure mode specifically.** Count events where the source states no end
|
||||
date but the output supplies one. Any occurrence is a blocking regression — this is the exact
|
||||
behavior `docs/PRD.md` § Quality bar exists to prevent.
|
||||
|
||||
6. **Record cost.** Token counts per version from the response `usage`. A prompt that is 10% more
|
||||
accurate and 3× more expensive is a real tradeoff the reader should get to weigh.
|
||||
|
||||
## Scoring
|
||||
|
||||
The axes are not equal, and the report must reflect that:
|
||||
|
||||
- **Hallucinated events and wrong dates are disqualifying.** Any increase blocks the change.
|
||||
- **Missed events are a regression to weigh** — worth accepting if hallucinations drop.
|
||||
- A change that only shortens the prompt with no accuracy movement is neutral. Say so; do not
|
||||
manufacture a recommendation. Check that the shortened system prompt is still above **512 tokens**,
|
||||
or prompt caching silently stops working.
|
||||
|
||||
## Report
|
||||
|
||||
A table of both versions across all three axes plus token cost, then a one-line verdict: ship,
|
||||
block, or inconclusive. If inconclusive, say exactly what additional inputs would settle it.
|
||||
|
||||
Report what you measured, faithfully. If the new version is worse, say so plainly. If the corpus was
|
||||
too small to distinguish the two, say that rather than reporting a difference within noise as a
|
||||
finding.
|
||||
Reference in New Issue
Block a user