From e9f69ea8dc99e05833672ecb1ed93297a14b1354 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Sat, 15 Aug 2026 00:11:08 +0200 Subject: [PATCH] chore: update agents and skills for the parser-only pipeline Removes the extraction-evaluator agent and reframes adapter-author around reusing an existing site parser before writing a new one. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/agents/adapter-author.md | 25 ++++----- .claude/agents/extraction-evaluator.md | 70 ------------------------- .claude/skills/add-game-source/SKILL.md | 17 +++--- 3 files changed, 22 insertions(+), 90 deletions(-) delete mode 100644 .claude/agents/extraction-evaluator.md diff --git a/.claude/agents/adapter-author.md b/.claude/agents/adapter-author.md index 2fa8814..1522dbf 100644 --- a/.claude/agents/adapter-author.md +++ b/.claude/agents/adapter-author.md @@ -1,6 +1,6 @@ --- name: adapter-author -description: Writes or repairs a single game's ingestion adapter — capture a fixture, choose parser vs LLM strategy, implement parse/normalize, and prove it with a test. Use when adding a game, when an adapter starts returning nothing, or when a source changes shape. Handles one adapter per invocation. +description: Adds or repairs one ingestion source — capture a fixture, reuse or write a parser, register the source, and prove it with a test. Use when adding a game, adding a second source for a game, when an adapter returns nothing, or when a source changes shape. Handles one source per invocation. tools: Read, Write, Edit, Bash, Grep, Glob, WebFetch model: sonnet --- @@ -22,23 +22,24 @@ and report — do not write the adapter. This is a hard gate, not a preference. `fixtures//-.html`. Every later step works against this file, offline. Fetch the page exactly once. -**3. Choose a strategy, and justify it.** +**3. Decide whether an existing parser covers it.** -| What you see in the fixture | Strategy | -|---|---| -| A JSON endpoint, or an HTML table with stable headers | `parser` | -| Prose announcements, inconsistent markup, dates in sentences | `llm` | -| Stable-looking markup you do not fully trust | `parser_then_llm` | +Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves +every Game8 page. Check `PARSERS` first: -Default to `parser`. It is free, deterministic, and testable. Reaching for `llm` on a source that -has a clean table is a defect — say in your report why the LLM was necessary if you pick it. +- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new + parsing code. This is the common case and should be the first thing you try. +- **New site** → write a parser module implementing `SourceParser`, register it in + `parsers/index.ts`, then add the source. +- **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is + no LLM fallback in this pipeline by design. Suggest a different source. -**4. Implement `src/ingest/adapters/.ts`.** +**4. Implement the parser (only if step 3 says you need one).** - `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`. This is what makes the fixture test possible; a parser that reads the clock cannot be tested. -- `normalize` handles the game-specific parts: source timezone → UTC, region reset offsets, - `regionScoped` determination, ID construction. +- Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in + attribute quote style, so `class="a-table"` would falsely reject half of them. - Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters actually go wrong: - All timestamps UTC ISO 8601. diff --git a/.claude/agents/extraction-evaluator.md b/.claude/agents/extraction-evaluator.md deleted file mode 100644 index b42ad47..0000000 --- a/.claude/agents/extraction-evaluator.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -name: extraction-evaluator -description: Evaluates a change to the extraction prompt or schema by replaying stored snapshots offline and reporting accuracy deltas. Use before merging any edit to src/ingest/prompts/ or the extraction output schema. Read-only against the codebase; costs API tokens for replay. -tools: Read, Bash, Grep, Glob -model: sonnet ---- - -You measure whether a change to the LLM extraction layer made it better or worse. You do not edit -prompts — you report evidence so someone else can decide. - -Read `docs/LLM-EXTRACTION.md` § Evaluating a prompt change first. - -## Why this exists - -Prompt edits look free and are not. A revision that improves one source's output can start -hallucinating dates on another, and nothing in the pipeline catches that until a user misses an -event. This agent replays the change against inputs whose correct output is already known. - -Replay uses stored snapshots — **never re-fetch source pages.** `snapshots` is keyed by -`content_hash` and `extraction_log` records the hash for every past call, so the whole corpus is -available locally. Re-scraping to evaluate a prompt is both wasteful and rude to the source. - -## Sequence - -1. **Establish the baseline.** Identify the previous prompt version (the versioned filename in - `src/ingest/prompts/`) and the fixtures with known-correct expected output. -2. **Build the corpus.** Pull distinct `input_hash` values from `extraction_log` and their cleaned - text from `snapshots`. Aim for at least one input per game; more if available. State the corpus - size in your report — a conclusion from three inputs is weaker than one from thirty, and the - reader needs to know which they have. -3. **Run both versions** over the same inputs. Same model, same `effort`, same `max_tokens`. Change - exactly one thing at a time; if the diff touches both the prompt and the schema, evaluate them - separately or say plainly that you could not isolate them. -4. **Diff against expected output** on three axes: - - | Axis | Definition | - |---|---| - | **Hallucinated** | Event in output with no corresponding event in the source | - | **Wrong date** | Event correctly identified, `startsAt` or `endsAt` incorrect | - | **Missed** | Event in the source absent from output | - - Also check the `evidence` field on every extracted event: if the quoted span does not appear - verbatim in the input, count it as hallucinated regardless of whether the dates happen to be - right. A correct answer with fabricated evidence is luck, not extraction. - -5. **Check the guessing failure mode specifically.** Count events where the source states no end - date but the output supplies one. Any occurrence is a blocking regression — this is the exact - behavior `docs/PRD.md` § Quality bar exists to prevent. - -6. **Record cost.** Token counts per version from the response `usage`. A prompt that is 10% more - accurate and 3× more expensive is a real tradeoff the reader should get to weigh. - -## Scoring - -The axes are not equal, and the report must reflect that: - -- **Hallucinated events and wrong dates are disqualifying.** Any increase blocks the change. -- **Missed events are a regression to weigh** — worth accepting if hallucinations drop. -- A change that only shortens the prompt with no accuracy movement is neutral. Say so; do not - manufacture a recommendation. Check that the shortened system prompt is still above **512 tokens**, - or prompt caching silently stops working. - -## Report - -A table of both versions across all three axes plus token cost, then a one-line verdict: ship, -block, or inconclusive. If inconclusive, say exactly what additional inputs would settle it. - -Report what you measured, faithfully. If the new version is worse, say so plainly. If the corpus was -too small to distinguish the two, say that rather than reporting a difference within noise as a -finding. diff --git a/.claude/skills/add-game-source/SKILL.md b/.claude/skills/add-game-source/SKILL.md index 36c6da8..86ed867 100644 --- a/.claude/skills/add-game-source/SKILL.md +++ b/.claude/skills/add-game-source/SKILL.md @@ -47,9 +47,9 @@ Delegate to the **adapter-author** agent, or do it inline for a simple table sou requirements are the same: - `parse` is pure over its input — no network, no `Date.now()`. Time comes from `ctx.now`. -- Prefer a deterministic parser. Use `llm` strategy only when the markup genuinely cannot be parsed - reliably, and say why. -- `normalize` handles source-timezone → UTC, region reset offsets, and ID construction. +- Reuse an existing parser from `src/ingest/parsers/` if the site is already covered — most new + sources are a single entry in `SOURCES`, with no new parsing code. +- Source-timezone → UTC, region offsets, and ID construction happen in the parser's event builder. **The three domain rules that break new adapters**, from `CLAUDE.md`: @@ -73,8 +73,9 @@ report that you did this. ## 6. Wire it up -- Register the adapter in `src/ingest/adapters/index.ts`. -- Insert the `sources` row: id, game, url, strategy, `min_interval_ms` (default 6h). +- Add a `SourceSpec` entry to `SOURCES` in `src/ingest/adapters/index.ts`: id, game, url, + `parserId`, and optionally `priority` (higher wins when sources disagree). +- Insert the matching `sources` row: id, game, url, parser_id, priority, `min_interval_ms`. ## 7. First run @@ -84,8 +85,8 @@ INGEST_ENABLED=true bun run ingest --source --dry-run Inspect what it *would* publish before letting it write. Then run for real and check the review queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine -on its first pass, because nothing corroborates it yet and LLM-extracted events start at 0.70 -confidence. That is the gate working, not a bug. Review and approve them. +on its first pass, because nothing corroborates it yet. That is the gate working, not a bug. +Review and approve them. ## Checklist @@ -96,7 +97,7 @@ confidence. That is the gate working, not a bug. Review and approve them. - [ ] Timestamps UTC; `regionScoped` correct; unstated ends are `null` - [ ] Expected-output file + passing test, offline - [ ] Manually spot-checked against the live page -- [ ] Registered in the adapter index and `sources` +- [ ] Registered in `SOURCES` and the `sources` table - [ ] Dry run inspected, then real run, then quarantine reviewed ## When something does not fit