Initial agent auxilliary files.
This commit is contained in:
Lucas Winther
@@ -0,0 +1,291 @@
|
||||
# LLM Extraction
|
||||
|
||||
Stage 3 of the pipeline, for sources whose markup is too unstable to parse deterministically. Read
|
||||
`docs/INGESTION.md` first for where this sits.
|
||||
|
||||
## Rules
|
||||
|
||||
1. **The model runs at ingestion time only.** No request path calls Anthropic. A page load must
|
||||
never trigger inference.
|
||||
2. **Deterministic parsers win.** If a source has a JSON API or a stable table, it gets a parser,
|
||||
not a prompt.
|
||||
3. **The model's job is transcription, not judgment.** It converts prose and tables into structured
|
||||
dates. It does not decide what is important, does not infer missing dates, and does not resolve
|
||||
contradictions — it reports them.
|
||||
4. **Confidence is computed from evidence in `reconcile`, not asserted by the model.** The output
|
||||
schema has no confidence field. A model claiming 0.95 confidence has predicted a token, not
|
||||
measured anything.
|
||||
|
||||
## Model and parameters
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---|---|---|
|
||||
| `model` | `claude-opus-5` | Exact, complete ID — never append a date suffix |
|
||||
| `max_tokens` | `16000` | Non-streaming; keeps the request under SDK HTTP timeouts |
|
||||
| `output_config.effort` | `"medium"` | Transcription, not reasoning. Sweep low/medium/high against fixtures before settling |
|
||||
| `output_config.format` | `zodOutputFormat(ExtractionResult)` | Schema-conformant output, validated by the SDK |
|
||||
| `thinking` | *omit* | On by default on `claude-opus-5`; the default is correct here |
|
||||
| `temperature` / `top_p` / `top_k` | **never set** | Removed on `claude-opus-5` — sending any of them returns 400 |
|
||||
|
||||
`thinking: {type: "enabled", budget_tokens: N}` is also removed and returns 400. If you want less
|
||||
thinking, lower `effort`.
|
||||
|
||||
Note that `max_tokens` caps thinking *and* output together. If extraction on a large page returns
|
||||
`stop_reason: "max_tokens"`, raise it rather than trimming the schema.
|
||||
|
||||
## The output schema
|
||||
|
||||
The model returns a list of candidate events plus explicit uncertainty. Note what is **absent**:
|
||||
no confidence score, no "importance" ranking, no summary of the page.
|
||||
|
||||
```ts
|
||||
// src/ingest/extract.ts
|
||||
import { z } from "zod";
|
||||
|
||||
const ExtractedEvent = z.object({
|
||||
title: z.string().describe("The event name exactly as written in the source, not paraphrased."),
|
||||
type: z.enum(["banner","story","rerun","challenge","login","shop","maintenance","other"]),
|
||||
summary: z.string().nullable()
|
||||
.describe("One sentence from the source describing the event. Null if the source gives none."),
|
||||
|
||||
startsAt: z.string().nullable()
|
||||
.describe("ISO 8601 UTC. Null only if the source truly does not state a start."),
|
||||
startPrecision: z.enum(["exact","day","unknown"])
|
||||
.describe("'exact' if a time of day is stated; 'day' if only a date; 'unknown' if neither."),
|
||||
|
||||
endsAt: z.string().nullable()
|
||||
.describe("ISO 8601 UTC. Null when the source says TBD, 'until further notice', or gives no end."),
|
||||
endPrecision: z.enum(["exact","day","unknown"]),
|
||||
|
||||
regionScoped: z.boolean()
|
||||
.describe("True if the end follows each server region's daily reset rather than one global instant."),
|
||||
|
||||
sourceTimezone: z.string().nullable()
|
||||
.describe("The timezone the source stated, e.g. 'UTC+8', 'server local'. Null if unstated."),
|
||||
evidence: z.string()
|
||||
.describe("The verbatim span from the input that gave you the dates. Must appear in the input."),
|
||||
});
|
||||
|
||||
const ExtractionResult = z.object({
|
||||
events: z.array(ExtractedEvent),
|
||||
ambiguities: z.array(z.object({
|
||||
title: z.string(),
|
||||
issue: z.string().describe("What is unclear or contradictory in the source."),
|
||||
})).describe("Events you could not confidently transcribe. These are held for human review."),
|
||||
});
|
||||
```
|
||||
|
||||
`evidence` is the load-bearing field. It is checked in `validate`: if the quoted span does not
|
||||
appear in the input text, the event is quarantined as `sanity_failed`. That check is what turns a
|
||||
fabricated date into a caught error instead of a shipped one.
|
||||
|
||||
## The system prompt
|
||||
|
||||
Kept in `src/ingest/prompts/extract-events.v1.md`, versioned in the filename, and logged as
|
||||
`prompt_version` in `extraction_log` so a regression can be traced to a specific revision.
|
||||
|
||||
It must stay **above 512 tokens** — that is the minimum cacheable prefix on `claude-opus-5`. Below
|
||||
it, prompt caching silently stops working with no error. After any prompt edit, check
|
||||
`usage.cache_read_input_tokens` is non-zero on the second call.
|
||||
|
||||
```markdown
|
||||
You extract scheduled in-game events from gacha game source pages into structured data.
|
||||
|
||||
Your output is consumed by a calendar that players rely on to avoid missing limited-time content.
|
||||
A wrong end date is worse than a missing event: a missing event sends someone to a wiki, a wrong
|
||||
one makes them miss content permanently. Transcribe what the source says; never supply what it
|
||||
omits.
|
||||
|
||||
## What counts as an event
|
||||
|
||||
Anything with a start and a bounded or open-ended run: character and weapon banners, story
|
||||
chapters, side events, login campaigns, limited shops, combat cycles, announced maintenance.
|
||||
|
||||
Not events: permanent features, general game descriptions, patch version numbers on their own,
|
||||
speculation or leaks, community posts, and anything phrased as expected, rumored, or datamined.
|
||||
|
||||
## Dates
|
||||
|
||||
- Emit UTC ISO 8601 with an explicit `Z`.
|
||||
- When the source states a timezone (commonly UTC+8 for Chinese-developed titles), convert to UTC
|
||||
and record what it stated in `sourceTimezone`.
|
||||
- When only a date is given, set the timestamp to 00:00:00Z and `precision: "day"`. Do not guess a
|
||||
time of day.
|
||||
- When the source says the end is TBD, "until further notice", "with the next version update", or
|
||||
gives no end at all: `endsAt: null` and `endPrecision: "unknown"`. This is a correct, expected
|
||||
answer. Do not compute a plausible date from a typical patch length.
|
||||
- `regionScoped` is true when the end is tied to each server's daily reset, false when the source
|
||||
gives one simultaneous global instant. Character banners are usually global; story and login
|
||||
events are usually region-scoped. Use what the source says over this heuristic when it says
|
||||
anything.
|
||||
|
||||
## Evidence
|
||||
|
||||
For every event, `evidence` must be a verbatim span copied from the input that contains the dates
|
||||
you reported. It is checked against the input automatically. If you cannot quote a span, the event
|
||||
belongs in `ambiguities` instead.
|
||||
|
||||
## Ambiguities
|
||||
|
||||
Put an entry in `ambiguities` — not in `events` — when the source contradicts itself, gives dates
|
||||
you cannot reconcile, or describes something that may not be a scheduled event. A human reviews
|
||||
these. Reporting uncertainty is a successful outcome, not a failure; guessing to avoid it is the
|
||||
one thing that breaks this system.
|
||||
|
||||
## Scope
|
||||
|
||||
Report every qualifying event on the page and nothing else. Do not rank them, do not summarize the
|
||||
page, do not comment on your process, and do not add fields the schema does not have. If the page
|
||||
contains no events, return empty arrays.
|
||||
```
|
||||
|
||||
### Why the prompt reads the way it does
|
||||
|
||||
`claude-opus-5` follows instructions literally and verifies its own work without being told, so the
|
||||
prompt states scope and boundaries plainly instead of adding emphasis or self-check scaffolding.
|
||||
Specifically: **do not add "double-check your answer" or "verify before responding"** here. On this
|
||||
model that produces over-verification with no accuracy gain. If extraction quality drops, change the
|
||||
schema descriptions or `effort` — not the volume of the prompt.
|
||||
|
||||
## Request shape
|
||||
|
||||
```ts
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
|
||||
|
||||
const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env
|
||||
|
||||
const response = await client.messages.parse({
|
||||
model: "claude-opus-5",
|
||||
max_tokens: 16000,
|
||||
system: [
|
||||
{
|
||||
type: "text",
|
||||
text: EXTRACTION_SYSTEM_PROMPT, // stable across every source — cached
|
||||
cache_control: { type: "ephemeral", ttl: "1h" },
|
||||
},
|
||||
],
|
||||
output_config: {
|
||||
effort: "medium",
|
||||
format: zodOutputFormat(ExtractionResult),
|
||||
},
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: [
|
||||
`Game: ${adapter.game}`,
|
||||
`Source: ${adapter.url}`,
|
||||
`Today (UTC): ${ctx.now}`,
|
||||
adapter.extractionHints ?? "",
|
||||
"",
|
||||
"--- SOURCE TEXT ---",
|
||||
cleanedText,
|
||||
].join("\n"),
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
// parsed_output is null if the model refused or hit max_tokens — check before use.
|
||||
const result = response.parsed_output;
|
||||
```
|
||||
|
||||
**Ordering matters for caching.** The system prompt is byte-identical across every source, so it
|
||||
sits first and stays cached. Everything volatile — game, URL, `now`, the page text — goes in the
|
||||
user turn, after the cache breakpoint. Interpolating `ctx.now` into the system prompt would
|
||||
invalidate the cache on every single call; it is in the user turn for exactly that reason.
|
||||
|
||||
## Batch mode
|
||||
|
||||
`EXTRACTION_MODE=batch` is the default for scheduled runs: 50% cheaper, results typically within an
|
||||
hour, which is irrelevant against a 6-hour cadence. The synchronous path above is for
|
||||
`refresh --now` and for local development.
|
||||
|
||||
```ts
|
||||
const batch = await client.messages.batches.create({
|
||||
requests: sourcesToExtract.map((s) => ({
|
||||
custom_id: s.runId, // key results by this — order is not guaranteed
|
||||
params: { /* same body as above */ },
|
||||
})),
|
||||
});
|
||||
|
||||
// Poll batches.retrieve(batch.id) until processing_status === "ended",
|
||||
// then stream batches.results(batch.id) and key each result by custom_id.
|
||||
```
|
||||
|
||||
Results arrive in **any order**. Key by `custom_id`, never by array position. The batch job's
|
||||
poll loop lives in `scheduler.ts` and persists `batch.id` so a process restart resumes rather than
|
||||
resubmitting.
|
||||
|
||||
## Handling non-success responses
|
||||
|
||||
Check `stop_reason` before touching `parsed_output`:
|
||||
|
||||
| `stop_reason` | Meaning | Action |
|
||||
|---|---|---|
|
||||
| `end_turn` | Normal | Proceed |
|
||||
| `max_tokens` | Output truncated; `parsed_output` unusable | Raise `max_tokens`, or split the page by section. Fail the run — never publish a partial list |
|
||||
| `refusal` | Safety classifier declined | Log `stop_details.category` to `extraction_log.refusal_category`, fail the run, alert. Vanishingly unlikely for game wiki content — if it fires, the input is probably not what you think it is |
|
||||
|
||||
`stop_details` can be `null` even on a refusal, so branch on `stop_reason` and treat `stop_details`
|
||||
as informational.
|
||||
|
||||
**Do not write a JSON-repair or regex-extraction fallback.** Structured outputs guarantee schema
|
||||
conformance; if parsing fails, the response was truncated or refused, and both cases are handled
|
||||
above. A repair path would silently paper over truncation and publish half a calendar.
|
||||
|
||||
## Cost
|
||||
|
||||
At `claude-opus-5` rates — $5/MTok input, $25/MTok output.
|
||||
|
||||
One extraction of a cleaned wiki page:
|
||||
|
||||
```
|
||||
input ~5,000 tok × $5/MTok = $0.025
|
||||
output ~1,500 tok × $25/MTok = $0.0375
|
||||
───────
|
||||
sync $0.063
|
||||
batch $0.031 (50% off)
|
||||
```
|
||||
|
||||
Six sources, four runs a day:
|
||||
|
||||
| Scenario | Extractions/day | Cost/day | Cost/month |
|
||||
|---|---|---|---|
|
||||
| Realistic — ~20% of runs see changed content | ~5 | $0.16 | **~$4.70** |
|
||||
| Worst case — every source changes every run | 24 | $0.74 | ~$22 |
|
||||
| No content-hash skip, no batch | 24 | $1.51 | ~$45 |
|
||||
|
||||
The gap between rows one and three is the whole argument for the skip check and batch mode. Track
|
||||
actuals by summing `extraction_log` token columns — do not rely on these estimates once there is
|
||||
real data.
|
||||
|
||||
Prompt caching contributes modestly (the ~1k-token system prompt at 0.1× on reads) but is free to
|
||||
keep. Its real value is that it makes the prompt cheap to grow if extraction quality needs more
|
||||
instruction.
|
||||
|
||||
## Evaluating a prompt change
|
||||
|
||||
`extraction_log` stores `input_hash` for every call, and `snapshots` stores the cleaned text keyed
|
||||
by the same hash. So a prompt revision is evaluated by replaying past inputs — no re-fetching, no
|
||||
new scraping load:
|
||||
|
||||
1. Pull the last N distinct `input_hash` values with known-correct expected output.
|
||||
2. Run the new prompt against each cleaned snapshot.
|
||||
3. Diff against `fixtures/*/*.expected.json`.
|
||||
4. Compare on three axes: dates correct, events missed, events hallucinated. **Hallucinated events
|
||||
and wrong dates are disqualifying; a missed event is a regression to weigh.** That asymmetry is
|
||||
the product rule from `docs/PRD.md` restated as an eval criterion.
|
||||
|
||||
Bump the prompt filename version and record it as `prompt_version` so the log distinguishes
|
||||
"extraction got worse" from "the source changed".
|
||||
|
||||
## Things not to do
|
||||
|
||||
- Do not ask the model to output a confidence score. Confidence is computed in `reconcile`.
|
||||
- Do not ask the model to resolve a contradiction between two sources. Route it to quarantine.
|
||||
- Do not send raw HTML. Always clean first — it is a 3× cost difference and it improves accuracy.
|
||||
- Do not add a second model call to "verify" the first. That is a scaffolding pattern this model
|
||||
does not need, and it doubles cost for no measured gain. The evidence-span check and the
|
||||
validator rules are the verification layer.
|
||||
- Do not lower `max_tokens` to save money. Output tokens scale with the number of events found;
|
||||
truncation costs a whole run.
|
||||
Reference in New Issue
Block a user