gacha-event-tracker/docs/LLM-EXTRACTION.md

# LLM Extraction

Stage 3 of the pipeline, for sources whose markup is too unstable to parse deterministically. Read
`docs/INGESTION.md` first for where this sits.

## Rules

1. **The model runs at ingestion time only.** No request path calls Anthropic. A page load must
   never trigger inference.
2. **Deterministic parsers win.** If a source has a JSON API or a stable table, it gets a parser,
   not a prompt.
3. **The model's job is transcription, not judgment.** It converts prose and tables into structured
   dates. It does not decide what is important, does not infer missing dates, and does not resolve
   contradictions — it reports them.
4. **Confidence is computed from evidence in `reconcile`, not asserted by the model.** The output
   schema has no confidence field. A model claiming 0.95 confidence has predicted a token, not
   measured anything.

## Model and parameters

| Setting | Value | Why |
|---|---|---|
| `model` | `claude-opus-5` | Exact, complete ID — never append a date suffix |
| `max_tokens` | `16000` | Non-streaming; keeps the request under SDK HTTP timeouts |
| `output_config.effort` | `"medium"` | Transcription, not reasoning. Sweep low/medium/high against fixtures before settling |
| `output_config.format` | `zodOutputFormat(ExtractionResult)` | Schema-conformant output, validated by the SDK |
| `thinking` | *omit* | On by default on `claude-opus-5`; the default is correct here |
| `temperature` / `top_p` / `top_k` | **never set** | Removed on `claude-opus-5` — sending any of them returns 400 |

`thinking: {type: "enabled", budget_tokens: N}` is also removed and returns 400. If you want less
thinking, lower `effort`.

Note that `max_tokens` caps thinking *and* output together. If extraction on a large page returns
`stop_reason: "max_tokens"`, raise it rather than trimming the schema.

## The output schema

The model returns a list of candidate events plus explicit uncertainty. Note what is **absent**:
no confidence score, no "importance" ranking, no summary of the page.

```ts
// src/ingest/extract.ts
import { z } from "zod";

const ExtractedEvent = z.object({
  title: z.string().describe("The event name exactly as written in the source, not paraphrased."),
  type: z.enum(["banner","story","rerun","challenge","login","shop","maintenance","other"]),
  summary: z.string().nullable()
    .describe("One sentence from the source describing the event. Null if the source gives none."),

  startsAt: z.string().nullable()
    .describe("ISO 8601 UTC. Null only if the source truly does not state a start."),
  startPrecision: z.enum(["exact","day","unknown"])
    .describe("'exact' if a time of day is stated; 'day' if only a date; 'unknown' if neither."),

  endsAt: z.string().nullable()
    .describe("ISO 8601 UTC. Null when the source says TBD, 'until further notice', or gives no end."),
  endPrecision: z.enum(["exact","day","unknown"]),

  regionScoped: z.boolean()
    .describe("True if the end follows each server region's daily reset rather than one global instant."),

  sourceTimezone: z.string().nullable()
    .describe("The timezone the source stated, e.g. 'UTC+8', 'server local'. Null if unstated."),
  evidence: z.string()
    .describe("The verbatim span from the input that gave you the dates. Must appear in the input."),
});

const ExtractionResult = z.object({
  events: z.array(ExtractedEvent),
  ambiguities: z.array(z.object({
    title: z.string(),
    issue: z.string().describe("What is unclear or contradictory in the source."),
  })).describe("Events you could not confidently transcribe. These are held for human review."),
});
```

`evidence` is the load-bearing field. It is checked in `validate`: if the quoted span does not
appear in the input text, the event is quarantined as `sanity_failed`. That check is what turns a
fabricated date into a caught error instead of a shipped one.

## The system prompt

Kept in `src/ingest/prompts/extract-events.v1.md`, versioned in the filename, and logged as
`prompt_version` in `extraction_log` so a regression can be traced to a specific revision.

It must stay **above 512 tokens** — that is the minimum cacheable prefix on `claude-opus-5`. Below
it, prompt caching silently stops working with no error. After any prompt edit, check
`usage.cache_read_input_tokens` is non-zero on the second call.

```markdown
You extract scheduled in-game events from gacha game source pages into structured data.

Your output is consumed by a calendar that players rely on to avoid missing limited-time content.
A wrong end date is worse than a missing event: a missing event sends someone to a wiki, a wrong
one makes them miss content permanently. Transcribe what the source says; never supply what it
omits.

## What counts as an event

Anything with a start and a bounded or open-ended run: character and weapon banners, story
chapters, side events, login campaigns, limited shops, combat cycles, announced maintenance.

Not events: permanent features, general game descriptions, patch version numbers on their own,
speculation or leaks, community posts, and anything phrased as expected, rumored, or datamined.

## Dates

- Emit UTC ISO 8601 with an explicit `Z`.
- When the source states a timezone (commonly UTC+8 for Chinese-developed titles), convert to UTC
  and record what it stated in `sourceTimezone`.
- When only a date is given, set the timestamp to 00:00:00Z and `precision: "day"`. Do not guess a
  time of day.
- When the source says the end is TBD, "until further notice", "with the next version update", or
  gives no end at all: `endsAt: null` and `endPrecision: "unknown"`. This is a correct, expected
  answer. Do not compute a plausible date from a typical patch length.
- `regionScoped` is true when the end is tied to each server's daily reset, false when the source
  gives one simultaneous global instant. Character banners are usually global; story and login
  events are usually region-scoped. Use what the source says over this heuristic when it says
  anything.

## Evidence

For every event, `evidence` must be a verbatim span copied from the input that contains the dates
you reported. It is checked against the input automatically. If you cannot quote a span, the event
belongs in `ambiguities` instead.

## Ambiguities

Put an entry in `ambiguities` — not in `events` — when the source contradicts itself, gives dates
you cannot reconcile, or describes something that may not be a scheduled event. A human reviews
these. Reporting uncertainty is a successful outcome, not a failure; guessing to avoid it is the
one thing that breaks this system.

## Scope

Report every qualifying event on the page and nothing else. Do not rank them, do not summarize the
page, do not comment on your process, and do not add fields the schema does not have. If the page
contains no events, return empty arrays.
```

### Why the prompt reads the way it does

`claude-opus-5` follows instructions literally and verifies its own work without being told, so the
prompt states scope and boundaries plainly instead of adding emphasis or self-check scaffolding.
Specifically: **do not add "double-check your answer" or "verify before responding"** here. On this
model that produces over-verification with no accuracy gain. If extraction quality drops, change the
schema descriptions or `effort` — not the volume of the prompt.

## Request shape

```ts
import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";

const client = new Anthropic();  // reads ANTHROPIC_API_KEY from env

const response = await client.messages.parse({
  model: "claude-opus-5",
  max_tokens: 16000,
  system: [
    {
      type: "text",
      text: EXTRACTION_SYSTEM_PROMPT,          // stable across every source — cached
      cache_control: { type: "ephemeral", ttl: "1h" },
    },
  ],
  output_config: {
    effort: "medium",
    format: zodOutputFormat(ExtractionResult),
  },
  messages: [
    {
      role: "user",
      content: [
        `Game: ${adapter.game}`,
        `Source: ${adapter.url}`,
        `Today (UTC): ${ctx.now}`,
        adapter.extractionHints ?? "",
        "",
        "--- SOURCE TEXT ---",
        cleanedText,
      ].join("\n"),
    },
  ],
});

// parsed_output is null if the model refused or hit max_tokens — check before use.
const result = response.parsed_output;
```

**Ordering matters for caching.** The system prompt is byte-identical across every source, so it
sits first and stays cached. Everything volatile — game, URL, `now`, the page text — goes in the
user turn, after the cache breakpoint. Interpolating `ctx.now` into the system prompt would
invalidate the cache on every single call; it is in the user turn for exactly that reason.

## Batch mode

`EXTRACTION_MODE=batch` is the default for scheduled runs: 50% cheaper, results typically within an
hour, which is irrelevant against a 6-hour cadence. The synchronous path above is for
`refresh --now` and for local development.

```ts
const batch = await client.messages.batches.create({
  requests: sourcesToExtract.map((s) => ({
    custom_id: s.runId,                    // key results by this — order is not guaranteed
    params: { /* same body as above */ },
  })),
});

// Poll batches.retrieve(batch.id) until processing_status === "ended",
// then stream batches.results(batch.id) and key each result by custom_id.
```

Results arrive in **any order**. Key by `custom_id`, never by array position. The batch job's
poll loop lives in `scheduler.ts` and persists `batch.id` so a process restart resumes rather than
resubmitting.

## Handling non-success responses

Check `stop_reason` before touching `parsed_output`:

| `stop_reason` | Meaning | Action |
|---|---|---|
| `end_turn` | Normal | Proceed |
| `max_tokens` | Output truncated; `parsed_output` unusable | Raise `max_tokens`, or split the page by section. Fail the run — never publish a partial list |
| `refusal` | Safety classifier declined | Log `stop_details.category` to `extraction_log.refusal_category`, fail the run, alert. Vanishingly unlikely for game wiki content — if it fires, the input is probably not what you think it is |

`stop_details` can be `null` even on a refusal, so branch on `stop_reason` and treat `stop_details`
as informational.

**Do not write a JSON-repair or regex-extraction fallback.** Structured outputs guarantee schema
conformance; if parsing fails, the response was truncated or refused, and both cases are handled
above. A repair path would silently paper over truncation and publish half a calendar.

## Cost

At `claude-opus-5` rates — $5/MTok input, $25/MTok output.

One extraction of a cleaned wiki page:

```
input   ~5,000 tok  × $5/MTok   = $0.025
output  ~1,500 tok  × $25/MTok  = $0.0375
                                  ───────
                          sync    $0.063
                          batch   $0.031   (50% off)
```

Six sources, four runs a day:

| Scenario | Extractions/day | Cost/day | Cost/month |
|---|---|---|---|
| Realistic — ~20% of runs see changed content | ~5 | $0.16 | **~$4.70** |
| Worst case — every source changes every run | 24 | $0.74 | ~$22 |
| No content-hash skip, no batch | 24 | $1.51 | ~$45 |

The gap between rows one and three is the whole argument for the skip check and batch mode. Track
actuals by summing `extraction_log` token columns — do not rely on these estimates once there is
real data.

Prompt caching contributes modestly (the ~1k-token system prompt at 0.1× on reads) but is free to
keep. Its real value is that it makes the prompt cheap to grow if extraction quality needs more
instruction.

## Evaluating a prompt change

`extraction_log` stores `input_hash` for every call, and `snapshots` stores the cleaned text keyed
by the same hash. So a prompt revision is evaluated by replaying past inputs — no re-fetching, no
new scraping load:

1. Pull the last N distinct `input_hash` values with known-correct expected output.
2. Run the new prompt against each cleaned snapshot.
3. Diff against `fixtures/*/*.expected.json`.
4. Compare on three axes: dates correct, events missed, events hallucinated. **Hallucinated events
   and wrong dates are disqualifying; a missed event is a regression to weigh.** That asymmetry is
   the product rule from `docs/PRD.md` restated as an eval criterion.

Bump the prompt filename version and record it as `prompt_version` so the log distinguishes
"extraction got worse" from "the source changed".

## Things not to do

- Do not ask the model to output a confidence score. Confidence is computed in `reconcile`.
- Do not ask the model to resolve a contradiction between two sources. Route it to quarantine.
- Do not send raw HTML. Always clean first — it is a 3× cost difference and it improves accuracy.
- Do not add a second model call to "verify" the first. That is a scaffolding pattern this model
  does not need, and it doubles cost for no measured gain. The evidence-span check and the
  validator rules are the verification layer.
- Do not lower `max_tokens` to save money. Output tokens scale with the number of events found;
  truncation costs a whole run.