lucasw89/gacha-event-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial agent auxilliary files.

Browse Source
This commit is contained in:
Lucas Winther
2026-08-14 23:45:35 +02:00
parent 308f288e6a
commit 5db608563f
16 changed files with 1662 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+161
View File
@@ -0,0 +1,161 @@
# Architecture
## Shape
One Bun process serves the static React build, exposes a read-only JSON API, and runs the
ingestion scheduler on a timer. SQLite is the only datastore. There is no auth layer because there
are no users.
```
┌──────────────────────────────────────────────┐
game wikis ───► │ Bun process │
news pages │ │
│ scheduler (every 6h) │
│ │ │
│ ▼ │
│ ingest pipeline │
│ fetch → clean → parse|extract → validate │
│ → reconcile → gate → publish │
│ │ │ │
│ │ └──► quarantine │
│ ▼ │ │
│ ┌─────────────┐ │ │
│ │ SQLite │◄────────────────┘ │
│ └─────────────┘ /review (127.0.0.1) │
│ │ │
│ ▼ │
│ GET /api/events static React build │
└────────┬─────────────────────────┬───────────┘
│ │
▼ ▼
browser fetch browser render
│
▼
localStorage: completions,
filters, region ← never leaves the device
```
Anthropic API calls happen only inside the ingest pipeline. **No request path — not `/api/events`,
not a page load — ever calls the model.** If a feature seems to need live inference, it needs a
precomputed field instead.
## Layout
```
src/
server/
index.ts Bun.serve entry: static + API + scheduler bootstrap
routes/
events.ts GET /api/events, /api/events.json
games.ts GET /api/games
review.ts /review UI + approve/reject (localhost-bound)
health.ts GET /api/health
db/
client.ts bun:sqlite handle, WAL, pragmas
migrations/ NNN-name.sql, applied in order at boot
queries.ts all SQL lives here — no SQL in route handlers
ingest/
scheduler.ts timer + jitter + per-source lock
pipeline.ts the 7 stages, orchestration only
clean.ts HTML → text reduction before extraction
extract.ts Anthropic client, prompts, batch submission
validate.ts zod parse + calendar sanity rules
reconcile.ts diff vs published, confidence, conflict detection
adapters/
index.ts registry: GameId → Adapter
genshin.ts
hsr.ts
zzz.ts
wuwa.ts
arknights.ts
endfield.ts
shared/
schema.ts zod schemas — the contract, imported by both sides
types.ts z.infer types only
time.ts region reset math, duration formatting
client/
main.tsx
App.tsx
views/
Timeline.tsx F1
EndingSoon.tsx F2
EventDetail.tsx
state/
completions.ts localStorage read/write + export/import
prefs.ts region, filters
api.ts typed fetch of /api/events
fixtures/<game>/ checked-in raw HTML + expected parse output
docs/
```
## Request paths
| Route | Purpose | Notes |
|---|---|---|
| `GET /` + assets | React SPA | Served from the `bun build` output |
| `GET /api/events?from&to&game` | Filtered feed | `ETag` + `Cache-Control: public, max-age=300` |
| `GET /api/events.json` | Whole published feed | Cheap; the client mostly uses this and filters locally |
| `GET /api/games` | Game metadata: id, name, color, lastUpdatedAt | Drives the freshness badges (F7) |
| `GET /api/health` | Per-source last-success, quarantine depth | For an operator, not the UI |
| `GET /review` | Quarantine review UI | **Bound to `127.0.0.1` only** |
| `POST /api/review/:id/approve` \| `/reject` | Promote or discard a quarantined event | Same binding |
### Why `/review` needs no auth
`Bun.serve` runs two listeners: the public one on `0.0.0.0:PORT` with the SPA and `/api/*`, and a
second on `127.0.0.1:ADMIN_PORT` with `/review` and `/api/review/*`. The review routes are not
registered on the public listener at all — they are unreachable from off-box, so there is nothing
to authenticate. This is the mechanism that satisfies "no logins" without leaving an open admin
endpoint on the internet.
**This is load-bearing.** If someone later puts a reverse proxy in front of the admin port, or
merges the two listeners "to simplify", the review UI becomes a public write endpoint. Any change
in that area needs an explicit auth story first.
## Data flow, concretely
1. **Scheduler** wakes every 6h (± jitter). For each source not fetched within its `minIntervalMs`,
it acquires a per-source lock row and enqueues a run.
2. **Pipeline** executes the seven stages in `docs/INGESTION.md`. Every stage writes to
`ingest_runs` so a failure is diagnosable after the fact without re-running.
3. **Publish** upserts into `events` by stable ID, bumping `version` and `updatedAt` when any field
changed. Events that vanish from a source are *not* deleted — they are marked
`status = 'delisted'` so a source outage cannot silently empty the calendar.
4. **Client** fetches the feed, merges the completion set from `localStorage` by event ID, and
renders. Merge is a client-side join; the server never learns what the user completed.
## Concurrency and failure
- One in-flight run per source, enforced by a lock row with a stale-lock timeout of 15 minutes.
- A source that fails keeps its previously published events. A failed run never deletes or blanks
data — worst case, the game's lane goes stale and gets a warning badge (F7).
- Three consecutive failures for one source raises its `health` to `failing` in `/api/health`. It
does not stop the schedule; a wiki being down for a day is normal.
- Extraction results are written to `extraction_log` with the input hash, so a prompt change can be
evaluated against previously-seen inputs without re-fetching or re-paying.
## Deployment
Single process, single SQLite file, no external services beyond the Anthropic API.
```
PORT=3000
ADMIN_PORT=3001 # bound to 127.0.0.1
DATABASE_PATH=./data/events.sqlite
ANTHROPIC_API_KEY=sk-ant-...
INGEST_INTERVAL_MS=21600000
INGEST_ENABLED=true # false for local UI work — never hits the network or the API
EXTRACTION_MODE=batch # batch | sync
CONFIDENCE_THRESHOLD=0.8
```
`INGEST_ENABLED=false` is the default for local development. Frontend work should run against a
seeded SQLite file and cost nothing.
## Deliberate non-choices
- **No ORM.** `bun:sqlite` plus hand-written SQL in `queries.ts`. The schema is six tables.
- **No Redis / job queue.** The scheduler is a timer and a lock row. Restarting the process resumes
cleanly because state is in SQLite.
- **No server-side rendering.** The feed is small and cacheable; a static SPA is enough.
- **No websockets.** Events change on a scale of hours; a 5-minute cache is more than adequate.
docs/DATA-MODEL.md
+248
View File
@@ -0,0 +1,248 @@
# Data Model
`src/shared/schema.ts` is the single source of truth. TypeScript types are derived with
`z.infer<>` — never hand-write an interface that duplicates a schema.
## The Event
```ts
import { z } from "zod";
export const GameId = z.enum([
"genshin", "hsr", "zzz", "wuwa", "arknights", "endfield",
]);
export const EventType = z.enum([
"banner", // limited character/weapon rate-up
"story", // main or side story chapter, limited-time
"rerun", // returning event
"challenge", // combat/endgame cycle (Abyss, Memory of Chaos, ...)
"login", // login rewards / check-in
"shop", // limited shop or exchange window
"maintenance", // server downtime
"other",
]);
export const Region = z.enum(["asia", "america", "europe"]);
/** How much we actually know about a boundary timestamp. */
export const Precision = z.enum([
"exact", // sourced to the minute
"day", // date known, time-of-day inferred from the game's reset
"unknown", // genuinely not announced — endsAt is null
]);
export const GachaEvent = z.object({
id: z.string(), // `${game}:${slug}:${YYYY-MM-DD}` — see Stability below
game: GameId,
title: z.string().min(1).max(200),
type: EventType,
summary: z.string().max(500).nullable(),
startsAt: z.string().datetime(), // UTC ISO 8601, always
startPrecision: Precision,
endsAt: z.string().datetime().nullable(),
endPrecision: Precision,
/** True when the end time follows each region's daily reset rather than a global instant. */
regionScoped: z.boolean(),
/** Populated only when regionScoped; per-region resolved UTC instants. */
regionEnds: z.record(Region, z.string().datetime()).nullable(),
sourceUrl: z.string().url(),
sourceId: z.string(), // which adapter/source produced this
status: z.enum(["published", "delisted"]),
confidence: z.number().min(0).max(1),
extractionMethod: z.enum(["parser", "llm", "manual"]),
version: z.number().int().positive(),
firstSeenAt: z.string().datetime(),
updatedAt: z.string().datetime(),
});
export type GachaEvent = z.infer<typeof GachaEvent>;
```
### Field notes that matter
**`endsAt: null` is a first-class state, not an error.** Many events are announced with "duration
TBD" or "until the next version update". The correct representation is `endsAt: null` with
`endPrecision: "unknown"`. The extractor is instructed to produce this and the UI renders it
distinctly (PRD F1). Filling in a plausible date instead is the single worst bug this codebase can
ship.
**`regionScoped` + `regionEnds`.** Character banners end at one global instant — `regionScoped:
false`, `regionEnds: null`. Story and login events end at each region's daily reset — `regionScoped:
true`, with `regionEnds` carrying the three resolved UTC instants. The client picks one using the
user's stored region (PRD F5). Collapsing these into a single timestamp loses up to 13 hours of
accuracy and will make the countdown wrong for two thirds of users.
**`confidence`** is assigned during reconciliation, not by the model's self-report. See
`docs/LLM-EXTRACTION.md` § Scoring — a model asserting "I am 0.95 confident" is not evidence.
**`status: "delisted"`** means the event stopped appearing at its source. It is never deleted,
because a source outage would otherwise silently empty the calendar. Delisted events are excluded
from the API feed but retained for debugging and for the case where a source flickers.
### ID stability — read before changing
```
`${game}:${slugify(title)}:${startsAt.slice(0, 10)}`
→ "genshin:windblume-festival:2026-03-14"
```
**Event IDs are the localStorage keys for completion state.** Changing the scheme orphans every
completion mark every user has ever made, silently, with no error and no way to recover it
server-side (the server never had the data). If the scheme must change, ship a client-side
migration that reads the old keys and remaps them, and keep that migration for at least a year.
The date suffix disambiguates reruns of the same event. Title is slugified from the *source's*
title, so a wiki renaming an event creates a new ID — reconciliation detects this as a near-match
(same game, overlapping dates, high title similarity) and treats it as an update rather than a new
event, preserving the original ID.
## SQLite schema
```sql
-- Published feed. One row per event.
CREATE TABLE events (
id TEXT PRIMARY KEY,
game TEXT NOT NULL,
title TEXT NOT NULL,
type TEXT NOT NULL,
summary TEXT,
starts_at TEXT NOT NULL,
start_precision TEXT NOT NULL,
ends_at TEXT,
end_precision TEXT NOT NULL,
region_scoped INTEGER NOT NULL DEFAULT 0,
region_ends TEXT, -- JSON object or NULL
source_url TEXT NOT NULL,
source_id TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'published',
confidence REAL NOT NULL,
extraction_method TEXT NOT NULL,
version INTEGER NOT NULL DEFAULT 1,
first_seen_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE INDEX idx_events_ends ON events (ends_at) WHERE status = 'published';
CREATE INDEX idx_events_game ON events (game, starts_at);
CREATE INDEX idx_events_window ON events (starts_at, ends_at);
-- Candidates held back by the review gate. Same shape plus why.
CREATE TABLE events_quarantine (
id TEXT PRIMARY KEY,
payload TEXT NOT NULL, -- full GachaEvent JSON
reason TEXT NOT NULL, -- 'low_confidence' | 'date_conflict' | 'sanity_failed' | 'novel_shape'
detail TEXT NOT NULL, -- human-readable explanation for the reviewer
conflicts_with TEXT, -- events.id, when reason = 'date_conflict'
run_id TEXT NOT NULL REFERENCES ingest_runs(id),
created_at TEXT NOT NULL,
resolved_at TEXT,
resolution TEXT -- 'approved' | 'rejected' | NULL
);
CREATE INDEX idx_quarantine_open ON events_quarantine (created_at) WHERE resolved_at IS NULL;
-- One row per configured source.
CREATE TABLE sources (
id TEXT PRIMARY KEY, -- 'genshin-wiki-events'
game TEXT NOT NULL,
url TEXT NOT NULL,
strategy TEXT NOT NULL, -- 'parser' | 'llm' | 'parser_then_llm'
min_interval_ms INTEGER NOT NULL DEFAULT 21600000,
etag TEXT,
last_modified TEXT,
content_hash TEXT, -- sha256 of cleaned content; the skip check
last_success_at TEXT,
last_attempt_at TEXT,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
health TEXT NOT NULL DEFAULT 'ok', -- 'ok' | 'degraded' | 'failing'
lock_holder TEXT,
lock_expires_at TEXT
);
-- One row per pipeline execution. The audit trail.
CREATE TABLE ingest_runs (
id TEXT PRIMARY KEY,
source_id TEXT NOT NULL REFERENCES sources(id),
started_at TEXT NOT NULL,
finished_at TEXT,
outcome TEXT, -- 'published' | 'skipped_unchanged' | 'quarantined' | 'failed'
stage_failed TEXT,
error TEXT,
events_seen INTEGER DEFAULT 0,
events_changed INTEGER DEFAULT 0,
events_held INTEGER DEFAULT 0
);
-- One row per LLM call. Enables replaying prompt changes against past inputs.
CREATE TABLE extraction_log (
id TEXT PRIMARY KEY,
run_id TEXT NOT NULL REFERENCES ingest_runs(id),
source_id TEXT NOT NULL,
model TEXT NOT NULL,
prompt_version TEXT NOT NULL,
input_hash TEXT NOT NULL, -- of the cleaned text sent to the model
input_tokens INTEGER,
output_tokens INTEGER,
cache_read_tokens INTEGER,
cache_write_tokens INTEGER,
stop_reason TEXT,
refusal_category TEXT,
duration_ms INTEGER,
created_at TEXT NOT NULL
);
CREATE INDEX idx_extraction_input ON extraction_log (input_hash);
-- Cached raw + cleaned snapshots so re-extraction never re-fetches.
CREATE TABLE snapshots (
content_hash TEXT PRIMARY KEY,
source_id TEXT NOT NULL,
fetched_at TEXT NOT NULL,
raw BLOB NOT NULL,
cleaned TEXT NOT NULL
);
```
`region_ends` and `payload` hold JSON as TEXT; parse them through the Zod schema on read so a
malformed row surfaces at the boundary rather than deep in the UI.
## Client-side storage
Namespaced, versioned, and small. Nothing here ever goes to the server.
```ts
"gacha-tracker:v1:completions" // { [eventId]: { completedAt: string } }
"gacha-tracker:v1:prefs" // { region, hiddenGames[], hiddenTypes[], showCompleted }
"gacha-tracker:v1:feedCache" // { fetchedAt, events } — offline fallback
```
The `v1` segment is the migration hook. On boot, the client checks for keys at older versions and
migrates them forward before reading. **Never delete an old-version key until the migration has
shipped and run** — a user who has not opened the app in six months still has their data under the
old key.
### Export format (PRD F6)
```json
{
"format": "gacha-tracker-export",
"version": 1,
"exportedAt": "2026-08-14T12:00:00.000Z",
"completions": { "genshin:windblume-festival:2026-03-14": { "completedAt": "..." } },
"prefs": { "region": "europe", "hiddenGames": [] }
}
```
Import **merges**: a completion present in either the file or the current device stays completed.
Import never removes a completion. Losing a user's marks to a bad import is unrecoverable, so the
merge is deliberately one-directional.
## Schema versioning
`/api/events` responses carry `{ schemaVersion: 1, generatedAt, events: [...] }`. The client
refuses to render a `schemaVersion` it does not know and shows a "refresh the page" prompt instead
of guessing at unfamiliar fields. Additive fields do not bump the version; removing or retyping a
field does.
docs/INGESTION.md
+221
View File
@@ -0,0 +1,221 @@
# Ingestion Pipeline
Seven stages, run per source. Every stage writes its outcome to `ingest_runs` so a failure two days
ago can be diagnosed without re-running or re-paying.
```
fetch → clean → parse|extract → validate → reconcile → gate → publish
│
└──► quarantine
```
## The adapter contract
An adapter is the only per-game code. Everything downstream of `parse` is shared.
```ts
export interface Adapter {
id: string; // 'genshin-wiki-events'
game: GameId;
url: string;
strategy: "parser" | "llm" | "parser_then_llm";
minIntervalMs?: number; // default 6h
/** Narrow the cleaned document to just the region containing event data. */
select?(cleaned: string): string;
/**
* Deterministic parse. Return null to fall through to LLM extraction
* (only meaningful when strategy is 'parser_then_llm').
* Pure over its input — no network, no clock, no randomness. This is what
* makes fixture tests possible.
*/
parse?(cleaned: string, ctx: ParseContext): RawEvent[] | null;
/** Extra instructions appended to the shared extraction prompt. */
extractionHints?: string;
/** Game-specific normalization: reset times, region offsets, patch cadence. */
normalize(raw: RawEvent, ctx: ParseContext): GachaEvent;
}
export interface ParseContext {
now: string; // injected, never Date.now() — keeps parse pure and testable
sourceUrl: string;
sourceId: string;
game: GameId;
}
```
**`parse` must not read the clock.** It takes `now` from `ctx`. This is what lets a fixture test
assert exact output for a page captured last March.
### Choosing a strategy
| Source shape | Strategy |
|---|---|
| JSON API, or a stable HTML table with consistent headers | `parser` |
| Free-form patch notes, announcement prose, inconsistent markup | `llm` |
| Mostly-stable markup that occasionally changes | `parser_then_llm` |
Prefer `parser`. It is free, deterministic, and instantly testable. The LLM exists for sources that
genuinely cannot be parsed reliably, not as the default. A source with a clean API that goes through
the model is a bug.
## Stage 1 — fetch
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
the run as `skipped_unchanged` with zero further cost.
- `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
- Honor `robots.txt`. Cache the parsed robots per host for 24h.
- 20s timeout; retry twice with exponential backoff on 5xx and network errors; never retry 4xx.
- Store the raw bytes in `snapshots`.
On failure: increment `consecutive_failures`, leave published events untouched, end the run as
`failed`. A source being down never mutates the feed.
## Stage 2 — clean
Reduce the document before it costs anything. This stage is the second-biggest cost lever after the
content-hash skip.
- Drop `<script>`, `<style>`, `<svg>`, `<noscript>`, comments, nav, header, footer, and known
wiki chrome (edit links, category boxes, reference lists).
- Collapse whitespace; convert tables to pipe-delimited text; keep headings as `#` markers so
section structure survives.
- Apply `adapter.select()` if present to isolate the event region.
- Hash the result (sha256) → `content_hash`.
**If `content_hash` matches `sources.content_hash`, end the run as `skipped_unchanged`.** This is
the check that keeps a 6-hourly schedule from costing anything on a quiet week — most runs should
end here.
A typical wiki page goes from ~15k tokens raw to ~5k cleaned. Verify with `messages.count_tokens`
when tuning, not by guessing.
## Stage 3 — parse or extract
Per strategy. `parse` produces `RawEvent[]` directly. `extract` sends the cleaned text to
`claude-opus-5` with a structured-output schema — see `docs/LLM-EXTRACTION.md` for the request
shape, prompt, and cost rules.
`parser_then_llm` calls `parse` first and falls through to `extract` only when it returns `null`.
When that fallthrough happens, log it loudly: it means the source changed shape and the parser needs
updating. A `parser_then_llm` source that is silently always falling through is paying LLM prices
for a parser that no longer works.
## Stage 4 — validate
Zod parse against `GachaEvent`, then calendar sanity rules. Anything failing a hard rule goes to
quarantine with `reason: 'sanity_failed'` — never to the feed.
**Hard rules (reject):**
| Rule | Rationale |
|---|---|
| `endsAt > startsAt` when both present | A backwards interval is always a parse error |
| Duration ≤ 180 days | Patch cycles are ~6 weeks; 180d means a year was misread as a range |
| `startsAt` within [now − 2y, now + 1y] | Catches century typos and relative-date misreads |
| `endsAt` null ⟺ `endPrecision === "unknown"` | The two fields must agree |
| `regionEnds` non-null ⟺ `regionScoped` | Same |
| All `regionEnds` values within 24h of each other | Region resets differ by hours, not days |
| `title` non-empty, ≤ 200 chars, not a placeholder ("TBD", "Event", "Unknown") | Catches header rows scraped as events |
**Soft rules (reduce confidence, do not reject):**
- Duration under 1 hour or over 60 days → −0.2
- `startPrecision` or `endPrecision` is `"day"` → −0.1
- Title very similar to another event in the same batch → −0.15 (likely a duplicate row)
## Stage 5 — reconcile
Diff the validated candidates against currently published events for this source.
1. **Exact ID match** → compare fields. Unchanged: no-op. Changed: candidate update.
2. **Near match** — same game, date windows overlap, title similarity ≥ 0.85 — → treat as an update
to the existing event, **keeping the existing ID**. This is what survives a wiki renaming an
event without orphaning every user's completion mark.
3. **No match** → new event.
4. **Published event absent from this run's candidates** → mark `status = 'delisted'`. Do not
delete.
**Conflict detection.** A candidate that changes an already-published `endsAt` by more than 24 hours
is a `date_conflict`. This is the case worth being paranoid about: the user may have already planned
around the old date, and a silent shift is exactly the failure the product exists to prevent. Route
it to quarantine regardless of confidence.
### Scoring
Confidence is computed here, from evidence — **not** taken from the model's self-report. A model
saying "confidence: 0.95" is a token prediction, not a measurement.
```
base parser → 0.95 llm → 0.70
+0.15 the same event was extracted identically from a previous run
+0.10 both timestamps have precision "exact"
+0.10 a second source for the same game corroborates within 1 hour
−0.20 any soft rule fired
−0.30 this is a date_conflict against a published event
```
Clamp to [0, 1]. `CONFIDENCE_THRESHOLD` (default 0.8) is the gate.
## Stage 6 — gate
| Condition | Destination |
|---|---|
| `confidence >= CONFIDENCE_THRESHOLD` and no conflict | publish |
| `confidence < CONFIDENCE_THRESHOLD` | quarantine — `low_confidence` |
| `date_conflict` | quarantine — `date_conflict`, with `conflicts_with` set |
| failed a hard rule | quarantine — `sanity_failed` |
| new event type or field the schema does not recognize | quarantine — `novel_shape` |
A quarantined event does not block its siblings. If eight events in a run pass and two are held, the
eight publish.
## Stage 7 — publish
Upsert by ID inside a transaction. Bump `version` and `updatedAt` only when a field actually
changed — an unchanged run must not churn `updatedAt`, or the freshness badge (PRD F7) becomes
meaningless. Update `sources.content_hash`, `etag`, `last_success_at`, and reset
`consecutive_failures`.
## The review gate
Quarantined events surface at `GET /review` on the admin listener (`127.0.0.1:ADMIN_PORT`). See
`docs/ARCHITECTURE.md` § Why `/review` needs no auth — the routes are simply not registered on the
public listener.
The review UI shows, per held event: the parsed fields, the reason and detail, the conflicting
published event side-by-side when applicable, a link to the source, and the exact cleaned text
excerpt the extraction came from. A reviewer needs to answer "is this date right?" without leaving
the page.
- `POST /api/review/:id/approve` — writes to `events` with `extraction_method: 'manual'` and
`confidence: 1.0`, and stamps `resolved_at` / `resolution`.
- `POST /api/review/:id/reject` — stamps resolution only. The event is not published, and the same
candidate will be re-held on the next run if the source has not changed.
Approving with edits is supported: the reviewer can correct a date before approving. That corrected
value is the one that publishes.
**Quarantine depth is the health signal for the whole pipeline.** A growing queue means a source
changed shape or the prompt regressed. `/api/health` exposes the count; watch it.
## Testing
Every adapter ships:
1. `fixtures/<game>/<source>-<YYYY-MM-DD>.html` — a real captured page.
2. `fixtures/<game>/<source>-<YYYY-MM-DD>.expected.json` — the exact `GachaEvent[]` it should
produce.
3. A test running `parse` + `normalize` against the fixture with a pinned `ctx.now`, asserting
deep equality.
`bun test` must pass with no network access. When a source changes shape, capture a new fixture
alongside the old one and keep both — the old fixture is the regression test proving the parser
still handles the previous format.
Validator rules get their own unit tests with deliberately broken inputs: backwards intervals,
1000-year durations, `endsAt` set with `endPrecision: "unknown"`. These rules are the last line of
defense before a wrong date reaches a user; test them like it.
docs/LLM-EXTRACTION.md
+291
View File
@@ -0,0 +1,291 @@
# LLM Extraction
Stage 3 of the pipeline, for sources whose markup is too unstable to parse deterministically. Read
`docs/INGESTION.md` first for where this sits.
## Rules
1. **The model runs at ingestion time only.** No request path calls Anthropic. A page load must
never trigger inference.
2. **Deterministic parsers win.** If a source has a JSON API or a stable table, it gets a parser,
not a prompt.
3. **The model's job is transcription, not judgment.** It converts prose and tables into structured
dates. It does not decide what is important, does not infer missing dates, and does not resolve
contradictions — it reports them.
4. **Confidence is computed from evidence in `reconcile`, not asserted by the model.** The output
schema has no confidence field. A model claiming 0.95 confidence has predicted a token, not
measured anything.
## Model and parameters
| Setting | Value | Why |
|---|---|---|
| `model` | `claude-opus-5` | Exact, complete ID — never append a date suffix |
| `max_tokens` | `16000` | Non-streaming; keeps the request under SDK HTTP timeouts |
| `output_config.effort` | `"medium"` | Transcription, not reasoning. Sweep low/medium/high against fixtures before settling |
| `output_config.format` | `zodOutputFormat(ExtractionResult)` | Schema-conformant output, validated by the SDK |
| `thinking` | *omit* | On by default on `claude-opus-5`; the default is correct here |
| `temperature` / `top_p` / `top_k` | **never set** | Removed on `claude-opus-5` — sending any of them returns 400 |
`thinking: {type: "enabled", budget_tokens: N}` is also removed and returns 400. If you want less
thinking, lower `effort`.
Note that `max_tokens` caps thinking *and* output together. If extraction on a large page returns
`stop_reason: "max_tokens"`, raise it rather than trimming the schema.
## The output schema
The model returns a list of candidate events plus explicit uncertainty. Note what is **absent**:
no confidence score, no "importance" ranking, no summary of the page.
```ts
// src/ingest/extract.ts
import { z } from "zod";
const ExtractedEvent = z.object({
title: z.string().describe("The event name exactly as written in the source, not paraphrased."),
type: z.enum(["banner","story","rerun","challenge","login","shop","maintenance","other"]),
summary: z.string().nullable()
.describe("One sentence from the source describing the event. Null if the source gives none."),
startsAt: z.string().nullable()
.describe("ISO 8601 UTC. Null only if the source truly does not state a start."),
startPrecision: z.enum(["exact","day","unknown"])
.describe("'exact' if a time of day is stated; 'day' if only a date; 'unknown' if neither."),
endsAt: z.string().nullable()
.describe("ISO 8601 UTC. Null when the source says TBD, 'until further notice', or gives no end."),
endPrecision: z.enum(["exact","day","unknown"]),
regionScoped: z.boolean()
.describe("True if the end follows each server region's daily reset rather than one global instant."),
sourceTimezone: z.string().nullable()
.describe("The timezone the source stated, e.g. 'UTC+8', 'server local'. Null if unstated."),
evidence: z.string()
.describe("The verbatim span from the input that gave you the dates. Must appear in the input."),
});
const ExtractionResult = z.object({
events: z.array(ExtractedEvent),
ambiguities: z.array(z.object({
title: z.string(),
issue: z.string().describe("What is unclear or contradictory in the source."),
})).describe("Events you could not confidently transcribe. These are held for human review."),
});
```
`evidence` is the load-bearing field. It is checked in `validate`: if the quoted span does not
appear in the input text, the event is quarantined as `sanity_failed`. That check is what turns a
fabricated date into a caught error instead of a shipped one.
## The system prompt
Kept in `src/ingest/prompts/extract-events.v1.md`, versioned in the filename, and logged as
`prompt_version` in `extraction_log` so a regression can be traced to a specific revision.
It must stay **above 512 tokens** — that is the minimum cacheable prefix on `claude-opus-5`. Below
it, prompt caching silently stops working with no error. After any prompt edit, check
`usage.cache_read_input_tokens` is non-zero on the second call.
```markdown
You extract scheduled in-game events from gacha game source pages into structured data.
Your output is consumed by a calendar that players rely on to avoid missing limited-time content.
A wrong end date is worse than a missing event: a missing event sends someone to a wiki, a wrong
one makes them miss content permanently. Transcribe what the source says; never supply what it
omits.
## What counts as an event
Anything with a start and a bounded or open-ended run: character and weapon banners, story
chapters, side events, login campaigns, limited shops, combat cycles, announced maintenance.
Not events: permanent features, general game descriptions, patch version numbers on their own,
speculation or leaks, community posts, and anything phrased as expected, rumored, or datamined.
## Dates
- Emit UTC ISO 8601 with an explicit `Z`.
- When the source states a timezone (commonly UTC+8 for Chinese-developed titles), convert to UTC
and record what it stated in `sourceTimezone`.
- When only a date is given, set the timestamp to 00:00:00Z and `precision: "day"`. Do not guess a
time of day.
- When the source says the end is TBD, "until further notice", "with the next version update", or
gives no end at all: `endsAt: null` and `endPrecision: "unknown"`. This is a correct, expected
answer. Do not compute a plausible date from a typical patch length.
- `regionScoped` is true when the end is tied to each server's daily reset, false when the source
gives one simultaneous global instant. Character banners are usually global; story and login
events are usually region-scoped. Use what the source says over this heuristic when it says
anything.
## Evidence
For every event, `evidence` must be a verbatim span copied from the input that contains the dates
you reported. It is checked against the input automatically. If you cannot quote a span, the event
belongs in `ambiguities` instead.
## Ambiguities
Put an entry in `ambiguities` — not in `events` — when the source contradicts itself, gives dates
you cannot reconcile, or describes something that may not be a scheduled event. A human reviews
these. Reporting uncertainty is a successful outcome, not a failure; guessing to avoid it is the
one thing that breaks this system.
## Scope
Report every qualifying event on the page and nothing else. Do not rank them, do not summarize the
page, do not comment on your process, and do not add fields the schema does not have. If the page
contains no events, return empty arrays.
```
### Why the prompt reads the way it does
`claude-opus-5` follows instructions literally and verifies its own work without being told, so the
prompt states scope and boundaries plainly instead of adding emphasis or self-check scaffolding.
Specifically: **do not add "double-check your answer" or "verify before responding"** here. On this
model that produces over-verification with no accuracy gain. If extraction quality drops, change the
schema descriptions or `effort` — not the volume of the prompt.
## Request shape
```ts
import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env
const response = await client.messages.parse({
model: "claude-opus-5",
max_tokens: 16000,
system: [
{
type: "text",
text: EXTRACTION_SYSTEM_PROMPT, // stable across every source — cached
cache_control: { type: "ephemeral", ttl: "1h" },
},
],
output_config: {
effort: "medium",
format: zodOutputFormat(ExtractionResult),
},
messages: [
{
role: "user",
content: [
`Game: ${adapter.game}`,
`Source: ${adapter.url}`,
`Today (UTC): ${ctx.now}`,
adapter.extractionHints ?? "",
"",
"--- SOURCE TEXT ---",
cleanedText,
].join("\n"),
},
],
});
// parsed_output is null if the model refused or hit max_tokens — check before use.
const result = response.parsed_output;
```
**Ordering matters for caching.** The system prompt is byte-identical across every source, so it
sits first and stays cached. Everything volatile — game, URL, `now`, the page text — goes in the
user turn, after the cache breakpoint. Interpolating `ctx.now` into the system prompt would
invalidate the cache on every single call; it is in the user turn for exactly that reason.
## Batch mode
`EXTRACTION_MODE=batch` is the default for scheduled runs: 50% cheaper, results typically within an
hour, which is irrelevant against a 6-hour cadence. The synchronous path above is for
`refresh --now` and for local development.
```ts
const batch = await client.messages.batches.create({
requests: sourcesToExtract.map((s) => ({
custom_id: s.runId, // key results by this — order is not guaranteed
params: { /* same body as above */ },
})),
});
// Poll batches.retrieve(batch.id) until processing_status === "ended",
// then stream batches.results(batch.id) and key each result by custom_id.
```
Results arrive in **any order**. Key by `custom_id`, never by array position. The batch job's
poll loop lives in `scheduler.ts` and persists `batch.id` so a process restart resumes rather than
resubmitting.
## Handling non-success responses
Check `stop_reason` before touching `parsed_output`:
| `stop_reason` | Meaning | Action |
|---|---|---|
| `end_turn` | Normal | Proceed |
| `max_tokens` | Output truncated; `parsed_output` unusable | Raise `max_tokens`, or split the page by section. Fail the run — never publish a partial list |
| `refusal` | Safety classifier declined | Log `stop_details.category` to `extraction_log.refusal_category`, fail the run, alert. Vanishingly unlikely for game wiki content — if it fires, the input is probably not what you think it is |
`stop_details` can be `null` even on a refusal, so branch on `stop_reason` and treat `stop_details`
as informational.
**Do not write a JSON-repair or regex-extraction fallback.** Structured outputs guarantee schema
conformance; if parsing fails, the response was truncated or refused, and both cases are handled
above. A repair path would silently paper over truncation and publish half a calendar.
## Cost
At `claude-opus-5` rates — $5/MTok input, $25/MTok output.
One extraction of a cleaned wiki page:
```
input ~5,000 tok × $5/MTok = $0.025
output ~1,500 tok × $25/MTok = $0.0375
───────
sync $0.063
batch $0.031 (50% off)
```
Six sources, four runs a day:
| Scenario | Extractions/day | Cost/day | Cost/month |
|---|---|---|---|
| Realistic — ~20% of runs see changed content | ~5 | $0.16 | **~$4.70** |
| Worst case — every source changes every run | 24 | $0.74 | ~$22 |
| No content-hash skip, no batch | 24 | $1.51 | ~$45 |
The gap between rows one and three is the whole argument for the skip check and batch mode. Track
actuals by summing `extraction_log` token columns — do not rely on these estimates once there is
real data.
Prompt caching contributes modestly (the ~1k-token system prompt at 0.1× on reads) but is free to
keep. Its real value is that it makes the prompt cheap to grow if extraction quality needs more
instruction.
## Evaluating a prompt change
`extraction_log` stores `input_hash` for every call, and `snapshots` stores the cleaned text keyed
by the same hash. So a prompt revision is evaluated by replaying past inputs — no re-fetching, no
new scraping load:
1. Pull the last N distinct `input_hash` values with known-correct expected output.
2. Run the new prompt against each cleaned snapshot.
3. Diff against `fixtures/*/*.expected.json`.
4. Compare on three axes: dates correct, events missed, events hallucinated. **Hallucinated events
and wrong dates are disqualifying; a missed event is a regression to weigh.** That asymmetry is
the product rule from `docs/PRD.md` restated as an eval criterion.
Bump the prompt filename version and record it as `prompt_version` so the log distinguishes
"extraction got worse" from "the source changed".
## Things not to do
- Do not ask the model to output a confidence score. Confidence is computed in `reconcile`.
- Do not ask the model to resolve a contradiction between two sources. Route it to quarantine.
- Do not send raw HTML. Always clean first — it is a 3× cost difference and it improves accuracy.
- Do not add a second model call to "verify" the first. That is a scaffolding pattern this model
does not need, and it doubles cost for no measured gain. The evidence-span check and the
validator rules are the verification layer.
- Do not lower `max_tokens` to save money. Output tokens scale with the number of events found;
truncation costs a whole run.
docs/PRD.md
+123
View File
@@ -0,0 +1,123 @@
# Gacha Event Tracker — Product Spec
## The problem
A player of three or four gacha games is tracking a dozen concurrent, overlapping, time-boxed
events across as many different in-game calendars. The information exists — on wikis, in patch
notes, in-game — but never in one place and never sorted by the thing that actually matters:
**what expires next.** The failure mode is missing a limited event by a day.
## What this app is
A single-page web app that answers three questions:
1. What is running right now, across all my games?
2. What ends soonest?
3. Which of these have I already finished?
## What this app is not
- Not an account system. There is no login, no profile, no cloud sync.
- Not a wiki. It does not explain how to complete an event, only that it exists and when it ends.
- Not a notification service. No push, no email, no background alerts. (A browser-local reminder
is a plausible v2; it is out of scope for v1.)
- Not a damage calculator, build planner, or pull tracker.
## Users
One persona: a player of 2–5 gacha games who checks in a few times a week, most often on mobile.
They care about accuracy of end dates above everything else — a wrong date is worse than a missing
event, because a missing event sends them to a wiki while a wrong one makes them miss content.
## Scope — v1
### Games at launch
| Game | ID |
|---|---|
| Genshin Impact | `genshin` |
| Honkai: Star Rail | `hsr` |
| Zenless Zone Zero | `zzz` |
| Wuthering Waves | `wuwa` |
| Arknights | `arknights` |
| Arknights: Endfield | `endfield` |
Adding a seventh game must require no schema change — only a new adapter. That is the test of
whether the data model is right.
### Features
**F1 — Calendar view (default).**
A horizontal timeline, one lane per game, spanning a scrollable date range with "today" pinned as a
vertical marker. Each event is a bar from `startsAt` to `endsAt`. Bars are colored by game, and
completed events render at reduced opacity with a check. Clicking a bar opens a detail panel with
title, type, exact start/end in the user's local timezone, source link, and a completion toggle.
An event with `endsAt: null` renders as a bar with a frayed right edge and the label "end date
unknown" — it must be visually distinct from an event that ends far in the future.
**F2 — Ends-soonest list.**
A flat list of all *currently running* events sorted ascending by end date, with a relative
countdown ("ends in 2 days", "ends in 4 hours"). Under 24 hours, the row is emphasized. This is the
view that justifies the app; it should be reachable in one tap from the calendar and is the better
default on narrow screens.
**F3 — Mark completed.**
A toggle on every event, in both views. State is written to `localStorage` immediately and
optimistically — there is no server round trip and no failure case. Completed events stay visible
but de-emphasized; a filter toggles them out entirely.
**F4 — Filters.**
Filter by game (multi-select, persisted) and by event type. Hiding a game hides it from both views.
Preferences persist in `localStorage`.
**F5 — Region selection.**
A user picks Asia / America / Europe once. For events where `regionScoped` is true, all displayed
end times resolve to that region's server reset. This is stored in `localStorage` and defaults to a
guess from the browser timezone, shown as a dismissible "showing America server times — change".
**F6 — Export / import.**
Because there are no accounts, moving between devices is manual: download a JSON file of completed
IDs and preferences, upload it elsewhere. Import merges rather than replaces, and never removes a
completion the user already has.
**F7 — Freshness disclosure.**
The footer shows when the feed was last updated, per game. If a game's data is more than 48 hours
stale, its lane carries a warning badge. Never present stale data as current — the whole value
proposition is trust in the dates.
## Out of scope for v1
Accounts and sync; push notifications; per-event checklists or progress tracking; in-game resource
or pull tracking; user-submitted events; mobile apps; localization beyond English.
## Success criteria
- A user can identify their next expiring event within **5 seconds** of load, on mobile.
- Published end dates are correct for **99%+** of events. This is a data-quality target, and it is
what the review gate in `docs/INGESTION.md` exists to protect. Prefer publishing nothing to
publishing a guess.
- Adding a new game is an adapter plus a fixture plus a test — no schema migration, no client
change.
## Quality bar for dates — the core product rule
The app's entire value is that the dates are right. Therefore:
- An event with an uncertain end date is published with `endsAt: null`, **not** with a plausible
guess.
- An event whose extraction confidence is below threshold is not published at all until a human
approves it.
- Every event links to its source so a skeptical user can verify in one click.
An empty calendar is a recoverable disappointment. A confidently wrong end date is the failure this
product exists to prevent.
## Open questions
- Does the calendar need a month/grid view, or is the timeline enough? (Assumption: timeline is
enough for v1; revisit after use.)
- Should events the user has hidden by game filter still count toward "ends soonest"?
(Assumption: no — the filter is global.)
- Is 6 hours the right refresh cadence? (Assumption: yes; events are announced days ahead, so
sub-hourly refresh buys nothing and costs API spend.)