diff --git a/.claude/agents/adapter-author.md b/.claude/agents/adapter-author.md index 1522dbf..f79e08f 100644 --- a/.claude/agents/adapter-author.md +++ b/.claude/agents/adapter-author.md @@ -24,8 +24,8 @@ Fetch the page exactly once. **3. Decide whether an existing parser covers it.** -Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves -every Game8 page. Check `PARSERS` first: +Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game. Two exist: `game8` +(six sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first: - **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new parsing code. This is the common case and should be the first thing you try. @@ -34,6 +34,10 @@ every Game8 page. Check `PARSERS` first: - **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is no LLM fallback in this pipeline by design. Suggest a different source. +**Check every table before concluding a page is undatable.** Endfield was written off on a pass that +only inspected its `Duration` rows; its real events were in a table further down, and a second +source (wiki.gg) turned out to publish ISO timestamps with per-region timers. + **4. Implement the parser (only if step 3 says you need one).** - `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`. @@ -48,9 +52,13 @@ every Game8 page. Check `PARSERS` first: - An unstated end is `endsAt: null` + `endPrecision: "unknown"`. **Never compute a plausible end from typical patch length.** This is the failure mode that makes the product worthless. -**5. Write the test.** `fixtures//-.expected.json` holds the exact -expected `GachaEvent[]`. The test runs `parse` + `normalize` against the fixture with a pinned -`ctx.now` and asserts deep equality. +**5. Write the test.** `fixtures//-events-.expected.json` holds the exact +expected `GachaEvent[]`. Add the source to `CASES` in `test/adapters/game8.test.ts`; the shared +assertions (schema, determinism, no backwards intervals, 180-day cap, unique IDs) then apply for +free. Regenerate the expected file with `bun run parse --json`. + +**Fixture names matter**: `build-feed` selects by `-*` within the game directory, so a game +with two sources needs distinct site prefixes. **6. Verify.** Run `bun test` and confirm it passes with no network. Then hand-check three or four events against the live page and state in your report that you did — a green test against an diff --git a/.claude/skills/add-game-source/SKILL.md b/.claude/skills/add-game-source/SKILL.md index 86ed867..7d6ce60 100644 --- a/.claude/skills/add-game-source/SKILL.md +++ b/.claude/skills/add-game-source/SKILL.md @@ -35,9 +35,13 @@ only schema edit a new game should require. If it needs more, that is a finding Fetch the source page **once** and save the raw HTML: ``` -fixtures//-.html +fixtures//-events-.html ``` +The `` prefix is load-bearing: `build-feed` picks fixtures by site within the game directory, +so a game with two sources whose files share a prefix will hand one site's page to the other's +parser. + Everything after this point works offline against that file. Do not re-fetch while iterating on the parser. @@ -77,16 +81,18 @@ report that you did this. `parserId`, and optionally `priority` (higher wins when sources disagree). - Insert the matching `sources` row: id, game, url, parser_id, priority, `min_interval_ms`. -## 7. First run +## 7. Rebuild the feed ``` -INGEST_ENABLED=true bun run ingest --source --dry-run +bun run build:feed # regenerates public/data/events.v1.json +bun run dev # build and serve on :3000 ``` -Inspect what it *would* publish before letting it write. Then run for real and check the review -queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine -on its first pass, because nothing corroborates it yet. That is the gate working, not a bug. -Review and approve them. +Check the event count and any conflicts the merge reports. A game with two sources will surface +disagreements — those are the gate working, not a bug. + +The scheduler, quarantine table and `/review` queue described in `docs/INGESTION.md` are **not built +yet**; today the feed is generated offline from fixtures. ## Checklist diff --git a/.claude/skills/review-quarantine/SKILL.md b/.claude/skills/review-quarantine/SKILL.md index e197eea..3ee633c 100644 --- a/.claude/skills/review-quarantine/SKILL.md +++ b/.claude/skills/review-quarantine/SKILL.md @@ -5,6 +5,11 @@ description: Work through the quarantined-event queue — triage held events by # Reviewing the quarantine queue +> **Not built yet.** The quarantine table, the `/review` route and the ingest scheduler are specified +> in `docs/INGESTION.md` but do not exist in the tree. Today the closest equivalent is the conflict +> list `bun run build:feed` prints when two sources disagree. Use this skill once the pipeline lands; +> until then, treat it as the spec for what that review flow should do. + Held events are candidates the pipeline declined to publish. Working the queue is both a data task (get these events onto the calendar) and a diagnostic one (**a growing queue means something upstream broke**). diff --git a/CLAUDE.md b/CLAUDE.md index 87bdf1f..f75ba1f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,9 +7,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co A web app that aggregates live and upcoming events across popular gacha games, plots them on a calendar, sorts them by end date, and lets a user mark events completed. -**Status: first vertical slice.** The schema, the Game8 parser, and two working adapters (Genshin -Impact, Neverness to Everness) exist and are tested. The server, database, and UI do not exist yet — -`docs/` specifies them. +**Status: working app, no live refresh.** Schema, two parsers, seven sources across six games, the +full interface, offline support, a static server, a Docker image and CI all exist and are tested. +The SQLite layer, the refresh scheduler and the review queue are specified in `docs/` but not built, +so the feed is generated offline from checked-in fixtures. ## Three constraints that shape everything @@ -39,18 +40,26 @@ parsing library — Bun covers all four. `tsconfig.json` runs `strict` plus ```bash bun install -bun test # full suite, offline, no network +bun test # full suite, offline, no network, no build needed bun run typecheck # tsc --noEmit +bun run dev # build then serve on :3000 +bun run build # feed + css + js + static into public/ -# Run an adapter against a checked-in fixture (offline, free) +# Run one source against its fixture (offline, free) bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html -bun run parse nte-game8-events fixtures/nte/game8-events-2026-08-14.html --json +bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json # Single test file / single test bun test test/dates.test.ts bun test --test-name-pattern "year-less" + +# Hosting under a subpath (GitHub Pages) +BASE_PATH=/gacha-event-tracker/ bun run build ``` +**Tests must never need build output.** They run before `bun run build` in CI; anything reading +`public/` must create its own fixture tree instead. + `bun run parse ... --json` is also how `.expected.json` fixtures are regenerated after an intentional parser change. Regenerating them makes the test self-consistent, not correct — always re-verify a sample against the live page afterward. @@ -58,19 +67,18 @@ re-verify a sample against the live page afterward. ## Current state of the code ``` -src/shared/schema.ts Zod GachaEvent, GameId, slugify, eventId ← the contract -src/ingest/html.ts flat-table HTML reader (no dependency) -src/ingest/dates.ts three date formats, null rather than guess -src/ingest/adapters/ - types.ts Adapter interface, ParseContext - game8.ts shared Game8 parser — handles 2 table shapes - index.ts adapter registry -scripts/parse-fixture.ts offline adapter runner -test/ 37 tests -fixtures// raw HTML + .expected.json per source +src/shared/ schema.ts (the contract), time.ts, games.ts, feed.ts +src/ingest/ html.ts, dates.ts (six formats), merge.ts + parsers/ game8.ts, wikigg.ts — keyed by SITE, not game + adapters/ index.ts — SOURCES registry binding url+game+parser +src/client/ React app, service worker, manifest +scripts/ build-feed.ts, parse-fixture.ts (both offline) +serve.ts static server + /api/health +test/ 112 tests +fixtures// raw HTML + .expected.json per source ``` -Not yet built: `src/server/**`, `src/client/**`, the SQLite layer, the scheduler, the review UI. +Not yet built: the SQLite layer, the ingest scheduler, the review UI. ## Domain rules that are not obvious from the code @@ -96,13 +104,16 @@ These come from how gacha games actually schedule things, and they cause most bu - **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a recoverable disappointment; a confidently wrong date is the failure this product exists to prevent. -- **Game8 has no single template.** Three shapes are known so far, and a page may mix them: - 1. Label/value detail tables (`Event Start` / `Event End`) — Genshin. - 2. Column tables (`Event | Duration | Event Details | Rewards`) — NTE. - 3. Image-grid schedules with a bare `MM/DD` and no end — unsupportable, yields nothing by design. - 4. Combined cells that fold label, range and blurb together - (`Period: 08/09/26 - 08/30/26 During the event...`) — Arknights: Endfield. - Before assuming a new Game8 page will work, dump its structure and check which shape it uses. +- **Parsers are keyed by site, not game.** One `game8` parser serves six sources; `wikigg` serves + one. Adding a source for a known site is one `SOURCES` entry; a new site is a parser module. +- **Game8 has no single template.** Five shapes are known and a page may mix them: label/value + detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb + cells, and rowspan Start/End pairs. Full table in `docs/INGESTION.md`. Before assuming a new + Game8 page will work, dump its structure and check **every** table — Endfield was written off as + undatable on a pass that only inspected its `Duration` rows, and its real events were further + down the page. +- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer + per server region, which is the only reason `regionEnds` carries real data anywhere. - **Silent drops are the dangerous failure.** A date format the parser does not recognise makes events vanish with no error. Abbreviated months (`Apr. 29 - May 13, 2026`) are supported for exactly this reason. When adding a source, compare the parser's event count against an diff --git a/README.md b/README.md index bd59412..60ce090 100644 --- a/README.md +++ b/README.md @@ -10,16 +10,19 @@ No account. No login. Your completed events are saved in your browser and never ## Status -Early. The parsing pipeline and the interface work end to end against checked-in fixtures; the -server, database and scheduler are specified but not built. +Early, but usable. The parsing pipeline and the interface work end to end against checked-in +fixtures, and the whole thing builds, serves, containerises and deploys. The database, refresh +scheduler and review queue are specified but not built, so the feed is generated offline rather than +refreshing itself. | Piece | State | |---|---| | Event schema, date parsing, Game8 parser | Built, tested | | Six game sources | Built, tested | | Cross-source merge and conflict detection | Built, tested | -| Web interface, offline support | Built | -| Bun server, SQLite, refresh scheduler, review queue | Specified in `docs/`, not built | +| Web interface, offline support, first-run picker | Built | +| Static server, Docker image, GitHub + GitLab CI | Built | +| SQLite, refresh scheduler, review queue | Specified in `docs/`, not built | Today the feed is generated offline from fixtures. That is deliberate: it let the interface be built against real parsed data, and it produces exactly the shape the server will serve. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a582e39..b0bba88 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -57,29 +57,48 @@ src/ scheduler.ts timer + jitter + per-source lock [not built] pipeline.ts the 6 stages, orchestration only [not built] html.ts flat-table HTML reader (no dependency) ✓ built - dates.ts deterministic date parsing ✓ built + dates.ts six deterministic date formats ✓ built + merge.ts cross-source dedupe, corroboration ✓ built validate.ts zod parse + calendar sanity rules [not built] reconcile.ts diff vs published, confidence, conflicts [not built] + parsers/ + types.ts SourceParser interface ✓ built + game8.ts game8.co article calendars ✓ built + wikigg.ts wiki.gg mp-event templates ✓ built + index.ts parser registry ✓ built adapters/ types.ts Adapter interface, ParseContext ✓ built - game8.ts shared Game8 parser (2 table shapes) ✓ built - index.ts registry: adapter id → Adapter ✓ built + index.ts SOURCES registry, parseGame() ✓ built shared/ schema.ts zod schemas — the contract, both sides ✓ built - time.ts region reset math, duration formatting [not built] - client/ - main.tsx - App.tsx - views/ - Timeline.tsx F1 - EndingSoon.tsx F2 - EventDetail.tsx + time.ts clocks, urgency, region resets, captions ✓ built + games.ts per-game name and hue ✓ built + feed.ts the /api/events.json wire contract ✓ built + client/ ✓ all built + main.tsx render + service worker registration + App.tsx shell, views, filters, onboarding gate + api.ts typed feed fetch, schemaVersion refusal + sw.js offline: shell cache, feed fallback + manifest.webmanifest, icon.svg + components/ + NextUp.tsx the hero countdown (PRD F1) + EventRow.tsx row + meter + caption (F2, F3) + Meter.tsx the depletion meter + Legend.tsx what the bars and colours mean + Timeline.tsx calendar lanes (F1) + EventDetail.tsx detail sheet, ignore action (F9) + Controls.tsx games, region, export/import(F4, F5, F6) + Welcome.tsx first-run game picker (F8) + Colophon.tsx credit, disclaimer, repo link state/ - completions.ts localStorage read/write + export/import - prefs.ts region, filters - api.ts typed fetch of /api/events + storage.ts namespaced, versioned localStorage + useMarkSet.ts completions and ignores (same shape) + usePrefs.ts region, filters, onboarding flags +serve.ts static server + /api/health ✓ built +scripts/ + build-feed.ts fixtures → public/data/events.v1.json ✓ built + parse-fixture.ts run one adapter offline ✓ built fixtures// checked-in raw HTML + expected parse output -docs/ ``` ## Request paths @@ -137,13 +156,40 @@ PORT=3000 ADMIN_PORT=3001 # bound to 127.0.0.1 DATABASE_PATH=./data/events.sqlite INGEST_INTERVAL_MS=21600000 -INGEST_ENABLED=true # false for local UI work — never hits the network or the API +INGEST_ENABLED=true # false for local UI work — never touches the network CONFIDENCE_THRESHOLD=0.8 +BASE_PATH=/ # trailing slash; set when hosting under a subpath ``` `INGEST_ENABLED=false` is the default for local development. Frontend work should run against a seeded SQLite file and cost nothing. +### Today + +`serve.ts` serves `public/` plus `/api/health`, and the feed is generated offline from fixtures by +`bun run build:feed`. It emits exactly the shape `/api/events.json` will, so the real server slots in +without the client changing. Reads are confined to `public/` by resolving the path and checking it +stays inside the root — string-matching `..` is not enough, because encodings and URL normalisation +both change what the string looks like. + +`Dockerfile` builds and serves this; the image runs typecheck and tests during build, ships no source +or toolchain, and runs unprivileged. `.github/workflows/ci.yml` and `.gitlab-ci.yml` run the same +gates and publish it. + +### Hosting under a subpath + +Assets resolve against a `` substituted at build time, the feed URL resolves against +`document.baseURI` so deep links work, and the service worker derives its paths from its own +registration scope. `BASE_PATH=/gacha-event-tracker/ bun run build` for GitHub Pages; without it a +subpath deploy 404s on every asset. + +### Offline + +The service worker caches the shell and webfonts (cache-first) and the feed (network-first, falling +back to the last copy seen). Countdowns run off the device clock, so the app stays useful with no +network. Offline state is surfaced in the header and above the footer — stale data must never be +presented as current. + ## Deliberate non-choices - **No ORM.** `bun:sqlite` plus hand-written SQL in `queries.ts`. The schema is six tables. diff --git a/docs/DATA-MODEL.md b/docs/DATA-MODEL.md index 0da8729..04da3a6 100644 --- a/docs/DATA-MODEL.md +++ b/docs/DATA-MODEL.md @@ -196,11 +196,19 @@ malformed row surfaces at the boundary rather than deep in the UI. Namespaced, versioned, and small. Nothing here ever goes to the server. ```ts -"gacha-tracker:v1:completions" // { [eventId]: { completedAt: string } } -"gacha-tracker:v1:prefs" // { region, hiddenGames[], hiddenTypes[], showCompleted } -"gacha-tracker:v1:feedCache" // { fetchedAt, events } — offline fallback +"gacha-tracker:v1:completions" // { [eventId]: { at: string } } — "I finished this" +"gacha-tracker:v1:ignored" // { [eventId]: { at: string } } — "stop showing me this" +"gacha-tracker:v1:prefs" // { region, hiddenGames[], showCompleted, showIgnored, + // regionConfirmed, onboarded } ``` +Completions and ignores are the same shape and share one implementation +(`useMarkSet`), but stay in separate stores because they mean different things: a completed event is +dimmed and still counted, an ignored one disappears from both views. + +Offline caching is the service worker's job, not localStorage's — it caches the feed response +itself, so there is no second copy of the events to keep in sync. + The `v1` segment is the migration hook. On boot, the client checks for keys at older versions and migrates them forward before reading. **Never delete an old-version key until the migration has shipped and run** — a user who has not opened the app in six months still has their data under the @@ -213,14 +221,16 @@ old key. "format": "gacha-tracker-export", "version": 1, "exportedAt": "2026-08-14T12:00:00.000Z", - "completions": { "genshin:windblume-festival:2026-03-14": { "completedAt": "..." } }, - "prefs": { "region": "europe", "hiddenGames": [] } + "completions": { "genshin:windblume-festival:2026-03-14": { "at": "..." } }, + "ignored": { "zzz:some-event-i-skip:2026-08-19": { "at": "..." } }, + "prefs": { "region": "europe", "hiddenGames": [], "onboarded": true } } ``` -Import **merges**: a completion present in either the file or the current device stays completed. -Import never removes a completion. Losing a user's marks to a bad import is unrecoverable, so the -merge is deliberately one-directional. +Import **merges** both sets: a mark present in either the file or the current device survives, and +import never removes one. Losing a user's marks to a bad import is unrecoverable, so the merge is +deliberately one-directional. A file whose `format` is unrecognised is refused outright rather than +partly applied. ## Schema versioning diff --git a/docs/INGESTION.md b/docs/INGESTION.md index b722982..ad40f63 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -41,6 +41,33 @@ Consequences worth internalising: - A game may have any number of sources. `parseGame(game, documents, now)` runs them all and merges. +### Parsers in the tree + +| Parser | Site | Sources using it | +|---|---|---| +| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE | +| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield | + +`wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server +region, so its events carry exact precision and real `regionEnds`. Prefer a source like that over a +prose wiki when both exist, and give it a higher `priority`. + +### Date formats understood + +All live in `src/ingest/dates.ts`, each returning null rather than inferring anything: + +| Function | Shape | Seen on | +|---|---|---| +| `parseMonthDayYear` | `August 12, 2026` | Genshin detail rows | +| `parseMonthDayRange` | `August 12 - September 21, 2026` (year on the end only) | Genshin, NTE | +| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves | +| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield | +| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events | +| `parseOpenRange` | `Jul. 24, 2026 - End of 4.6`, `July 10, 2026 - Permanent` | Star Rail, Wuthering Waves | + +`parseOpenRange` is tried last because it is the most permissive — it accepts any leading full date +and reports no end. + ### The parser interface ```ts @@ -87,7 +114,7 @@ produced it. | Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing | | Free-form prose with no table structure | Find a different source | -Game8 uses at least three page templates and a game's page may use any of them: +Game8 uses at least four page templates and a game's page may use any of them: 1. **Label/value detail tables** — `Event Start` / `Event End` rows under a per-event `h3`, full dates with year. *(Genshin Impact)* @@ -96,8 +123,10 @@ Game8 uses at least three page templates and a game's page may use any of them: 3. **Image-grid schedules** — a bare `MM/DD`, no year, no end date. **Unsupportable.** 4. **Combined cells** — one cell holding label, range and blurb (`Period: 08/09/26 - 08/30/26 During the event...`). *(Arknights: Endfield)* +5. **Rowspan Start/End pairs** — the event name spans two rows, so a flat cell reader sees + `[title, "Start", date]` then `["End", date]`. *(Zenless Zone Zero)* -Shapes 1, 2 and 4 are handled. Before assuming a new Game8 page will work, dump its heading/table +Shapes 1, 2, 4 and 5 are handled. Before assuming a new Game8 page will work, dump its heading/table structure and check which shape it uses — and check **every** table, not just the obvious one. Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its two real events were in a table further down. diff --git a/docs/PRD.md b/docs/PRD.md index 4151a16..8634678 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -83,6 +83,29 @@ Because there are no accounts, moving between devices is manual: download a JSON IDs and preferences, upload it elsewhere. Import merges rather than replaces, and never removes a completion the user already has. +**F8 — First-run game picker.** +Before any events are shown, the reader picks which games they play. A calendar full of games they +don't play is worse than an empty one — it buries the thing they came for. Nothing is preselected +and the button stays disabled until something is chosen; guessing on their behalf and hoping they +notice is worse than asking. The choice is stored as *hidden* games, the inverse, so a game added +later appears by default rather than staying invisible forever. + +**F9 — Ignore an event.** +Distinct from completing one. "Done" keeps an event visible and counted; "not interested" removes it +from both views entirely. Ignored events stay recoverable: a count and a reveal toggle appear in +settings once there is something to reveal. + +**F10 — Works offline.** +The reader's question is answered entirely by data already on the device, and countdowns run off the +local clock, so losing signal should not lose the app. A service worker caches the shell and serves +the last feed it downloaded. Offline is disclosed in the header and above the footer — see F7; stale +data must never be presented as current. + +**F11 — Credit and disclaimer.** +The sources that compile these calendars, and the studios that make the games, are named on the same +screen as the data rather than one navigation step away. The page states plainly that it is +unofficial and unaffiliated, and that the source page is the authority when the two disagree. + **F7 — Freshness disclosure.** The footer shows when the feed was last updated, per game. If a game's data is more than 48 hours stale, its lane carries a warning badge. Never present stale data as current — the whole value @@ -91,7 +114,8 @@ proposition is trust in the dates. ## Out of scope for v1 Accounts and sync; push notifications; per-event checklists or progress tracking; in-game resource -or pull tracking; user-submitted events; mobile apps; localization beyond English. +or pull tracking; user-submitted events; native mobile apps (the web app installs to a home screen, +which is enough); localization beyond English. ## Success criteria @@ -119,6 +143,8 @@ product exists to prevent. - Does the calendar need a month/grid view, or is the timeline enough? (Assumption: timeline is enough for v1; revisit after use.) +- Should an ignored event still count toward the "N live" header total? (Assumption: no — ignoring + means gone.) - Should events the user has hidden by game filter still count toward "ends soonest"? (Assumption: no — the filter is global.) - Is 6 hours the right refresh cadence? (Assumption: yes; events are announced days ahead, so