docs: bring every markdown file up to date with the code

The docs had drifted in ways that would mislead: DATA-MODEL documented a
localStorage shape the code stopped using (completedAt, no ignored store),
INGESTION claimed three Game8 templates when five are known, ARCHITECTURE
still listed the whole client and time.ts as unbuilt, and the review-quarantine
skill described a pipeline that does not exist yet without saying so.

Adds the parser roster and the six date formats as tables, documents the
subpath/base-href and offline behaviour, and records the new product surface
(first-run picker, ignore, offline, credit) as PRD features.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-15 01:27:56 +02:00
co-authored by Claude Opus 5
parent f2cf9ba0cb
commit 692b5a83d5
9 changed files with 211 additions and 67 deletions
+13 -5
View File
@@ -24,8 +24,8 @@ Fetch the page exactly once.
**3. Decide whether an existing parser covers it.**
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game — one Game8 parser serves
every Game8 page. Check `PARSERS` first:
Parsers live in `src/ingest/parsers/` and are keyed by *site*, not game. Two exist: `game8`
(six sources) and `wikigg` (wiki.gg MediaWiki `mp-event` templates). Check `PARSERS` first:
- **Existing parser handles it** → add one entry to `SOURCES` in `adapters/index.ts`. No new
parsing code. This is the common case and should be the first thing you try.
@@ -34,6 +34,10 @@ every Game8 page. Check `PARSERS` first:
- **Undatable source** (no year, no end date, image-grid schedule) → **stop and report.** There is
no LLM fallback in this pipeline by design. Suggest a different source.
**Check every table before concluding a page is undatable.** Endfield was written off on a pass that
only inspected its `Duration` rows; its real events were in a table further down, and a second
source (wiki.gg) turned out to publish ISO timestamps with per-region timers.
**4. Implement the parser (only if step 3 says you need one).**
- `parse` must be **pure**: no network, no `Date.now()`, no randomness. Time comes from `ctx.now`.
@@ -48,9 +52,13 @@ every Game8 page. Check `PARSERS` first:
- An unstated end is `endsAt: null` + `endPrecision: "unknown"`. **Never compute a plausible end
from typical patch length.** This is the failure mode that makes the product worthless.
**5. Write the test.** `fixtures/<game>/<source-id>-<YYYY-MM-DD>.expected.json` holds the exact
expected `GachaEvent[]`. The test runs `parse` + `normalize` against the fixture with a pinned
`ctx.now` and asserts deep equality.
**5. Write the test.** `fixtures/<game>/<site>-events-<YYYY-MM-DD>.expected.json` holds the exact
expected `GachaEvent[]`. Add the source to `CASES` in `test/adapters/game8.test.ts`; the shared
assertions (schema, determinism, no backwards intervals, 180-day cap, unique IDs) then apply for
free. Regenerate the expected file with `bun run parse <id> <fixture> --json`.
**Fixture names matter**: `build-feed` selects by `<site>-*` within the game directory, so a game
with two sources needs distinct site prefixes.
**6. Verify.** Run `bun test` and confirm it passes with no network. Then hand-check three or four
events against the live page and state in your report that you did — a green test against an
+13 -7
View File
@@ -35,9 +35,13 @@ only schema edit a new game should require. If it needs more, that is a finding
Fetch the source page **once** and save the raw HTML:
```
fixtures/<game>/<source-id>-<YYYY-MM-DD>.html
fixtures/<game>/<site>-events-<YYYY-MM-DD>.html
```
The `<site>` prefix is load-bearing: `build-feed` picks fixtures by site within the game directory,
so a game with two sources whose files share a prefix will hand one site's page to the other's
parser.
Everything after this point works offline against that file. Do not re-fetch while iterating on the
parser.
@@ -77,16 +81,18 @@ report that you did this.
`parserId`, and optionally `priority` (higher wins when sources disagree).
- Insert the matching `sources` row: id, game, url, parser_id, priority, `min_interval_ms`.
## 7. First run
## 7. Rebuild the feed
```
INGEST_ENABLED=true bun run ingest --source <source-id> --dry-run
bun run build:feed # regenerates public/data/events.v1.json
bun run dev # build and serve on :3000
```
Inspect what it *would* publish before letting it write. Then run for real and check the review
queue at `http://127.0.0.1:$ADMIN_PORT/review` — a new source commonly lands events in quarantine
on its first pass, because nothing corroborates it yet. That is the gate working, not a bug.
Review and approve them.
Check the event count and any conflicts the merge reports. A game with two sources will surface
disagreements — those are the gate working, not a bug.
The scheduler, quarantine table and `/review` queue described in `docs/INGESTION.md` are **not built
yet**; today the feed is generated offline from fixtures.
## Checklist
@@ -5,6 +5,11 @@ description: Work through the quarantined-event queue — triage held events by
# Reviewing the quarantine queue
> **Not built yet.** The quarantine table, the `/review` route and the ingest scheduler are specified
> in `docs/INGESTION.md` but do not exist in the tree. Today the closest equivalent is the conflict
> list `bun run build:feed` prints when two sources disagree. Use this skill once the pipeline lands;
> until then, treat it as the spec for what that review flow should do.
Held events are candidates the pipeline declined to publish. Working the queue is both a data task
(get these events onto the calendar) and a diagnostic one (**a growing queue means something
upstream broke**).
+35 -24
View File
@@ -7,9 +7,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
A web app that aggregates live and upcoming events across popular gacha games, plots them on a
calendar, sorts them by end date, and lets a user mark events completed.
**Status: first vertical slice.** The schema, the Game8 parser, and two working adapters (Genshin
Impact, Neverness to Everness) exist and are tested. The server, database, and UI do not exist yet —
`docs/` specifies them.
**Status: working app, no live refresh.** Schema, two parsers, seven sources across six games, the
full interface, offline support, a static server, a Docker image and CI all exist and are tested.
The SQLite layer, the refresh scheduler and the review queue are specified in `docs/` but not built,
so the feed is generated offline from checked-in fixtures.
## Three constraints that shape everything
@@ -39,18 +40,26 @@ parsing library — Bun covers all four. `tsconfig.json` runs `strict` plus
```bash
bun install
bun test # full suite, offline, no network
bun test # full suite, offline, no network, no build needed
bun run typecheck # tsc --noEmit
bun run dev # build then serve on :3000
bun run build # feed + css + js + static into public/
# Run an adapter against a checked-in fixture (offline, free)
# Run one source against its fixture (offline, free)
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
bun run parse nte-game8-events fixtures/nte/game8-events-2026-08-14.html --json
bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
# Single test file / single test
bun test test/dates.test.ts
bun test --test-name-pattern "year-less"
# Hosting under a subpath (GitHub Pages)
BASE_PATH=/gacha-event-tracker/ bun run build
```
**Tests must never need build output.** They run before `bun run build` in CI; anything reading
`public/` must create its own fixture tree instead.
`bun run parse ... --json` is also how `.expected.json` fixtures are regenerated after an
intentional parser change. Regenerating them makes the test self-consistent, not correct — always
re-verify a sample against the live page afterward.
@@ -58,19 +67,18 @@ re-verify a sample against the live page afterward.
## Current state of the code
```
src/shared/schema.ts Zod GachaEvent, GameId, slugify, eventId ← the contract
src/ingest/html.ts flat-table HTML reader (no dependency)
src/ingest/dates.ts three date formats, null rather than guess
src/ingest/adapters/
types.ts Adapter interface, ParseContext
game8.ts shared Game8 parser — handles 2 table shapes
index.ts adapter registry
scripts/parse-fixture.ts offline adapter runner
test/ 37 tests
fixtures/<game>/ raw HTML + .expected.json per source
src/shared/ schema.ts (the contract), time.ts, games.ts, feed.ts
src/ingest/ html.ts, dates.ts (six formats), merge.ts
parsers/ game8.ts, wikigg.ts — keyed by SITE, not game
adapters/ index.ts — SOURCES registry binding url+game+parser
src/client/ React app, service worker, manifest
scripts/ build-feed.ts, parse-fixture.ts (both offline)
serve.ts static server + /api/health
test/ 112 tests
fixtures/<game>/ raw HTML + .expected.json per source
```
Not yet built: `src/server/**`, `src/client/**`, the SQLite layer, the scheduler, the review UI.
Not yet built: the SQLite layer, the ingest scheduler, the review UI.
## Domain rules that are not obvious from the code
@@ -96,13 +104,16 @@ These come from how gacha games actually schedule things, and they cause most bu
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
- **Game8 has no single template.** Three shapes are known so far, and a page may mix them:
1. Label/value detail tables (`Event Start` / `Event End`) — Genshin.
2. Column tables (`Event | Duration | Event Details | Rewards`) — NTE.
3. Image-grid schedules with a bare `MM/DD` and no end — unsupportable, yields nothing by design.
4. Combined cells that fold label, range and blurb together
(`Period: 08/09/26 - 08/30/26 During the event...`) — Arknights: Endfield.
Before assuming a new Game8 page will work, dump its structure and check which shape it uses.
- **Parsers are keyed by site, not game.** One `game8` parser serves six sources; `wikigg` serves
one. Adding a source for a known site is one `SOURCES` entry; a new site is a parser module.
- **Game8 has no single template.** Five shapes are known and a page may mix them: label/value
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
cells, and rowspan Start/End pairs. Full table in `docs/INGESTION.md`. Before assuming a new
Game8 page will work, dump its structure and check **every** table — Endfield was written off as
undatable on a pass that only inspected its `Duration` rows, and its real events were further
down the page.
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
per server region, which is the only reason `regionEnds` carries real data anywhere.
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
events vanish with no error. Abbreviated months (`Apr. 29 - May 13, 2026`) are supported for
exactly this reason. When adding a source, compare the parser's event count against an
+7 -4
View File
@@ -10,16 +10,19 @@ No account. No login. Your completed events are saved in your browser and never
## Status
Early. The parsing pipeline and the interface work end to end against checked-in fixtures; the
server, database and scheduler are specified but not built.
Early, but usable. The parsing pipeline and the interface work end to end against checked-in
fixtures, and the whole thing builds, serves, containerises and deploys. The database, refresh
scheduler and review queue are specified but not built, so the feed is generated offline rather than
refreshing itself.
| Piece | State |
|---|---|
| Event schema, date parsing, Game8 parser | Built, tested |
| Six game sources | Built, tested |
| Cross-source merge and conflict detection | Built, tested |
| Web interface, offline support | Built |
| Bun server, SQLite, refresh scheduler, review queue | Specified in `docs/`, not built |
| Web interface, offline support, first-run picker | Built |
| Static server, Docker image, GitHub + GitLab CI | Built |
| SQLite, refresh scheduler, review queue | Specified in `docs/`, not built |
Today the feed is generated offline from fixtures. That is deliberate: it let the interface be built
against real parsed data, and it produces exactly the shape the server will serve.
+62 -16
View File
@@ -57,29 +57,48 @@ src/
scheduler.ts timer + jitter + per-source lock [not built]
pipeline.ts the 6 stages, orchestration only [not built]
html.ts flat-table HTML reader (no dependency) ✓ built
dates.ts deterministic date parsing ✓ built
dates.ts six deterministic date formats ✓ built
merge.ts cross-source dedupe, corroboration ✓ built
validate.ts zod parse + calendar sanity rules [not built]
reconcile.ts diff vs published, confidence, conflicts [not built]
parsers/
types.ts SourceParser interface ✓ built
game8.ts game8.co article calendars ✓ built
wikigg.ts wiki.gg mp-event templates ✓ built
index.ts parser registry ✓ built
adapters/
types.ts Adapter interface, ParseContext ✓ built
game8.ts shared Game8 parser (2 table shapes) ✓ built
index.ts registry: adapter id → Adapter ✓ built
index.ts SOURCES registry, parseGame() ✓ built
shared/
schema.ts zod schemas — the contract, both sides ✓ built
time.ts region reset math, duration formatting [not built]
client/
main.tsx
App.tsx
views/
Timeline.tsx F1
EndingSoon.tsx F2
EventDetail.tsx
time.ts clocks, urgency, region resets, captions ✓ built
games.ts per-game name and hue ✓ built
feed.ts the /api/events.json wire contract ✓ built
client/ ✓ all built
main.tsx render + service worker registration
App.tsx shell, views, filters, onboarding gate
api.ts typed feed fetch, schemaVersion refusal
sw.js offline: shell cache, feed fallback
manifest.webmanifest, icon.svg
components/
NextUp.tsx the hero countdown (PRD F1)
EventRow.tsx row + meter + caption (F2, F3)
Meter.tsx the depletion meter
Legend.tsx what the bars and colours mean
Timeline.tsx calendar lanes (F1)
EventDetail.tsx detail sheet, ignore action (F9)
Controls.tsx games, region, export/import(F4, F5, F6)
Welcome.tsx first-run game picker (F8)
Colophon.tsx credit, disclaimer, repo link
state/
completions.ts localStorage read/write + export/import
prefs.ts region, filters
api.ts typed fetch of /api/events
storage.ts namespaced, versioned localStorage
useMarkSet.ts completions and ignores (same shape)
usePrefs.ts region, filters, onboarding flags
serve.ts static server + /api/health ✓ built
scripts/
build-feed.ts fixtures → public/data/events.v1.json ✓ built
parse-fixture.ts run one adapter offline ✓ built
fixtures/<game>/ checked-in raw HTML + expected parse output
docs/
```
## Request paths
@@ -137,13 +156,40 @@ PORT=3000
ADMIN_PORT=3001 # bound to 127.0.0.1
DATABASE_PATH=./data/events.sqlite
INGEST_INTERVAL_MS=21600000
INGEST_ENABLED=true # false for local UI work — never hits the network or the API
INGEST_ENABLED=true # false for local UI work — never touches the network
CONFIDENCE_THRESHOLD=0.8
BASE_PATH=/ # trailing slash; set when hosting under a subpath
```
`INGEST_ENABLED=false` is the default for local development. Frontend work should run against a
seeded SQLite file and cost nothing.
### Today
`serve.ts` serves `public/` plus `/api/health`, and the feed is generated offline from fixtures by
`bun run build:feed`. It emits exactly the shape `/api/events.json` will, so the real server slots in
without the client changing. Reads are confined to `public/` by resolving the path and checking it
stays inside the root — string-matching `..` is not enough, because encodings and URL normalisation
both change what the string looks like.
`Dockerfile` builds and serves this; the image runs typecheck and tests during build, ships no source
or toolchain, and runs unprivileged. `.github/workflows/ci.yml` and `.gitlab-ci.yml` run the same
gates and publish it.
### Hosting under a subpath
Assets resolve against a `<base href>` substituted at build time, the feed URL resolves against
`document.baseURI` so deep links work, and the service worker derives its paths from its own
registration scope. `BASE_PATH=/gacha-event-tracker/ bun run build` for GitHub Pages; without it a
subpath deploy 404s on every asset.
### Offline
The service worker caches the shell and webfonts (cache-first) and the feed (network-first, falling
back to the last copy seen). Countdowns run off the device clock, so the app stays useful with no
network. Offline state is surfaced in the header and above the footer — stale data must never be
presented as current.
## Deliberate non-choices
- **No ORM.** `bun:sqlite` plus hand-written SQL in `queries.ts`. The schema is six tables.
+18 -8
View File
@@ -196,11 +196,19 @@ malformed row surfaces at the boundary rather than deep in the UI.
Namespaced, versioned, and small. Nothing here ever goes to the server.
```ts
"gacha-tracker:v1:completions" // { [eventId]: { completedAt: string } }
"gacha-tracker:v1:prefs" // { region, hiddenGames[], hiddenTypes[], showCompleted }
"gacha-tracker:v1:feedCache" // { fetchedAt, events } — offline fallback
"gacha-tracker:v1:completions" // { [eventId]: { at: string } } — "I finished this"
"gacha-tracker:v1:ignored" // { [eventId]: { at: string } } — "stop showing me this"
"gacha-tracker:v1:prefs" // { region, hiddenGames[], showCompleted, showIgnored,
// regionConfirmed, onboarded }
```
Completions and ignores are the same shape and share one implementation
(`useMarkSet`), but stay in separate stores because they mean different things: a completed event is
dimmed and still counted, an ignored one disappears from both views.
Offline caching is the service worker's job, not localStorage's — it caches the feed response
itself, so there is no second copy of the events to keep in sync.
The `v1` segment is the migration hook. On boot, the client checks for keys at older versions and
migrates them forward before reading. **Never delete an old-version key until the migration has
shipped and run** — a user who has not opened the app in six months still has their data under the
@@ -213,14 +221,16 @@ old key.
"format": "gacha-tracker-export",
"version": 1,
"exportedAt": "2026-08-14T12:00:00.000Z",
"completions": { "genshin:windblume-festival:2026-03-14": { "completedAt": "..." } },
"prefs": { "region": "europe", "hiddenGames": [] }
"completions": { "genshin:windblume-festival:2026-03-14": { "at": "..." } },
"ignored": { "zzz:some-event-i-skip:2026-08-19": { "at": "..." } },
"prefs": { "region": "europe", "hiddenGames": [], "onboarded": true }
}
```
Import **merges**: a completion present in either the file or the current device stays completed.
Import never removes a completion. Losing a user's marks to a bad import is unrecoverable, so the
merge is deliberately one-directional.
Import **merges** both sets: a mark present in either the file or the current device survives, and
import never removes one. Losing a user's marks to a bad import is unrecoverable, so the merge is
deliberately one-directional. A file whose `format` is unrecognised is refused outright rather than
partly applied.
## Schema versioning
+31 -2
View File
@@ -41,6 +41,33 @@ Consequences worth internalising:
- A game may have any number of sources. `parseGame(game, documents, now)` runs them all and
merges.
### Parsers in the tree
| Parser | Site | Sources using it |
|---|---|---|
| `game8` | game8.co article calendars | Genshin, Star Rail, Wuthering Waves, ZZZ, Endfield, NTE |
| `wikigg` | wiki.gg MediaWiki `mp-event` templates | Endfield |
`wikigg` is the better shape by a distance: it emits ISO timestamps with one timer per server
region, so its events carry exact precision and real `regionEnds`. Prefer a source like that over a
prose wiki when both exist, and give it a higher `priority`.
### Date formats understood
All live in `src/ingest/dates.ts`, each returning null rather than inferring anything:
| Function | Shape | Seen on |
|---|---|---|
| `parseMonthDayYear` | `August 12, 2026` | Genshin detail rows |
| `parseMonthDayRange` | `August 12 - September 21, 2026` (year on the end only) | Genshin, NTE |
| `parseFullRange` | `Aug. 14, 2026 - Aug. 24, 2026` (a year each side) | Star Rail, Wuthering Waves |
| `parseShortSlashRange` | `08/09/26 - 08/30/26` | Endfield |
| `parseSlashDateTimeRange` | `2021/01/16 04:00 - 2021/01/31 03:59` | Genshin past events |
| `parseOpenRange` | `Jul. 24, 2026 - End of 4.6`, `July 10, 2026 - Permanent` | Star Rail, Wuthering Waves |
`parseOpenRange` is tried last because it is the most permissive — it accepts any leading full date
and reports no end.
### The parser interface
```ts
@@ -87,7 +114,7 @@ produced it.
| Dates without a year, or no end date at all | **Unsupportable** — yields nothing rather than guessing |
| Free-form prose with no table structure | Find a different source |
Game8 uses at least three page templates and a game's page may use any of them:
Game8 uses at least four page templates and a game's page may use any of them:
1. **Label/value detail tables**`Event Start` / `Event End` rows under a per-event `h3`, full
dates with year. *(Genshin Impact)*
@@ -96,8 +123,10 @@ Game8 uses at least three page templates and a game's page may use any of them:
3. **Image-grid schedules** — a bare `MM/DD`, no year, no end date. **Unsupportable.**
4. **Combined cells** — one cell holding label, range and blurb
(`Period: 08/09/26 - 08/30/26 During the event...`). *(Arknights: Endfield)*
5. **Rowspan Start/End pairs** — the event name spans two rows, so a flat cell reader sees
`[title, "Start", date]` then `["End", date]`. *(Zenless Zone Zero)*
Shapes 1, 2 and 4 are handled. Before assuming a new Game8 page will work, dump its heading/table
Shapes 1, 2, 4 and 5 are handled. Before assuming a new Game8 page will work, dump its heading/table
structure and check which shape it uses — and check **every** table, not just the obvious one.
Endfield was written off as undatable on a first pass that only inspected its `Duration` rows; its
two real events were in a table further down.
+27 -1
View File
@@ -83,6 +83,29 @@ Because there are no accounts, moving between devices is manual: download a JSON
IDs and preferences, upload it elsewhere. Import merges rather than replaces, and never removes a
completion the user already has.
**F8 — First-run game picker.**
Before any events are shown, the reader picks which games they play. A calendar full of games they
don't play is worse than an empty one — it buries the thing they came for. Nothing is preselected
and the button stays disabled until something is chosen; guessing on their behalf and hoping they
notice is worse than asking. The choice is stored as *hidden* games, the inverse, so a game added
later appears by default rather than staying invisible forever.
**F9 — Ignore an event.**
Distinct from completing one. "Done" keeps an event visible and counted; "not interested" removes it
from both views entirely. Ignored events stay recoverable: a count and a reveal toggle appear in
settings once there is something to reveal.
**F10 — Works offline.**
The reader's question is answered entirely by data already on the device, and countdowns run off the
local clock, so losing signal should not lose the app. A service worker caches the shell and serves
the last feed it downloaded. Offline is disclosed in the header and above the footer — see F7; stale
data must never be presented as current.
**F11 — Credit and disclaimer.**
The sources that compile these calendars, and the studios that make the games, are named on the same
screen as the data rather than one navigation step away. The page states plainly that it is
unofficial and unaffiliated, and that the source page is the authority when the two disagree.
**F7 — Freshness disclosure.**
The footer shows when the feed was last updated, per game. If a game's data is more than 48 hours
stale, its lane carries a warning badge. Never present stale data as current — the whole value
@@ -91,7 +114,8 @@ proposition is trust in the dates.
## Out of scope for v1
Accounts and sync; push notifications; per-event checklists or progress tracking; in-game resource
or pull tracking; user-submitted events; mobile apps; localization beyond English.
or pull tracking; user-submitted events; native mobile apps (the web app installs to a home screen,
which is enough); localization beyond English.
## Success criteria
@@ -119,6 +143,8 @@ product exists to prevent.
- Does the calendar need a month/grid view, or is the timeline enough? (Assumption: timeline is
enough for v1; revisit after use.)
- Should an ignored event still count toward the "N live" header total? (Assumption: no — ignoring
means gone.)
- Should events the user has hidden by game filter still count toward "ends soonest"?
(Assumption: no — the filter is global.)
- Is 6 hours the right refresh cadence? (Assumption: yes; events are announced days ahead, so