lucasw89/gacha-event-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: cover the refresh pipeline, sanitisation and dailies

Browse Source 
The status sections claimed the feed was generated from fixtures and the
scheduler unbuilt, which stopped being true. Also documents the
sanitisation stage and the two new key spaces — `dailies:<game>` and
game-day keys — beside the existing warning about event IDs, since they
carry the same "no server-side recovery" property.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-15 21:19:08 +02:00
co-authored by Claude Opus 5
parent 8e42afdd7a
commit 02863ed008
4 changed files with 243 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+73 -13
View File
@@ -5,12 +5,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## What this is
A web app that aggregates live and upcoming events across popular gacha games, plots them on a
calendar, sorts them by end date, and lets a user mark events completed.
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
progress on events that repeat daily, and lets a user mark events completed.
**Status: working app, no live refresh.** Schema, two parsers, seven sources across six games, the
full interface, offline support, a static server, a Docker image and CI all exist and are tested.
The SQLite layer, the refresh scheduler and the review queue are specified in `docs/` but not built,
so the feed is generated offline from checked-in fixtures.
**Status: working app, refreshing itself on a schedule.** Schema, two parsers, seven sources across
six games, the full interface, offline support, a static server, a Docker image and CI all exist and
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
feed is a static JSON file built from snapshots, falling back to checked-in fixtures.
## Three constraints that shape everything
@@ -45,6 +48,11 @@ bun run typecheck # tsc --noEmit
bun run dev # build then serve on :3000
bun run build # feed + css + js + static into public/
# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping
# conduct before running it, and prefer --dry-run.
bun run refresh --dry-run
bun run refresh --only genshin-game8-events
# Run one source against its fixture (offline, free)
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
@@ -67,18 +75,21 @@ re-verify a sample against the live page afterward.
## Current state of the code
```
src/shared/ schema.ts (the contract), time.ts, games.ts, feed.ts
src/ingest/ html.ts, dates.ts (six formats), merge.ts
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
src/ingest/ html.ts, dates.ts (six formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
parsers/ game8.ts, wikigg.ts — keyed by SITE, not game
adapters/ index.ts — SOURCES registry binding url+game+parser
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
src/client/ React app, service worker, manifest
scripts/ build-feed.ts, parse-fixture.ts (both offline)
state/ progress, daily log, ignores, prefs, sort — all localStorage
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
serve.ts static server + /api/health
test/ 112 tests
fixtures/<game>/ raw HTML + .expected.json per source
test/ 257 tests
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
snapshots/ current page per source, rewritten by refresh — see its README
```
Not yet built: the SQLite layer, the ingest scheduler, the review UI.
Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on
disk.
## Domain rules that are not obvious from the code
@@ -132,6 +143,18 @@ recovery**, because the server never had the data. If it must change, ship a cli
that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such
change.
Two more key spaces have the same property, for the same reason:
- **`dailies:<game>`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore.
Two segments, so it cannot collide with an event ID.
- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at
04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the
format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day.
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
fixtures, that is a data-loss bug, not a diff to regenerate.
## Scraping conduct
Sources are community wikis. Treat them as a guest would:
@@ -149,6 +172,35 @@ rate, and do not add an LLM that consumes page content.
A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything
that would make it fetch more often is a change to this section first.
## Untrusted input
Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust
boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single
seam every source passes through — **do not sanitize inside a parser**, and do not add a code path
that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup.
The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is
the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5.
## Events that repeat daily
Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed
day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published —
`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an
event's length. It adds **no schema field**, so the feed contract is untouched.
- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks
the wrong box for four hours every night.
- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null`
rule applies here exactly as it does to a countdown.
- **A tick is never removed except by the reader**, including ticks outside the window the feed now
claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere
else.
## Conventions
- **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never
@@ -156,4 +208,12 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and
- Every adapter ships a fixture in `fixtures/<game>/` and a test asserting parsed output. This is
how a source silently changing shape gets caught.
- Keep old fixtures when a source changes shape — the old one is the regression test proving the
parser still handles the previous format.
parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the
current page and gets overwritten. Do not conflate them.
- **A list row is one target.** The event row opens the event and does nothing else — status,
effort, notes and the daily checklist all live in the detail sheet. A second control inside a
full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens"
without adding a second stop for keyboard and screen-reader users.
- **Sorting groups, it never reorders within a group.** Every mode falls back to
`endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product
exists for.
README.md
+87 -21
View File
@@ -6,27 +6,33 @@ You play three or four gacha games. Each has its own calendar, none of them talk
the only question that actually matters — *what runs out first?* — takes four browser tabs to
answer. This does it in one screen.
No account. No login. What you've finished, what you're partway through, and how much work you
reckon each event is are saved in your browser and never leave your device.
No account. No login. What you've finished, what you're partway through, how much work you reckon
each event is, and which day of a daily you've ticked off are saved in your browser and never leave
your device.
## Status
Early, but usable. The parsing pipeline and the interface work end to end against checked-in
fixtures, and the whole thing builds, serves, containerises and deploys. The database, refresh
scheduler and review queue are specified but not built, so the feed is generated offline rather than
refreshing itself.
Usable, and it now keeps itself up to date. The parsing pipeline and the interface work end to end,
a scheduled job refreshes the sources twice a day, and the whole thing builds, serves, containerises
and deploys. The database and review queue are specified but not built.
| Piece | State |
|---|---|
| Event schema, date parsing, Game8 parser | Built, tested |
| Six game sources | Built, tested |
| Cross-source merge and conflict detection | Built, tested |
| Web interface, offline support, first-run picker | Built |
| Input sanitization at the ingest boundary | Built, tested |
| Scheduled refresh — robots, snapshots, commit-on-change | Built, tested offline |
| Web interface, daily checklists, offline support | Built |
| Static server, Docker image, GitHub + GitLab CI | Built |
| SQLite, refresh scheduler, review queue | Specified in `docs/`, not built |
| SQLite, review queue | Specified in `docs/`, not built |
Today the feed is generated offline from fixtures. That is deliberate: it let the interface be built
against real parsed data, and it produces exactly the shape the server will serve.
The feed is still a static JSON file rather than a database read: the refresh job commits the raw
pages it fetched, CI rebuilds the feed from them, and a clean checkout with no snapshots falls back
to the checked-in fixtures — so the build stays offline and reproducible either way.
The refresh runner is tested against a fake fetch, never a live wiki. Its first real run against
game8.co and wiki.gg is unproven.
## Try it
@@ -53,7 +59,11 @@ reproducible and a wiki being down never breaks it.
bun test # full suite, offline, no network
bun run typecheck # tsc --noEmit
bun run build # feed + css + js + html into public/
bun run build:feed # regenerate public/data/events.v1.json from fixtures
bun run build:feed # regenerate public/data/events.v1.json from snapshots, else fixtures
# Fetch the sources. Makes real requests, so read "Conduct" first.
bun run refresh --dry-run # plan only: no requests, no writes
bun run refresh --only genshin-game8-events
# Run one source against its fixture and print what it yields
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
@@ -63,11 +73,12 @@ bun run parse nte-game8-events fixtures/nte/game8-events-2026-08-14.html --j
## How it works
```
game wikis ──► fetch ──► parse ──► merge ──► validate ──► gate ──► feed ──► browser
│ │ │ │
per-site per-game hold anything localStorage:
parser corroboration uncertain for what you've
and conflicts human review finished
game wikis ─► fetch ─► parse ─► sanitize ─► merge ─► validate ─► gate ─► feed ─► browser
│ │ │ │ │ │
robots, 6h, per-site untrusted per-game hold anything localStorage:
conditional, parser text corroboration uncertain for what you've done,
snapshots bounded and conflicts human review day by day if it
repeats daily
```
**Parsers are deterministic code.** There is no LLM anywhere in the pipeline — no API key, no
@@ -126,6 +137,39 @@ Arknights is defined in the schema and awaiting a source.
Full walkthrough in `docs/INGESTION.md`, or run the `add-game-source` skill.
## Dailies
Some things are not one job with a deadline. A login campaign is twenty small jobs on twenty
separate deadlines, and a day you miss is gone whatever you do afterwards — which a single "done"
tick cannot express.
So events that repeat get a checklist instead: today's tick, a strip of every day in the run showing
what you got and what you missed, your streak, and how many chances are left. Past days stay
editable, because people tick up later and a checklist you can't correct stops being trusted after
the first mistake.
Alongside them sits **today's dailies** — commissions, sanity, daily training — one tick per game.
No wiki publishes those, so they are a fixed list in the app rather than scraped data, and they are
the only thing on the page that expires tonight rather than next patch.
Both roll over at **04:00 server time** in your region, not midnight, because that is when the games
roll over. Finishing at 02:00 still counts as yesterday.
Repeating events are recognised from what the source actually printed — a login event type, or
wording like "daily", "check-in", "7-day". Nothing is assumed from a game's habits, and an event
whose end was never announced gets a day count rather than a checklist of invented length.
## Sorting
Two orders, and the toggle sits with the list rather than in settings:
- **Ending soonest** — the default, and the reason this app exists.
- **Doing first** — what you're partway through, floated to the top. Ticking a daily counts as
"doing it" without your having to say so twice.
Sorting only ever *groups*. Deadline order survives inside every group, so choosing an order can
never cost you the one thing you came for.
## Offline
The app works with no network. A service worker caches the shell and webfonts, and serves the last
@@ -137,8 +181,13 @@ It installs to a home screen as a standalone app.
## Conduct
Sources are community wikis, treated as a guest would: `robots.txt` honoured, a descriptive
`User-Agent`, one request per source per six hours, conditional requests, and raw snapshots cached
so iteration never re-fetches. Every event links back to its source.
`User-Agent` with a contact URL, one request per source per six hours, conditional requests, and raw
snapshots cached so iteration never re-fetches. Every event links back to its source.
`bun run refresh` enforces all of that in code rather than leaving it to good intentions: the
six-hour floor is checked per source, there are no retries (a retry is a second request), and a
`robots.txt` that cannot be read means *do not fetch* rather than *assume yes*. Text scraped from a
page is sanitized at the ingest boundary before it reaches the feed, the browser or your disk.
## Documentation
@@ -164,9 +213,26 @@ Until then the `pages` job is skipped and the pipeline stays green. Pages is una
repositories on the free plan.
The feed job fails if the event count collapses. A source that quietly stops yielding events is the
failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Everything
runs offline against checked-in fixtures, so a red pipeline always means the code changed rather
than a wiki being down.
failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Tests run
offline against checked-in fixtures, so a red pipeline always means the code changed rather than a
wiki being down.
### Refreshing the data
GitHub Actions only — the GitLab pipeline still runs the gates, but nothing there fetches.
`.github/workflows/refresh.yml` runs `bun run refresh` twice a day (and on demand, with a dry-run
input). It fetches each source at most once per cycle, and **commits only when a page's bytes
actually changed** — a `304`, an identical body, or a fetch that fails to parse all leave the
working tree clean and produce no commit. When something did change it commits the raw snapshots and
dispatches `ci.yml`, which typechecks, tests, rebuilds the feed and deploys through the path that
already existed; none of that logic is duplicated.
A body that yields zero events is rejected and the previous snapshot kept, so a wiki redesign shows
up as a stale timestamp rather than an empty calendar. One source being down is a warning; every
source being down fails the run, so a bad cycle never gets committed.
The schedule is off for forks (it is pinned to this repository) — a fork owner can still dispatch it
by hand and take responsibility for the traffic.
### Hosting under a subpath
docs/DATA-MODEL.md
+41 -4
View File
@@ -197,8 +197,9 @@ Namespaced, versioned, and small. Nothing here ever goes to the server.
```ts
"gacha-tracker:v1:progress" // { [eventId]: { status?, effort?, note?, at } }
"gacha-tracker:v1:daily" // { [id]: { days: ["2026-08-15", ...], at } }
"gacha-tracker:v1:ignored" // { [eventId]: { at } } — "stop showing me this"
"gacha-tracker:v1:prefs" // { region, hiddenGames[], showCompleted, showIgnored,
"gacha-tracker:v1:prefs" // { region, hiddenGames[], sort, showCompleted, showIgnored,
// regionConfirmed, onboarded }
"gacha-tracker:v1:completions" // SUPERSEDED — read once to migrate, never written
```
@@ -225,6 +226,36 @@ about it would be fabricating the reader's own input.
Ignores stay in a separate store because they mean something different: a done event is dimmed and
still counted, an ignored one disappears from both views.
### Daily checklists
Some events are not one job with a deadline but twenty small jobs on twenty separate deadlines, and
a missed day is gone whatever you do afterwards. `daily` records which game-days the reader ticked
off, keyed by:
- an **event ID**, for a repeating event in the feed (`src/shared/daily.ts` § `isDaily`), or
- **`dailies:<game>`**, the standing per-game chore — commissions, sanity, daily training. No source
publishes these, so they are a fixed client-side list, never feed data. The two-segment shape
cannot collide with an event ID, which is always `game:slug:date`.
Day keys are `YYYY-MM-DD` in **game-day space, not UTC**: gacha servers roll the day at 04:00 local
server time, so a player finishing at 02:00 is still on the previous day's dailies, and the key is
computed against the reader's chosen region (`RESET_HOUR_LOCAL`, `dayKey`). Keys sort
lexicographically, which is what "how many days are left" and streak counting rely on.
Three rules this store keeps, for the same reason the rest of the client does — nothing else holds
a copy:
- **A tick is never removed except by the reader.** Ticks that fall outside the window the feed now
claims still count; a source quietly moving a date must not erase a fortnight's streak.
- **An unannounced end yields no checklist**, not a checklist of guessed length. `dailyDays` returns
null when `endsAt` is null, and the UI says how many days are ticked instead of how many are left.
- **Past days stay editable.** People log in and tick up later, and a checklist that cannot be
corrected stops being trusted after the first mistake.
Dailiness is derived from the published event — `type: "login"`, or wording like "daily",
"check-in", "7-day" in the title or summary — and never from a game's habits or an event's length.
It adds no schema field, so nothing about the feed contract or the event ID changes.
### Migration from `completions`
`completions` used membership to mean "done", which cannot express "started". `progress` replaces it
@@ -254,13 +285,19 @@ old key.
"genshin:windblume-festival:2026-03-14": { "status": "done", "at": "..." },
"hsr:garden-of-plenty:2026-08-14": { "status": "doing", "effort": "grind", "at": "..." }
},
"daily": {
"endfield:bedazzling-dawnstar:2026-08-12": { "days": ["2026-08-13", "2026-08-14"], "at": "..." },
"dailies:genshin": { "days": ["2026-08-14", "2026-08-15"], "at": "..." }
},
"ignored": { "zzz:some-event-i-skip:2026-08-19": { "at": "..." } },
"prefs": { "region": "europe", "hiddenGames": [], "onboarded": true }
"prefs": { "region": "europe", "hiddenGames": [], "sort": "ending", "onboarded": true }
}
```
Import **merges** both sets: a mark present in either the file or the current device survives, and
import never removes one. Losing a user's marks to a bad import is unrecoverable, so the merge is
Import **merges** every set: a mark present in either the file or the current device survives, and
import never removes one. `daily` merges as a **union of days per ID** — every day either side
recorded is a day the reader actually played. An export written before daily checklists existed
simply has no `daily` key, which is not an error. Losing a user's marks to a bad import is unrecoverable, so the merge is
deliberately one-directional. A file whose `format` is unrecognised is refused outright rather than
partly applied.
docs/INGESTION.md
+42 -4
View File
@@ -136,13 +136,24 @@ two real events were in a table further down.
- Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
the run as `skipped_unchanged`.
- `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
- Honor `robots.txt`; cache parsed robots per host for 24h.
- 20s timeout; retry twice with backoff on 5xx and network errors; never retry 4xx.
- Store raw bytes in `snapshots`.
- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
5xxs or times out means "do not fetch", because a permission we could not read is not a
permission we have. A 404 means no restrictions.
- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
one per source per cycle. A failed source waits for the next cycle instead.
- Store raw bytes in `snapshots/<source-id>.html`, with hash/ETag/Last-Modified alongside it.
On failure: increment `consecutive_failures`, leave published events untouched, end as `failed`. A
On failure: increment the failure streak, leave published events untouched, end as `failed`. A
source being down never mutates the feed.
**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
`.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by
injection, so the whole runner is tested offline against a fake fetch. A fetched body is *rejected*
— the previous snapshot survives — when it fails `canParse`, throws, or yields zero events; storing
an empty parse would make the feed build prefer it over the fixture and silently empty a game's
calendar. One source down is a warning and exit 0; every source failing is exit 1, so CI never
commits a cycle that learned nothing.
## Stage 2 — parse
Hash the raw body (sha256) → `content_hash`. **If it matches `sources.content_hash`, end as
@@ -160,6 +171,33 @@ no error — the parser simply matches nothing. Compare each run's `events_seen`
run and flag a large drop. A source that went from 13 events to 2 has broken, not quieted down.
This is the most likely real failure mode of a parser-only pipeline, and nothing else surfaces it.
### Stage 2.5 — sanitize
Everything a parser returns came from a page we do not control, and it is about to become React
text, JSON on disk, a `localStorage` key and eventually a SQLite row. `src/ingest/sanitize.ts` is
the trust boundary, applied in `toAdapter()`'s `parse` wrapper in `src/ingest/adapters/index.ts` —
the one seam every source passes through, so a source added tomorrow is sanitized without its
author doing anything and no parser can opt out. It runs after `canParse` and before validation.
What it does: removes script/style/comment content and residual tags; decodes entities **to a fixed
point** so `&amp;lt;script&amp;gt;` cannot resurrect as markup in a later decoder; NFKC-normalizes;
strips control, zero-width and bidi-override characters (an RTL override visually spoofs a title);
collapses whitespace; truncates to the schema's own caps at a word boundary; and requires
`sourceUrl` to be absolute http(s), falling back to the source's registered URL rather than
dropping the event.
Three constraints it holds:
- **It never touches a date.** Not a timestamp, not a precision, not `regionEnds`. Dates are the
product's promise and the sanitizer's job stops at prose and URLs.
- **It cleans rather than drops.** The only drop is a title that sanitizes to nothing, and every
repair and drop emits a note whose default sink is `console.warn` — silence is not something a
future caller gets for free (§ Silent drops).
- **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the
title *and* the incoming ID was minted the standard way. All seven fixtures pass through with
zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys
and moving one orphans a reader's marks with no server-side recovery.
## Stage 3 — merge
Only meaningful when a game has more than one source; a single-source game passes straight through.