From 02863ed008a2f9eb1f3bd64537f451b04e3129b1 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Sat, 15 Aug 2026 21:19:08 +0200 Subject: [PATCH] docs: cover the refresh pipeline, sanitisation and dailies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The status sections claimed the feed was generated from fixtures and the scheduler unbuilt, which stopped being true. Also documents the sanitisation stage and the two new key spaces — `dailies:` and game-day keys — beside the existing warning about event IDs, since they carry the same "no server-side recovery" property. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 86 ++++++++++++++++++++++++++++++------ README.md | 108 ++++++++++++++++++++++++++++++++++++--------- docs/DATA-MODEL.md | 45 +++++++++++++++++-- docs/INGESTION.md | 46 +++++++++++++++++-- 4 files changed, 243 insertions(+), 42 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f75ba1f..c9ad50c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,12 +5,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## What this is A web app that aggregates live and upcoming events across popular gacha games, plots them on a -calendar, sorts them by end date, and lets a user mark events completed. +calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day +progress on events that repeat daily, and lets a user mark events completed. -**Status: working app, no live refresh.** Schema, two parsers, seven sources across six games, the -full interface, offline support, a static server, a Docker image and CI all exist and are tested. -The SQLite layer, the refresh scheduler and the review queue are specified in `docs/` but not built, -so the feed is generated offline from checked-in fixtures. +**Status: working app, refreshing itself on a schedule.** Schema, two parsers, seven sources across +six games, the full interface, offline support, a static server, a Docker image and CI all exist and +are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the +feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually +changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the +feed is a static JSON file built from snapshots, falling back to checked-in fixtures. ## Three constraints that shape everything @@ -45,6 +48,11 @@ bun run typecheck # tsc --noEmit bun run dev # build then serve on :3000 bun run build # feed + css + js + static into public/ +# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping +# conduct before running it, and prefer --dry-run. +bun run refresh --dry-run +bun run refresh --only genshin-game8-events + # Run one source against its fixture (offline, free) bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json @@ -67,18 +75,21 @@ re-verify a sample against the live page afterward. ## Current state of the code ``` -src/shared/ schema.ts (the contract), time.ts, games.ts, feed.ts -src/ingest/ html.ts, dates.ts (six formats), merge.ts +src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts +src/ingest/ html.ts, dates.ts (six formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts parsers/ game8.ts, wikigg.ts — keyed by SITE, not game - adapters/ index.ts — SOURCES registry binding url+game+parser + adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam src/client/ React app, service worker, manifest -scripts/ build-feed.ts, parse-fixture.ts (both offline) + state/ progress, daily log, ignores, prefs, sort — all localStorage +scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 112 tests -fixtures// raw HTML + .expected.json per source +test/ 257 tests +fixtures// raw HTML + .expected.json per source — pinned, kept forever +snapshots/ current page per source, rewritten by refresh — see its README ``` -Not yet built: the SQLite layer, the ingest scheduler, the review UI. +Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on +disk. ## Domain rules that are not obvious from the code @@ -132,6 +143,18 @@ recovery**, because the server never had the data. If it must change, ship a cli that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such change. +Two more key spaces have the same property, for the same reason: + +- **`dailies:`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore. + Two segments, so it cannot collide with an event ID. +- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at + 04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the + format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day. + +The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually +changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real +fixtures, that is a data-loss bug, not a diff to regenerate. + ## Scraping conduct Sources are community wikis. Treat them as a guest would: @@ -149,6 +172,35 @@ rate, and do not add an LLM that consumes page content. A source whose ToS forbids automated access does not get an adapter. Flag it and ask. +`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no +retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything +that would make it fetch more often is a change to this section first. + +## Untrusted input + +Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust +boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single +seam every source passes through — **do not sanitize inside a parser**, and do not add a code path +that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup. + +The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is +the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5. + +## Events that repeat daily + +Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed +day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published — +`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an +event's length. It adds **no schema field**, so the feed contract is untouched. + +- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks + the wrong box for four hours every night. +- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null` + rule applies here exactly as it does to a countdown. +- **A tick is never removed except by the reader**, including ticks outside the window the feed now + claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere + else. + ## Conventions - **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never @@ -156,4 +208,12 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and - Every adapter ships a fixture in `fixtures//` and a test asserting parsed output. This is how a source silently changing shape gets caught. - Keep old fixtures when a source changes shape — the old one is the regression test proving the - parser still handles the previous format. + parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the + current page and gets overwritten. Do not conflate them. +- **A list row is one target.** The event row opens the event and does nothing else — status, + effort, notes and the daily checklist all live in the detail sheet. A second control inside a + full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens" + without adding a second stop for keyboard and screen-reader users. +- **Sorting groups, it never reorders within a group.** Every mode falls back to + `endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product + exists for. diff --git a/README.md b/README.md index 3d7b230..4c21238 100644 --- a/README.md +++ b/README.md @@ -6,27 +6,33 @@ You play three or four gacha games. Each has its own calendar, none of them talk the only question that actually matters — *what runs out first?* — takes four browser tabs to answer. This does it in one screen. -No account. No login. What you've finished, what you're partway through, and how much work you -reckon each event is are saved in your browser and never leave your device. +No account. No login. What you've finished, what you're partway through, how much work you reckon +each event is, and which day of a daily you've ticked off are saved in your browser and never leave +your device. ## Status -Early, but usable. The parsing pipeline and the interface work end to end against checked-in -fixtures, and the whole thing builds, serves, containerises and deploys. The database, refresh -scheduler and review queue are specified but not built, so the feed is generated offline rather than -refreshing itself. +Usable, and it now keeps itself up to date. The parsing pipeline and the interface work end to end, +a scheduled job refreshes the sources twice a day, and the whole thing builds, serves, containerises +and deploys. The database and review queue are specified but not built. | Piece | State | |---|---| | Event schema, date parsing, Game8 parser | Built, tested | | Six game sources | Built, tested | | Cross-source merge and conflict detection | Built, tested | -| Web interface, offline support, first-run picker | Built | +| Input sanitization at the ingest boundary | Built, tested | +| Scheduled refresh — robots, snapshots, commit-on-change | Built, tested offline | +| Web interface, daily checklists, offline support | Built | | Static server, Docker image, GitHub + GitLab CI | Built | -| SQLite, refresh scheduler, review queue | Specified in `docs/`, not built | +| SQLite, review queue | Specified in `docs/`, not built | -Today the feed is generated offline from fixtures. That is deliberate: it let the interface be built -against real parsed data, and it produces exactly the shape the server will serve. +The feed is still a static JSON file rather than a database read: the refresh job commits the raw +pages it fetched, CI rebuilds the feed from them, and a clean checkout with no snapshots falls back +to the checked-in fixtures — so the build stays offline and reproducible either way. + +The refresh runner is tested against a fake fetch, never a live wiki. Its first real run against +game8.co and wiki.gg is unproven. ## Try it @@ -53,7 +59,11 @@ reproducible and a wiki being down never breaks it. bun test # full suite, offline, no network bun run typecheck # tsc --noEmit bun run build # feed + css + js + html into public/ -bun run build:feed # regenerate public/data/events.v1.json from fixtures +bun run build:feed # regenerate public/data/events.v1.json from snapshots, else fixtures + +# Fetch the sources. Makes real requests, so read "Conduct" first. +bun run refresh --dry-run # plan only: no requests, no writes +bun run refresh --only genshin-game8-events # Run one source against its fixture and print what it yields bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html @@ -63,11 +73,12 @@ bun run parse nte-game8-events fixtures/nte/game8-events-2026-08-14.html --j ## How it works ``` -game wikis ──► fetch ──► parse ──► merge ──► validate ──► gate ──► feed ──► browser - │ │ │ │ - per-site per-game hold anything localStorage: - parser corroboration uncertain for what you've - and conflicts human review finished +game wikis ─► fetch ─► parse ─► sanitize ─► merge ─► validate ─► gate ─► feed ─► browser + │ │ │ │ │ │ + robots, 6h, per-site untrusted per-game hold anything localStorage: + conditional, parser text corroboration uncertain for what you've done, + snapshots bounded and conflicts human review day by day if it + repeats daily ``` **Parsers are deterministic code.** There is no LLM anywhere in the pipeline — no API key, no @@ -126,6 +137,39 @@ Arknights is defined in the schema and awaiting a source. Full walkthrough in `docs/INGESTION.md`, or run the `add-game-source` skill. +## Dailies + +Some things are not one job with a deadline. A login campaign is twenty small jobs on twenty +separate deadlines, and a day you miss is gone whatever you do afterwards — which a single "done" +tick cannot express. + +So events that repeat get a checklist instead: today's tick, a strip of every day in the run showing +what you got and what you missed, your streak, and how many chances are left. Past days stay +editable, because people tick up later and a checklist you can't correct stops being trusted after +the first mistake. + +Alongside them sits **today's dailies** — commissions, sanity, daily training — one tick per game. +No wiki publishes those, so they are a fixed list in the app rather than scraped data, and they are +the only thing on the page that expires tonight rather than next patch. + +Both roll over at **04:00 server time** in your region, not midnight, because that is when the games +roll over. Finishing at 02:00 still counts as yesterday. + +Repeating events are recognised from what the source actually printed — a login event type, or +wording like "daily", "check-in", "7-day". Nothing is assumed from a game's habits, and an event +whose end was never announced gets a day count rather than a checklist of invented length. + +## Sorting + +Two orders, and the toggle sits with the list rather than in settings: + +- **Ending soonest** — the default, and the reason this app exists. +- **Doing first** — what you're partway through, floated to the top. Ticking a daily counts as + "doing it" without your having to say so twice. + +Sorting only ever *groups*. Deadline order survives inside every group, so choosing an order can +never cost you the one thing you came for. + ## Offline The app works with no network. A service worker caches the shell and webfonts, and serves the last @@ -137,8 +181,13 @@ It installs to a home screen as a standalone app. ## Conduct Sources are community wikis, treated as a guest would: `robots.txt` honoured, a descriptive -`User-Agent`, one request per source per six hours, conditional requests, and raw snapshots cached -so iteration never re-fetches. Every event links back to its source. +`User-Agent` with a contact URL, one request per source per six hours, conditional requests, and raw +snapshots cached so iteration never re-fetches. Every event links back to its source. + +`bun run refresh` enforces all of that in code rather than leaving it to good intentions: the +six-hour floor is checked per source, there are no retries (a retry is a second request), and a +`robots.txt` that cannot be read means *do not fetch* rather than *assume yes*. Text scraped from a +page is sanitized at the ingest boundary before it reaches the feed, the browser or your disk. ## Documentation @@ -164,9 +213,26 @@ Until then the `pages` job is skipped and the pipeline stays green. Pages is una repositories on the free plan. The feed job fails if the event count collapses. A source that quietly stops yielding events is the -failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Everything -runs offline against checked-in fixtures, so a red pipeline always means the code changed rather -than a wiki being down. +failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Tests run +offline against checked-in fixtures, so a red pipeline always means the code changed rather than a +wiki being down. + +### Refreshing the data + +GitHub Actions only — the GitLab pipeline still runs the gates, but nothing there fetches. +`.github/workflows/refresh.yml` runs `bun run refresh` twice a day (and on demand, with a dry-run +input). It fetches each source at most once per cycle, and **commits only when a page's bytes +actually changed** — a `304`, an identical body, or a fetch that fails to parse all leave the +working tree clean and produce no commit. When something did change it commits the raw snapshots and +dispatches `ci.yml`, which typechecks, tests, rebuilds the feed and deploys through the path that +already existed; none of that logic is duplicated. + +A body that yields zero events is rejected and the previous snapshot kept, so a wiki redesign shows +up as a stale timestamp rather than an empty calendar. One source being down is a warning; every +source being down fails the run, so a bad cycle never gets committed. + +The schedule is off for forks (it is pinned to this repository) — a fork owner can still dispatch it +by hand and take responsibility for the traffic. ### Hosting under a subpath diff --git a/docs/DATA-MODEL.md b/docs/DATA-MODEL.md index 8f654c5..90c7262 100644 --- a/docs/DATA-MODEL.md +++ b/docs/DATA-MODEL.md @@ -197,8 +197,9 @@ Namespaced, versioned, and small. Nothing here ever goes to the server. ```ts "gacha-tracker:v1:progress" // { [eventId]: { status?, effort?, note?, at } } +"gacha-tracker:v1:daily" // { [id]: { days: ["2026-08-15", ...], at } } "gacha-tracker:v1:ignored" // { [eventId]: { at } } — "stop showing me this" -"gacha-tracker:v1:prefs" // { region, hiddenGames[], showCompleted, showIgnored, +"gacha-tracker:v1:prefs" // { region, hiddenGames[], sort, showCompleted, showIgnored, // regionConfirmed, onboarded } "gacha-tracker:v1:completions" // SUPERSEDED — read once to migrate, never written ``` @@ -225,6 +226,36 @@ about it would be fabricating the reader's own input. Ignores stay in a separate store because they mean something different: a done event is dimmed and still counted, an ignored one disappears from both views. +### Daily checklists + +Some events are not one job with a deadline but twenty small jobs on twenty separate deadlines, and +a missed day is gone whatever you do afterwards. `daily` records which game-days the reader ticked +off, keyed by: + +- an **event ID**, for a repeating event in the feed (`src/shared/daily.ts` § `isDaily`), or +- **`dailies:`**, the standing per-game chore — commissions, sanity, daily training. No source + publishes these, so they are a fixed client-side list, never feed data. The two-segment shape + cannot collide with an event ID, which is always `game:slug:date`. + +Day keys are `YYYY-MM-DD` in **game-day space, not UTC**: gacha servers roll the day at 04:00 local +server time, so a player finishing at 02:00 is still on the previous day's dailies, and the key is +computed against the reader's chosen region (`RESET_HOUR_LOCAL`, `dayKey`). Keys sort +lexicographically, which is what "how many days are left" and streak counting rely on. + +Three rules this store keeps, for the same reason the rest of the client does — nothing else holds +a copy: + +- **A tick is never removed except by the reader.** Ticks that fall outside the window the feed now + claims still count; a source quietly moving a date must not erase a fortnight's streak. +- **An unannounced end yields no checklist**, not a checklist of guessed length. `dailyDays` returns + null when `endsAt` is null, and the UI says how many days are ticked instead of how many are left. +- **Past days stay editable.** People log in and tick up later, and a checklist that cannot be + corrected stops being trusted after the first mistake. + +Dailiness is derived from the published event — `type: "login"`, or wording like "daily", +"check-in", "7-day" in the title or summary — and never from a game's habits or an event's length. +It adds no schema field, so nothing about the feed contract or the event ID changes. + ### Migration from `completions` `completions` used membership to mean "done", which cannot express "started". `progress` replaces it @@ -254,13 +285,19 @@ old key. "genshin:windblume-festival:2026-03-14": { "status": "done", "at": "..." }, "hsr:garden-of-plenty:2026-08-14": { "status": "doing", "effort": "grind", "at": "..." } }, + "daily": { + "endfield:bedazzling-dawnstar:2026-08-12": { "days": ["2026-08-13", "2026-08-14"], "at": "..." }, + "dailies:genshin": { "days": ["2026-08-14", "2026-08-15"], "at": "..." } + }, "ignored": { "zzz:some-event-i-skip:2026-08-19": { "at": "..." } }, - "prefs": { "region": "europe", "hiddenGames": [], "onboarded": true } + "prefs": { "region": "europe", "hiddenGames": [], "sort": "ending", "onboarded": true } } ``` -Import **merges** both sets: a mark present in either the file or the current device survives, and -import never removes one. Losing a user's marks to a bad import is unrecoverable, so the merge is +Import **merges** every set: a mark present in either the file or the current device survives, and +import never removes one. `daily` merges as a **union of days per ID** — every day either side +recorded is a day the reader actually played. An export written before daily checklists existed +simply has no `daily` key, which is not an error. Losing a user's marks to a bad import is unrecoverable, so the merge is deliberately one-directional. A file whose `format` is unrecognised is refused outright rather than partly applied. diff --git a/docs/INGESTION.md b/docs/INGESTION.md index ad40f63..0403a80 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -136,13 +136,24 @@ two real events were in a table further down. - Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends the run as `skipped_unchanged`. - `User-Agent: gacha-event-tracker/1.0 (+https://github.com//gacha-event-tracker)`. -- Honor `robots.txt`; cache parsed robots per host for 24h. -- 20s timeout; retry twice with backoff on 5xx and network errors; never retry 4xx. -- Store raw bytes in `snapshots`. +- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that + 5xxs or times out means "do not fetch", because a permission we could not read is not a + permission we have. A 404 means no restrictions. +- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says + one per source per cycle. A failed source waits for the next cycle instead. +- Store raw bytes in `snapshots/.html`, with hash/ETag/Last-Modified alongside it. -On failure: increment `consecutive_failures`, leave published events untouched, end as `failed`. A +On failure: increment the failure streak, leave published events untouched, end as `failed`. A source being down never mutates the feed. +**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by +`.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by +injection, so the whole runner is tested offline against a fake fetch. A fetched body is *rejected* +— the previous snapshot survives — when it fails `canParse`, throws, or yields zero events; storing +an empty parse would make the feed build prefer it over the fixture and silently empty a game's +calendar. One source down is a warning and exit 0; every source failing is exit 1, so CI never +commits a cycle that learned nothing. + ## Stage 2 — parse Hash the raw body (sha256) → `content_hash`. **If it matches `sources.content_hash`, end as @@ -160,6 +171,33 @@ no error — the parser simply matches nothing. Compare each run's `events_seen` run and flag a large drop. A source that went from 13 events to 2 has broken, not quieted down. This is the most likely real failure mode of a parser-only pipeline, and nothing else surfaces it. +### Stage 2.5 — sanitize + +Everything a parser returns came from a page we do not control, and it is about to become React +text, JSON on disk, a `localStorage` key and eventually a SQLite row. `src/ingest/sanitize.ts` is +the trust boundary, applied in `toAdapter()`'s `parse` wrapper in `src/ingest/adapters/index.ts` — +the one seam every source passes through, so a source added tomorrow is sanitized without its +author doing anything and no parser can opt out. It runs after `canParse` and before validation. + +What it does: removes script/style/comment content and residual tags; decodes entities **to a fixed +point** so `&lt;script&gt;` cannot resurrect as markup in a later decoder; NFKC-normalizes; +strips control, zero-width and bidi-override characters (an RTL override visually spoofs a title); +collapses whitespace; truncates to the schema's own caps at a word boundary; and requires +`sourceUrl` to be absolute http(s), falling back to the source's registered URL rather than +dropping the event. + +Three constraints it holds: + +- **It never touches a date.** Not a timestamp, not a precision, not `regionEnds`. Dates are the + product's promise and the sanitizer's job stops at prose and URLs. +- **It cleans rather than drops.** The only drop is a title that sanitizes to nothing, and every + repair and drop emits a note whose default sink is `console.warn` — silence is not something a + future caller gets for free (§ Silent drops). +- **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the + title *and* the incoming ID was minted the standard way. All seven fixtures pass through with + zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys + and moving one orphans a reader's marks with no server-side recovery. + ## Stage 3 — merge Only meaningful when a game has more than one source; a single-source game passes straight through.