docs: cover the refresh pipeline, sanitisation and dailies

The status sections claimed the feed was generated from fixtures and the
scheduler unbuilt, which stopped being true. Also documents the
sanitisation stage and the two new key spaces — `dailies:<game>` and
game-day keys — beside the existing warning about event IDs, since they
carry the same "no server-side recovery" property.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

CLAUDE.md

@@ -5,12 +5,15 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## What this is
 
 A web app that aggregates live and upcoming events across popular gacha games, plots them on a
-calendar, sorts them by end date, and lets a user mark events completed.
+calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
+progress on events that repeat daily, and lets a user mark events completed.
 
-**Status: working app, no live refresh.** Schema, two parsers, seven sources across six games, the
+**Status: working app, refreshing itself on a schedule.** Schema, two parsers, seven sources across
-full interface, offline support, a static server, a Docker image and CI all exist and are tested.
+six games, the full interface, offline support, a static server, a Docker image and CI all exist and
-The SQLite layer, the refresh scheduler and the review queue are specified in `docs/` but not built,
+are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
-so the feed is generated offline from checked-in fixtures.
+feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
+changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
+feed is a static JSON file built from snapshots, falling back to checked-in fixtures.
 
 ## Three constraints that shape everything
 
@@ -45,6 +48,11 @@ bun run typecheck             # tsc --noEmit
 bun run dev                   # build then serve on :3000
 bun run build                 # feed + css + js + static into public/
 
+# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping
+# conduct before running it, and prefer --dry-run.
+bun run refresh --dry-run
+bun run refresh --only genshin-game8-events
+
 # Run one source against its fixture (offline, free)
 bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
 bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
@@ -67,18 +75,21 @@ re-verify a sample against the live page afterward.
 ## Current state of the code
 
 ```
-src/shared/       schema.ts (the contract), time.ts, games.ts, feed.ts
+src/shared/       schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
-src/ingest/       html.ts, dates.ts (six formats), merge.ts
+src/ingest/       html.ts, dates.ts (six formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
   parsers/        game8.ts, wikigg.ts — keyed by SITE, not game
-  adapters/       index.ts — SOURCES registry binding url+game+parser
+  adapters/       index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
 src/client/       React app, service worker, manifest
-scripts/          build-feed.ts, parse-fixture.ts (both offline)
+  state/          progress, daily log, ignores, prefs, sort — all localStorage
+scripts/          build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
 serve.ts          static server + /api/health
-test/             112 tests
+test/             257 tests
-fixtures/<game>/  raw HTML + .expected.json per source
+fixtures/<game>/  raw HTML + .expected.json per source — pinned, kept forever
+snapshots/        current page per source, rewritten by refresh — see its README
 ```
 
-Not yet built: the SQLite layer, the ingest scheduler, the review UI.
+Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on
+disk.
 
 ## Domain rules that are not obvious from the code
 
@@ -132,6 +143,18 @@ recovery**, because the server never had the data. If it must change, ship a cli
 that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such
 change.
 
+Two more key spaces have the same property, for the same reason:
+
+- **`dailies:<game>`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore.
+  Two segments, so it cannot collide with an event ID.
+- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at
+  04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the
+  format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day.
+
+The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
+changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
+fixtures, that is a data-loss bug, not a diff to regenerate.
+
 ## Scraping conduct
 
 Sources are community wikis. Treat them as a guest would:
@@ -149,6 +172,35 @@ rate, and do not add an LLM that consumes page content.
 
 A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
 
+`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
+retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything
+that would make it fetch more often is a change to this section first.
+
+## Untrusted input
+
+Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust
+boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single
+seam every source passes through — **do not sanitize inside a parser**, and do not add a code path
+that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup.
+
+The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is
+the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5.
+
+## Events that repeat daily
+
+Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed
+day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published —
+`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an
+event's length. It adds **no schema field**, so the feed contract is untouched.
+
+- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks
+  the wrong box for four hours every night.
+- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null`
+  rule applies here exactly as it does to a countdown.
+- **A tick is never removed except by the reader**, including ticks outside the window the feed now
+  claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere
+  else.
+
 ## Conventions
 
 - **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never
@@ -156,4 +208,12 @@ A source whose ToS forbids automated access does not get an adapter. Flag it and
 - Every adapter ships a fixture in `fixtures/<game>/` and a test asserting parsed output. This is
   how a source silently changing shape gets caught.
 - Keep old fixtures when a source changes shape — the old one is the regression test proving the
-  parser still handles the previous format.
+  parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the
+  current page and gets overwritten. Do not conflate them.
+- **A list row is one target.** The event row opens the event and does nothing else — status,
+  effort, notes and the daily checklist all live in the detail sheet. A second control inside a
+  full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens"
+  without adding a second stop for keyboard and screen-reader users.
+- **Sorting groups, it never reorders within a group.** Every mode falls back to
+  `endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product
+  exists for.

README.md

@@ -6,27 +6,33 @@ You play three or four gacha games. Each has its own calendar, none of them talk
 the only question that actually matters — *what runs out first?* — takes four browser tabs to
 answer. This does it in one screen.
 
-No account. No login. What you've finished, what you're partway through, and how much work you
+No account. No login. What you've finished, what you're partway through, how much work you reckon
-reckon each event is are saved in your browser and never leave your device.
+each event is, and which day of a daily you've ticked off are saved in your browser and never leave
+your device.
 
 ## Status
 
-Early, but usable. The parsing pipeline and the interface work end to end against checked-in
+Usable, and it now keeps itself up to date. The parsing pipeline and the interface work end to end,
-fixtures, and the whole thing builds, serves, containerises and deploys. The database, refresh
+a scheduled job refreshes the sources twice a day, and the whole thing builds, serves, containerises
-scheduler and review queue are specified but not built, so the feed is generated offline rather than
+and deploys. The database and review queue are specified but not built.
-refreshing itself.
 
 | Piece | State |
 |---|---|
 | Event schema, date parsing, Game8 parser | Built, tested |
 | Six game sources | Built, tested |
 | Cross-source merge and conflict detection | Built, tested |
-| Web interface, offline support, first-run picker | Built |
+| Input sanitization at the ingest boundary | Built, tested |
+| Scheduled refresh — robots, snapshots, commit-on-change | Built, tested offline |
+| Web interface, daily checklists, offline support | Built |
 | Static server, Docker image, GitHub + GitLab CI | Built |
-| SQLite, refresh scheduler, review queue | Specified in `docs/`, not built |
+| SQLite, review queue | Specified in `docs/`, not built |
 
-Today the feed is generated offline from fixtures. That is deliberate: it let the interface be built
+The feed is still a static JSON file rather than a database read: the refresh job commits the raw
-against real parsed data, and it produces exactly the shape the server will serve.
+pages it fetched, CI rebuilds the feed from them, and a clean checkout with no snapshots falls back
+to the checked-in fixtures — so the build stays offline and reproducible either way.
+
+The refresh runner is tested against a fake fetch, never a live wiki. Its first real run against
+game8.co and wiki.gg is unproven.
 
 ## Try it
 
@@ -53,7 +59,11 @@ reproducible and a wiki being down never breaks it.
 bun test                  # full suite, offline, no network
 bun run typecheck         # tsc --noEmit
 bun run build             # feed + css + js + html into public/
-bun run build:feed        # regenerate public/data/events.v1.json from fixtures
+bun run build:feed        # regenerate public/data/events.v1.json from snapshots, else fixtures
+
+# Fetch the sources. Makes real requests, so read "Conduct" first.
+bun run refresh --dry-run             # plan only: no requests, no writes
+bun run refresh --only genshin-game8-events
 
 # Run one source against its fixture and print what it yields
 bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
@@ -63,11 +73,12 @@ bun run parse nte-game8-events     fixtures/nte/game8-events-2026-08-14.html --j
 ## How it works
 
 ```
-game wikis ──► fetch ──► parse ──► merge ──► validate ──► gate ──► feed ──► browser
+game wikis ─► fetch ─► parse ─► sanitize ─► merge ─► validate ─► gate ─► feed ─► browser
-                          │          │                     │                  │
+               │         │         │          │                   │                 │
-                    per-site      per-game            hold anything    localStorage:
+          robots, 6h, per-site  untrusted  per-game        hold anything    localStorage:
-                     parser      corroboration        uncertain for      what you've
+          conditional,  parser    text     corroboration   uncertain for   what you've done,
-                                 and conflicts        human review        finished
+           snapshots            bounded    and conflicts   human review    day by day if it
+                                                                            repeats daily
 ```
 
 **Parsers are deterministic code.** There is no LLM anywhere in the pipeline — no API key, no
@@ -126,6 +137,39 @@ Arknights is defined in the schema and awaiting a source.
 
 Full walkthrough in `docs/INGESTION.md`, or run the `add-game-source` skill.
 
+## Dailies
+
+Some things are not one job with a deadline. A login campaign is twenty small jobs on twenty
+separate deadlines, and a day you miss is gone whatever you do afterwards — which a single "done"
+tick cannot express.
+
+So events that repeat get a checklist instead: today's tick, a strip of every day in the run showing
+what you got and what you missed, your streak, and how many chances are left. Past days stay
+editable, because people tick up later and a checklist you can't correct stops being trusted after
+the first mistake.
+
+Alongside them sits **today's dailies** — commissions, sanity, daily training — one tick per game.
+No wiki publishes those, so they are a fixed list in the app rather than scraped data, and they are
+the only thing on the page that expires tonight rather than next patch.
+
+Both roll over at **04:00 server time** in your region, not midnight, because that is when the games
+roll over. Finishing at 02:00 still counts as yesterday.
+
+Repeating events are recognised from what the source actually printed — a login event type, or
+wording like "daily", "check-in", "7-day". Nothing is assumed from a game's habits, and an event
+whose end was never announced gets a day count rather than a checklist of invented length.
+
+## Sorting
+
+Two orders, and the toggle sits with the list rather than in settings:
+
+- **Ending soonest** — the default, and the reason this app exists.
+- **Doing first** — what you're partway through, floated to the top. Ticking a daily counts as
+  "doing it" without your having to say so twice.
+
+Sorting only ever *groups*. Deadline order survives inside every group, so choosing an order can
+never cost you the one thing you came for.
+
 ## Offline
 
 The app works with no network. A service worker caches the shell and webfonts, and serves the last
@@ -137,8 +181,13 @@ It installs to a home screen as a standalone app.
 ## Conduct
 
 Sources are community wikis, treated as a guest would: `robots.txt` honoured, a descriptive
-`User-Agent`, one request per source per six hours, conditional requests, and raw snapshots cached
+`User-Agent` with a contact URL, one request per source per six hours, conditional requests, and raw
-so iteration never re-fetches. Every event links back to its source.
+snapshots cached so iteration never re-fetches. Every event links back to its source.
+
+`bun run refresh` enforces all of that in code rather than leaving it to good intentions: the
+six-hour floor is checked per source, there are no retries (a retry is a second request), and a
+`robots.txt` that cannot be read means *do not fetch* rather than *assume yes*. Text scraped from a
+page is sanitized at the ingest boundary before it reaches the feed, the browser or your disk.
 
 ## Documentation
 
@@ -164,9 +213,26 @@ Until then the `pages` job is skipped and the pipeline stays green. Pages is una
 repositories on the free plan.
 
 The feed job fails if the event count collapses. A source that quietly stops yielding events is the
-failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Everything
+failure mode a parser-only pipeline is most prone to, and nothing else would surface it. Tests run
-runs offline against checked-in fixtures, so a red pipeline always means the code changed rather
+offline against checked-in fixtures, so a red pipeline always means the code changed rather than a
-than a wiki being down.
+wiki being down.
+
+### Refreshing the data
+
+GitHub Actions only — the GitLab pipeline still runs the gates, but nothing there fetches.
+`.github/workflows/refresh.yml` runs `bun run refresh` twice a day (and on demand, with a dry-run
+input). It fetches each source at most once per cycle, and **commits only when a page's bytes
+actually changed** — a `304`, an identical body, or a fetch that fails to parse all leave the
+working tree clean and produce no commit. When something did change it commits the raw snapshots and
+dispatches `ci.yml`, which typechecks, tests, rebuilds the feed and deploys through the path that
+already existed; none of that logic is duplicated.
+
+A body that yields zero events is rejected and the previous snapshot kept, so a wiki redesign shows
+up as a stale timestamp rather than an empty calendar. One source being down is a warning; every
+source being down fails the run, so a bad cycle never gets committed.
+
+The schedule is off for forks (it is pinned to this repository) — a fork owner can still dispatch it
+by hand and take responsibility for the traffic.
 
 ### Hosting under a subpath
 

docs/DATA-MODEL.md

@@ -197,8 +197,9 @@ Namespaced, versioned, and small. Nothing here ever goes to the server.
 
 ```ts
 "gacha-tracker:v1:progress"     // { [eventId]: { status?, effort?, note?, at } }
+"gacha-tracker:v1:daily"        // { [id]: { days: ["2026-08-15", ...], at } }
 "gacha-tracker:v1:ignored"      // { [eventId]: { at } }  — "stop showing me this"
-"gacha-tracker:v1:prefs"        // { region, hiddenGames[], showCompleted, showIgnored,
+"gacha-tracker:v1:prefs"        // { region, hiddenGames[], sort, showCompleted, showIgnored,
                                 //   regionConfirmed, onboarded }
 "gacha-tracker:v1:completions"  // SUPERSEDED — read once to migrate, never written
 ```
@@ -225,6 +226,36 @@ about it would be fabricating the reader's own input.
 Ignores stay in a separate store because they mean something different: a done event is dimmed and
 still counted, an ignored one disappears from both views.
 
+### Daily checklists
+
+Some events are not one job with a deadline but twenty small jobs on twenty separate deadlines, and
+a missed day is gone whatever you do afterwards. `daily` records which game-days the reader ticked
+off, keyed by:
+
+- an **event ID**, for a repeating event in the feed (`src/shared/daily.ts` § `isDaily`), or
+- **`dailies:<game>`**, the standing per-game chore — commissions, sanity, daily training. No source
+  publishes these, so they are a fixed client-side list, never feed data. The two-segment shape
+  cannot collide with an event ID, which is always `game:slug:date`.
+
+Day keys are `YYYY-MM-DD` in **game-day space, not UTC**: gacha servers roll the day at 04:00 local
+server time, so a player finishing at 02:00 is still on the previous day's dailies, and the key is
+computed against the reader's chosen region (`RESET_HOUR_LOCAL`, `dayKey`). Keys sort
+lexicographically, which is what "how many days are left" and streak counting rely on.
+
+Three rules this store keeps, for the same reason the rest of the client does — nothing else holds
+a copy:
+
+- **A tick is never removed except by the reader.** Ticks that fall outside the window the feed now
+  claims still count; a source quietly moving a date must not erase a fortnight's streak.
+- **An unannounced end yields no checklist**, not a checklist of guessed length. `dailyDays` returns
+  null when `endsAt` is null, and the UI says how many days are ticked instead of how many are left.
+- **Past days stay editable.** People log in and tick up later, and a checklist that cannot be
+  corrected stops being trusted after the first mistake.
+
+Dailiness is derived from the published event — `type: "login"`, or wording like "daily",
+"check-in", "7-day" in the title or summary — and never from a game's habits or an event's length.
+It adds no schema field, so nothing about the feed contract or the event ID changes.
+
 ### Migration from `completions`
 
 `completions` used membership to mean "done", which cannot express "started". `progress` replaces it
@@ -254,13 +285,19 @@ old key.
     "genshin:windblume-festival:2026-03-14": { "status": "done", "at": "..." },
     "hsr:garden-of-plenty:2026-08-14": { "status": "doing", "effort": "grind", "at": "..." }
   },
+  "daily": {
+    "endfield:bedazzling-dawnstar:2026-08-12": { "days": ["2026-08-13", "2026-08-14"], "at": "..." },
+    "dailies:genshin": { "days": ["2026-08-14", "2026-08-15"], "at": "..." }
+  },
   "ignored": { "zzz:some-event-i-skip:2026-08-19": { "at": "..." } },
-  "prefs": { "region": "europe", "hiddenGames": [], "onboarded": true }
+  "prefs": { "region": "europe", "hiddenGames": [], "sort": "ending", "onboarded": true }
 }
 ```
 
-Import **merges** both sets: a mark present in either the file or the current device survives, and
+Import **merges** every set: a mark present in either the file or the current device survives, and
-import never removes one. Losing a user's marks to a bad import is unrecoverable, so the merge is
+import never removes one. `daily` merges as a **union of days per ID** — every day either side
+recorded is a day the reader actually played. An export written before daily checklists existed
+simply has no `daily` key, which is not an error. Losing a user's marks to a bad import is unrecoverable, so the merge is
 deliberately one-directional. A file whose `format` is unrecognised is refused outright rather than
 partly applied.
 

docs/INGESTION.md

@@ -136,13 +136,24 @@ two real events were in a table further down.
 - Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends
   the run as `skipped_unchanged`.
 - `User-Agent: gacha-event-tracker/1.0 (+https://github.com/<owner>/gacha-event-tracker)`.
-- Honor `robots.txt`; cache parsed robots per host for 24h.
+- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
-- 20s timeout; retry twice with backoff on 5xx and network errors; never retry 4xx.
+  5xxs or times out means "do not fetch", because a permission we could not read is not a
-- Store raw bytes in `snapshots`.
+  permission we have. A 404 means no restrictions.
+- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
+  one per source per cycle. A failed source waits for the next cycle instead.
+- Store raw bytes in `snapshots/<source-id>.html`, with hash/ETag/Last-Modified alongside it.
 
-On failure: increment `consecutive_failures`, leave published events untouched, end as `failed`. A
+On failure: increment the failure streak, leave published events untouched, end as `failed`. A
 source being down never mutates the feed.
 
+**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
+`.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by
+injection, so the whole runner is tested offline against a fake fetch. A fetched body is *rejected*
+— the previous snapshot survives — when it fails `canParse`, throws, or yields zero events; storing
+an empty parse would make the feed build prefer it over the fixture and silently empty a game's
+calendar. One source down is a warning and exit 0; every source failing is exit 1, so CI never
+commits a cycle that learned nothing.
+
 ## Stage 2 — parse
 
 Hash the raw body (sha256) → `content_hash`. **If it matches `sources.content_hash`, end as
@@ -160,6 +171,33 @@ no error — the parser simply matches nothing. Compare each run's `events_seen`
 run and flag a large drop. A source that went from 13 events to 2 has broken, not quieted down.
 This is the most likely real failure mode of a parser-only pipeline, and nothing else surfaces it.
 
+### Stage 2.5 — sanitize
+
+Everything a parser returns came from a page we do not control, and it is about to become React
+text, JSON on disk, a `localStorage` key and eventually a SQLite row. `src/ingest/sanitize.ts` is
+the trust boundary, applied in `toAdapter()`'s `parse` wrapper in `src/ingest/adapters/index.ts` —
+the one seam every source passes through, so a source added tomorrow is sanitized without its
+author doing anything and no parser can opt out. It runs after `canParse` and before validation.
+
+What it does: removes script/style/comment content and residual tags; decodes entities **to a fixed
+point** so `&amp;lt;script&amp;gt;` cannot resurrect as markup in a later decoder; NFKC-normalizes;
+strips control, zero-width and bidi-override characters (an RTL override visually spoofs a title);
+collapses whitespace; truncates to the schema's own caps at a word boundary; and requires
+`sourceUrl` to be absolute http(s), falling back to the source's registered URL rather than
+dropping the event.
+
+Three constraints it holds:
+
+- **It never touches a date.** Not a timestamp, not a precision, not `regionEnds`. Dates are the
+  product's promise and the sanitizer's job stops at prose and URLs.
+- **It cleans rather than drops.** The only drop is a title that sanitizes to nothing, and every
+  repair and drop emits a note whose default sink is `console.warn` — silence is not something a
+  future caller gets for free (§ Silent drops).
+- **It does not move event IDs.** An ID is recomputed only when sanitizing actually changed the
+  title *and* the incoming ID was minted the standard way. All seven fixtures pass through with
+  zero repairs and byte-identical output, which is the regression guard: IDs are localStorage keys
+  and moving one orphans a reader's marks with no server-side recovery.
+
 ## Stage 3 — merge
 
 Only meaningful when a game has more than one source; a single-source game passes straight through.