docs: move the working agreement to AGENTS.md
The guidance in here is not Claude-specific — it is what this project is, what it refuses to do, and the rules that are invisible from the code. Any agent working here needs it, and AGENTS.md is the name they all look for. CLAUDE.md stays as a pointer so Claude Code still finds it, and holds no guidance of its own; two copies would drift and the wrong one would be read. Every cross-reference in the source comments, docs, workflow and skills now names AGENTS.md, and the image ignores both. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
c2740760bd
commit
025605aa09
@@ -44,7 +44,7 @@ source (wiki.gg) turned out to publish ISO timestamps with per-region timers.
|
|||||||
This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
|
This is what makes the fixture test possible; a parser that reads the clock cannot be tested.
|
||||||
- Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in
|
- Implement `canParse` as a *structural* check, and do not over-fit it. Game8's own pages differ in
|
||||||
attribute quote style, so `class="a-table"` would falsely reject half of them.
|
attribute quote style, so `class="a-table"` would falsely reject half of them.
|
||||||
- Get the domain rules right — they are in `CLAUDE.md` § Domain rules and they are where adapters
|
- Get the domain rules right — they are in `AGENTS.md` § Domain rules and they are where adapters
|
||||||
actually go wrong:
|
actually go wrong:
|
||||||
- All timestamps UTC ISO 8601.
|
- All timestamps UTC ISO 8601.
|
||||||
- Banners are usually global (`regionScoped: false`); story/login events usually follow per-region
|
- Banners are usually global (`regionScoped: false`); story/login events usually follow per-region
|
||||||
|
|||||||
@@ -55,7 +55,7 @@ requirements are the same:
|
|||||||
sources are a single entry in `SOURCES`, with no new parsing code.
|
sources are a single entry in `SOURCES`, with no new parsing code.
|
||||||
- Source-timezone → UTC, region offsets, and ID construction happen in the parser's event builder.
|
- Source-timezone → UTC, region offsets, and ID construction happen in the parser's event builder.
|
||||||
|
|
||||||
**The three domain rules that break new adapters**, from `CLAUDE.md`:
|
**The three domain rules that break new adapters**, from `AGENTS.md`:
|
||||||
|
|
||||||
1. Everything stored as UTC ISO 8601.
|
1. Everything stored as UTC ISO 8601.
|
||||||
2. Banners are usually one global end instant; story and login events usually end at each region's
|
2. Banners are usually one global end instant; story and login events usually end at each region's
|
||||||
|
|||||||
@@ -11,4 +11,5 @@ data
|
|||||||
docs
|
docs
|
||||||
.claude
|
.claude
|
||||||
README.md
|
README.md
|
||||||
|
AGENTS.md
|
||||||
CLAUDE.md
|
CLAUDE.md
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ name: Refresh sources
|
|||||||
# any of it, it just hands CI fresher input.
|
# any of it, it just hands CI fresher input.
|
||||||
#
|
#
|
||||||
# Twelve hours apart is deliberately well clear of the six-hour-per-source floor
|
# Twelve hours apart is deliberately well clear of the six-hour-per-source floor
|
||||||
# in CLAUDE.md § Scraping conduct, and the runner enforces that floor itself, so
|
# in AGENTS.md § Scraping conduct, and the runner enforces that floor itself, so
|
||||||
# a manual dispatch on top of a scheduled run cannot double up on a wiki.
|
# a manual dispatch on top of a scheduled run cannot double up on a wiki.
|
||||||
on:
|
on:
|
||||||
schedule:
|
schedule:
|
||||||
@@ -73,7 +73,7 @@ jobs:
|
|||||||
- name: Refresh
|
- name: Refresh
|
||||||
id: refresh
|
id: refresh
|
||||||
env:
|
env:
|
||||||
# Identify the crawler with a contact URL, per CLAUDE.md.
|
# Identify the crawler with a contact URL, per AGENTS.md.
|
||||||
REFRESH_CONTACT_URL: ${{ github.server_url }}/${{ github.repository }}
|
REFRESH_CONTACT_URL: ${{ github.server_url }}/${{ github.repository }}
|
||||||
# Passed through the environment rather than interpolated into the
|
# Passed through the environment rather than interpolated into the
|
||||||
# run script, so a dispatch input cannot become shell.
|
# run script, so a dispatch input cannot become shell.
|
||||||
|
|||||||
@@ -0,0 +1,348 @@
|
|||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
This file provides guidance to coding agents working in this repository. It is the working
|
||||||
|
agreement: what this project is, the constraints it holds to, and the rules that are not visible
|
||||||
|
from the code alone. Read it before changing anything.
|
||||||
|
|
||||||
|
`CLAUDE.md` points here, so Claude Code picks it up too — keep the guidance in this file and leave
|
||||||
|
that one a pointer.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
A web app that aggregates live and upcoming events across popular gacha games, plots them on a
|
||||||
|
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
||||||
|
progress on events that repeat daily, and lets a user mark events completed.
|
||||||
|
|
||||||
|
**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across
|
||||||
|
nine games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
||||||
|
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
||||||
|
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
||||||
|
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
||||||
|
feed is a static JSON file built from snapshots, falling back to checked-in fixtures.
|
||||||
|
|
||||||
|
## Three constraints that shape everything
|
||||||
|
|
||||||
|
1. **No accounts, no logins, no user records.** Completion state lives in the browser's
|
||||||
|
`localStorage`, keyed by event ID. There is no user table and no session. Any request implying
|
||||||
|
"sync across devices" is solved with export/import JSON, not a server-side user.
|
||||||
|
2. **No LLM in the pipeline.** Event data is extracted by deterministic code-based parsers only.
|
||||||
|
There is no Anthropic dependency, no API key, and no per-run inference cost. A source that
|
||||||
|
cannot be parsed deterministically does not get an adapter — see `docs/INGESTION.md` § No LLM.
|
||||||
|
3. **A server is allowed** (Bun) and owns fetching, parsing, and SQLite. The client only ever calls
|
||||||
|
this app's own `/api/*`.
|
||||||
|
|
||||||
|
## Stack
|
||||||
|
|
||||||
|
| Layer | Choice |
|
||||||
|
|---|---|
|
||||||
|
| Runtime / server / bundler / test runner | Bun 1.3 (`Bun.serve`, `bun:sqlite`, `bun test`, `bun build`) |
|
||||||
|
| UI | React 19 + TypeScript (strict) + Tailwind |
|
||||||
|
| Storage | SQLite via `bun:sqlite` (gitignored — `*.sqlite`) |
|
||||||
|
| Validation | Zod — one schema module shared by server and client |
|
||||||
|
|
||||||
|
The only runtime dependency is `zod`. Do not add a bundler, test runner, HTTP client, or HTML
|
||||||
|
parsing library — Bun covers all four. `tsconfig.json` runs `strict` plus
|
||||||
|
`noUncheckedIndexedAccess` and `exactOptionalPropertyTypes`.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun install
|
||||||
|
bun test # full suite, offline, no network, no build needed
|
||||||
|
bun run typecheck # tsc --noEmit
|
||||||
|
bun run dev # build then serve on :3000
|
||||||
|
bun run build # feed + css + js + static into public/
|
||||||
|
|
||||||
|
# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping
|
||||||
|
# conduct before running it, and prefer --dry-run.
|
||||||
|
bun run refresh --dry-run
|
||||||
|
bun run refresh --only genshin-game8-events
|
||||||
|
|
||||||
|
# Run one source against its fixture (offline, free)
|
||||||
|
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
|
||||||
|
bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
|
||||||
|
|
||||||
|
# Single test file / single test
|
||||||
|
bun test test/dates.test.ts
|
||||||
|
bun test --test-name-pattern "year-less"
|
||||||
|
|
||||||
|
# Hosting under a subpath (GitHub Pages)
|
||||||
|
BASE_PATH=/gacha-event-tracker/ bun run build
|
||||||
|
```
|
||||||
|
|
||||||
|
**Tests must never need build output.** They run before `bun run build` in CI; anything reading
|
||||||
|
`public/` must create its own fixture tree instead.
|
||||||
|
|
||||||
|
`bun run parse ... --json` is also how `.expected.json` fixtures are regenerated after an
|
||||||
|
intentional parser change. Regenerating them makes the test self-consistent, not correct — always
|
||||||
|
re-verify a sample against the live page afterward.
|
||||||
|
|
||||||
|
## Current state of the code
|
||||||
|
|
||||||
|
```
|
||||||
|
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
||||||
|
custom.ts — reader-authored games and events, and their key spaces
|
||||||
|
src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
||||||
|
parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game
|
||||||
|
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
||||||
|
src/client/ React app, service worker, manifest
|
||||||
|
state/ progress, daily log, ignores, prefs, sort — all localStorage
|
||||||
|
useCustom.ts — the reader's own games and events (PRD F13)
|
||||||
|
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
||||||
|
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
||||||
|
serve.ts static server + /api/health
|
||||||
|
test/ 401 tests
|
||||||
|
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
||||||
|
snapshots/ current page per source, rewritten by refresh — see its README
|
||||||
|
```
|
||||||
|
|
||||||
|
Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on
|
||||||
|
disk.
|
||||||
|
|
||||||
|
## Domain rules that are not obvious from the code
|
||||||
|
|
||||||
|
These come from how gacha games actually schedule things, and they cause most bugs here:
|
||||||
|
|
||||||
|
- **Store every timestamp as UTC ISO 8601.** Sources publish in a mix of UTC+8, server-local, and
|
||||||
|
"after maintenance".
|
||||||
|
- **Banner ends are usually global and simultaneous; event ends are usually per-region.** Character
|
||||||
|
banners end at one instant worldwide; story/login events end at each region's daily reset (Asia /
|
||||||
|
America / Europe differ by hours). `regionScoped` and `regionEnds` exist for this — do not
|
||||||
|
collapse them into one timestamp.
|
||||||
|
- **`endsAt: null` is a correct, expected value.** An event whose end is genuinely unannounced gets
|
||||||
|
`endsAt: null` and `endPrecision: "unknown"`. **Never invent a plausible date to satisfy a
|
||||||
|
non-null type.** This is the worst failure mode this codebase has, because the user's entire
|
||||||
|
reason for visiting is trusting the end date.
|
||||||
|
- **Patch cycles are ~6 weeks.** Any event over 180 days is a parse error, not a long event. The
|
||||||
|
validator and the tests both reject it.
|
||||||
|
|
||||||
|
## Working on parsers
|
||||||
|
|
||||||
|
- **Parsers are pure.** No network, no `Date.now()`, no randomness — time arrives as `ctx.now`.
|
||||||
|
This is what makes fixture tests meaningful; a parser that reads the clock cannot be tested.
|
||||||
|
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
|
||||||
|
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
|
||||||
|
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
|
||||||
|
- **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg` and
|
||||||
|
`akwiki` serve one each — same host family, entirely different templates. Adding a source for a
|
||||||
|
known site is one `SOURCES` entry; a new site is a parser module.
|
||||||
|
- **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on
|
||||||
|
every row, five months apart. Publish the one our readers are on and skip the row that lacks it —
|
||||||
|
a CN date on a Global calendar is a confidently wrong date, not a near miss.
|
||||||
|
- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value
|
||||||
|
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
|
||||||
|
cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `<hr>`-separated date pairs.
|
||||||
|
Full table in `docs/INGESTION.md`. Before assuming a new
|
||||||
|
Game8 page will work, dump its structure and check **every** table — Endfield was written off as
|
||||||
|
undatable on a pass that only inspected its `Duration` rows, and its real events were further
|
||||||
|
down the page.
|
||||||
|
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
|
||||||
|
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
|
||||||
|
accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but
|
||||||
|
never names one — an unrecognised `h4` must leave the current event title alone.
|
||||||
|
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
|
||||||
|
per server region, which is the only reason `regionEnds` carries real data anywhere.
|
||||||
|
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
|
||||||
|
events vanish with no error. Abbreviated months (`Apr. 29 - May 13, 2026`) are supported for
|
||||||
|
exactly this reason. When adding a source, compare the parser's event count against an
|
||||||
|
independent count of the page.
|
||||||
|
|
||||||
|
## Event IDs are localStorage keys
|
||||||
|
|
||||||
|
```
|
||||||
|
`${game}:${slugify(title)}:${startsAt.slice(0, 10)}`
|
||||||
|
→ "genshin:mutual-aid-in-bloom-into-the-frostlands:2026-08-12"
|
||||||
|
```
|
||||||
|
|
||||||
|
Changing `slugify` or `eventId` in `src/shared/schema.ts` — including seemingly cosmetic changes to
|
||||||
|
the slug rules — **silently orphans every completion mark every user has, with no server-side
|
||||||
|
recovery**, because the server never had the data. If it must change, ship a client-side migration
|
||||||
|
that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such
|
||||||
|
change.
|
||||||
|
|
||||||
|
Two more key spaces have the same property, for the same reason:
|
||||||
|
|
||||||
|
- **`dailies:<game>`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore.
|
||||||
|
Two segments, so it cannot collide with an event ID.
|
||||||
|
- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at
|
||||||
|
04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the
|
||||||
|
format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day.
|
||||||
|
A game whose server map differs lists the affected regions in `resetOffsets` (`games.ts`) —
|
||||||
|
Endfield serves Europe off the Americas machine, so `europe` is UTC-5 there and its reset is
|
||||||
|
09:00 UTC, not 03:00. Keep that override **per region**: a blanket per-game offset drags the
|
||||||
|
regions that do have their own server onto someone else's clock. Every day-key function takes an
|
||||||
|
optional `game` — **anything reading or writing a tick must pass it**, or it writes under one
|
||||||
|
clock and reads under another. A day that drops out of `dailyDays` renders no pip, so a tick on it
|
||||||
|
becomes unreachable; check real fixture windows before changing an offset.
|
||||||
|
|
||||||
|
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
|
||||||
|
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
|
||||||
|
fixtures, that is a data-loss bug, not a diff to regenerate.
|
||||||
|
|
||||||
|
## Scraping conduct
|
||||||
|
|
||||||
|
Sources are community wikis. Treat them as a guest would:
|
||||||
|
|
||||||
|
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
||||||
|
- One request per source per refresh cycle, minimum 6 hours apart.
|
||||||
|
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
|
||||||
|
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
|
||||||
|
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
||||||
|
what a burst looks like from the far end regardless of our intent.
|
||||||
|
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
||||||
|
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
||||||
|
- Record `sourceUrl` on every event and surface attribution in the UI.
|
||||||
|
|
||||||
|
Note that game8.co disallows `GPTBot` and `Google-Extended` in `robots.txt` — it has opted out of
|
||||||
|
AI-training crawlers. Our use is a low-rate personal aggregator with attribution and no model
|
||||||
|
training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch
|
||||||
|
rate, and do not add an LLM that consumes page content.
|
||||||
|
|
||||||
|
**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns
|
||||||
|
`202 Accepted` with a bot-management body to every one of the eight game8 sources, from the first
|
||||||
|
scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse
|
||||||
|
cleanly from a normal address. So `robots.txt` permits us and the network does not, and those eight
|
||||||
|
games have only ever been built from checked-in fixtures in CI.
|
||||||
|
|
||||||
|
The per-host spacing above does not fix this and was not meant to: a 202 on the very first request
|
||||||
|
of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a
|
||||||
|
proxy, or a residential egress would each be defeating a deliberate access control, which is the
|
||||||
|
same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose
|
||||||
|
`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the
|
||||||
|
refresh from an address game8 will serve, or to find those games another source.
|
||||||
|
|
||||||
|
A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
|
||||||
|
|
||||||
|
**Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass:
|
||||||
|
|
||||||
|
| Source | Verdict |
|
||||||
|
|---|---|
|
||||||
|
| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source |
|
||||||
|
| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` |
|
||||||
|
| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have |
|
||||||
|
| `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
||||||
|
|
||||||
|
wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference`
|
||||||
|
with `Allow: /`, and disallow `ClaudeBot` and other AI crawlers by name. Our fetcher is neither: it
|
||||||
|
trains nothing, and no LLM reads the page content — constraint 2 is what keeps that true, so it is
|
||||||
|
load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive,
|
||||||
|
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
||||||
|
|
||||||
|
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
|
||||||
|
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
|
||||||
|
read). Anything that would make it fetch more often is a change to this section first.
|
||||||
|
|
||||||
|
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
|
||||||
|
never blank a calendar or stop the sources that did answer from being committed — so a failure is
|
||||||
|
exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
|
||||||
|
cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
|
||||||
|
checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
|
||||||
|
a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
|
||||||
|
the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
|
||||||
|
runner instead would skip the commit and throw away the pages that did arrive. This tier exists
|
||||||
|
because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
|
||||||
|
opens the log to read is not a signal.
|
||||||
|
|
||||||
|
## Untrusted input
|
||||||
|
|
||||||
|
Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust
|
||||||
|
boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single
|
||||||
|
seam every source passes through — **do not sanitize inside a parser**, and do not add a code path
|
||||||
|
that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup.
|
||||||
|
|
||||||
|
The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is
|
||||||
|
the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5.
|
||||||
|
|
||||||
|
## Events that repeat daily
|
||||||
|
|
||||||
|
Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed
|
||||||
|
day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published —
|
||||||
|
`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an
|
||||||
|
event's length. It adds **no schema field**, so the feed contract is untouched.
|
||||||
|
|
||||||
|
- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks
|
||||||
|
the wrong box for four hours every night.
|
||||||
|
- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null`
|
||||||
|
rule applies here exactly as it does to a countdown.
|
||||||
|
- **A tick is never removed except by the reader**, including ticks outside the window the feed now
|
||||||
|
claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere
|
||||||
|
else.
|
||||||
|
- **A repeating event the reader marked done leaves the strip.** They have said there is nothing
|
||||||
|
left to do; keeping a tickable chip for it is the app arguing with them. Their logged days are
|
||||||
|
untouched, so unmarking it brings the chip and the streak straight back.
|
||||||
|
- **Detection is a guess, not a verdict — and it ships off.** `prefs.detectDaily` defaults to
|
||||||
|
`false` and the control is labelled experimental: wording is a weak signal and gets it wrong in
|
||||||
|
both directions, so a new reader opts in rather than out. The default moves nothing for an
|
||||||
|
existing reader, whose stored `prefs` wins. The reader can mark any event as repeating, or unmark
|
||||||
|
one detection got wrong (`progress.daily`, resolved by `resolveDaily`), whether the guessing is
|
||||||
|
on or off. Store an override only when it *disagrees* with detection —
|
||||||
|
recording agreement would freeze today's guess and stop a better parser from ever reaching that
|
||||||
|
event. Neither control ever deletes a mark or a logged day, so both are reversible.
|
||||||
|
|
||||||
|
## Events the reader entered themselves
|
||||||
|
|
||||||
|
No adapter list covers a ten-game player, so a reader can define a game and type in events
|
||||||
|
(PRD F13, `src/shared/custom.ts`, `src/client/state/useCustom.ts`). They join the same lists,
|
||||||
|
timeline, sort, filters, progress, ignore and daily stores as scraped events. Four rules:
|
||||||
|
|
||||||
|
- **Their ids live in their own spaces**: `mygame:<slug>` and `myevent:<random>`. Never
|
||||||
|
`${game}:${slug}:${date}` — a reader can type a scraped event's exact title and date, and that
|
||||||
|
collision would silently share one completion mark and one streak between two events. Random also
|
||||||
|
means renaming their own event never moves its id. `dailies`, `mygame` and `myevent` are reserved
|
||||||
|
first segments and **none may ever become a `GameId`**; a test pins this.
|
||||||
|
- **Nothing they type enters the ingest pipeline.** `sanitize.ts` and `merge.ts` are for pages we do
|
||||||
|
not control. Their events are not fetched, parsed, merged, scored or quarantined.
|
||||||
|
- **A hand-entered date is never attributed to a source.** No `sourceUrl`, no source link, and the
|
||||||
|
row and detail sheet both say it is theirs. `"I don't know when it ends"` is an offered answer, for
|
||||||
|
the same reason the parsers are forbidden from guessing one.
|
||||||
|
- **They are in the export.** These exist in one browser and nowhere else, so an export without them
|
||||||
|
is a lossy backup. Import merges by id and never removes.
|
||||||
|
|
||||||
|
A lane may now be a game the reader invented, so `gameMeta` is a context resolver (`metaFor`, pure
|
||||||
|
and total) rather than a direct lookup — a lane can outlive its game when an import carries an event
|
||||||
|
whose game did not come with it.
|
||||||
|
|
||||||
|
## Shipping a new version
|
||||||
|
|
||||||
|
The shell is cached cache-first, so a reader with the tab open keeps the bundle they first loaded.
|
||||||
|
An old app presented as current is the same failure as old events presented as current, so a waiting
|
||||||
|
version is disclosed and reloaded on a tap (PRD F14, `docs/ARCHITECTURE.md` § Shipping a new version
|
||||||
|
to an open page). Four things hold it up:
|
||||||
|
|
||||||
|
- **`sw.js` must not `skipWaiting()` on install.** It activates only on the `skip-waiting` message
|
||||||
|
the reader's tap sends. Claiming an open page unasked runs the old bundle against the new cache and
|
||||||
|
says nothing.
|
||||||
|
- **`__BUILD__` must stay in `sw.js`.** `scripts/build-static.ts` substitutes a hash of the built
|
||||||
|
shell for it, which is what makes a deploy's worker bytes differ and therefore detectable. It
|
||||||
|
throws if the placeholder is gone — do not "fix" that by dropping the substitution. There is no
|
||||||
|
`CACHE_VERSION` bump ritual any more; the cache name is a namespace, and per-build names would
|
||||||
|
discard the stored feed an offline reader is reading.
|
||||||
|
- **The feed is not part of the build id.** It changes twice a day and needs no reload; announcing it
|
||||||
|
as a new version teaches readers to dismiss the notice unread.
|
||||||
|
- **The app never reloads itself.** Someone may be mid-way through typing an event in.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Commit straight to `main`.** This is a solo repo and its history is a single line; do not open a
|
||||||
|
branch for a change unless asked for one. Committing still waits to be asked, and each commit is
|
||||||
|
self-contained — one coherent change, typechecking and passing tests on its own.
|
||||||
|
- **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never
|
||||||
|
hand-write an interface that duplicates a schema.
|
||||||
|
- Every adapter ships a fixture in `fixtures/<game>/` and a test asserting parsed output. This is
|
||||||
|
how a source silently changing shape gets caught.
|
||||||
|
- Keep old fixtures when a source changes shape — the old one is the regression test proving the
|
||||||
|
parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the
|
||||||
|
current page and gets overwritten. Do not conflate them.
|
||||||
|
- **A list row is one target.** The event row opens the event and does nothing else — status,
|
||||||
|
effort, notes and the daily checklist all live in the detail sheet. A second control inside a
|
||||||
|
full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens"
|
||||||
|
without adding a second stop for keyboard and screen-reader users.
|
||||||
|
- **Sorting groups, it never reorders within a group.** Every mode falls back to
|
||||||
|
`endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product
|
||||||
|
exists for.
|
||||||
|
- **Telling the reader to do something is not the same as showing it to them.** `showCompleted` and
|
||||||
|
`showIgnored` decide what they can *look at*; the "next to expire" headline and the dailies strip
|
||||||
|
are *instructions*, so both drop anything done or ignored regardless (`outstanding` in
|
||||||
|
`src/client/state/lens.ts`). Being pointed at a job you already finished is the bug either way.
|
||||||
|
For the same reason "next to expire" reads the minimum end date rather than the head of the list,
|
||||||
|
which under "doing first" is a different event entirely.
|
||||||
@@ -1,343 +1,7 @@
|
|||||||
# CLAUDE.md
|
# CLAUDE.md
|
||||||
|
|
||||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
See [AGENTS.md](AGENTS.md). It holds the working agreement for this repository — what this is, the
|
||||||
|
three constraints, the domain rules, and the conventions — and it applies to Claude Code exactly as
|
||||||
|
written.
|
||||||
|
|
||||||
## What this is
|
Nothing lives in this file. Add guidance to `AGENTS.md` so every agent reads the same copy.
|
||||||
|
|
||||||
A web app that aggregates live and upcoming events across popular gacha games, plots them on a
|
|
||||||
calendar, sorts them by end date or by what the reader is partway through, tracks day-by-day
|
|
||||||
progress on events that repeat daily, and lets a user mark events completed.
|
|
||||||
|
|
||||||
**Status: working app, refreshing itself on a schedule.** Schema, three parsers, ten sources across
|
|
||||||
nine games, the full interface, offline support, a static server, a Docker image and CI all exist and
|
|
||||||
are tested. The refresh runner (`bun run refresh`) fetches, caches raw snapshots and rebuilds the
|
|
||||||
feed; `.github/workflows/refresh.yml` runs it twice a day and commits only when a page actually
|
|
||||||
changed. The SQLite layer and the review queue are still specified in `docs/` but not built, so the
|
|
||||||
feed is a static JSON file built from snapshots, falling back to checked-in fixtures.
|
|
||||||
|
|
||||||
## Three constraints that shape everything
|
|
||||||
|
|
||||||
1. **No accounts, no logins, no user records.** Completion state lives in the browser's
|
|
||||||
`localStorage`, keyed by event ID. There is no user table and no session. Any request implying
|
|
||||||
"sync across devices" is solved with export/import JSON, not a server-side user.
|
|
||||||
2. **No LLM in the pipeline.** Event data is extracted by deterministic code-based parsers only.
|
|
||||||
There is no Anthropic dependency, no API key, and no per-run inference cost. A source that
|
|
||||||
cannot be parsed deterministically does not get an adapter — see `docs/INGESTION.md` § No LLM.
|
|
||||||
3. **A server is allowed** (Bun) and owns fetching, parsing, and SQLite. The client only ever calls
|
|
||||||
this app's own `/api/*`.
|
|
||||||
|
|
||||||
## Stack
|
|
||||||
|
|
||||||
| Layer | Choice |
|
|
||||||
|---|---|
|
|
||||||
| Runtime / server / bundler / test runner | Bun 1.3 (`Bun.serve`, `bun:sqlite`, `bun test`, `bun build`) |
|
|
||||||
| UI | React 19 + TypeScript (strict) + Tailwind |
|
|
||||||
| Storage | SQLite via `bun:sqlite` (gitignored — `*.sqlite`) |
|
|
||||||
| Validation | Zod — one schema module shared by server and client |
|
|
||||||
|
|
||||||
The only runtime dependency is `zod`. Do not add a bundler, test runner, HTTP client, or HTML
|
|
||||||
parsing library — Bun covers all four. `tsconfig.json` runs `strict` plus
|
|
||||||
`noUncheckedIndexedAccess` and `exactOptionalPropertyTypes`.
|
|
||||||
|
|
||||||
## Commands
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bun install
|
|
||||||
bun test # full suite, offline, no network, no build needed
|
|
||||||
bun run typecheck # tsc --noEmit
|
|
||||||
bun run dev # build then serve on :3000
|
|
||||||
bun run build # feed + css + js + static into public/
|
|
||||||
|
|
||||||
# Fetch sources and refresh the snapshots. Makes real requests — see § Scraping
|
|
||||||
# conduct before running it, and prefer --dry-run.
|
|
||||||
bun run refresh --dry-run
|
|
||||||
bun run refresh --only genshin-game8-events
|
|
||||||
|
|
||||||
# Run one source against its fixture (offline, free)
|
|
||||||
bun run parse genshin-game8-events fixtures/genshin/game8-events-2026-08-14.html
|
|
||||||
bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15.html --json
|
|
||||||
|
|
||||||
# Single test file / single test
|
|
||||||
bun test test/dates.test.ts
|
|
||||||
bun test --test-name-pattern "year-less"
|
|
||||||
|
|
||||||
# Hosting under a subpath (GitHub Pages)
|
|
||||||
BASE_PATH=/gacha-event-tracker/ bun run build
|
|
||||||
```
|
|
||||||
|
|
||||||
**Tests must never need build output.** They run before `bun run build` in CI; anything reading
|
|
||||||
`public/` must create its own fixture tree instead.
|
|
||||||
|
|
||||||
`bun run parse ... --json` is also how `.expected.json` fixtures are regenerated after an
|
|
||||||
intentional parser change. Regenerating them makes the test self-consistent, not correct — always
|
|
||||||
re-verify a sample against the live page afterward.
|
|
||||||
|
|
||||||
## Current state of the code
|
|
||||||
|
|
||||||
```
|
|
||||||
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
|
|
||||||
custom.ts — reader-authored games and events, and their key spaces
|
|
||||||
src/ingest/ html.ts, dates.ts (nine formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
|
|
||||||
parsers/ game8.ts, wikigg.ts, akwiki.ts — keyed by SITE, not game
|
|
||||||
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
|
|
||||||
src/client/ React app, service worker, manifest
|
|
||||||
state/ progress, daily log, ignores, prefs, sort — all localStorage
|
|
||||||
useCustom.ts — the reader's own games and events (PRD F13)
|
|
||||||
lens.ts — who sees which rows (focus, outstanding, next-to-expire); pure
|
|
||||||
scripts/ build-feed.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
|
|
||||||
serve.ts static server + /api/health
|
|
||||||
test/ 401 tests
|
|
||||||
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
|
|
||||||
snapshots/ current page per source, rewritten by refresh — see its README
|
|
||||||
```
|
|
||||||
|
|
||||||
Not yet built: the SQLite layer and the review UI. Everything upstream of them runs as files on
|
|
||||||
disk.
|
|
||||||
|
|
||||||
## Domain rules that are not obvious from the code
|
|
||||||
|
|
||||||
These come from how gacha games actually schedule things, and they cause most bugs here:
|
|
||||||
|
|
||||||
- **Store every timestamp as UTC ISO 8601.** Sources publish in a mix of UTC+8, server-local, and
|
|
||||||
"after maintenance".
|
|
||||||
- **Banner ends are usually global and simultaneous; event ends are usually per-region.** Character
|
|
||||||
banners end at one instant worldwide; story/login events end at each region's daily reset (Asia /
|
|
||||||
America / Europe differ by hours). `regionScoped` and `regionEnds` exist for this — do not
|
|
||||||
collapse them into one timestamp.
|
|
||||||
- **`endsAt: null` is a correct, expected value.** An event whose end is genuinely unannounced gets
|
|
||||||
`endsAt: null` and `endPrecision: "unknown"`. **Never invent a plausible date to satisfy a
|
|
||||||
non-null type.** This is the worst failure mode this codebase has, because the user's entire
|
|
||||||
reason for visiting is trusting the end date.
|
|
||||||
- **Patch cycles are ~6 weeks.** Any event over 180 days is a parse error, not a long event. The
|
|
||||||
validator and the tests both reject it.
|
|
||||||
|
|
||||||
## Working on parsers
|
|
||||||
|
|
||||||
- **Parsers are pure.** No network, no `Date.now()`, no randomness — time arrives as `ctx.now`.
|
|
||||||
This is what makes fixture tests meaningful; a parser that reads the clock cannot be tested.
|
|
||||||
- **Skip, never guess.** Every function in `dates.ts` returns `null` rather than inferring a missing
|
|
||||||
year, month, or end. `readColumnTable` drops a row it cannot date. An omitted event is a
|
|
||||||
recoverable disappointment; a confidently wrong date is the failure this product exists to prevent.
|
|
||||||
- **Parsers are keyed by site, not game.** One `game8` parser serves eight sources; `wikigg` and
|
|
||||||
`akwiki` serve one each — same host family, entirely different templates. Adding a source for a
|
|
||||||
known site is one `SOURCES` entry; a new site is a parser module.
|
|
||||||
- **A source may publish more than one region's schedule.** Arknights' wiki lists CN and Global on
|
|
||||||
every row, five months apart. Publish the one our readers are on and skip the row that lacks it —
|
|
||||||
a CN date on a Global calendar is a confidently wrong date, not a near miss.
|
|
||||||
- **Game8 has no single template.** Seven shapes are known and a page may mix them: label/value
|
|
||||||
detail tables, column tables, image-grid schedules (unsupportable), combined label+range+blurb
|
|
||||||
cells, rowspan Start/End pairs, labelled `Start: … End: …` cells, and `<hr>`-separated date pairs.
|
|
||||||
Full table in `docs/INGESTION.md`. Before assuming a new
|
|
||||||
Game8 page will work, dump its structure and check **every** table — Endfield was written off as
|
|
||||||
undatable on a pass that only inspected its `Duration` rows, and its real events were further
|
|
||||||
down the page.
|
|
||||||
- **Check what fences a section off.** Inclusion is decided by headings, and the level varies: Persona
|
|
||||||
5 hides fifty finished events behind nothing but an `<h4>Finished Events</h4>` in a collapsed
|
|
||||||
accordion, while Genshin uses `h4` for sub-headings *inside* one event. So `h4` gates sections but
|
|
||||||
never names one — an unrecognised `h4` must leave the current event title alone.
|
|
||||||
- **Prefer a source that states machine-readable times.** wiki.gg emits ISO timestamps with a timer
|
|
||||||
per server region, which is the only reason `regionEnds` carries real data anywhere.
|
|
||||||
- **Silent drops are the dangerous failure.** A date format the parser does not recognise makes
|
|
||||||
events vanish with no error. Abbreviated months (`Apr. 29 - May 13, 2026`) are supported for
|
|
||||||
exactly this reason. When adding a source, compare the parser's event count against an
|
|
||||||
independent count of the page.
|
|
||||||
|
|
||||||
## Event IDs are localStorage keys
|
|
||||||
|
|
||||||
```
|
|
||||||
`${game}:${slugify(title)}:${startsAt.slice(0, 10)}`
|
|
||||||
→ "genshin:mutual-aid-in-bloom-into-the-frostlands:2026-08-12"
|
|
||||||
```
|
|
||||||
|
|
||||||
Changing `slugify` or `eventId` in `src/shared/schema.ts` — including seemingly cosmetic changes to
|
|
||||||
the slug rules — **silently orphans every completion mark every user has, with no server-side
|
|
||||||
recovery**, because the server never had the data. If it must change, ship a client-side migration
|
|
||||||
that remaps old keys and keep it for at least a year. Use the **schema-guardian** agent on any such
|
|
||||||
change.
|
|
||||||
|
|
||||||
Two more key spaces have the same property, for the same reason:
|
|
||||||
|
|
||||||
- **`dailies:<game>`** (`dailiesId` in `src/shared/daily.ts`) keys a game's standing daily chore.
|
|
||||||
Two segments, so it cannot collide with an event ID.
|
|
||||||
- **Game-day keys** (`dayKey`) are `YYYY-MM-DD` in *server-reset space*, not UTC — the day rolls at
|
|
||||||
04:00 local server time. They are storage keys *and* they are compared with `<` and sorted, so the
|
|
||||||
format is fixed. Changing the reset hour or the offsets moves every reader's streak by a day.
|
|
||||||
A game whose server map differs lists the affected regions in `resetOffsets` (`games.ts`) —
|
|
||||||
Endfield serves Europe off the Americas machine, so `europe` is UTC-5 there and its reset is
|
|
||||||
09:00 UTC, not 03:00. Keep that override **per region**: a blanket per-game offset drags the
|
|
||||||
regions that do have their own server onto someone else's clock. Every day-key function takes an
|
|
||||||
optional `game` — **anything reading or writing a tick must pass it**, or it writes under one
|
|
||||||
clock and reads under another. A day that drops out of `dailyDays` renders no pip, so a tick on it
|
|
||||||
becomes unreachable; check real fixture windows before changing an offset.
|
|
||||||
|
|
||||||
The sanitizer at the ingest boundary recomputes an event ID only when a sanitized title actually
|
|
||||||
changed *and* the ID was minted the standard way. If a change to it starts moving IDs on real
|
|
||||||
fixtures, that is a data-loss bug, not a diff to regenerate.
|
|
||||||
|
|
||||||
## Scraping conduct
|
|
||||||
|
|
||||||
Sources are community wikis. Treat them as a guest would:
|
|
||||||
|
|
||||||
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
|
|
||||||
- One request per source per refresh cycle, minimum 6 hours apart.
|
|
||||||
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
|
|
||||||
sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
|
|
||||||
eight back-to-back requests to a single site — which is the shape an edge network throttles, and
|
|
||||||
what a burst looks like from the far end regardless of our intent.
|
|
||||||
- Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
|
|
||||||
- Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
|
|
||||||
- Record `sourceUrl` on every event and surface attribution in the UI.
|
|
||||||
|
|
||||||
Note that game8.co disallows `GPTBot` and `Google-Extended` in `robots.txt` — it has opted out of
|
|
||||||
AI-training crawlers. Our use is a low-rate personal aggregator with attribution and no model
|
|
||||||
training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch
|
|
||||||
rate, and do not add an LLM that consumes page content.
|
|
||||||
|
|
||||||
**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns
|
|
||||||
`202 Accepted` with a bot-management body to every one of the eight game8 sources, from the first
|
|
||||||
scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse
|
|
||||||
cleanly from a normal address. So `robots.txt` permits us and the network does not, and those eight
|
|
||||||
games have only ever been built from checked-in fixtures in CI.
|
|
||||||
|
|
||||||
The per-host spacing above does not fix this and was not meant to: a 202 on the very first request
|
|
||||||
of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a
|
|
||||||
proxy, or a residential egress would each be defeating a deliberate access control, which is the
|
|
||||||
same reason `uma.moe` was declined below — and unlike `uma.moe` we would be doing it to a host whose
|
|
||||||
`robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the
|
|
||||||
refresh from an address game8 will serve, or to find those games another source.
|
|
||||||
|
|
||||||
A source whose ToS forbids automated access does not get an adapter. Flag it and ask.
|
|
||||||
|
|
||||||
**Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass:
|
|
||||||
|
|
||||||
| Source | Verdict |
|
|
||||||
|---|---|
|
|
||||||
| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source |
|
|
||||||
| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` |
|
|
||||||
| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have |
|
|
||||||
| `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h |
|
|
||||||
|
|
||||||
wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference`
|
|
||||||
with `Allow: /`, and disallow `ClaudeBot` and other AI crawlers by name. Our fetcher is neither: it
|
|
||||||
trains nothing, and no LLM reads the page content — constraint 2 is what keeps that true, so it is
|
|
||||||
load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive,
|
|
||||||
Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.
|
|
||||||
|
|
||||||
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
|
|
||||||
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
|
|
||||||
read). Anything that would make it fetch more often is a change to this section first.
|
|
||||||
|
|
||||||
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
|
|
||||||
never blank a calendar or stop the sources that did answer from being committed — so a failure is
|
|
||||||
exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
|
|
||||||
cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
|
|
||||||
checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
|
|
||||||
a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
|
|
||||||
the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
|
|
||||||
runner instead would skip the commit and throw away the pages that did arrive. This tier exists
|
|
||||||
because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
|
|
||||||
opens the log to read is not a signal.
|
|
||||||
|
|
||||||
## Untrusted input
|
|
||||||
|
|
||||||
Every string on an event came from a page we do not control. `src/ingest/sanitize.ts` is the trust
|
|
||||||
boundary and it is wired into `toAdapter()` in `src/ingest/adapters/index.ts`, which is the single
|
|
||||||
seam every source passes through — **do not sanitize inside a parser**, and do not add a code path
|
|
||||||
that reaches `parser.parse` directly. Parsers stay pure readers of one site's markup.
|
|
||||||
|
|
||||||
The sanitizer never touches a date, cleans rather than drops (a title that sanitizes to nothing is
|
|
||||||
the only drop), and logs every repair and drop by default. See `docs/INGESTION.md` § Stage 2.5.
|
|
||||||
|
|
||||||
## Events that repeat daily
|
|
||||||
|
|
||||||
Some events are twenty small jobs on twenty deadlines, not one job with an end date, and a missed
|
|
||||||
day is unrecoverable. `src/shared/daily.ts` decides dailiness from what the source published —
|
|
||||||
`type: "login"`, or "daily"/"check-in"/"7-day" wording — and never from a game's habits or an
|
|
||||||
event's length. It adds **no schema field**, so the feed contract is untouched.
|
|
||||||
|
|
||||||
- The day rolls at **04:00 server time** (`RESET_HOUR_LOCAL`), per region. Getting this wrong ticks
|
|
||||||
the wrong box for four hours every night.
|
|
||||||
- **An unannounced end yields no checklist**, not a checklist of guessed length — the `endsAt: null`
|
|
||||||
rule applies here exactly as it does to a countdown.
|
|
||||||
- **A tick is never removed except by the reader**, including ticks outside the window the feed now
|
|
||||||
claims. A source quietly moving a date must not erase a fortnight's streak that exists nowhere
|
|
||||||
else.
|
|
||||||
- **A repeating event the reader marked done leaves the strip.** They have said there is nothing
|
|
||||||
left to do; keeping a tickable chip for it is the app arguing with them. Their logged days are
|
|
||||||
untouched, so unmarking it brings the chip and the streak straight back.
|
|
||||||
- **Detection is a guess, not a verdict — and it ships off.** `prefs.detectDaily` defaults to
|
|
||||||
`false` and the control is labelled experimental: wording is a weak signal and gets it wrong in
|
|
||||||
both directions, so a new reader opts in rather than out. The default moves nothing for an
|
|
||||||
existing reader, whose stored `prefs` wins. The reader can mark any event as repeating, or unmark
|
|
||||||
one detection got wrong (`progress.daily`, resolved by `resolveDaily`), whether the guessing is
|
|
||||||
on or off. Store an override only when it *disagrees* with detection —
|
|
||||||
recording agreement would freeze today's guess and stop a better parser from ever reaching that
|
|
||||||
event. Neither control ever deletes a mark or a logged day, so both are reversible.
|
|
||||||
|
|
||||||
## Events the reader entered themselves
|
|
||||||
|
|
||||||
No adapter list covers a ten-game player, so a reader can define a game and type in events
|
|
||||||
(PRD F13, `src/shared/custom.ts`, `src/client/state/useCustom.ts`). They join the same lists,
|
|
||||||
timeline, sort, filters, progress, ignore and daily stores as scraped events. Four rules:
|
|
||||||
|
|
||||||
- **Their ids live in their own spaces**: `mygame:<slug>` and `myevent:<random>`. Never
|
|
||||||
`${game}:${slug}:${date}` — a reader can type a scraped event's exact title and date, and that
|
|
||||||
collision would silently share one completion mark and one streak between two events. Random also
|
|
||||||
means renaming their own event never moves its id. `dailies`, `mygame` and `myevent` are reserved
|
|
||||||
first segments and **none may ever become a `GameId`**; a test pins this.
|
|
||||||
- **Nothing they type enters the ingest pipeline.** `sanitize.ts` and `merge.ts` are for pages we do
|
|
||||||
not control. Their events are not fetched, parsed, merged, scored or quarantined.
|
|
||||||
- **A hand-entered date is never attributed to a source.** No `sourceUrl`, no source link, and the
|
|
||||||
row and detail sheet both say it is theirs. `"I don't know when it ends"` is an offered answer, for
|
|
||||||
the same reason the parsers are forbidden from guessing one.
|
|
||||||
- **They are in the export.** These exist in one browser and nowhere else, so an export without them
|
|
||||||
is a lossy backup. Import merges by id and never removes.
|
|
||||||
|
|
||||||
A lane may now be a game the reader invented, so `gameMeta` is a context resolver (`metaFor`, pure
|
|
||||||
and total) rather than a direct lookup — a lane can outlive its game when an import carries an event
|
|
||||||
whose game did not come with it.
|
|
||||||
|
|
||||||
## Shipping a new version
|
|
||||||
|
|
||||||
The shell is cached cache-first, so a reader with the tab open keeps the bundle they first loaded.
|
|
||||||
An old app presented as current is the same failure as old events presented as current, so a waiting
|
|
||||||
version is disclosed and reloaded on a tap (PRD F14, `docs/ARCHITECTURE.md` § Shipping a new version
|
|
||||||
to an open page). Four things hold it up:
|
|
||||||
|
|
||||||
- **`sw.js` must not `skipWaiting()` on install.** It activates only on the `skip-waiting` message
|
|
||||||
the reader's tap sends. Claiming an open page unasked runs the old bundle against the new cache and
|
|
||||||
says nothing.
|
|
||||||
- **`__BUILD__` must stay in `sw.js`.** `scripts/build-static.ts` substitutes a hash of the built
|
|
||||||
shell for it, which is what makes a deploy's worker bytes differ and therefore detectable. It
|
|
||||||
throws if the placeholder is gone — do not "fix" that by dropping the substitution. There is no
|
|
||||||
`CACHE_VERSION` bump ritual any more; the cache name is a namespace, and per-build names would
|
|
||||||
discard the stored feed an offline reader is reading.
|
|
||||||
- **The feed is not part of the build id.** It changes twice a day and needs no reload; announcing it
|
|
||||||
as a new version teaches readers to dismiss the notice unread.
|
|
||||||
- **The app never reloads itself.** Someone may be mid-way through typing an event in.
|
|
||||||
|
|
||||||
## Conventions
|
|
||||||
|
|
||||||
- **Commit straight to `main`.** This is a solo repo and its history is a single line; do not open a
|
|
||||||
branch for a change unless asked for one. Committing still waits to be asked, and each commit is
|
|
||||||
self-contained — one coherent change, typechecking and passing tests on its own.
|
|
||||||
- **Zod schemas are the single source of truth for types.** Derive with `z.infer<>`; never
|
|
||||||
hand-write an interface that duplicates a schema.
|
|
||||||
- Every adapter ships a fixture in `fixtures/<game>/` and a test asserting parsed output. This is
|
|
||||||
how a source silently changing shape gets caught.
|
|
||||||
- Keep old fixtures when a source changes shape — the old one is the regression test proving the
|
|
||||||
parser still handles the previous format. Fixtures are pinned and permanent; `snapshots/` is the
|
|
||||||
current page and gets overwritten. Do not conflate them.
|
|
||||||
- **A list row is one target.** The event row opens the event and does nothing else — status,
|
|
||||||
effort, notes and the daily checklist all live in the detail sheet. A second control inside a
|
|
||||||
full-bleed row target is a mis-tap waiting to happen, and a decorative chevron says "this opens"
|
|
||||||
without adding a second stop for keyboard and screen-reader users.
|
|
||||||
- **Sorting groups, it never reorders within a group.** Every mode falls back to
|
|
||||||
`endingSoonestFirst`, so choosing one can never cost the reader the deadline order the product
|
|
||||||
exists for.
|
|
||||||
- **Telling the reader to do something is not the same as showing it to them.** `showCompleted` and
|
|
||||||
`showIgnored` decide what they can *look at*; the "next to expire" headline and the dailies strip
|
|
||||||
are *instructions*, so both drop anything done or ignored regardless (`outstanding` in
|
|
||||||
`src/client/state/lens.ts`). Being pointed at a job you already finished is the bug either way.
|
|
||||||
For the same reason "next to expire" reads the minimum end date rather than the head of the list,
|
|
||||||
which under "doing first" is a different event entirely.
|
|
||||||
|
|||||||
@@ -213,7 +213,7 @@ page is sanitized at the ingest boundary before it reaches the feed, the browser
|
|||||||
|
|
||||||
| Document | Covers |
|
| Document | Covers |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `CLAUDE.md` | Working agreements, domain rules, conventions |
|
| `AGENTS.md` | Working agreements, domain rules, conventions (`CLAUDE.md` points here) |
|
||||||
| `docs/PRD.md` | What this is, who for, what's out of scope |
|
| `docs/PRD.md` | What this is, who for, what's out of scope |
|
||||||
| `docs/ARCHITECTURE.md` | Process shape, routes, deployment |
|
| `docs/ARCHITECTURE.md` | Process shape, routes, deployment |
|
||||||
| `docs/DATA-MODEL.md` | Event schema, SQLite tables, client storage |
|
| `docs/DATA-MODEL.md` | Event schema, SQLite tables, client storage |
|
||||||
|
|||||||
+4
-4
@@ -91,7 +91,7 @@ first thing to find out.
|
|||||||
2. `bun run refresh --dry-run` locally to see the plan and the robots verdict per source without
|
2. `bun run refresh --dry-run` locally to see the plan and the robots verdict per source without
|
||||||
making a request.
|
making a request.
|
||||||
3. If Game8 is being refused, that is a scraping-conduct question before it is a code question —
|
3. If Game8 is being refused, that is a scraping-conduct question before it is a code question —
|
||||||
re-read `CLAUDE.md` § Scraping conduct and decide, rather than working around it.
|
re-read `AGENTS.md` § Scraping conduct and decide, rather than working around it.
|
||||||
4. Regardless of cause: surface it in the UI. `Colophon.tsx` already receives `staleCount`, and
|
4. Regardless of cause: surface it in the UI. `Colophon.tsx` already receives `staleCount`, and
|
||||||
`App.tsx` computes `staleSources` at a two-day threshold. Verify a reader actually sees that
|
`App.tsx` computes `staleSources` at a two-day threshold. Verify a reader actually sees that
|
||||||
banner today, because if five of six games are on four-day-old fixtures, the app is currently
|
banner today, because if five of six games are on four-day-old fixtures, the app is currently
|
||||||
@@ -177,7 +177,7 @@ So this is a **default and framing problem**, not a missing-feature problem.
|
|||||||
the minimum end date rather than the head of the list — under "doing first" those differ.
|
the minimum end date rather than the head of the list — under "doing first" those differ.
|
||||||
2. Cap the "Running now" section at a handful of rows with an explicit "show all N" expander, rather
|
2. Cap the "Running now" section at a handful of rows with an explicit "show all N" expander, rather
|
||||||
than rendering 21. Grouping stays as-is; this is truncation of the view, not a re-sort, so the
|
than rendering 21. Grouping stays as-is; this is truncation of the view, not a re-sort, so the
|
||||||
deadline-order guarantee in `CLAUDE.md` § Conventions is untouched.
|
deadline-order guarantee in `AGENTS.md` § Conventions is untouched.
|
||||||
3. **Persist the view.** `const [view, setView] = useState<View>("soon")` in `App.tsx:68` is
|
3. **Persist the view.** `const [view, setView] = useState<View>("soon")` in `App.tsx:68` is
|
||||||
component state, so every reload throws the reader back to the list even if they chose the
|
component state, so every reload throws the reader back to the list even if they chose the
|
||||||
timeline last time. It belongs in `prefs` next to `sort` and `focusGame`.
|
timeline last time. It belongs in `prefs` next to `sort` and `focusGame`.
|
||||||
@@ -239,7 +239,7 @@ scraping, no ToS question, and no server.
|
|||||||
2. **Never mint a user event's ID the standard way.** `${game}:${slug}:${date}` and `dailies:<game>`
|
2. **Never mint a user event's ID the standard way.** `${game}:${slug}:${date}` and `dailies:<game>`
|
||||||
are live key spaces and a collision corrupts real marks. Give user events their own prefix, and
|
are live key spaces and a collision corrupts real marks. Give user events their own prefix, and
|
||||||
run the **schema-guardian** agent on the change — this is exactly the class of change
|
run the **schema-guardian** agent on the change — this is exactly the class of change
|
||||||
`CLAUDE.md` § Event IDs flags.
|
`AGENTS.md` § Event IDs flags.
|
||||||
3. **Mark provenance in the UI.** A hand-entered date must be visibly the reader's own, never
|
3. **Mark provenance in the UI.** A hand-entered date must be visibly the reader's own, never
|
||||||
attributed to a source, and must not flow into merge or sanitisation. The trust boundary at
|
attributed to a source, and must not flow into merge or sanitisation. The trust boundary at
|
||||||
`src/ingest/sanitize.ts` is for pages we do not control; this is a different path entirely.
|
`src/ingest/sanitize.ts` is for pages we do not control; this is a different path entirely.
|
||||||
@@ -265,7 +265,7 @@ scraping, no ToS question, and no server.
|
|||||||
"I haven't touched Wuwa, zzz, HSR and arknights in a long time," and a third reader's plan to play
|
"I haven't touched Wuwa, zzz, HSR and arknights in a long time," and a third reader's plan to play
|
||||||
one gacha in concentrated bursts. The `status` ("partway through") and `effort` fields plus the
|
one gacha in concentrated bursts. The `status` ("partway through") and `effort` fields plus the
|
||||||
"doing first" sort already serve this. Resist adding a "you haven't played X in N days" nudge:
|
"doing first" sort already serve this. Resist adding a "you haven't played X in N days" nudge:
|
||||||
`CLAUDE.md` draws a hard line between what the app *shows* and what it *tells you to do*, and
|
`AGENTS.md` draws a hard line between what the app *shows* and what it *tells you to do*, and
|
||||||
nagging someone about a game they consciously dropped is the app arguing with them.
|
nagging someone about a game they consciously dropped is the app arguing with them.
|
||||||
- **The daily-checklist feature got no signal at all** — not one comment, positive or negative, in 26.
|
- **The daily-checklist feature got no signal at all** — not one comment, positive or negative, in 26.
|
||||||
It shipped detection-off and experimental two days ago, which is the right posture. Do not invest
|
It shipped detection-off and experimental two days ago, which is the right posture. Do not invest
|
||||||
|
|||||||
+2
-2
@@ -175,7 +175,7 @@ section but must never claim the event title.
|
|||||||
- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
|
- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
|
||||||
5xxs or times out means "do not fetch", because a permission we could not read is not a
|
5xxs or times out means "do not fetch", because a permission we could not read is not a
|
||||||
permission we have. A 404 means no restrictions.
|
permission we have. A 404 means no restrictions.
|
||||||
- 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
|
- 20s timeout. **No retries**: a retry is a second request, and AGENTS.md § Scraping conduct says
|
||||||
one per source per cycle. A failed source waits for the next cycle instead.
|
one per source per cycle. A failed source waits for the next cycle instead.
|
||||||
- **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole
|
- **Only `200` is a page** (plus `304` for "unchanged"). Not `response.ok` — that admits the whole
|
||||||
2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge
|
2xx range, and `202 Accepted` is what an edge bot-manager answers with while it serves a challenge
|
||||||
@@ -195,7 +195,7 @@ identically whether the page moved behind a login or a CDN decided the runner is
|
|||||||
**The failure streak is read, not just written.** `consecutiveFailures` reaching
|
**The failure streak is read, not just written.** `consecutiveFailures` reaching
|
||||||
`BROKEN_AFTER_FAILURES` (3, so ~36h at two cycles a day) promotes a source from "down" to `broken`:
|
`BROKEN_AFTER_FAILURES` (3, so ~36h at two cycles a day) promotes a source from "down" to `broken`:
|
||||||
annotated on the run page, listed in the job summary with its status code, and counted in the
|
annotated on the run page, listed in the job summary with its status code, and counted in the
|
||||||
`broken` step output that `refresh.yml` fails on *after* committing. See CLAUDE.md § Scraping
|
`broken` step output that `refresh.yml` fails on *after* committing. See AGENTS.md § Scraping
|
||||||
conduct for why that ordering is load-bearing.
|
conduct for why that ordering is load-bearing.
|
||||||
|
|
||||||
**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
|
**Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
|
||||||
|
|||||||
+2
-2
@@ -12,7 +12,7 @@ Raw pages, exactly as fetched. `scripts/refresh-sources.ts` writes them; nothing
|
|||||||
what those bytes are in — the `Content-Type` header, else a `<meta charset>` in the page, else
|
what those bytes are in — the `Content-Type` header, else a `<meta charset>` in the page, else
|
||||||
UTF-8 — and `bytes` is the served length. A page in Shift_JIS or Latin-1 decoded as UTF-8 would be
|
UTF-8 — and `bytes` is the served length. A page in Shift_JIS or Latin-1 decoded as UTF-8 would be
|
||||||
stored as a field of U+FFFD with the original bytes gone; re-parsing could never recover it, and the
|
stored as a field of U+FFFD with the original bytes gone; re-parsing could never recover it, and the
|
||||||
mojibake would reach `slugify`, moving every event ID for that source (CLAUDE.md § Event IDs are
|
mojibake would reach `slugify`, moving every event ID for that source (AGENTS.md § Event IDs are
|
||||||
localStorage keys).
|
localStorage keys).
|
||||||
|
|
||||||
Every file is written to a sibling `.tmp-*` and renamed into place, body before metadata, so an
|
Every file is written to a sibling `.tmp-*` and renamed into place, body before metadata, so an
|
||||||
@@ -23,7 +23,7 @@ directory, so otherwise a run killed mid-write would pin a half-page in git fore
|
|||||||
Three reasons this is committed rather than cached:
|
Three reasons this is committed rather than cached:
|
||||||
|
|
||||||
- **Re-parsing never re-fetches.** Iterating on a parser reads these files, not the wikis
|
- **Re-parsing never re-fetches.** Iterating on a parser reads these files, not the wikis
|
||||||
(CLAUDE.md § Scraping conduct).
|
(AGENTS.md § Scraping conduct).
|
||||||
- **A refresh is reviewable.** The commit diff is the page diff, so "an event vanished" is a
|
- **A refresh is reviewable.** The commit diff is the page diff, so "an event vanished" is a
|
||||||
question you can answer from git rather than from a wiki that has since changed again.
|
question you can answer from git rather than from a wiki that has since changed again.
|
||||||
- **The build stays offline.** `bun run build:feed` parses whichever of these exists and falls back
|
- **The build stays offline.** `bun run build:feed` parses whichever of these exists and falls back
|
||||||
|
|||||||
@@ -118,7 +118,7 @@ function toAdapter(spec: SourceSpec): Adapter {
|
|||||||
// keeps parsers what they are — pure readers of one site's markup.
|
// keeps parsers what they are — pure readers of one site's markup.
|
||||||
//
|
//
|
||||||
// `sanitizeEvents` logs to console.warn by default, so a repaired or
|
// `sanitizeEvents` logs to console.warn by default, so a repaired or
|
||||||
// dropped event is never silent (CLAUDE.md § Silent drops).
|
// dropped event is never silent (AGENTS.md § Silent drops).
|
||||||
return sanitizeEvents(parser.parse(html, ctx), {
|
return sanitizeEvents(parser.parse(html, ctx), {
|
||||||
sourceId: ctx.sourceId,
|
sourceId: ctx.sourceId,
|
||||||
fallbackUrl: ctx.sourceUrl,
|
fallbackUrl: ctx.sourceUrl,
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
* robots.txt: parsing, matching, and a per-host cache.
|
* robots.txt: parsing, matching, and a per-host cache.
|
||||||
*
|
*
|
||||||
* Sources are community wikis and this project's standing rule is to behave as
|
* Sources are community wikis and this project's standing rule is to behave as
|
||||||
* a guest would (CLAUDE.md § Scraping conduct). That starts with actually
|
* a guest would (AGENTS.md § Scraping conduct). That starts with actually
|
||||||
* reading robots.txt rather than assuming a path is fair game.
|
* reading robots.txt rather than assuming a path is fair game.
|
||||||
*
|
*
|
||||||
* Parsing is a pure function over text, deliberately separated from fetching,
|
* Parsing is a pure function over text, deliberately separated from fetching,
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ import { decodeEntities } from "./html.ts";
|
|||||||
* sanitiser's job stops at prose and URLs.
|
* sanitiser's job stops at prose and URLs.
|
||||||
* 2. **Clean, do not drop.** A hostile title is truncated and stripped, not
|
* 2. **Clean, do not drop.** A hostile title is truncated and stripped, not
|
||||||
* rejected — an event vanishing without a trace is the failure mode this
|
* rejected — an event vanishing without a trace is the failure mode this
|
||||||
* codebase fears most (CLAUDE.md § Silent drops). The one unrecoverable
|
* codebase fears most (AGENTS.md § Silent drops). The one unrecoverable
|
||||||
* case is a title that sanitises to nothing, and that emits a note the
|
* case is a title that sanitises to nothing, and that emits a note the
|
||||||
* caller is expected to surface.
|
* caller is expected to surface.
|
||||||
* 3. **Never throw on junk.** Malformed entities, lone surrogates, absurd code
|
* 3. **Never throw on junk.** Malformed entities, lone surrogates, absurd code
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
*
|
*
|
||||||
* Every fetched page is stored verbatim on disk so that re-parsing — which is
|
* Every fetched page is stored verbatim on disk so that re-parsing — which is
|
||||||
* the thing we actually iterate on — never costs the source another request
|
* the thing we actually iterate on — never costs the source another request
|
||||||
* (CLAUDE.md § Scraping conduct). A snapshot plus its metadata is also what
|
* (AGENTS.md § Scraping conduct). A snapshot plus its metadata is also what
|
||||||
* makes a conditional request possible on the next cycle: we keep the ETag and
|
* makes a conditional request possible on the next cycle: we keep the ETag and
|
||||||
* Last-Modified the server gave us and hand them back.
|
* Last-Modified the server gave us and hand them back.
|
||||||
*
|
*
|
||||||
@@ -119,7 +119,7 @@ export interface DecodedBody {
|
|||||||
* comes back as a field of U+FFFD. That is not merely ugly: mojibake in a title
|
* comes back as a field of U+FFFD. That is not merely ugly: mojibake in a title
|
||||||
* flows through `slugify` into the event ID, which is a localStorage key, so a
|
* flows through `slugify` into the event ID, which is a localStorage key, so a
|
||||||
* mis-decoded fetch silently orphans every completion mark for that source
|
* mis-decoded fetch silently orphans every completion mark for that source
|
||||||
* (CLAUDE.md § Event IDs are localStorage keys).
|
* (AGENTS.md § Event IDs are localStorage keys).
|
||||||
*
|
*
|
||||||
* Header first, then a `<meta charset>` sniff, then UTF-8. An encoding label
|
* Header first, then a `<meta charset>` sniff, then UTF-8. An encoding label
|
||||||
* the runtime does not know falls back to UTF-8 rather than throwing — the raw
|
* the runtime does not know falls back to UTF-8 rather than throwing — the raw
|
||||||
@@ -275,7 +275,7 @@ export class SnapshotStore {
|
|||||||
/**
|
/**
|
||||||
* Has enough time passed to fetch this source again?
|
* Has enough time passed to fetch this source again?
|
||||||
*
|
*
|
||||||
* The floor is six hours per source (CLAUDE.md). A source we have never
|
* The floor is six hours per source (AGENTS.md). A source we have never
|
||||||
* checked is always due.
|
* checked is always due.
|
||||||
*/
|
*/
|
||||||
isDue(state: SnapshotState, nowMs: number, minIntervalMs: number): boolean {
|
isDue(state: SnapshotState, nowMs: number, minIntervalMs: number): boolean {
|
||||||
@@ -300,7 +300,7 @@ export class SnapshotStore {
|
|||||||
* one. The validators are the exception: a server is free to rotate an ETag
|
* one. The validators are the exception: a server is free to rotate an ETag
|
||||||
* while serving the very same bytes, and keeping the old one would mean
|
* while serving the very same bytes, and keeping the old one would mean
|
||||||
* sending a stale `If-None-Match` forever — every cycle costing the wiki a
|
* sending a stale `If-None-Match` forever — every cycle costing the wiki a
|
||||||
* full body where a 304 was the whole point (CLAUDE.md § Scraping conduct).
|
* full body where a 304 was the whole point (AGENTS.md § Scraping conduct).
|
||||||
* So the metadata is refreshed, and `changed` stays false.
|
* So the metadata is refreshed, and `changed` stays false.
|
||||||
*/
|
*/
|
||||||
async save(sourceId: string, input: SaveInput): Promise<SaveResult> {
|
async save(sourceId: string, input: SaveInput): Promise<SaveResult> {
|
||||||
|
|||||||
+1
-1
@@ -25,7 +25,7 @@ import { DAY, HOUR, REGION_RESET_UTC_OFFSET } from "./time.ts";
|
|||||||
export const RESET_HOUR_LOCAL = 4;
|
export const RESET_HOUR_LOCAL = 4;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* A 180-day event is already a parse error (see CLAUDE.md § Domain rules), so
|
* A 180-day event is already a parse error (see AGENTS.md § Domain rules), so
|
||||||
* this only ever fires on data that is wrong; it exists so a bad end date
|
* this only ever fires on data that is wrong; it exists so a bad end date
|
||||||
* cannot make the client allocate an unbounded array.
|
* cannot make the client allocate an unbounded array.
|
||||||
*/
|
*/
|
||||||
|
|||||||
+1
-1
@@ -186,7 +186,7 @@ Allow: /wiki/Event
|
|||||||
|
|
||||||
describe("the sources we actually fetch", () => {
|
describe("the sources we actually fetch", () => {
|
||||||
// Game8 opts out of AI-training crawlers by name and leaves everyone else
|
// Game8 opts out of AI-training crawlers by name and leaves everyone else
|
||||||
// alone (CLAUDE.md § Scraping conduct). If that ever changes, this is where
|
// alone (AGENTS.md § Scraping conduct). If that ever changes, this is where
|
||||||
// it should be noticed.
|
// it should be noticed.
|
||||||
const game8 = parseRobots(`
|
const game8 = parseRobots(`
|
||||||
User-agent: GPTBot
|
User-agent: GPTBot
|
||||||
|
|||||||
Reference in New Issue
Block a user