feat(refresh): --force, to ask before the 6h floor is up

The interval gate has no override, so a page you know has just changed cannot
be fetched until six hours after the last attempt. The only workaround was
deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and
lastConfirmedAt — resetting the broken-source streak and making the footer age
every source from when its bytes last changed rather than when we last
confirmed them. A flag that says what it means is better than a side effect
nobody documented.

It sets aside the interval and nothing else. Conditional headers still go out,
which is what makes forcing defensible at all: the host is asked, not
re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one
request per source and the no-retry rule all still apply — a source that was
not due and is also disallowed stays skipped, for the reason that matters.

Refused under CI, like --assume-robots-on-403 and for the same reason: a
schedule that forces every cycle is a shorter interval with extra steps, and
the interval is the obligation, not the default. So AGENTS.md § Scraping
conduct is amended rather than left to be quietly contradicted by a flag.

Every source asked early is named in summary.forced and warned about. A run
that was due anyway is never reported as forced — a summary that cried "forced"
on an ordinary run would train the reader to ignore the word.

Also repoints the "unknown flag" test, which used --force as its example and
stopped testing anything the moment --force existed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-19 04:59:52 +02:00
co-authored by Claude Opus 5
parent 240e4d59d1
commit 6552528acb
4 changed files with 187 additions and 10 deletions
+15 -5
View File
@@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest
theme.ts — dark or light, and what a game hue reads as on each
scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
serve.ts static server + /api/health
test/ 724 tests
test/ 730 tests
fixtures/<game>/ raw HTML + .expected.json per source — pinned, kept forever
snapshots/ current page per source, rewritten by refresh — see its README
```
@@ -261,7 +261,15 @@ fixtures, that is a data-loss bug, not a diff to regenerate.
Sources are community wikis. Treat them as a guest would:
- Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
- One request per source per refresh cycle, minimum 6 hours apart.
- One request per source per refresh cycle, minimum 6 hours apart. **`--force` sets that floor
aside for one run, and only a person at a keyboard may pass it** — it is refused under CI, because
a schedule that forces every cycle is just a shorter interval with extra steps, and the interval is
the obligation. What makes it defensible is what it does *not* change: conditional headers still go
out, so a page that has not moved costs the host a `304` rather than a re-serve, and per-host
spacing, robots, the one-request-per-source rule and the no-retry rule all still apply. Prefer it
with `--only`: forcing nineteen sources to re-ask a question they answered an hour ago is the
behaviour this bullet exists to prevent, whatever flag authorised it. The run names every source it
asked early, and a run that was due anyway is never reported as forced.
- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Nine of the
nineteen sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive
as nine back-to-back requests to a single site — which is the shape an edge network throttles, and
@@ -568,9 +576,11 @@ Stella Sora takes no `resetOffsets` either, and for the opposite reason to most:
outright, and the offset is `-07:00` — US Pacific, which shifts by an hour twice a year. That is the
Fate/Grand Order problem arriving through a source that looks like it answered the question.
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first.
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
(failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
described in § Fandom). Both overrides are interactive-only, refused under CI, and reported by name
in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
never blank a calendar or stop the sources that did answer from being committed — so a failure is