feat(refresh): --force, to ask before the 6h floor is up

The interval gate has no override, so a page you know has just changed cannot
be fetched until six hours after the last attempt. The only workaround was
deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and
lastConfirmedAt — resetting the broken-source streak and making the footer age
every source from when its bytes last changed rather than when we last
confirmed them. A flag that says what it means is better than a side effect
nobody documented.

It sets aside the interval and nothing else. Conditional headers still go out,
which is what makes forcing defensible at all: the host is asked, not
re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one
request per source and the no-retry rule all still apply — a source that was
not due and is also disallowed stays skipped, for the reason that matters.

Refused under CI, like --assume-robots-on-403 and for the same reason: a
schedule that forces every cycle is a shorter interval with extra steps, and
the interval is the obligation, not the default. So AGENTS.md § Scraping
conduct is amended rather than left to be quietly contradicted by a flag.

Every source asked early is named in summary.forced and warned about. A run
that was due anyway is never reported as forced — a summary that cried "forced"
on an ordinary run would train the reader to ignore the word.

Also repoints the "unknown flag" test, which used --force as its example and
stopped testing anything the moment --force existed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-19 04:59:52 +02:00
co-authored by Claude Opus 5
parent 240e4d59d1
commit 6552528acb
4 changed files with 187 additions and 10 deletions
+7
View File
@@ -270,6 +270,13 @@ section but must never claim the event title.
- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
5xxs or times out means "do not fetch", because a permission we could not read is not a
permission we have. A 404 means no restrictions.
- **`--force` sets the 6h floor aside for one run**, and nothing else: conditional headers still go
out (so an unchanged page is a `304`, not a re-serve), per-host spacing still applies, robots still
decides, and there are still no retries. A source that was not due and is *also* disallowed stays
skipped, for the reason that actually matters. Refused under CI — a schedule that forces every
cycle is a shorter interval with extra steps. Every source asked early is listed in
`summary.forced` and warned about; a run that was due anyway is never reported as forced, because a
summary that cried "forced" on an ordinary run would train the reader to ignore the word.
- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre
address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with
a `200`, so the gate fails closed and four sources can never refresh — even though their rules are