feat(refresh): --force, to ask before the 6h floor is up
The interval gate has no override, so a page you know has just changed cannot be fetched until six hours after the last attempt. The only workaround was deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and lastConfirmedAt — resetting the broken-source streak and making the footer age every source from when its bytes last changed rather than when we last confirmed them. A flag that says what it means is better than a side effect nobody documented. It sets aside the interval and nothing else. Conditional headers still go out, which is what makes forcing defensible at all: the host is asked, not re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one request per source and the no-retry rule all still apply — a source that was not due and is also disallowed stays skipped, for the reason that matters. Refused under CI, like --assume-robots-on-403 and for the same reason: a schedule that forces every cycle is a shorter interval with extra steps, and the interval is the obligation, not the default. So AGENTS.md § Scraping conduct is amended rather than left to be quietly contradicted by a flag. Every source asked early is named in summary.forced and warned about. A run that was due anyway is never reported as forced — a summary that cried "forced" on an ordinary run would train the reader to ignore the word. Also repoints the "unknown flag" test, which used --force as its example and stopped testing anything the moment --force existed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
240e4d59d1
commit
6552528acb
@@ -270,6 +270,13 @@ section but must never claim the event title.
|
||||
- Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
|
||||
5xxs or times out means "do not fetch", because a permission we could not read is not a
|
||||
permission we have. A 404 means no restrictions.
|
||||
- **`--force` sets the 6h floor aside for one run**, and nothing else: conditional headers still go
|
||||
out (so an unchanged page is a `304`, not a re-serve), per-host spacing still applies, robots still
|
||||
decides, and there are still no retries. A source that was not due and is *also* disallowed stays
|
||||
skipped, for the reason that actually matters. Refused under CI — a schedule that forces every
|
||||
cycle is a shorter interval with extra steps. Every source asked early is listed in
|
||||
`summary.forced` and warned about; a run that was due anyway is never reported as forced, because a
|
||||
summary that cried "forced" on an ordinary run would train the reader to ignore the word.
|
||||
- **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre
|
||||
address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with
|
||||
a `200`, so the gate fails closed and four sources can never refresh — even though their rules are
|
||||
|
||||
Reference in New Issue
Block a user