From 6552528acb385af791f3f7cd3265c0e3e7d34174 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Wed, 19 Aug 2026 04:59:52 +0200 Subject: [PATCH] feat(refresh): --force, to ask before the 6h floor is up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The interval gate has no override, so a page you know has just changed cannot be fetched until six hours after the last attempt. The only workaround was deleting snapshots/.state.json, which also wipes consecutiveFailures and lastConfirmedAt — resetting the broken-source streak and making the footer age every source from when its bytes last changed rather than when we last confirmed them. A flag that says what it means is better than a side effect nobody documented. It sets aside the interval and nothing else. Conditional headers still go out, which is what makes forcing defensible at all: the host is asked, not re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one request per source and the no-retry rule all still apply — a source that was not due and is also disallowed stays skipped, for the reason that matters. Refused under CI, like --assume-robots-on-403 and for the same reason: a schedule that forces every cycle is a shorter interval with extra steps, and the interval is the obligation, not the default. So AGENTS.md § Scraping conduct is amended rather than left to be quietly contradicted by a flag. Every source asked early is named in summary.forced and warned about. A run that was due anyway is never reported as forced — a summary that cried "forced" on an ordinary run would train the reader to ignore the word. Also repoints the "unknown flag" test, which used --force as its example and stopped testing anything the moment --force existed. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 20 ++++++-- docs/INGESTION.md | 7 +++ scripts/refresh-sources.ts | 77 +++++++++++++++++++++++++++++-- test/refresh.test.ts | 93 +++++++++++++++++++++++++++++++++++++- 4 files changed, 187 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5a9963a..62d2119 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -119,7 +119,7 @@ src/client/ React app, service worker, manifest theme.ts — dark or light, and what a game hue reads as on each scripts/ build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches) serve.ts static server + /api/health -test/ 724 tests +test/ 730 tests fixtures// raw HTML + .expected.json per source — pinned, kept forever snapshots/ current page per source, rewritten by refresh — see its README ``` @@ -261,7 +261,15 @@ fixtures, that is a data-loss bug, not a diff to regenerate. Sources are community wikis. Treat them as a guest would: - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL. -- One request per source per refresh cycle, minimum 6 hours apart. +- One request per source per refresh cycle, minimum 6 hours apart. **`--force` sets that floor + aside for one run, and only a person at a keyboard may pass it** — it is refused under CI, because + a schedule that forces every cycle is just a shorter interval with extra steps, and the interval is + the obligation. What makes it defensible is what it does *not* change: conditional headers still go + out, so a page that has not moved costs the host a `304` rather than a re-serve, and per-host + spacing, robots, the one-request-per-source rule and the no-retry rule all still apply. Prefer it + with `--only`: forcing nineteen sources to re-ask a question they answered an hour ago is the + behaviour this bullet exists to prevent, whatever flag authorised it. The run names every source it + asked early, and a run that was due anyway is never reported as forced. - **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Nine of the nineteen sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as nine back-to-back requests to a single site — which is the shape an edge network throttles, and @@ -568,9 +576,11 @@ Stella Sora takes no `resetOffsets` either, and for the opposite reason to most: outright, and the offset is `-07:00` — US Pacific, which shifts by an hour twice a year. That is the Fate/Grand Order problem arriving through a source that looks like it answered the question. -`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no -retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be -read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first. +`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the +opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots +(failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403` +described in § Fandom). Both overrides are interactive-only, refused under CI, and reported by name +in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first. **A source down is a warning; a source down for days is a broken build.** One wiki failing must never blank a calendar or stop the sources that did answer from being committed — so a failure is diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 906ff56..a2f72c4 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -270,6 +270,13 @@ section but must never claim the event title. - Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that 5xxs or times out means "do not fetch", because a permission we could not read is not a permission we have. A 404 means no restrictions. +- **`--force` sets the 6h floor aside for one run**, and nothing else: conditional headers still go + out (so an unchanged page is a `304`, not a re-serve), per-host spacing still applies, robots still + decides, and there are still no retries. A source that was not due and is *also* disallowed stays + skipped, for the reason that actually matters. Refused under CI — a schedule that forces every + cycle is a shorter interval with extra steps. Every source asked early is listed in + `summary.forced` and warned about; a run that was due anyway is never reported as forced, because a + summary that cried "forced" on an ordinary run would train the reader to ignore the word. - **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with a `200`, so the gate fails closed and four sources can never refresh — even though their rules are diff --git a/scripts/refresh-sources.ts b/scripts/refresh-sources.ts index 1c43d43..bb786b8 100644 --- a/scripts/refresh-sources.ts +++ b/scripts/refresh-sources.ts @@ -5,6 +5,7 @@ * bun run refresh --dry-run # plan only, no requests, no writes * bun run refresh --only genshin-game8-events * bun run refresh --assume-robots-on-403 # see § the flag, below + * bun run refresh --force --only nikki-fandom-events # ignore the 6h floor * * This is the scheduled half of the pipeline (docs/INGESTION.md stages 1-2). * The rules it enforces are etiquette obligations, not preferences: @@ -19,7 +20,10 @@ * - at most ONE request per source per cycle, and never sooner than six hours * after the last attempt. There is deliberately no retry: a retry is a * second request, and the next cycle is minutes-cheap compared to being a - * bad guest. + * bad guest. `--force` is the one way past the six hours, interactive-only + * and refused under CI, and it sets aside the interval and nothing else — + * conditional headers, per-host spacing, robots and the no-retry rule all + * still apply, and every source asked early is named in the summary. * - requests to one host are spaced, honouring its `Crawl-delay`. Eight of the * twelve sources are game8.co pages, so without this one cycle is eight * back-to-back requests to a single site — inside the per-source floor and @@ -106,6 +110,11 @@ export interface RefreshSummary { * and on every CI run — the flag is refused there. */ assumedRobots: string[]; + /** + * Sources asked before their interval was up, under `--force`. Empty on every + * normal run, and on every CI run — the flag is refused there. + */ + forced: string[]; } export interface RobotsGate { @@ -134,6 +143,12 @@ export interface RefreshOptions { sleep: (ms: number) => Promise; dryRun: boolean; only: string | null; + /** + * Ignore the per-source interval floor for this run. Interactive only — see + * `--force` in USAGE, and AGENTS.md § Scraping conduct, which the flag amends + * rather than quietly contradicts. + */ + force: boolean; timeoutMs: number; log: (line: string) => void; /** Called once when something changed. Null skips the rebuild (tests). */ @@ -163,6 +178,8 @@ interface Cycle { requestedHosts: Set; /** Hosts fetched on `--assume-robots-on-403` rather than on a file we read. */ assumedRobots: Set; + /** Sources asked before their interval was up, under `--force`. */ + forced: Set; } export async function runRefresh( @@ -177,9 +194,14 @@ export async function runRefresh( broken: [], hardFailure: null, assumedRobots: [], + forced: [], }; - const cycle: Cycle = { requestedHosts: new Set(), assumedRobots: new Set() }; + const cycle: Cycle = { + requestedHosts: new Set(), + assumedRobots: new Set(), + forced: new Set(), + }; const selected = options.only === null @@ -248,6 +270,14 @@ export async function runRefresh( // Named in the summary rather than only in the per-source log, so it survives // into the job summary and cannot be scrolled past. + summary.forced = [...cycle.forced].sort(); + if (summary.forced.length > 0) { + summary.warnings.push( + `--force: asked ${summary.forced.length} source(s) before their interval ` + + `was up (${summary.forced.join(", ")})`, + ); + } + summary.assumedRobots = [...cycle.assumedRobots].sort(); for (const host of summary.assumedRobots) { summary.warnings.push( @@ -294,7 +324,8 @@ async function refreshOne( const state = await store.readState(adapter.id); const headers = store.conditionalHeaders(meta); - if (!store.isDue(state, now.getTime(), adapter.minIntervalMs)) { + const due = store.isDue(state, now.getTime(), adapter.minIntervalMs); + if (!due && !options.force) { const dueAt = new Date(store.dueAt(state, adapter.minIntervalMs)); return { sourceId: adapter.id, @@ -304,6 +335,10 @@ async function refreshOne( eventCount: meta?.eventCount ?? null, }; } + // Asking early is the one obligation `--force` sets aside, and only for a + // source that would otherwise have been skipped — a run that was due anyway + // is an ordinary run and must not be reported as forced. + if (!due) cycle.forced.add(adapter.id); if (options.dryRun) { const conditional = Object.keys(headers); @@ -663,6 +698,8 @@ interface Args { help: boolean; /** See `--assume-robots-on-403` in USAGE, and § Scraping conduct. */ assumeRobotsOn403: boolean; + /** See `--force` in USAGE, and § Scraping conduct. */ + force: boolean; } export function parseArgs(argv: readonly string[]): Args { @@ -674,6 +711,7 @@ export function parseArgs(argv: readonly string[]): Args { rebuild: true, help: false, assumeRobotsOn403: false, + force: false, }; // A flag whose value is missing is a mistake, never a default. `--only` with @@ -711,6 +749,9 @@ export function parseArgs(argv: readonly string[]): Args { case "--assume-robots-on-403": args.assumeRobotsOn403 = true; break; + case "--force": + args.force = true; + break; case "--help": case "-h": args.help = true; @@ -728,6 +769,7 @@ export function parseArgs(argv: readonly string[]): Args { const USAGE = `usage: bun run refresh [--dry-run] [--only ] [--snapshots ] [--user-agent ] [--no-feed] [--assume-robots-on-403] + [--force] --dry-run report what each source would do; no requests, no writes --only refresh a single source (${ADAPTERS.map((a) => a.id).join(", ")}) @@ -739,7 +781,12 @@ const USAGE = `usage: bun run refresh [--dry-run] [--only ] [--snapsho /robots.txt itself, proceed on the permission recorded in AGENTS.md instead of failing closed. Refused under CI. Does NOT override a robots.txt we could read: a file that - disallows us still says no.`; + disallows us still says no. + --force temporary, interactive-only. Ignore the 6h per-source floor + and ask now. Refused under CI. Everything else about being a + guest still holds: one request per source, per-host spacing, + conditional headers (so an unchanged page still costs a 304), + robots, no retries. Prefer it with --only.`; async function main(): Promise { let args: Args; @@ -767,6 +814,19 @@ async function main(): Promise { // asserting a permission nobody re-checked is how "temporary" becomes // permanent — so CI is refused the option outright rather than trusted not to // pass it. AGENTS.md § Scraping conduct is the argument. + // Same reasoning as the robots override: this is a person deciding, once, + // that a page has moved and they want it now. A schedule deciding that every + // run is just a shorter interval with extra steps, and the interval is the + // obligation. + if (args.force && isCi()) { + console.error( + "--force is interactive-only and refused under CI.\n" + + "The 6h floor is what the scheduled runner is for; change the schedule, " + + "not the floor.", + ); + return 2; + } + if (args.assumeRobotsOn403 && isCi()) { console.error( "--assume-robots-on-403 is interactive-only and refused under CI.\n" + @@ -783,6 +843,14 @@ async function main(): Promise { assumeAllowedWhenForbidden: args.assumeRobotsOn403, }); + if (args.force) { + const n = args.only === null ? ADAPTERS.length : 1; + console.warn( + ` ! --force: ignoring the 6h floor for ${n} source${n === 1 ? "" : "s"}. ` + + `Conditional headers still apply, so an unchanged page costs a 304.`, + ); + } + if (args.assumeRobotsOn403) { console.warn( " ! --assume-robots-on-403: a host answering 403 to /robots.txt will be\n" + @@ -808,6 +876,7 @@ async function main(): Promise { sleep: (ms) => Bun.sleep(ms), dryRun: args.dryRun, only: args.only, + force: args.force, timeoutMs: 20_000, log: (line) => console.log(line), rebuildFeed: args.dryRun || !args.rebuild ? null : rebuildFeedViaScript, diff --git a/test/refresh.test.ts b/test/refresh.test.ts index e4902e6..6591919 100644 --- a/test/refresh.test.ts +++ b/test/refresh.test.ts @@ -100,6 +100,7 @@ function options( }, dryRun: false, only: null, + force: false, timeoutMs: 1000, log: () => {}, rebuildFeed: async () => { @@ -202,6 +203,88 @@ describe("one request per source per six hours", () => { expect(summary.hardFailure).toBeNull(); }); + test("--force asks a source that was not due, and says which", async () => { + await store.recordCheck("genshin-game8-events", { + at: new Date(NOW.getTime() - 1000).toISOString(), + status: 200, + ok: true, + }); + + const { opts, calls } = options({ force: true }); + const summary = await runRefresh(opts); + + expect(calls).toHaveLength(1); + expect(summary.outcomes[0]?.result).not.toBe("skipped_interval"); + // Named, not merely permitted. Overriding an etiquette obligation quietly + // is how the obligation stops being one. + expect(summary.forced).toEqual(["genshin-game8-events"]); + expect(summary.warnings.some((w) => w.includes("--force"))).toBe(true); + }); + + test("--force still sends conditional headers, so an unchanged page is a 304", async () => { + // The whole reason forcing is defensible: the host is asked, not re-served. + await seed("", NOW.toISOString(), 1); + await store.recordCheck("genshin-game8-events", { + at: new Date(NOW.getTime() - 1000).toISOString(), + status: 200, + ok: true, + }); + + const { opts, calls } = options({ + force: true, + responder: () => new Response(null, { status: 304 }), + }); + const summary = await runRefresh(opts); + + expect(calls[0]?.headers["If-None-Match"]).toBe('W/"v1"'); + expect(summary.outcomes[0]?.result).toBe("unchanged"); + }); + + test("--force sets aside the interval and nothing else", async () => { + // robots is the gate it must never touch. A source that was not due AND is + // disallowed stays skipped for the reason that actually matters. + await store.recordCheck("genshin-game8-events", { + at: new Date(NOW.getTime() - 1000).toISOString(), + status: 200, + ok: true, + }); + + const { opts, calls } = options({ + force: true, + robots: { + allows: async () => ({ allowed: false, reason: "disallowed by robots" }), + }, + }); + const summary = await runRefresh(opts); + + expect(calls).toHaveLength(0); + expect(summary.outcomes[0]?.result).toBe("skipped_robots"); + }); + + test("a run that was due anyway is not reported as forced", async () => { + // --force is a description of what happened, not of what was passed. A + // summary that cried "forced" on an ordinary run would train the reader to + // ignore the word. + await store.recordCheck("genshin-game8-events", { + at: new Date(NOW.getTime() - SIX_HOURS_MS).toISOString(), + status: 200, + ok: true, + }); + + const { opts, calls } = options({ force: true }); + const summary = await runRefresh(opts); + + expect(calls).toHaveLength(1); + expect(summary.forced).toEqual([]); + expect(summary.warnings).toEqual([]); + }); + + test("an ordinary run reports nothing forced", async () => { + const { opts } = options({}); + const summary = await runRefresh(opts); + expect(summary.forced).toEqual([]); + }); + test("fetches again once the interval has elapsed", async () => { await store.recordCheck("genshin-game8-events", { at: new Date(NOW.getTime() - SIX_HOURS_MS).toISOString(), @@ -798,6 +881,7 @@ describe("what the runner reports to the runner", () => { ], hardFailure: null, assumedRobots: [], + forced: [], }; test("a broken source becomes an annotation on the run page", () => { @@ -962,8 +1046,15 @@ describe("flags", () => { expect(parseArgs(["--assume-robots-on-403"]).assumeRobotsOn403).toBe(true); }); + test("parseArgs reads --force, and it is off by default", () => { + expect(parseArgs([]).force).toBe(false); + expect(parseArgs(["--force"]).force).toBe(true); + }); + test("parseArgs rejects an unknown flag rather than ignoring it", () => { - expect(() => parseArgs(["--force"])).toThrow("unknown flag"); + // Deliberately a flag nobody would add. This case used to be spelled + // `--force`, which stopped testing anything the day --force was built. + expect(() => parseArgs(["--yolo"])).toThrow("unknown flag"); }); test("parseArgs rejects a flag whose value is missing", () => {