feat(refresh): --force, to ask before the 6h floor is up

The interval gate has no override, so a page you know has just changed cannot
be fetched until six hours after the last attempt. The only workaround was
deleting snapshots/<id>.state.json, which also wipes consecutiveFailures and
lastConfirmedAt — resetting the broken-source streak and making the footer age
every source from when its bytes last changed rather than when we last
confirmed them. A flag that says what it means is better than a side effect
nobody documented.

It sets aside the interval and nothing else. Conditional headers still go out,
which is what makes forcing defensible at all: the host is asked, not
re-served, and an unchanged page costs it a 304. Per-host spacing, robots, one
request per source and the no-retry rule all still apply — a source that was
not due and is also disallowed stays skipped, for the reason that matters.

Refused under CI, like --assume-robots-on-403 and for the same reason: a
schedule that forces every cycle is a shorter interval with extra steps, and
the interval is the obligation, not the default. So AGENTS.md § Scraping
conduct is amended rather than left to be quietly contradicted by a flag.

Every source asked early is named in summary.forced and warned about. A run
that was due anyway is never reported as forced — a summary that cried "forced"
on an ordinary run would train the reader to ignore the word.

Also repoints the "unknown flag" test, which used --force as its example and
stopped testing anything the moment --force existed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

AGENTS.md

@@ -119,7 +119,7 @@ src/client/       React app, service worker, manifest
                   theme.ts — dark or light, and what a game hue reads as on each
 scripts/          build-feed.ts, build-static.ts, parse-fixture.ts (offline), refresh-sources.ts (fetches)
 serve.ts          static server + /api/health
-test/             724 tests
+test/             730 tests
 fixtures/<game>/  raw HTML + .expected.json per source — pinned, kept forever
 snapshots/        current page per source, rewritten by refresh — see its README
 ```
@@ -261,7 +261,15 @@ fixtures, that is a data-loss bug, not a diff to regenerate.
 Sources are community wikis. Treat them as a guest would:
 
 - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
-- One request per source per refresh cycle, minimum 6 hours apart.
+- One request per source per refresh cycle, minimum 6 hours apart. **`--force` sets that floor
+  aside for one run, and only a person at a keyboard may pass it** — it is refused under CI, because
+  a schedule that forces every cycle is just a shorter interval with extra steps, and the interval is
+  the obligation. What makes it defensible is what it does *not* change: conditional headers still go
+  out, so a page that has not moved costs the host a `304` rather than a re-serve, and per-host
+  spacing, robots, the one-request-per-source rule and the no-retry rule all still apply. Prefer it
+  with `--only`: forcing nineteen sources to re-ask a question they answered an hour ago is the
+  behaviour this bullet exists to prevent, whatever flag authorised it. The run names every source it
+  asked early, and a run that was due anyway is never reported as forced.
 - **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Nine of the
   nineteen sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive
   as nine back-to-back requests to a single site — which is the shape an edge network throttles, and
@@ -568,9 +576,11 @@ Stella Sora takes no `resetOffsets` either, and for the opposite reason to most:
 outright, and the offset is `-07:00` — US Pacific, which shifts by an hour twice a year. That is the
 Fate/Grand Order problem arriving through a source that looks like it answered the question.
 
-`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
+`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
-retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
+opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
-read, except under the opt-in `--assume-robots-on-403` described in § Fandom). Anything that would make it fetch more often is a change to this section first.
+(failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
+described in § Fandom). Both overrides are interactive-only, refused under CI, and reported by name
+in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
 
 **A source down is a warning; a source down for days is a broken build.** One wiki failing must
 never blank a calendar or stop the sources that did answer from being committed — so a failure is

docs/INGESTION.md

@@ -270,6 +270,13 @@ section but must never claim the event title.
 - Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that
   5xxs or times out means "do not fetch", because a permission we could not read is not a
   permission we have. A 404 means no restrictions.
+- **`--force` sets the 6h floor aside for one run**, and nothing else: conditional headers still go
+  out (so an unchanged page is a `304`, not a re-serve), per-host spacing still applies, robots still
+  decides, and there are still no retries. A source that was not due and is *also* disallowed stays
+  skipped, for the reason that actually matters. Refused under CI — a schedule that forces every
+  cycle is a shorter interval with extra steps. Every source asked early is listed in
+  `summary.forced` and warned about; a run that was due anyway is never reported as forced, because a
+  summary that cried "forced" on an ordinary run would train the reader to ignore the word.
 - **One narrow exception, opt-in per run: `--assume-robots-on-403`.** Fandom answers a datacentre
   address `403` on `/robots.txt` itself while `api.php?action=parse` answers our own User-Agent with
   a `200`, so the gate fails closed and four sources can never refresh — even though their rules are

scripts/refresh-sources.ts

@@ -5,6 +5,7 @@
  *   bun run refresh --dry-run                # plan only, no requests, no writes
  *   bun run refresh --only genshin-game8-events
  *   bun run refresh --assume-robots-on-403   # see § the flag, below
+ *   bun run refresh --force --only nikki-fandom-events   # ignore the 6h floor
  *
  * This is the scheduled half of the pipeline (docs/INGESTION.md stages 1-2).
  * The rules it enforces are etiquette obligations, not preferences:
@@ -19,7 +20,10 @@
  *   - at most ONE request per source per cycle, and never sooner than six hours
  *     after the last attempt. There is deliberately no retry: a retry is a
  *     second request, and the next cycle is minutes-cheap compared to being a
- *     bad guest.
+ *     bad guest. `--force` is the one way past the six hours, interactive-only
+ *     and refused under CI, and it sets aside the interval and nothing else —
+ *     conditional headers, per-host spacing, robots and the no-retry rule all
+ *     still apply, and every source asked early is named in the summary.
  *   - requests to one host are spaced, honouring its `Crawl-delay`. Eight of the
  *     twelve sources are game8.co pages, so without this one cycle is eight
  *     back-to-back requests to a single site — inside the per-source floor and
@@ -106,6 +110,11 @@ export interface RefreshSummary {
    * and on every CI run — the flag is refused there.
    */
   assumedRobots: string[];
+  /**
+   * Sources asked before their interval was up, under `--force`. Empty on every
+   * normal run, and on every CI run — the flag is refused there.
+   */
+  forced: string[];
 }
 
 export interface RobotsGate {
@@ -134,6 +143,12 @@ export interface RefreshOptions {
   sleep: (ms: number) => Promise<void>;
   dryRun: boolean;
   only: string | null;
+  /**
+   * Ignore the per-source interval floor for this run. Interactive only — see
+   * `--force` in USAGE, and AGENTS.md § Scraping conduct, which the flag amends
+   * rather than quietly contradicts.
+   */
+  force: boolean;
   timeoutMs: number;
   log: (line: string) => void;
   /** Called once when something changed. Null skips the rebuild (tests). */
@@ -163,6 +178,8 @@ interface Cycle {
   requestedHosts: Set<string>;
   /** Hosts fetched on `--assume-robots-on-403` rather than on a file we read. */
   assumedRobots: Set<string>;
+  /** Sources asked before their interval was up, under `--force`. */
+  forced: Set<string>;
 }
 
 export async function runRefresh(
@@ -177,9 +194,14 @@ export async function runRefresh(
     broken: [],
     hardFailure: null,
     assumedRobots: [],
+    forced: [],
   };
 
-  const cycle: Cycle = { requestedHosts: new Set(), assumedRobots: new Set() };
+  const cycle: Cycle = {
+    requestedHosts: new Set(),
+    assumedRobots: new Set(),
+    forced: new Set(),
+  };
 
   const selected =
     options.only === null
@@ -248,6 +270,14 @@ export async function runRefresh(
 
   // Named in the summary rather than only in the per-source log, so it survives
   // into the job summary and cannot be scrolled past.
+  summary.forced = [...cycle.forced].sort();
+  if (summary.forced.length > 0) {
+    summary.warnings.push(
+      `--force: asked ${summary.forced.length} source(s) before their interval ` +
+        `was up (${summary.forced.join(", ")})`,
+    );
+  }
+
   summary.assumedRobots = [...cycle.assumedRobots].sort();
   for (const host of summary.assumedRobots) {
     summary.warnings.push(
@@ -294,7 +324,8 @@ async function refreshOne(
   const state = await store.readState(adapter.id);
   const headers = store.conditionalHeaders(meta);
 
-  if (!store.isDue(state, now.getTime(), adapter.minIntervalMs)) {
+  const due = store.isDue(state, now.getTime(), adapter.minIntervalMs);
+  if (!due && !options.force) {
     const dueAt = new Date(store.dueAt(state, adapter.minIntervalMs));
     return {
       sourceId: adapter.id,
@@ -304,6 +335,10 @@ async function refreshOne(
       eventCount: meta?.eventCount ?? null,
     };
   }
+  // Asking early is the one obligation `--force` sets aside, and only for a
+  // source that would otherwise have been skipped — a run that was due anyway
+  // is an ordinary run and must not be reported as forced.
+  if (!due) cycle.forced.add(adapter.id);
 
   if (options.dryRun) {
     const conditional = Object.keys(headers);
@@ -663,6 +698,8 @@ interface Args {
   help: boolean;
   /** See `--assume-robots-on-403` in USAGE, and § Scraping conduct. */
   assumeRobotsOn403: boolean;
+  /** See `--force` in USAGE, and § Scraping conduct. */
+  force: boolean;
 }
 
 export function parseArgs(argv: readonly string[]): Args {
@@ -674,6 +711,7 @@ export function parseArgs(argv: readonly string[]): Args {
     rebuild: true,
     help: false,
     assumeRobotsOn403: false,
+    force: false,
   };
 
   // A flag whose value is missing is a mistake, never a default. `--only` with
@@ -711,6 +749,9 @@ export function parseArgs(argv: readonly string[]): Args {
       case "--assume-robots-on-403":
         args.assumeRobotsOn403 = true;
         break;
+      case "--force":
+        args.force = true;
+        break;
       case "--help":
       case "-h":
         args.help = true;
@@ -728,6 +769,7 @@ export function parseArgs(argv: readonly string[]): Args {
 
 const USAGE = `usage: bun run refresh [--dry-run] [--only <sourceId>] [--snapshots <dir>]
                        [--user-agent <ua>] [--no-feed] [--assume-robots-on-403]
+                       [--force]
 
   --dry-run       report what each source would do; no requests, no writes
   --only <id>     refresh a single source (${ADAPTERS.map((a) => a.id).join(", ")})
@@ -739,7 +781,12 @@ const USAGE = `usage: bun run refresh [--dry-run] [--only <sourceId>] [--snapsho
                   /robots.txt itself, proceed on the permission recorded in
                   AGENTS.md instead of failing closed. Refused under CI.
                   Does NOT override a robots.txt we could read: a file that
-                  disallows us still says no.`;
+                  disallows us still says no.
+  --force         temporary, interactive-only. Ignore the 6h per-source floor
+                  and ask now. Refused under CI. Everything else about being a
+                  guest still holds: one request per source, per-host spacing,
+                  conditional headers (so an unchanged page still costs a 304),
+                  robots, no retries. Prefer it with --only.`;
 
 async function main(): Promise<number> {
   let args: Args;
@@ -767,6 +814,19 @@ async function main(): Promise<number> {
   // asserting a permission nobody re-checked is how "temporary" becomes
   // permanent — so CI is refused the option outright rather than trusted not to
   // pass it. AGENTS.md § Scraping conduct is the argument.
+  // Same reasoning as the robots override: this is a person deciding, once,
+  // that a page has moved and they want it now. A schedule deciding that every
+  // run is just a shorter interval with extra steps, and the interval is the
+  // obligation.
+  if (args.force && isCi()) {
+    console.error(
+      "--force is interactive-only and refused under CI.\n" +
+        "The 6h floor is what the scheduled runner is for; change the schedule, " +
+        "not the floor.",
+    );
+    return 2;
+  }
+
   if (args.assumeRobotsOn403 && isCi()) {
     console.error(
       "--assume-robots-on-403 is interactive-only and refused under CI.\n" +
@@ -783,6 +843,14 @@ async function main(): Promise<number> {
     assumeAllowedWhenForbidden: args.assumeRobotsOn403,
   });
 
+  if (args.force) {
+    const n = args.only === null ? ADAPTERS.length : 1;
+    console.warn(
+      `  ! --force: ignoring the 6h floor for ${n} source${n === 1 ? "" : "s"}. ` +
+        `Conditional headers still apply, so an unchanged page costs a 304.`,
+    );
+  }
+
   if (args.assumeRobotsOn403) {
     console.warn(
       "  ! --assume-robots-on-403: a host answering 403 to /robots.txt will be\n" +
@@ -808,6 +876,7 @@ async function main(): Promise<number> {
     sleep: (ms) => Bun.sleep(ms),
     dryRun: args.dryRun,
     only: args.only,
+    force: args.force,
     timeoutMs: 20_000,
     log: (line) => console.log(line),
     rebuildFeed: args.dryRun || !args.rebuild ? null : rebuildFeedViaScript,

test/refresh.test.ts

@@ -100,6 +100,7 @@ function options(
     },
     dryRun: false,
     only: null,
+    force: false,
     timeoutMs: 1000,
     log: () => {},
     rebuildFeed: async () => {
@@ -202,6 +203,88 @@ describe("one request per source per six hours", () => {
     expect(summary.hardFailure).toBeNull();
   });
 
+  test("--force asks a source that was not due, and says which", async () => {
+    await store.recordCheck("genshin-game8-events", {
+      at: new Date(NOW.getTime() - 1000).toISOString(),
+      status: 200,
+      ok: true,
+    });
+
+    const { opts, calls } = options({ force: true });
+    const summary = await runRefresh(opts);
+
+    expect(calls).toHaveLength(1);
+    expect(summary.outcomes[0]?.result).not.toBe("skipped_interval");
+    // Named, not merely permitted. Overriding an etiquette obligation quietly
+    // is how the obligation stops being one.
+    expect(summary.forced).toEqual(["genshin-game8-events"]);
+    expect(summary.warnings.some((w) => w.includes("--force"))).toBe(true);
+  });
+
+  test("--force still sends conditional headers, so an unchanged page is a 304", async () => {
+    // The whole reason forcing is defensible: the host is asked, not re-served.
+    await seed("<html><event></event></html>", NOW.toISOString(), 1);
+    await store.recordCheck("genshin-game8-events", {
+      at: new Date(NOW.getTime() - 1000).toISOString(),
+      status: 200,
+      ok: true,
+    });
+
+    const { opts, calls } = options({
+      force: true,
+      responder: () => new Response(null, { status: 304 }),
+    });
+    const summary = await runRefresh(opts);
+
+    expect(calls[0]?.headers["If-None-Match"]).toBe('W/"v1"');
+    expect(summary.outcomes[0]?.result).toBe("unchanged");
+  });
+
+  test("--force sets aside the interval and nothing else", async () => {
+    // robots is the gate it must never touch. A source that was not due AND is
+    // disallowed stays skipped for the reason that actually matters.
+    await store.recordCheck("genshin-game8-events", {
+      at: new Date(NOW.getTime() - 1000).toISOString(),
+      status: 200,
+      ok: true,
+    });
+
+    const { opts, calls } = options({
+      force: true,
+      robots: {
+        allows: async () => ({ allowed: false, reason: "disallowed by robots" }),
+      },
+    });
+    const summary = await runRefresh(opts);
+
+    expect(calls).toHaveLength(0);
+    expect(summary.outcomes[0]?.result).toBe("skipped_robots");
+  });
+
+  test("a run that was due anyway is not reported as forced", async () => {
+    // --force is a description of what happened, not of what was passed. A
+    // summary that cried "forced" on an ordinary run would train the reader to
+    // ignore the word.
+    await store.recordCheck("genshin-game8-events", {
+      at: new Date(NOW.getTime() - SIX_HOURS_MS).toISOString(),
+      status: 200,
+      ok: true,
+    });
+
+    const { opts, calls } = options({ force: true });
+    const summary = await runRefresh(opts);
+
+    expect(calls).toHaveLength(1);
+    expect(summary.forced).toEqual([]);
+    expect(summary.warnings).toEqual([]);
+  });
+
+  test("an ordinary run reports nothing forced", async () => {
+    const { opts } = options({});
+    const summary = await runRefresh(opts);
+    expect(summary.forced).toEqual([]);
+  });
+
   test("fetches again once the interval has elapsed", async () => {
     await store.recordCheck("genshin-game8-events", {
       at: new Date(NOW.getTime() - SIX_HOURS_MS).toISOString(),
@@ -798,6 +881,7 @@ describe("what the runner reports to the runner", () => {
     ],
     hardFailure: null,
     assumedRobots: [],
+    forced: [],
   };
 
   test("a broken source becomes an annotation on the run page", () => {
@@ -962,8 +1046,15 @@ describe("flags", () => {
     expect(parseArgs(["--assume-robots-on-403"]).assumeRobotsOn403).toBe(true);
   });
 
+  test("parseArgs reads --force, and it is off by default", () => {
+    expect(parseArgs([]).force).toBe(false);
+    expect(parseArgs(["--force"]).force).toBe(true);
+  });
+
   test("parseArgs rejects an unknown flag rather than ignoring it", () => {
-    expect(() => parseArgs(["--force"])).toThrow("unknown flag");
+    // Deliberately a flag nobody would add. This case used to be spelled
+    // `--force`, which stopped testing anything the day --force was built.
+    expect(() => parseArgs(["--yolo"])).toThrow("unknown flag");
   });
 
   test("parseArgs rejects a flag whose value is missing", () => {