fix(refresh): make a source that stopped answering turn the run red

The only snapshot the scheduled refresh has ever committed is endfield-wikigg-events. Seven sources existed at that run; the six game8.co ones yielded nothing, and no cycle since has committed anything. All of them fetch fine from a laptop, so whatever is happening happens on the runner — meanwhile eight of ten games were served from checked-in fixtures for three days behind a green tick. Nine failures out of ten was exit 0 with warnings buried in a log nobody opens. `consecutiveFailures` was already tracked and never read. A source that has failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported as `broken`: a GitHub annotation, a job-summary row carrying its status code, and a `broken` step output. The runner still exits 0 on it and `refresh.yml` fails on that output in a final step, after the commit and the CI dispatch — exiting non-zero from the runner would skip the commit and throw away the pages that did arrive, which is the opposite of what "one wiki down never blanks a calendar" is for. The streak is read from the store rather than from this cycle's outcome, so a source dead for days that happens to be inside its six-hour window has not recovered. A non-ok response now records what turned us away — the Server header, whether a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads identically whether the page moved behind a login or a CDN decided the runner is a bot farm, and that is the open question here. Values are trimmed and capped: the note lands in a workflow command and a markdown cell, and it came from a host we do not control. Also space requests to a host already asked this cycle, honouring its Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the per-source floor alone still permitted one cycle to arrive as eight back-to-back requests to a single site — which is what a burst looks like from the far end regardless of our intent, and is plausibly self-inflicted here. The wait is taken after the interval and robots gates, so a source we then skip costs nothing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-17 21:56:01 +02:00
parent d2615c606c
commit 2a0ea1a796
5 changed files with 564 additions and 15 deletions
@@ -71,6 +71,7 @@ jobs:
          restore-keys: refresh-state-

      - name: Refresh
+        id: refresh
        env:
          # Identify the crawler with a contact URL, per CLAUDE.md.
          REFRESH_CONTACT_URL: ${{ github.server_url }}/${{ github.repository }}
@@ -155,3 +156,28 @@ jobs:
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: gh workflow run ci.yml --ref "${{ github.ref_name }}"
+
+      # A source that has failed three cycles running is broken, not down: that
+      # game's calendar has been built from a checked-in fixture for a day and a
+      # half while every run showed a green tick. `bun run refresh` exits 0 on
+      # this so the steps above still commit and publish what did work; turning
+      # the run red is this step's job, and it is last for that reason.
+      #
+      # `always()` so it still reports when an earlier step failed — but note it
+      # cannot report when the Refresh step itself hard-failed, since the output
+      # is then unset and the job is already red on its own account.
+      - name: Report source health
+        if: always()
+        env:
+          BROKEN: ${{ steps.refresh.outputs.broken }}
+          REFRESH_OUTCOME: ${{ steps.refresh.outcome }}
+        run: |
+          if [ "$REFRESH_OUTCOME" != "success" ]; then
+            echo "refresh did not complete ($REFRESH_OUTCOME); no health to report"
+            exit 0
+          fi
+          if [ -n "$BROKEN" ] && [ "$BROKEN" != "0" ]; then
+            echo "::error::$BROKEN source(s) have stopped answering; see the job summary"
+            exit 1
+          fi
+          echo "every source is answering"
@@ -180,6 +180,10 @@ Sources are community wikis. Treat them as a guest would:

 - Honor `robots.txt`; set a descriptive `User-Agent` with a contact URL.
 - One request per source per refresh cycle, minimum 6 hours apart.
+- **Space requests to one host**, honouring its `Crawl-delay` and defaulting to 2s. Eight of the ten
+  sources are game8.co pages, so the per-source floor alone still permits one cycle to arrive as
+  eight back-to-back requests to a single site — which is the shape an edge network throttles, and
+  what a burst looks like from the far end regardless of our intent.
 - Send `If-None-Match` / `If-Modified-Since`; treat `304` as "skip, unchanged".
 - Cache raw snapshots so re-parsing never re-fetches. **Iterate against fixtures, not the network.**
 - Record `sourceUrl` on every event and surface attribution in the UI.
@@ -207,8 +211,19 @@ load-bearing here and not only a cost decision. Note also that Reverse: 1999, Bl
 Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401.

 `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no
-retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything
-that would make it fetch more often is a change to this section first.
+retries, conditional headers, per-host spacing, robots (failing closed when `robots.txt` cannot be
+read). Anything that would make it fetch more often is a change to this section first.
+
+**A source down is a warning; a source down for days is a broken build.** One wiki failing must
+never blank a calendar or stop the sources that did answer from being committed — so a failure is
+exit 0 and the previous snapshot stands. But a source that has failed `BROKEN_AFTER_FAILURES` (3)
+cycles running is not having a bad afternoon: that game's calendar has been quietly built from a
+checked-in fixture for a day and a half. The runner reports those as `broken` — a GitHub annotation,
+a row in the job summary with the status code, and a `broken` step output — and `refresh.yml` fails
+the run on it in a **final** step, after the commit and the CI dispatch. Exiting non-zero from the
+runner instead would skip the commit and throw away the pages that did arrive. This tier exists
+because six of seven sources failed every cycle for three days behind a green tick; a warning nobody
+opens the log to read is not a signal.

 ## Untrusted input

@@ -177,10 +177,21 @@ section but must never claim the event title.
  permission we have. A 404 means no restrictions.
 - 20s timeout. **No retries**: a retry is a second request, and CLAUDE.md § Scraping conduct says
  one per source per cycle. A failed source waits for the next cycle instead.
+- **Space requests to a host we have already asked this cycle** — the host's `Crawl-delay` if it
+  states one, else `DEFAULT_HOST_GAP_MS` (2s). The wait is taken after the interval and robots gates,
+  so a source we then skip costs nothing.
 - Store raw bytes in `snapshots/<source-id>.html`, with hash/ETag/Last-Modified alongside it.

 On failure: increment the failure streak, leave published events untouched, end as `failed`. A
-source being down never mutates the feed.
+source being down never mutates the feed. A non-`ok` status also records what turned us away — the
+`Server` header, whether a `CF-Ray` was present, any `Retry-After` — because a bare `HTTP 403` reads
+identically whether the page moved behind a login or a CDN decided the runner is a bot farm.
+
+**The failure streak is read, not just written.** `consecutiveFailures` reaching
+`BROKEN_AFTER_FAILURES` (3, so ~36h at two cycles a day) promotes a source from "down" to `broken`:
+annotated on the run page, listed in the job summary with its status code, and counted in the
+`broken` step output that `refresh.yml` fails on *after* committing. See CLAUDE.md § Scraping
+conduct for why that ordering is load-bearing.

 **Built: `scripts/refresh-sources.ts`** (`bun run refresh`), scheduled by
 `.github/workflows/refresh.yml`. It takes its adapters, store, robots gate, fetch and clock by
@@ -14,14 +14,27 @@
 *     after the last attempt. There is deliberately no retry: a retry is a
 *     second request, and the next cycle is minutes-cheap compared to being a
 *     bad guest.
+ *   - requests to one host are spaced, honouring its `Crawl-delay`. Eight of the
+ *     ten sources are game8.co pages, so without this one cycle is eight
+ *     back-to-back requests to a single site — inside the per-source floor and
+ *     still the behaviour an edge network throttles.
 *   - conditional requests always, so an unchanged page costs the wiki a 304.
 *   - a descriptive User-Agent carrying a contact URL.
 *
- * Failure policy: one wiki being down is a warning. The previous snapshot stays
- * in place and the feed keeps its events — a source outage must never blank the
- * calendar. Hard failures (bad arguments, an unwritable cache, a feed that will
- * not rebuild) exit non-zero so CI stops before committing anything.
+ * Failure policy has two tiers, because they want opposite things from CI:
+ *
+ *   - One wiki being down is a warning and exit 0. The previous snapshot stays
+ *     in place and the feed keeps its events — a source outage must never blank
+ *     the calendar, and the sources that did answer must still be committed.
+ *   - A source failing `BROKEN_AFTER_FAILURES` cycles in a row is not "down",
+ *     it is broken, and the feed has been quietly serving a stale fixture for a
+ *     day and a half. That is reported as `broken` and annotated, and the
+ *     workflow turns the run red *after* committing what did work. Exiting
+ *     non-zero here instead would skip the commit and throw the good pages away.
+ *   - Hard failures (bad arguments, every source failing, a feed that will not
+ *     rebuild) exit non-zero so CI stops before committing anything.
 */
+import { appendFile } from "node:fs/promises";
 import {
  ADAPTERS,
  adapterById,
@@ -54,6 +67,17 @@ export interface SourceOutcome {
  eventCount: number | null;
 }

+/**
+ * A source that has stopped answering for long enough that the feed is now
+ * knowingly stale for that game.
+ */
+export interface BrokenSource {
+  sourceId: string;
+  consecutiveFailures: number;
+  lastStatus: number | null;
+  lastConfirmedAt: string | null;
+}
+
 export interface RefreshSummary {
  outcomes: SourceOutcome[];
  /** Sources whose stored bytes changed — the only reason to commit. */
@@ -63,12 +87,23 @@ export interface RefreshSummary {
  /** Sources that answered (200 or 304). */
  confirmed: number;
  warnings: string[];
+  /**
+   * Sources failing for `BROKEN_AFTER_FAILURES` cycles running. Not a hard
+   * failure — the workflow reports it after the commit, so a broken source
+   * cannot cost a working one its snapshot.
+   */
+  broken: BrokenSource[];
  /** Set when the run should exit non-zero. */
  hardFailure: string | null;
 }

 export interface RobotsGate {
-  allows(url: string): Promise<{ allowed: boolean; reason: string }>;
+  allows(url: string): Promise<{
+    allowed: boolean;
+    reason: string;
+    /** From the host's `Crawl-delay`, when it states one. */
+    crawlDelayMs?: number | null;
+  }>;
 }

 export interface RefreshOptions {
@@ -79,6 +114,11 @@ export interface RefreshOptions {
  userAgent: string;
  /** Injected clock — the runner is testable, like the parsers it drives. */
  now: () => Date;
+  /**
+   * Injected timer, for the same reason as the clock: the per-host gap is real
+   * seconds on a runner and must cost a test nothing.
+   */
+  sleep: (ms: number) => Promise<void>;
  dryRun: boolean;
  only: string | null;
  timeoutMs: number;
@@ -90,6 +130,26 @@ export interface RefreshOptions {
 /** A drop this steep means the page changed shape, not that events ended. */
 const DROP_WARNING_RATIO = 0.5;

+/**
+ * Cycles of failure that separate "the wiki is down" from "this source is
+ * broken". At two cycles a day, three is a day and a half of a game's calendar
+ * silently coming from a checked-in fixture — long enough to be certain, short
+ * enough to still be worth hearing about.
+ */
+export const BROKEN_AFTER_FAILURES = 3;
+
+/**
+ * Gap between two requests to the same host when its robots.txt names none.
+ * The per-source floor is six hours, but eight sources share game8.co, so
+ * without this they arrive as one burst.
+ */
+export const DEFAULT_HOST_GAP_MS = 2_000;
+
+/** Per-cycle state shared across sources: which hosts we have already asked. */
+interface Cycle {
+  requestedHosts: Set<string>;
+}
+
 export async function runRefresh(
  options: RefreshOptions,
 ): Promise<RefreshSummary> {
@@ -99,9 +159,12 @@ export async function runRefresh(
    attempted: 0,
    confirmed: 0,
    warnings: [],
+    broken: [],
    hardFailure: null,
  };

+  const cycle: Cycle = { requestedHosts: new Set() };
+
  const selected =
    options.only === null
      ? [...options.adapters]
@@ -120,7 +183,7 @@ export async function runRefresh(
    // no summary and no record of what was already asked.
    let outcome: SourceOutcome;
    try {
-      outcome = await refreshOne(adapter, options);
+      outcome = await refreshOne(adapter, options, cycle);
    } catch (error) {
      outcome = {
        sourceId: adapter.id,
@@ -149,6 +212,22 @@ export async function runRefresh(
    options.log(
      `  ${adapter.id.padEnd(24)} ${outcome.result.padEnd(17)} ${outcome.note}`,
    );
+
+    // Read from the store rather than from this cycle's outcome: the streak is
+    // the point, and a source that has failed for days and is now skipped for
+    // the interval is still broken. A successful fetch resets it to zero, so
+    // this reports a standing condition, not one bad afternoon.
+    if (!options.dryRun) {
+      const health = await options.store.readState(adapter.id);
+      if (health.consecutiveFailures >= BROKEN_AFTER_FAILURES) {
+        summary.broken.push({
+          sourceId: adapter.id,
+          consecutiveFailures: health.consecutiveFailures,
+          lastStatus: health.lastStatus,
+          lastConfirmedAt: health.lastConfirmedAt,
+        });
+      }
+    }
  }

  // Every source failing is not "a wiki is down", it is us: no network, a bad
@@ -181,6 +260,7 @@ export async function runRefresh(
 async function refreshOne(
  adapter: Adapter,
  options: RefreshOptions,
+  cycle: Cycle,
 ): Promise<SourceOutcome> {
  const { store } = options;
  const now = options.now();
@@ -224,6 +304,15 @@ async function refreshOne(
    };
  }

+  // Space requests to a host we have already asked this cycle. This sits after
+  // the interval and robots gates on purpose: waiting on behalf of a source we
+  // then skip would buy the host nothing and cost the run a minute.
+  const host = new URL(adapter.url).host;
+  if (cycle.requestedHosts.has(host)) {
+    await options.sleep(decision.crawlDelayMs ?? DEFAULT_HOST_GAP_MS);
+  }
+  cycle.requestedHosts.add(host);
+
  let response: Response;
  try {
    response = await options.fetchImpl(adapter.url, {
@@ -266,7 +355,7 @@ async function refreshOne(
    return {
      sourceId: adapter.id,
      result: "failed",
-      note: `HTTP ${response.status}`,
+      note: `HTTP ${response.status}${describeRejection(response)}`,
      status: response.status,
      eventCount: meta?.eventCount ?? null,
    };
@@ -395,6 +484,112 @@ async function refreshOne(
  };
 }

+/**
+ * The few header words that tell a wiki being down from an edge network turning
+ * us away.
+ *
+ * `HTTP 403` alone cannot be acted on: it reads the same whether the page moved
+ * behind a login or whether a CDN has decided the runner's address is a bot
+ * farm. Naming the server in the note means the answer is in the run log and
+ * the step summary rather than in a request someone has to reproduce by hand.
+ */
+function describeRejection(response: Response): string {
+  // A header value is a string from a host we do not control, and this one ends
+  // up inside a `::warning::` workflow command and a markdown table cell. HTTP
+  // forbids a bare newline in a value, so this is belt and braces rather than a
+  // live hole — but a note is not worth trusting a stranger's bytes over.
+  const tidy = (value: string | null): string | null => {
+    if (value === null) return null;
+    const clean = value.replace(/[^\x20-\x7e]+/g, " ").trim().slice(0, 40);
+    return clean === "" ? null : clean;
+  };
+
+  const parts: string[] = [];
+  const server = tidy(response.headers.get("Server"));
+  if (server !== null) parts.push(server);
+  if (response.headers.get("CF-Ray") !== null) parts.push("cf-ray");
+  const retryAfter = tidy(response.headers.get("Retry-After"));
+  if (retryAfter !== null) parts.push(`retry-after ${retryAfter}`);
+  return parts.length === 0 ? "" : ` (${parts.join(", ")})`;
+}
+
+/**
+ * Workflow-command lines for a GitHub runner.
+ *
+ * Warnings printed to stdout are invisible unless someone opens the log, which
+ * is how six of seven sources failed every cycle for three days under a green
+ * tick. An annotation shows on the run page itself. `::error::` annotates
+ * without failing the step, which is what lets the commit still happen.
+ */
+export function annotations(summary: RefreshSummary): string[] {
+  const lines: string[] = [];
+  for (const b of summary.broken) {
+    lines.push(
+      `::error title=${b.sourceId} has stopped answering::` +
+        `${b.consecutiveFailures} cycles failing in a row; ` +
+        `last status ${b.lastStatus ?? "none"}; ` +
+        `last confirmed ${b.lastConfirmedAt ?? "never"}. ` +
+        `This game's calendar is being built from a checked-in fixture.`,
+    );
+  }
+  for (const warning of summary.warnings) {
+    lines.push(`::warning title=refresh::${warning}`);
+  }
+  return lines;
+}
+
+/** `$GITHUB_STEP_SUMMARY` markdown: one row per source, statuses included. */
+export function stepSummary(summary: RefreshSummary): string {
+  const cell = (text: string) => text.replaceAll("|", "\\|");
+  const rows = summary.outcomes.map(
+    (o) =>
+      `| \`${o.sourceId}\` | ${o.result} | ${o.status ?? "—"} | ` +
+      `${o.eventCount ?? "—"} | ${cell(o.note)} |`,
+  );
+
+  const head =
+    `### Refresh: ${summary.changed} changed, ` +
+    `${summary.confirmed}/${summary.attempted} confirmed, ` +
+    `${summary.broken.length} broken\n\n` +
+    `| source | result | status | events | note |\n` +
+    `| --- | --- | --- | --- | --- |\n`;
+
+  const tail =
+    summary.hardFailure === null
+      ? ""
+      : `\n**Hard failure:** ${summary.hardFailure}\n`;
+
+  return `${head}${rows.join("\n")}\n${tail}`;
+}
+
+/**
+ * `$GITHUB_OUTPUT` values the workflow branches on.
+ *
+ * `broken` is deliberately an output rather than an exit code: the workflow has
+ * to commit the sources that did work before it turns the run red.
+ */
+export function outputs(summary: RefreshSummary): string[] {
+  return [
+    `changed=${summary.changed}`,
+    `attempted=${summary.attempted}`,
+    `confirmed=${summary.confirmed}`,
+    `broken=${summary.broken.length}`,
+  ];
+}
+
+/** Append to a file named by an env var, if the runner set one. */
+async function appendToEnvFile(name: string, text: string): Promise<void> {
+  const path = process.env[name];
+  if (path === undefined || path === "") return;
+  try {
+    await appendFile(path, text.endsWith("\n") ? text : `${text}\n`);
+  } catch (error) {
+    // Reporting is not the job. A read-only summary file must not turn a
+    // successful refresh into a failed one.
+    console.warn(`could not write ${name}: ${String(error)}`);
+  }
+}
+
 /** Regenerate public/data/events.v1.json from whatever is now cached. */
 export async function rebuildFeedViaScript(): Promise<void> {
  const proc = Bun.spawn(["bun", "run", "scripts/build-feed.ts"], {
@@ -521,6 +716,7 @@ async function main(): Promise<number> {
    fetchImpl: (input, init) => fetch(input, init),
    userAgent: args.userAgent,
    now: () => new Date(),
+    sleep: (ms) => Bun.sleep(ms),
    dryRun: args.dryRun,
    only: args.only,
    timeoutMs: 20_000,
@@ -529,12 +725,27 @@ async function main(): Promise<number> {
  });

  console.log(
-    `\n${summary.changed} changed, ${summary.confirmed}/${summary.attempted} confirmed, ${summary.warnings.length} warnings`,
+    `\n${summary.changed} changed, ${summary.confirmed}/${summary.attempted} confirmed, ` +
+      `${summary.warnings.length} warnings, ${summary.broken.length} broken`,
  );
  for (const warning of summary.warnings) console.warn(`  ! ${warning}`);
+  for (const b of summary.broken) {
+    console.error(
+      `  !! ${b.sourceId} has failed ${b.consecutiveFailures} cycles running ` +
+        `(last confirmed ${b.lastConfirmedAt ?? "never"})`,
+    );
+  }

-  // The workflow reads this line to decide whether to commit; `git status` is
-  // the authority, but this makes a skipped commit legible in the log.
+  // Only on a runner: the `!` and `!!` lines above already said all of this to a
+  // human, and `::warning title=…::` in a terminal is noise that reads as a bug.
+  if (process.env["GITHUB_ACTIONS"] === "true") {
+    for (const line of annotations(summary)) console.log(line);
+  }
+  await appendToEnvFile("GITHUB_STEP_SUMMARY", stepSummary(summary));
+  await appendToEnvFile("GITHUB_OUTPUT", outputs(summary).join("\n"));
+
+  // Kept for a human reading the log; `git status` is what the workflow trusts
+  // to decide whether to commit.
  console.log(`changed=${summary.changed}`);

  if (summary.hardFailure !== null) {
@@ -542,6 +753,9 @@ async function main(): Promise<number> {
    return 1;
  }

+  // Broken sources exit 0 on purpose. The workflow reads the `broken` output and
+  // fails the run *after* committing, so one dead wiki cannot stop nine live
+  // ones from reaching the site.
  return 0;
 }

@@ -3,8 +3,13 @@ import { mkdtemp, rm } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
 import {
+  annotations,
+  BROKEN_AFTER_FAILURES,
+  DEFAULT_HOST_GAP_MS,
+  outputs,
  parseArgs,
  runRefresh,
+  stepSummary,
  type RefreshOptions,
  type RobotsGate,
 } from "../scripts/refresh-sources.ts";
@@ -60,9 +65,15 @@ interface Call {

 function options(
  over: Partial<RefreshOptions> & { responder?: (call: Call) => Response },
-): { opts: RefreshOptions; calls: Call[]; rebuilds: { count: number } } {
+): {
+  opts: RefreshOptions;
+  calls: Call[];
+  rebuilds: { count: number };
+  naps: number[];
+} {
  const calls: Call[] = [];
  const rebuilds = { count: 0 };
+  const naps: number[] = [];
  const responder =
    over.responder ?? (() => new Response("<html><event></event></html>"));

@@ -83,6 +94,10 @@ function options(
    },
    userAgent: UA,
    now: () => NOW,
+    // Recorded, never waited: the per-host gap is real seconds on a runner.
+    sleep: async (ms: number) => {
+      naps.push(ms);
+    },
    dryRun: false,
    only: null,
    timeoutMs: 1000,
@@ -93,7 +108,7 @@ function options(
    ...over,
  };

-  return { opts, calls, rebuilds };
+  return { opts, calls, rebuilds, naps };
 }

 async function seed(html: string, at: string, eventCount: number | null) {
@@ -521,6 +536,251 @@ describe("a page that is not UTF-8", () => {
  });
 });

+describe("requests to one host are spaced", () => {
+  // Eight of the ten sources are game8.co pages. Each one is inside the
+  // six-hour-per-source floor and the burst is still the shape a CDN throttles.
+  const twoOnOneHost = () => [
+    adapter({ id: "genshin-game8-events", url: "https://game8.co/a" }),
+    adapter({ id: "hsr-game8-events", game: "hsr", url: "https://game8.co/b" }),
+  ];
+
+  test("the second request to a host waits", async () => {
+    const { opts, calls, naps } = options({ adapters: twoOnOneHost() });
+    await runRefresh(opts);
+
+    expect(calls).toHaveLength(2);
+    expect(naps).toEqual([DEFAULT_HOST_GAP_MS]);
+  });
+
+  test("the first request to a host does not", async () => {
+    const { opts, naps } = options({
+      adapters: [
+        adapter({ id: "genshin-game8-events", url: "https://game8.co/a" }),
+        adapter({
+          id: "endfield-wikigg-events",
+          game: "endfield",
+          url: "https://endfield.wiki.gg/wiki/Event",
+        }),
+      ],
+    });
+    await runRefresh(opts);
+    expect(naps).toEqual([]);
+  });
+
+  test("a host's own Crawl-delay wins over our default", async () => {
+    const { opts, naps } = options({
+      adapters: twoOnOneHost(),
+      robots: {
+        allows: async () => ({
+          allowed: true,
+          reason: "robots.txt ok",
+          crawlDelayMs: 10_000,
+        }),
+      },
+    });
+    await runRefresh(opts);
+    expect(naps).toEqual([10_000]);
+  });
+
+  test("a source we skip costs no wait", async () => {
+    // Sleeping on behalf of a request we are not about to make buys the host
+    // nothing and costs the cycle a minute.
+    await store.recordCheck("hsr-game8-events", {
+      at: NOW.toISOString(),
+      status: 200,
+      ok: true,
+    });
+    const { opts, calls, naps } = options({ adapters: twoOnOneHost() });
+    await runRefresh(opts);
+
+    expect(calls).toHaveLength(1);
+    expect(naps).toEqual([]);
+  });
+});
+
+describe("a source that has stopped answering", () => {
+  // Genshin is turned away; Endfield answers. A cycle where *every* source fails
+  // is already a hard failure, and this is the case that hid for three days: a
+  // green run carrying one live source and the rest of the games on fixtures.
+  const failing = () =>
+    options({
+      adapters: [
+        adapter(),
+        adapter({
+          id: "endfield-wikigg-events",
+          game: "endfield",
+          url: "https://endfield.wiki.gg/wiki/Event",
+        }),
+      ],
+      responder: (call) =>
+        call.url.includes("Genshin")
+          ? new Response("", {
+              status: 403,
+              headers: { Server: "cloudflare", "CF-Ray": "8f2a-CPH" },
+            })
+          : new Response("<html><event></event></html>"),
+    });
+
+  test("one bad cycle is a warning, not a verdict", async () => {
+    const summary = await runRefresh(failing().opts);
+    expect(summary.warnings).toHaveLength(1);
+    expect(summary.broken).toEqual([]);
+    expect(summary.hardFailure).toBeNull();
+  });
+
+  test("names what turned us away, so 403 can be acted on", async () => {
+    const summary = await runRefresh(failing().opts);
+    // "HTTP 403" alone reads the same whether the page moved behind a login or
+    // a CDN decided the runner is a bot farm.
+    expect(summary.outcomes[0]?.note).toContain("HTTP 403");
+    expect(summary.outcomes[0]?.note).toContain("cloudflare");
+    expect(summary.outcomes[0]?.note).toContain("cf-ray");
+  });
+
+  test("a verbose header cannot run away with the note", async () => {
+    // The note goes into a `::warning::` workflow command and a markdown table
+    // cell, and Server is a string from a host we do not control. A value
+    // carrying a control character is refused by the runtime — a `Response`
+    // cannot be constructed with one — so length is the part left to hold.
+    const { opts } = options({
+      responder: () =>
+        new Response("", {
+          status: 503,
+          headers: { Server: "cloudflare-".repeat(20) },
+        }),
+    });
+    const summary = await runRefresh(opts);
+    const note = summary.outcomes[0]?.note ?? "";
+
+    expect(note).toStartWith("HTTP 503 (cloudflare-");
+    expect(note.length).toBeLessThan(60);
+    for (const line of annotations({ ...summary, broken: [] })) {
+      expect(line).not.toInclude("\n");
+    }
+  });
+
+  test("three cycles running is broken, and still exits without a hard failure", async () => {
+    let summary = await runRefresh(failing().opts);
+    for (let i = 1; i < BROKEN_AFTER_FAILURES; i += 1) {
+      // Each cycle is a fresh run six hours later, so the interval is clear.
+      const later = new Date(NOW.getTime() + i * SIX_HOURS_MS);
+      summary = await runRefresh({ ...failing().opts, now: () => later });
+    }
+
+    expect(summary.broken).toHaveLength(1);
+    expect(summary.broken[0]?.consecutiveFailures).toBe(BROKEN_AFTER_FAILURES);
+    expect(summary.broken[0]?.lastStatus).toBe(403);
+    expect(summary.broken[0]?.lastConfirmedAt).toBeNull();
+    // Not a hard failure: the workflow has to commit the sources that did work
+    // before it turns the run red.
+    expect(summary.hardFailure).toBeNull();
+  });
+
+  test("stays broken while it is being skipped for the interval", async () => {
+    // The streak is the point. A source dead for days that happens to be inside
+    // its six-hour window this cycle has not recovered.
+    for (let i = 0; i < BROKEN_AFTER_FAILURES; i += 1) {
+      const at = new Date(NOW.getTime() + i * SIX_HOURS_MS);
+      await runRefresh({ ...failing().opts, now: () => at });
+    }
+    const soonAfter = new Date(
+      NOW.getTime() + (BROKEN_AFTER_FAILURES - 1) * SIX_HOURS_MS + 60_000,
+    );
+    const summary = await runRefresh({ ...failing().opts, now: () => soonAfter });
+
+    expect(summary.outcomes[0]?.result).toBe("skipped_interval");
+    expect(summary.broken).toHaveLength(1);
+  });
+
+  test("one good cycle clears it", async () => {
+    for (let i = 0; i < BROKEN_AFTER_FAILURES; i += 1) {
+      const at = new Date(NOW.getTime() + i * SIX_HOURS_MS);
+      await runRefresh({ ...failing().opts, now: () => at });
+    }
+    const recovered = new Date(
+      NOW.getTime() + BROKEN_AFTER_FAILURES * SIX_HOURS_MS,
+    );
+    const summary = await runRefresh({
+      ...options({}).opts,
+      now: () => recovered,
+    });
+
+    expect(summary.outcomes[0]?.result).toBe("fetched");
+    expect(summary.broken).toEqual([]);
+  });
+
+  test("a dry run reports no health, because it asked nothing", async () => {
+    for (let i = 0; i < BROKEN_AFTER_FAILURES; i += 1) {
+      const at = new Date(NOW.getTime() + i * SIX_HOURS_MS);
+      await runRefresh({ ...failing().opts, now: () => at });
+    }
+    const summary = await runRefresh(options({ dryRun: true }).opts);
+    expect(summary.broken).toEqual([]);
+  });
+});
+
+describe("what the runner reports to the runner", () => {
+  // Warnings on stdout are invisible unless someone opens the log, which is how
+  // six of seven sources failed every cycle for three days under a green tick.
+  const broken = {
+    outcomes: [
+      {
+        sourceId: "genshin-game8-events",
+        result: "failed" as const,
+        note: "HTTP 403 (cloudflare, cf-ray)",
+        status: 403,
+        eventCount: 9,
+      },
+    ],
+    changed: 0,
+    attempted: 1,
+    confirmed: 0,
+    warnings: ["genshin-game8-events: HTTP 403 (cloudflare, cf-ray)"],
+    broken: [
+      {
+        sourceId: "genshin-game8-events",
+        consecutiveFailures: 4,
+        lastStatus: 403,
+        lastConfirmedAt: "2026-08-14T05:27:00.000Z",
+      },
+    ],
+    hardFailure: null,
+  };
+
+  test("a broken source becomes an annotation on the run page", () => {
+    const lines = annotations(broken);
+    expect(lines[0]).toStartWith("::error title=genshin-game8-events");
+    expect(lines[0]).toContain("4 cycles failing");
+    expect(lines[0]).toContain("last status 403");
+    expect(lines.some((l) => l.startsWith("::warning"))).toBe(true);
+    // A newline inside an annotation truncates it at the runner.
+    for (const line of lines) expect(line).not.toInclude("\n");
+  });
+
+  test("nothing to say means no annotations", () => {
+    expect(annotations({ ...broken, warnings: [], broken: [] })).toEqual([]);
+  });
+
+  test("the step summary carries the status codes", () => {
+    const md = stepSummary(broken);
+    expect(md).toContain("1 broken");
+    expect(md).toContain("genshin-game8-events");
+    expect(md).toContain("403");
+    // A note holding a pipe would otherwise split the row into new columns.
+    expect(stepSummary({
+      ...broken,
+      outcomes: [{ ...broken.outcomes[0]!, note: "a | b" }],
+    })).toContain("a \\| b");
+  });
+
+  test("broken is an output, not an exit code", () => {
+    // Exiting non-zero would skip the commit and throw away the pages that did
+    // arrive; the workflow fails on this output after committing instead.
+    expect(outputs(broken)).toContain("broken=1");
+    expect(outputs(broken)).toContain("changed=0");
+  });
+});
+
 describe("the workflows that drive the refresh", () => {
  // These three defects live in YAML, and each one is silent: nothing fails,
  // the site just quietly carries wrong or stale data. Asserting on the file
@@ -567,6 +827,29 @@ describe("the workflows that drive the refresh", () => {
    expect(refresh).not.toContain("--force");
    expect(refresh).not.toContain("-f origin");
  });
+
+  test("refresh.yml turns red on a broken source only after committing", async () => {
+    // Six of seven sources failed every cycle for three days and every run
+    // showed a green tick. Reporting it must not cost the sources that did work
+    // their snapshot, so the health check has to be the last step.
+    const refresh = await read("refresh.yml");
+    const health = refresh.indexOf("Report source health");
+    const commit = refresh.indexOf("Commit refreshed snapshots");
+    const publish = refresh.indexOf("Publish the refreshed feed");
+
+    expect(health).toBeGreaterThan(commit);
+    expect(health).toBeGreaterThan(publish);
+    expect(refresh.slice(health)).toContain("steps.refresh.outputs.broken");
+    expect(refresh.slice(health)).toContain("exit 1");
+  });
+
+  test("ci.yml fails when any one source yields no events", async () => {
+    // The total-event floor is blind to one source going to zero while nine
+    // others hold the number up, which shows the reader an empty calendar for
+    // that game.
+    const ci = await read("ci.yml");
+    expect(ci).toContain("eventCount === 0");
+  });
 });

 describe("flags", () => {