lucasw89/gacha-event-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: stop failing the build on a page that says it has no events

Browse Source 
Infinity Nikki's wiki prints "There are no Events in this category"
between versions, and the refresh runner has honoured that since
2026-09-03 — it stores the empty parse as the source's real answer
rather than letting a quiet lane reach the broken tier. The feed build
never asked, so the same page arrived at CI as parsedCount 0,
indistinguishable from a parser that has stopped reading a redesigned
page, and brokenSources failed every build while every refresh stayed
green.

The runner's verdict cannot travel on its own: only a parser has seen
the page, and by the time brokenSources runs there is nothing left but
the feed. So the fact rides on SourceHealth, defaulted so an older feed
the service worker cached keeps validating and reads as the strict
answer.

Both ends now ask it the same way — of an empty parse only, from the
page's own words only — because a redesign yields zero rows too, and
excusing that is the silently emptied calendar the gate exists for.

The rule sits in a module rather than in build-feed.ts, which writes
public/ and so runs a build if a test imports it. That is how the two
ends drifted apart with nothing to catch it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-09-06 18:02:16 +02:00
co-authored by Claude Opus 5
parent e51585f2b2
commit 1156c358a7
9 changed files with 304 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+13
View File
@@ -131,6 +131,7 @@ re-verify a sample against the live page afterward.
src/shared/ schema.ts (the contract), time.ts, daily.ts, effort.ts, games.ts, feed.ts
custom.ts — reader-authored games and events, and their key spaces
src/ingest/ html.ts, dates.ts (sixteen formats), merge.ts, sanitize.ts, robots.ts, snapshots.ts
health.ts — which of the three empties a source's zero was; pure
parsers/ game8.ts, wikigg.ts, akwiki.ts, fandom.ts, bawiki.ts, holodori.ts, iopwiki.ts,
stellasora.ts — keyed by SITE, not game
adapters/ index.ts — SOURCES registry binding url+game+parser, and the sanitize seam
@@ -643,6 +644,18 @@ and both generalise past this wiki:
on the page's own words — a redesign yields zero rows too, and storing *that* is the silent
emptying the zero-events gate exists to prevent. Only this template implements it; the other three
say nothing either way when empty and keep the strict gate.
- **Two things judge a zero, and telling only one of them is half a fix.** The runner learned this on
2026-09-03; `scripts/build-feed.ts` did not, and CI fails the build on `brokenSources`
(`parsedCount === 0`). So for four days every green refresh was followed by a red CI run over a lane
that was correctly empty — the same rule, contradicting itself across two scripts. The runner's
verdict cannot travel on its own: only a parser has seen the page, and by the time `brokenSources`
runs there is nothing left but the feed. `SourceHealth.statesNoEvents` carries it,
`src/ingest/health.ts` sets it, and both ends now ask it the same way — of an empty parse only, from
the page's own words only.
It lives in a module because `build-feed.ts` writes `public/`, so importing it from a test runs a
build and the rule sat where no test could reach it. That is how the two ends drifted, and it is
the lesson `brokenSources` itself already carried: it was inline in `ci.yml` and pinned by grepping
that file for a string, which proved the check existed and never that it was right.
An empty Nikki lane is therefore now the truth rather than a gap, exactly as GFL2's thin weeks are.
The lane refills on its own when 2.8 is listed, with no parser change.
docs/DATA-MODEL.md
+19 -4
View File
@@ -608,7 +608,22 @@ partly applied.
## Schema versioning
`/api/events` responses carry `{ schemaVersion: 1, generatedAt, events: [...] }`. The client
refuses to render a `schemaVersion` it does not know and shows a "refresh the page" prompt instead
of guessing at unfamiliar fields. Additive fields do not bump the version; removing or retyping a
field does.
`/api/events` responses carry `{ schemaVersion: 1, generatedAt, events: [...], sources: [...] }`.
The client refuses to render a `schemaVersion` it does not know and shows a "refresh the page"
prompt instead of guessing at unfamiliar fields. Additive fields do not bump the version; removing
or retyping a field does.
`sources` is a `SourceHealth` per registered source (`src/shared/feed.ts`), and it is what the
footer's freshness line and CI's build gate both read. Three of its fields describe one source's
last document and only make sense together: `eventCount` is what it contributed after expired events
were dropped, `parsedCount` is what the same bytes yield when parsed as of their own capture date,
and `statesNoEvents` says the page itself declared it currently lists none. That triple is what
separates a parser that has stopped reading a redesigned page (a fault) from one whose events have
all since ended, and from a game between versions (neither) — see `docs/INGESTION.md` § Stage 1.
**Every field added here is defaulted, and that is a rule rather than a habit.** The service worker
serves the last feed it downloaded, so a required field fails validation on every cached feed and
takes the offline promise down with it. A default must also be the *strict* reading of silence:
`parsedCount` defaults to `null` — an older feed never recorded the figure, which is an absence of
information and not evidence of health — and `statesNoEvents` to `false`, because a feed built
before the field existed never asked the page and cannot be read as having got an answer.
docs/INGESTION.md
+22
View File
@@ -406,6 +406,28 @@ the page's statement and never a row count, so a redesign still rejects, and the
the two it saw: `0 events — the page states it currently lists none`. One source down is a warning and exit 0; every source failing is exit 1, so CI never
commits a cycle that learned nothing.
**The fact has to reach the feed, because the runner is not the only thing that judges a zero.**
`scripts/build-feed.ts` parses the same snapshot again and records a `SourceHealth` per source, and
CI fails the build on `brokenSources` — `parsedCount === 0`. For four days from 2026-09-03 the
runner excused Infinity Nikki's quiet page and the feed build did not, so every green refresh was
followed by a red CI run on a lane that was correctly empty. The runner's verdict could not travel:
only a parser has seen the page, and by the time `brokenSources` runs there is nothing left but the
feed. So `SourceHealth.statesNoEvents` carries it, `src/ingest/health.ts` sets it, and the same rule
applies at both ends — asked only of an empty parse, taken only from the page's own words.
Two properties of that field are load-bearing rather than incidental. It is **defaulted, never
required**, for the reason `parsedCount` is: the service worker serves the last feed it downloaded,
and a required field would fail every cached feed's validation and take the offline promise with it.
And an absent value reads as `false` — the strict answer — because a feed built before the field
existed never asked the question and must not be read as having answered it.
**The rule lives in a module rather than in the script, and that is the point.**
`scripts/build-feed.ts` writes `public/`, so importing it from a test runs a build; the rule sat
where no test could reach it, which is exactly how the two ends drifted apart. `src/ingest/health.ts`
is a pure function over an adapter and a document, and `test/feed.test.ts` exercises it. Same lesson
as `brokenSources` itself, which was inline in `ci.yml` and pinned by grepping that file for a
string — proof the check existed, never that it was right.
## Stage 2 — parse
Hash the raw body (sha256) → `content_hash`. **If it matches `sources.content_hash`, end as
scripts/build-feed.ts
+11 -32
View File
@@ -10,6 +10,7 @@
* bun run build:feed
*/
import { ADAPTERS } from "../src/ingest/adapters/index.ts";
import { sourceHealth } from "../src/ingest/health.ts";
import { mergeEvents } from "../src/ingest/merge.ts";
import { SnapshotStore, freshnessAt } from "../src/ingest/snapshots.ts";
import { EventFeed, SCHEMA_VERSION, type SourceHealth } from "../src/shared/feed.ts";
@@ -69,44 +70,22 @@ for (const adapter of ADAPTERS) {
game: adapter.game,
});
// Parsed a second time as of the document's own capture date, when nothing
// in it had expired yet. That figure is what separates "this parser has
// stopped reading the page" from "this page's events have all finished
// since it was captured" — the two are the same zero once expiry has been
// applied, and only the first means our code is wrong.
// Null when we do not know when these bytes were current: there is no date
// to parse "as of", and inventing one would manufacture a figure the check
// then trusts. Unknown is a real answer here, and `brokenSources` declines
// to fail a build on it.
const parsedCount =
at === null
? null
: adapter.parse(html, {
now: at,
sourceUrl: adapter.url,
sourceId: adapter.id,
game: adapter.game,
}).length;
// Which of the three empties this is, decided in a module a test can reach
// rather than here — see `src/ingest/health.ts` for why that matters.
const health = sourceHealth(adapter, html, at, events.length);
const { parsedCount } = health;
const groups = byGame.get(adapter.game) ?? [];
groups.push(events);
byGame.set(adapter.game, groups);
sources.push({
sourceId: adapter.id,
game: adapter.game,
url: adapter.url,
// When the bytes were last confirmed live; a fixture's capture date when
// this source has never been refreshed.
lastSuccessAt: at,
eventCount: events.length,
parsedCount,
});
sources.push(health);
// A source that parsed events and then lost them all to the calendar says
// so on the build log, because a bare "0 events" reads as a fault.
const note =
events.length === 0 && parsedCount !== null && parsedCount > 0
// A source that came back with nothing says which nothing it was, because a
// bare "0 events" reads as a fault and two of the three are not one.
const note = health.statesNoEvents
? " (the page states it currently lists none)"
: events.length === 0 && parsedCount !== null && parsedCount > 0
? ` (all ${parsedCount} have ended — stale page)`
: "";
console.log(
src/ingest/health.ts
+89
View File
@@ -0,0 +1,89 @@
/**
* What the feed records about a source, from the document it just parsed.
*
* This is one function rather than a few lines inside `scripts/build-feed.ts`
* because of how the rule it carries went wrong. The refresh runner and the
* feed builder both have to tell three zeros apart — a parser that can no
* longer read a redesigned page, a page whose events have all ended, and a page
* that says it currently lists none — and the runner learned the third on
* 2026-09-03 while the builder did not. Nothing caught that: the builder is a
* top-level script that fetches nothing but writes `public/`, so importing it
* from a test runs a build, and the rule sat where no test could reach it.
*
* That is the same argument the `ci.yml` comment already makes about
* `brokenSources`: behaviour belongs where behaviour can be exercised.
* `docs/INGESTION.md` § Stage 1 carries the rule itself.
*/
import type { SourceHealth } from "../shared/feed.ts";
import type { GachaEvent, GameId } from "../shared/schema.ts";
/**
* The part of an `Adapter` this needs. Narrow on purpose — it keeps the
* function testable with a stub and says plainly that nothing here fetches.
*/
export interface HealthAdapter {
id: string;
game: GameId;
url: string;
parse(
html: string,
ctx: { now: string; sourceUrl: string; sourceId: string; game: GameId },
): GachaEvent[];
statesNoEvents?(html: string): boolean;
}
/**
* @param html the document the feed was built from — a live snapshot, or a
* checked-in fixture on a clean checkout.
* @param at when those bytes were last confirmed current, or null when
* we do not know.
* @param eventCount events this source contributed to the feed, after expiry.
*/
export function sourceHealth(
adapter: HealthAdapter,
html: string,
at: string | null,
eventCount: number,
): SourceHealth {
// Parsed a second time as of the document's own capture date, when nothing in
// it had expired yet. That figure is what separates "this parser has stopped
// reading the page" from "this page's events have all finished since it was
// captured" — the two are the same zero once expiry has been applied, and
// only the first means our code is wrong.
//
// Null when we do not know when these bytes were current: there is no date to
// parse "as of", and inventing one would manufacture a figure the check then
// trusts. Unknown is a real answer here, and `brokenSources` declines to fail
// a build on it.
const parsedCount =
at === null
? null
: adapter.parse(html, {
now: at,
sourceUrl: adapter.url,
sourceId: adapter.id,
game: adapter.game,
}).length;
// Asked only of a zero, exactly as `scripts/refresh-sources.ts` asks it. The
// flag qualifies an empty parse — "this zero is the page's own answer" — and
// claims nothing on its own, so a loose implementation that keeps matching
// after a redesign can never excuse a source that is still producing rows.
//
// Null takes the strict reading for the same reason it does above: with no
// date for the bytes there is no parse to qualify.
const statesNoEvents =
parsedCount === 0 && adapter.statesNoEvents?.(html) === true;
return {
sourceId: adapter.id,
game: adapter.game,
url: adapter.url,
// When the bytes were last confirmed live; a fixture's capture date when
// this source has never been refreshed.
lastSuccessAt: at,
eventCount,
parsedCount,
statesNoEvents,
};
}
src/shared/feed.ts
+27 -1
View File
@@ -34,6 +34,26 @@ export const SourceHealth = z.object({
* than evidence of a fault.
*/
parsedCount: z.number().int().nonnegative().nullable().default(null),
/**
* The page itself says it currently lists no events.
*
* The third of the three ways a source can read zero, and the only one the
* feed could not previously express. A redesigned page the parser can no
* longer read, a page whose events have all ended, and a page printing
* "There are no Events in this category" all arrive as `parsedCount: 0` —
* and the last one is a source *answering*, not failing.
*
* `scripts/refresh-sources.ts` has drawn this distinction since 2026-09-03
* and the feed did not, so a correctly quiet lane reddened CI every build.
* Carried here because `brokenSources` runs against the feed and has nothing
* else to go on: only the parser has seen the page.
*
* **Set from the page's own words, never from a row count** — a redesign
* yields zero rows too, and excusing *that* is the silently emptied calendar
* the check exists for. Defaulted rather than required, for the reason
* `parsedCount` is: an older cached feed must keep validating.
*/
statesNoEvents: z.boolean().default(false),
});
export const EventFeed = z.object({
@@ -120,9 +140,15 @@ export function freshness(
* page and a broken parser arrived as the same zero. Only an explicit zero
* counts here; a null is an older feed that never recorded the figure, and
* failing on missing information would be the same mistake in a new place.
*
* Nor is a page that states its own emptiness, which is that same mistake a
* third time: a gacha calendar goes quiet between versions, and Infinity
* Nikki's wiki says so in words. `statesNoEvents` is the page answering, so it
* is excused here exactly as the refresh runner already excuses it — see that
* field, and `scripts/refresh-sources.ts`.
*/
export function brokenSources(sources: readonly SourceHealth[]): SourceHealth[] {
return sources.filter((s) => s.parsedCount === 0);
return sources.filter((s) => s.parsedCount === 0 && !s.statesNoEvents);
}
/**
test/custom-ui.test.tsx
+1
View File
@@ -240,6 +240,7 @@ describe("Colophon freshness notice (PRD F7)", () => {
eventCount: 9,
parsedCount: 9,
statesNoEvents: false,
};
test("states when the data was refreshed, unprompted", () => {
test/feed.test.ts
+121 -2
View File
@@ -4,9 +4,10 @@ import {
freshness,
staleSources,
STALE_AFTER_MS,
type SourceHealth,
SourceHealth,
} from "../src/shared/feed.ts";
import type { GameId } from "../src/shared/schema.ts";
import { sourceHealth } from "../src/ingest/health.ts";
import type { GachaEvent, GameId } from "../src/shared/schema.ts";
/**
* Freshness disclosure (PRD F7).
@@ -34,6 +35,7 @@ function source(
eventCount: 3,
parsedCount: 3,
statesNoEvents: false,
};
}
@@ -132,6 +134,7 @@ describe("telling a broken source from a stale one", () => {
lastSuccessAt: "2026-08-19T00:00:00.000Z",
eventCount: 0,
parsedCount: 7,
statesNoEvents: false,
...over,
});
@@ -143,6 +146,29 @@ describe("telling a broken source from a stale one", () => {
]);
});
test("a source whose page states it lists none is not broken", () => {
// The third empty, and the one the refresh runner already knew about while
// the feed did not. Infinity Nikki's wiki replaced both event tables with
// "There are no Events in this category" between 2.7 and 2.8, so the page
// parses to zero and is *answering*. A redesign yields zero too, which is
// why only the page's own words may say which — never the row count.
expect(
brokenSources([health({ parsedCount: 0, statesNoEvents: true })]),
).toEqual([]);
});
test("a page that states its emptiness still fails once it parses nothing at all", () => {
// Guard on the guard. `statesNoEvents` excuses an empty parse, so a parser
// whose selectors all broke must not be able to reach it: the flag is only
// ever set from the page's own declaration, and a source claiming both a
// declaration and rows is a contradiction we do not have to honour.
expect(
brokenSources([health({ parsedCount: 0, statesNoEvents: false })]).map(
(s) => s.sourceId,
),
).toEqual(["nikki-fandom-events"]);
});
test("a source whose events have all ended is not broken", () => {
// The Nikki case exactly. The parser did its job; the calendar moved past
// everything the page still lists.
@@ -164,6 +190,23 @@ describe("telling a broken source from a stale one", () => {
]);
});
test("a feed cached before the field existed is not called broken", () => {
// The service worker serves the last feed it downloaded, so a feed built
// before `statesNoEvents` shipped still has to validate and still has to
// mean what it meant. Absent parses to `false`, which is the strict
// reading — an old feed cannot vouch for a page it never asked.
const cached = SourceHealth.parse({
sourceId: "nikki-fandom-events",
game: "nikki",
url: "https://example.test/nikki",
lastSuccessAt: "2026-08-19T00:00:00.000Z",
eventCount: 0,
parsedCount: 7,
});
expect(cached.statesNoEvents).toBe(false);
expect(brokenSources([cached])).toEqual([]);
});
test("a feed that never recorded the count is not called broken", () => {
// An older feed — one the service worker cached before this field existed
// — says nothing either way, and absence of information is not evidence of
@@ -172,3 +215,79 @@ describe("telling a broken source from a stale one", () => {
expect(staleSources([health({ parsedCount: null })])).toEqual([]);
});
});
/**
* What the feed builder records about a source it just parsed.
*
* The distinction above is only worth having if something sets it, and this is
* the seam where it was missing: `scripts/refresh-sources.ts` had asked
* `statesNoEvents` since 2026-09-03 while `scripts/build-feed.ts` never did, so
* a page declaring itself empty reached CI as a bare zero and failed the build.
* The rule was in a script nothing could import, which is why it now lives in a
* module and is exercised here rather than grepped for.
*/
describe("recording a source's health at build time", () => {
const EMPTY = "<p>There are no Events in this category</p>";
const FULL = "<p>Song of the Wandering Sky</p>";
const adapter = (over: Partial<Parameters<typeof sourceHealth>[0]> = {}) => ({
id: "nikki-fandom-events",
game: "nikki" as GameId,
url: "https://example.test/nikki",
parse: (html: string) => (html === FULL ? ([{}] as unknown as GachaEvent[]) : []),
statesNoEvents: (html: string) => html.includes("There are no Events"),
...over,
});
test("a page that declares itself empty is recorded as having answered", () => {
const health = sourceHealth(adapter(), EMPTY, "2026-09-06T15:39:19.376Z", 0);
expect(health.parsedCount).toBe(0);
expect(health.statesNoEvents).toBe(true);
expect(brokenSources([health])).toEqual([]);
});
test("a page that reads empty without saying so is left to fail", () => {
// The redesign case, and the whole reason the flag may not be inferred
// from the row count: this parse is zero too.
const health = sourceHealth(
adapter({ statesNoEvents: () => false }),
EMPTY,
"2026-09-06T15:39:19.376Z",
0,
);
expect(health.statesNoEvents).toBe(false);
expect(brokenSources([health]).map((s) => s.sourceId)).toEqual([
"nikki-fandom-events",
]);
});
test("a source with rows is never marked as declaring itself empty", () => {
// A loose `statesNoEvents` — one matching prose that survives a redesign —
// must not be able to excuse a source that is working. The flag qualifies
// an empty parse and states nothing on its own, so it is asked only of a
// zero, exactly as the refresh runner asks it.
const health = sourceHealth(
adapter({ statesNoEvents: () => true }),
FULL,
"2026-09-06T15:39:19.376Z",
1,
);
expect(health.parsedCount).toBe(1);
expect(health.statesNoEvents).toBe(false);
});
test("a document of unknown age records neither count nor declaration", () => {
// No capture date means no "as of" to parse against, and inventing one
// manufactures a figure the check then trusts. `parsedCount` is null for
// that reason and the declaration goes with it: both answer a question
// about bytes we cannot date.
const health = sourceHealth(adapter(), EMPTY, null, 0);
expect(health.parsedCount).toBeNull();
expect(health.statesNoEvents).toBe(false);
expect(brokenSources([health])).toEqual([]);
});
});
test/issue-templates.test.tsx
+1
View File
@@ -168,6 +168,7 @@ describe("the app's links into them", () => {
eventCount: 9,
parsedCount: 9,
statesNoEvents: false,
},
]}
now={NOW}