Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.
Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.
The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.
Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Four Fandom sources have been skipping on a challenged robots.txt every cycle,
and the only thing that clears it is a person passing the recorded permission.
Until now there was no way to do that from the workflow, so it meant checking
out the repo and running the refresh by hand. Both overrides are dispatch
inputs now, guarded on the literal string "true" and empty on the cron, so the
schedule still cannot reach either one.
The force-push guard needed narrowing to allow this. It asserted the workflow
contains no "--force" anywhere, as a proxy for never rewriting the branch, and
the refresh runner's own --force flag trips that substring while having nothing
to do with pushing. It now matches force on a git push specifically, including
--force-with-lease, which is still one — a tighter assertion than the string it
replaces, checked against both a real force push and the flag it must ignore.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The guidance in here is not Claude-specific — it is what this project is,
what it refuses to do, and the rules that are invisible from the code. Any
agent working here needs it, and AGENTS.md is the name they all look for.
CLAUDE.md stays as a pointer so Claude Code still finds it, and holds no
guidance of its own; two copies would drift and the wrong one would be read.
Every cross-reference in the source comments, docs, workflow and skills now
names AGENTS.md, and the image ignores both.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The only snapshot the scheduled refresh has ever committed is
endfield-wikigg-events. Seven sources existed at that run; the six game8.co
ones yielded nothing, and no cycle since has committed anything. All of them
fetch fine from a laptop, so whatever is happening happens on the runner —
meanwhile eight of ten games were served from checked-in fixtures for three
days behind a green tick. Nine failures out of ten was exit 0 with warnings
buried in a log nobody opens.
`consecutiveFailures` was already tracked and never read. A source that has
failed BROKEN_AFTER_FAILURES (3, so ~36h at two cycles a day) is now reported
as `broken`: a GitHub annotation, a job-summary row carrying its status code,
and a `broken` step output. The runner still exits 0 on it and `refresh.yml`
fails on that output in a final step, after the commit and the CI dispatch —
exiting non-zero from the runner would skip the commit and throw away the pages
that did arrive, which is the opposite of what "one wiki down never blanks a
calendar" is for. The streak is read from the store rather than from this
cycle's outcome, so a source dead for days that happens to be inside its
six-hour window has not recovered.
A non-ok response now records what turned us away — the Server header, whether
a CF-Ray was present, any Retry-After — because a bare `HTTP 403` reads
identically whether the page moved behind a login or a CDN decided the runner
is a bot farm, and that is the open question here. Values are trimmed and
capped: the note lands in a workflow command and a markdown cell, and it came
from a host we do not control.
Also space requests to a host already asked this cycle, honouring its
Crawl-delay and defaulting to 2s. Eight sources share game8.co, so the
per-source floor alone still permitted one cycle to arrive as eight
back-to-back requests to a single site — which is what a burst looks like from
the far end regardless of our intent, and is plausibly self-inflicted here. The
wait is taken after the interval and robots gates, so a source we then skip
costs nothing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed sanity check only had a floor on the total, which is blind to the
failure it exists to catch: nine healthy sources hold the number comfortably
over twenty while the tenth has gone to zero and that game shows an empty
calendar. Check per source, and print the per-source counts so a drop is
legible in the log before it is a failure.
Zero is unambiguous here. A live snapshot that parsed to nothing is refused by
the refresh runner and never stored, so a source at zero in the feed means the
checked-in fixture stopped parsing — a regression in our code, not a quiet week
for that game.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`lastConfirmedAt` lives in gitignored bookkeeping that only refresh.yml
restored, so the workflow that actually builds and deploys never saw it:
every source reported its last *content change* as its last success, and
the UI flagged anything whose bytes had not moved in two days as stale —
which is most wiki pages most of the time. ci.yml now restores the same
cache read-only before building the feed.
The refresh push was a bare `git push`, so a human push landing in
between made it non-fast-forward: the job failed and threw away pages it
had just fetched, while the bookkeeping had already been saved, so those
sources would not be re-asked for six hours. Rebase and retry instead —
never force.
The cache save key used run_id, which is stable across re-runs, so a
re-run saved nothing and the run after it restored stale bookkeeping.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.
bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.
Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.
A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.
Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The image build ran the tests without serve.ts present, so the server the
tests spawn never started and the hook stalled until it timed out. The
runtime stage would have failed on the same missing file immediately after.
The test now bails the moment the process exits and reports its stderr, so
this shows up as "serve.ts exited with 1 before listening" in 53ms rather
than an unexplained hook timeout after five seconds.
serve.ts is also in tsconfig's include now — it was outside it, so a type
error in the file that serves the app would only have surfaced at runtime.
Adding it immediately caught one in the test.
Verified by replaying the build stage against exactly the copied file set.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
configure-pages fails with a 404 when Pages has never been switched on for
the repository. It can enable it given the pages: write permission the job
already holds, which beats requiring a trip through settings before CI can
go green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
github.repository preserves the owner's casing and GHCR rejects any
uppercase in a repository name, so the image job failed on tag validation
before it built anything.
Also drops the review-quarantine skill's references to evidence spans and the
extraction-evaluator agent, both of which went away with the LLM layer.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Mirrors the GitLab pipeline — typecheck, tests, feed sanity — and adds a
Pages deploy.
Pages serves from /<repo>/, and the app used absolute asset paths, so that
job would have shipped a site that 404s on everything. Assets now resolve
against a <base href> the build substitutes, the feed URL resolves against
document.baseURI so deep links work too, and the service worker derives its
paths from its own registration scope. Root-hosted builds are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Lucas Winther