lucasw89/gacha-event-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refresh: let the cron stand on the recorded robots permission

Browse Source 
Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.

Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.

The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.

Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-20 00:12:28 +02:00
co-authored by Claude Opus 5
parent f12398a29b
commit 3e463adf47
5 changed files with 102 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/refresh.yml
+14 -7
View File
@@ -21,18 +21,21 @@ on:
description: "Refresh a single source id (blank = all)"
type: string
default: ""
# Both of these are refused on the scheduled run and only work here, on a
# dispatch, because a dispatch is a person and GitHub records which one.
# See AGENTS.md § Scraping conduct: an override on every cycle is just a
# new default. Prefer pairing either with `only`.
# `force` is refused on the scheduled run and only works here, on a
# dispatch, because a dispatch is a person and GitHub records which one —
# a schedule that forces every cycle is just a shorter interval with extra
# steps. Prefer pairing it with `only`.
force:
description: "Ask sources before their 6h floor is up (one run only)"
type: boolean
default: false
# The scheduled run passes this one anyway (see the Refresh step), so this
# input exists to turn it OFF for a dispatch — which is how you see which
# hosts genuinely cannot be read, rather than which ones we assumed.
assume_robots_on_403:
description: "Treat a challenged robots.txt 403 as the recorded permission"
type: boolean
default: false
default: true
# Never two refreshes at once: they would both fetch, and the second would race
# the first's commit. Queue instead of cancelling — a half-finished refresh that
@@ -91,9 +94,13 @@ jobs:
# run script, so a dispatch input cannot become shell.
ONLY: ${{ inputs.only }}
DRY_RUN: ${{ inputs.dry_run }}
# Empty on a scheduled run, so neither flag is ever passed by the cron.
# Empty on a scheduled run, so --force is never passed by the cron.
FORCE: ${{ inputs.force }}
# Also empty on a schedule, which is why EVENT_NAME decides there. The
# cron stands on the permission recorded in AGENTS.md § Scraping
# conduct; a dispatch can set the input to false to see the real state.
ASSUME_ROBOTS_ON_403: ${{ inputs.assume_robots_on_403 }}
EVENT_NAME: ${{ github.event_name }}
run: |
args=()
if [ "$DRY_RUN" = "true" ]; then
@@ -105,7 +112,7 @@ jobs:
if [ "$FORCE" = "true" ]; then
args+=(--force)
fi
if [ "$ASSUME_ROBOTS_ON_403" = "true" ]; then
if [ "$ASSUME_ROBOTS_ON_403" = "true" ] || [ "$EVENT_NAME" = "schedule" ]; then
args+=(--assume-robots-on-403)
fi
bun run refresh "${args[@]}"
AGENTS.md
+25 -9
View File
@@ -483,14 +483,30 @@ in `GITHUB_ACTOR`, and a `schedule` — or any other runner event — is not. Th
were unreachable from the workflow at all. The run prints which override was used and who authorised
it.
**The cron stays refused, and for the robots override that is not ceremony.** From an address that
gets a challenge we never receive `robots.txt` at all — so a schedule standing on the recorded
permission has no way to notice the host withdrawing it. The recorded permission has no expiry, and a
person re-reading the file in a browser is the only thing that ever re-validates it. A twice-daily job
asserting it forever would be fetching on a snapshot of consent taken on 2026-08-19. Note what the
challenge-vs-refusal narrowing does and does not cover here: a plain `403` still stops us, but a
`robots.txt` *edited* to disallow us would be invisible, because we get a challenge instead of a file.
That gap is the whole reason a person has to be the one asking.
**The two overrides are not symmetrical, and `--force` is the stricter one.** A schedule may never
force: asking every cycle before the interval is up is a shorter interval with extra steps, and the
interval is the obligation. That one stays reachable only from a person or a dispatch.
**The cron does pass `--assume-robots-on-403`, and that is a decision with a named cost** (taken by
the repository owner, 2026-08-20). Four Fandom sources skip on a challenged `robots.txt` every cycle,
and the alternative was four calendars sitting as stale as the last manual run — which for a product
whose whole promise is a trustworthy end date is the worse failure. So the schedule stands on the
permission recorded in this section, and **the owner re-reads those files by hand over time** rather
than the code pretending to.
Be exact about the risk that buys, because it is not "we might be crawling against robots.txt":
- A plain `403` still fails closed. The challenge-or-refusal split is what guarantees that much, so a
host that actually turns us away still stops the run.
- What is invisible is a `robots.txt` **edited** to disallow us. From a challenged address the file
never arrives, so a withdrawal would look exactly like the challenge we already expect. Nothing in
the pipeline can catch that; only the manual re-read can.
- **The per-cycle warning is therefore the compensating control, not a nicety.** Every run names each
host it stood on a hand-recorded permission for, says the file was not read, and tells the reader to
re-read it — as a `::warning` annotation on the run page, so it survives a completely green cycle
where nothing else draws the eye. `test/refresh.test.ts` pins both the one-warning-per-host rule and
that it reaches the run page. **Do not demote it to a log line**, and do not let a future change
make it conditional: it is the only thing that ever prompts the re-read this arrangement depends on.
**A `403` is two answers wearing one status code, and only one of them is covered.** A managed
challenge means "we cannot tell what you are" — the question a human answers by reading the file in a
@@ -677,7 +693,7 @@ Fate/Grand Order problem arriving through a source that looks like it answered t
`scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor (except under the
opt-in `--force` above), one request, no retries, conditional headers, per-host spacing, robots
(failing closed when `robots.txt` cannot be read, except under the opt-in `--assume-robots-on-403`
described in § Fandom, which covers a challenged `403` and never a plain refusal). Both overrides are refused on an unattended run — a `schedule` gets neither, a person or a `workflow_dispatch` may pass both (`runAttendance`) — and reported by name
described in § Fandom, which covers a challenged `403` and never a plain refusal). The two overrides differ on who may pass them: `--force` is refused on an unattended run (`runAttendance` — a person or a `workflow_dispatch`, never a `schedule`), while `--assume-robots-on-403` is passed by the cron by standing decision and is reported by name
in the run's warnings — an override that reports nothing is one nobody withdraws. Anything that would make it fetch more often is a change to this section first.
**A source down is a warning; a source down for days is a broken build.** One wiki failing must
docs/INGESTION.md
+8 -7
View File
@@ -292,13 +292,14 @@ section but must never claim the event title.
challenge page's markers; a `403` whose body cannot be read is unclassifiable and therefore not
excused.
- It never overrides a `robots.txt` we *could* read, so a file that disallows us still says no.
- **It is refused on an unattended run, not on every runner.** `runAttendance` asks who asked: a
local shell and a `workflow_dispatch` are both a person (the latter better evidenced — GitHub
names the actor), while a `schedule` or any other runner event is not. The `schedule` refusal is
load-bearing rather than ceremonial: from a challenged address `robots.txt` never arrives, so a
cron standing on the recorded permission could never see the host withdraw it, and the recorded
permission has no expiry. `refresh.yml` therefore exposes both overrides as
`workflow_dispatch` inputs, which are empty on the cron and so can never be passed by it.
- **The cron passes it, unlike `--force`.** A standing decision (2026-08-20): four Fandom sources
skip on a challenged `robots.txt` every cycle, and four stale calendars are the worse failure for
a product that sells trustworthy end dates. `--force` remains gated on `runAttendance` — a person
or a `workflow_dispatch`, never a `schedule` — because forcing every cycle is just a shorter
interval. The accepted risk is narrow and named: a plain `403` still fails closed, but a
`robots.txt` *edited* to disallow us is invisible from a challenged address, so the owner re-reads
those files by hand. The per-cycle `::warning` naming every assumed host is the compensating
control and is pinned by tests; the dispatch input can be set to `false` to see the real state.
Every host it applied to is named in the run's warnings and in `summary.assumedRobots` — an
override that reports nothing is one nobody withdraws. It changes no other obligation: still one
scripts/refresh-sources.ts
+15 -9
View File
@@ -864,15 +864,21 @@ async function main(): Promise<number> {
return 2;
}
if (args.assumeRobotsOn403 && !attendance.attended) {
console.error(
`--assume-robots-on-403 needs a person to ask for it; this run came from ` +
`${attendance.why}.\n` +
"It asserts a permission a person read by hand, and a schedule cannot " +
"re-read it. Dispatch the workflow by hand to pass it.",
);
return 2;
}
// `--assume-robots-on-403` is deliberately NOT gated on attendance, unlike
// `--force` above. The four Fandom sources skip on a challenged robots.txt
// every cycle, and the repository owner has taken on re-reading those files by
// hand over time rather than have four calendars go stale between manual runs
// (decided 2026-08-20). That is a standing decision about this project's own
// conduct, so the code records it instead of re-litigating it per run.
//
// What keeps it honest is the disclosure, which is why that is not optional:
// every cycle names each host it stood on a hand-recorded permission for and
// says the file was not read this run. The risk being accepted is specific —
// from a challenged address robots.txt never arrives, so a file *edited* to
// disallow us would be invisible (a plain 403 still fails closed, and the
// challenge-or-refusal split is what guarantees that much). The per-cycle
// warning is the only thing that ever prompts the re-read, so do not demote it
// to a log line.
// Who authorised the override is part of the record, not a detail. A local
// shell leaves no trace anybody else can read; a dispatch names the actor.
test/refresh.test.ts
+40 -12
View File
@@ -901,6 +901,31 @@ describe("what the runner reports to the runner", () => {
for (const line of lines) expect(line).not.toInclude("\n");
});
test("an assumed robots permission annotates an otherwise green run", () => {
// This is now the scheduled run's standing state, not an exception: the cron
// passes --assume-robots-on-403, so every cycle fetches four Fandom sources
// on a permission it could not re-read. That warning is the whole
// compensating control for the risk being accepted (AGENTS.md § Scraping
// conduct), and a summary line can be scrolled past where a run-page
// annotation cannot — so it has to survive as one even when nothing is
// broken and there is no error to draw the eye.
const green = {
...broken,
broken: [],
warnings: [
"reverse1999.fandom.com: fetched on --assume-robots-on-403 — its " +
"robots.txt was NOT read this run. Re-read it in a browser.",
],
assumedRobots: ["reverse1999.fandom.com"],
};
const lines = annotations(green);
expect(lines).toHaveLength(1);
expect(lines[0]).toStartWith("::warning title=refresh::");
expect(lines[0]).toContain("--assume-robots-on-403");
expect(lines[0]).toContain("reverse1999.fandom.com");
expect(lines[0]).not.toInclude("\n");
});
test("nothing to say means no annotations", () => {
expect(annotations({ ...broken, warnings: [], broken: [] })).toEqual([]);
});
@@ -978,12 +1003,14 @@ describe("the workflows that drive the refresh", () => {
expect(refresh).not.toContain("push --force");
});
test("refresh.yml offers both overrides to a dispatch and to no cron", async () => {
// The scheduled run must never be able to pass either one: an override on
// every cycle is the new default with extra steps, and for the robots one a
// cron cannot re-read the permission it would be standing on. Both flags are
// therefore reachable only through `workflow_dispatch` inputs, which are
// empty on a schedule, and each is guarded on the literal string "true".
test("refresh.yml lets the cron assume robots, and never lets it force", async () => {
// The two overrides are deliberately not symmetrical any more. --force stays
// reachable only through a dispatch input: a schedule that forces every cycle
// is a shorter interval with extra steps, and the interval is the obligation.
// --assume-robots-on-403 is passed by the cron on purpose (owner's decision,
// 2026-08-20) so four Fandom calendars do not sit stale between manual runs;
// the compensating control is the per-cycle warning naming every host, which
// is what prompts the manual re-read.
const refresh = await read("refresh.yml");
const dispatch = refresh.slice(
@@ -993,15 +1020,16 @@ describe("the workflows that drive the refresh", () => {
expect(dispatch).toContain("force:");
expect(dispatch).toContain("assume_robots_on_403:");
// Every occurrence of either flag is inside a test on its input variable.
// --force: guarded on its input alone, so a schedule (where it is empty)
// can never reach it.
expect(refresh).toMatch(/if \[ "\$FORCE" = "true" \]; then\n\s*args\+=\(--force\)/);
expect(refresh).not.toMatch(/FORCE" = "true" \] \|\| \[ "\$EVENT_NAME"/);
// --assume-robots-on-403: its input OR the run being a schedule.
expect(refresh).toMatch(
/if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \]; then\n\s*args\+=\(--assume-robots-on-403\)/,
/if \[ "\$ASSUME_ROBOTS_ON_403" = "true" \] \|\| \[ "\$EVENT_NAME" = "schedule" \]; then\n\s*args\+=\(--assume-robots-on-403\)/,
);
// The cron block itself carries no flags.
const cron = refresh.slice(refresh.indexOf("schedule:"), refresh.indexOf("workflow_dispatch:"));
expect(cron).not.toContain("force");
expect(cron).not.toContain("assume");
expect(refresh).toContain("EVENT_NAME: ${{ github.event_name }}");
});
test("refresh.yml turns red on a broken source only after committing", async () => {