refresh: let the cron stand on the recorded robots permission

Four Fandom sources have skipped on a challenged robots.txt every cycle, so
their calendars are only ever as fresh as the last manual run. For a product
whose promise is a trustworthy end date, four stale lanes are the worse failure
— so the scheduled run now passes --assume-robots-on-403 and the owner re-reads
those files by hand over time.

Kept asymmetric on purpose: --force is still refused on an unattended run.
Forcing every cycle really is just a shorter interval with extra steps, and
nothing about this decision touches that.

The accepted risk is narrower than "crawling against robots.txt", and worth
stating precisely. A plain 403 still fails closed, so a host that turns us away
still stops the run — that is what the challenge-or-refusal split buys. What is
invisible is a robots.txt edited to disallow us, because from a challenged
address a withdrawal looks exactly like the challenge we already expect. No
code can catch that; only the re-read can.

Which makes the per-cycle warning the compensating control rather than a
courtesy, so it is now pinned as one: every host named, and surviving as a
run-page annotation on a completely green cycle where nothing else draws the
eye. A dispatch can set the input to false to see the real state.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.github/workflows/refresh.yml

@@ -21,18 +21,21 @@ on:
         description: "Refresh a single source id (blank = all)"
         type: string
         default: ""
-      # Both of these are refused on the scheduled run and only work here, on a
-      # dispatch, because a dispatch is a person and GitHub records which one.
-      # See AGENTS.md § Scraping conduct: an override on every cycle is just a
-      # new default. Prefer pairing either with `only`.
+      # `force` is refused on the scheduled run and only works here, on a
+      # dispatch, because a dispatch is a person and GitHub records which one —
+      # a schedule that forces every cycle is just a shorter interval with extra
+      # steps. Prefer pairing it with `only`.
       force:
         description: "Ask sources before their 6h floor is up (one run only)"
         type: boolean
         default: false
+      # The scheduled run passes this one anyway (see the Refresh step), so this
+      # input exists to turn it OFF for a dispatch — which is how you see which
+      # hosts genuinely cannot be read, rather than which ones we assumed.
       assume_robots_on_403:
         description: "Treat a challenged robots.txt 403 as the recorded permission"
         type: boolean
-        default: false
+        default: true
 
 # Never two refreshes at once: they would both fetch, and the second would race
 # the first's commit. Queue instead of cancelling — a half-finished refresh that
@@ -91,9 +94,13 @@ jobs:
           # run script, so a dispatch input cannot become shell.
           ONLY: ${{ inputs.only }}
           DRY_RUN: ${{ inputs.dry_run }}
-          # Empty on a scheduled run, so neither flag is ever passed by the cron.
+          # Empty on a scheduled run, so --force is never passed by the cron.
           FORCE: ${{ inputs.force }}
+          # Also empty on a schedule, which is why EVENT_NAME decides there. The
+          # cron stands on the permission recorded in AGENTS.md § Scraping
+          # conduct; a dispatch can set the input to false to see the real state.
           ASSUME_ROBOTS_ON_403: ${{ inputs.assume_robots_on_403 }}
+          EVENT_NAME: ${{ github.event_name }}
         run: |
           args=()
           if [ "$DRY_RUN" = "true" ]; then
@@ -105,7 +112,7 @@ jobs:
           if [ "$FORCE" = "true" ]; then
             args+=(--force)
           fi
-          if [ "$ASSUME_ROBOTS_ON_403" = "true" ]; then
+          if [ "$ASSUME_ROBOTS_ON_403" = "true" ] || [ "$EVENT_NAME" = "schedule" ]; then
             args+=(--assume-robots-on-403)
           fi
           bun run refresh "${args[@]}"