feat: refresh sources on a schedule

The feed was generated from checked-in fixtures and only moved when
somebody captured a page by hand. This fetches.

bun run refresh caches each page raw under snapshots/ and rebuilds the
feed from what it cached; build-feed prefers a snapshot and falls back to
the fixture, so a clean checkout and the container build stay offline and
reproducible. The workflow runs it twice a day and commits only when a
page's bytes actually changed — a 304, an identical body or a rejected
parse all leave the tree clean — then dispatches ci.yml, which already
knows how to test, build and deploy.

Scraping conduct is enforced in code rather than left to good
intentions: one request per source per cycle, a six-hour floor checked
per source, conditional requests, a User-Agent with a contact URL, and
robots.txt honoured — failing closed, because a permission we could not
read is not a permission we have. No retries; a retry is a second
request.

A body that yields zero events is rejected and the previous snapshot
kept, so a redesigned wiki shows up as a stale timestamp rather than an
emptied calendar. One source down is a warning; all of them down fails
the run, so a cycle that learned nothing is never committed.

Tested entirely offline against an injected fetch and clock — no request
has ever been made to a live wiki from this code.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Lucas Winther
2026-08-15 21:15:25 +02:00
co-authored by Claude Opus 5
parent 18b9652aed
commit b085087b05
12 changed files with 2251 additions and 12 deletions
+1
View File
@@ -7,6 +7,7 @@
"typecheck": "tsc --noEmit",
"parse": "bun run scripts/parse-fixture.ts",
"build:feed": "bun run scripts/build-feed.ts",
"refresh": "bun run scripts/refresh-sources.ts",
"build:css": "bunx @tailwindcss/cli -i src/client/styles.css -o public/styles.css --minify",
"build:js": "bun build src/client/main.tsx --outfile public/main.js --minify",
"build": "bun run build:feed && bun run build:css && bun run build:js && bun run build:static",