feat: refresh sources on a schedule
The feed was generated from checked-in fixtures and only moved when somebody captured a page by hand. This fetches. bun run refresh caches each page raw under snapshots/ and rebuilds the feed from what it cached; build-feed prefers a snapshot and falls back to the fixture, so a clean checkout and the container build stay offline and reproducible. The workflow runs it twice a day and commits only when a page's bytes actually changed — a 304, an identical body or a rejected parse all leave the tree clean — then dispatches ci.yml, which already knows how to test, build and deploy. Scraping conduct is enforced in code rather than left to good intentions: one request per source per cycle, a six-hour floor checked per source, conditional requests, a User-Agent with a contact URL, and robots.txt honoured — failing closed, because a permission we could not read is not a permission we have. No retries; a retry is a second request. A body that yields zero events is rejected and the previous snapshot kept, so a redesigned wiki shows up as a stale timestamp rather than an emptied calendar. One source down is a warning; all of them down fails the run, so a cycle that learned nothing is never committed. Tested entirely offline against an injected fetch and clock — no request has ever been made to a live wiki from this code. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
18b9652aed
commit
b085087b05
+10
-3
@@ -1,9 +1,10 @@
|
||||
# Build the static site, then serve it from a distroless-ish runtime.
|
||||
#
|
||||
# Two stages so the image ships the built assets and nothing else: no source,
|
||||
# no fixtures, no toolchain. The build is fully offline — it parses checked-in
|
||||
# fixtures rather than fetching anything — so the image is reproducible and
|
||||
# needs no network at build time.
|
||||
# no fixtures, no toolchain. The build is fully offline — it parses the
|
||||
# committed snapshots, falling back to checked-in fixtures, rather than
|
||||
# fetching anything — so the image is reproducible and needs no network at
|
||||
# build time.
|
||||
|
||||
FROM oven/bun:1.3-alpine AS build
|
||||
WORKDIR /app
|
||||
@@ -16,6 +17,12 @@ COPY tsconfig.json index.html serve.ts ./
|
||||
COPY src ./src
|
||||
COPY scripts ./scripts
|
||||
COPY fixtures ./fixtures
|
||||
# Whatever the last refresh committed. The directory always exists (it carries
|
||||
# a README), so this cannot break a build made before the first refresh — it
|
||||
# just leaves build:feed on the fixture fallback, which is what the image did
|
||||
# before. Without it the container would serve fixture-era data while the site
|
||||
# served fresh, with nothing to say why.
|
||||
COPY snapshots ./snapshots
|
||||
COPY test ./test
|
||||
|
||||
# Fail the image on a type error or a failing test rather than shipping it.
|
||||
|
||||
Reference in New Issue
Block a user