Lucas WintherandClaude Opus 5
b085087b05
The feed was generated from checked-in fixtures and only moved when somebody captured a page by hand. This fetches. bun run refresh caches each page raw under snapshots/ and rebuilds the feed from what it cached; build-feed prefers a snapshot and falls back to the fixture, so a clean checkout and the container build stay offline and reproducible. The workflow runs it twice a day and commits only when a page's bytes actually changed — a 304, an identical body or a rejected parse all leave the tree clean — then dispatches ci.yml, which already knows how to test, build and deploy. Scraping conduct is enforced in code rather than left to good intentions: one request per source per cycle, a six-hour floor checked per source, conditional requests, a User-Agent with a contact URL, and robots.txt honoured — failing closed, because a permission we could not read is not a permission we have. No retries; a retry is a second request. A body that yields zero events is rejected and the previous snapshot kept, so a redesigned wiki shows up as a stale timestamp rather than an emptied calendar. One source down is a warning; all of them down fails the run, so a cycle that learned nothing is never committed. Tested entirely offline against an injected fetch and clock — no request has ever been made to a live wiki from this code. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
26 lines
1.3 KiB
Markdown
26 lines
1.3 KiB
Markdown
# Snapshots
|
|
|
|
Raw pages, exactly as fetched. `scripts/refresh-sources.ts` writes them; nothing else fetches.
|
|
|
|
```
|
|
<source-id>.html the body verbatim — tracked
|
|
<source-id>.meta.json hash, size, ETag, Last-Modified, when the bytes last changed — tracked
|
|
<source-id>.state.json when we last checked, and failure streak — gitignored
|
|
```
|
|
|
|
Three reasons this is committed rather than cached:
|
|
|
|
- **Re-parsing never re-fetches.** Iterating on a parser reads these files, not the wikis
|
|
(CLAUDE.md § Scraping conduct).
|
|
- **A refresh is reviewable.** The commit diff is the page diff, so "an event vanished" is a
|
|
question you can answer from git rather than from a wiki that has since changed again.
|
|
- **The build stays offline.** `bun run build:feed` parses whichever of these exists and falls back
|
|
to `fixtures/` otherwise, so a clean checkout and the container build work with no network.
|
|
|
|
The `.state.json` files are the exception: they change every cycle whether or not a page did, and
|
|
committing them would mean a commit per run saying nothing happened. CI keeps them in the actions
|
|
cache instead.
|
|
|
|
Fixtures are not the same thing. A fixture is pinned to a date and kept forever as the regression
|
|
test for a page shape; a snapshot is the current page and is overwritten each time it changes.
|