fix: harden the fetch pipeline for a real server

A review of code that has never made a live request. Each of these was
reproduced before it was fixed.

robots.txt matched a group by testing whether our whole User-Agent
contained the group's name. Our contact URL carries "StereotypicalCat",
so a wiki writing `User-agent: cat` matched us — and because a named
group *replaces* the wildcard group, that silently discarded every rule
under `User-agent: *`. Match the RFC 9309 product token instead. A 200
carrying an HTML "not found" page also parsed to zero rules and read as
permission; it now fails closed, while a genuinely empty body still
means "no restrictions".

The response body was read outside the try that guarded the request, so
one truncated body aborted the whole cycle: later sources were never
fetched, and the failed source never recorded its check, meaning a
re-dispatch would ask that wiki again minutes later.

Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A
page served in a legacy charset became replacement characters with the
original bytes gone — and mojibake in a title flows into slugify and
moves every localStorage key for that source. Decode by the declared
charset, keep the served bytes verbatim, and hash those.

Unchanged bytes skipped the metadata write, so once a server rotated its
ETag we sent a stale validator forever and it served full bodies instead
of 304s.

Also: snapshot writes go through a temp file and a rename, and `--only`
with no value is an argument error rather than "all sources".

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

snapshots/README.md

@@ -4,10 +4,21 @@ Raw pages, exactly as fetched. `scripts/refresh-sources.ts` writes them; nothing
 
 ```
 <source-id>.html        the body verbatim — tracked
-<source-id>.meta.json   hash, size, ETag, Last-Modified, when the bytes last changed — tracked
+<source-id>.meta.json   hash, size, charset, ETag, Last-Modified, when the bytes last changed — tracked
 <source-id>.state.json  when we last checked, and failure streak — gitignored
 ```
 
+"Verbatim" means the bytes as served, not text we re-encoded. `charset` in the metadata records
+what those bytes are in — the `Content-Type` header, else a `<meta charset>` in the page, else
+UTF-8 — and `bytes` is the served length. A page in Shift_JIS or Latin-1 decoded as UTF-8 would be
+stored as a field of U+FFFD with the original bytes gone; re-parsing could never recover it, and the
+mojibake would reach `slugify`, moving every event ID for that source (CLAUDE.md § Event IDs are
+localStorage keys).
+
+Every file is written to a sibling `.tmp-*` and renamed into place, body before metadata, so an
+interrupted run leaves a stray temp file rather than a truncated snapshot or metadata describing
+bytes that were never stored.
+
 Three reasons this is committed rather than cached:
 
 - **Re-parsing never re-fetches.** Iterating on a parser reads these files, not the wikis
@@ -19,7 +30,15 @@ Three reasons this is committed rather than cached:
 
 The `.state.json` files are the exception: they change every cycle whether or not a page did, and
 committing them would mean a commit per run saying nothing happened. CI keeps them in the actions
-cache instead.
+cache instead: `refresh.yml` saves that cache, and `ci.yml` restores it read-only before building
+the feed. Both halves matter — `lastConfirmedAt` lives only there, and without the restore the feed
+falls back to `contentChangedAt` and the UI calls every source stale two days after its bytes last
+moved.
+
+The metadata is rewritten on an unchanged page in one case: the server rotating an `ETag` or
+`Last-Modified` while serving the same bytes. Keeping the old validator would mean sending a stale
+`If-None-Match` forever and being served the whole page every cycle, so that diff is worth the
+commit — `contentChangedAt` and the body stay put, so it is still visibly not a content change.
 
 Fixtures are not the same thing. A fixture is pinned to a date and kept forever as the regression
 test for a page shape; a snapshot is the current page and is overwritten each time it changes.