fix(refresh): name assumed hosts on article reads, and keep times on a bare 404
Also corrects the doc sentences the feature made false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
24327ea99b
commit
4ee1bf2d2a
@@ -134,9 +134,11 @@ scripts/
|
||||
build-feed.ts fixtures → public/data/events.v1.json ✓ built
|
||||
build-static.ts shell + worker into public/, build-stamped ✓ built
|
||||
parse-fixture.ts run one adapter offline ✓ built
|
||||
refresh-sources.ts fetch, cache, rebuild — the only network ✓ built
|
||||
refresh-sources.ts fetch, cache, rebuild ✓ built
|
||||
refresh-articles.ts Genshin event articles, for exact ends ✓ built
|
||||
fixtures/<game>/ checked-in raw HTML + expected parse output
|
||||
snapshots/ the current page per source, rewritten by refresh
|
||||
articles/ Genshin event articles' lead sections, rewritten by refresh
|
||||
```
|
||||
|
||||
## Request paths
|
||||
@@ -214,8 +216,8 @@ changing. Reads are confined to `public/` by resolving the path and checking it
|
||||
— string-matching `..` is not enough, because encodings and URL normalisation both change what the
|
||||
string looks like.
|
||||
|
||||
`scripts/refresh-sources.ts` (`bun run refresh`) is what fills `snapshots/`, and it is the only code
|
||||
here that touches the network. `.gitea/workflows/refresh.yml` runs it at 05:27 and 17:27 UTC and
|
||||
`scripts/refresh-sources.ts` (`bun run refresh`) is what fills `snapshots/`, and with
|
||||
`scripts/refresh-articles.ts`, which it calls, it is the only code here that touches the network. `.gitea/workflows/refresh.yml` runs it at 05:27 and 17:27 UTC and
|
||||
commits only when a page actually changed. It stands in for the unbuilt scheduler and enforces the
|
||||
same conduct in code — the 6h floor, one request, no retries, conditional headers, per-host spacing,
|
||||
robots failing closed. A source that has failed `BROKEN_AFTER_FAILURES` (3) cycles running is
|
||||
|
||||
+5
-3
@@ -754,11 +754,13 @@ keeps it listed, and an unlisted article is deleted.
|
||||
|
||||
| Result of a read | Stored article | Row shows |
|
||||
|---|---|---|
|
||||
| Network error, non-200 other than 404, robots refusal, or a 200 that is not the API's answer | Kept | The last good time, if any |
|
||||
| 404 or `missingtitle` | Body deleted; metadata kept for the 6-hour floor | Fallback |
|
||||
| Network error, any non-200 including 404, robots refusal, or a 200 that is not the API's answer | Kept | The last good time, if any |
|
||||
| The API's `missingtitle`, which it sends with a 200 | Body deleted; metadata kept for the 6-hour floor | Fallback |
|
||||
| A real article with no usable End | Overwritten | Fallback, logged but not warned: stubs are ordinary |
|
||||
|
||||
None of these touches a source's failure streak, health or broken count.
|
||||
None of these touches a source's failure streak, health or broken count. A read on an assumed robots
|
||||
permission names its host in the run's warning like any page does, even when the table itself was
|
||||
skipped for its interval.
|
||||
|
||||
**Parse.** `parseArticleEnd` reads only the `<td data-source="time_end">` cell, in two shapes:
|
||||
`December 15, 2026 14:59 (Server Time)` gives `regionEnds` at each server's clock, with `endsAt` the
|
||||
|
||||
@@ -121,8 +121,8 @@ Approved with this document unless marked as the agent's call.
|
||||
|
||||
| Result of a read | Stored article | Row shows |
|
||||
|---|---|---|
|
||||
| Network error, 5xx, refused, or robots-blocked: the wiki told us nothing about the article | Kept untouched | The last good time, if any. Due again after the 6-hour floor |
|
||||
| 404 or a missing-page API error: the article is gone or renamed | Deleted | Fallback |
|
||||
| Network error, any non-200 including 404, refused, or robots-blocked: the wiki told us nothing about the article | Kept untouched | The last good time, if any. Due again after the 6-hour floor |
|
||||
| The API's `missingtitle`, sent with a 200: the article is gone or renamed. Changed from "404" in final review: a bare 404 is an edge or outage, and one would wipe every row's time | Deleted | Fallback |
|
||||
| `200` whose End value is absent or unusable: the article no longer states a time | Overwritten | Fallback. Due again under the no-time rule |
|
||||
|
||||
`.meta.json` records the last attempt separately from the last successful read, so the
|
||||
|
||||
Reference in New Issue
Block a user