diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 90c2ff5..f58ac7a 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -78,7 +78,7 @@ body: id: where attributes: label: Where are you running it, and in what browser? - placeholder: stereotypicalcat.github.io/gacha-event-tracker — Firefox 141 on Android + placeholder: localhost:3000 (container image) — Firefox 141 on Android validations: required: false diff --git a/AGENTS.md b/AGENTS.md index 3859c55..a66c572 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,7 +97,7 @@ bun run parse endfield-wikigg-events fixtures/endfield/wikigg-events-2026-08-15. bun test test/dates.test.ts bun test --test-name-pattern "year-less" -# Hosting under a subpath (GitHub Pages) +# Hosting under a subpath (CI does not set this — the image serves from /) BASE_PATH=/gacha-event-tracker/ bun run build ``` @@ -348,11 +348,11 @@ AI-training crawlers. Our use is a low-rate personal aggregator with attribution training, and no `User-agent: *` rule applies to our paths. Keep it that way: do not raise the fetch rate, and do not add an LLM that consumes page content. -**game8.co does not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returns +**game8.co did not answer a GitHub Actions runner** (confirmed 2026-08-17). Its edge returned `202 Accepted` with a bot-management body to every one of the nine game8 sources, from the first scheduled cycle onward — `last confirmed: never` — while the same URLs return `200` and parse -cleanly from a normal address. So `robots.txt` permits us and the network does not, and those nine -games have only ever been built from checked-in fixtures in CI. +cleanly from a normal address. So `robots.txt` permitted us and the network did not, and those nine +games had only ever been built from checked-in fixtures in CI. The per-host spacing above does not fix this and was not meant to: a 202 on the very first request of a cycle is address reputation, not rate. **Do not work around it.** Browser-shaped headers, a @@ -361,6 +361,16 @@ same reason `uma.moe` was declined below — and unlike `uma.moe` we would be do `robots.txt` was welcoming, which makes it worse, not better. The legitimate options are to run the refresh from an address game8 will serve, or to find those games another source. +**CI moved to a Gitea runner on 2026-09-16, which is the first of those two options — and its result +is not yet known.** A different address is exactly the variable this failure turns on, so the nine +game8 sources may now refresh on schedule, or may not. Nothing here should be rewritten on the +strength of that hope: the tense above stays past because the GitHub evidence is real and the Gitea +evidence does not exist yet. Establish it the same way it was established before — let a cycle run, +then read `git log` on `snapshots/` for which hosts actually landed — and only then update this +section, `README.md` § Status, `docs/SOURCES.md` § Method and `docs/FEEDBACK.md`'s P0 row together. +A single manual `--only` dispatch is enough to answer it for one source; note that `--dry-run` +cannot, because it makes no requests at all by design. + A source whose ToS forbids automated access does not get an adapter. Flag it and ask. **Sources assessed and declined** (2026-08-17, extended 2026-08-19), so these are not diff --git a/README.md b/README.md index eafe240..e692780 100644 --- a/README.md +++ b/README.md @@ -26,23 +26,33 @@ review queue are specified but not built. | Input sanitization at the ingest boundary | Built, tested | | Scheduled refresh — robots, snapshots, commit-on-change | Built, tested offline | | Web interface, daily checklists, offline support | Built | -| Static server, Docker image, GitHub + GitLab CI | Built | +| Static server, Docker image, Gitea + GitLab CI | Built | | SQLite, review queue | Specified in `docs/`, not built | The feed is still a static JSON file rather than a database read: the refresh job commits the raw pages it fetched, CI rebuilds the feed from them, and a clean checkout with no snapshots falls back to the checked-in fixtures — so the build stays offline and reproducible either way. -**The scheduled refresh does not reach every source.** game8.co answers the GitHub Actions runner -with a bot-management `202` instead of the page, so the nine Game8 sources only move when someone -runs `bun run refresh` by hand from an address it will talk to. The wiki sources refresh on schedule +**The scheduled refresh does not reach every source.** game8.co answered the GitHub Actions runner +with a bot-management `202` instead of the page, so the nine Game8 sources only moved when someone +ran `bun run refresh` by hand from an address it will talk to. The wiki sources refresh on schedule as intended. `docs/SOURCES.md` records which hosts answer the runner and which do not; the footer shows when each lane was last refreshed, so a stale one is visible rather than silent. +Since the move to Gitea the refresh runs from a **different address**, which is one of the two +remedies `AGENTS.md` § Scraping conduct allows for this. Whether game8.co serves that address is +**not yet established** — it needs one real single-source run on the runner, and until that has +happened the paragraph above still describes what we know. + ## Try it -Live at ****, deployed from `main` by the -Pages job below. +Deployed from `main` as a container image, published to +`gitea.lucaswinther.info/lucasw89/gacha-event-tracker:latest` by the CI job below. Run it with +`docker run -p 3000:3000 gitea.lucaswinther.info/lucasw89/gacha-event-tracker:latest` and open +. + +> The previous GitHub Pages deploy at is no +> longer updated by this pipeline; it holds whatever `main` looked like when the repo moved. Or run it yourself: @@ -298,9 +308,9 @@ page is sanitized at the ingest boundary before it reaches the feed, the browser ## Found a problem, or want something? -- **[Report a problem](https://github.com/StereotypicalCat/gacha-event-tracker/issues/new?template=bug_report.yml)** +- **[Report a problem](https://gitea.lucaswinther.info/lucasw89/gacha-event-tracker/issues/new?template=bug_report.yml)** — a wrong date, a missing event, a lost tick, anything the app got wrong. -- **[Request a feature](https://github.com/StereotypicalCat/gacha-event-tracker/issues/new?template=feature_request.yml)** +- **[Request a feature](https://gitea.lucaswinther.info/lucasw89/gacha-event-tracker/issues/new?template=feature_request.yml)** — including a game it does not cover yet, which is the most common ask by a distance. Both are forms rather than a blank box, for one reason: nothing you mark, type or tick ever leaves @@ -331,15 +341,22 @@ Blank issues are still open for anything that fits neither form. The templates t Both `.github/workflows/ci.yml` and `.gitlab-ci.yml` run the same gates on every push — typecheck, tests, and a feed sanity check — then build and publish a container image from the default branch. -GitHub Actions can also deploy to Pages, but that needs two one-time steps it cannot do for itself — -the default `GITHUB_TOKEN` is not allowed to create a Pages site: -1. **Settings → Pages → Source: GitHub Actions.** -2. **Settings → Secrets and variables → Actions → Variables:** add `DEPLOY_PAGES` = `true`. +The workflows run on **Gitea Actions**, not GitHub's. They stay under `.github/workflows/` because +Gitea reads that directory natively, and because `test/refresh.test.ts` asserts on those two files by +path — three defects that live in YAML and fail silently otherwise. Two consequences worth knowing +before editing them: -Until then the `pages` job is skipped and the pipeline stays green. Pages is unavailable for private -repositories on the free plan. Both steps are done here, and the deploy lands at -. +- **`uses:` steps are written as full URLs** (`https://github.com/actions/checkout@v4`). A bare + `actions/checkout@v4` resolves against the instance's `DEFAULT_ACTIONS_URL`, which is configuration + we do not control from here. The runner needs outbound access to github.com to fetch them. +- **The image is the deploy artefact.** There is no `pages` job: Gitea has no Pages equivalent, so + the container image published to Gitea's registry is what gets deployed. `bun run build` therefore + runs without `BASE_PATH` — the image serves from `/`. + +The image lands at `gitea.lucaswinther.info/lucasw89/gacha-event-tracker`, tagged `latest` and with +the commit SHA. The registry host is derived from `github.server_url` rather than hardcoded, so a +fork on another instance pushes to its own registry. The feed job fails if the event count collapses, or if any single source parses to nothing — nine healthy sources hide a tenth that has gone quiet, and the total stays comfortably over the floor @@ -354,13 +371,19 @@ pipeline always means the code changed rather than a wiki being down. ### Refreshing the data -GitHub Actions only — the GitLab pipeline still runs the gates, but nothing there fetches. +Gitea Actions only — the GitLab pipeline still runs the gates, but nothing there fetches. `.github/workflows/refresh.yml` runs `bun run refresh` twice a day (and on demand, with a dry-run input). It fetches each source at most once per cycle, and **commits only when a page's bytes actually changed** — a `304`, an identical body, or a fetch that fails to parse all leave the working tree clean and produce no commit. When something did change it commits the raw snapshots and -dispatches `ci.yml`, which typechecks, tests, rebuilds the feed and deploys through the path that -already existed; none of that logic is duplicated. +dispatches `ci.yml`, which typechecks, tests, rebuilds the feed and republishes the image through the +path that already existed; none of that logic is duplicated. + +That dispatch is a REST call rather than `gh workflow run`, there being no `gh` on a Gitea runner. +It is also, strictly, belt and braces: unlike GitHub, Gitea does **not** suppress workflow triggers +on a push made with the Actions token, so the snapshot commit will usually have started `ci.yml` +already. A duplicate run is harmless — `ci.yml`'s concurrency group is per-ref with +`cancel-in-progress`, so the second supersedes the first. A body that yields zero events is rejected and the previous snapshot kept, so a wiki redesign shows up as a stale timestamp rather than an empty calendar. One source being down is a warning; every diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 61e5732..57c3566 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -37,7 +37,7 @@ are no users. ``` **That is the design, and the scheduler half of it is not built yet.** What runs today is -`scripts/refresh-sources.ts` on a GitHub Actions cron, writing snapshots to disk, and a feed built +`scripts/refresh-sources.ts` on a Gitea Actions cron, writing snapshots to disk, and a feed built from those files rather than from SQLite — see § Today. The stages, the layering and the review gate are unchanged by that; only what wakes them up and where they land is. @@ -230,8 +230,10 @@ gates and publish it. Assets resolve against a `` substituted at build time, the feed URL resolves against `document.baseURI` so deep links work, and the service worker derives its paths from its own -registration scope. `BASE_PATH=/gacha-event-tracker/ bun run build` for GitHub Pages; without it a -subpath deploy 404s on every asset. +registration scope. `BASE_PATH=/gacha-event-tracker/ bun run build` for any host that serves the app +from a subpath; without it a subpath deploy 404s on every asset. CI no longer sets it — the container +image is what gets deployed and `serve.ts` serves from `/` — but the mechanism stays, because a +subpath deploy is a hosting decision rather than a build one. ### Offline @@ -263,9 +265,11 @@ mode of the `fetch()` that follows it, and a mismatch is not a no-op — the bro preload and fetches the feed a second time, which is worse than not preloading. If that link is ever edited, count the feed requests in a real browser; nothing in the test suite can see this. -Compression is the server's job and both hosts do it: GitHub Pages transparently, and `serve.ts` — -which is what the Docker image runs — by negotiating `accept-encoding` per request. It matters more -than it sounds: the bundle is 344 KB raw and 100 KB gzipped. +Compression is the server's job, and `serve.ts` — which is what the Docker image runs, and so what +serves the deployed site — does it by negotiating `accept-encoding` per request. It matters more +than it sounds: the bundle is 344 KB raw and 100 KB gzipped. A static host put in front of the built +`public/` instead must do its own; GitHub Pages did this transparently while it was the deploy +target, which is why nothing here ever had to. ### The theme, before the bundle arrives diff --git a/docs/FEEDBACK.md b/docs/FEEDBACK.md index 584f8f6..ade479a 100644 --- a/docs/FEEDBACK.md +++ b/docs/FEEDBACK.md @@ -299,7 +299,7 @@ diagnosis in each item still holds — what changed is whether it has been acted | Item | Status | |---|---| -| P0 refresh pipeline | **Diagnosed, half acted on.** game8.co answers a GitHub Actions runner with `202` and a bot-management body, so those eight sources have only ever built from fixtures in CI — see `AGENTS.md` § Scraping conduct, including why it is not to be worked around. The `broken` tier now makes a source failing three cycles fail the run. Step 5 (a build assertion on snapshot age) is **not built** | +| P0 refresh pipeline | **Diagnosed, half acted on.** game8.co answered a GitHub Actions runner with `202` and a bot-management body, so those eight sources had only ever built from fixtures in CI — see `AGENTS.md` § Scraping conduct, including why it is not to be worked around. CI moved to a Gitea runner on 2026-09-16, which changes the address this failure turns on; whether game8 serves it is **unverified**, and the row stays as-is until a cycle has run. The `broken` tier now makes a source failing three cycles fail the run. Step 5 (a build assertion on snapshot age) is **not built** | | P1a Arknights | **Done.** `arknights-akwiki-events`, via the new `akwiki` parser | | P1b `NextUp` → three | **Done** (2026-08-18). One headline and two behind it, off `nextToExpire` | | P1b cap the long list | **Done** (2026-08-18). Both sections cap at six with "show all N" — truncation of the view, not a re-sort | diff --git a/docs/INGESTION.md b/docs/INGESTION.md index 7178ce7..475386a 100644 --- a/docs/INGESTION.md +++ b/docs/INGESTION.md @@ -318,7 +318,7 @@ on the section and leave every `h4` beneath it unable to take it back — publis - Send `If-None-Match` / `If-Modified-Since` from `sources.etag` / `last_modified`. A `304` ends the run as `skipped_unchanged`. -- `User-Agent: gacha-event-tracker/1.0 (+https://github.com//gacha-event-tracker)`. +- `User-Agent: gacha-event-tracker/1.0 (+https://gitea.lucaswinther.info//gacha-event-tracker)`. - Honor `robots.txt`; cache parsed robots per host for 24h. **Fail closed** — a `robots.txt` that 5xxs or times out means "do not fetch", because a permission we could not read is not a permission we have. A 404 means no restrictions. diff --git a/docs/SOURCES.md b/docs/SOURCES.md index 59c4752..1365f46 100644 --- a/docs/SOURCES.md +++ b/docs/SOURCES.md @@ -27,23 +27,33 @@ the ruling. ## Method Every page below was fetched from this machine with the refresh runner's own `User-Agent` -(`gacha-event-tracker/1.0 (+https://github.com/StereotypicalCat/gacha-event-tracker)`), one or two -requests per host, `robots.txt` first. No browser-shaped headers, no proxy, no JS execution — +(`gacha-event-tracker/1.0 (+https://github.com/StereotypicalCat/gacha-event-tracker)` — the contact +URL at the time; the repo has since moved to Gitea and the runner now sends that address instead), +one or two requests per host, `robots.txt` first. No browser-shaped headers, no proxy, no JS execution — anything that only answers a browser is treated as closed, per the `uma.moe` precedent. Where an existing parser could plausibly read a page, it was run against the fetched bytes offline (`src/ingest/parsers/index.ts`) rather than guessed at. Two things this method cannot tell you, and both matter: -- **Whether a host answers the GitHub Actions runner.** game8.co does not (`AGENTS.md` § Scraping - conduct). The only hard evidence we hold is what CI has actually committed: `git log` on - `snapshots/` shows `github-actions[bot]` landing **arknights.wiki.gg**, **endfield.wiki.gg** and - **bluearchive.wiki** (Miraheze). Fandom sources have never refreshed in CI, because `robots.txt` - itself is challenged from a datacentre address and the gate fails closed — and as of 2026-08-19 - that challenge covers every Fandom wiki from *any* address tried here, not just CI's. - Of the 2026-08-19 additions, **Stella Sora** is Miraheze and so is the one likely to refresh in CI; - **IOP Wiki** is permissive but unproven; **Chaos Zero Nightmare** and **Umamusume** are game8 and - therefore blind in CI by construction. +- **Whether a host answers the CI runner.** This is a property of the runner's address, not of the + page, so it has to be re-established whenever that address changes — and it just did: CI moved from + GitHub Actions to a Gitea runner (`AGENTS.md` § Scraping conduct). **Everything in this bullet + below is evidence about the old GitHub address.** Treat it as the prior, not the current state, + until a cycle has run on Gitea. + + game8.co did not answer the GitHub runner. The only hard evidence we held was what CI had actually + committed: `git log` on `snapshots/` shows `github-actions[bot]` landing **arknights.wiki.gg**, + **endfield.wiki.gg** and **bluearchive.wiki** (Miraheze). Fandom sources never refreshed in CI, + because `robots.txt` itself is challenged from a datacentre address and the gate fails closed — and + as of 2026-08-19 that challenge covered every Fandom wiki from *any* address tried here, not just + CI's. Of the 2026-08-19 additions, **Stella Sora** is Miraheze and so was the one likely to refresh + in CI; **IOP Wiki** is permissive but unproven; **Chaos Zero Nightmare** and **Umamusume** are + game8 and were therefore blind in CI by construction. + + The Gitea runner may well be served where GitHub's was not — that is one of the two remedies + `AGENTS.md` allows — but "may well be" is not evidence. The check is the same one that produced the + list above: run a cycle, then read `git log` on `snapshots/` for what actually landed. - **Whether a page will still look like this in six weeks.** Each recommendation below names the assertion `canParse` should make, so a redesign fails the source loudly instead of emptying a lane. @@ -836,7 +846,7 @@ if it ever prints a year (§ 13). None is worth a pass until something changes u ## Appendix — reproducing the checks ```bash -UA='gacha-event-tracker/1.0 (+https://github.com/StereotypicalCat/gacha-event-tracker)' +UA='gacha-event-tracker/1.0 (+https://gitea.lucaswinther.info/lucasw89/gacha-event-tracker)' # conduct curl -sS -A "$UA" https://iopwiki.com/robots.txt diff --git a/serve.ts b/serve.ts index c2b3b07..3292947 100644 --- a/serve.ts +++ b/serve.ts @@ -23,10 +23,12 @@ const ROOT_DIR = resolve(ROOT); * times the bytes and, on anything slower than a laptop on wifi, three times the * download. * - * GitHub Pages compresses for us, so the deployed site never had this problem; - * the Docker image serves through this file and did. An image whose payload is - * 3x the site's is not a placeholder detail — it is the only thing a self-hoster - * ever sees. + * This used to be the self-hoster's problem only: GitHub Pages compressed for + * us, so the deployed site never had it, while the Docker image serves through + * this file and did. Since the move to Gitea the image *is* the deploy, so this + * is now the only compression the deployed site gets — there is no host behind + * it doing the job silently. Serving 3x the bytes is not a placeholder detail + * when it is what every reader gets. */ const COMPRESSIBLE = new Set([ ".html", diff --git a/src/ingest/robots.ts b/src/ingest/robots.ts index 40b2526..f2b1c5a 100644 --- a/src/ingest/robots.ts +++ b/src/ingest/robots.ts @@ -142,12 +142,15 @@ export function agentToken(userAgent: string): string { * * RFC 9309 § 2.2.1 matches the *product token* — the header up to the first * `/` — not the header text. Matching anywhere in the header is actively - * dangerous here: our contact URL contains the string `StereotypicalCat`, so a - * `User-agent: cat` group elsewhere in the file would be treated as naming us, - * and because a named group replaces the `*` group outright, that unrelated - * group's rules would *discard* every rule the site actually wrote for us. - * Erring towards obeying more rules means never letting a coincidence take a - * `*` group away. + * dangerous here, because our header carries a contact URL and a URL is a long + * string of arbitrary substrings. The case that caught it: the contact URL was + * once `…/StereotypicalCat/…`, so a `User-agent: cat` group elsewhere in the + * file was treated as naming us, and because a named group replaces the `*` + * group outright, that unrelated group's rules *discarded* every rule the site + * actually wrote for us. The URL has since changed and that particular + * coincidence is gone; the hazard is not, since the next contact URL is just as + * arbitrary. Erring towards obeying more rules means never letting a + * coincidence take a `*` group away. * * A robots.txt that names us with a version (`gacha-event-tracker/1.0`) is * still honoured: the group's own product token is compared too. diff --git a/test/serve.test.ts b/test/serve.test.ts index ef8a915..16555e6 100644 --- a/test/serve.test.ts +++ b/test/serve.test.ts @@ -117,12 +117,13 @@ describe("static server", () => { }); /** - * Compression, which is the whole difference between the Docker image and the - * deployed site. + * Compression, which used to be the whole difference between the Docker image + * and the deployed site. * - * GitHub Pages gzips on our behalf, so the bundle crosses the wire at a third of - * its size there and did not here — and this file is what the image runs. Three - * times the bytes is the only thing a self-hoster would ever have seen. + * GitHub Pages gzipped on our behalf, so the bundle crossed the wire at a third + * of its size there and did not here — and serve.ts is what the image runs. + * Since the move to Gitea the image is the deploy, so these assertions cover + * every reader rather than only a self-hoster: nothing else compresses now. */ describe("static server: compression", () => { test("gzips a text asset for a client that asks", async () => {