From adaa9eaadaf277d237786e10fd28449aa2edcd77 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Mon, 17 Aug 2026 18:26:21 +0200 Subject: [PATCH] docs: record which sources were assessed and declined MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five games were surveyed for adapters before the scope changed. Three of those sources are off-limits and the reasons are not visible from the code: Azur Lane's wiki declines AI input outright, uma.moe gates its API behind a Turnstile proof, and Fandom's robots.txt 403s — which fails closed, since a permission we could not read is not a permission we have. Also notes that four of the games named in the release thread have no wiki.gg wiki at all, so nobody repeats that search. Co-Authored-By: Claude Opus 5 (1M context) --- CLAUDE.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 5d21171..60668a4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -191,6 +191,21 @@ rate, and do not add an LLM that consumes page content. A source whose ToS forbids automated access does not get an adapter. Flag it and ask. +**Sources assessed and declined** (2026-08-17), so these are not re-litigated each pass: + +| Source | Verdict | +|---|---| +| `azurlane.koumakan.jp` | **Declined.** `Content-Signal: ai-input=no` — an explicit refusal of collecting content as model input, which is what capturing a fixture to read amounts to. Stronger than game8's or wiki.gg's signal. Find Azur Lane another source | +| `uma.moe` | **Declined.** Data comes from an API behind a Cloudflare Turnstile proof header; an adapter would mean defeating a deliberate access control. The `robots.txt` is permissive, but the gate is not in `robots.txt` | +| `reverse1999.fandom.com` | **Declined for now.** `robots.txt` returns 403, and an unreadable robots means "do not fetch" — a permission we could not read is not a permission we have | +| `bluearchive.wiki`, `prydwen.gg`, `gametora.com` | **Cleared, unbuilt.** `User-agent: *` allows the paths we would want. prydwen sets `Crawl-delay: 10`, far below our one-per-6h | + +wiki.gg hosts (`arknights`, `endfield`) carry `Content-Signal: search=yes, ai-train=no, use=reference` +with `Allow: /`, and disallow `ClaudeBot` and other AI crawlers by name. Our fetcher is neither: it +trains nothing, and no LLM reads the page content — constraint 2 is what keeps that true, so it is +load-bearing here and not only a cost decision. Note also that Reverse: 1999, Blue Archive, +Umamusume and Nikke have **no wiki.gg wiki** — those subdomains 401. + `scripts/refresh-sources.ts` enforces all of the above in code — the 6h floor, one request, no retries, conditional headers, robots (failing closed when `robots.txt` cannot be read). Anything that would make it fetch more often is a change to this section first.