fix: harden the fetch pipeline for a real server

A review of code that has never made a live request. Each of these was reproduced before it was fixed. robots.txt matched a group by testing whether our whole User-Agent contained the group's name. Our contact URL carries "StereotypicalCat", so a wiki writing `User-agent: cat` matched us — and because a named group *replaces* the wildcard group, that silently discarded every rule under `User-agent: *`. Match the RFC 9309 product token instead. A 200 carrying an HTML "not found" page also parsed to zero rules and read as permission; it now fails closed, while a genuinely empty body still means "no restrictions". The response body was read outside the try that guarded the request, so one truncated body aborted the whole cycle: later sources were never fetched, and the failed source never recorded its check, meaning a re-dispatch would ask that wiki again minutes later. Bodies were decoded as UTF-8 unconditionally and stored re-encoded. A page served in a legacy charset became replacement characters with the original bytes gone — and mojibake in a title flows into slugify and moves every localStorage key for that source. Decode by the declared charset, keep the served bytes verbatim, and hash those. Unchanged bytes skipped the metadata write, so once a server rotated its ETag we sent a stale validator forever and it served full bodies instead of 304s. Also: snapshot writes go through a temp file and a rename, and `--only` with no value is an argument error rather than "all sources". Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-15 21:53:34 +02:00
parent 99786f6fc9
commit c85ec0b2d1
7 changed files with 820 additions and 48 deletions
@@ -77,6 +77,48 @@ Disallow: /admin
    expect(isAllowed(robots, UA, "/games/x")).toBe(true);
  });

+  test("an unrelated named group cannot steal the wildcard group", () => {
+    // Our contact URL contains "StereotypicalCat". Matching a group name
+    // anywhere in the header made `User-agent: cat` look like our group, and
+    // because a named group replaces `*`, that coincidence *discarded* the
+    // rules the site actually wrote for everyone.
+    const ua =
+      "gacha-event-tracker/1.0 (+https://github.com/StereotypicalCat/gacha-event-tracker)";
+    const robots = parseRobots(`
+User-agent: *
+Disallow: /games/
+
+User-agent: cat
+Disallow: /litter
+`);
+    expect(groupFor(robots, ua)?.agents).toEqual(["*"]);
+    expect(isAllowed(robots, ua, "/games/x")).toBe(false);
+    expect(isAllowed(robots, ua, "/litter")).toBe(true);
+  });
+
+  test("a substring of the product token does not name us either", () => {
+    const robots = parseRobots(`
+User-agent: *
+Disallow: /games/
+
+User-agent: gacha
+Allow: /
+`);
+    expect(isAllowed(robots, UA, "/games/x")).toBe(false);
+  });
+
+  test("a group naming us with a version still binds", () => {
+    const robots = parseRobots(`
+User-agent: *
+Disallow: /
+
+User-agent: gacha-event-tracker/1.0
+Disallow: /admin
+`);
+    expect(isAllowed(robots, UA, "/games/x")).toBe(true);
+    expect(isAllowed(robots, UA, "/admin/panel")).toBe(false);
+  });
+
  test("the longest matching agent name wins", () => {
    const robots = parseRobots(`
 User-agent: googlebot
@@ -227,6 +269,47 @@ describe("RobotsCache", () => {
    expect(decision.reason).toContain("503");
  });

+  test("an empty 200 body is a valid robots.txt that restricts nothing", async () => {
+    const { cache } = cacheWith(() => new Response("   \n", { status: 200 }));
+    const decision = await cache.allows("https://x.test/wiki/Event");
+    expect(decision.allowed).toBe(true);
+    expect(decision.reason).toBe("robots.txt ok");
+  });
+
+  test("fails closed on a soft 404 — an HTML page served as 200", async () => {
+    // The commonest robots.txt misconfiguration there is. It parses to zero
+    // groups, which is indistinguishable from "nothing is restricted", so
+    // reading it as permission is exactly the fail-open this class forbids.
+    const { cache } = cacheWith(
+      () =>
+        new Response(
+          "<!DOCTYPE html>\n<html><head><title>404 Not Found</title></head><body>Not found</body></html>",
+          { status: 200, headers: { "Content-Type": "text/html" } },
+        ),
+    );
+    const decision = await cache.allows("https://x.test/wiki/Event");
+    expect(decision.allowed).toBe(false);
+    expect(decision.reason).toContain("HTML");
+  });
+
+  test("fails closed when the body dies mid-read", async () => {
+    const { cache } = cacheWith(
+      () =>
+        new Response(
+          new ReadableStream({
+            start(controller) {
+              controller.enqueue(new TextEncoder().encode("User-agent: *\n"));
+              controller.error(new Error("ECONNRESET"));
+            },
+          }),
+          { status: 200 },
+        ),
+    );
+    const decision = await cache.allows("https://x.test/wiki/Event");
+    expect(decision.allowed).toBe(false);
+    expect(decision.reason).toContain("unreadable");
+  });
+
  test("fails closed when robots.txt is unreachable", async () => {
    const { cache } = cacheWith(() => {
      throw new Error("ECONNREFUSED");