From b2eb621ebbf787333a111b1a68cdb27922528e4f Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Thu, 15 Oct 2026 01:50:01 +0200 Subject: [PATCH] docs: remove the finished plan (the spec is the record) Co-Authored-By: Claude Opus 5.5 --- docs/plans/001-webtoons-covers.md | 6088 ----------------------------- 1 file changed, 6088 deletions(-) delete mode 100644 docs/plans/001-webtoons-covers.md diff --git a/docs/plans/001-webtoons-covers.md b/docs/plans/001-webtoons-covers.md deleted file mode 100644 index 1a16f9a..0000000 --- a/docs/plans/001-webtoons-covers.md +++ /dev/null @@ -1,6088 +0,0 @@ ---- -status: draft ---- - -# Webtoons covers Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build `mangabaka-seeder` v0.1: given a MangaBaka series link and a Webtoons series link, find every current and historical Webtoons cover, check and deduplicate them against MangaBaka, rehost them on request, and hand them over as copyable entries or (with a token) as a previewed edit request for review — through a static page served by a self-hosted Bun server, and through a userscript. - -**Architecture:** A pure TypeScript core (`src/core/`) does all the work through an injected `Transport`, so the same code runs in the page (via our `/api/fetch` proxy), in the userscript (via `GM_xmlhttpRequest`) and in tests (via a fake transport serving captured fixtures). `src/core/session.ts` holds the review state both front ends share. The server is one web-standard handler (`src/server/app.ts`) run by `Bun.serve` (`src/server/bun.ts`), with covers stored in a size-capped folder. - -**Tech Stack:** Bun 1.3 (runtime, `bun test`, `bun build`), TypeScript 5 (strict, `tsc --noEmit`), no runtime dependencies. Dev dependencies: `typescript`, `@types/bun`. - -**Spec:** `docs/specs/001-webtoons-covers.md` (approved 2026-10-14, amended the same day after testing against the real services). Read it before Task 1; this plan argues from it and quotes it. - -## Global Constraints - -- Runtime and tooling are **Bun**: `bun`, `bun test`, `bun run …`. Never `npm`, `npx`, `node`, `vitest`, `jest`. -- **No runtime dependencies.** HTML is parsed with small regex helpers (`src/core/html.ts`), not a DOM library. Adding any dependency needs the user's approval (AGENTS.md). -- `src/core/` uses no DOM globals (`document`, `window`, `localStorage`), no `Bun.*`, no `node:*`, no `GM_*`, and imports nothing from `web/`, `server/`, `userscript/`, `cli/`. Hashing is `crypto.subtle`. Enforced by `test/boundaries.test.ts` (Task 1). -- `src/server/app.ts`, `storage.ts`, `rate-limit.ts`, `config.ts`, `assets-path.ts` use only web-standard APIs; Bun/`node:*` only in `bun.ts` and `storage-dir.ts`. -- **Tests never touch the network.** Fixtures in `test/fixtures/` are captured from real responses (Task 2), never hand-written. The two MangaBaka responses we cannot capture without a token (snapshot, preview) are built inline in the tests from the OpenAPI schema, marked as such, and listed in `docs/TODO.md` for capture. -- Never call the real MangaBaka save endpoint (`POST /v0/my/submissions/…` without `/preview`), never send a real token, never upload to real litterbox/catbox while implementing. The manual smoke test (Task 23) stops before submitting. -- MangaBaka saves always use `save_mode: "review"`; the `images` array is the **whole** set; exactly one image has `is_default: true` when the set is non-empty. -- Webtoons image URLs are normalised to `https://swebtoon-phinf.pstatic.net` with the query stripped (from `webtoon-phinf.pstatic.net`, `swebtoon.phinf.naver.net`, `webtoon.phinf.naver.net`). -- Wayback gets the honest User-Agent `mangabaka-seeder/ (+https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder)` and at most one request per 10 s; Webtoons gets a browser User-Agent. -- Retries (2, with backoff, honouring `Retry-After`) only for `GET`/`HEAD`. -- Per-image download cap: 20 MB (`20 * 1024 * 1024`). -- History granularity: every 1, 3, 6 or 12 months; default **6**. -- Classification by h/w: > 1.6 tall (`other`, not selected, "landing-page art"); 1.15–1.6 `volume`; 0.85–1.15 `other`; < 0.5 `banner`; otherwise `other`. -- Server env vars: `MBSEED_PORT` (8787), `MBSEED_HOST` (127.0.0.1), `MBSEED_PUBLIC_URL`, `MBSEED_STORAGE_DIR`, `MBSEED_STORAGE_MAX_BYTES` (1073741824), `MBSEED_TRUST_PROXY` (`1` = trust `X-Forwarded-For`), `MBSEED_ASSETS_DIR` (`dist/web`). Covers expire 30 days after their last write; full store answers `507`. -- Deferred, do not build: password gate (spec §4.11), Cloudflare `worker.ts`, CLI, userscript form-filling (see Task 22 note), per-IP upload quotas. -- Commits: Conventional Commits, small, tests green at each, ending with `Co-Authored-By: Claude Opus 5.5 `. Branch: `feat/webtoons-covers`. Never commit to `main`. -- Docs duties in the same commit as the code (AGENTS.md): `docs/ARCHITECTURE.md` when modules land, `docs/DECISIONS.md` for non-obvious choices, `docs/TODO.md` for anything deliberately not done. - -## Review Focus - -Inputs the spec implies but its happy path doesn't exercise, most likely to bite first. Each has a test in the task named. - -1. **Wayback prefix matches other series.** CDX `matchType=prefix` on `…?title_no=95` also returns `title_no=950`, `951`… Expect those rows dropped before snapshots are picked. → Task 9 (`waybackHistory drops other title_no rows`). -2. **A proxied redirect leaves the allowlist.** An allowlisted host answers 302 to `https://evil.example/`. Expect `/api/fetch` to return 403 without fetching it. → Task 17 (`redirect to a non-allowlisted host is refused`). -3. **A retried POST files a second edit request.** MangaBaka answers 502 to the save. Expect exactly one POST, the 502 surfaced. → Task 3 (`never retries POST`). -4. **Spoofed client IP.** Behind the reverse proxy, a client sends its own `X-Forwarded-For: 1.2.3.4`; the proxy appends the real IP. Expect the rate limiter to key on the **last** entry, and to ignore the header entirely when `MBSEED_TRUST_PROXY` is unset. → Task 16 (`clientIp`). -5. **Path traversal into the server's files.** `GET /..%2f.env`, `/%2e%2e/package.json`, `/covers/../../x`. Expect 404 for all; only files under `dist/web/` and names matching `^[0-9a-f]{64}\.(jpg|png|gif|webp)$` are served. → Task 16 (`resolveAssetPath`) and Task 17 (`covers names`). - -Also pinned (lower likelihood): MangaBaka images paginated over two pages (Task 10); same image under two URLs, e.g. canvas `og:image` = `twitter:image` (Task 8); a user-supplied Webtoons slug that doesn't exist hangs Webtoons — we always fetch the `/-/-/-/list?title_no=N` form, which 301s to the canonical page (Task 6); two concurrent uploads racing past the size cap (Task 15). - ---- - -## File structure - -``` -package.json, tsconfig.json, bun.lock Task 1 -scripts/capture-fixtures.sh Task 2 — re-captures test/fixtures (network; run by hand) -scripts/build-userscript.ts Task 21 -src/core/ - version.ts VERSION, REPO_URL - model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, …) - transport.ts Transport, withPolicy (rate limit, timeout, retries, token scope), User-Agents - links.ts LinkError, parseLink, parseMangaBakaSeriesId - html.ts decodeEntities, parseAttributes, findTags, metaContent, textOf - images.ts probeImage, sha256Hex, shapeOf, typeForShape, readCapped, inspectCandidate, inspectAll - covers.ts mergeCandidates, mergeByUrl, dedupeByHash, splitExisting, pickDefault, coverNote - pipeline.ts runCovers — the only place the stages are composed - session.ts Session — review state shared by the page and the userscript - sources/source.ts Source, SourceContext, SourceError - sources/registry.ts SOURCES, findSource, allowedFetchHosts - sources/webtoons/urls.ts URL parsing, language map, image URL normalisation - sources/webtoons/parse.ts HTML → { title, canonical, canvas, mature, covers } - sources/webtoons/index.ts the Webtoons Source - history/wayback.ts CDX, granularity buckets, snapshots → candidates - mangabaka/types.ts API types (subset of the OpenAPI spec) - mangabaka/client.ts MangaBakaClient, MangaBakaError, coverFormUrl - seed/covers.ts buildCoverSeed, withHostedUrls, manualEntryText - hosting/host.ts ImageHost, HostContext, HostError, viaServer - hosting/direct.ts, litterbox.ts, catbox.ts, server.ts - hosting/registry.ts HOSTS - hosting/rehost.ts rehostAll -src/server/ - storage.ts Storage, MemoryStorage, COVER_NAME, StorageFullError - storage-dir.ts DirStorage (folder, cap, sweep) — Bun only - rate-limit.ts RateLimiter - config.ts readConfig, clientIp - assets-path.ts resolveAssetPath - app.ts createApp(env) → (Request) => Promise - bun.ts Bun.serve entry -src/web/ - index.html, styles.css, main.ts the page (design via impeccable, Task 19) - transport.ts pageTransport - view-model.ts pure helpers shared with the userscript panel - render.ts DOM rendering of the review list (shared with the userscript) -src/userscript/ - gm.ts GM types, gmTransport, parseHeaders - main.ts panel on mangabaka.org series pages -test/ - helpers/fake-transport.ts, helpers/fixtures.ts - fixtures/{webtoons,wayback,mangabaka,images}/… - *.test.ts -``` - ---- - -## Part A — Core - -### Task 1: Bun scaffold, boundary test, verified commands - -**Files:** -- Create: `package.json`, `tsconfig.json`, `src/core/version.ts`, `test/boundaries.test.ts`, `test/version.test.ts` -- Modify: `AGENTS.md` (Commands section), `docs/PERMISSIONS.md`, `.claude/settings.json`, `.gemini/settings.json` - -**Interfaces:** -- Produces: `VERSION: string`, `REPO_URL: string` from `src/core/version.ts`; scripts `test`, `typecheck`, `build:web`, `build:userscript`, `build`, `start`. - -- [ ] **Step 1: Create `package.json`** - -```json -{ - "name": "mangabaka-seeder", - "version": "0.1.0", - "private": true, - "type": "module", - "scripts": { - "test": "bun test", - "typecheck": "tsc --noEmit", - "build:web": "bun build ./src/web/index.html --outdir dist/web --minify", - "build:userscript": "bun run scripts/build-userscript.ts", - "build": "bun run build:web && bun run build:userscript", - "start": "bun run src/server/bun.ts" - } -} -``` - -- [ ] **Step 2: Add the dev dependencies** (asks for approval — `bun add` is in the Ask tier) - -Run: `bun add -d typescript @types/bun` -Expected: `bun.lock` created, `devDependencies` added to `package.json`. - -- [ ] **Step 3: Create `tsconfig.json`** - -```json -{ - "compilerOptions": { - "target": "ES2022", - "module": "ESNext", - "moduleResolution": "bundler", - "lib": ["ES2022", "DOM", "DOM.Iterable"], - "types": ["bun"], - "strict": true, - "noUncheckedIndexedAccess": true, - "noImplicitOverride": true, - "verbatimModuleSyntax": true, - "resolveJsonModule": true, - "skipLibCheck": true, - "noEmit": true - }, - "include": ["src", "test", "scripts"] -} -``` - -- [ ] **Step 4: Write the failing tests** - -`test/version.test.ts`: -```ts -import { expect, test } from "bun:test"; -import pkg from "../package.json"; -import { VERSION } from "../src/core/version"; - -test("core VERSION matches package.json", () => { - expect(VERSION).toBe(pkg.version); -}); -``` - -`test/boundaries.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { Glob } from "bun"; - -// AGENTS.md: src/core runs in a browser, a userscript and Bun. -const CORE_FORBIDDEN: [RegExp, string][] = [ - [/\bdocument\b|\bwindow\b|\blocalStorage\b|\bsessionStorage\b/, "DOM global"], - [/\bBun\./, "Bun API"], - [/from\s+["']node:|import\(["']node:/, "node:* import"], - [/\bGM_\w+|\bGM\./, "GM_* API"], - [/from\s+["'](?:\.\.\/)+(?:web|server|userscript|cli)\//, "import from a front end"], -]; -// spec §4: these server files stay web-standard so a Workers entry can be added later. -const PORTABLE_SERVER = ["app.ts", "storage.ts", "rate-limit.ts", "config.ts", "assets-path.ts"]; -const PORTABLE_FORBIDDEN: [RegExp, string][] = [ - [/\bBun\./, "Bun API"], - [/from\s+["']node:|import\(["']node:/, "node:* import"], -]; - -async function violations(pattern: string, rules: [RegExp, string][], only?: string[]) { - const problems: string[] = []; - for await (const file of new Glob(pattern).scan(".")) { - if (only && !only.some((name) => file.endsWith(`/${name}`))) continue; - const text = await Bun.file(file).text(); - for (const [re, what] of rules) if (re.test(text)) problems.push(`${file}: ${what}`); - } - return problems; -} - -test("src/core uses no platform APIs and imports no front end", async () => { - expect(await violations("src/core/**/*.ts", CORE_FORBIDDEN)).toEqual([]); -}); - -test("portable server files use only web-standard APIs", async () => { - expect(await violations("src/server/*.ts", PORTABLE_FORBIDDEN, PORTABLE_SERVER)).toEqual([]); -}); -``` - -- [ ] **Step 5: Run to verify the version test fails** - -Run: `bun test` -Expected: FAIL — `Cannot find module '../src/core/version'`. (The boundary tests pass: no files yet.) - -- [ ] **Step 6: Create `src/core/version.ts`** - -```ts -export const VERSION = "0.1.0"; -export const REPO_URL = "https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder"; -``` - -- [ ] **Step 7: Run tests and typecheck** - -Run: `bun test` → Expected: 3 pass, 0 fail. -Run: `bun run typecheck` → Expected: exit 0, no output. -Run: `bun install --frozen-lockfile` → Expected: exit 0 ("no changes"). - -- [ ] **Step 8: Record the verified commands** - -In `AGENTS.md`, replace the whole `## Commands` section body with: -```markdown -Verified 2026-10-14 (see `docs/PERMISSIONS.md`): -`bun install --frozen-lockfile` · `bun test` · `bun test test/.test.ts` · -`bun run typecheck` · `bun run build` · `bun run start` (server; env vars in `docs/ARCHITECTURE.md`). -Use `bun`/`bunx` — never `npm`, `npx`, `node` or `vitest`. -``` -(`bun run build`/`start` are added to this line in Tasks 18 and 21 when first run; until then list only the first four and keep the last sentence.) - -In `docs/PERMISSIONS.md`, replace the three `bun` rows of the Verified table and the `TODO(maintainer)` line with: -```markdown -| `bun install --frozen-lockfile` | worked | -| `bun test` | worked | -| `bun run typecheck` (`tsc --noEmit`) | worked | -``` -and add `bun install --frozen-lockfile`, `bun test`, `bun run typecheck` to the Allow row of the Tiers table. Add to Known gaps: "Codex: the allowed bun commands need no rule — they run inside the `workspace-write` sandbox." - -In `.claude/settings.json` add to `allow`: `"Bash(bun install --frozen-lockfile)", "Bash(bun test:*)", "Bash(bun run typecheck)"`. -In `.gemini/settings.json` add to `allowed`: `"run_shell_command(bun install --frozen-lockfile)", "run_shell_command(bun test)", "run_shell_command(bun run typecheck)"`. - -- [ ] **Step 9: Commit** - -```bash -git add package.json bun.lock tsconfig.json src/core/version.ts test/boundaries.test.ts test/version.test.ts AGENTS.md docs/PERMISSIONS.md .claude/settings.json .gemini/settings.json -git commit -m "chore: scaffold Bun + TypeScript with core boundary test; record verified commands" -``` - ---- - -### Task 2: Capture fixtures from the real services - -**Files:** -- Create: `scripts/capture-fixtures.sh`, `test/fixtures/**`, `test/helpers/fixtures.ts`, `test/fixtures.test.ts` - -**Interfaces:** -- Produces: `fixtureText(path): Promise`, `fixtureBytes(path): Promise`, `fixtureJson(path): Promise` from `test/helpers/fixtures.ts`. Fixture paths used by later tasks: - - `webtoons/tower-of-god.html`, `webtoons/crow-time-canvas.html`, `webtoons/zombie-revelation-mature.html` - - `wayback/tower-of-god-cdx-www.txt`, `wayback/tower-of-god-cdx-m.txt`, `wayback/tower-of-god-2015.html`, `wayback/tower-of-god-2023.html` - - `mangabaka/series-9999.json`, `mangabaka/links-9999.json`, `mangabaka/images-9999-page1.json`, `mangabaka/images-9999-page2.json` - - `images/poster-95.jpg` (480×623), `images/foreground-95.png` (1200×240), `images/landing-2023.jpg` (720×1230) - -- [ ] **Step 1: Write `scripts/capture-fixtures.sh`** - -```bash -#!/usr/bin/env bash -# Re-captures test/fixtures from the real services. Run by hand, never from tests. -# Wayback: honest User-Agent and 10 s between requests (spec §2.2), or it flags us as a bot. -set -euo pipefail -cd "$(dirname "$0")/.." -F=test/fixtures -mkdir -p "$F/webtoons" "$F/wayback" "$F/mangabaka" "$F/images" -BROWSER='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36' -TOOL='mangabaka-seeder/0.1.0 (+https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder)' -get() { curl -sSfL --max-time 90 -A "$1" -o "$2" "$3"; echo "ok $2 ($(wc -c <"$2") bytes)"; } -wb() { get "$TOOL" "$1" "$2"; sleep 10; } - -# Webtoons: always the /-/-/-/ form; it 301s to the canonical page. -get "$BROWSER" "$F/webtoons/tower-of-god.html" 'https://www.webtoons.com/-/-/-/list?title_no=95' -get "$BROWSER" "$F/webtoons/crow-time-canvas.html" 'https://www.webtoons.com/-/-/-/list?title_no=693372' -get "$BROWSER" "$F/webtoons/zombie-revelation-mature.html" 'https://www.webtoons.com/-/-/-/list?title_no=6065' - -CDX='https://web.archive.org/cdx/search/cdx?matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original&url=' -wb "$F/wayback/tower-of-god-cdx-www.txt" "${CDX}www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95" -wb "$F/wayback/tower-of-god-cdx-m.txt" "${CDX}m.webtoons.com/en/fantasy/tower-of-god/list?title_no=95" -row() { grep -m1 "^$1" "$F/wayback/tower-of-god-cdx-www.txt"; } -read -r TS ORIG <<<"$(row 2015)"; wb "$F/wayback/tower-of-god-2015.html" "https://web.archive.org/web/${TS}id_/${ORIG}" -read -r TS ORIG <<<"$(row 2023)"; wb "$F/wayback/tower-of-god-2023.html" "https://web.archive.org/web/${TS}id_/${ORIG}" - -API='https://api.mangabaka.org/v1/series/9999' -get "$TOOL" "$F/mangabaka/series-9999.json" "$API" -get "$TOOL" "$F/mangabaka/links-9999.json" "$API/links?limit=100" -get "$TOOL" "$F/mangabaka/images-9999-page1.json" "$API/images?limit=24&page=1" -get "$TOOL" "$F/mangabaka/images-9999-page2.json" "$API/images?limit=24&page=2" - -IMG='https://swebtoon-phinf.pstatic.net' -get "$BROWSER" "$F/images/poster-95.jpg" "$IMG/20250204_46/17386458444547o1b2_JPEG/95.jpg" -get "$BROWSER" "$F/images/foreground-95.png" "$IMG/20230324_64/1679612177711Smf0u_PNG/6TowerOfGod_landingpage_desktop_fg.png" -get "$BROWSER" "$F/images/landing-2023.jpg" "$IMG/20230324_114/1679612146968KhLLz_JPEG/6TowerOfGod_landingpage_mobile.jpg" -echo "done" -``` - -- [ ] **Step 2: Run it** - -Run: `bash scripts/capture-fixtures.sh` -Expected: 14 `ok` lines and `done`. If `crow-time-canvas.html` is not the Crow Time page (check `grep -c 'isChallenge : true' test/fixtures/webtoons/crow-time-canvas.html` ≥ 1), the `/-/-/-/` redirect doesn't work for canvas: change that line to `https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372`, rerun, and in Task 6 make `resolve` fall back to the user's URL (rewritten to `www.`) when the `/-/-/-/` fetch is not 2xx. Note it in `docs/DECISIONS.md`. - -- [ ] **Step 3: Write the fixture helper and a sanity test** - -`test/helpers/fixtures.ts`: -```ts -const dir = new URL("../fixtures/", import.meta.url); - -export const fixtureText = (path: string): Promise => Bun.file(new URL(path, dir)).text(); -export const fixtureBytes = async (path: string): Promise => - new Uint8Array(await Bun.file(new URL(path, dir)).arrayBuffer()); -export const fixtureJson = async (path: string): Promise => JSON.parse(await fixtureText(path)) as T; -``` - -`test/fixtures.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { fixtureBytes, fixtureJson, fixtureText } from "./helpers/fixtures"; - -test("captured fixtures are the pages and payloads we expect", async () => { - expect(await fixtureText("webtoons/tower-of-god.html")).toContain('property="og:image"'); - expect(await fixtureText("webtoons/crow-time-canvas.html")).toMatch(/isChallenge\s*:\s*true/); - expect(await fixtureText("webtoons/zombie-revelation-mature.html")).toMatch(/ageGradeNotice\s*:\s*true/); - expect(await fixtureText("wayback/tower-of-god-cdx-www.txt")).toMatch(/^2015\d{10} /m); - const page1 = await fixtureJson<{ data: unknown[]; pagination: { next: string | null } }>("mangabaka/images-9999-page1.json"); - expect(page1.data.length).toBe(24); - expect(page1.pagination.next).toContain("page=2"); - expect((await fixtureBytes("images/poster-95.jpg"))[0]).toBe(0xff); -}); -``` - -- [ ] **Step 4: Run it** - -Run: `bun test test/fixtures.test.ts` -Expected: PASS. If the mature check fails, open the page and find what marks it mature now; update spec §2.1 and this test together. - -- [ ] **Step 5: Commit** - -```bash -git add scripts/capture-fixtures.sh test/fixtures test/helpers/fixtures.ts test/fixtures.test.ts -git commit -m "test: capture Webtoons, Wayback and MangaBaka fixtures" -``` - ---- - -### Task 3: Model types, transport policy, fake transport - -**Files:** -- Create: `src/core/model.ts`, `src/core/transport.ts`, `test/helpers/fake-transport.ts`, `test/transport.test.ts` - -**Interfaces:** -- Consumes: `VERSION`, `REPO_URL` (Task 1). -- Produces (`src/core/model.ts`): types `CoverRole`, `ImageType`, `ContentRating`, `CONTENT_RATINGS`, `Shape`, `ImageExt`, `Granularity`, `GRANULARITIES`, `SourceSeries`, `CoverCandidate`, `InspectedCover`. -- Produces (`src/core/transport.ts`): `FetchInit`, `Transport`, `withPolicy(inner, opts?)`, `PolicyOptions`, `TransportError`, `abortableSleep(ms, signal?)`, `hostOf(url)`, `userAgentFor(url)`, `assertTokenScope(host, headers?)`, constants `MANGABAKA_API_HOST`, `WAYBACK_HOST`, `CORS_OPEN_HOSTS`, `BROWSER_USER_AGENT`, `TOOL_USER_AGENT`. -- Produces (`test/helpers/fake-transport.ts`): `FakeTransport` with `.on(match, respond)`, `.calls`; helpers `text()`, `json()`, `bytes()`. - -- [ ] **Step 1: Write `src/core/model.ts`** (types only — no test of its own; the typecheck covers it) - -```ts -export type CoverRole = "poster" | "square" | "banner" | "foreground" | "other"; -export type ImageType = "volume" | "volume_back" | "chapter" | "season" | "banner" | "audiobook" | "other"; -export type ContentRating = "safe" | "suggestive" | "erotica" | "pornographic"; -export const CONTENT_RATINGS: readonly ContentRating[] = ["safe", "suggestive", "erotica", "pornographic"]; -export type Shape = "tall" | "portrait" | "square" | "wide" | "other"; -export type ImageExt = "jpg" | "png" | "gif" | "webp"; -/** Wayback history: one snapshot per this many months (spec §4.3). */ -export type Granularity = 1 | 3 | 6 | 12; -export const GRANULARITIES: readonly Granularity[] = [1, 3, 6, 12]; - -export interface SourceSeries { - source: string; // Source.id - externalId: string; // e.g. Webtoons title_no "95" - title: string; - language: string; // MangaBaka language code, e.g. "en", "zh-Hant", or "unknown" - canonicalUrl: string; - archiveUrls: string[]; // URL prefixes (no scheme) to look up in Wayback - mature: boolean; - flags: Record; // e.g. { canvas: true } -} - -export interface CoverCandidate { - url: string; // normalised, hotlinkable - role: CoverRole; - current: boolean; // shown on the live page now - seenAt?: string; // earliest archive timestamp (YYYYMMDDhhmmss) if historical - lastSeenAt?: string; // latest archive timestamp after merging - originalUrl?: string; // as written in the page (absolute), before normalisation - archivedUrl?: string; // Wayback copy of the image itself -} - -export interface InspectedCover { - candidate: CoverCandidate; - fetchedFrom: string; // URL that served the bytes (candidate.url, or archivedUrl) - live: boolean; // true if fetchedFrom === candidate.url - width: number; - height: number; - sha256: string; // lowercase hex - contentType: string; // image/jpeg | image/png | image/gif | image/webp - ext: ImageExt; - bytes: Uint8Array; - shape: Shape; - type: ImageType; // suggested MangaBaka type -} -``` - -- [ ] **Step 2: Write the fake transport** - -`test/helpers/fake-transport.ts`: -```ts -import type { FetchInit, Transport } from "../../src/core/transport"; - -type Matcher = string | RegExp | ((url: string, init: FetchInit) => boolean); -type Responder = (url: string, init: FetchInit) => Response | Promise; - -/** Serves canned responses; throws on any request without a route, so tests can't hit the network. */ -export class FakeTransport implements Transport { - readonly calls: { url: string; init: FetchInit }[] = []; - private readonly routes: { match: Matcher; respond: Responder }[] = []; - - on(match: Matcher, respond: Responder): this { - this.routes.push({ match, respond }); - return this; - } - - async fetch(url: string, init: FetchInit = {}): Promise { - this.calls.push({ url, init }); - for (const { match, respond } of this.routes) { - const hit = typeof match === "string" ? match === url : match instanceof RegExp ? match.test(url) : match(url, init); - if (hit) return respond(url, init); - } - throw new Error(`FakeTransport: no route for ${init.method ?? "GET"} ${url}`); - } - - urls(): string[] { - return this.calls.map((c) => c.url); - } -} - -export const text = (body: string, status = 200, headers: Record = {}) => - new Response(body, { status, headers: { "content-type": "text/html; charset=utf-8", ...headers } }); -export const json = (body: unknown, status = 200) => - new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); -export const bytes = (body: Uint8Array, contentType = "image/jpeg", status = 200) => - new Response(body as Uint8Array, { status, headers: { "content-type": contentType } }); -``` - -- [ ] **Step 3: Write the failing transport tests** - -`test/transport.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { - TOOL_USER_AGENT, TransportError, userAgentFor, withPolicy, type PolicyOptions, -} from "../src/core/transport"; -import { FakeTransport, text } from "./helpers/fake-transport"; - -function fakeClock() { - let now = 0; - const slept: number[] = []; - const opts: PolicyOptions = { - now: () => now, - sleep: async (ms) => { slept.push(ms); now += ms; }, - }; - return { opts, slept, advance: (ms: number) => { now += ms; } }; -} - -describe("withPolicy", () => { - test("spaces requests to the same host, not across hosts", async () => { - const fake = new FakeTransport().on(() => true, () => text("ok")); - const clock = fakeClock(); - const t = withPolicy(fake, clock.opts); - await t.fetch("https://www.webtoons.com/a"); - await t.fetch("https://www.webtoons.com/b"); - await t.fetch("https://swebtoon-phinf.pstatic.net/c"); - await t.fetch("https://www.webtoons.com/d"); - expect(clock.slept).toEqual([1000, 1000]); - }); - - test("waits 10 s between Wayback requests", async () => { - const fake = new FakeTransport().on(() => true, () => text("ok")); - const clock = fakeClock(); - const t = withPolicy(fake, clock.opts); - await t.fetch("https://web.archive.org/a"); - await t.fetch("https://web.archive.org/b"); - expect(clock.slept).toEqual([10_000]); - }); - - test("retries GET on 429/5xx, honouring Retry-After", async () => { - let n = 0; - const fake = new FakeTransport().on(() => true, () => - ++n === 1 ? text("slow down", 429, { "retry-after": "7" }) : n === 2 ? text("oops", 503) : text("ok")); - const clock = fakeClock(); - const res = await withPolicy(fake, { ...clock.opts, minIntervalMs: () => 0 }).fetch("https://www.webtoons.com/x"); - expect(res.status).toBe(200); - expect(fake.calls.length).toBe(3); - expect(clock.slept).toEqual([7000, 4000]); // Retry-After, then backoff 2000 * 2^1 - }); - - test("returns the last 5xx after the retries are used up", async () => { - const fake = new FakeTransport().on(() => true, () => text("down", 502)); - const clock = fakeClock(); - const res = await withPolicy(fake, { ...clock.opts, minIntervalMs: () => 0 }).fetch("https://www.webtoons.com/x"); - expect(res.status).toBe(502); - expect(fake.calls.length).toBe(3); - }); - - test("never retries POST", async () => { - const fake = new FakeTransport().on(() => true, () => text("bad gateway", 502)); - const clock = fakeClock(); - const res = await withPolicy(fake, clock.opts).fetch("https://api.mangabaka.org/v0/my/submissions/series-images/1", { - method: "POST", body: "{}", - }); - expect(res.status).toBe(502); - expect(fake.calls.length).toBe(1); - }); - - test("retries network errors on GET, then throws TransportError", async () => { - const fake = new FakeTransport().on(() => true, () => { throw new TypeError("connection reset"); }); - const clock = fakeClock(); - const t = withPolicy(fake, { ...clock.opts, minIntervalMs: () => 0 }); - await expect(t.fetch("https://www.webtoons.com/x")).rejects.toBeInstanceOf(TransportError); - expect(fake.calls.length).toBe(3); - }); - - test("times out a hanging request", async () => { - const hang = { fetch: (_url: string, init?: { signal?: AbortSignal }) => - new Promise((_, reject) => init?.signal?.addEventListener("abort", () => reject(init.signal!.reason))) }; - const t = withPolicy(hang, { timeoutMs: 30, retries: 0, minIntervalMs: () => 0 }); - await expect(t.fetch("https://www.webtoons.com/x")).rejects.toBeInstanceOf(TransportError); - }); - - test("refuses to send a token anywhere but api.mangabaka.org", async () => { - const fake = new FakeTransport().on(() => true, () => text("ok")); - const t = withPolicy(fake, { minIntervalMs: () => 0 }); - await expect(t.fetch("https://evil.example/x", { headers: { "X-Api-Key": "mb-secret" } })).rejects.toThrow(/credentials/); - expect(fake.calls.length).toBe(0); - await t.fetch("https://api.mangabaka.org/v1/series/1", { headers: { "x-api-key": "mb-secret" } }); - expect(fake.calls.length).toBe(1); - }); - - test("a caller abort is not retried", async () => { - const controller = new AbortController(); - const fake = new FakeTransport().on(() => true, () => { controller.abort(); throw new DOMException("Aborted", "AbortError"); }); - const t = withPolicy(fake, { minIntervalMs: () => 0 }); - await expect(t.fetch("https://www.webtoons.com/x", { signal: controller.signal })).rejects.toThrow(); - expect(fake.calls.length).toBe(1); - }); -}); - -test("Wayback gets the honest User-Agent, everything else a browser one", () => { - expect(userAgentFor("https://web.archive.org/cdx/search/cdx?url=x")).toBe(TOOL_USER_AGENT); - expect(TOOL_USER_AGENT).toMatch(/^mangabaka-seeder\/\d+\.\d+\.\d+ \(\+https:\/\//); - expect(userAgentFor("https://www.webtoons.com/en/x")).toMatch(/^Mozilla\/5\.0/); -}); -``` - -- [ ] **Step 4: Run to verify they fail** - -Run: `bun test test/transport.test.ts` -Expected: FAIL — `Cannot find module '../src/core/transport'`. - -- [ ] **Step 5: Write `src/core/transport.ts`** - -```ts -import { REPO_URL, VERSION } from "./version"; - -export interface FetchInit { - method?: string; - headers?: Record; - body?: BodyInit | null; - signal?: AbortSignal; - /** Overrides the policy timeout for this one request (uploads). */ - timeoutMs?: number; -} - -/** How a front end reaches the network (spec §4.2). Returns a standard Response. */ -export interface Transport { - fetch(url: string, init?: FetchInit): Promise; -} - -export const MANGABAKA_API_HOST = "api.mangabaka.org"; -export const WAYBACK_HOST = "web.archive.org"; -/** Hosts a browser page may call directly because they send CORS headers (spec §2.3, §2.4). */ -export const CORS_OPEN_HOSTS: ReadonlySet = new Set([MANGABAKA_API_HOST, "litterbox.catbox.moe"]); - -export const BROWSER_USER_AGENT = - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36"; -export const TOOL_USER_AGENT = `mangabaka-seeder/${VERSION} (+${REPO_URL})`; - -/** Wayback flags fake browser User-Agents as bots (spec §2.2); sources get a browser one. */ -export function userAgentFor(url: string): string { - return hostOf(url) === WAYBACK_HOST ? TOOL_USER_AGENT : BROWSER_USER_AGENT; -} - -export function hostOf(url: string): string { - try { - return new URL(url).hostname; - } catch { - return ""; - } -} - -export class TransportError extends Error { - constructor(message: string, readonly url: string, readonly status?: number) { - super(message); - this.name = "TransportError"; - } -} - -export interface PolicyOptions { - /** Minimum gap between requests to one host. Default: 10 s for Wayback, 1 s otherwise. */ - minIntervalMs?: (host: string) => number; - timeoutMs?: number; - retries?: number; - backoffMs?: number; - now?: () => number; - sleep?: (ms: number, signal?: AbortSignal) => Promise; -} - -export const defaultInterval = (host: string): number => (host === WAYBACK_HOST ? 10_000 : 1_000); - -export function abortableSleep(ms: number, signal?: AbortSignal): Promise { - return new Promise((resolve, reject) => { - if (signal?.aborted) return reject(signal.reason); - const onAbort = () => { - clearTimeout(timer); - reject(signal!.reason); - }; - const timer = setTimeout(() => { - signal?.removeEventListener("abort", onAbort); - resolve(); - }, ms); - signal?.addEventListener("abort", onAbort, { once: true }); - }); -} - -export function assertTokenScope(host: string, headers?: Record): void { - if (!headers) return; - const carriesToken = Object.keys(headers).some((k) => ["x-api-key", "authorization"].includes(k.toLowerCase())); - if (carriesToken && host !== MANGABAKA_API_HOST) { - throw new TransportError(`refusing to send credentials to ${host || "an unknown host"}`, host); - } -} - -const RETRYABLE_METHODS = new Set(["GET", "HEAD"]); -const isRetryableStatus = (status: number) => status === 429 || status >= 500; - -/** Wraps any transport with the shared behaviour of spec §4.2. */ -export function withPolicy(inner: Transport, opts: PolicyOptions = {}): Transport { - const interval = opts.minIntervalMs ?? defaultInterval; - const timeoutMs = opts.timeoutMs ?? 20_000; - const retries = opts.retries ?? 2; - const backoffMs = opts.backoffMs ?? 2_000; - const now = opts.now ?? Date.now; - const sleep = opts.sleep ?? abortableSleep; - const nextSlot = new Map(); - - async function waitTurn(host: string, signal?: AbortSignal) { - const t = now(); - const at = Math.max(t, nextSlot.get(host) ?? 0); - nextSlot.set(host, at + interval(host)); - if (at > t) await sleep(at - t, signal); - } - - function attempt(url: string, init: FetchInit): Promise { - const { timeoutMs: perRequest, ...rest } = init; - const timeout = AbortSignal.timeout(perRequest ?? timeoutMs); - const signal = init.signal ? AbortSignal.any([init.signal, timeout]) : timeout; - return inner.fetch(url, { ...rest, signal }); - } - - return { - async fetch(url, init = {}) { - const host = hostOf(url); - assertTokenScope(host, init.headers); - const method = (init.method ?? "GET").toUpperCase(); - const attempts = RETRYABLE_METHODS.has(method) ? retries + 1 : 1; - let lastError: unknown; - for (let i = 0; i < attempts; i++) { - await waitTurn(host, init.signal); - try { - const res = await attempt(url, init); - if (!isRetryableStatus(res.status) || i === attempts - 1) return res; - await res.body?.cancel(); - await sleep(retryDelay(res, backoffMs, i), init.signal); - } catch (err) { - if (init.signal?.aborted) throw err; - lastError = err; - if (i < attempts - 1) await sleep(backoffMs * 2 ** i, init.signal); - } - } - throw new TransportError(`request to ${host} failed: ${describe(lastError)}`, url); - }, - }; -} - -function retryDelay(res: Response, base: number, attempt: number): number { - const seconds = Number(res.headers.get("retry-after")); - return Number.isFinite(seconds) && seconds > 0 ? Math.min(seconds, 120) * 1000 : base * 2 ** attempt; -} - -function describe(err: unknown): string { - return err instanceof Error ? err.message : String(err); -} -``` - -- [ ] **Step 6: Run tests** - -Run: `bun test test/transport.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 7: Commit** - -```bash -git add src/core/model.ts src/core/transport.ts test/helpers/fake-transport.ts test/transport.test.ts -git commit -m "feat(core): model types and transport policy (pacing, retries, timeouts, token scope)" -``` - ---- - -### Task 4: Link parsing and Webtoons URLs - -**Files:** -- Create: `src/core/links.ts`, `src/core/sources/webtoons/urls.ts`, `test/links.test.ts`, `test/webtoons-urls.test.ts` - -**Interfaces:** -- Produces (`links.ts`): `class LinkError extends Error`, `parseLink(input: string): URL`, `parseMangaBakaSeriesId(input: string): number`. -- Produces (`webtoons/urls.ts`): `WEBTOONS_HOSTS`, `IMAGE_HOST = "swebtoon-phinf.pstatic.net"`, `interface WebtoonsRef { titleNo: string; languageSegment: string | null; canvas: boolean }`, `isWebtoonsSeriesUrl(url: URL): boolean`, `parseWebtoonsUrl(url: URL): WebtoonsRef`, `mangabakaLanguage(segment: string | null): string`, `fetchUrlFor(titleNo: string): string`, `archivePrefixes(canonicalUrl: string): string[]`, `normalizeImageUrl(raw: string, base?: string | URL): string`, `isSameWebtoonsSeries(linkUrl: string, titleNo: string): boolean`. - -- [ ] **Step 1: Write the failing tests** - -`test/links.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { LinkError, parseLink, parseMangaBakaSeriesId } from "../src/core/links"; - -test.each([ - ["9999", 9999], - [" 9999 ", 9999], - ["https://mangabaka.org/9999", 9999], - ["https://mangabaka.org/manhwa/9999/Tower-of-God", 9999], - ["mangabaka.org/manhwa/9999/Tower-of-God", 9999], - ["https://www.mangabaka.org/manhwa/9999", 9999], - ["https://mangabaka.org/data/submissions/series/9999/images", 9999], -])("parseMangaBakaSeriesId(%p) = %p", (input, id) => { - expect(parseMangaBakaSeriesId(input)).toBe(id); -}); - -test.each(["", "abc", "https://example.org/9999", "https://mangabaka.org/manhwa/Tower-of-God", "0", "-5"])( - "parseMangaBakaSeriesId(%p) throws LinkError", - (input) => { - expect(() => parseMangaBakaSeriesId(input)).toThrow(LinkError); - }, -); - -test("parseLink adds https:// when the scheme is missing", () => { - expect(parseLink("www.webtoons.com/en/x/list?title_no=95").href).toBe("https://www.webtoons.com/en/x/list?title_no=95"); - expect(() => parseLink(" ")).toThrow(LinkError); -}); -``` - -`test/webtoons-urls.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { LinkError } from "../src/core/links"; -import { - archivePrefixes, fetchUrlFor, isSameWebtoonsSeries, isWebtoonsSeriesUrl, mangabakaLanguage, - normalizeImageUrl, parseWebtoonsUrl, -} from "../src/core/sources/webtoons/urls"; - -test("parses official, canvas, mobile and canonicalised links", () => { - expect(parseWebtoonsUrl(new URL("https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95"))) - .toEqual({ titleNo: "95", languageSegment: "en", canvas: false }); - expect(parseWebtoonsUrl(new URL("https://m.webtoons.com/fr/romance/x/list?title_no=1&page=3"))) - .toEqual({ titleNo: "1", languageSegment: "fr", canvas: false }); - expect(parseWebtoonsUrl(new URL("https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372"))) - .toEqual({ titleNo: "693372", languageSegment: "en", canvas: true }); - expect(parseWebtoonsUrl(new URL("https://www.webtoons.com/-/-/-/list?title_no=95"))) - .toEqual({ titleNo: "95", languageSegment: null, canvas: false }); - expect(parseWebtoonsUrl(new URL("https://www.webtoons.com/zh-hant/fantasy/x/list?title_no=160")).languageSegment) - .toBe("zh-hant"); -}); - -test("rejects links without a numeric title_no or on other hosts", () => { - expect(() => parseWebtoonsUrl(new URL("https://www.webtoons.com/en/fantasy/tower-of-god/list"))).toThrow(LinkError); - expect(() => parseWebtoonsUrl(new URL("https://www.webtoons.com/en/x/list?title_no=abc"))).toThrow(LinkError); - expect(() => parseWebtoonsUrl(new URL("https://tapas.io/series/x"))).toThrow(LinkError); - expect(isWebtoonsSeriesUrl(new URL("https://www.webtoons.com/en/x/list?title_no=95"))).toBe(true); - expect(isWebtoonsSeriesUrl(new URL("https://www.webtoons.com/en/x/viewer?title_no=95&episode_no=1"))).toBe(true); - expect(isWebtoonsSeriesUrl(new URL("https://tapas.io/?title_no=95"))).toBe(false); -}); - -test("maps Webtoons language segments to MangaBaka codes", () => { - expect(mangabakaLanguage("en")).toBe("en"); - expect(mangabakaLanguage("zh-hant")).toBe("zh-Hant"); - expect(mangabakaLanguage("de")).toBe("de"); - expect(mangabakaLanguage(null)).toBe("unknown"); - expect(mangabakaLanguage("xx")).toBe("unknown"); -}); - -test("always fetches the /-/-/-/ form, which redirects to the canonical page", () => { - expect(fetchUrlFor("95")).toBe("https://www.webtoons.com/-/-/-/list?title_no=95"); -}); - -test("archive prefixes cover www. and m. without a scheme", () => { - expect(archivePrefixes("https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95")).toEqual([ - "www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - "m.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - ]); -}); - -test.each([ - ["https://webtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg?type=crop540_540", - "https://swebtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg"], - ["http://swebtoon.phinf.naver.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg", - "https://swebtoon-phinf.pstatic.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg"], - ["http://webtoon.phinf.naver.net/20140915_195/1410759727783HvB4R_JPEG/11_EC8BA0EC9D98ED8391_bg.jpg", - "https://swebtoon-phinf.pstatic.net/20140915_195/1410759727783HvB4R_JPEG/11_EC8BA0EC9D98ED8391_bg.jpg"], - ["//swebtoon-phinf.pstatic.net/a/b c).jpg", "https://swebtoon-phinf.pstatic.net/a/b%20c).jpg"], - ["https://example.org/x.jpg?keep=1", "https://example.org/x.jpg?keep=1"], -])("normalizeImageUrl(%p)", (raw, expected) => { - expect(normalizeImageUrl(raw, "https://www.webtoons.com/en/x/list?title_no=1")).toBe(expected); -}); - -test("isSameWebtoonsSeries compares title_no exactly, including MangaBaka's canonical form", () => { - expect(isSameWebtoonsSeries("https://www.webtoons.com/-/-/-/list?title_no=95", "95")).toBe(true); - expect(isSameWebtoonsSeries("http://www.webtoons.com:80/en/fantasy/tower-of-god/list?title_no=95", "95")).toBe(true); - expect(isSameWebtoonsSeries("https://www.webtoons.com/-/-/-/list?title_no=950", "95")).toBe(false); - expect(isSameWebtoonsSeries("https://comic.naver.com/webtoon/list?titleId=95", "95")).toBe(false); - expect(isSameWebtoonsSeries("not a url", "95")).toBe(false); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/links.test.ts test/webtoons-urls.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/core/links.ts`** - -```ts -export class LinkError extends Error { - override name = "LinkError"; -} - -/** Parses a pasted link; a missing scheme means https. */ -export function parseLink(input: string): URL { - const s = input.trim(); - if (!s) throw new LinkError("Paste a link first."); - try { - return new URL(/^[a-z][a-z0-9+.-]*:\/\//i.test(s) ? s : `https://${s}`); - } catch { - throw new LinkError(`That doesn't look like a link: ${s}`); - } -} - -/** - * Accepts a bare id or any mangabaka.org link to a series. `/9999` redirects to the canonical - * `/manhwa/9999/Tower-of-God` and `/series/9999` is a 404 (spec §4.5), so we take the first - * all-digits path segment. - */ -export function parseMangaBakaSeriesId(input: string): number { - const s = input.trim(); - if (/^\d+$/.test(s)) return positive(Number(s), s); - const url = parseLink(s); - if (!/^(www\.)?mangabaka\.org$/i.test(url.hostname)) { - throw new LinkError(`Not a MangaBaka link: ${url.hostname}. Paste a mangabaka.org series link or its number.`); - } - const segment = url.pathname.split("/").find((p) => /^\d+$/.test(p)); - if (!segment) throw new LinkError("That MangaBaka link has no series number in it."); - return positive(Number(segment), s); -} - -function positive(n: number, input: string): number { - if (!Number.isSafeInteger(n) || n <= 0) throw new LinkError(`Not a MangaBaka series number: ${input}`); - return n; -} -``` - -- [ ] **Step 4: Write `src/core/sources/webtoons/urls.ts`** - -```ts -import { LinkError } from "../../links"; - -export const WEBTOONS_HOSTS: ReadonlySet = new Set(["www.webtoons.com", "m.webtoons.com", "webtoons.com"]); -export const IMAGE_HOST = "swebtoon-phinf.pstatic.net"; -/** Hosts that serve the same image paths; only IMAGE_HOST works without a Webtoons Referer (spec §2.1). */ -const IMAGE_HOSTS: ReadonlySet = new Set([ - IMAGE_HOST, "webtoon-phinf.pstatic.net", "swebtoon.phinf.naver.net", "webtoon.phinf.naver.net", -]); -/** Webtoons path segment → MangaBaka language code. */ -const LANGUAGES: Readonly> = { - en: "en", es: "es", fr: "fr", de: "de", th: "th", id: "id", "zh-hant": "zh-Hant", -}; - -export interface WebtoonsRef { - titleNo: string; - languageSegment: string | null; - canvas: boolean; -} - -export function isWebtoonsSeriesUrl(url: URL): boolean { - return WEBTOONS_HOSTS.has(url.hostname) && /^\d+$/.test(url.searchParams.get("title_no") ?? ""); -} - -export function parseWebtoonsUrl(url: URL): WebtoonsRef { - if (!WEBTOONS_HOSTS.has(url.hostname)) throw new LinkError(`Not a WEBTOON link: ${url.hostname}`); - const titleNo = url.searchParams.get("title_no") ?? ""; - if (!/^\d+$/.test(titleNo)) { - throw new LinkError("That WEBTOON link has no title_no. Use the series page (…/list?title_no=…)."); - } - const segments = url.pathname.split("/").filter(Boolean).map((s) => s.toLowerCase()); - const first = segments[0]; - return { - titleNo, - languageSegment: first && LANGUAGES[first] ? first : null, - canvas: segments.includes("canvas") || segments.includes("challenge"), - }; -} - -export function mangabakaLanguage(segment: string | null): string { - return (segment && LANGUAGES[segment]) || "unknown"; -} - -/** - * The `/-/-/-/` form 301s to the canonical page (verified 2026-10-14); a made-up slug like - * `/en/x/x/` made Webtoons hang instead, so we never fetch the user's own path. - */ -export function fetchUrlFor(titleNo: string): string { - return `https://www.webtoons.com/-/-/-/list?title_no=${titleNo}`; -} - -export function archivePrefixes(canonicalUrl: string): string[] { - const u = new URL(canonicalUrl); - const rest = `${u.pathname}?title_no=${u.searchParams.get("title_no") ?? ""}`; - return [`www.webtoons.com${rest}`, `m.webtoons.com${rest}`]; -} - -/** Largest, hotlinkable form: IMAGE_HOST, same path, no `?type=` resize query. */ -export function normalizeImageUrl(raw: string, base?: string | URL): string { - let u: URL; - try { - u = new URL(raw.trim(), base); - } catch { - return raw.trim(); - } - return IMAGE_HOSTS.has(u.hostname) ? `https://${IMAGE_HOST}${u.pathname}` : u.toString(); -} - -export function isSameWebtoonsSeries(linkUrl: string, titleNo: string): boolean { - try { - const u = new URL(linkUrl); - return WEBTOONS_HOSTS.has(u.hostname) && u.searchParams.get("title_no") === titleNo; - } catch { - return false; - } -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/links.test.ts test/webtoons-urls.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/links.ts src/core/sources/webtoons/urls.ts test/links.test.ts test/webtoons-urls.test.ts -git commit -m "feat(core): parse MangaBaka and Webtoons links; normalise Webtoons image URLs" -``` - ---- - -### Task 5: HTML helpers and the Webtoons page parser - -**Files:** -- Create: `src/core/html.ts`, `src/core/sources/webtoons/parse.ts`, `test/html.test.ts`, `test/webtoons-parse.test.ts` - -**Interfaces:** -- Consumes: `normalizeImageUrl` (Task 4), `CoverCandidate`, `CoverRole` (Task 3). -- Produces (`html.ts`): `decodeEntities(s)`, `parseAttributes(tag): Record`, `findTags(html, name): Record[]`, `metaContent(html, key): string | undefined`, `textOf(fragment): string`. -- Produces (`parse.ts`): `interface ParsedPage { title: string | null; canonicalUrl: string | null; canvas: boolean; mature: boolean; covers: CoverCandidate[] }`, `parseSeriesPage(html: string, pageUrl: URL): ParsedPage`, `extractCovers(html: string, pageUrl: URL, canvas?: boolean): CoverCandidate[]` (all with `current: false`; callers set it). - -- [ ] **Step 1: Write the failing tests** - -`test/html.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { decodeEntities, findTags, metaContent, parseAttributes, textOf } from "../src/core/html"; - -test("decodes named and numeric entities", () => { - expect(decodeEntities("a & b <c> "d" 'e' 'f' &bogus;")).toBe(`a & b "d" 'e' 'f' &bogus;`); -}); - -test("parses attributes in any order and quoting style", () => { - expect(parseAttributes(``)) - .toEqual({ content: "x&y", property: "og:image", "data-a": "1", hidden: "" }); - expect(parseAttributes(`
`).style) - .toBe("background:url('https://a/b.jpg') repeat-x"); -}); - -test("finds tags and meta content by property or name", () => { - const html = ``; - expect(findTags(html, "meta")).toHaveLength(2); - expect(metaContent(html, "og:image")).toBe("o.jpg"); - expect(metaContent(html, "twitter:image")).toBe("t.jpg"); - expect(metaContent(html, "og:title")).toBeUndefined(); -}); - -test("textOf strips tags and collapses whitespace", () => { - expect(textOf("Tower\n of & God ")).toBe("Tower of & God"); -}); -``` - -`test/webtoons-parse.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { parseSeriesPage } from "../src/core/sources/webtoons/parse"; -import { fixtureText } from "./helpers/fixtures"; - -// Expected URLs are what the fixtures captured on 2026-10-14. If Webtoons changed since and you -// re-captured, the fixture is the truth: update these strings to what it now contains. -const P = "https://swebtoon-phinf.pstatic.net"; - -test("official series: poster, square, banner, foreground; title, canonical, flags", async () => { - const page = parseSeriesPage(await fixtureText("webtoons/tower-of-god.html"), - new URL("https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95")); - expect(page.title).toBe("Tower of God"); - expect(page.canonicalUrl).toBe("https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95"); - expect(page.canvas).toBe(false); - expect(page.mature).toBe(false); - expect(page.covers.map((c) => [c.role, c.url])).toEqual([ - ["poster", `${P}/20250204_46/17386458444547o1b2_JPEG/95.jpg`], - ["square", `${P}/20230324_171/1679612128056zKhRj_JPEG/1TowerOfGod_thumbnail_mobile.jpg`], - ["banner", `${P}/20230408_147/1680918244853P9prU_JPEG/0TowerOfGod_landingpage_desktop_bg.jpg`], - ["foreground", `${P}/20230324_64/1679612177711Smf0u_PNG/6TowerOfGod_landingpage_desktop_fg.png`], - ]); - expect(page.covers[0]!.originalUrl).toContain("?type=crop540_540"); - expect(page.covers.every((c) => c.current === false)).toBe(true); -}); - -test("canvas series: squares only, hero image deduplicated by URL", async () => { - const page = parseSeriesPage(await fixtureText("webtoons/crow-time-canvas.html"), - new URL("https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372")); - expect(page.canvas).toBe(true); - expect(page.covers.map((c) => c.role)).toEqual(["square", "square"]); - expect(page.covers[0]!.url).toBe(`${P}/20240408_279/1712548929813q5cnb_JPEG/thumbnail.jpg`); -}); - -test("mature series is flagged", async () => { - const page = parseSeriesPage(await fixtureText("webtoons/zombie-revelation-mature.html"), - new URL("https://www.webtoons.com/en/action/zombie-revelation-82-08/list?title_no=6065")); - expect(page.mature).toBe(true); - expect(page.covers.length).toBeGreaterThan(0); -}); - -test("2015 Wayback snapshot: unquoted url(), old naver.net hosts", async () => { - const page = parseSeriesPage(await fixtureText("wayback/tower-of-god-2015.html"), - new URL("http://www.webtoons.com:80/en/fantasy/tower-of-god/list?title_no=95")); - const byRole = Object.fromEntries(page.covers.map((c) => [c.role, c])); - expect(byRole.poster!.url).toBe(`${P}/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg`); - expect(byRole.poster!.originalUrl).toBe("http://swebtoon.phinf.naver.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg"); - expect(byRole.banner!.url).toBe(`${P}/20140915_195/1410759727783HvB4R_JPEG/11_EC8BA0EC9D98ED8391_bg.jpg`); - expect(byRole.foreground!.url).toBe(`${P}/20140915_275/1410759723052oa5KP_PNG/11_EC8BA0EC9D98ED8391.png`); -}); - -test("pages without the markup give no covers, not an error", () => { - const page = parseSeriesPage("x", new URL("https://www.webtoons.com/en/x/list?title_no=1")); - expect(page).toEqual({ title: null, canonicalUrl: null, canvas: false, mature: false, covers: [] }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/html.test.ts test/webtoons-parse.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/core/html.ts`** - -```ts -// Small, forgiving HTML helpers. Not a parser: enough for , ,
, . - -const NAMED: Readonly> = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'" }; - -export function decodeEntities(s: string): string { - return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (match, entity: string) => { - const e = entity.toLowerCase(); - if (e.startsWith("#x")) return String.fromCodePoint(parseInt(e.slice(2), 16)); - if (e.startsWith("#")) return String.fromCodePoint(parseInt(e.slice(1), 10)); - return NAMED[e] ?? match; - }); -} - -export function parseAttributes(tag: string): Record { - const out: Record = {}; - const body = tag.replace(/^<\s*[\w-]+/, "").replace(/\/?\s*>$/, ""); - for (const m of body.matchAll(/([^\s=/"'>]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'>]+)))?/g)) { - const name = m[1]!.toLowerCase(); - if (!(name in out)) out[name] = decodeEntities(m[2] ?? m[3] ?? m[4] ?? ""); - } - return out; -} - -export function findTags(html: string, name: string): Record[] { - return [...html.matchAll(new RegExp(`<${name}\\b[^>]*>`, "gi"))].map((m) => parseAttributes(m[0])); -} - -/** `` */ -export function metaContent(html: string, key: string): string | undefined { - for (const a of findTags(html, "meta")) { - if ((a.property ?? a.name) === key && a.content) return a.content; - } - return undefined; -} - -export function textOf(fragment: string): string { - return decodeEntities(fragment.replace(/<[^>]*>/g, " ")).replace(/\s+/g, " ").trim(); -} -``` - -- [ ] **Step 4: Write `src/core/sources/webtoons/parse.ts`** - -```ts -import { findTags, metaContent, textOf } from "../../html"; -import type { CoverCandidate, CoverRole } from "../../model"; -import { normalizeImageUrl } from "./urls"; - -export interface ParsedPage { - title: string | null; - canonicalUrl: string | null; - canvas: boolean; - mature: boolean; - covers: CoverCandidate[]; -} - -const isCanvas = (html: string, pageUrl: URL) => - /\/(canvas|challenge)\//i.test(pageUrl.pathname) || /isChallenge\s*:\s*true/.test(html); - -export function parseSeriesPage(html: string, pageUrl: URL): ParsedPage { - const canvas = isCanvas(html, pageUrl); - return { - title: titleOf(html), - canonicalUrl: canonicalOf(html), - canvas, - mature: /ageGradeNotice\s*:\s*true/.test(html), - covers: extractCovers(html, pageUrl, canvas), - }; -} - -/** Spec §2.1 table. Canvas pages have no portrait poster: their og:image is a square. */ -export function extractCovers(html: string, pageUrl: URL, canvas = isCanvas(html, pageUrl)): CoverCandidate[] { - const found: { role: CoverRole; raw: string | undefined }[] = [ - { role: canvas ? "square" : "poster", raw: metaContent(html, "og:image") }, - { role: "square", raw: metaContent(html, "twitter:image") }, - { role: "banner", raw: backgroundOf(html) }, - { role: canvas ? "square" : "foreground", raw: heroImageOf(html) }, - ]; - const seen = new Set(); - const out: CoverCandidate[] = []; - for (const { role, raw } of found) { - if (!raw || /^\s*data:/i.test(raw)) continue; - let originalUrl: string; - try { - originalUrl = new URL(raw.trim(), pageUrl).toString(); - } catch { - continue; - } - const url = normalizeImageUrl(originalUrl); - if (seen.has(url)) continue; - seen.add(url); - out.push({ url, role, current: false, originalUrl }); - } - return out; -} - -function backgroundOf(html: string): string | undefined { - for (const div of findTags(html, "div")) { - if (!/(^|\s)detail_bg(\s|$)/.test(div.class ?? "")) continue; - const m = /url\(\s*['"]?([^'")\s]+)['"]?\s*\)/i.exec(div.style ?? ""); - if (m) return m[1]; - } - return undefined; -} - -function heroImageOf(html: string): string | undefined { - const i = html.search(/class="detail_header\b/); - if (i < 0) return undefined; - return findTags(html.slice(i, i + 3000), "img")[0]?.src || undefined; -} - -function titleOf(html: string): string | null { - const og = metaContent(html, "og:title"); - if (og?.trim()) return og.trim(); - const m = /]*class="[^"]*\bsubj\b[^"]*"[^>]*>([\s\S]*?)<\/h[13]>/i.exec(html); - return m ? textOf(m[1]!) || null : null; -} - -function canonicalOf(html: string): string | null { - for (const link of findTags(html, "link")) { - if ((link.rel ?? "").toLowerCase() === "canonical" && link.href) return link.href; - } - return null; -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/html.test.ts test/webtoons-parse.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/html.ts src/core/sources/webtoons/parse.ts test/html.test.ts test/webtoons-parse.test.ts -git commit -m "feat(core): parse Webtoons series pages (current and archived) into cover candidates" -``` - ---- - -### Task 6: Source interface, registry, the Webtoons source - -**Files:** -- Create: `src/core/sources/source.ts`, `src/core/sources/registry.ts`, `src/core/sources/webtoons/index.ts`, `test/webtoons-source.test.ts` - -**Interfaces:** -- Consumes: Tasks 3–5. -- Produces (`source.ts`): `interface Source` exactly as spec §4.1 (`id`, `name`, `matches`, `resolve`, `currentCovers`, `extractCovers?`, `historicalCovers?`, `normalizeImageUrl`, `isSameSeries`, `hosts`), `interface SourceContext { transport: Transport; signal?: AbortSignal }`, `class SourceError extends Error`. -- Produces (`registry.ts`): `SOURCES: readonly Source[]`, `findSource(url: URL, sources?): Source | undefined`, `allowedFetchHosts(sources?): Set` (source hosts + `web.archive.org`). -- Produces (`webtoons/index.ts`): `webtoons: Source`. - -- [ ] **Step 1: Write the failing tests** - -`test/webtoons-source.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { allowedFetchHosts, findSource } from "../src/core/sources/registry"; -import { SourceError } from "../src/core/sources/source"; -import { webtoons } from "../src/core/sources/webtoons"; -import { FakeTransport, text } from "./helpers/fake-transport"; -import { fixtureText } from "./helpers/fixtures"; - -const TOG = "https://www.webtoons.com/-/-/-/list?title_no=95"; - -test("resolve fetches the /-/-/-/ form and reads the canonical page", async () => { - const fake = new FakeTransport().on(TOG, async () => text(await fixtureText("webtoons/tower-of-god.html"))); - const series = await webtoons.resolve(new URL("https://m.webtoons.com/en/made-up-slug/list?title_no=95"), { transport: fake }); - expect(fake.urls()).toEqual([TOG]); - expect(series).toEqual({ - source: "webtoons", - externalId: "95", - title: "Tower of God", - language: "en", - canonicalUrl: "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - archiveUrls: [ - "www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - "m.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - ], - mature: false, - flags: { canvas: false }, - }); -}); - -test("currentCovers reuses the page fetched by resolve and marks covers current", async () => { - const fake = new FakeTransport().on(TOG, async () => text(await fixtureText("webtoons/tower-of-god.html"))); - const series = await webtoons.resolve(new URL(TOG), { transport: fake }); - const covers = await webtoons.currentCovers(series, { transport: fake }); - expect(fake.calls.length).toBe(1); - expect(covers.map((c) => c.role)).toEqual(["poster", "square", "banner", "foreground"]); - expect(covers.every((c) => c.current)).toBe(true); -}); - -test("canvas and mature flags come through", async () => { - const fake = new FakeTransport() - .on("https://www.webtoons.com/-/-/-/list?title_no=693372", async () => text(await fixtureText("webtoons/crow-time-canvas.html"))) - .on("https://www.webtoons.com/-/-/-/list?title_no=6065", async () => text(await fixtureText("webtoons/zombie-revelation-mature.html"))); - const canvas = await webtoons.resolve(new URL("https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372"), { transport: fake }); - expect(canvas.flags.canvas).toBe(true); - const mature = await webtoons.resolve(new URL("https://www.webtoons.com/-/-/-/list?title_no=6065"), { transport: fake }); - expect(mature.mature).toBe(true); -}); - -test("a Webtoons error is a SourceError with the status", async () => { - const fake = new FakeTransport().on(TOG, () => text("gone", 404)); - await expect(webtoons.resolve(new URL(TOG), { transport: fake })).rejects.toThrow(SourceError); - await expect(webtoons.resolve(new URL(TOG), { transport: fake })).rejects.toThrow(/404/); -}); - -test("registry finds the source and builds the proxy allowlist", () => { - expect(findSource(new URL("https://www.webtoons.com/en/x/list?title_no=1"))).toBe(webtoons); - expect(findSource(new URL("https://tapas.io/series/x"))).toBeUndefined(); - expect([...allowedFetchHosts()].sort()).toEqual( - ["m.webtoons.com", "swebtoon-phinf.pstatic.net", "web.archive.org", "www.webtoons.com"]); -}); - -test("isSameSeries checks title_no", () => { - const series = { externalId: "95" } as Parameters[1]; - expect(webtoons.isSameSeries("https://www.webtoons.com/-/-/-/list?title_no=95", series)).toBe(true); - expect(webtoons.isSameSeries("https://www.webtoons.com/-/-/-/list?title_no=318", series)).toBe(false); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/webtoons-source.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/core/sources/source.ts`** - -```ts -import type { CoverCandidate, SourceSeries } from "../model"; -import type { Transport } from "../transport"; - -export interface SourceContext { - transport: Transport; - signal?: AbortSignal; -} - -export class SourceError extends Error { - override name = "SourceError"; -} - -/** Extension point (spec §4.1): a new source is one module plus one line in registry.ts. */ -export interface Source { - /** Stable id, e.g. "webtoons". Used in notes and output. */ - readonly id: string; - /** Human-readable name, e.g. "WEBTOON". */ - readonly name: string; - /** Hosts this source fetches from; the server proxy allowlist is built from these. */ - readonly hosts: readonly string[]; - matches(url: URL): boolean; - resolve(url: URL, ctx: SourceContext): Promise; - currentCovers(series: SourceSeries, ctx: SourceContext): Promise; - /** Pure extraction from a series page's HTML. Sources that implement it get Wayback history. */ - extractCovers?(html: string, pageUrl: URL): CoverCandidate[]; - historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise; - normalizeImageUrl(url: string): string; - isSameSeries(linkUrl: string, series: SourceSeries): boolean; -} -``` - -- [ ] **Step 4: Write `src/core/sources/webtoons/index.ts`** - -```ts -import type { CoverCandidate, SourceSeries } from "../../model"; -import { SourceError, type Source, type SourceContext } from "../source"; -import { extractCovers, parseSeriesPage } from "./parse"; -import { - IMAGE_HOST, archivePrefixes, fetchUrlFor, isSameWebtoonsSeries, isWebtoonsSeriesUrl, mangabakaLanguage, - normalizeImageUrl, parseWebtoonsUrl, type WebtoonsRef, -} from "./urls"; - -/** Covers parsed by resolve(), so currentCovers() doesn't fetch the page twice. */ -const currentBySeries = new WeakMap(); - -async function fetchPage(titleNo: string, ctx: SourceContext): Promise { - const res = await ctx.transport.fetch(fetchUrlFor(titleNo), { signal: ctx.signal }); - if (!res.ok) { - await res.body?.cancel(); - throw new SourceError(`WEBTOON answered HTTP ${res.status} for title_no ${titleNo}.`); - } - return res.text(); -} - -export const webtoons: Source = { - id: "webtoons", - name: "WEBTOON", - hosts: ["www.webtoons.com", "m.webtoons.com", IMAGE_HOST], - - matches: (url) => isWebtoonsSeriesUrl(url), - - async resolve(url, ctx) { - const ref = parseWebtoonsUrl(url); - const page = parseSeriesPage(await fetchPage(ref.titleNo, ctx), url); - const canonicalUrl = page.canonicalUrl ?? url.toString(); - let canonical: WebtoonsRef = ref; - try { - canonical = parseWebtoonsUrl(new URL(canonicalUrl)); - } catch { - // keep the user's link if the page has no usable canonical URL - } - if (canonical.titleNo !== ref.titleNo) { - throw new SourceError(`WEBTOON sent title_no ${canonical.titleNo} when asked for ${ref.titleNo}.`); - } - const series: SourceSeries = { - source: "webtoons", - externalId: ref.titleNo, - title: page.title ?? `title_no ${ref.titleNo}`, - language: mangabakaLanguage(canonical.languageSegment ?? ref.languageSegment), - canonicalUrl, - archiveUrls: archivePrefixes(canonicalUrl), - mature: page.mature, - flags: { canvas: page.canvas || canonical.canvas }, - }; - currentBySeries.set(series, page.covers.map((c) => ({ ...c, current: true }))); - return series; - }, - - async currentCovers(series, ctx) { - const cached = currentBySeries.get(series); - if (cached) return cached; - const html = await fetchPage(series.externalId, ctx); - return extractCovers(html, new URL(series.canonicalUrl), series.flags.canvas).map((c) => ({ ...c, current: true })); - }, - - extractCovers: (html, pageUrl) => extractCovers(html, pageUrl), - normalizeImageUrl: (url) => normalizeImageUrl(url), - isSameSeries: (linkUrl, series) => isSameWebtoonsSeries(linkUrl, series.externalId), -}; -``` - -- [ ] **Step 5: Write `src/core/sources/registry.ts`** - -```ts -import { WAYBACK_HOST } from "../transport"; -import type { Source } from "./source"; -import { webtoons } from "./webtoons"; - -export const SOURCES: readonly Source[] = [webtoons]; - -export function findSource(url: URL, sources: readonly Source[] = SOURCES): Source | undefined { - return sources.find((s) => s.matches(url)); -} - -/** What `/api/fetch` may fetch: every source's hosts plus Wayback (spec §4.10). */ -export function allowedFetchHosts(sources: readonly Source[] = SOURCES): Set { - return new Set([...sources.flatMap((s) => s.hosts), WAYBACK_HOST]); -} -``` - -- [ ] **Step 6: Run tests** - -Run: `bun test test/webtoons-source.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 7: Commit** - -```bash -git add src/core/sources test/webtoons-source.test.ts -git commit -m "feat(core): Source interface, registry and the Webtoons source" -``` - ---- - -### Task 7: Image inspection and classification - -**Files:** -- Create: `src/core/images.ts`, `test/images.test.ts` - -**Interfaces:** -- Consumes: `CoverCandidate`, `InspectedCover`, `ImageExt`, `ImageType`, `Shape` (Task 3), `Transport`. -- Produces: `MAX_IMAGE_BYTES`, `class ImageError`, `interface ImageInfo { width; height; contentType; ext }`, `probeImage(bytes): ImageInfo | null`, `sha256Hex(bytes): Promise`, `shapeOf(width, height): Shape`, `typeForShape(shape): ImageType`, `readCapped(res, max?): Promise`, `inspectCandidate(c, transport, signal?): Promise`, `inspectAll(candidates, transport, opts: { warn(msg: string): void; onProgress?(done: number, total: number): void; signal?: AbortSignal }): Promise`. The server (Task 17) reuses `probeImage`, `sha256Hex`, `readCapped`, `MAX_IMAGE_BYTES`. - -- [ ] **Step 1: Write the failing tests** - -`test/images.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { - ImageError, inspectAll, inspectCandidate, probeImage, readCapped, sha256Hex, shapeOf, typeForShape, -} from "../src/core/images"; -import type { CoverCandidate } from "../src/core/model"; -import { FakeTransport, bytes, text } from "./helpers/fake-transport"; -import { fixtureBytes } from "./helpers/fixtures"; - -describe("probeImage", () => { - test("real JPEG and PNG fixtures", async () => { - expect(probeImage(await fixtureBytes("images/poster-95.jpg"))) - .toEqual({ width: 480, height: 623, contentType: "image/jpeg", ext: "jpg" }); - expect(probeImage(await fixtureBytes("images/foreground-95.png"))) - .toEqual({ width: 1200, height: 240, contentType: "image/png", ext: "png" }); - expect(probeImage(await fixtureBytes("images/landing-2023.jpg"))) - .toMatchObject({ width: 720, height: 1230 }); - }); - - // Synthetic headers: these test the format decoders, they are not captured service responses. - test("GIF and WebP (VP8X) headers", () => { - const gif = new Uint8Array([0x47, 0x49, 0x46, 0x38, 0x39, 0x61, 0x2c, 0x01, 0xc8, 0x00]); - expect(probeImage(gif)).toEqual({ width: 300, height: 200, contentType: "image/gif", ext: "gif" }); - const webp = new Uint8Array(30); - webp.set([...new TextEncoder().encode("RIFF")], 0); - webp.set([...new TextEncoder().encode("WEBPVP8X")], 8); - webp.set([0x2b, 0x01, 0x00, 0xc7, 0x00, 0x00], 24); // 299+1 × 199+1 - expect(probeImage(webp)).toEqual({ width: 300, height: 200, contentType: "image/webp", ext: "webp" }); - }); - - test("HTML error pages and empty bodies are not images", () => { - expect(probeImage(new TextEncoder().encode("404"))).toBeNull(); - expect(probeImage(new Uint8Array())).toBeNull(); - }); -}); - -test.each([ - [480, 623, "portrait", "volume"], - [720, 1230, "tall", "other"], - [510, 510, "square", "other"], - [240, 254, "square", "other"], - [1920, 321, "wide", "banner"], - [1200, 240, "wide", "banner"], - [1000, 700, "other", "other"], -] as const)("%i×%i is %s → %s", (w, h, shape, type) => { - expect(shapeOf(w, h)).toBe(shape); - expect(typeForShape(shapeOf(w, h))).toBe(type); -}); - -test("sha256Hex", async () => { - expect(await sha256Hex(new TextEncoder().encode("abc"))) - .toBe("ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"); -}); - -describe("readCapped", () => { - test("refuses a declared size over the cap", async () => { - const res = new Response("x", { headers: { "content-length": "999999" } }); - await expect(readCapped(res, 1000)).rejects.toBeInstanceOf(ImageError); - }); - test("refuses a stream that grows over the cap", async () => { - const stream = new ReadableStream({ start(c) { c.enqueue(new Uint8Array(600)); c.enqueue(new Uint8Array(600)); c.close(); } }); - await expect(readCapped(new Response(stream), 1000)).rejects.toBeInstanceOf(ImageError); - }); - test("returns the bytes under the cap", async () => { - expect((await readCapped(new Response(new Uint8Array(10)), 1000)).byteLength).toBe(10); - }); -}); - -describe("inspectCandidate", () => { - const live = "https://swebtoon-phinf.pstatic.net/a/95.jpg"; - const archived = "https://web.archive.org/web/20150312154616id_/http://swebtoon.phinf.naver.net/a/95.jpg"; - const candidate: CoverCandidate = { url: live, role: "poster", current: false, seenAt: "20150312154616", archivedUrl: archived }; - - test("live image: measured, hashed, classified", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - const fake = new FakeTransport().on(live, () => bytes(poster)); - const c = await inspectCandidate(candidate, fake); - expect(c).toMatchObject({ fetchedFrom: live, live: true, width: 480, height: 623, ext: "jpg", shape: "portrait", type: "volume" }); - expect(c.sha256).toBe(await sha256Hex(poster)); - }); - - test("falls back to the archived copy when the live URL is gone or not an image", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - for (const gone of [() => text("nope", 404), () => text("image not found")]) { - const fake = new FakeTransport().on(live, gone).on(archived, () => bytes(poster)); - const c = await inspectCandidate(candidate, fake); - expect(c).toMatchObject({ fetchedFrom: archived, live: false }); - } - }); - - test("throws ImageError naming every URL tried", async () => { - const fake = new FakeTransport().on(() => true, () => text("nope", 404)); - await expect(inspectCandidate(candidate, fake)).rejects.toThrow(/95\.jpg.*404.*404/); - }); - - test("inspectAll drops failures with a warning and reports progress", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - const fake = new FakeTransport().on(live, () => bytes(poster)).on(() => true, () => text("nope", 404)); - const warnings: string[] = []; - const progress: string[] = []; - const out = await inspectAll( - [candidate, { url: "https://swebtoon-phinf.pstatic.net/missing.jpg", role: "square", current: true }], - fake, - { warn: (m) => warnings.push(m), onProgress: (d, t) => progress.push(`${d}/${t}`) }, - ); - expect(out).toHaveLength(1); - expect(warnings).toHaveLength(1); - expect(warnings[0]).toContain("missing.jpg"); - expect(progress).toEqual(["1/2", "2/2"]); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/images.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/core/images.ts`** - -```ts -import type { CoverCandidate, ImageExt, ImageType, InspectedCover, Shape } from "./model"; -import type { Transport } from "./transport"; - -export const MAX_IMAGE_BYTES = 20 * 1024 * 1024; - -export class ImageError extends Error { - override name = "ImageError"; -} - -export interface ImageInfo { - width: number; - height: number; - contentType: string; - ext: ImageExt; -} - -const u16be = (b: Uint8Array, i: number) => (b[i]! << 8) | b[i + 1]!; -const u16le = (b: Uint8Array, i: number) => b[i]! | (b[i + 1]! << 8); -const u24le = (b: Uint8Array, i: number) => b[i]! | (b[i + 1]! << 8) | (b[i + 2]! << 16); -const u32be = (b: Uint8Array, i: number) => ((b[i]! << 24) | (b[i + 1]! << 16) | (b[i + 2]! << 8) | b[i + 3]!) >>> 0; -const ascii = (b: Uint8Array, i: number, n: number) => String.fromCharCode(...b.subarray(i, i + n)); - -/** Dimensions from the header bytes, so no DOM is needed (spec §4.4). */ -export function probeImage(b: Uint8Array): ImageInfo | null { - if (b.length >= 24 && u32be(b, 0) === 0x89504e47) { - return { width: u32be(b, 16), height: u32be(b, 20), contentType: "image/png", ext: "png" }; - } - if (b.length >= 4 && b[0] === 0xff && b[1] === 0xd8) return probeJpeg(b); - if (b.length >= 10 && ascii(b, 0, 3) === "GIF") { - return { width: u16le(b, 6), height: u16le(b, 8), contentType: "image/gif", ext: "gif" }; - } - if (b.length >= 30 && ascii(b, 0, 4) === "RIFF" && ascii(b, 8, 4) === "WEBP") return probeWebp(b); - return null; -} - -function probeJpeg(b: Uint8Array): ImageInfo | null { - let i = 2; - while (i + 9 < b.length) { - if (b[i] !== 0xff) { - i++; - continue; - } - const marker = b[i + 1]!; - if (marker === 0xff) { - i++; - continue; - } - if (marker === 0xd8 || marker === 0x01 || (marker >= 0xd0 && marker <= 0xd7)) { - i += 2; - continue; - } - // SOF0–SOF15, except DHT (C4), JPG (C8) and DAC (CC) - if (marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8 && marker !== 0xcc) { - return { width: u16be(b, i + 7), height: u16be(b, i + 5), contentType: "image/jpeg", ext: "jpg" }; - } - i += 2 + u16be(b, i + 2); - } - return null; -} - -function probeWebp(b: Uint8Array): ImageInfo | null { - const base = { contentType: "image/webp", ext: "webp" as const }; - switch (ascii(b, 12, 4)) { - case "VP8 ": - return { width: u16le(b, 26) & 0x3fff, height: u16le(b, 28) & 0x3fff, ...base }; - case "VP8L": { - const bits = (b[21]! | (b[22]! << 8) | (b[23]! << 16) | (b[24]! << 24)) >>> 0; - return { width: (bits & 0x3fff) + 1, height: ((bits >>> 14) & 0x3fff) + 1, ...base }; - } - case "VP8X": - return { width: u24le(b, 24) + 1, height: u24le(b, 27) + 1, ...base }; - default: - return null; - } -} - -export async function sha256Hex(bytes: Uint8Array): Promise { - const digest = await crypto.subtle.digest("SHA-256", bytes as Uint8Array); - return [...new Uint8Array(digest)].map((x) => x.toString(16).padStart(2, "0")).join(""); -} - -/** Spec §4.4: h/w > 1.6 is landing-page art; real posters are ~1.3. */ -export function shapeOf(width: number, height: number): Shape { - const r = height / width; - if (r > 1.6) return "tall"; - if (r >= 1.15) return "portrait"; - if (r >= 0.85) return "square"; - if (r < 0.5) return "wide"; - return "other"; -} - -export function typeForShape(shape: Shape): ImageType { - return shape === "portrait" ? "volume" : shape === "wide" ? "banner" : "other"; -} - -export async function readCapped(res: Response, max = MAX_IMAGE_BYTES): Promise { - const declared = Number(res.headers.get("content-length")); - if (Number.isFinite(declared) && declared > max) { - await res.body?.cancel(); - throw new ImageError(`larger than ${Math.round(max / 1024 / 1024)} MB`); - } - if (!res.body) return new Uint8Array(0); - const reader = res.body.getReader(); - const parts: Uint8Array[] = []; - let total = 0; - for (;;) { - const { done, value } = await reader.read(); - if (done) break; - total += value.byteLength; - if (total > max) { - await reader.cancel(); - throw new ImageError(`larger than ${Math.round(max / 1024 / 1024)} MB`); - } - parts.push(value); - } - const out = new Uint8Array(total); - let offset = 0; - for (const p of parts) { - out.set(p, offset); - offset += p.byteLength; - } - return out; -} - -/** Live URL first, then the Wayback copy (spec §4.4). */ -export async function inspectCandidate(c: CoverCandidate, transport: Transport, signal?: AbortSignal): Promise { - const attempts: [string, boolean][] = [[c.url, true]]; - if (c.archivedUrl) attempts.push([c.archivedUrl, false]); - const tried: string[] = []; - for (const [url, live] of attempts) { - try { - const res = await transport.fetch(url, { signal }); - if (!res.ok) { - await res.body?.cancel(); - tried.push(`${url}: HTTP ${res.status}`); - continue; - } - const bytes = await readCapped(res); - const info = probeImage(bytes); - if (!info) { - tried.push(`${url}: not an image`); - continue; - } - const shape = shapeOf(info.width, info.height); - return { - candidate: c, fetchedFrom: url, live, width: info.width, height: info.height, - sha256: await sha256Hex(bytes), contentType: info.contentType, ext: info.ext, bytes, shape, type: typeForShape(shape), - }; - } catch (err) { - if (signal?.aborted) throw err; - tried.push(`${url}: ${err instanceof Error ? err.message : String(err)}`); - } - } - throw new ImageError(`Could not download ${c.url} (${tried.join("; ")})`); -} - -export async function inspectAll( - candidates: readonly CoverCandidate[], - transport: Transport, - opts: { warn(message: string): void; onProgress?(done: number, total: number): void; signal?: AbortSignal }, -): Promise { - const out: InspectedCover[] = []; - let done = 0; - for (const c of candidates) { - try { - out.push(await inspectCandidate(c, transport, opts.signal)); - } catch (err) { - if (opts.signal?.aborted) throw err; - opts.warn(err instanceof Error ? err.message : String(err)); - } - opts.onProgress?.(++done, candidates.length); - } - return out; -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/images.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/core/images.ts test/images.test.ts -git commit -m "feat(core): inspect images (size from headers, sha256, shape incl. landing-page art)" -``` - ---- - -### Task 8: Merge, dedupe, compare with MangaBaka, default, notes - -**Files:** -- Create: `src/core/covers.ts`, `test/helpers/covers.ts`, `test/covers.test.ts` - -**Interfaces:** -- Consumes: Task 3 types. -- Produces: `interface ExistingImage { id: number; sha256: string | null; language: string; type: ImageType }`, `mergeCandidates(a, b): CoverCandidate`, `mergeByUrl(list): CoverCandidate[]`, `dedupeByHash(list): InspectedCover[]`, `splitExisting(list, existing): { fresh: InspectedCover[]; present: { cover: InspectedCover; existing: ExistingImage }[] }`, `pickDefault(list): InspectedCover | null`, `formatMonth(ts): string`, `coverNote(cover, sourceName): string`. -- Produces (test helper): `inspected(overrides?: { candidate?: Partial } & Partial>): InspectedCover`. - -- [ ] **Step 1: Write the test helper** - -`test/helpers/covers.ts`: -```ts -import type { CoverCandidate, InspectedCover } from "../../src/core/model"; - -let n = 0; -/** A plausible InspectedCover for unit tests; override what the test is about. */ -export function inspected( - overrides: { candidate?: Partial } & Partial> = {}, -): InspectedCover { - n++; - const { candidate, ...rest } = overrides; - const url = candidate?.url ?? `https://swebtoon-phinf.pstatic.net/x/${n}.jpg`; - return { - candidate: { url, role: "poster", current: true, ...candidate }, - fetchedFrom: url, - live: true, - width: 480, - height: 623, - sha256: n.toString(16).padStart(64, "0"), - contentType: "image/jpeg", - ext: "jpg", - bytes: new Uint8Array([n]), - shape: "portrait", - type: "volume", - ...rest, - }; -} -``` - -- [ ] **Step 2: Write the failing tests** - -`test/covers.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { coverNote, dedupeByHash, mergeByUrl, pickDefault, splitExisting } from "../src/core/covers"; -import { inspected } from "./helpers/covers"; - -describe("mergeByUrl", () => { - test("merges current and historical sightings of one URL", () => { - const url = "https://swebtoon-phinf.pstatic.net/a.jpg"; - const merged = mergeByUrl([ - { url, role: "poster", current: false, seenAt: "20190301000000", lastSeenAt: "20190301000000", archivedUrl: "A" }, - { url, role: "poster", current: true }, - { url, role: "poster", current: false, seenAt: "20210101000000", lastSeenAt: "20210101000000", archivedUrl: "B" }, - { url: "https://swebtoon-phinf.pstatic.net/b.jpg", role: "banner", current: true }, - ]); - expect(merged).toHaveLength(2); - expect(merged[0]).toEqual({ - url, role: "poster", current: true, seenAt: "20190301000000", lastSeenAt: "20210101000000", archivedUrl: "A", - }); - }); -}); - -describe("dedupeByHash", () => { - test("same bytes under two URLs become one cover; the live, current one is kept", () => { - const sha = "f".repeat(64); - const archivedOnly = inspected({ sha256: sha, live: false, candidate: { url: "https://x/old.jpg", current: false, seenAt: "20150101000000" } }); - const current = inspected({ sha256: sha, candidate: { url: "https://x/new.jpg", current: true } }); - const out = dedupeByHash([archivedOnly, current]); - expect(out).toHaveLength(1); - expect(out[0]!.candidate).toMatchObject({ url: "https://x/new.jpg", current: true, seenAt: "20150101000000" }); - expect(out[0]!.live).toBe(true); - }); -}); - -test("splitExisting drops what MangaBaka already has, by sha256", () => { - const a = inspected(); - const b = inspected(); - const { fresh, present } = splitExisting([a, b], [ - { id: 7193, sha256: b.sha256.toUpperCase(), language: "en", type: "volume" }, - { id: 1, sha256: null, language: "ko", type: "volume" }, - ]); - expect(fresh).toEqual([a]); - expect(present).toEqual([{ cover: b, existing: { id: 7193, sha256: b.sha256.toUpperCase(), language: "en", type: "volume" } }]); -}); - -describe("pickDefault", () => { - test("current live poster", () => { - const square = inspected({ candidate: { role: "square" }, shape: "square" }); - const poster = inspected({ candidate: { role: "poster" } }); - expect(pickDefault([square, poster])).toBe(poster); - }); - test("canvas: current live square when there is no poster", () => { - const square = inspected({ candidate: { role: "square" }, shape: "square" }); - expect(pickDefault([square])).toBe(square); - }); - test("never landing-page art, a historical cover, or an archive-only copy", () => { - expect(pickDefault([ - inspected({ shape: "tall" }), - inspected({ candidate: { current: false } }), - inspected({ live: false }), - ])).toBeNull(); - }); -}); - -describe("coverNote", () => { - test.each([ - [{ current: true }, "portrait", "WEBTOON poster (current)"], - [{ current: false, seenAt: "20190318000000", lastSeenAt: "20210127000000" }, "portrait", "WEBTOON poster, seen 2019-03 – 2021-01"], - [{ current: false, seenAt: "20190318000000", lastSeenAt: "20190330000000" }, "portrait", "WEBTOON poster, seen 2019-03"], - [{ current: true, seenAt: "20230401000000" }, "portrait", "WEBTOON poster (current; archived since 2023-04)"], - [{ current: false, seenAt: "20230401000000", role: "poster" as const }, "tall", "WEBTOON poster (landing-page art), seen 2023-04"], - [{ current: true, role: "foreground" as const }, "wide", "WEBTOON foreground art (current)"], - ] as const)("%o", (candidate, shape, note) => { - expect(coverNote(inspected({ candidate, shape }), "WEBTOON")).toBe(note); - }); -}); -``` - -- [ ] **Step 3: Run to verify they fail** - -Run: `bun test test/covers.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 4: Write `src/core/covers.ts`** - -```ts -import type { CoverCandidate, CoverRole, ImageType, InspectedCover } from "./model"; - -/** A cover already on MangaBaka (from the public images endpoint). */ -export interface ExistingImage { - id: number; - sha256: string | null; - language: string; - type: ImageType; -} - -const earlier = (a?: string, b?: string) => (!a ? b : !b ? a : a < b ? a : b); -const later = (a?: string, b?: string) => (!a ? b : !b ? a : a > b ? a : b); - -/** Keeps `a`'s URL; widest seen range; current if either is. */ -export function mergeCandidates(a: CoverCandidate, b: CoverCandidate): CoverCandidate { - const merged: CoverCandidate = { - ...a, - role: a.current || !b.current ? a.role : b.role, - current: a.current || b.current, - }; - const seenAt = earlier(a.seenAt, b.seenAt); - const lastSeenAt = later(a.lastSeenAt ?? a.seenAt, b.lastSeenAt ?? b.seenAt); - const originalUrl = a.originalUrl ?? b.originalUrl; - const archivedUrl = a.archivedUrl ?? b.archivedUrl; - if (seenAt) merged.seenAt = seenAt; - if (lastSeenAt) merged.lastSeenAt = lastSeenAt; - if (originalUrl) merged.originalUrl = originalUrl; - if (archivedUrl) merged.archivedUrl = archivedUrl; - return merged; -} - -export function mergeByUrl(list: readonly CoverCandidate[]): CoverCandidate[] { - const byUrl = new Map(); - for (const c of list) { - const prev = byUrl.get(c.url); - byUrl.set(c.url, prev ? mergeCandidates(prev, c) : { ...c }); - } - return [...byUrl.values()]; -} - -const rank = (c: InspectedCover) => (c.live ? 2 : 0) + (c.candidate.current ? 1 : 0); - -/** Same bytes = same cover (spec §4.5): keep the live URL and the widest seen range. */ -export function dedupeByHash(list: readonly InspectedCover[]): InspectedCover[] { - const bySha = new Map(); - for (const c of list) { - const prev = bySha.get(c.sha256); - if (!prev) { - bySha.set(c.sha256, c); - continue; - } - const [keep, other] = rank(c) > rank(prev) ? [c, prev] : [prev, c]; - bySha.set(c.sha256, { ...keep, candidate: mergeCandidates(keep.candidate, other.candidate) }); - } - return [...bySha.values()]; -} - -export function splitExisting(list: readonly InspectedCover[], existing: readonly ExistingImage[]) { - const bySha = new Map(existing.filter((e) => e.sha256).map((e) => [e.sha256!.toLowerCase(), e])); - const fresh: InspectedCover[] = []; - const present: { cover: InspectedCover; existing: ExistingImage }[] = []; - for (const cover of list) { - const e = bySha.get(cover.sha256); - if (e) present.push({ cover, existing: e }); - else fresh.push(cover); - } - return { fresh, present }; -} - -/** Current live poster, else current live square (canvas). Never landing-page art. */ -export function pickDefault(list: readonly InspectedCover[]): InspectedCover | null { - const live = list.filter((c) => c.candidate.current && c.live && c.shape !== "tall"); - return live.find((c) => c.candidate.role === "poster") ?? live.find((c) => c.candidate.role === "square") ?? null; -} - -const ROLE_LABEL: Readonly> = { - poster: "poster", square: "square thumbnail", banner: "banner", foreground: "foreground art", other: "image", -}; - -export const formatMonth = (ts: string): string => `${ts.slice(0, 4)}-${ts.slice(4, 6)}`; - -export function coverNote(cover: InspectedCover, sourceName: string): string { - const { role, current, seenAt, lastSeenAt } = cover.candidate; - const base = `${sourceName} ${ROLE_LABEL[role]}${cover.shape === "tall" ? " (landing-page art)" : ""}`; - if (!seenAt) return `${base} (current)`; - const from = formatMonth(seenAt); - if (current) return `${base} (current; archived since ${from})`; - const to = formatMonth(lastSeenAt ?? seenAt); - return from === to ? `${base}, seen ${from}` : `${base}, seen ${from} – ${to}`; -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/covers.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/covers.ts test/helpers/covers.ts test/covers.test.ts -git commit -m "feat(core): merge, dedupe, compare with MangaBaka, pick the default, write notes" -``` - ---- - -### Task 9: Wayback history with granularity - -**Files:** -- Create: `src/core/history/wayback.ts`, `test/wayback.test.ts` - -**Interfaces:** -- Consumes: `Source` (Task 6), `mergeByUrl`, `formatMonth` (Task 8), `Granularity`, `Transport`. -- Produces: `WAYBACK = "https://web.archive.org"`, `interface CdxRow { timestamp: string; original: string }`, `cdxUrl(prefix)`, `parseCdx(text): CdxRow[]`, `pickSnapshots(rows, every: Granularity, preferHost: string): CdxRow[]`, `snapshotUrl(row)`, `archivedImageUrl(timestamp, originalImageUrl)`, `interface HistoryOptions { every: Granularity; transport: Transport; warn(msg: string): void; onProgress?(done: number, total: number): void; signal?: AbortSignal }`, `waybackHistory(source, series, opts): Promise`. - -- [ ] **Step 1: Write the failing tests** - -`test/wayback.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { cdxUrl, parseCdx, pickSnapshots, waybackHistory } from "../src/core/history/wayback"; -import type { SourceSeries } from "../src/core/model"; -import { webtoons } from "../src/core/sources/webtoons"; -import { FakeTransport, text } from "./helpers/fake-transport"; -import { fixtureText } from "./helpers/fixtures"; - -const SERIES: SourceSeries = { - source: "webtoons", externalId: "95", title: "Tower of God", language: "en", - canonicalUrl: "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", - archiveUrls: ["www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", "m.webtoons.com/en/fantasy/tower-of-god/list?title_no=95"], - mature: false, flags: { canvas: false }, -}; -const W = "http://www.webtoons.com:80/en/fantasy/tower-of-god/list?title_no=95"; -const M = "http://m.webtoons.com/en/fantasy/tower-of-god/list?title_no=95"; - -test("cdxUrl asks for one capture per month of the prefix", () => { - const u = new URL(cdxUrl(SERIES.archiveUrls[0]!)); - expect(u.origin + u.pathname).toBe("https://web.archive.org/cdx/search/cdx"); - expect(Object.fromEntries(u.searchParams)).toEqual({ - url: SERIES.archiveUrls[0], matchType: "prefix", filter: "statuscode:200", collapse: "timestamp:6", fl: "timestamp,original", - }); -}); - -test("parseCdx reads the captured index and skips junk lines", async () => { - const rows = parseCdx(await fixtureText("wayback/tower-of-god-cdx-www.txt")); - expect(rows.length).toBeGreaterThan(100); - expect(rows[0]!.timestamp).toMatch(/^2015\d{10}$/); - expect(parseCdx("garbage\n\n20150312154616 http://x/list?title_no=95\n")).toEqual([ - { timestamp: "20150312154616", original: "http://x/list?title_no=95" }, - ]); -}); - -describe("pickSnapshots", () => { - // Synthetic rows: they test the bucketing arithmetic, not a service response. - const rows = [ - { timestamp: "20150312000000", original: W }, - { timestamp: "20150401000000", original: M }, - { timestamp: "20150501000000", original: W }, - { timestamp: "20150901000000", original: M }, - { timestamp: "20151001000000", original: W }, - { timestamp: "20160301000000", original: W }, - ]; - test("one capture per bucket of N months from the first capture, www. preferred", () => { - expect(pickSnapshots(rows, 6, "www.webtoons.com").map((r) => r.timestamp)) - .toEqual(["20150312000000", "20151001000000", "20160301000000"]); - expect(pickSnapshots(rows, 12, "www.webtoons.com").map((r) => r.timestamp)) - .toEqual(["20150312000000", "20160301000000"]); - expect(pickSnapshots(rows, 1, "www.webtoons.com")).toHaveLength(6); - }); - test("falls back to m. when a bucket has no www. capture", () => { - expect(pickSnapshots([rows[1]!, rows[3]!], 3, "www.webtoons.com").map((r) => r.original)).toEqual([M, M]); - }); - test("real index, every 6 months: one per half-year bucket, in order", async () => { - const real = parseCdx(await fixtureText("wayback/tower-of-god-cdx-www.txt")); - const picks = pickSnapshots(real, 6, "www.webtoons.com"); - const months = new Set(real.map((r) => r.timestamp.slice(0, 6))).size; - expect(picks.length).toBeLessThanOrEqual(Math.ceil(months / 6) + 12); - expect(picks.length).toBeGreaterThanOrEqual(15); - expect([...picks].sort((a, b) => a.timestamp.localeCompare(b.timestamp))).toEqual(picks); - }); -}); - -function waybackFake(cdxWww: string, snapshot: (url: string) => Response) { - return new FakeTransport() - .on((u) => u.startsWith("https://web.archive.org/cdx/") && u.includes("www.webtoons.com"), () => text(cdxWww)) - .on((u) => u.startsWith("https://web.archive.org/cdx/"), () => text("")) - .on((u) => u.includes("id_/"), (u) => snapshot(u)); -} - -describe("waybackHistory", () => { - test("drops other title_no rows before picking snapshots", async () => { - // Review Focus 1: matchType=prefix on title_no=95 also returns title_no=950. - const cdx = `20150312154616 ${W}\n20160101000000 http://www.webtoons.com/en/fantasy/x/list?title_no=950\n`; - const page2015 = await fixtureText("wayback/tower-of-god-2015.html"); - const fake = waybackFake(cdx, () => text(page2015)); - await waybackHistory(webtoons, SERIES, { every: 1, transport: fake, warn: () => {} }); - const snapshots = fake.urls().filter((u) => u.includes("id_/")); - expect(snapshots).toEqual([`https://web.archive.org/web/20150312154616id_/${W}`]); - }); - - test("turns snapshots into merged historical candidates with archive fallbacks", async () => { - const cdx = `20150312154616 ${W}\n20150901000000 ${W}\n20230401000000 ${W}\n`; - const [p2015, p2023] = [await fixtureText("wayback/tower-of-god-2015.html"), await fixtureText("wayback/tower-of-god-2023.html")]; - const fake = waybackFake(cdx, (u) => text(u.includes("/2023") ? p2023 : p2015)); - const progress: string[] = []; - const found = await waybackHistory(webtoons, SERIES, { - every: 1, transport: fake, warn: () => {}, onProgress: (d, t) => progress.push(`${d}/${t}`), - }); - expect(progress).toEqual(["1/3", "2/3", "3/3"]); - const poster2015 = found.find((c) => c.url.endsWith("01_EC8BA0EC9D98ED8391.jpg"))!; - expect(poster2015).toMatchObject({ - role: "poster", current: false, seenAt: "20150312154616", lastSeenAt: "20150901000000", - archivedUrl: "https://web.archive.org/web/20150312154616id_/http://swebtoon.phinf.naver.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg", - }); - expect(found.some((c) => c.seenAt?.startsWith("2023"))).toBe(true); - }); - - test("a refused snapshot stops the history and keeps what was found", async () => { - const cdx = `20150312154616 ${W}\n20160301000000 ${W}\n20170301000000 ${W}\n`; - const page2015 = await fixtureText("wayback/tower-of-god-2015.html"); - const fake = waybackFake(cdx, (u) => (u.includes("/2015") ? text(page2015) : text("blocked", 429))); - const warnings: string[] = []; - const found = await waybackHistory(webtoons, SERIES, { every: 1, transport: fake, warn: (m) => warnings.push(m) }); - expect(found.length).toBeGreaterThan(0); - expect(fake.urls().filter((u) => u.includes("id_/"))).toHaveLength(2); - expect(warnings).toEqual([expect.stringContaining("history stops at 2016-03")]); - }); - - test("an unavailable index is a warning, not an error", async () => { - const fake = new FakeTransport().on(() => true, () => text("busy", 503)); - const warnings: string[] = []; - expect(await waybackHistory(webtoons, SERIES, { every: 6, transport: fake, warn: (m) => warnings.push(m) })).toEqual([]); - expect(warnings).toHaveLength(2); - }); - - test("cancelling keeps what was found so far", async () => { - const cdx = `20150312154616 ${W}\n20160301000000 ${W}\n`; - const page2015 = await fixtureText("wayback/tower-of-god-2015.html"); - const controller = new AbortController(); - const fake = waybackFake(cdx, () => { - controller.abort(); - return text(page2015); - }); - const found = await waybackHistory(webtoons, SERIES, { every: 1, transport: fake, warn: () => {}, signal: controller.signal }); - expect(found.length).toBeGreaterThan(0); - expect(fake.urls().filter((u) => u.includes("id_/"))).toHaveLength(1); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/wayback.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/core/history/wayback.ts`** - -```ts -import { formatMonth, mergeByUrl } from "../covers"; -import type { CoverCandidate, Granularity, SourceSeries } from "../model"; -import type { Source } from "../sources/source"; -import type { Transport } from "../transport"; - -export const WAYBACK = "https://web.archive.org"; - -export interface CdxRow { - timestamp: string; - original: string; -} - -export interface HistoryOptions { - every: Granularity; - transport: Transport; - warn(message: string): void; - onProgress?(done: number, total: number): void; - signal?: AbortSignal; -} - -/** Every month with a capture (collapse=timestamp:6) — one cheap request per prefix (spec §4.3). */ -export function cdxUrl(prefix: string): string { - const q = new URLSearchParams({ - url: prefix, matchType: "prefix", filter: "statuscode:200", collapse: "timestamp:6", fl: "timestamp,original", - }); - return `${WAYBACK}/cdx/search/cdx?${q}`; -} - -export function parseCdx(text: string): CdxRow[] { - const rows: CdxRow[] = []; - for (const line of text.split("\n")) { - const [timestamp, original] = line.trim().split(/\s+/); - if (timestamp && original && /^\d{14}$/.test(timestamp)) rows.push({ timestamp, original }); - } - return rows; -} - -const monthIndex = (ts: string) => Number(ts.slice(0, 4)) * 12 + Number(ts.slice(4, 6)) - 1; -const hostIs = (row: CdxRow, host: string) => { - try { - return new URL(row.original).hostname === host; - } catch { - return false; - } -}; - -/** Buckets of `every` months from the first capture; first capture per bucket, preferring `preferHost`. */ -export function pickSnapshots(rows: readonly CdxRow[], every: Granularity, preferHost: string): CdxRow[] { - const sorted = [...rows].sort((a, b) => a.timestamp.localeCompare(b.timestamp)); - if (!sorted.length) return []; - const first = monthIndex(sorted[0]!.timestamp); - const buckets = new Map(); - for (const row of sorted) { - const bucket = Math.floor((monthIndex(row.timestamp) - first) / every); - const current = buckets.get(bucket); - if (!current || (!hostIs(current, preferHost) && hostIs(row, preferHost))) buckets.set(bucket, row); - } - return [...buckets.values()].sort((a, b) => a.timestamp.localeCompare(b.timestamp)); -} - -export const snapshotUrl = (row: CdxRow): string => `${WAYBACK}/web/${row.timestamp}id_/${row.original}`; -export const archivedImageUrl = (timestamp: string, originalImageUrl: string): string => - `${WAYBACK}/web/${timestamp}id_/${originalImageUrl}`; - -const message = (err: unknown) => (err instanceof Error ? err.message : String(err)); - -export async function waybackHistory(source: Source, series: SourceSeries, opts: HistoryOptions): Promise { - if (!source.extractCovers) return []; - const rows: CdxRow[] = []; - for (const prefix of series.archiveUrls) { - try { - const res = await opts.transport.fetch(cdxUrl(prefix), { signal: opts.signal }); - if (!res.ok) { - await res.body?.cancel(); - opts.warn(`The Wayback index didn't answer (HTTP ${res.status}); no history from ${prefix}.`); - continue; - } - // matchType=prefix on title_no=95 also matches title_no=950 (spec §4.3). - rows.push(...parseCdx(await res.text()).filter((r) => source.isSameSeries(r.original, series))); - } catch (err) { - if (opts.signal?.aborted) return []; - opts.warn(`The Wayback index didn't answer (${message(err)}); no history from ${prefix}.`); - } - } - - const preferHost = series.archiveUrls[0]?.split("/")[0] ?? ""; - const picks = pickSnapshots(rows, opts.every, preferHost); - const found: CoverCandidate[] = []; - let done = 0; - for (const row of picks) { - if (opts.signal?.aborted) break; - let res: Response; - try { - res = await opts.transport.fetch(snapshotUrl(row), { signal: opts.signal }); - } catch (err) { - if (opts.signal?.aborted) break; - opts.warn(`Wayback stopped answering (${message(err)}); history stops at ${formatMonth(row.timestamp)}.`); - break; - } - if (res.status === 429 || res.status >= 500) { - await res.body?.cancel(); - opts.warn(`Wayback is refusing requests (HTTP ${res.status}); history stops at ${formatMonth(row.timestamp)}.`); - break; - } - if (res.ok) { - const pageUrl = new URL(row.original); - for (const c of source.extractCovers(await res.text(), pageUrl)) { - const candidate: CoverCandidate = { ...c, current: false, seenAt: row.timestamp, lastSeenAt: row.timestamp }; - if (c.originalUrl) candidate.archivedUrl = archivedImageUrl(row.timestamp, c.originalUrl); - found.push(candidate); - } - } else { - await res.body?.cancel(); - opts.warn(`Snapshot ${formatMonth(row.timestamp)} is unavailable (HTTP ${res.status}).`); - } - opts.onProgress?.(++done, picks.length); - } - return mergeByUrl(found); -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/wayback.test.ts` → Expected: all PASS. If "real index, every 6 months" fails on the bounds, print `picks.length` and `months`: the bounds are deliberately loose; tighten them to the observed values only if they are clearly right for the captured index. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/core/history/wayback.ts test/wayback.test.ts -git commit -m "feat(core): Wayback history with 1/3/6/12-month granularity; stop on refusal" -``` - ---- - -### Task 10: MangaBaka types and client - -**Files:** -- Create: `src/core/mangabaka/types.ts`, `src/core/mangabaka/client.ts`, `test/mangabaka-client.test.ts` - -**Interfaces:** -- Consumes: `Transport` (Task 3), `ImageType`, `ContentRating`, `ExistingImage` (Task 8). -- Produces (`types.ts`): `SeriesSummary { id: number; title: string; type: string; content_rating: string | null }`, `SeriesLink { url: string; name: string; language: string | null }`, `PublicImage { id: number; series_id: number; type: ImageType; language: string; note: string | null; content_rating: ContentRating; hashes: { sha256?: string | null } | null }`, `SubmissionImage` (spec §4.8 shape), `ImageSnapshot { version: number; images: SubmissionImage[] }`, `SubmissionChange`, `PreviewResult { has_changes: boolean; changes: SubmissionChange[] }`, `SaveResult { submission_id: number; status: string; changes: SubmissionChange[] }`, `toExisting(img: PublicImage): ExistingImage`. -- Produces (`client.ts`): `API_BASE`, `SITE_BASE`, `coverFormUrl(seriesId)`, `class MangaBakaError { status: number; body?: unknown }`, `class MangaBakaClient(transport, token?)` with `getSeries(id)`, `getLinks(id)`, `getImages(id)`, `getImageSnapshot(id)`, `previewImages(id, version, images)`, `saveImagesForReview(id, version, images, userNote)`. - -- [ ] **Step 1: Write the failing tests** - -`test/mangabaka-client.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { MangaBakaClient, MangaBakaError, coverFormUrl } from "../src/core/mangabaka/client"; -import { toExisting, type SubmissionImage } from "../src/core/mangabaka/types"; -import { FakeTransport, json } from "./helpers/fake-transport"; -import { fixtureJson } from "./helpers/fixtures"; - -const API = "https://api.mangabaka.org"; - -function publicFake() { - return new FakeTransport() - .on(`${API}/v1/series/9999`, async () => json(await fixtureJson("mangabaka/series-9999.json"))) - .on(`${API}/v1/series/9999/links?limit=100`, async () => json(await fixtureJson("mangabaka/links-9999.json"))) - .on(/\/v1\/series\/9999\/images\?.*page=2/, async () => json(await fixtureJson("mangabaka/images-9999-page2.json"))) - .on(`${API}/v1/series/9999/images?limit=50`, async () => json(await fixtureJson("mangabaka/images-9999-page1.json"))); -} - -describe("public reads (captured fixtures)", () => { - test("series, links, and every page of images", async () => { - const fake = publicFake(); - const mb = new MangaBakaClient(fake); - expect((await mb.getSeries(9999)).title).toBe("Tower of God"); - expect((await mb.getLinks(9999)).map((l) => l.url)).toContain("https://www.webtoons.com/-/-/-/list?title_no=95"); - const images = await mb.getImages(9999); - expect(images).toHaveLength(31); - expect(fake.calls.some((c) => c.init.headers?.["x-api-key"])).toBe(false); - expect(toExisting(images[0]!)).toEqual({ - id: images[0]!.id, sha256: images[0]!.hashes!.sha256, language: images[0]!.language, type: images[0]!.type, - }); - }); - - test("a next link to another origin is not followed", async () => { - const fake = new FakeTransport().on(`${API}/v1/series/1/images?limit=50`, () => - json({ status: 200, data: [], pagination: { next: "https://evil.example/page2" } })); - expect(await new MangaBakaClient(fake).getImages(1)).toEqual([]); - expect(fake.calls).toHaveLength(1); - }); - - test("404 is a MangaBakaError", async () => { - const fake = new FakeTransport().on(() => true, () => json({ status: 404, message: "Not found" }, 404)); - const err = await new MangaBakaClient(fake).getSeries(1).catch((e) => e); - expect(err).toBeInstanceOf(MangaBakaError); - expect(err.status).toBe(404); - }); -}); - -// Not capturable without a token: shapes built from the OpenAPI spec (spec §2.3). See docs/TODO.md. -const existing: SubmissionImage = { - id: 7193, series_id: 9999, work_id: null, url: "https://example.org/old.jpg", index: "0.1", index_numeric: 0.1, - language: "ko", type: "volume", note: "Season Cover 3", content_rating: "safe", is_default: true, -}; -const snapshotBody = { status: 200, data: { data: { images: [existing] }, version: 1760000000 } }; -const previewBody = { status: 200, data: { has_changes: true, changes: [{ field: "images", type: "added", identity: "https://x/new.jpg" }] } }; - -describe("submissions (token)", () => { - const SUB = `${API}/v0/my/submissions/series-images/9999`; - - test("snapshot, preview and save send the token and the right bodies", async () => { - const fake = new FakeTransport() - .on((u, i) => u === SUB && (i.method ?? "GET") === "GET", () => json(snapshotBody)) - .on(`${SUB}/preview`, () => json(previewBody)) - .on((u, i) => u === SUB && i.method === "POST", () => - json({ status: 200, data: { submission_id: 5, status: "pending", changes: [] } })); - const mb = new MangaBakaClient(fake, "mb-test-token"); - const snapshot = await mb.getImageSnapshot(9999); - expect(snapshot).toEqual({ version: 1760000000, images: [existing] }); - expect((await mb.previewImages(9999, snapshot.version, snapshot.images)).has_changes).toBe(true); - const saved = await mb.saveImagesForReview(9999, snapshot.version, snapshot.images, "Add WEBTOON covers"); - expect(saved.submission_id).toBe(5); - - expect(fake.calls.every((c) => c.init.headers?.["x-api-key"] === "mb-test-token")).toBe(true); - expect(JSON.parse(fake.calls[1]!.init.body as string)).toEqual({ version: 1760000000, data: { images: [existing] } }); - expect(JSON.parse(fake.calls[2]!.init.body as string)).toEqual({ - version: 1760000000, user_note: "Add WEBTOON covers", save_mode: "review", data: { images: [existing] }, - }); - }); - - test("no token: refuses before sending anything", async () => { - const fake = new FakeTransport(); - await expect(new MangaBakaClient(fake).getImageSnapshot(9999)).rejects.toMatchObject({ status: 401 }); - expect(fake.calls).toHaveLength(0); - }); - - test.each([ - [403, /25 approved submissions/], - [409, /changed since/], - [401, /token/], - ])("HTTP %i explains itself", async (status, message) => { - const fake = new FakeTransport().on(() => true, () => json({ status, message: "x" }, status)); - const err = await new MangaBakaClient(fake, "mb-t").previewImages(9999, 1, []).catch((e) => e); - expect(err).toBeInstanceOf(MangaBakaError); - expect(err.status).toBe(status); - expect(err.message).toMatch(message); - }); -}); - -test("coverFormUrl", () => { - expect(coverFormUrl(9999)).toBe("https://mangabaka.org/data/submissions/series/9999/images"); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/mangabaka-client.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/core/mangabaka/types.ts`** - -```ts -// Subset of https://mangabaka.org/api.json (downloaded 2026-10-14) that we use. See spec §2.3. -import type { ExistingImage } from "../covers"; -import type { ContentRating, ImageType } from "../model"; - -export interface SeriesSummary { - id: number; - title: string; - type: string; - content_rating: string | null; -} - -export interface SeriesLink { - url: string; - name: string; - language: string | null; -} - -/** GET /v1/series/{id}/images item. Has no `is_default`. */ -export interface PublicImage { - id: number; - series_id: number; - type: ImageType; - language: string; - note: string | null; - content_rating: ContentRating; - hashes: { sha256?: string | null } | null; -} - -/** OpenAPI `series-images` item. `work_id`, `note` and `index_numeric` keys are required, even when null. */ -export interface SubmissionImage { - id: number | null; - series_id: number | null; - work_id: string | null; - url: string; - index: string | null; - index_numeric: number | null; - language: string; - type: ImageType; - note: string | null; - content_rating: ContentRating; - is_default: boolean; -} - -export interface ImageSnapshot { - version: number; - images: SubmissionImage[]; -} - -export interface SubmissionChange { - field: string; - type: "added" | "removed" | "changed"; - identity?: string; - label?: string; - details?: Record; -} - -export interface PreviewResult { - has_changes: boolean; - changes: SubmissionChange[]; -} - -export interface SaveResult { - submission_id: number; - status: string; - changes: SubmissionChange[]; -} - -export function toExisting(img: PublicImage): ExistingImage { - return { id: img.id, sha256: img.hashes?.sha256 ?? null, language: img.language, type: img.type }; -} -``` - -- [ ] **Step 4: Write `src/core/mangabaka/client.ts`** - -```ts -import type { Transport } from "../transport"; -import type { - ImageSnapshot, PreviewResult, PublicImage, SaveResult, SeriesLink, SeriesSummary, SubmissionImage, -} from "./types"; - -export const API_BASE = "https://api.mangabaka.org"; -export const SITE_BASE = "https://mangabaka.org"; -export const coverFormUrl = (seriesId: number): string => `${SITE_BASE}/data/submissions/series/${seriesId}/images`; - -export class MangaBakaError extends Error { - constructor(message: string, readonly status: number, readonly body?: unknown) { - super(message); - this.name = "MangaBakaError"; - } -} - -const MAX_PAGES = 50; - -/** Talks to api.mangabaka.org directly (CORS is open); the token never goes anywhere else. */ -export class MangaBakaClient { - constructor(private readonly transport: Transport, private readonly token?: string) {} - - async getSeries(id: number): Promise { - return (await this.get<{ data: SeriesSummary }>(`/v1/series/${id}`)).data; - } - - getLinks(id: number): Promise { - return this.getAllPages(`/v1/series/${id}/links?limit=100`); - } - - /** Paginated, `limit` ≤ 50 (spec §2.3). */ - getImages(id: number): Promise { - return this.getAllPages(`/v1/series/${id}/images?limit=50`); - } - - async getImageSnapshot(id: number): Promise { - const body = await this.get<{ data: { data: { images: SubmissionImage[] }; version: number } }>( - `/v0/my/submissions/series-images/${id}`, true); - return { version: body.data.version, images: body.data.data.images }; - } - - async previewImages(id: number, version: number, images: SubmissionImage[]): Promise { - return (await this.post<{ data: PreviewResult }>(`/v0/my/submissions/series-images/${id}/preview`, - { version, data: { images } })).data; - } - - /** Files a real edit request, always for review. Call only after the user confirmed the preview. */ - async saveImagesForReview(id: number, version: number, images: SubmissionImage[], userNote: string): Promise { - return (await this.post<{ data: SaveResult }>(`/v0/my/submissions/series-images/${id}`, - { version, user_note: userNote, save_mode: "review", data: { images } })).data; - } - - private headers(auth: boolean): Record { - const headers: Record = { accept: "application/json" }; - if (auth) { - if (!this.token) throw new MangaBakaError("This needs a MangaBaka personal access token.", 401); - headers["x-api-key"] = this.token; - } - return headers; - } - - private async get(path: string, auth = false): Promise { - const headers = this.headers(auth); - return this.parse(await this.transport.fetch(`${API_BASE}${path}`, { headers }), path); - } - - private async post(path: string, body: unknown): Promise { - const headers = { ...this.headers(true), "content-type": "application/json" }; - const res = await this.transport.fetch(`${API_BASE}${path}`, { method: "POST", headers, body: JSON.stringify(body) }); - return this.parse(res, path); - } - - private async getAllPages(path: string): Promise { - const out: T[] = []; - let url: string | null = `${API_BASE}${path}`; - for (let page = 0; url && page < MAX_PAGES; page++) { - const res = await this.transport.fetch(url, { headers: this.headers(false) }); - const body = await this.parse<{ data: T[]; pagination?: { next?: string | null } }>(res, path); - out.push(...body.data); - const next = body.pagination?.next ?? null; - url = next && new URL(next).origin === API_BASE ? next : null; - } - return out; - } - - private async parse(res: Response, path: string): Promise { - const text = await res.text(); - let body: unknown; - try { - body = text ? JSON.parse(text) : undefined; - } catch { - body = undefined; - } - if (!res.ok) throw new MangaBakaError(explain(res.status, body, path), res.status, body); - if (body === undefined) throw new MangaBakaError(`MangaBaka sent something that isn't JSON for ${path}.`, res.status); - return body as T; - } -} - -function explain(status: number, body: unknown, path: string): string { - const detail = body && typeof body === "object" && "message" in body ? String((body as { message: unknown }).message) : ""; - switch (status) { - case 401: - return "MangaBaka rejected the token (401). Check your personal access token."; - case 403: - return "MangaBaka refused (403): the submissions API needs the contributor role or 25 approved submissions. Use the copy-and-paste path instead."; - case 404: - return `MangaBaka has no such series (${path}).`; - case 409: - return "The series' covers changed since they were loaded (409)."; - default: - return `MangaBaka answered HTTP ${status}${detail ? `: ${detail}` : ""}.`; - } -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/mangabaka-client.test.ts` → Expected: all PASS. If the image count isn't 31, the series gained images since capture: use `page1.data.length + page2.data.length` from the fixtures instead of 31. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/mangabaka test/mangabaka-client.test.ts -git commit -m "feat(core): MangaBaka client (paginated public reads; snapshot, preview, save for review)" -``` - ---- - -### Task 11: The cover seed - -**Files:** -- Create: `src/core/seed/covers.ts`, `test/seed.test.ts` - -**Interfaces:** -- Consumes: `SubmissionImage` (Task 10), `InspectedCover`, `SourceSeries`, `ImageType`, `ContentRating` (Task 3). -- Produces: `interface CoverChoice { cover: InspectedCover; url: string; type: ImageType; note: string; contentRating: ContentRating }`, `type DefaultChoice = { kind: "new"; sha256: string } | { kind: "existing"; id: number } | null`, `interface CoverSeed { seriesId: number; userNote: string; images: SubmissionImage[]; added: CoverChoice[] }`, `class SeedError`, `buildCoverSeed(input): CoverSeed`, `withHostedUrls(seed, hosted: ReadonlyMap): CoverSeed`, `manualEntryText(choices, language, defaultSha256: string | null): string`. - -- [ ] **Step 1: Write the failing tests** - -`test/seed.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import type { SubmissionImage } from "../src/core/mangabaka/types"; -import type { SourceSeries } from "../src/core/model"; -import { SeedError, buildCoverSeed, manualEntryText, withHostedUrls, type CoverChoice } from "../src/core/seed/covers"; -import { inspected } from "./helpers/covers"; - -const series = { language: "en", canonicalUrl: "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95" } as SourceSeries; -const old: SubmissionImage = { - id: 7193, series_id: 9999, work_id: null, url: "https://example.org/old.jpg", index: "0.1", index_numeric: 0.1, - language: "ko", type: "volume", note: "Season Cover 3", content_rating: "safe", is_default: true, -}; -const choice = (overrides: Partial = {}): CoverChoice => { - const cover = inspected(); - return { cover, url: cover.candidate.url, type: "volume", note: "WEBTOON poster (current)", contentRating: "safe", ...overrides }; -}; -const base = { seriesId: 9999, series, sourceName: "WEBTOON", existing: [old] }; - -describe("buildCoverSeed", () => { - test("keeps every existing image and adds the new ones (images replaces the whole set)", () => { - const a = choice(); - const seed = buildCoverSeed({ ...base, added: [a], makeDefault: null }); - expect(seed.userNote).toBe("Add WEBTOON covers from https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95"); - expect(seed.images).toEqual([old, { - id: null, series_id: 9999, work_id: null, url: a.url, index: null, index_numeric: null, - language: "en", type: "volume", note: "WEBTOON poster (current)", content_rating: "safe", is_default: false, - }]); - }); - - test("a new default takes over: exactly one is_default", () => { - const a = choice(); - const seed = buildCoverSeed({ ...base, added: [a], makeDefault: { kind: "new", sha256: a.cover.sha256 } }); - expect(seed.images.map((i) => i.is_default)).toEqual([false, true]); - }); - - test("an already-present cover can be made the default", () => { - const other = { ...old, id: 8000, is_default: false, url: "https://example.org/other.jpg" }; - const seed = buildCoverSeed({ ...base, existing: [old, other], added: [choice()], makeDefault: { kind: "existing", id: 8000 } }); - expect(seed.images.map((i) => i.is_default)).toEqual([false, true, false]); - }); - - test("a default that was deselected is ignored; existing defaults stay", () => { - const seed = buildCoverSeed({ ...base, added: [choice()], makeDefault: { kind: "new", sha256: "0".repeat(64) } }); - expect(seed.images.map((i) => i.is_default)).toEqual([true, false]); - }); - - test("refuses a set without exactly one default", () => { - expect(() => buildCoverSeed({ ...base, existing: [{ ...old, is_default: false }], added: [choice()], makeDefault: null })) - .toThrow(SeedError); - }); -}); - -test("withHostedUrls swaps in rehosted URLs for new images only", () => { - const a = choice(); - const b = choice(); - const seed = buildCoverSeed({ ...base, added: [a, b], makeDefault: null }); - const hosted = withHostedUrls(seed, new Map([[a.cover.sha256, "https://litter.catbox.moe/abc.jpg"]])); - expect(hosted.images.map((i) => i.url)).toEqual([old.url, "https://litter.catbox.moe/abc.jpg", b.url]); - expect(hosted.added[0]!.url).toBe("https://litter.catbox.moe/abc.jpg"); -}); - -test("manualEntryText: one block per cover, ready to paste into the form", () => { - const a = choice({ url: "https://files.catbox.moe/a.jpg" }); - const b = choice({ url: "https://files.catbox.moe/b.png", type: "banner", note: "WEBTOON banner, seen 2019-03", contentRating: "suggestive" }); - expect(manualEntryText([a, b], "en", a.cover.sha256)).toBe([ - "https://files.catbox.moe/a.jpg", - "type: volume · language: en · rating: safe · main cover", - "note: WEBTOON poster (current)", - "", - "https://files.catbox.moe/b.png", - "type: banner · language: en · rating: suggestive", - "note: WEBTOON banner, seen 2019-03", - ].join("\n")); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/seed.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/core/seed/covers.ts`** - -```ts -import type { SubmissionImage } from "../mangabaka/types"; -import type { ContentRating, ImageType, InspectedCover, SourceSeries } from "../model"; - -/** One new cover as the user left it in the review list. */ -export interface CoverChoice { - cover: InspectedCover; - url: string; - type: ImageType; - note: string; - contentRating: ContentRating; -} - -/** Which image becomes the series' one main cover; null leaves existing defaults alone. */ -export type DefaultChoice = { kind: "new"; sha256: string } | { kind: "existing"; id: number } | null; - -export interface CoverSeed { - seriesId: number; - userNote: string; - images: SubmissionImage[]; - added: CoverChoice[]; -} - -export class SeedError extends Error { - override name = "SeedError"; -} - -/** Spec §4.8. `images` replaces the whole set, so every existing image is carried over. */ -export function buildCoverSeed(input: { - seriesId: number; - series: SourceSeries; - sourceName: string; - existing: readonly SubmissionImage[]; - added: readonly CoverChoice[]; - makeDefault: DefaultChoice; -}): CoverSeed { - const { seriesId, series, existing, added } = input; - const target = effectiveDefault(input.makeDefault, existing, added); - const images: SubmissionImage[] = [ - ...existing.map((e) => ({ ...e, is_default: target ? target.kind === "existing" && target.id === e.id : e.is_default })), - ...added.map((c): SubmissionImage => ({ - id: null, series_id: seriesId, work_id: null, url: c.url, index: null, index_numeric: null, - language: series.language, type: c.type, note: c.note, content_rating: c.contentRating, - is_default: target?.kind === "new" && target.sha256 === c.cover.sha256, - })), - ]; - const defaults = images.filter((i) => i.is_default).length; - if (images.length > 0 && defaults !== 1) { - throw new SeedError(`MangaBaka needs exactly one main cover; this set would have ${defaults}. Tick "Make this the series' main cover".`); - } - return { seriesId, userNote: `Add ${input.sourceName} covers from ${series.canonicalUrl}`, images, added: [...added] }; -} - -function effectiveDefault(d: DefaultChoice, existing: readonly SubmissionImage[], added: readonly CoverChoice[]): DefaultChoice { - if (d?.kind === "new" && added.some((c) => c.cover.sha256 === d.sha256)) return d; - if (d?.kind === "existing" && existing.some((e) => e.id === d.id)) return d; - return null; -} - -export function withHostedUrls(seed: CoverSeed, hosted: ReadonlyMap): CoverSeed { - const added = seed.added.map((c) => ({ ...c, url: hosted.get(c.cover.sha256) ?? c.url })); - const newUrl = new Map(seed.added.map((c, i) => [c.url, added[i]!.url])); - const images = seed.images.map((img) => (img.id === null ? { ...img, url: newUrl.get(img.url) ?? img.url } : img)); - return { ...seed, added, images }; -} - -/** For users without API access: what to paste into MangaBaka's cover form, cover by cover. */ -export function manualEntryText(choices: readonly CoverChoice[], language: string, defaultSha256: string | null): string { - return choices - .map((c) => [ - c.url, - `type: ${c.type} · language: ${language} · rating: ${c.contentRating}${c.cover.sha256 === defaultSha256 ? " · main cover" : ""}`, - `note: ${c.note}`, - ].join("\n")) - .join("\n\n"); -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/seed.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/core/seed/covers.ts test/seed.test.ts -git commit -m "feat(core): build the cover seed (whole set, one default) and manual-entry text" -``` - ---- - -### Task 12: Image hosts and rehosting - -**Files:** -- Create: `src/core/hosting/host.ts`, `direct.ts`, `litterbox.ts`, `catbox.ts`, `server.ts`, `registry.ts`, `rehost.ts`; `test/hosting.test.ts` - -**Interfaces:** -- Consumes: `Transport`, `FetchInit` (Task 3). -- Produces (`host.ts`): `type HostId = "server" | "litterbox" | "catbox" | "direct"`, `type LitterboxTime = "1h" | "12h" | "24h" | "72h"`, `LITTERBOX_TIMES`, `interface HostOptions { litterboxTime?: LitterboxTime; catboxUserhash?: string }`, `interface HostImage { sourceUrl: string; live: boolean; bytes: Uint8Array; sha256: string; contentType: string; ext: string }`, `interface HostContext { transport: Transport; serverBase: string | null; catboxVia: "server" | "direct"; options: HostOptions }`, `interface ImageHost { id: HostId; lifetime(opts: HostOptions): string; upload(image, ctx): Promise<{ url: string }> }`, `class HostError { status?: number }`, `UPLOAD_TIMEOUT_MS`, `fileFor(image): Blob`, `viaServer(image, ctx, body)`. -- Produces: `direct`, `litterbox`, `catbox`, `serverHost` hosts; `uploadToCatbox(transport, file, name, userhash?)` (used by the server, Task 17); `CATBOX_API`, `LITTERBOX_API`; `HOSTS: Record`; `rehostAll(images, host, ctx, cache, onProgress?): Promise` with `RehostResult { hosted: Map; failures: { sha256: string; message: string }[]; allFailed: boolean }`; `cacheKey(hostId, options, sha256): string | null`. - -- [ ] **Step 1: Write the failing tests** - -`test/hosting.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { catbox } from "../src/core/hosting/catbox"; -import { direct } from "../src/core/hosting/direct"; -import { HostError, type HostContext, type HostImage } from "../src/core/hosting/host"; -import { litterbox } from "../src/core/hosting/litterbox"; -import { HOSTS } from "../src/core/hosting/registry"; -import { rehostAll } from "../src/core/hosting/rehost"; -import { serverHost } from "../src/core/hosting/server"; -import { FakeTransport, json, text } from "./helpers/fake-transport"; - -const image = (n = 1, live = true): HostImage => ({ - sourceUrl: `https://swebtoon-phinf.pstatic.net/x/${n}.jpg`, live, bytes: new Uint8Array([n, 2, 3]), - sha256: String(n).repeat(64).slice(0, 64), contentType: "image/jpeg", ext: "jpg", -}); -const ctx = (transport: FakeTransport, over: Partial = {}): HostContext => - ({ transport, serverBase: "https://mbseed.example.org", catboxVia: "direct", options: {}, ...over }); -const LITTERBOX = "https://litterbox.catbox.moe/resources/internals/api.php"; -const CATBOX = "https://catbox.moe/user/api.php"; -const UPLOAD = "https://mbseed.example.org/api/upload"; - -test("lifetimes shown next to each host", () => { - expect(HOSTS.server.lifetime({})).toBe("30 days"); - expect(HOSTS.litterbox.lifetime({})).toBe("72 hours"); - expect(HOSTS.litterbox.lifetime({ litterboxTime: "12h" })).toBe("12 hours"); - expect(HOSTS.catbox.lifetime({})).toBe("permanent"); - expect(HOSTS.direct.lifetime({})).toBe("as long as the source keeps it"); -}); - -test("direct: the live source URL; refuses archive-only images", async () => { - const fake = new FakeTransport(); - expect(await direct.upload(image(), ctx(fake))).toEqual({ url: image().sourceUrl }); - await expect(direct.upload(image(1, false), ctx(fake))).rejects.toBeInstanceOf(HostError); - expect(fake.calls).toHaveLength(0); -}); - -describe("litterbox", () => { - test("multipart upload of the checked bytes, 72h by default", async () => { - const fake = new FakeTransport().on(LITTERBOX, () => text("https://litter.catbox.moe/abc123.jpg\n")); - expect(await litterbox.upload(image(), ctx(fake))).toEqual({ url: "https://litter.catbox.moe/abc123.jpg" }); - const form = fake.calls[0]!.init.body as FormData; - expect(fake.calls[0]!.init.method).toBe("POST"); - expect(form.get("reqtype")).toBe("fileupload"); - expect(form.get("time")).toBe("72h"); - expect((form.get("fileToUpload") as File).size).toBe(3); - }); - test("an error body is a HostError", async () => { - const fake = new FakeTransport().on(LITTERBOX, () => text("File too large", 413)); - await expect(litterbox.upload(image(), ctx(fake))).rejects.toThrow(/File too large/); - }); -}); - -describe("catbox", () => { - test("direct (userscript): multipart with the userhash", async () => { - const fake = new FakeTransport().on(CATBOX, () => text("https://files.catbox.moe/xyz.jpg")); - const url = await catbox.upload(image(), ctx(fake, { options: { catboxUserhash: "abc123" } })); - expect(url).toEqual({ url: "https://files.catbox.moe/xyz.jpg" }); - expect((fake.calls[0]!.init.body as FormData).get("userhash")).toBe("abc123"); - }); - test("via our server (page): sends the source URL, never bytes; checks the sha256", async () => { - const fake = new FakeTransport().on(UPLOAD, () => json({ url: "https://files.catbox.moe/xyz.jpg", sha256: image().sha256 })); - await catbox.upload(image(), ctx(fake, { catboxVia: "server", options: { catboxUserhash: "abc123" } })); - expect(JSON.parse(fake.calls[0]!.init.body as string)).toEqual({ url: image().sourceUrl, host: "catbox", userhash: "abc123" }); - }); -}); - -describe("server storage", () => { - test("posts the source URL with host store", async () => { - const fake = new FakeTransport().on(UPLOAD, () => - json({ url: `https://mbseed.example.org/covers/${image().sha256}.jpg`, sha256: image().sha256 })); - expect((await serverHost.upload(image(), ctx(fake))).url).toContain("/covers/"); - expect(JSON.parse(fake.calls[0]!.init.body as string)).toEqual({ url: image().sourceUrl, host: "store" }); - }); - test("rejects different bytes", async () => { - const fake = new FakeTransport().on(UPLOAD, () => json({ url: "https://x/covers/y.jpg", sha256: "0".repeat(64) })); - await expect(serverHost.upload(image(), ctx(fake))).rejects.toThrow(/different bytes/); - }); - test("storage full is HostError 507", async () => { - const fake = new FakeTransport().on(UPLOAD, () => json({ error: "server storage is full" }, 507)); - await expect(serverHost.upload(image(), ctx(fake))).rejects.toMatchObject({ status: 507 }); - }); - test("no server configured", async () => { - await expect(serverHost.upload(image(), ctx(new FakeTransport(), { serverBase: null }))).rejects.toBeInstanceOf(HostError); - }); -}); - -describe("rehostAll", () => { - test("failures are per image; the cache avoids uploading twice", async () => { - let n = 0; - const fake = new FakeTransport().on(LITTERBOX, () => - ++n === 2 ? text("error", 500) : text(`https://litter.catbox.moe/${n}.jpg`)); - const cache = new Map(); - const first = await rehostAll([image(1), image(2), image(3)], litterbox, ctx(fake), cache); - expect([...first.hosted.keys()]).toEqual([image(1).sha256, image(3).sha256]); - expect(first.failures).toEqual([{ sha256: image(2).sha256, message: expect.stringContaining("litterbox") }]); - expect(first.allFailed).toBe(false); - await rehostAll([image(1)], litterbox, ctx(fake), cache); - expect(fake.calls).toHaveLength(3); - }); - test("a full store stops the remaining uploads", async () => { - const fake = new FakeTransport().on(UPLOAD, () => json({ error: "server storage is full" }, 507)); - const res = await rehostAll([image(1), image(2)], serverHost, ctx(fake), new Map()); - expect(fake.calls).toHaveLength(1); - expect(res.failures).toHaveLength(2); - expect(res.allFailed).toBe(true); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/hosting.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/core/hosting/host.ts`** - -```ts -import type { Transport } from "../transport"; - -export type HostId = "server" | "litterbox" | "catbox" | "direct"; -export type LitterboxTime = "1h" | "12h" | "24h" | "72h"; -export const LITTERBOX_TIMES: readonly LitterboxTime[] = ["1h", "12h", "24h", "72h"]; - -export interface HostOptions { - litterboxTime?: LitterboxTime; - catboxUserhash?: string; -} - -/** One inspected image, ready to rehost. `sourceUrl` is where its bytes came from. */ -export interface HostImage { - sourceUrl: string; - live: boolean; - bytes: Uint8Array; - sha256: string; - contentType: string; - ext: string; -} - -export interface HostContext { - transport: Transport; - /** Our server's origin, or null when there is none (userscript without a server). */ - serverBase: string | null; - /** catbox has no CORS: the page goes through our server, the userscript straight there (spec §4.6). */ - catboxVia: "server" | "direct"; - options: HostOptions; -} - -export interface ImageHost { - readonly id: HostId; - /** Shown next to the host in the UI. */ - lifetime(opts: HostOptions): string; - upload(image: HostImage, ctx: HostContext): Promise<{ url: string }>; -} - -export class HostError extends Error { - constructor(message: string, readonly status?: number) { - super(message); - this.name = "HostError"; - } -} - -export const UPLOAD_TIMEOUT_MS = 120_000; - -export const fileFor = (image: HostImage): Blob => new Blob([image.bytes as Uint8Array], { type: image.contentType }); - -/** `POST /api/upload`: the server fetches `sourceUrl` itself, so it can't be used to host arbitrary bytes. */ -export async function viaServer( - image: HostImage, - ctx: HostContext, - body: { host: "store" | "catbox"; userhash?: string }, -): Promise<{ url: string }> { - if (ctx.serverBase === null) throw new HostError("No mangabaka-seeder server is configured."); - const res = await ctx.transport.fetch(`${ctx.serverBase}/api/upload`, { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ url: image.sourceUrl, ...body }), - timeoutMs: UPLOAD_TIMEOUT_MS, - }); - const reply = (await res.json().catch(() => null)) as { url?: string; sha256?: string; error?: string } | null; - if (!res.ok || !reply?.url) throw new HostError(reply?.error ?? `Upload failed (HTTP ${res.status}).`, res.status); - if (reply.sha256 !== image.sha256) throw new HostError("The server fetched different bytes than the ones checked; not using them."); - return { url: reply.url }; -} -``` - -- [ ] **Step 4: Write the four hosts, the registry and `rehostAll`** - -`src/core/hosting/direct.ts`: -```ts -import { HostError, type ImageHost } from "./host"; - -/** No rehosting: submit the source URL itself. */ -export const direct: ImageHost = { - id: "direct", - lifetime: () => "as long as the source keeps it", - async upload(image) { - if (!image.live) throw new HostError("This image is only on archive.org now; pick a host that rehosts it."); - return { url: image.sourceUrl }; - }, -}; -``` - -`src/core/hosting/litterbox.ts`: -```ts -import { HostError, UPLOAD_TIMEOUT_MS, fileFor, type ImageHost, type LitterboxTime } from "./host"; - -export const LITTERBOX_API = "https://litterbox.catbox.moe/resources/internals/api.php"; -const LABEL: Readonly> = { "1h": "1 hour", "12h": "12 hours", "24h": "24 hours", "72h": "72 hours" }; - -/** Temporary (≤ 72 h); CORS is open, so the page uploads straight from the browser. */ -export const litterbox: ImageHost = { - id: "litterbox", - lifetime: (opts) => LABEL[opts.litterboxTime ?? "72h"], - async upload(image, ctx) { - const form = new FormData(); - form.set("reqtype", "fileupload"); - form.set("time", ctx.options.litterboxTime ?? "72h"); - form.set("fileToUpload", fileFor(image), `${image.sha256}.${image.ext}`); - const res = await ctx.transport.fetch(LITTERBOX_API, { method: "POST", body: form, timeoutMs: UPLOAD_TIMEOUT_MS }); - const body = (await res.text()).trim(); - if (!res.ok || !/^https:\/\/litter\.catbox\.moe\/\S+$/.test(body)) { - throw new HostError(`litterbox: ${body.slice(0, 200) || `HTTP ${res.status}`}`, res.status); - } - return { url: body }; - }, -}; -``` - -`src/core/hosting/catbox.ts`: -```ts -import type { Transport } from "../transport"; -import { HostError, UPLOAD_TIMEOUT_MS, fileFor, viaServer, type ImageHost } from "./host"; - -export const CATBOX_API = "https://catbox.moe/user/api.php"; - -/** Used directly by the userscript and by our server (for the page). */ -export async function uploadToCatbox(transport: Transport, file: Blob, name: string, userhash?: string): Promise<{ url: string }> { - const form = new FormData(); - form.set("reqtype", "fileupload"); - if (userhash) form.set("userhash", userhash); - form.set("fileToUpload", file, name); - const res = await transport.fetch(CATBOX_API, { method: "POST", body: form, timeoutMs: UPLOAD_TIMEOUT_MS }); - const body = (await res.text()).trim(); - if (!res.ok || !/^https:\/\/files\.catbox\.moe\/\S+$/.test(body)) { - throw new HostError(`catbox: ${body.slice(0, 200) || `HTTP ${res.status}`}`, res.status); - } - return { url: body }; -} - -export const catbox: ImageHost = { - id: "catbox", - lifetime: () => "permanent", - upload(image, ctx) { - const userhash = ctx.options.catboxUserhash?.trim() || undefined; - if (ctx.catboxVia === "server") return viaServer(image, ctx, { host: "catbox", ...(userhash ? { userhash } : {}) }); - return uploadToCatbox(ctx.transport, fileFor(image), `${image.sha256}.${image.ext}`, userhash); - }, -}; -``` - -`src/core/hosting/server.ts`: -```ts -import { viaServer, type ImageHost } from "./host"; - -/** Our server's covers folder; swept after 30 days (spec §4.6). */ -export const serverHost: ImageHost = { - id: "server", - lifetime: () => "30 days", - upload: (image, ctx) => viaServer(image, ctx, { host: "store" }), -}; -``` - -`src/core/hosting/registry.ts`: -```ts -import { catbox } from "./catbox"; -import { direct } from "./direct"; -import type { HostId, ImageHost } from "./host"; -import { litterbox } from "./litterbox"; -import { serverHost } from "./server"; - -/** Adding a host: one file in this folder plus one line here. */ -export const HOSTS: Readonly> = { server: serverHost, litterbox, catbox, direct }; -``` - -`src/core/hosting/rehost.ts`: -```ts -import { HostError, type HostContext, type HostId, type HostImage, type HostOptions, type ImageHost } from "./host"; - -export interface RehostResult { - hosted: Map; - failures: { sha256: string; message: string }[]; - allFailed: boolean; -} - -/** litterbox and catbox uploads are remembered by hash; server storage is re-uploaded on purpose (restarts its 30 days). */ -export function cacheKey(host: HostId, options: HostOptions, sha256: string): string | null { - if (host === "litterbox") return `litterbox:${options.litterboxTime ?? "72h"}:${sha256}`; - if (host === "catbox") return `catbox:${sha256}`; - return null; -} - -/** Failures are per image (spec §4.6). A full store (507) stops the rest: they would fail the same way. */ -export async function rehostAll( - images: readonly HostImage[], - host: ImageHost, - ctx: HostContext, - cache: Map, - onProgress?: (done: number, total: number) => void, -): Promise { - const hosted = new Map(); - const failures: { sha256: string; message: string }[] = []; - let stopped: string | null = null; - let done = 0; - for (const image of images) { - if (stopped) { - failures.push({ sha256: image.sha256, message: stopped }); - } else { - const key = cacheKey(host.id, ctx.options, image.sha256); - try { - const url = (key && cache.get(key)) || (await host.upload(image, ctx)).url; - if (key) cache.set(key, url); - hosted.set(image.sha256, url); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - failures.push({ sha256: image.sha256, message }); - if (err instanceof HostError && err.status === 507) stopped = message; - } - } - onProgress?.(++done, images.length); - } - return { hosted, failures, allFailed: images.length > 0 && hosted.size === 0 }; -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/hosting.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/hosting test/hosting.test.ts -git commit -m "feat(core): image hosts (server, litterbox, catbox, direct) and per-image rehosting" -``` - ---- - -### Task 13: The pipeline - -**Files:** -- Create: `src/core/pipeline.ts`, `test/pipeline.test.ts` - -**Interfaces:** -- Consumes: everything in Tasks 3–10. -- Produces: `interface RunInput { mangabakaLink: string; sourceLink: string; history: Granularity | null }` (`null` = history off), `type Stage = "series" | "source" | "history" | "inspect" | "compare"`, `interface RunEvents { onStage?(stage: Stage): void; onProgress?(stage: Stage, done: number, total: number): void; onWarning?(message: string): void; signal?: AbortSignal; stopHistory?: AbortSignal }`, `interface RunResult { seriesId: number; mbSeries: SeriesSummary; source: Source; series: SourceSeries; linkMatches: boolean; fresh: InspectedCover[]; present: { cover: InspectedCover; existing: ExistingImage }[]; suggestedDefault: InspectedCover | null; warnings: string[] }`, `runCovers(input, deps: { transport: Transport; sources?: readonly Source[] }, events?): Promise`. - -- [ ] **Step 1: Write the failing tests** - -`test/pipeline.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { sha256Hex } from "../src/core/images"; -import { LinkError } from "../src/core/links"; -import { runCovers, type Stage } from "../src/core/pipeline"; -import { FakeTransport, bytes, json, text } from "./helpers/fake-transport"; -import { fixtureBytes, fixtureJson, fixtureText } from "./helpers/fixtures"; - -const API = "https://api.mangabaka.org"; -const P = "https://swebtoon-phinf.pstatic.net"; -const POSTER = `${P}/20250204_46/17386458444547o1b2_JPEG/95.jpg`; -const FOREGROUND = `${P}/20230324_64/1679612177711Smf0u_PNG/6TowerOfGod_landingpage_desktop_fg.png`; - -/** Tower of God end to end on captured fixtures. Images other than three captured ones 404 (and are dropped). */ -async function towerOfGod() { - const [p2015, p2023] = [await fixtureText("wayback/tower-of-god-2015.html"), await fixtureText("wayback/tower-of-god-2023.html")]; - return new FakeTransport() - .on(`${API}/v1/series/9999`, async () => json(await fixtureJson("mangabaka/series-9999.json"))) - .on(`${API}/v1/series/9999/links?limit=100`, async () => json(await fixtureJson("mangabaka/links-9999.json"))) - .on(/\/v1\/series\/9999\/images\?.*page=2/, async () => json(await fixtureJson("mangabaka/images-9999-page2.json"))) - .on(`${API}/v1/series/9999/images?limit=50`, async () => json(await fixtureJson("mangabaka/images-9999-page1.json"))) - .on("https://www.webtoons.com/-/-/-/list?title_no=95", async () => text(await fixtureText("webtoons/tower-of-god.html"))) - .on((u) => u.startsWith("https://web.archive.org/cdx/") && u.includes("www.webtoons.com"), - async () => text(await fixtureText("wayback/tower-of-god-cdx-www.txt"))) - .on((u) => u.startsWith("https://web.archive.org/cdx/"), async () => text(await fixtureText("wayback/tower-of-god-cdx-m.txt"))) - .on((u) => /\/web\/20(1\d)\d+id_\//.test(u) && !/\.(jpe?g|png)$/i.test(u), () => text(p2015)) - .on((u) => /\/web\/20(2\d)\d+id_\//.test(u) && !/\.(jpe?g|png)$/i.test(u), () => text(p2023)) - .on(POSTER, async () => bytes(await fixtureBytes("images/poster-95.jpg"))) - .on(FOREGROUND, async () => bytes(await fixtureBytes("images/foreground-95.png"), "image/png")) - .on((u) => u.endsWith("6TowerOfGod_landingpage_mobile.jpg"), async () => bytes(await fixtureBytes("images/landing-2023.jpg"))) - .on(() => true, () => text("not found", 404)); -} - -describe("runCovers on Tower of God", () => { - test("current + yearly history: checked, deduplicated, compared, default suggested", async () => { - const fake = await towerOfGod(); - const stages: Stage[] = []; - const r = await runCovers( - { mangabakaLink: "https://mangabaka.org/manhwa/9999/Tower-of-God", sourceLink: "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", history: 12 }, - { transport: fake }, - { onStage: (s) => stages.push(s) }, - ); - expect(stages).toEqual(["series", "source", "history", "inspect", "compare"]); - expect(r.seriesId).toBe(9999); - expect(r.mbSeries.title).toBe("Tower of God"); - expect(r.series.language).toBe("en"); - expect(r.linkMatches).toBe(true); - - const all = [...r.fresh, ...r.present.map((p) => p.cover)]; - const poster = all.find((c) => c.candidate.url === POSTER)!; - expect(poster).toMatchObject({ live: true, shape: "portrait", type: "volume", candidate: { current: true, role: "poster" } }); - expect(r.suggestedDefault?.sha256).toBe(await sha256Hex(await fixtureBytes("images/poster-95.jpg"))); - expect(all.find((c) => c.candidate.url.endsWith("landingpage_mobile.jpg"))).toMatchObject({ shape: "tall", candidate: { current: false } }); - expect(all.find((c) => c.candidate.url === FOREGROUND)!.candidate.current).toBe(true); - expect(new Set(all.map((c) => c.sha256)).size).toBe(all.length); - expect(r.warnings.some((w) => w.startsWith("Could not download"))).toBe(true); - expect(fake.calls.some((c) => c.init.headers?.["x-api-key"])).toBe(false); - }); - - test("history off: Wayback is never asked", async () => { - const fake = await towerOfGod(); - await runCovers({ mangabakaLink: "9999", sourceLink: "https://www.webtoons.com/-/-/-/list?title_no=95", history: null }, { transport: fake }); - expect(fake.urls().some((u) => u.includes("web.archive.org"))).toBe(false); - }); - - test("'skip the rest of the history' keeps going with what was found", async () => { - const fake = await towerOfGod(); - const stop = new AbortController(); - const r = await runCovers( - { mangabakaLink: "9999", sourceLink: "https://www.webtoons.com/-/-/-/list?title_no=95", history: 1 }, - { transport: fake }, - { stopHistory: stop.signal, onProgress: (stage) => stage === "history" && stop.abort() }, - ); - const snapshots = fake.urls().filter((u) => /id_\/http:\/\/(www|m)\.webtoons\.com/.test(u)); - expect(snapshots).toHaveLength(1); - expect([...r.fresh, ...r.present.map((p) => p.cover)].some((c) => c.candidate.url === POSTER)).toBe(true); - }); - - test("warns when MangaBaka has no link to the Webtoons series", async () => { - const fake = await towerOfGod(); - const r = await runCovers({ mangabakaLink: "9999", sourceLink: "https://www.webtoons.com/-/-/-/list?title_no=95", history: null }, { - transport: { fetch: (u, i) => (u.includes("/links") ? Promise.resolve(json({ status: 200, data: [], pagination: { next: null } })) : fake.fetch(u, i)) }, - }); - expect(r.linkMatches).toBe(false); - expect(r.warnings[0]).toMatch(/has no link to this WEBTOON series/); - }); -}); - -test("unsupported source link is a LinkError naming the supported sources", async () => { - await expect(runCovers({ mangabakaLink: "9999", sourceLink: "https://tapas.io/series/x", history: null }, { transport: new FakeTransport() })) - .rejects.toThrow(/Supported: WEBTOON/); - await expect(runCovers({ mangabakaLink: "nope", sourceLink: "https://tapas.io/series/x", history: null }, { transport: new FakeTransport() })) - .rejects.toBeInstanceOf(LinkError); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/pipeline.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/core/pipeline.ts`** - -```ts -import { dedupeByHash, mergeByUrl, pickDefault, splitExisting, type ExistingImage } from "./covers"; -import { waybackHistory } from "./history/wayback"; -import { inspectAll } from "./images"; -import { LinkError, parseLink, parseMangaBakaSeriesId } from "./links"; -import { MangaBakaClient } from "./mangabaka/client"; -import { toExisting, type SeriesSummary } from "./mangabaka/types"; -import type { CoverCandidate, Granularity, InspectedCover, SourceSeries } from "./model"; -import { SOURCES, findSource } from "./sources/registry"; -import type { Source } from "./sources/source"; -import type { Transport } from "./transport"; - -export interface RunInput { - mangabakaLink: string; - sourceLink: string; - /** Wayback granularity in months, or null for no history. */ - history: Granularity | null; -} - -export type Stage = "series" | "source" | "history" | "inspect" | "compare"; - -export interface RunEvents { - onStage?(stage: Stage): void; - onProgress?(stage: Stage, done: number, total: number): void; - onWarning?(message: string): void; - /** Cancels the whole run. */ - signal?: AbortSignal; - /** Stops the history early and carries on with what it found (spec §4.3). */ - stopHistory?: AbortSignal; -} - -export interface RunResult { - seriesId: number; - mbSeries: SeriesSummary; - source: Source; - series: SourceSeries; - linkMatches: boolean; - fresh: InspectedCover[]; - present: { cover: InspectedCover; existing: ExistingImage }[]; - suggestedDefault: InspectedCover | null; - warnings: string[]; -} - -/** Spec §3. The only place the stages are composed. */ -export async function runCovers( - input: RunInput, - deps: { transport: Transport; sources?: readonly Source[] }, - events: RunEvents = {}, -): Promise { - const warnings: string[] = []; - const warn = (message: string) => { - warnings.push(message); - events.onWarning?.(message); - }; - const sources = deps.sources ?? SOURCES; - const seriesId = parseMangaBakaSeriesId(input.mangabakaLink); - const sourceUrl = parseLink(input.sourceLink); - const source = findSource(sourceUrl, sources); - if (!source) { - throw new LinkError(`No supported source for ${sourceUrl.hostname}. Supported: ${sources.map((s) => s.name).join(", ")}.`); - } - const { transport } = deps; - const ctx = { transport, signal: events.signal }; - - events.onStage?.("series"); - const mb = new MangaBakaClient(transport); - const [mbSeries, links, images] = await Promise.all([mb.getSeries(seriesId), mb.getLinks(seriesId), mb.getImages(seriesId)]); - - events.onStage?.("source"); - const series = await source.resolve(sourceUrl, ctx); - const linkMatches = links.some((l) => source.isSameSeries(l.url, series)); - if (!linkMatches) { - warn(`MangaBaka series ${seriesId} has no link to this ${source.name} series (${series.canonicalUrl}). Check that the two links belong together.`); - } - if (series.language === "unknown") warn(`Couldn't tell the language of ${series.canonicalUrl}; covers will be marked "unknown".`); - const current = await source.currentCovers(series, ctx); - - let historical: CoverCandidate[] = []; - if (input.history !== null) { - events.onStage?.("history"); - const signals = [events.signal, events.stopHistory].filter((s): s is AbortSignal => !!s); - historical = await waybackHistory(source, series, { - every: input.history, - transport, - warn, - signal: signals.length ? AbortSignal.any(signals) : undefined, - onProgress: (done, total) => events.onProgress?.("history", done, total), - }); - if (events.signal?.aborted) throw events.signal.reason; - if (source.historicalCovers) historical.push(...(await source.historicalCovers(series, ctx))); - } - - events.onStage?.("inspect"); - const inspected = await inspectAll(mergeByUrl([...current, ...historical]), transport, { - warn, - signal: events.signal, - onProgress: (done, total) => events.onProgress?.("inspect", done, total), - }); - - events.onStage?.("compare"); - const { fresh, present } = splitExisting(dedupeByHash(inspected), images.map(toExisting)); - const suggestedDefault = pickDefault([...fresh, ...present.map((p) => p.cover)]); - return { seriesId, mbSeries, source, series, linkMatches, fresh, present, suggestedDefault, warnings }; -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/pipeline.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/core/pipeline.ts test/pipeline.test.ts -git commit -m "feat(core): compose the pipeline (links → series → current + history → inspect → compare)" -``` - ---- - -### Task 14: The shared review session - -**Files:** -- Create: `src/core/session.ts`, `test/session.test.ts` - -**Interfaces:** -- Consumes: `runCovers`, `RunResult`, `RunInput`, `RunEvents` (Task 13); `HOSTS`, `rehostAll`, `HostError`, `HostId`, `HostOptions` (Task 12); `buildCoverSeed`, `manualEntryText`, `SeedError` (Task 11); `MangaBakaClient`, `MangaBakaError` (Task 10); `coverNote` (Task 8). -- Produces: `interface KeyValueStore { get(k): string | null; set(k, v): void; remove(k): void }`, `memoryStore()`, `TOKEN_KEY`, `HOSTED_KEY`, `loadToken(store)`, `saveToken(store, token | null)`, `interface ReviewCard { cover; selected; type; note; contentRating; hostedUrl: string | null; error: string | null }`, `interface SessionDeps { transport; serverBase: string | null; catboxVia: "server" | "direct"; store: KeyValueStore; persistHostedUrls: boolean; sources? }`, `interface PreparedSubmission { version: number; existing: SubmissionImage[]; seed: CoverSeed; preview: PreviewResult }`, `type SubmitOutcome = { status: "saved"; result: SaveResult } | { status: "changed"; prepared: PreparedSubmission; reason: string }`, `class Session` with `result`, `cards`, `makeDefault`, `run(input, events?, rating?)`, `load(result, rating?)`, `update(sha256, patch)`, `setRatingForAll(rating)`, `selected()`, `defaultChoice()`, `choices()`, `manualText()`, `rehost(hostId, options, onProgress?)`, `preview(token)`, `submit(token, prepared, hostId, options)`. - -- [ ] **Step 1: Write the failing tests** - -`test/session.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { HostError } from "../src/core/hosting/host"; -import { MangaBakaError } from "../src/core/mangabaka/client"; -import type { SubmissionImage } from "../src/core/mangabaka/types"; -import type { SourceSeries } from "../src/core/model"; -import type { RunResult } from "../src/core/pipeline"; -import { HOSTED_KEY, Session, memoryStore, type SessionDeps } from "../src/core/session"; -import { webtoons } from "../src/core/sources/webtoons"; -import { inspected } from "./helpers/covers"; -import { FakeTransport, json, text } from "./helpers/fake-transport"; - -const series: SourceSeries = { - source: "webtoons", externalId: "95", title: "Tower of God", language: "en", - canonicalUrl: "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95", archiveUrls: [], mature: false, flags: {}, -}; -function result(): RunResult { - const poster = inspected({ candidate: { role: "poster", current: true } }); - const landing = inspected({ shape: "tall", type: "other", candidate: { role: "poster", current: false, seenAt: "20230401000000" } }); - const already = inspected({ candidate: { role: "square" }, shape: "square", type: "other" }); - return { - seriesId: 9999, mbSeries: { id: 9999, title: "Tower of God", type: "manhwa", content_rating: "safe" }, source: webtoons, series, - linkMatches: true, fresh: [poster, landing], present: [{ cover: already, existing: { id: 8000, sha256: already.sha256, language: "en", type: "other" } }], - suggestedDefault: poster, warnings: [], - }; -} -const deps = (transport: FakeTransport, over: Partial = {}): SessionDeps => - ({ transport, serverBase: null, catboxVia: "direct", store: memoryStore(), persistHostedUrls: false, ...over }); - -describe("review cards", () => { - test("landing-page art starts unselected; notes and rating filled in", () => { - const s = new Session(deps(new FakeTransport())); - s.load(result(), "suggestive"); - expect(s.cards.map((c) => c.selected)).toEqual([true, false]); - expect(s.cards.map((c) => c.note)).toEqual(["WEBTOON poster (current)", "WEBTOON poster (landing-page art), seen 2023-04"]); - expect(s.cards.every((c) => c.contentRating === "suggestive")).toBe(true); - }); - - test("main cover: the suggested poster, unless it's unticked or deselected", () => { - const s = new Session(deps(new FakeTransport())); - const r = result(); - s.load(r); - expect(s.defaultChoice()).toEqual({ kind: "new", sha256: r.suggestedDefault!.sha256 }); - s.update(r.suggestedDefault!.sha256, { selected: false }); - expect(s.defaultChoice()).toBeNull(); - s.update(r.suggestedDefault!.sha256, { selected: true }); - s.makeDefault = false; - expect(s.defaultChoice()).toBeNull(); - }); - - test("main cover already on MangaBaka: that existing image is marked", () => { - const s = new Session(deps(new FakeTransport())); - const r = result(); - s.load({ ...r, suggestedDefault: r.present[0]!.cover }); - expect(s.defaultChoice()).toEqual({ kind: "existing", id: 8000 }); - }); -}); - -describe("rehost and manual entry", () => { - test("rehosted URLs are used in the copy-and-paste text and remembered (catbox only)", async () => { - const fake = new FakeTransport().on("https://catbox.moe/user/api.php", () => text("https://files.catbox.moe/p.jpg")); - const store = memoryStore(); - const r = result(); - const s = new Session(deps(fake, { store, persistHostedUrls: true })); - s.load(r); - const res = await s.rehost("catbox", {}); - expect(res.failures).toEqual([]); - expect(s.cards[0]!.hostedUrl).toBe("https://files.catbox.moe/p.jpg"); - expect(s.manualText()).toStartWith("https://files.catbox.moe/p.jpg\ntype: volume · language: en · rating: safe · main cover"); - expect(JSON.parse(store.get(HOSTED_KEY)!)).toEqual({ [`catbox:${s.cards[0]!.cover.sha256}`]: "https://files.catbox.moe/p.jpg" }); - - const again = new Session(deps(fake, { store, persistHostedUrls: true })); - again.load(r); - await again.rehost("catbox", {}); - expect(fake.calls).toHaveLength(1); - }); -}); - -// Snapshot/preview/save shapes from the OpenAPI spec (not capturable without a token). -const old: SubmissionImage = { - id: 7193, series_id: 9999, work_id: null, url: "https://example.org/old.jpg", index: "0.1", index_numeric: 0.1, - language: "ko", type: "volume", note: "Season Cover 3", content_rating: "safe", is_default: true, -}; -const SUB = "https://api.mangabaka.org/v0/my/submissions/series-images/9999"; -const LITTERBOX = "https://litterbox.catbox.moe/resources/internals/api.php"; -function mangabaka(saveStatuses: number[]) { - return new FakeTransport() - .on((u, i) => u === SUB && (i.method ?? "GET") === "GET", () => json({ status: 200, data: { data: { images: [old] }, version: 42 } })) - .on(`${SUB}/preview`, () => json({ status: 200, data: { has_changes: true, changes: [] } })) - .on((u, i) => u === SUB && i.method === "POST", () => { - const status = saveStatuses.shift() ?? 200; - return status === 200 - ? json({ status: 200, data: { submission_id: 77, status: "pending", changes: [] } }) - : json({ status, message: "x" }, status); - }); -} - -describe("submit with a token", () => { - test("preview, then save for review with the whole set and one main cover", async () => { - const fake = mangabaka([]); - const s = new Session(deps(fake)); - s.load(result()); - const prepared = await s.preview("mb-token"); - expect(prepared.seed.images.map((i) => [i.id, i.is_default])).toEqual([[7193, false], [null, true]]); - const outcome = await s.submit("mb-token", prepared, "direct", {}); - expect(outcome).toMatchObject({ status: "saved", result: { submission_id: 77 } }); - const save = JSON.parse(fake.calls.at(-1)!.init.body as string); - expect(save.save_mode).toBe("review"); - expect(save.version).toBe(42); - expect(save.data.images).toHaveLength(2); - }); - - test("409 once: refetch and ask again; 409 twice: give up", async () => { - const fake = mangabaka([409, 409]); - const s = new Session(deps(fake)); - s.load(result()); - const first = await s.submit("mb-token", await s.preview("mb-token"), "direct", {}); - expect(first.status).toBe("changed"); - if (first.status !== "changed") throw new Error("unreachable"); - await expect(s.submit("mb-token", first.prepared, "direct", {})).rejects.toBeInstanceOf(MangaBakaError); - }); - - test("a failed upload is left out and the user gets a fresh preview; nothing is saved yet", async () => { - let uploads = 0; - const fake = mangabaka([]).on(LITTERBOX, () => (++uploads === 1 ? text("https://litter.catbox.moe/ok.jpg") : text("error", 500))); - const s = new Session(deps(fake)); - const r = result(); - s.load(r); - s.update(r.fresh[1]!.sha256, { selected: true }); - const outcome = await s.submit("mb-token", await s.preview("mb-token"), "litterbox", {}); - expect(outcome.status).toBe("changed"); - expect(s.cards[1]!.selected).toBe(false); - expect(fake.calls.some((c) => c.url === SUB && c.init.method === "POST")).toBe(false); - }); - - test("every upload failing aborts the action", async () => { - const fake = mangabaka([]).on(LITTERBOX, () => text("error", 500)); - const s = new Session(deps(fake)); - s.load(result()); - await expect(s.submit("mb-token", await s.preview("mb-token"), "litterbox", {})).rejects.toBeInstanceOf(HostError); - expect(fake.calls.some((c) => c.url === SUB && c.init.method === "POST")).toBe(false); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/session.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/core/session.ts`** - -```ts -import { coverNote } from "./covers"; -import { HostError, type HostId, type HostImage, type HostOptions } from "./hosting/host"; -import { HOSTS } from "./hosting/registry"; -import { rehostAll, type RehostResult } from "./hosting/rehost"; -import { MangaBakaClient, MangaBakaError } from "./mangabaka/client"; -import type { PreviewResult, SaveResult, SubmissionImage } from "./mangabaka/types"; -import type { ContentRating, ImageType, InspectedCover } from "./model"; -import { runCovers, type RunEvents, type RunInput, type RunResult } from "./pipeline"; -import { SeedError, buildCoverSeed, manualEntryText, type CoverChoice, type CoverSeed, type DefaultChoice } from "./seed/covers"; -import type { Source } from "./sources/source"; -import type { Transport } from "./transport"; - -/** Browser storage on the page, GM storage in the userscript, a Map in tests. */ -export interface KeyValueStore { - get(key: string): string | null; - set(key: string, value: string): void; - remove(key: string): void; -} - -export function memoryStore(): KeyValueStore { - const m = new Map(); - return { get: (k) => m.get(k) ?? null, set: (k, v) => void m.set(k, v), remove: (k) => void m.delete(k) }; -} - -export const TOKEN_KEY = "mbseed.token"; -export const HOSTED_KEY = "mbseed.hosted"; -const HOSTED_LIMIT = 500; - -export const loadToken = (store: KeyValueStore): string | null => store.get(TOKEN_KEY); -export function saveToken(store: KeyValueStore, token: string | null): void { - if (token) store.set(TOKEN_KEY, token); - else store.remove(TOKEN_KEY); -} - -export interface ReviewCard { - cover: InspectedCover; - selected: boolean; - type: ImageType; - note: string; - contentRating: ContentRating; - hostedUrl: string | null; - error: string | null; -} - -export interface SessionDeps { - transport: Transport; - serverBase: string | null; - catboxVia: "server" | "direct"; - store: KeyValueStore; - /** The userscript remembers catbox uploads across runs; the page only within a session (spec §4.6). */ - persistHostedUrls: boolean; - sources?: readonly Source[]; -} - -export interface PreparedSubmission { - version: number; - existing: SubmissionImage[]; - seed: CoverSeed; - preview: PreviewResult; -} - -export type SubmitOutcome = - | { status: "saved"; result: SaveResult } - | { status: "changed"; prepared: PreparedSubmission; reason: string }; - -const hostImage = (c: InspectedCover): HostImage => ({ - sourceUrl: c.fetchedFrom, live: c.live, bytes: c.bytes, sha256: c.sha256, contentType: c.contentType, ext: c.ext, -}); - -/** The review state behind both front ends: cards, main cover, rehosting, manual text, token flow. */ -export class Session { - result: RunResult | null = null; - cards: ReviewCard[] = []; - /** "Make this the series' main cover" (spec §4.5). */ - makeDefault = true; - private conflicts = 0; - private readonly hostedCache: Map; - - constructor(private readonly deps: SessionDeps) { - this.hostedCache = deps.persistHostedUrls ? loadHosted(deps.store) : new Map(); - } - - async run(input: RunInput, events: RunEvents = {}, rating: ContentRating = "safe"): Promise { - const result = await runCovers(input, { transport: this.deps.transport, sources: this.deps.sources }, events); - this.load(result, rating); - return result; - } - - load(result: RunResult, rating: ContentRating = "safe"): void { - this.result = result; - this.conflicts = 0; - this.makeDefault = result.suggestedDefault !== null; - this.cards = result.fresh.map((cover) => ({ - cover, - selected: cover.shape !== "tall", - type: cover.type, - note: coverNote(cover, result.source.name), - contentRating: rating, - hostedUrl: null, - error: null, - })); - } - - update(sha256: string, patch: Partial>): void { - const card = this.cards.find((c) => c.cover.sha256 === sha256); - if (!card) throw new Error(`No cover ${sha256}`); - Object.assign(card, patch); - } - - setRatingForAll(rating: ContentRating): void { - for (const c of this.cards) c.contentRating = rating; - } - - selected(): ReviewCard[] { - return this.cards.filter((c) => c.selected); - } - - defaultChoice(): DefaultChoice { - const suggested = this.result?.suggestedDefault; - if (!this.makeDefault || !suggested) return null; - const present = this.result!.present.find((p) => p.cover.sha256 === suggested.sha256); - if (present) return { kind: "existing", id: present.existing.id }; - return this.selected().some((c) => c.cover.sha256 === suggested.sha256) ? { kind: "new", sha256: suggested.sha256 } : null; - } - - choices(): CoverChoice[] { - return this.selected().map((c) => ({ - cover: c.cover, url: c.hostedUrl ?? c.cover.fetchedFrom, type: c.type, note: c.note, contentRating: c.contentRating, - })); - } - - manualText(): string { - const d = this.defaultChoice(); - return manualEntryText(this.choices(), this.requireResult().series.language, d?.kind === "new" ? d.sha256 : null); - } - - /** Explicit user action only (spec §4.6). */ - async rehost(hostId: HostId, options: HostOptions, onProgress?: (done: number, total: number) => void): Promise { - const cards = this.selected(); - const res = await rehostAll( - cards.map((c) => hostImage(c.cover)), - HOSTS[hostId], - { transport: this.deps.transport, serverBase: this.deps.serverBase, catboxVia: this.deps.catboxVia, options }, - this.hostedCache, - onProgress, - ); - for (const c of cards) { - c.hostedUrl = res.hosted.get(c.cover.sha256) ?? null; - c.error = res.failures.find((f) => f.sha256 === c.cover.sha256)?.message ?? null; - } - if (this.deps.persistHostedUrls) saveHosted(this.deps.store, this.hostedCache); - return res; - } - - async preview(token: string): Promise { - const r = this.requireResult(); - if (!this.selected().length) throw new SeedError("Select at least one cover first."); - const mb = new MangaBakaClient(this.deps.transport, token); - const snapshot = await mb.getImageSnapshot(r.seriesId); - const seed = this.seed(snapshot.images); - const preview = await mb.previewImages(r.seriesId, snapshot.version, seed.images); - return { version: snapshot.version, existing: snapshot.images, seed, preview }; - } - - /** After the user confirmed `prepared`: rehost, then file the edit request for review (spec §4.9). */ - async submit(token: string, prepared: PreparedSubmission, hostId: HostId, options: HostOptions): Promise { - const r = this.requireResult(); - const uploads = await this.rehost(hostId, options); - if (uploads.allFailed) throw new HostError(`Every upload failed: ${uploads.failures[0]?.message ?? "unknown error"}`); - if (uploads.failures.length) { - for (const f of uploads.failures) this.update(f.sha256, { selected: false }); - return { - status: "changed", - prepared: await this.preview(token), - reason: `${uploads.failures.length} upload(s) failed and were left out. Check the new preview.`, - }; - } - const seed = this.seed(prepared.existing); - const mb = new MangaBakaClient(this.deps.transport, token); - try { - return { status: "saved", result: await mb.saveImagesForReview(r.seriesId, prepared.version, seed.images, seed.userNote) }; - } catch (err) { - if (err instanceof MangaBakaError && err.status === 409 && this.conflicts++ === 0) { - return { status: "changed", prepared: await this.preview(token), reason: "The series' covers changed meanwhile. Check the new preview." }; - } - throw err; - } - } - - private seed(existing: SubmissionImage[]): CoverSeed { - const r = this.requireResult(); - return buildCoverSeed({ - seriesId: r.seriesId, series: r.series, sourceName: r.source.name, existing, added: this.choices(), makeDefault: this.defaultChoice(), - }); - } - - private requireResult(): RunResult { - if (!this.result) throw new Error("Run the covers search first."); - return this.result; - } -} - -/** Only catbox links are kept across runs: litterbox links die within 72 hours. */ -function loadHosted(store: KeyValueStore): Map { - try { - const raw = JSON.parse(store.get(HOSTED_KEY) ?? "{}") as Record; - return new Map(Object.entries(raw).filter((e): e is [string, string] => typeof e[1] === "string")); - } catch { - return new Map(); - } -} - -function saveHosted(store: KeyValueStore, cache: Map): void { - const kept = [...cache].filter(([k]) => k.startsWith("catbox:")).slice(-HOSTED_LIMIT); - store.set(HOSTED_KEY, JSON.stringify(Object.fromEntries(kept))); -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/session.test.ts` → Expected: all PASS. -Run: `bun test` → Expected: the whole suite passes. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Update `docs/ARCHITECTURE.md`** (docs duty: the core now exists) - -Replace the Overview's "No code exists yet." paragraph with: "The core (`src/core/`) is built and tested: `pipeline.ts` composes the stages of spec §3, `session.ts` holds the review state both front ends share. The server, page and userscript follow." In the Map table, add a row `| src/core/session.ts | Review state shared by the page and the userscript: cards, main cover, rehosting, manual text, token flow (preview → confirm → save for review). |`. - -- [ ] **Step 6: Commit** - -```bash -git add src/core/session.ts test/session.test.ts docs/ARCHITECTURE.md -git commit -m "feat(core): shared review session (cards, main cover, rehost, manual text, token flow)" -``` - ---- - -## Part B — Server - -### Task 15: Cover storage (memory and folder) - -**Files:** -- Create: `src/server/storage.ts`, `src/server/storage-dir.ts`, `test/storage.test.ts` - -**Interfaces:** -- Produces (`storage.ts`, web-standard): `COVER_NAME = /^[0-9a-f]{64}\.(jpg|png|gif|webp)$/`, `CONTENT_TYPES: Record`, `FULL_MARGIN_BYTES = 1024 * 1024`, `class StorageFullError`, `interface Storage { put(name, bytes): Promise; get(name): Promise; isFull(): boolean }`, `class MemoryStorage(maxBytes)` (+ `usedBytes()`). -- Produces (`storage-dir.ts`, Bun only): `RETENTION_MS` (30 days), `class DirStorage` with `static open(dir, maxBytes, now?)`, `put`, `get`, `isFull`, `usedBytes()`, `sweep(maxAgeMs?): Promise`. - -- [ ] **Step 1: Write the failing tests** - -`test/storage.test.ts`: -```ts -import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { mkdtemp, readdir, rm, utimes, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { MemoryStorage, StorageFullError, type Storage } from "../src/server/storage"; -import { DirStorage, RETENTION_MS } from "../src/server/storage-dir"; - -const name = (c: string, ext = "jpg") => `${c.repeat(64)}.${ext}`; -const bytes = (n: number) => new Uint8Array(n).fill(7); - -let dir: string; -beforeEach(async () => { dir = await mkdtemp(join(tmpdir(), "mbseed-test-")); }); -afterEach(async () => { await rm(dir, { recursive: true, force: true }); }); - -const kinds: [string, (max: number) => Promise][] = [ - ["memory", async (max) => new MemoryStorage(max)], - ["folder", (max) => DirStorage.open(dir, max)], -]; - -describe.each(kinds)("%s storage", (_kind, make) => { - test("put then get", async () => { - const s = await make(10_000); - await s.put(name("a"), bytes(100)); - expect((await s.get(name("a")))!.size).toBe(100); - expect(await s.get(name("b"))).toBeNull(); - }); - test("refuses past the cap; re-uploading the same file doesn't count twice", async () => { - const s = await make(1000); - await s.put(name("a"), bytes(600)); - await s.put(name("a"), bytes(600)); - expect(s.usedBytes()).toBe(600); - await expect(s.put(name("b"), bytes(600))).rejects.toBeInstanceOf(StorageFullError); - expect(s.usedBytes()).toBe(600); - }); - test("concurrent uploads can't race past the cap", async () => { - const s = await make(1000); - const results = await Promise.allSettled([s.put(name("a"), bytes(600)), s.put(name("b"), bytes(600))]); - expect(results.map((r) => r.status).sort()).toEqual(["fulfilled", "rejected"]); - expect(s.usedBytes()).toBe(600); - }); - test("rejects names that aren't .", async () => { - const s = await make(1000); - for (const bad of ["../x.jpg", `${"A".repeat(64)}.jpg`, `${"a".repeat(64)}.svg`, "x"]) { - await expect(s.put(bad, bytes(1))).rejects.toThrow(); - } - }); - test("isFull when less than 1 MiB is left", async () => { - const s = await make(1024 * 1024 + 10); - expect(s.isFull()).toBe(false); - await s.put(name("a"), bytes(20)); - expect(s.isFull()).toBe(true); - }); -}); - -describe("folder storage", () => { - test("measures existing files at startup and removes stale temp files", async () => { - await writeFile(join(dir, name("a")), bytes(300)); - await writeFile(join(dir, `${name("b")}.123.tmp`), bytes(50)); - await writeFile(join(dir, "README"), "not a cover"); - const s = await DirStorage.open(dir, 10_000); - expect(s.usedBytes()).toBe(300); - expect((await readdir(dir)).sort()).toEqual([name("a"), "README"].sort()); - }); - test("sweep deletes files older than 30 days and frees their space", async () => { - const now = Date.now(); - const s = await DirStorage.open(dir, 10_000, () => now); - await s.put(name("a"), bytes(100)); - await s.put(name("b"), bytes(200)); - const old = new Date(now - RETENTION_MS - 60_000); - await utimes(join(dir, name("a")), old, old); - expect(await s.sweep()).toBe(1); - expect(await s.get(name("a"))).toBeNull(); - expect(s.usedBytes()).toBe(200); - }); - test("a re-upload restarts the 30 days", async () => { - const now = Date.now(); - const s = await DirStorage.open(dir, 10_000, () => now); - await s.put(name("a"), bytes(100)); - const old = new Date(now - RETENTION_MS - 60_000); - await utimes(join(dir, name("a")), old, old); - await s.put(name("a"), bytes(100)); - expect(await s.sweep()).toBe(0); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/storage.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/server/storage.ts`** - -```ts -// Web-standard: the Storage interface and an in-memory implementation for tests. - -/** Covers are stored as .; nothing else is ever read or written (spec §4.6). */ -export const COVER_NAME = /^[0-9a-f]{64}\.(jpg|png|gif|webp)$/; -export const CONTENT_TYPES: Readonly> = { - jpg: "image/jpeg", png: "image/png", gif: "image/gif", webp: "image/webp", -}; -/** `/api/config` reports full when less than this is left; a typical cover is ≤ 650 KB. */ -export const FULL_MARGIN_BYTES = 1024 * 1024; - -export class StorageFullError extends Error { - override name = "StorageFullError"; -} - -export interface Storage { - /** Throws StorageFullError when the write would pass the cap; never evicts. */ - put(name: string, bytes: Uint8Array): Promise; - get(name: string): Promise; - isFull(): boolean; -} - -export function assertCoverName(name: string): void { - if (!COVER_NAME.test(name)) throw new Error(`Not a cover name: ${name}`); -} - -export class MemoryStorage implements Storage { - private readonly files = new Map(); - private used = 0; - - constructor(private readonly maxBytes: number) {} - - async put(name: string, bytes: Uint8Array): Promise { - assertCoverName(name); - const previous = this.files.get(name)?.byteLength ?? 0; - if (this.used - previous + bytes.byteLength > this.maxBytes) { - throw new StorageFullError(`Storage is full (${this.used} of ${this.maxBytes} bytes used).`); - } - this.files.set(name, bytes); - this.used += bytes.byteLength - previous; - } - - async get(name: string): Promise { - const b = this.files.get(name); - return b ? new Blob([b as Uint8Array]) : null; - } - - isFull(): boolean { - return this.maxBytes - this.used < FULL_MARGIN_BYTES; - } - - usedBytes(): number { - return this.used; - } -} -``` - -- [ ] **Step 4: Write `src/server/storage-dir.ts`** - -```ts -// The covers folder on our server (spec §4.6). Bun/node APIs: used only by bun.ts. -import { mkdir, readdir, rename, stat, unlink } from "node:fs/promises"; -import { join } from "node:path"; -import { COVER_NAME, FULL_MARGIN_BYTES, StorageFullError, assertCoverName, type Storage } from "./storage"; - -export const RETENTION_MS = 30 * 24 * 60 * 60 * 1000; - -export class DirStorage implements Storage { - private used = 0; - private readonly sizes = new Map(); - /** Writes and sweeps run one at a time, so two uploads can't race past the cap. */ - private queue: Promise = Promise.resolve(); - - private constructor(readonly dir: string, readonly maxBytes: number, private readonly now: () => number) {} - - static async open(dir: string, maxBytes: number, now: () => number = Date.now): Promise { - await mkdir(dir, { recursive: true }); - const storage = new DirStorage(dir, maxBytes, now); - for (const entry of await readdir(dir)) { - const path = join(dir, entry); - if (entry.endsWith(".tmp")) { - await unlink(path).catch(() => {}); - continue; - } - if (!COVER_NAME.test(entry)) continue; - const { size } = await stat(path); - storage.sizes.set(entry, size); - storage.used += size; - } - return storage; - } - - put(name: string, bytes: Uint8Array): Promise { - return this.serial(async () => { - assertCoverName(name); - const previous = this.sizes.get(name) ?? 0; - if (this.used - previous + bytes.byteLength > this.maxBytes) { - throw new StorageFullError(`The covers folder is full (${this.used} of ${this.maxBytes} bytes used).`); - } - // Temp file + rename: a reader never sees half a file. Rewriting also restarts the 30 days. - const tmp = join(this.dir, `${name}.${crypto.randomUUID()}.tmp`); - await Bun.write(tmp, bytes); - await rename(tmp, join(this.dir, name)); - this.sizes.set(name, bytes.byteLength); - this.used += bytes.byteLength - previous; - }); - } - - async get(name: string): Promise { - if (!COVER_NAME.test(name) || !this.sizes.has(name)) return null; - const file = Bun.file(join(this.dir, name)); - return (await file.exists()) ? file : null; - } - - isFull(): boolean { - return this.maxBytes - this.used < FULL_MARGIN_BYTES; - } - - usedBytes(): number { - return this.used; - } - - /** Deletes covers last written more than `maxAgeMs` ago. Returns how many. */ - sweep(maxAgeMs = RETENTION_MS): Promise { - return this.serial(async () => { - const cutoff = this.now() - maxAgeMs; - let removed = 0; - for (const name of [...this.sizes.keys()]) { - const path = join(this.dir, name); - let mtime: number; - try { - mtime = (await stat(path)).mtimeMs; - } catch { - this.forget(name); - continue; - } - if (mtime < cutoff) { - await unlink(path); - this.forget(name); - removed++; - } - } - return removed; - }); - } - - private forget(name: string): void { - this.used -= this.sizes.get(name) ?? 0; - this.sizes.delete(name); - } - - private serial(fn: () => Promise): Promise { - const run = this.queue.then(fn, fn); - this.queue = run.catch(() => {}); - return run; - } -} -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/storage.test.ts` → Expected: all PASS. -Run: `bun test test/boundaries.test.ts` → Expected: PASS (`storage.ts` has no Bun/node APIs). -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 6: Commit** - -```bash -git add src/server/storage.ts src/server/storage-dir.ts test/storage.test.ts -git commit -m "feat(server): size-capped covers folder with a 30-day sweep; in-memory storage for tests" -``` - ---- - -### Task 16: Rate limiter, configuration, client IP, asset paths - -**Files:** -- Create: `src/server/rate-limit.ts`, `src/server/config.ts`, `src/server/assets-path.ts`, `test/server-config.test.ts` - -**Interfaces:** -- Produces: `class RateLimiter(limit, windowMs, now?)` with `take(key): boolean`; `interface ServerConfig { port; hostname; publicUrl: string | null; storageDir: string | null; storageMaxBytes; trustProxy; assetsDir }`, `class ConfigError`, `DEFAULT_STORAGE_MAX_BYTES`, `readConfig(env): ServerConfig`, `clientIp(req, trustProxy, peer?): string`; `resolveAssetPath(pathname): string | null` (relative path under the assets root, or null). - -- [ ] **Step 1: Write the failing tests** - -`test/server-config.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { resolveAssetPath } from "../src/server/assets-path"; -import { ConfigError, clientIp, readConfig } from "../src/server/config"; -import { RateLimiter } from "../src/server/rate-limit"; - -test("RateLimiter: N per window per key", () => { - let now = 0; - const limiter = new RateLimiter(2, 60_000, () => now); - expect([limiter.take("a"), limiter.take("a"), limiter.take("a"), limiter.take("b")]).toEqual([true, true, false, true]); - now = 60_000; - expect(limiter.take("a")).toBe(true); -}); - -describe("readConfig", () => { - test("defaults", () => { - expect(readConfig({})).toEqual({ - port: 8787, hostname: "127.0.0.1", publicUrl: null, storageDir: null, - storageMaxBytes: 1073741824, trustProxy: false, assetsDir: "dist/web", - }); - }); - test("everything set", () => { - expect(readConfig({ - MBSEED_PORT: "9000", MBSEED_HOST: "0.0.0.0", MBSEED_PUBLIC_URL: "https://mbseed.example.org/", - MBSEED_STORAGE_DIR: "/srv/mbseed/covers", MBSEED_STORAGE_MAX_BYTES: "5000000", MBSEED_TRUST_PROXY: "1", MBSEED_ASSETS_DIR: "/srv/mbseed/web", - })).toEqual({ - port: 9000, hostname: "0.0.0.0", publicUrl: "https://mbseed.example.org", storageDir: "/srv/mbseed/covers", - storageMaxBytes: 5000000, trustProxy: true, assetsDir: "/srv/mbseed/web", - }); - }); - test.each([ - [{ MBSEED_STORAGE_DIR: "/x" }, /MBSEED_PUBLIC_URL/], - [{ MBSEED_PUBLIC_URL: "mbseed.example.org" }, /not a URL/], - [{ MBSEED_PUBLIC_URL: "https://example.org/sub/path" }, /origin/], - [{ MBSEED_PORT: "eighty" }, /whole number/], - [{ MBSEED_STORAGE_MAX_BYTES: "1GB" }, /whole number/], - ])("%o is refused", (env, message) => { - expect(() => readConfig(env)).toThrow(ConfigError); - expect(() => readConfig(env)).toThrow(message); - }); -}); - -describe("clientIp", () => { - const req = (xff?: string) => new Request("http://x/", { headers: xff ? { "x-forwarded-for": xff } : {} }); - test("ignores X-Forwarded-For unless the proxy is trusted", () => { - expect(clientIp(req("1.2.3.4"), false, "10.0.0.1")).toBe("10.0.0.1"); - }); - test("trusts only the last entry, the one our proxy appended", () => { - // Review Focus 4: the client sent "1.2.3.4" itself; the proxy appended the real address. - expect(clientIp(req("1.2.3.4, 203.0.113.9"), true, "127.0.0.1")).toBe("203.0.113.9"); - expect(clientIp(req(), true, "127.0.0.1")).toBe("127.0.0.1"); - expect(clientIp(req(), false)).toBe("unknown"); - }); -}); - -describe("resolveAssetPath", () => { - test.each([ - ["/", "index.html"], - ["/index.html", "index.html"], - ["/main-ab12cd.js", "main-ab12cd.js"], - ["/mbseed.user.js", "mbseed.user.js"], - ["/fonts/x.woff2", "fonts/x.woff2"], - ])("%p → %p", (path, file) => { - expect(resolveAssetPath(path)).toBe(file); - }); - // Review Focus 5 - test.each(["/..%2f.env", "/%2e%2e/package.json", "/a/../../b", "/.env", "/%00x", "/a\\..\\b", "/%E0%A4%A"])("%p is refused", (path) => { - expect(resolveAssetPath(path)).toBeNull(); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/server-config.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write the three modules** - -`src/server/rate-limit.ts`: -```ts -/** Fixed-window counter per key (client IP). In memory: one server process. */ -export class RateLimiter { - private readonly hits = new Map(); - - constructor(readonly limit: number, readonly windowMs: number, private readonly now: () => number = Date.now) {} - - take(key: string): boolean { - const t = this.now(); - const hit = this.hits.get(key); - if (!hit || t - hit.start >= this.windowMs) { - if (this.hits.size > 10_000) this.prune(t); - this.hits.set(key, { start: t, count: 1 }); - return true; - } - if (hit.count >= this.limit) return false; - hit.count++; - return true; - } - - private prune(t: number): void { - for (const [key, hit] of this.hits) if (t - hit.start >= this.windowMs) this.hits.delete(key); - } -} -``` - -`src/server/config.ts`: -```ts -// Web-standard: parses the environment; no Bun or node APIs. - -export interface ServerConfig { - port: number; - hostname: string; - publicUrl: string | null; - storageDir: string | null; - storageMaxBytes: number; - trustProxy: boolean; - assetsDir: string; -} - -export class ConfigError extends Error { - override name = "ConfigError"; -} - -export const DEFAULT_STORAGE_MAX_BYTES = 1024 ** 3; - -export function readConfig(env: Record): ServerConfig { - const publicUrl = parsePublicUrl(env.MBSEED_PUBLIC_URL); - const storageDir = env.MBSEED_STORAGE_DIR?.trim() || null; - if (storageDir && !publicUrl) { - throw new ConfigError("MBSEED_STORAGE_DIR needs MBSEED_PUBLIC_URL: MangaBaka must be able to download stored covers."); - } - return { - port: wholeNumber(env, "MBSEED_PORT", 8787), - hostname: env.MBSEED_HOST?.trim() || "127.0.0.1", - publicUrl, - storageDir, - storageMaxBytes: wholeNumber(env, "MBSEED_STORAGE_MAX_BYTES", DEFAULT_STORAGE_MAX_BYTES), - trustProxy: env.MBSEED_TRUST_PROXY === "1", - assetsDir: env.MBSEED_ASSETS_DIR?.trim() || "dist/web", - }; -} - -function parsePublicUrl(raw: string | undefined): string | null { - const value = raw?.trim(); - if (!value) return null; - let url: URL; - try { - url = new URL(value); - } catch { - throw new ConfigError(`MBSEED_PUBLIC_URL is not a URL: ${value}`); - } - if (!["http:", "https:"].includes(url.protocol) || url.pathname !== "/" || url.search || url.hash) { - throw new ConfigError(`MBSEED_PUBLIC_URL must be an origin like https://mbseed.example.org, got ${value}`); - } - return url.origin; -} - -function wholeNumber(env: Record, name: string, fallback: number): number { - const raw = env[name]?.trim(); - if (!raw) return fallback; - if (!/^\d+$/.test(raw)) throw new ConfigError(`${name} must be a whole number, got "${raw}".`); - return Number(raw); -} - -/** - * The address rate limits key on. Behind our reverse proxy (MBSEED_TRUST_PROXY=1) that is the - * last X-Forwarded-For entry — the one the proxy appended; earlier entries come from the client. - */ -export function clientIp(req: Request, trustProxy: boolean, peer?: string): string { - if (trustProxy) { - const last = (req.headers.get("x-forwarded-for") ?? "").split(",").map((s) => s.trim()).filter(Boolean).at(-1); - if (last) return last; - } - return peer ?? "unknown"; -} -``` - -`src/server/assets-path.ts`: -```ts -/** Maps a URL path to a file path relative to the assets root, or null. No dot-segments, no dotfiles. */ -export function resolveAssetPath(pathname: string): string | null { - let decoded: string; - try { - decoded = decodeURIComponent(pathname); - } catch { - return null; - } - if (decoded.includes("\0") || decoded.includes("\\")) return null; - const segments: string[] = []; - for (const segment of decoded.split("/")) { - if (segment === "") continue; - if (segment.startsWith(".")) return null; - segments.push(segment); - } - return segments.length ? segments.join("/") : "index.html"; -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/server-config.test.ts test/boundaries.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/server/rate-limit.ts src/server/config.ts src/server/assets-path.ts test/server-config.test.ts -git commit -m "feat(server): config from env, trusted client IP, per-IP rate limiter, safe asset paths" -``` - ---- - -### Task 17: The web-standard request handler - -**Files:** -- Create: `src/server/app.ts`, `test/server-app.test.ts` - -**Interfaces:** -- Consumes: `allowedFetchHosts` (Task 6), `probeImage`, `sha256Hex`, `readCapped`, `MAX_IMAGE_BYTES`, `ImageError` (Task 7), `userAgentFor`, `TOOL_USER_AGENT`, `Transport` (Task 3), `uploadToCatbox` (Task 12), Task 15–16 modules. -- Produces: `type Upstream = (url: string, init?: RequestInit) => Promise`, `interface ServerEnv { storage: Storage | null; publicUrl: string | null; assets(relativePath: string): Promise; clientIp(req: Request): string; upstream?: Upstream; allowedHosts?: ReadonlySet; limits?: { fetch: RateLimiter; upload: RateLimiter } }`, `MAX_REDIRECTS = 5`, `createApp(env): (req: Request) => Promise`. (Spec §4.10 names it `handle(req, env)`; `createApp` returns that handler with the rate-limit state bound once.) - -- [ ] **Step 1: Write the failing tests** - -`test/server-app.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { sha256Hex } from "../src/core/images"; -import { TOOL_USER_AGENT } from "../src/core/transport"; -import { createApp, type ServerEnv, type Upstream } from "../src/server/app"; -import { RateLimiter } from "../src/server/rate-limit"; -import { MemoryStorage } from "../src/server/storage"; -import { fixtureBytes } from "./helpers/fixtures"; - -const ORIGIN = "https://mbseed.example.org"; -const POSTER = "https://swebtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg"; - -function setup(over: Partial = {}, routes: Record Response | Promise> = {}) { - const calls: { url: string; init?: RequestInit }[] = []; - const upstream: Upstream = async (url, init) => { - calls.push({ url, init }); - const route = routes[url]; - if (!route) throw new Error(`unexpected upstream ${url}`); - return route(); - }; - const app = createApp({ - storage: new MemoryStorage(10 * 1024 * 1024), - publicUrl: ORIGIN, - assets: async (path) => (path === "index.html" ? new Response("page", { headers: { "content-type": "text/html" } }) : null), - clientIp: () => "203.0.113.9", - upstream, - ...over, - }); - const call = (path: string, init?: RequestInit) => app(new Request(`${ORIGIN}${path}`, init)); - return { app, call, calls }; -} -const fetchPath = (url: string) => `/api/fetch?url=${encodeURIComponent(url)}`; -const uploadInit = (body: unknown): RequestInit => ({ method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }); - -describe("/api/config", () => { - test("reports storage and whether it is full", async () => { - expect(await (await setup().call("/api/config")).json()).toEqual({ storage: true, storageFull: false }); - expect(await (await setup({ storage: null }).call("/api/config")).json()).toEqual({ storage: false, storageFull: false }); - expect(await (await setup({ storage: new MemoryStorage(10) }).call("/api/config")).json()).toEqual({ storage: true, storageFull: true }); - }); -}); - -describe("/api/fetch", () => { - test("fetches allowlisted hosts with the right User-Agent and passes status and type through", async () => { - const { call, calls } = setup({}, { - "https://www.webtoons.com/-/-/-/list?title_no=95": () => new Response("", { headers: { "content-type": "text/html" } }), - "https://web.archive.org/cdx/search/cdx?url=x": () => new Response("", { status: 503 }), - }); - const page = await call(fetchPath("https://www.webtoons.com/-/-/-/list?title_no=95")); - expect(page.status).toBe(200); - expect(page.headers.get("content-type")).toBe("text/html"); - expect(await page.text()).toBe(""); - expect(new Headers(calls[0]!.init!.headers).get("user-agent")).toMatch(/^Mozilla/); - expect((await call(fetchPath("https://web.archive.org/cdx/search/cdx?url=x"))).status).toBe(503); - expect(new Headers(calls[1]!.init!.headers).get("user-agent")).toBe(TOOL_USER_AGENT); - }); - - test.each(["https://evil.example/x", "http://169.254.169.254/latest", "ftp://www.webtoons.com/x", "https://www.webtoons.com:8443/x", "not a url"])( - "%p is refused without being fetched", async (url) => { - const { call, calls } = setup(); - expect([400, 403]).toContain((await call(fetchPath(url))).status); - expect(calls).toHaveLength(0); - }); - - test("follows redirects within the allowlist", async () => { - const { call } = setup({}, { - "https://www.webtoons.com/-/-/-/list?title_no=95": () => new Response(null, { status: 301, headers: { location: "/en/fantasy/tower-of-god/list?title_no=95" } }), - "https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95": () => new Response("canonical"), - }); - expect(await (await call(fetchPath("https://www.webtoons.com/-/-/-/list?title_no=95"))).text()).toBe("canonical"); - }); - - test("redirect to a non-allowlisted host is refused", async () => { - // Review Focus 2 - const { call, calls } = setup({}, { - "https://www.webtoons.com/x": () => new Response(null, { status: 302, headers: { location: "https://evil.example/" } }), - }); - expect((await call(fetchPath("https://www.webtoons.com/x"))).status).toBe(403); - expect(calls.map((c) => c.url)).toEqual(["https://www.webtoons.com/x"]); - }); - - test("responses over 20 MB are refused", async () => { - const { call } = setup({}, { [POSTER]: () => new Response("x", { headers: { "content-length": String(21 * 1024 * 1024) } }) }); - expect((await call(fetchPath(POSTER))).status).toBe(413); - }); - - test("rejects other origins; allows ours, mangabaka.org and none", async () => { - const { call } = setup({}, { [POSTER]: () => new Response("x") }); - expect((await call(fetchPath(POSTER), { headers: { origin: "https://evil.example" } })).status).toBe(403); - expect((await call(fetchPath(POSTER), { headers: { origin: ORIGIN } })).status).toBe(200); - expect((await call(fetchPath(POSTER), { headers: { origin: "https://mangabaka.org" } })).status).toBe(200); - expect((await call(fetchPath(POSTER))).status).toBe(200); - }); - - test("rate-limited per IP", async () => { - const { call } = setup({ limits: { fetch: new RateLimiter(1, 60_000), upload: new RateLimiter(1, 60_000) } }, { [POSTER]: () => new Response("x") }); - expect((await call(fetchPath(POSTER))).status).toBe(200); - expect((await call(fetchPath(POSTER))).status).toBe(429); - }); - - test("proxy errors are marked so the page can tell them from upstream answers", async () => { - const res = await setup().call(fetchPath("https://evil.example/")); - expect(res.headers.get("x-mbseed-error")).toBe("1"); - expect(await res.json()).toEqual({ error: expect.stringContaining("evil.example") }); - }); -}); - -describe("/api/upload", () => { - test("store: fetches the source itself, keeps it by hash, returns its public URL", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - const sha = await sha256Hex(poster); - const { call } = setup({}, { [POSTER]: () => new Response(poster as Uint8Array) }); - const res = await call("/api/upload", uploadInit({ url: POSTER, host: "store" })); - expect(await res.json()).toEqual({ url: `${ORIGIN}/covers/${sha}.jpg`, sha256: sha }); - const cover = await call(`/covers/${sha}.jpg`); - expect(cover.headers.get("content-type")).toBe("image/jpeg"); - expect((await cover.arrayBuffer()).byteLength).toBe(poster.byteLength); - }); - - test("store full: 507", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - const { call } = setup({ storage: new MemoryStorage(1000) }, { [POSTER]: () => new Response(poster as Uint8Array) }); - expect((await call("/api/upload", uploadInit({ url: POSTER, host: "store" }))).status).toBe(507); - }); - - test("catbox: forwards the userhash to catbox only", async () => { - const poster = await fixtureBytes("images/poster-95.jpg"); - const { call, calls } = setup({}, { - [POSTER]: () => new Response(poster as Uint8Array), - "https://catbox.moe/user/api.php": () => new Response("https://files.catbox.moe/abc.jpg"), - }); - const res = await call("/api/upload", uploadInit({ url: POSTER, host: "catbox", userhash: "abc123" })); - expect((await res.json()).url).toBe("https://files.catbox.moe/abc.jpg"); - expect((calls[1]!.init!.body as FormData).get("userhash")).toBe("abc123"); - }); - - test.each([ - [{ url: POSTER, host: "imgur" }, 400], - [{ host: "store" }, 400], - [{ url: "https://evil.example/x.jpg", host: "store" }, 403], - [{ url: POSTER, host: "catbox", userhash: "../../x" }, 400], - ])("%o → %i", async (body, status) => { - expect((await setup().call("/api/upload", uploadInit(body))).status).toBe(status); - }); - - test("no storage configured: 404 for store", async () => { - expect((await setup({ storage: null }).call("/api/upload", uploadInit({ url: POSTER, host: "store" }))).status).toBe(404); - }); - - test("not an image: 415", async () => { - const { call } = setup({}, { [POSTER]: () => new Response("image not found") }); - expect((await call("/api/upload", uploadInit({ url: POSTER, host: "store" }))).status).toBe(415); - }); -}); - -describe("covers and assets", () => { - test.each(["/covers/../../x", `/covers/${"A".repeat(64)}.jpg`, `/covers/${"a".repeat(64)}.svg`, "/covers/"])("%p → 404", async (path) => { - expect((await setup().call(path)).status).toBe(404); - }); - test("static assets via env.assets; unknown paths 404; writes 405", async () => { - const { call } = setup(); - expect(await (await call("/")).text()).toBe("page"); - expect((await call("/nope.js")).status).toBe(404); - expect((await call("/", { method: "POST" })).status).toBe(405); - expect((await call("/..%2f.env")).status).toBe(404); - }); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/server-app.test.ts` -Expected: FAIL — module not found. - -- [ ] **Step 3: Write `src/server/app.ts`** - -```ts -// The portable request handler (spec §4.10): web-standard APIs only, so a Workers entry can wrap it later. -import { uploadToCatbox } from "../core/hosting/catbox"; -import { ImageError, MAX_IMAGE_BYTES, probeImage, readCapped, sha256Hex } from "../core/images"; -import { allowedFetchHosts } from "../core/sources/registry"; -import { TOOL_USER_AGENT, userAgentFor, type Transport } from "../core/transport"; -import { resolveAssetPath } from "./assets-path"; -import { RateLimiter } from "./rate-limit"; -import { CONTENT_TYPES, COVER_NAME, StorageFullError, type Storage } from "./storage"; - -export type Upstream = (url: string, init?: RequestInit) => Promise; - -export interface ServerEnv { - storage: Storage | null; - publicUrl: string | null; - /** Serves a built file by its path relative to the assets root, or null. */ - assets(relativePath: string): Promise; - clientIp(req: Request): string; - upstream?: Upstream; - allowedHosts?: ReadonlySet; - limits?: { fetch: RateLimiter; upload: RateLimiter }; -} - -export const MAX_REDIRECTS = 5; -const UPSTREAM_TIMEOUT_MS = 30_000; -/** Pages whose scripts may call our API besides our own (the userscript runs on mangabaka.org). */ -const EXTRA_ORIGINS = new Set(["https://mangabaka.org"]); - -const error = (status: number, message: string) => - Response.json({ error: message }, { status, headers: { "x-mbseed-error": "1", "cache-control": "no-store" } }); - -export function createApp(env: ServerEnv): (req: Request) => Promise { - const hosts = env.allowedHosts ?? allowedFetchHosts(); - // A run is ~60–100 proxied requests over several minutes (spec §5). - const limits = env.limits ?? { fetch: new RateLimiter(240, 60_000), upload: new RateLimiter(60, 60_000) }; - const upstream: Upstream = env.upstream ?? ((url, init) => fetch(url, init)); - const ownOrigin = (req: Request) => env.publicUrl ?? new URL(req.url).origin; - - const allowed = (u: URL) => - (u.protocol === "https:" || u.protocol === "http:") && hosts.has(u.hostname) && ["", "80", "443"].includes(u.port); - - /** GET through the allowlist, re-checking every redirect hop. */ - async function proxyGet(target: string | null): Promise { - if (!target) return error(400, "Missing url."); - let url: URL; - try { - url = new URL(target); - } catch { - return error(400, "Not a URL."); - } - for (let hop = 0; hop <= MAX_REDIRECTS; hop++) { - if (!allowed(url)) return error(403, `Host not allowed: ${url.hostname}`); - let res: Response; - try { - res = await upstream(url.toString(), { - redirect: "manual", - headers: { "user-agent": userAgentFor(url.toString()), accept: "*/*" }, - signal: AbortSignal.timeout(UPSTREAM_TIMEOUT_MS), - }); - } catch (err) { - return error(502, `Upstream failed: ${err instanceof Error ? err.message : String(err)}`); - } - const location = res.headers.get("location"); - if (res.status >= 300 && res.status < 400 && location) { - await res.body?.cancel(); - url = new URL(location, url); - continue; - } - let body: Uint8Array; - try { - body = await readCapped(res, MAX_IMAGE_BYTES); - } catch (err) { - if (err instanceof ImageError) return error(413, "Response larger than 20 MB."); - throw err; - } - return new Response(body as Uint8Array, { - status: res.status, - headers: { - "content-type": res.headers.get("content-type") ?? "application/octet-stream", - "cache-control": "no-store", - "x-mbseed-final-url": url.toString(), - }, - }); - } - return error(502, "Too many redirects."); - } - - const catboxTransport: Transport = { - fetch: (url, init = {}) => - upstream(url, { - method: init.method, - headers: { "user-agent": TOOL_USER_AGENT, ...init.headers }, - body: init.body, - signal: AbortSignal.timeout(init.timeoutMs ?? UPSTREAM_TIMEOUT_MS), - }), - }; - - async function upload(req: Request): Promise { - let body: { url?: unknown; host?: unknown; userhash?: unknown }; - try { - body = await req.json(); - } catch { - return error(400, "Body must be JSON."); - } - if (typeof body.url !== "string") return error(400, "url is required."); - if (body.host !== "store" && body.host !== "catbox") return error(400, 'host must be "store" or "catbox".'); - if (body.userhash !== undefined && (typeof body.userhash !== "string" || !/^[A-Za-z0-9]{1,64}$/.test(body.userhash))) { - return error(400, "Bad userhash."); - } - if (body.host === "store" && !env.storage) return error(404, "This server has no storage."); - - // The client sends the source URL, never bytes, so this can't host arbitrary files (spec §4.6). - const fetched = await proxyGet(body.url); - if (fetched.headers.get("x-mbseed-error")) return fetched; - if (!fetched.ok) return error(502, `Could not fetch the image (HTTP ${fetched.status}).`); - const bytes = new Uint8Array(await fetched.arrayBuffer()); - const info = probeImage(bytes); - if (!info) return error(415, "That URL is not an image."); - const sha256 = await sha256Hex(bytes); - - if (body.host === "store") { - const name = `${sha256}.${info.ext}`; - try { - await env.storage!.put(name, bytes); - } catch (err) { - if (err instanceof StorageFullError) return error(507, "This server's storage is full; use litterbox or catbox."); - throw err; - } - return Response.json({ url: `${env.publicUrl}/covers/${name}`, sha256 }); - } - try { - const blob = new Blob([bytes as Uint8Array], { type: info.contentType }); - const { url } = await uploadToCatbox(catboxTransport, blob, `${sha256}.${info.ext}`, body.userhash as string | undefined); - return Response.json({ url, sha256 }); - } catch (err) { - return error(502, err instanceof Error ? err.message : "catbox upload failed."); - } - } - - async function cover(name: string): Promise { - if (!env.storage || !COVER_NAME.test(name)) return error(404, "Not found."); - const blob = await env.storage.get(name); - if (!blob) return error(404, "Not found."); - return new Response(blob, { - headers: { - "content-type": CONTENT_TYPES[name.split(".").pop()!]!, - "cache-control": "public, max-age=86400", - "x-content-type-options": "nosniff", - }, - }); - } - - async function route(req: Request): Promise { - const url = new URL(req.url); - const path = url.pathname; - const isRead = req.method === "GET" || req.method === "HEAD"; - - if (path.startsWith("/api/")) { - const origin = req.headers.get("origin"); - if (origin && origin !== ownOrigin(req) && !EXTRA_ORIGINS.has(origin)) return error(403, "Cross-origin requests are not allowed."); - if (path === "/api/config" && isRead) { - return Response.json({ storage: env.storage !== null, storageFull: env.storage?.isFull() ?? false }, { headers: { "cache-control": "no-store" } }); - } - if (path === "/api/fetch" && req.method === "GET") { - if (!limits.fetch.take(env.clientIp(req))) return error(429, "Too many requests; slow down."); - return proxyGet(url.searchParams.get("url")); - } - if (path === "/api/upload" && req.method === "POST") { - if (!limits.upload.take(env.clientIp(req))) return error(429, "Too many uploads; slow down."); - return upload(req); - } - return error(404, "No such API route."); - } - if (!isRead) return error(405, "Method not allowed."); - if (path.startsWith("/covers/")) return cover(path.slice("/covers/".length)); - const file = resolveAssetPath(path); - return (file && (await env.assets(file))) || error(404, "Not found."); - } - - return async (req) => { - try { - return await route(req); - } catch (err) { - console.error("request failed:", req.method, new URL(req.url).pathname, err); - return error(500, "Internal error."); - } - }; -} -``` - -- [ ] **Step 4: Run tests** - -Run: `bun test test/server-app.test.ts test/boundaries.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 5: Commit** - -```bash -git add src/server/app.ts test/server-app.test.ts -git commit -m "feat(server): web-standard handler (config, allowlisted fetch proxy, upload, covers, assets)" -``` - ---- - -### Task 18: Bun entry and a local smoke run - -**Files:** -- Create: `src/server/bun.ts` -- Modify: `docs/ARCHITECTURE.md`, `AGENTS.md` (Commands), `docs/PERMISSIONS.md`, `.claude/settings.json`, `.gemini/settings.json` - -**Interfaces:** -- Consumes: `createApp` (Task 17), `readConfig`, `clientIp` (Task 16), `DirStorage` (Task 15), `resolveAssetPath` is applied inside `createApp`; `bun.ts` only maps a relative path to a file. - -- [ ] **Step 1: Write `src/server/bun.ts`** (thin adapter; its parts are tested in Tasks 15–17) - -```ts -// Self-hosted entry: Bun.serve behind our reverse proxy (spec §4.10). Keep it thin. -import { join, resolve } from "node:path"; -import { createApp } from "./app"; -import { clientIp, readConfig } from "./config"; -import { DirStorage } from "./storage-dir"; - -const config = readConfig(process.env); -const storage = config.storageDir ? await DirStorage.open(config.storageDir, config.storageMaxBytes) : null; - -if (storage) { - const sweep = () => - storage - .sweep() - .then((n) => n > 0 && console.log(`swept ${n} expired cover(s)`)) - .catch((err) => console.error("sweep failed:", err)); - await sweep(); - setInterval(sweep, 24 * 60 * 60 * 1000); -} - -const assetsRoot = resolve(config.assetsDir); -let server: ReturnType; - -const app = createApp({ - storage, - publicUrl: config.publicUrl, - // `relativePath` is already checked by resolveAssetPath (no dot-segments, no dotfiles). - assets: async (relativePath) => { - const file = Bun.file(join(assetsRoot, relativePath)); - return (await file.exists()) ? new Response(file) : null; - }, - clientIp: (req) => clientIp(req, config.trustProxy, server.requestIP(req)?.address), -}); - -server = Bun.serve({ port: config.port, hostname: config.hostname, fetch: app }); -console.log( - `mangabaka-seeder on http://${server.hostname}:${server.port} · storage: ${ - storage ? `${config.storageDir} (${storage.usedBytes()} of ${config.storageMaxBytes} bytes)` : "off" - }`, -); -``` - -- [ ] **Step 2: Typecheck** - -Run: `bun run typecheck` → Expected: exit 0. - -- [ ] **Step 3: Smoke-run it locally** (local only; fetches one real Webtoons page and one real image through the proxy, stores it in a temp folder — no third-party uploads) - -Run in the background: `MBSEED_STORAGE_DIR=$(mktemp -d) MBSEED_PUBLIC_URL=http://127.0.0.1:8787 bun run start` -Expected log: `mangabaka-seeder on http://127.0.0.1:8787 · storage: /tmp/… (0 of 1073741824 bytes)` - -Then: -```bash -curl -s http://127.0.0.1:8787/api/config -# {"storage":true,"storageFull":false} -curl -s -o /dev/null -w '%{http_code} %{content_type}\n' 'http://127.0.0.1:8787/api/fetch?url=https%3A%2F%2Fwww.webtoons.com%2F-%2F-%2F-%2Flist%3Ftitle_no%3D95' -# 200 text/html;charset=UTF-8 -curl -s -o /dev/null -w '%{http_code}\n' 'http://127.0.0.1:8787/api/fetch?url=https%3A%2F%2Fexample.org%2F' -# 403 -curl -s -X POST -H 'content-type: application/json' -d '{"url":"https://swebtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg","host":"store"}' http://127.0.0.1:8787/api/upload -# {"url":"http://127.0.0.1:8787/covers/<64 hex>.jpg","sha256":"<64 hex>"} -curl -s -o /dev/null -w '%{http_code} %{content_type}\n' -# 200 image/jpeg -``` -Stop the server. If any line differs, debug with superpowers:systematic-debugging before going on. - -- [ ] **Step 4: Record what was verified** - -`AGENTS.md` Commands: add `bun run start` (server; env vars in `docs/ARCHITECTURE.md`). `docs/PERMISSIONS.md` Verified table: add `| bun run start | worked (local smoke run) |`. Leave `start` out of the Allow tier (it opens a port and fetches from the internet — keep it a prompt); add a sentence saying so under Known gaps. - -`docs/ARCHITECTURE.md`: in the Map, describe `src/server/` as built: files and their roles (`app.ts` handler, `storage.ts`/`storage-dir.ts`, `config.ts`, `rate-limit.ts`, `assets-path.ts`, `bun.ts`). Add a "Running the server" section listing every `MBSEED_*` variable with its default (copy from Global Constraints) and: "Put it behind a reverse proxy that terminates TLS and **appends** the client address to `X-Forwarded-For` (nginx `$proxy_add_x_forwarded_for`, Caddy's default), then set `MBSEED_TRUST_PROXY=1`." - -`docs/DECISIONS.md`: append an entry "Server API details" covering: `createApp(env)` instead of `handle(req, env)` (binds rate-limit state once); `mangabaka.org` accepted as an `Origin` for the userscript; the last `X-Forwarded-For` entry is trusted; `/api/fetch` buffers up to 20 MB instead of streaming (simpler cap; pages and covers are well below it). - -- [ ] **Step 5: Commit** - -```bash -git add src/server/bun.ts AGENTS.md docs/PERMISSIONS.md docs/ARCHITECTURE.md docs/DECISIONS.md -git commit -m "feat(server): Bun.serve entry with folder storage and daily sweep; document running it" -``` - ---- - -## Part C — The page - -### Task 19: Working page (template, wiring, view model, rendering, transport) - -The page works end to end here with plain styling; Task 20 gives it its design. Both front ends mount the same template and wiring (`mountApp`), so the userscript (Part D) reuses all of it. - -**Files:** -- Create: `src/web/template.ts`, `src/web/view-model.ts`, `src/web/render.ts`, `src/web/app.ts`, `src/web/transport.ts`, `src/web/main.ts`, `src/web/index.html`, `src/web/styles.css`, `src/types/text-imports.d.ts` -- Test: `test/web-transport.test.ts`, `test/view-model.test.ts`, `test/template.test.ts` - -**Interfaces:** -- Consumes: `Session`, `loadToken`, `saveToken`, `KeyValueStore`, `PreparedSubmission` (Task 14); `HOSTS`, `LITTERBOX_TIMES`, `HostId`, `HostOptions`, `LitterboxTime` (Task 12); `coverFormUrl` (Task 10); `formatMonth` (Task 8); `CONTENT_RATINGS`, `GRANULARITIES` (Task 3); `withPolicy`, `CORS_OPEN_HOSTS`, `TransportError` (Task 3); `Stage` (Task 13). -- Produces: `APP_HTML: string` (root element `.mbseed-app`); `interface ServerConfig { storage: boolean; storageFull: boolean }`, `interface HostChoice { id: HostId; label: string; lifetime: string; disabled: string | null; warning: string | null }`, `hostChoices(config, opts, { catboxNeedsServer })`, `defaultHost(choices)`, `stageText(stage, done?, total?)`, `GRANULARITY_LABELS`, `cardFacts(card): string[]`; `renderCards(container, session, onEdit)`, `renderPresent(container, present)`, `renderPreview(container, preview)`; `USERHASH_KEY`, `interface AppOptions { root: ParentNode; session: Session; store: KeyValueStore; config: ServerConfig | null; catboxNeedsServer: boolean; seriesLink?: string }`, `mountApp(options)`; `pageTransport(origin, rawFetch?)`, `localStore(): KeyValueStore`. - -- [ ] **Step 1: Write the failing tests** - -`test/web-transport.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { TransportError } from "../src/core/transport"; -import { pageTransport } from "../src/web/transport"; - -const ORIGIN = "https://mbseed.example.org"; -function raw(respond: (url: string) => Response = () => new Response("ok")) { - const seen: { url: string; init?: RequestInit }[] = []; - const fn = async (url: string, init?: RequestInit) => { - seen.push({ url, init }); - return respond(url); - }; - return { fn, seen }; -} - -test("MangaBaka, litterbox and our own server directly; everything else through /api/fetch", async () => { - const r = raw(); - const t = pageTransport(ORIGIN, r.fn); - await t.fetch("https://api.mangabaka.org/v1/series/1", { headers: { "x-api-key": "mb-x" } }); - await t.fetch("https://litterbox.catbox.moe/resources/internals/api.php", { method: "POST", body: "x" }); - await t.fetch(`${ORIGIN}/api/config`); - await t.fetch("https://www.webtoons.com/-/-/-/list?title_no=95"); - expect(r.seen.map((s) => s.url)).toEqual([ - "https://api.mangabaka.org/v1/series/1", - "https://litterbox.catbox.moe/resources/internals/api.php", - `${ORIGIN}/api/config`, - `${ORIGIN}/api/fetch?url=${encodeURIComponent("https://www.webtoons.com/-/-/-/list?title_no=95")}`, - ]); - expect(new Headers(r.seen[3]!.init?.headers).get("x-api-key")).toBeNull(); -}); - -test("a POST to a host without CORS is refused in the page", async () => { - const t = pageTransport(ORIGIN, raw().fn); - await expect(t.fetch("https://catbox.moe/user/api.php", { method: "POST", body: "x" })).rejects.toBeInstanceOf(TransportError); -}); - -test("a proxy error becomes a TransportError with the server's message", async () => { - const t = pageTransport(ORIGIN, raw(() => - Response.json({ error: "Host not allowed: evil.example" }, { status: 403, headers: { "x-mbseed-error": "1" } })).fn); - await expect(t.fetch("https://www.webtoons.com/x")).rejects.toThrow(/Host not allowed/); -}); - -test("an upstream status (e.g. Wayback 429) passes through as a response", async () => { - const t = pageTransport(ORIGIN, raw(() => new Response("slow down", { status: 429 })).fn); - expect((await t.fetch("https://web.archive.org/web/1id_/x")).status).toBe(429); -}); -``` - -`test/view-model.test.ts`: -```ts -import { expect, test } from "bun:test"; -import type { ReviewCard } from "../src/core/session"; -import { GRANULARITY_LABELS, cardFacts, defaultHost, hostChoices, stageText } from "../src/web/view-model"; -import { inspected } from "./helpers/covers"; - -test("hosts: server first when it has room; hidden without storage; disabled when full", () => { - const ids = (c: ReturnType) => c.map((h) => `${h.id}${h.disabled ? "(disabled)" : ""}`); - expect(ids(hostChoices({ storage: true, storageFull: false }, {}, { catboxNeedsServer: true }))).toEqual(["server", "litterbox", "catbox", "direct"]); - expect(ids(hostChoices({ storage: false, storageFull: false }, {}, { catboxNeedsServer: true }))).toEqual(["litterbox", "catbox", "direct"]); - const full = hostChoices({ storage: true, storageFull: true }, {}, { catboxNeedsServer: true }); - expect(ids(full)).toEqual(["server(disabled)", "litterbox", "catbox", "direct"]); - expect(defaultHost(full)).toBe("litterbox"); - expect(ids(hostChoices(null, {}, { catboxNeedsServer: true }))).toEqual(["litterbox", "catbox(disabled)", "direct"]); - expect(ids(hostChoices(null, {}, { catboxNeedsServer: false }))).toEqual(["litterbox", "catbox", "direct"]); -}); - -test("litterbox carries the review-time warning and its chosen lifetime", () => { - const litterbox = hostChoices(null, { litterboxTime: "24h" }, { catboxNeedsServer: false })[0]!; - expect(litterbox.lifetime).toBe("24 hours"); - expect(litterbox.warning).toMatch(/review can take longer/); -}); - -test("stage text with and without progress", () => { - expect(stageText("history")).toBe("Looking through the Wayback Machine…"); - expect(stageText("history", 3, 24)).toBe("Looking through the Wayback Machine… 3/24"); -}); - -test("granularity labels", () => { - expect(GRANULARITY_LABELS).toEqual({ 1: "every month", 3: "every 3 months", 6: "every 6 months", 12: "every year" }); -}); - -test("card facts: size, shape, when seen, archive-only", () => { - const card = (cover: ReturnType): ReviewCard => - ({ cover, selected: true, type: cover.type, note: "", contentRating: "safe", hostedUrl: null, error: null }); - expect(cardFacts(card(inspected({ bytes: new Uint8Array(352813) })))).toEqual(["480×623", "345 KB", "on the page now"]); - expect(cardFacts(card(inspected({ - width: 720, height: 1230, shape: "tall", live: false, bytes: new Uint8Array(2048), - candidate: { current: false, seenAt: "20230401000000", lastSeenAt: "20260301000000" }, - })))).toEqual(["720×1230", "2 KB", "landing-page art", "seen 2023-04 – 2026-03", "only on archive.org"]); -}); -``` - -`test/template.test.ts`: -```ts -import { expect, test } from "bun:test"; -import { APP_HTML } from "../src/web/template"; - -test("every element app.ts looks up exists in the template exactly once", async () => { - const app = await Bun.file("src/web/app.ts").text(); - const wanted = [...app.matchAll(/\bq<[^>]+>\("([a-z-]+)"\)/g)].map((m) => m[1]!); - expect(wanted.length).toBeGreaterThan(30); - for (const id of wanted) { - expect({ id, count: APP_HTML.split(`id="${id}"`).length - 1 }).toEqual({ id, count: 1 }); - } -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/web-transport.test.ts test/view-model.test.ts test/template.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/web/transport.ts`** - -```ts -import { CORS_OPEN_HOSTS, TransportError, type Transport } from "../core/transport"; - -type RawFetch = (url: string, init?: RequestInit) => Promise; - -/** - * Spec §4.2: MangaBaka and litterbox send CORS headers, so the page calls them directly (the - * token never touches our server). Everything else is read through our /api/fetch proxy. - */ -export function pageTransport(origin: string, rawFetch: RawFetch = (url, init) => fetch(url, init)): Transport { - return { - async fetch(url, init = {}) { - const target = new URL(url, origin); - const { timeoutMs: _timeout, ...rest } = init; - if (target.origin === origin || CORS_OPEN_HOSTS.has(target.hostname)) return rawFetch(target.toString(), rest); - if ((init.method ?? "GET").toUpperCase() !== "GET") { - throw new TransportError(`The page can't ${init.method} to ${target.hostname} (no CORS).`, url); - } - const res = await rawFetch(`${origin}/api/fetch?url=${encodeURIComponent(target.toString())}`, { signal: init.signal }); - if (res.headers.get("x-mbseed-error")) { - const body = (await res.json().catch(() => ({}))) as { error?: string }; - throw new TransportError(`Server proxy: ${body.error ?? `HTTP ${res.status}`}`, url, res.status); - } - return res; - }, - }; -} - -/** localStorage, but never fatal: private windows and blocked storage just forget. */ -export function localStore() { - return { - get: (k: string) => { - try { - return localStorage.getItem(k); - } catch { - return null; - } - }, - set: (k: string, v: string) => { - try { - localStorage.setItem(k, v); - } catch { - // storage unavailable: remembering is a convenience - } - }, - remove: (k: string) => { - try { - localStorage.removeItem(k); - } catch { - // see above - } - }, - }; -} -``` - -- [ ] **Step 4: Write `src/web/view-model.ts`** - -```ts -import { formatMonth } from "../core/covers"; -import type { HostId, HostOptions } from "../core/hosting/host"; -import { HOSTS } from "../core/hosting/registry"; -import type { Granularity } from "../core/model"; -import type { Stage } from "../core/pipeline"; -import type { ReviewCard } from "../core/session"; - -export interface ServerConfig { - storage: boolean; - storageFull: boolean; -} - -export interface HostChoice { - id: HostId; - label: string; - lifetime: string; - /** Why it can't be used right now, or null. */ - disabled: string | null; - warning: string | null; -} - -/** Spec §4.6 / §4.10: server storage only when configured; a full store is shown disabled with the reason. */ -export function hostChoices(config: ServerConfig | null, opts: HostOptions, front: { catboxNeedsServer: boolean }): HostChoice[] { - const choices: HostChoice[] = []; - if (config?.storage) { - choices.push({ - id: "server", label: "This server", lifetime: HOSTS.server.lifetime(opts), - disabled: config.storageFull ? "Storage is full until older covers expire. Use litterbox or catbox." : null, warning: null, - }); - } - choices.push( - { - id: "litterbox", label: "litterbox", lifetime: HOSTS.litterbox.lifetime(opts), disabled: null, - warning: "A MangaBaka review can take longer than this; the links then stop working.", - }, - { - id: "catbox", label: "catbox.moe", lifetime: HOSTS.catbox.lifetime(opts), - disabled: front.catboxNeedsServer && !config ? "Needs this site's server, which isn't answering." : null, warning: null, - }, - { - id: "direct", label: "No rehosting", lifetime: HOSTS.direct.lifetime(opts), disabled: null, - warning: "Submits WEBTOON's own image URL. Covers only on archive.org can't use this.", - }, - ); - return choices; -} - -export const defaultHost = (choices: readonly HostChoice[]): HostId | null => choices.find((c) => !c.disabled)?.id ?? null; - -const STAGE_TEXT: Readonly> = { - series: "Reading the MangaBaka series", - source: "Reading the source page", - history: "Looking through the Wayback Machine", - inspect: "Checking images", - compare: "Comparing with MangaBaka", -}; - -export function stageText(stage: Stage, done?: number, total?: number): string { - return total ? `${STAGE_TEXT[stage]}… ${done ?? 0}/${total}` : `${STAGE_TEXT[stage]}…`; -} - -export const GRANULARITY_LABELS: Readonly> = { - 1: "every month", 3: "every 3 months", 6: "every 6 months", 12: "every year", -}; - -/** The evidence shown on each card (PRODUCT.md: show where and when each cover was seen). */ -export function cardFacts(card: ReviewCard): string[] { - const c = card.cover; - const facts = [`${c.width}×${c.height}`, `${Math.max(1, Math.round(c.bytes.byteLength / 1024))} KB`]; - if (c.shape === "tall") facts.push("landing-page art"); - if (c.candidate.current) { - facts.push("on the page now"); - } else { - const from = formatMonth(c.candidate.seenAt ?? ""); - const to = formatMonth(c.candidate.lastSeenAt ?? c.candidate.seenAt ?? ""); - facts.push(from === to ? `seen ${from}` : `seen ${from} – ${to}`); - } - if (!c.live) facts.push("only on archive.org"); - return facts; -} -``` - -- [ ] **Step 5: Write `src/web/template.ts`** (the markup both front ends mount; Task 20 restyles it but must keep every `id`) - -```ts -/** Mounted by the page (into #app) and by the userscript (into a shadow root). Keep every id: app.ts looks them up. */ -export const APP_HTML = /* html */ ` -
-
- - -
- - - -
-
- - - -
-
- -
-

- -
    -
    - - -
    `; -``` - -- [ ] **Step 6: Write `src/web/render.ts`** - -```ts -import type { ExistingImage } from "../core/covers"; -import type { PreviewResult } from "../core/mangabaka/types"; -import { CONTENT_RATINGS, type ImageType, type InspectedCover } from "../core/model"; -import type { Session } from "../core/session"; -import { cardFacts } from "./view-model"; - -const IMAGE_TYPES: readonly ImageType[] = ["volume", "volume_back", "chapter", "season", "banner", "audiobook", "other"]; - -function el( - tag: K, - props: Partial = {}, - ...children: (Node | string)[] -): HTMLElementTagNameMap[K] { - const node = Object.assign(document.createElement(tag), props); - node.append(...children); - return node; -} - -function picker(values: readonly T[], value: T, label: string, onChange: (v: T) => void): HTMLSelectElement { - const select = el("select", { ariaLabel: label }, ...values.map((v) => new Option(v, v, false, v === value))); - select.addEventListener("change", () => onChange(select.value as T)); - return select; -} - -/** One card per new cover: the evidence, and the fields the user may change. */ -export function renderCards(container: HTMLElement, session: Session, onEdit: () => void): void { - const suggested = session.result?.suggestedDefault?.sha256; - container.replaceChildren( - ...session.cards.map((card) => { - const sha = card.cover.sha256; - const edit = (patch: Parameters[1]) => { - session.update(sha, patch); - onEdit(); - }; - const include = el("input", { type: "checkbox", checked: card.selected, ariaLabel: "Include this cover" }); - include.addEventListener("change", () => edit({ selected: include.checked })); - const note = el("input", { type: "text", value: card.note, ariaLabel: "Note for the moderator" }); - note.addEventListener("change", () => edit({ note: note.value })); - const classes = ["mbseed-card", card.selected && "is-selected", sha === suggested && "is-default"].filter(Boolean).join(" "); - return el("article", { className: classes }, - el("label", { className: "mbseed-card-pick" }, - include, - el("img", { src: card.hostedUrl ?? card.cover.fetchedFrom, alt: card.note, loading: "lazy", referrerPolicy: "no-referrer" })), - el("ul", { className: "mbseed-card-facts" }, ...cardFacts(card).map((f) => el("li", { textContent: f }))), - el("div", { className: "mbseed-card-fields" }, - picker(IMAGE_TYPES, card.type, "Type", (type) => edit({ type })), - picker(CONTENT_RATINGS, card.contentRating, "Content rating", (contentRating) => edit({ contentRating })), - note), - card.hostedUrl ? el("a", { className: "mbseed-card-hosted", href: card.hostedUrl, target: "_blank", rel: "noopener", textContent: card.hostedUrl }) : "", - card.error ? el("p", { className: "mbseed-card-error", role: "alert", textContent: card.error }) : ""); - }), - ); -} - -export function renderPresent(container: HTMLElement, present: readonly { cover: InspectedCover; existing: ExistingImage }[]): void { - container.replaceChildren( - present.length - ? el("ul", {}, ...present.map((p) => el("li", {}, - el("img", { src: p.cover.fetchedFrom, alt: "", loading: "lazy", referrerPolicy: "no-referrer", width: 60 }), - ` ${p.cover.width}×${p.cover.height} — MangaBaka image #${p.existing.id} (${p.existing.language}, ${p.existing.type})`))) - : el("p", { textContent: "None of the found covers is on MangaBaka yet." }), - ); -} - -export function renderPreview(container: HTMLElement, preview: PreviewResult): void { - container.replaceChildren( - preview.has_changes - ? el("ul", { className: "mbseed-diff" }, ...preview.changes.map((c) => - el("li", { className: `is-${c.type}`, textContent: `${c.type}: ${c.label ?? c.identity ?? c.field}` }))) - : el("p", { textContent: "MangaBaka sees no changes: these covers are already there." }), - ); -} -``` - -- [ ] **Step 7: Write `src/web/app.ts`** - -```ts -import { LITTERBOX_TIMES, type HostId, type HostOptions, type LitterboxTime } from "../core/hosting/host"; -import { HOSTS } from "../core/hosting/registry"; -import { coverFormUrl } from "../core/mangabaka/client"; -import { CONTENT_RATINGS, GRANULARITIES, type ContentRating, type Granularity } from "../core/model"; -import { loadToken, saveToken, type KeyValueStore, type PreparedSubmission, type Session } from "../core/session"; -import { renderCards, renderPresent, renderPreview } from "./render"; -import { GRANULARITY_LABELS, defaultHost, hostChoices, stageText, type ServerConfig } from "./view-model"; - -export const USERHASH_KEY = "mbseed.catbox-userhash"; - -export interface AppOptions { - root: ParentNode; - session: Session; - store: KeyValueStore; - config: ServerConfig | null; - /** The page reaches catbox through our server; the userscript goes direct. */ - catboxNeedsServer: boolean; - /** Userscript: the MangaBaka series page it runs on. */ - seriesLink?: string; -} - -/** Wires APP_HTML to a Session. Nothing is uploaded or sent without a click (spec §1). */ -export function mountApp(o: AppOptions): void { - const q = (id: string): T => { - const found = o.root.querySelector(`#${id}`); - if (!found) throw new Error(`mbseed: #${id} is missing from the template`); - return found; - }; - const ui = { - form: q("run-form"), - mbLink: q("mb-link"), - sourceLink: q("source-link"), - historyOn: q("history-on"), - historyEvery: q("history-every"), - ratingAll: q("rating-all"), - run: q("run"), - cancel: q("cancel"), - skipHistory: q("skip-history"), - stage: q("stage"), - progress: q("progress"), - warnings: q("warnings"), - review: q("review"), - seriesTitle: q("series-title"), - matureWarning: q("mature-warning"), - linkWarning: q("link-warning"), - linkConfirm: q("link-confirm"), - makeDefault: q("make-default"), - defaultNote: q("default-note"), - selectedCount: q("selected-count"), - cards: q("cards"), - present: q("present-list"), - hosts: q("hosts"), - litterboxTime: q("litterbox-time"), - userhash: q("catbox-userhash"), - userhashRemember: q("catbox-remember"), - rehost: q("rehost"), - rehostStatus: q("rehost-status"), - manualText: q("manual-text"), - copy: q("copy"), - formLink: q("form-link"), - token: q("token"), - tokenRemember: q("token-remember"), - preview: q("preview"), - previewDiff: q("preview-diff"), - confirm: q("confirm"), - submitStatus: q("submit-status"), - }; - let running: AbortController | null = null; - let stopHistory: AbortController | null = null; - let prepared: PreparedSubmission | null = null; - - const fill = (select: HTMLSelectElement, values: readonly (string | number)[], label: (v: string) => string, selected: string) => - select.replaceChildren(...values.map(String).map((v) => new Option(label(v), v, false, v === selected))); - fill(ui.historyEvery, GRANULARITIES, (v) => GRANULARITY_LABELS[Number(v) as Granularity], "6"); - fill(ui.ratingAll, CONTENT_RATINGS, (v) => v, "safe"); - fill(ui.litterboxTime, LITTERBOX_TIMES, (v) => HOSTS.litterbox.lifetime({ litterboxTime: v as LitterboxTime }), "72h"); - if (o.seriesLink) { - ui.mbLink.value = o.seriesLink; - ui.mbLink.readOnly = true; - } - const savedToken = loadToken(o.store); - if (savedToken) { - ui.token.value = savedToken; - ui.tokenRemember.checked = true; - } - const savedUserhash = o.store.get(USERHASH_KEY); - if (savedUserhash) { - ui.userhash.value = savedUserhash; - ui.userhashRemember.checked = true; - } - - const hostOptions = (): HostOptions => ({ - litterboxTime: ui.litterboxTime.value as LitterboxTime, - catboxUserhash: ui.userhash.value.trim() || undefined, - }); - const selectedHost = (): HostId | null => - (o.root.querySelector('input[name="mbseed-host"]:checked')?.value as HostId | undefined) ?? null; - - function drawHosts(): void { - const choices = hostChoices(o.config, hostOptions(), { catboxNeedsServer: o.catboxNeedsServer }); - const keep = selectedHost(); - const current = choices.some((c) => c.id === keep && !c.disabled) ? keep : defaultHost(choices); - ui.hosts.replaceChildren(...choices.map((c) => { - const input = Object.assign(document.createElement("input"), { - type: "radio", name: "mbseed-host", value: c.id, checked: c.id === current, disabled: c.disabled !== null, - }); - const label = document.createElement("label"); - label.append(input, ` ${c.label} — ${c.lifetime}`); - const why = c.disabled ?? c.warning; - if (why) label.append(Object.assign(document.createElement("small"), { textContent: ` ${why}` })); - return label; - })); - } - - const status = (where: HTMLElement, text: string, isError = false) => { - where.textContent = text; - where.classList.toggle("is-error", isError); - }; - const errorText = (err: unknown) => (err instanceof Error ? err.message : String(err)); - - /** Any edit after a preview invalidates it: what is confirmed must be what was previewed. */ - function edited(): void { - prepared = null; - ui.confirm.hidden = true; - ui.previewDiff.replaceChildren(); - draw(); - } - - function draw(): void { - const r = o.session.result; - if (!r) return; - renderCards(ui.cards, o.session, edited); - renderPresent(ui.present, r.present); - const selected = o.session.selected().length; - ui.selectedCount.textContent = `${selected} of ${o.session.cards.length} new cover(s) selected`; - ui.linkWarning.hidden = r.linkMatches; - ui.makeDefault.checked = o.session.makeDefault; - ui.makeDefault.disabled = !r.suggestedDefault; - ui.defaultNote.textContent = r.suggestedDefault - ? "MangaBaka has one main cover per series, not per language: this replaces the current main cover for every language." - : "No current poster was found to suggest as the main cover."; - ui.manualText.value = o.session.manualText(); - const blocked = selected === 0 || (!r.linkMatches && !ui.linkConfirm.checked); - ui.rehost.disabled = blocked; - ui.preview.disabled = blocked; - } - - async function run(): Promise { - running = new AbortController(); - stopHistory = new AbortController(); - prepared = null; - ui.run.disabled = true; - ui.cancel.hidden = false; - ui.warnings.replaceChildren(); - ui.review.hidden = true; - ui.confirm.hidden = true; - ui.previewDiff.replaceChildren(); - ui.linkConfirm.checked = false; - status(ui.stage, "Starting…"); - try { - const r = await o.session.run( - { - mangabakaLink: ui.mbLink.value, - sourceLink: ui.sourceLink.value, - history: ui.historyOn.checked ? (Number(ui.historyEvery.value) as Granularity) : null, - }, - { - signal: running.signal, - stopHistory: stopHistory.signal, - onStage: (stage) => { - status(ui.stage, stageText(stage)); - ui.progress.removeAttribute("value"); - ui.skipHistory.hidden = stage !== "history"; - }, - onProgress: (stage, done, total) => { - status(ui.stage, stageText(stage, done, total)); - ui.progress.max = total; - ui.progress.value = done; - }, - onWarning: (message) => ui.warnings.append(Object.assign(document.createElement("li"), { textContent: message })), - }, - ui.ratingAll.value as ContentRating, - ); - status(ui.stage, `Found ${r.fresh.length} new cover(s); ${r.present.length} already on MangaBaka.`); - ui.progress.max = 1; - ui.progress.value = 1; - ui.seriesTitle.textContent = `${r.mbSeries.title} ← ${r.source.name} “${r.series.title}” (${r.series.language})`; - ui.matureWarning.hidden = !r.series.mature; - ui.formLink.href = coverFormUrl(r.seriesId); - ui.review.hidden = false; - draw(); - } catch (err) { - if (running?.signal.aborted) status(ui.stage, "Cancelled."); - else status(ui.stage, errorText(err), true); - } finally { - running = null; - stopHistory = null; - ui.run.disabled = false; - ui.cancel.hidden = true; - ui.skipHistory.hidden = true; - } - } - - function token(): string | null { - const t = ui.token.value.trim(); - saveToken(o.store, ui.tokenRemember.checked && t ? t : null); - return t || null; - } - - function rememberUserhash(): void { - const hash = ui.userhash.value.trim(); - if (ui.userhashRemember.checked && hash) o.store.set(USERHASH_KEY, hash); - else o.store.remove(USERHASH_KEY); - } - - ui.form.addEventListener("submit", (e) => { - e.preventDefault(); - void run(); - }); - ui.cancel.addEventListener("click", () => running?.abort()); - ui.skipHistory.addEventListener("click", () => stopHistory?.abort()); - ui.linkConfirm.addEventListener("change", draw); - ui.makeDefault.addEventListener("change", () => { - o.session.makeDefault = ui.makeDefault.checked; - edited(); - }); - ui.ratingAll.addEventListener("change", () => { - if (!o.session.result) return; - o.session.setRatingForAll(ui.ratingAll.value as ContentRating); - edited(); - }); - ui.litterboxTime.addEventListener("change", drawHosts); - ui.userhash.addEventListener("change", rememberUserhash); - ui.userhashRemember.addEventListener("change", rememberUserhash); - ui.tokenRemember.addEventListener("change", () => void token()); - - ui.rehost.addEventListener("click", async () => { - const host = selectedHost(); - if (!host) return status(ui.rehostStatus, "Pick a host first.", true); - ui.rehost.disabled = true; - status(ui.rehostStatus, "Uploading…"); - try { - const res = await o.session.rehost(host, hostOptions(), (done, total) => status(ui.rehostStatus, `Uploading… ${done}/${total}`)); - if (res.allFailed) status(ui.rehostStatus, `Every upload failed: ${res.failures[0]?.message ?? "unknown error"}`, true); - else status(ui.rehostStatus, `${res.hosted.size} rehosted${res.failures.length ? `, ${res.failures.length} failed (see the covers)` : ""}.`); - } catch (err) { - status(ui.rehostStatus, errorText(err), true); - } finally { - edited(); - } - }); - - ui.copy.addEventListener("click", async () => { - try { - await navigator.clipboard.writeText(ui.manualText.value); - ui.copy.textContent = "Copied"; - setTimeout(() => (ui.copy.textContent = "Copy all selected"), 1500); - } catch { - ui.manualText.select(); - } - }); - - ui.preview.addEventListener("click", async () => { - const t = token(); - if (!t) return status(ui.submitStatus, "Paste your MangaBaka personal access token first.", true); - ui.preview.disabled = true; - status(ui.submitStatus, "Asking MangaBaka for a preview…"); - try { - prepared = await o.session.preview(t); - renderPreview(ui.previewDiff, prepared.preview); - ui.confirm.hidden = false; - status(ui.submitStatus, "Check the changes. Confirming rehosts the covers and files the edit request for review."); - } catch (err) { - status(ui.submitStatus, errorText(err), true); - } finally { - ui.preview.disabled = false; - } - }); - - ui.confirm.addEventListener("click", async () => { - const t = token(); - const host = selectedHost(); - if (!t || !prepared || !host) return; - ui.confirm.disabled = true; - status(ui.submitStatus, "Rehosting and filing the edit request…"); - try { - const outcome = await o.session.submit(t, prepared, host, hostOptions()); - if (outcome.status === "saved") { - status(ui.submitStatus, `Filed edit request #${outcome.result.submission_id} (${outcome.result.status}). A MangaBaka moderator will review it.`); - prepared = null; - ui.confirm.hidden = true; - } else { - prepared = outcome.prepared; - renderPreview(ui.previewDiff, prepared.preview); - status(ui.submitStatus, outcome.reason, true); - } - } catch (err) { - status(ui.submitStatus, errorText(err), true); - } finally { - ui.confirm.disabled = false; - renderCards(ui.cards, o.session, edited); - } - }); - - drawHosts(); -} -``` - -- [ ] **Step 8: Write the page entry, HTML, minimal CSS and the text-import declaration** - -`src/web/main.ts`: -```ts -import { Session } from "../core/session"; -import { withPolicy } from "../core/transport"; -import { mountApp } from "./app"; -import { APP_HTML } from "./template"; -import { localStore, pageTransport } from "./transport"; -import type { ServerConfig } from "./view-model"; - -const app = document.getElementById("app")!; -app.innerHTML = APP_HTML; -const config = await fetch("/api/config") - .then((r) => (r.ok ? (r.json() as Promise) : null)) - .catch(() => null); -const store = localStore(); -const session = new Session({ - transport: withPolicy(pageTransport(location.origin)), - serverBase: location.origin, - catboxVia: "server", - store, - persistHostedUrls: false, -}); -mountApp({ root: document, session, store, config, catboxNeedsServer: true }); -``` - -`src/web/index.html`: -```html - - - - - - MangaBaka Seeder - - - - -
    -

    MangaBaka Seeder

    -

    Find every cover a WEBTOON series has used and turn them into a MangaBaka cover edit request.

    -
    - -
    - - -``` - -`src/web/styles.css` (functional only; Task 20 replaces it): -```css -.mbseed-app { font: 16px/1.5 system-ui, sans-serif; max-width: 72rem; margin: 0 auto; padding: 0 16px; } -.mbseed-app label { display: block; margin: 0.5rem 0; } -.mbseed-app .mbseed-inline { display: inline-block; margin-right: 1rem; } -.mbseed-app input:not([type]), .mbseed-app input[type="text"], .mbseed-app input[type="password"] { width: 100%; box-sizing: border-box; } -.mbseed-app .mbseed-cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(14rem, 1fr)); gap: 1rem; } -.mbseed-app .mbseed-card { border: 1px solid #ccc; padding: 0.5rem; opacity: 0.6; } -.mbseed-app .mbseed-card.is-selected { opacity: 1; } -.mbseed-app .mbseed-card.is-default { outline: 3px solid #2a7; } -.mbseed-app .mbseed-card img { display: block; max-width: 100%; max-height: 16rem; margin: 0.5rem auto; } -.mbseed-app .is-error, .mbseed-app .mbseed-card-error { color: #b00020; } -.mbseed-app .mbseed-warning { background: #fff4ce; padding: 0.5rem; } -.mbseed-app textarea { width: 100%; box-sizing: border-box; font-family: ui-monospace, monospace; } -.mbseed-app [hidden] { display: none !important; } -``` - -`src/types/text-imports.d.ts`: -```ts -// `import css from "./styles.css" with { type: "text" }` (Bun) — used by the userscript build. -declare module "*.css" { - const text: string; - export default text; -} -``` - -- [ ] **Step 9: Run tests, typecheck, build** - -Run: `bun test` → Expected: whole suite PASS (incl. the three new files). -Run: `bun run typecheck` → Expected: exit 0. -Run: `bun run build:web` → Expected: `dist/web/index.html` plus a hashed `.js` and `.css`; no errors. - -- [ ] **Step 10: Try it** - -Start the server as in Task 18 Step 3 (with storage), open `http://127.0.0.1:8787/` in a browser, and run Tower of God (`https://mangabaka.org/manhwa/9999/Tower-of-God`, `https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95`) with history **every year**. Expect: progress through the stages, ~10 snapshots, cards with images, the 720×1230 landing art unselected, the current poster outlined as main cover. Then **Rehost selected** with "This server" and check the cards show `http://127.0.0.1:8787/covers/…` links that open. Do **not** paste a token or press Preview. If you can't open a browser, ask the user to do this check and report what they see. - -- [ ] **Step 11: Commit** - -```bash -git add src/web src/types test/web-transport.test.ts test/view-model.test.ts test/template.test.ts -git commit -m "feat(web): working page (two links → review cards → rehost → copy or preview/confirm)" -``` - ---- - -### Task 20: Design the page with impeccable - -**Files:** -- Modify: `src/web/styles.css`, `src/web/template.ts` (structure and classes only — every `id` stays), `src/web/index.html`, `src/web/render.ts` (classes only) -- Modify: `DESIGN.md`, `PRODUCT.md` (if the design work sharpens it) - -- [ ] **Step 1: Design with impeccable `craft`** - -Invoke the `impeccable` skill in `craft` mode for the page. Brief it with: `PRODUCT.md`; the users (MangaBaka contributors, most without API access, so the copy-and-paste path is the main path); the flow (two links → progress that can take ~5 minutes with Wayback → review cards → rehost → copy or token flow); the evidence each card must show (`cardFacts`); and these constraints: -- Keep every `id` in `src/web/template.ts` and the behaviour of `src/web/app.ts`; `test/template.test.ts` must stay green. -- All rules are scoped under `.mbseed-app` (or `:host` for the panel) — the same CSS is injected into a shadow root on mangabaka.org by the userscript, where the panel should sit comfortably next to MangaBaka's own pages. -- No external fonts or scripts without the user's approval (no runtime dependencies). -- Layout works from 360 px wide up; light and dark (`prefers-color-scheme`). - -- [ ] **Step 2: Harden** - -Run impeccable `harden` on the page: long titles and notes, 0 / 1 / 40 cards, every warning shown at once, slow progress, error states for each stage, keyboard-only use, screen-reader labels. - -- [ ] **Step 3: Check** - -Run: `bun test` → PASS. `bun run typecheck` → exit 0. `bun run build:web` → builds. -Repeat Task 19 Step 10 and look at the page at 360 px and at desktop width, light and dark. Ask the user to look at it as well before moving on. - -- [ ] **Step 4: Record the design** - -Use the `impeccable-documenter` agent (or the impeccable skill's documenting step) to write `DESIGN.md` from what shipped: tokens, type, components, the shadow-root constraint. Remove the `TODO(maintainer)` line in `DESIGN.md`. - -- [ ] **Step 5: Commit** - -```bash -git add src/web DESIGN.md PRODUCT.md -git commit -m "feat(web): page design (impeccable craft + harden); record DESIGN.md" -``` - ---- - -## Part D — Userscript - -### Task 21: GM transport, store and the userscript build - -**Files:** -- Create: `src/userscript/gm.ts`, `src/userscript/gm-globals.d.ts`, `scripts/userscript-banner.ts`, `scripts/build-userscript.ts`, `test/gm.test.ts` - -**Interfaces:** -- Consumes: `Transport`, `TransportError`, `userAgentFor` (Task 3), `KeyValueStore` (Task 14). -- Produces: `interface GmResponse`, `interface GmRequest`, `type GmXhr`, `parseHeaders(raw): Headers`, `gmTransport(gm: GmXhr): Transport`, `gmStore(get, set, del): KeyValueStore`; `userscriptBanner(version: string, serverUrl?: string): string`; `bun run build:userscript` → `dist/web/mbseed.user.js`. - -- [ ] **Step 1: Write the failing tests** - -`test/gm.test.ts`: -```ts -import { describe, expect, test } from "bun:test"; -import { TOOL_USER_AGENT, TransportError } from "../src/core/transport"; -import { gmStore, gmTransport, parseHeaders, type GmRequest } from "../src/userscript/gm"; -import { userscriptBanner } from "../scripts/userscript-banner"; - -function fakeGm(respond: (req: GmRequest) => void) { - const requests: GmRequest[] = []; - const gm = (req: GmRequest) => { - requests.push(req); - queueMicrotask(() => respond(req)); - return { abort: () => req.onabort() }; - }; - return { gm, requests }; -} - -describe("gmTransport", () => { - test("wraps GM_xmlhttpRequest into a Response", async () => { - const body = new TextEncoder().encode("").buffer; - const { gm, requests } = fakeGm((r) => - r.onload({ status: 200, statusText: "OK", responseHeaders: "Content-Type: text/html\r\nX-Two: a: b\r\n", response: body })); - const res = await gmTransport(gm).fetch("https://www.webtoons.com/-/-/-/list?title_no=95"); - expect(res.status).toBe(200); - expect(res.headers.get("content-type")).toBe("text/html"); - expect(await res.text()).toBe(""); - expect(requests[0]).toMatchObject({ method: "GET", responseType: "arraybuffer", anonymous: true }); - expect(requests[0]!.headers!["user-agent"]).toMatch(/^Mozilla/); - }); - - test("Wayback gets the honest User-Agent", async () => { - const { gm, requests } = fakeGm((r) => r.onload({ status: 200, statusText: "OK", responseHeaders: "", response: new ArrayBuffer(0) })); - await gmTransport(gm).fetch("https://web.archive.org/cdx/search/cdx?url=x"); - expect(requests[0]!.headers!["user-agent"]).toBe(TOOL_USER_AGENT); - }); - - test("network errors and timeouts reject with TransportError; odd statuses become 502", async () => { - await expect(gmTransport(fakeGm((r) => r.onerror({})).gm).fetch("https://x/")).rejects.toBeInstanceOf(TransportError); - await expect(gmTransport(fakeGm((r) => r.ontimeout()).gm).fetch("https://x/")).rejects.toBeInstanceOf(TransportError); - const res = await gmTransport(fakeGm((r) => r.onload({ status: 0, statusText: "", responseHeaders: "", response: null })).gm).fetch("https://x/"); - expect(res.status).toBe(502); - }); - - test("an abort signal aborts the request", async () => { - const controller = new AbortController(); - const { gm } = fakeGm(() => controller.abort()); - await expect(gmTransport(gm).fetch("https://x/", { signal: controller.signal })).rejects.toThrow(); - }); -}); - -test("parseHeaders keeps colons inside values", () => { - expect(parseHeaders("A: 1\r\nB: x: y\r\nbroken\r\n").get("b")).toBe("x: y"); -}); - -test("gmStore maps to GM_getValue / GM_setValue / GM_deleteValue", () => { - const values = new Map(); - const store = gmStore((k) => values.get(k), (k, v) => void values.set(k, v), (k) => void values.delete(k)); - store.set("a", "1"); - expect(store.get("a")).toBe("1"); - store.remove("a"); - expect(store.get("a")).toBeNull(); -}); - -test("banner: matches mangabaka.org, grants GM APIs, connects to every host it needs", () => { - const banner = userscriptBanner("0.1.0", "https://mbseed.example.org"); - expect(banner).toStartWith("// ==UserScript==\n"); - expect(banner).toContain("// @version 0.1.0\n"); - expect(banner).toContain("// @match https://mangabaka.org/*\n"); - for (const grant of ["GM_xmlhttpRequest", "GM_getValue", "GM_setValue", "GM_deleteValue"]) expect(banner).toContain(`// @grant ${grant}\n`); - for (const host of ["api.mangabaka.org", "webtoons.com", "pstatic.net", "web.archive.org", "catbox.moe", "mbseed.example.org"]) { - expect(banner).toContain(`// @connect ${host}\n`); - } - expect(banner).toEndWith("// ==/UserScript==\n\n"); - expect(userscriptBanner("0.1.0")).not.toContain("mbseed.example.org"); -}); -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bun test test/gm.test.ts` -Expected: FAIL — modules not found. - -- [ ] **Step 3: Write `src/userscript/gm.ts`** - -```ts -import type { KeyValueStore } from "../core/session"; -import { TransportError, userAgentFor, type Transport } from "../core/transport"; - -export interface GmResponse { - status: number; - statusText: string; - responseHeaders: string; - response: ArrayBuffer | null; -} - -export interface GmRequest { - method: string; - url: string; - headers?: Record; - data?: unknown; - responseType: "arraybuffer"; - anonymous?: boolean; - timeout?: number; - onload(r: GmResponse): void; - onerror(r: unknown): void; - ontimeout(): void; - onabort(): void; -} - -export type GmXhr = (req: GmRequest) => { abort(): void }; - -export function parseHeaders(raw: string): Headers { - const headers = new Headers(); - for (const line of raw.split(/\r?\n/)) { - const i = line.indexOf(":"); - if (i <= 0) continue; - try { - headers.append(line.slice(0, i).trim(), line.slice(i + 1).trim()); - } catch { - // skip header names the Headers API rejects - } - } - return headers; -} - -const NULL_BODY = new Set([204, 205, 304]); - -/** GM_xmlhttpRequest is not bound by CORS (spec §4.2), so the userscript needs no proxy. */ -export function gmTransport(gm: GmXhr): Transport { - return { - fetch(url, init = {}) { - return new Promise((resolve, reject) => { - if (init.signal?.aborted) return reject(init.signal.reason); - const request = gm({ - method: (init.method ?? "GET").toUpperCase(), - url, - headers: { "user-agent": userAgentFor(url), ...init.headers }, - data: init.body ?? undefined, - responseType: "arraybuffer", - anonymous: true, - timeout: init.timeoutMs, - onload: (r) => { - const status = r.status >= 200 && r.status <= 599 ? r.status : 502; - resolve(new Response(NULL_BODY.has(status) ? null : r.response, { - status, statusText: r.statusText, headers: parseHeaders(r.responseHeaders), - })); - }, - onerror: () => reject(new TransportError(`Network error: ${url}`, url)), - ontimeout: () => reject(new TransportError(`Timed out: ${url}`, url)), - onabort: () => reject(init.signal?.reason ?? new DOMException("Aborted", "AbortError")), - }); - init.signal?.addEventListener("abort", () => request.abort(), { once: true }); - }); - }, - }; -} - -export function gmStore( - get: (key: string) => string | undefined, - set: (key: string, value: string) => void, - del: (key: string) => void, -): KeyValueStore { - return { get: (k) => get(k) ?? null, set, remove: del }; -} -``` - -`src/userscript/gm-globals.d.ts`: -```ts -import type { GmXhr } from "./gm"; - -declare global { - const GM_xmlhttpRequest: GmXhr; - function GM_getValue(key: string): string | undefined; - function GM_setValue(key: string, value: string): void; - function GM_deleteValue(key: string): void; -} -``` - -- [ ] **Step 4: Write the banner and the build script** - -`scripts/userscript-banner.ts`: -```ts -/** Tampermonkey/Violentmonkey metadata. @connect lists every host GM_xmlhttpRequest may reach. */ -export function userscriptBanner(version: string, serverUrl?: string): string { - const connects = ["api.mangabaka.org", "webtoons.com", "pstatic.net", "web.archive.org", "catbox.moe"]; - if (serverUrl) connects.push(new URL(serverUrl).hostname); - const lines = [ - "// ==UserScript==", - "// @name MangaBaka Seeder", - "// @namespace https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder", - `// @version ${version}`, - "// @description Find every WEBTOON cover of a series and turn them into a MangaBaka cover edit request", - "// @match https://mangabaka.org/*", - "// @grant GM_xmlhttpRequest", - "// @grant GM_getValue", - "// @grant GM_setValue", - "// @grant GM_deleteValue", - ...connects.map((host) => `// @connect ${host}`), - "// @run-at document-idle", - "// ==/UserScript==", - ]; - return `${lines.join("\n")}\n\n`; -} -``` - -`scripts/build-userscript.ts`: -```ts -// bun run build:userscript → dist/web/mbseed.user.js (served by our server at /mbseed.user.js). -// Set MBSEED_PUBLIC_URL to add our server to @connect. -import pkg from "../package.json"; -import { userscriptBanner } from "./userscript-banner"; - -const result = await Bun.build({ entrypoints: ["src/userscript/main.ts"], target: "browser", format: "iife" }); -if (!result.success) { - for (const log of result.logs) console.error(log); - process.exit(1); -} -const code = await result.outputs[0]!.text(); -const out = userscriptBanner(pkg.version, process.env.MBSEED_PUBLIC_URL || undefined) + code; -await Bun.write("dist/web/mbseed.user.js", out); -console.log(`wrote dist/web/mbseed.user.js (${Math.round(out.length / 1024)} KB)`); -``` - -- [ ] **Step 5: Run tests** - -Run: `bun test test/gm.test.ts` → Expected: all PASS. -Run: `bun run typecheck` → Expected: exit 0. (`build:userscript` needs `src/userscript/main.ts` from Task 22.) - -- [ ] **Step 6: Commit** - -```bash -git add src/userscript/gm.ts src/userscript/gm-globals.d.ts scripts/userscript-banner.ts scripts/build-userscript.ts test/gm.test.ts -git commit -m "feat(userscript): GM_xmlhttpRequest transport, GM storage, metadata banner and build" -``` - ---- - -### Task 22: The panel on mangabaka.org - -**Files:** -- Create: `src/userscript/main.ts` -- Modify: `src/web/styles.css` (panel chrome: `.mbseed-panel`, `.mbseed-panel-bar`, `.mbseed-settings`), `AGENTS.md`, `docs/PERMISSIONS.md`, `.claude/settings.json`, `.gemini/settings.json` - -**Not in this task (deferred, see `docs/TODO.md`):** filling MangaBaka's cover form from the userscript (spec §4.9, last bullet). It needs the form's markup, which is only visible when logged in, and whether the form accepts programmatic input is unverified (spec §7.5). The panel offers the same copy-and-paste text as the page instead. - -**Interfaces:** -- Consumes: `mountApp`, `APP_HTML`, `ServerConfig` (Task 19), `Session` (Task 14), `withPolicy` (Task 3), `gmTransport`, `gmStore` (Task 21), styles as text. - -- [ ] **Step 1: Write `src/userscript/main.ts`** (DOM glue; verified by building and by the user in Step 4) - -```ts -import { Session } from "../core/session"; -import { withPolicy } from "../core/transport"; -import { mountApp } from "../web/app"; -import styles from "../web/styles.css" with { type: "text" }; -import { APP_HTML } from "../web/template"; -import type { ServerConfig } from "../web/view-model"; -import { gmStore, gmTransport } from "./gm"; - -const SERVER_KEY = "mbseed.server"; -/** MangaBaka series pages: ///. */ -const SERIES_PATH = /^\/(manga|manhwa|manhua|novel|oel|other)\/\d+(\/|$)/; - -const store = gmStore(GM_getValue, GM_setValue, GM_deleteValue); -const transport = withPolicy(gmTransport(GM_xmlhttpRequest)); -let panel: HTMLElement | null = null; -let panelPath: string | null = null; - -async function loadConfig(server: string | null): Promise { - if (!server) return null; - try { - const res = await transport.fetch(`${server}/api/config`); - return res.ok ? ((await res.json()) as ServerConfig) : null; - } catch { - return null; - } -} - -function closePanel(): void { - panel?.remove(); - panel = null; - panelPath = null; -} - -async function openPanel(): Promise { - closePanel(); - panelPath = location.pathname; - const server = store.get(SERVER_KEY); - panel = document.createElement("mbseed-panel"); - panel.style.cssText = "position:fixed;top:16px;right:16px;bottom:16px;width:min(40rem,calc(100vw - 32px));overflow:auto;z-index:2147483647;"; - const shadow = panel.attachShadow({ mode: "open" }); - shadow.innerHTML = ` -
    -
    Seed covers
    -
    Settings - - Only needed to rehost on that server. litterbox and catbox work without one. -
    - ${APP_HTML} -
    `; - document.body.append(panel); - - const serverInput = shadow.querySelector("#mbseed-server")!; - serverInput.value = server ?? ""; - serverInput.addEventListener("change", () => { - const value = serverInput.value.trim().replace(/\/+$/, ""); - if (value) store.set(SERVER_KEY, value); - else store.remove(SERVER_KEY); - void openPanel(); - }); - shadow.querySelector("#mbseed-close")!.addEventListener("click", closePanel); - - const session = new Session({ transport, serverBase: server, catboxVia: "direct", store, persistHostedUrls: true }); - mountApp({ root: shadow, session, store, config: await loadConfig(server), catboxNeedsServer: false, seriesLink: location.origin + location.pathname }); -} - -const launcher = document.createElement("button"); -launcher.type = "button"; -launcher.textContent = "Seed covers"; -launcher.style.cssText = "position:fixed;right:16px;bottom:16px;z-index:2147483646;padding:8px 14px;border-radius:999px;"; -launcher.addEventListener("click", () => void openPanel()); -document.body.append(launcher); - -/** MangaBaka is a single-page app: follow client-side navigation. */ -function sync(): void { - const onSeries = SERIES_PATH.test(location.pathname); - launcher.hidden = !onSeries; - if (panel && panelPath !== location.pathname) closePanel(); -} -sync(); -setInterval(sync, 1000); -``` - -- [ ] **Step 2: Panel chrome styles** - -Append to `src/web/styles.css` styles for `.mbseed-panel` (background, border, shadow, padding, its own colour scheme so MangaBaka's page styles don't leak in — shadow DOM already isolates selectors), `.mbseed-panel-bar` (sticky header with the close button) and `.mbseed-settings`. Follow the tokens in `DESIGN.md` from Task 20. - -- [ ] **Step 3: Build and check the output** - -Run: `bun run typecheck` → exit 0. -Run: `bun run build` → Expected: `wrote dist/web/mbseed.user.js (… KB)` and the web build. -Run: `head -20 dist/web/mbseed.user.js` → Expected: the banner, then `(() => {`. -Run: `grep -c "^import " dist/web/mbseed.user.js` → Expected: `0`. -Run: `bun test` → Expected: PASS. - -- [ ] **Step 4: Ask the user to try it** (an agent can't drive Tampermonkey) - -Ask the user to install `http://127.0.0.1:8787/mbseed.user.js` (server from Task 18 running, after `bun run build`) in Tampermonkey or Violentmonkey, open `https://mangabaka.org/manhwa/9999/Tower-of-God`, click **Seed covers**, paste `https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95`, untick history, run, pick **No rehosting**, and **Copy all selected** — without a token and without uploading. Expected: the panel appears only on series pages, follows navigation, finds the current covers, and the copied text lists pstatic URLs. Fix what they report before going on. - -- [ ] **Step 5: Record and commit** - -`AGENTS.md` Commands: add `bun run build` (page + userscript). `docs/PERMISSIONS.md`: `| bun run build | worked |`, add it to Allow; `.claude/settings.json` allow `"Bash(bun run build)", "Bash(bun run build:web)", "Bash(bun run build:userscript)"`; `.gemini/settings.json` the same three as `run_shell_command(...)`. -`docs/TODO.md`: add under Next: "Userscript: fill MangaBaka's cover form for users without a token (spec §4.9) — needs the logged-in form's markup and a check that it accepts programmatic input (spec §7.5)." - -```bash -git add src/userscript/main.ts src/web/styles.css AGENTS.md docs/PERMISSIONS.md docs/TODO.md .claude/settings.json .gemini/settings.json -git commit -m "feat(userscript): Seed covers panel on MangaBaka series pages" -``` - ---- - -## Part E — Finish - -### Task 23: Smoke test, docs, review, hand-off - -**Files:** -- Modify: `README.md`, `docs/ARCHITECTURE.md`, `docs/TODO.md`, `docs/DECISIONS.md` (if anything new), `docs/specs/001-webtoons-covers.md` (only facts learned while building) -- Delete (before the merge request): `docs/plans/001-webtoons-covers.md` - -- [ ] **Step 1: Full verification** - -Run: `bun install --frozen-lockfile && bun test && bun run typecheck && bun run build` -Expected: all green; note the test count. - -- [ ] **Step 2: Manual smoke test with the user** (spec §6) - -With the server running locally (Task 18 Step 3), the user runs the page against Tower of God (`9999` / `title_no=95`) with history every 6 months, rehosts to **this server**, copies the text, and opens two of the `/covers/…` links. No token, no litterbox/catbox upload (those need the user's go-ahead — AGENTS.md). Record anything surprising in `docs/TODO.md` (Known issues) or fix it with a test first. - -- [ ] **Step 3: Docs duties** - -- `README.md`: status → "v0.1 — works; not deployed yet". Add **Running it** (`bun install`, `bun run build`, `bun run start`, env vars → `docs/ARCHITECTURE.md`) and **Userscript** (`/mbseed.user.js`, the optional server setting). Keep the existing explanation. -- `docs/ARCHITECTURE.md`: Map rows for `src/web/` and `src/userscript/` as built; remove "(planned)"; Data flow names the real modules (`runCovers` → `Session`); Known rough edges → link the TODO items. -- `docs/TODO.md`: tick done items under Next; add: - - "Capture MangaBaka snapshot / preview / save responses once someone with API access runs the token flow; replace the inline OpenAPI-shaped bodies in `test/mangabaka-client.test.ts` and `test/session.test.ts`." - - "Check whether MangaBaka's `hashes.sha256` matches the original bytes (spec §7.7): compare a known submitted image." - - "Deploy (needs the user: server, domain, `MBSEED_PUBLIC_URL`, covers folder, cap)." -- `docs/DECISIONS.md`: anything decided during implementation that isn't recorded yet. - -- [ ] **Step 4: Review** - -Use `superpowers:requesting-code-review` for the whole branch against `main`, with the spec and this plan. Fix findings test-first; record deliberately skipped ones in `docs/TODO.md`. - -- [ ] **Step 5: Remove the plan and commit** - -```bash -git rm docs/plans/001-webtoons-covers.md -git add README.md docs -git commit -m "docs: v0.1 docs; remove the finished plan (the spec is the record)" -``` -`git status` → clean. - -- [ ] **Step 6: Hand off** - -Ask the user before pushing (`git push` is in the Ask tier) and before opening a merge request on Gitea. Report: what was built, test count, what was verified by hand and by whom, the decisions made during implementation (from `docs/DECISIONS.md`), and what is deferred (`docs/TODO.md`).