From dd2d5a94bc347f9cbf3bb70e27614488c4dc32a7 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Wed, 14 Oct 2026 22:10:20 +0200 Subject: [PATCH] docs: approve spec 001; self-hosted Bun server, defer password gate The user signed off the spec. The server runs self-hosted under Bun; the password gate and the Cloudflare Workers entry stay designed but deferred until abuse or traffic calls for them. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 14 +++--- README.md | 4 +- docs/ARCHITECTURE.md | 10 ++--- docs/DECISIONS.md | 13 ++++++ docs/TODO.md | 22 +++++---- docs/specs/001-webtoons-covers.md | 74 ++++++++++++++++++------------- 6 files changed, 80 insertions(+), 57 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f780fa7..ca76e8b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,8 +1,8 @@ # mangabaka-seeder Seeds cover edits into MangaBaka from a Webtoons series. TypeScript on **Bun** (not Node): -a pure core, a static page served by a small server (Cloudflare Worker or self-hosted Bun), and a userscript. No code yet — the -approved design is `docs/specs/001-webtoons-covers.md`. +a pure core, a static page served by a small self-hosted Bun server, and a userscript. +No code yet — the approved design is `docs/specs/001-webtoons-covers.md`. `AGENTS.md` is the single source of truth for agents here. `CLAUDE.md` and `GEMINI.md` only point here. Deeper context on demand: `docs/ARCHITECTURE.md`, `docs/DECISIONS.md`, `docs/TODO.md`. @@ -30,10 +30,10 @@ Use impeccable for the static page (`src/web/`) and the userscript panel — `cr ## Rules -- `src/core/` runs in a browser, a Worker, a userscript and Bun: no DOM, no `Bun.*`, no `node:*`, +- `src/core/` runs in a browser, a userscript and Bun: no DOM, no `Bun.*`, no `node:*`, no `GM_*`. All network goes through the injected `Transport`; hashing uses `crypto.subtle`. - `core/` never imports from `web/`, `server/`, `userscript/` or `cli/`. `server/app.ts` uses only - web-standard APIs so it runs on Workers and `Bun.serve`; `worker.ts`/`bun.ts` stay thin adapters. + web-standard APIs (a Workers entry may come later); `bun.ts` stays a thin adapter. - The MangaBaka token goes straight to `api.mangabaka.org` (CORS is open) — never through our server. - MangaBaka `images` **replaces the whole set**: a save body must include every existing image. Saves always use `save_mode: "review"`. @@ -41,7 +41,7 @@ Use impeccable for the static page (`src/web/`) and the userscript panel — `cr it files a real edit request. Tests use the fake transport; tests never touch the network. - Fixtures in `test/fixtures/` are captured from real responses, never hand-written. - `/api/fetch` only proxies allowlisted hosts; `/api/upload` takes a source URL, never raw bytes. -- When `MBSEED_PASSWORD` is set the gate covers **every** route except `/login` — static assets too. +- The password gate (spec §4.11) and the Cloudflare entry are deferred: don't build them unasked. - Webtoons image URLs: strip the `?type=` query and use the `swebtoon-phinf.pstatic.net` host: ```ts @@ -53,7 +53,7 @@ Use impeccable for the static page (`src/web/`) and the userscript panel — `cr ## Architecture src/core/ -> pure pipeline: sources, Wayback history, inspect, dedupe, seed, MangaBaka client -src/web/ -> static page · src/server/ -> gate, assets, `/api/fetch`, `/api/upload` (Workers or Bun) +src/web/ -> static page · src/server/ -> assets, `/api/fetch`, `/api/upload` (`Bun.serve`) src/core/hosting/ -> server storage (30 days), litterbox (≤72h), catbox (permanent), direct src/userscript/ -> panel on mangabaka.org, `GM_xmlhttpRequest` · src/cli/ -> later @@ -64,7 +64,7 @@ Full picture: `docs/ARCHITECTURE.md` and the spec — read them before your firs Never commit `.env*`, `.dev.vars`, tokens, passwords, S3 keys or Cloudflare credentials. Never force-push, rewrite pushed history, or commit directly to `main`. -Ask first: new runtime dependencies · `wrangler deploy` or any Cloudflare/R2/S3 configuration · +Ask first: new runtime dependencies · deploying, or any server/bucket configuration · real uploads to litterbox/catbox · anything sent to MangaBaka with a real token · anything destructive (`rm -rf`, `git reset --hard`) · work the approved plan does not cover. diff --git a/README.md b/README.md index abae7ef..bb95214 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,8 @@ poster becomes the main (default) cover for the edition's language. - **Web page** (primary). Open the page, paste the two links, review the covers. A small server behind the page reads Webtoons and the Wayback - Machine for you, because browsers can't. It runs on Cloudflare Workers or on - your own machine with Bun, and can be locked with a password. + Machine for you, because browsers can't. It runs on your own machine with + Bun. - **Userscript.** Install `mbseed.user.js` in Tampermonkey or Violentmonkey. On a MangaBaka series page it adds a **Seed covers** panel; paste the Webtoons link. It reads Webtoons directly from your browser, and with diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index dbdc45a..863c5ef 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -5,16 +5,15 @@ How `mangabaka-seeder` is built today. Update in the same commit as any structur ## Overview No code exists yet. The design is in [`specs/001-webtoons-covers.md`](specs/001-webtoons-covers.md): one pure TypeScript core (`src/core/`) that turns a MangaBaka series link and a Webtoons series -link into a cover submission, used by three front ends — a static page served by a small, -optionally password-gated server that runs on Cloudflare Workers or self-hosted under Bun -(primary), a userscript, and later a Bun CLI. Replace this section with reality as modules land. +link into a cover submission, used by three front ends — a static page served by a small +self-hosted Bun server (primary), a userscript, and later a Bun CLI. Replace this section with reality as modules land. ## Map (planned — spec §4) | Path | Responsibility | |---|---| | `src/core/` | Pure pipeline: sources, Wayback history, image inspection, dedupe, seed, MangaBaka client. No DOM, Bun, Node or `GM_*` APIs. | | `src/web/` | Static page UI. | -| `src/server/` | Portable `fetch` handler (`app.ts`) + password gate; entries `worker.ts` (Cloudflare, R2) and `bun.ts` (self-hosted, S3). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`. | +| `src/server/` | Web-standard `fetch` handler (`app.ts`) run by `bun.ts` (`Bun.serve`, behind our reverse proxy). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`. Gate and Workers entry deferred. | | `src/userscript/` | Userscript on mangabaka.org; network via `GM_xmlhttpRequest`. | | `src/cli/` | Later: Bun CLI. | | `test/fixtures/` | Saved real responses; tests never hit the network. | @@ -29,13 +28,12 @@ sha256) → dedupe against MangaBaka → seed → user reviews → rehost (expli | `api.mangabaka.org` | direct (CORS open) | Nothing works. | | Webtoons HTML / `swebtoon-phinf.pstatic.net` | server proxy or `GM_xmlhttpRequest` | Nothing works. | | Wayback (`web.archive.org`) | same | Warning; current covers only. | -| Server storage (R2 binding or S3 bucket) | server | `server` host hidden; litterbox/catbox/direct still work. | +| Server storage (S3-compatible bucket) | server | `server` host hidden; litterbox/catbox/direct still work. | | litterbox / catbox | browser directly (litterbox), server or `GM_xmlhttpRequest` (catbox) | That host fails; others still work. | ## Invariants - `src/core/` never imports from `web/`, `server/`, `userscript/` or `cli/`. - The MangaBaka token is only ever sent to `api.mangabaka.org`, never through our server. -- With `MBSEED_PASSWORD` set, every route except `/login` requires a session (or Bearer password). - Nothing is uploaded or submitted without an explicit user action; saves use `save_mode: "review"`. - `/api/fetch` only fetches allowlisted hosts; `/api/upload` only takes a source URL, never bytes. diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index ead8d89..3da62ed 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -16,6 +16,19 @@ the repo every session — measurably slower and more tokens). **Consequences:** Rule changes happen in one file; pointer files must stay thin. Plans are working artefacts: kept after merge they go stale and contradict the code, so the spec is the record. +## 2026-10-14 — Spec signed off; self-hosted only; password gate deferred +**Status:** accepted (narrows the entry below) +**Context:** The user signed off spec 001 and said the app will run self-hosted, and that +password-gating should wait: "if it gets abused or we see too big amounts of traffic, we will deal +with it then." +**Decision:** First version runs `server/app.ts` under `Bun.serve` on our own machine behind our +reverse proxy. No gate, no Cloudflare entry; both stay designed in the spec (§4.10, §4.11) and are +listed under Someday in `docs/TODO.md`. The handler stays web-standard so either can be added later. +**Alternatives:** Build the gate now (cost without a demonstrated need). +**Consequences:** The site is open: the host allowlist, the 20 MB cap and the per-IP rate limit are +the only protections. The rate limit needs the real client IP from our reverse proxy +(`MBSEED_TRUST_PROXY`). + ## 2026-10-14 — Also support catbox/litterbox; optional password gate; self-hostable server **Status:** accepted (extends the two entries below) **Context:** The user asked for catbox/litterbox support as well, and for a way to password-gate the diff --git a/docs/TODO.md b/docs/TODO.md index ff1bdc6..f2ec0a0 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -5,12 +5,11 @@ Small, specific, independently actionable. Remove only when actually done. Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). ## Now -- [ ] User signs off the revised spec `docs/specs/001-webtoons-covers.md` (status: draft). - [ ] Write the implementation plan with `superpowers:writing-plans` into `docs/plans/001-webtoons-covers.md`, and **show it to the user before implementing** (the user asked for this in the design session). ## Next (rough order; the plan will refine it) -- [ ] **Scaffold with Bun.** `package.json`, `bun.lock`, TypeScript, `wrangler`, `.gitignore`. +- [ ] **Scaffold with Bun.** `package.json`, `bun.lock`, TypeScript, `.gitignore` (already exists). Scripts for test, typecheck, build (page, Worker, userscript). Then fill the Commands in `AGENTS.md` and the Verified commands in `docs/PERMISSIONS.md` from runs that worked. - [ ] **Capture fixtures from real responses:** Webtoons pages (Tower of God, crow-time canvas, @@ -23,23 +22,19 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). registry, webtoons (urls, parse, index), history/wayback, images, covers, mangabaka client, hosting (server, litterbox, catbox, direct), seed. - [ ] `server/`: `app.ts` (assets, `/api/config`, `/api/fetch` allowlist, `/api/upload` store|catbox, - per-IP rate limit), `gate.ts` (login, signed cookie, Bearer), storage (R2 binding, S3), - entries `worker.ts` and `bun.ts`. -- [ ] `web/`: login page, two-link form, review cards, host picker with lifetimes, rehost, copy, + per-IP rate limit), storage, entry `bun.ts`. Decide in the plan: S3-compatible bucket or + local directory with a 30-day sweep (spec §7.7). +- [ ] `web/`: two-link form, review cards, host picker with lifetimes, rehost, copy, token flow (preview → confirm → save). - [ ] `userscript/`: panel on mangabaka.org series pages, `GM_xmlhttpRequest` transport, token in - `GM_setValue`, direct litterbox/catbox uploads, optional server URL + password, + `GM_setValue`, direct litterbox/catbox uploads, optional server URL, build to `mbseed.user.js` with metadata banner. - [ ] Manual smoke test against Tower of God (9999 / title_no 95). - [ ] Update README, `docs/ARCHITECTURE.md` and `PRODUCT.md`/`DESIGN.md` to match reality. ## Verify with real services (once possible) -- [ ] Do Webtoons and Wayback serve requests from Cloudflare Worker IPs? -- [ ] Does catbox accept uploads from Cloudflare Worker IPs? -- [ ] Workers static assets: confirm the config key that runs the Worker before assets - (`run_worker_first`) so the gate covers them. -- [ ] Choose the deployment: Cloudflare, our own server, or both; and the gate (password or - reverse-proxy auth). +- [ ] Which of our servers and domain hosts it; which bucket (or local dir) backs `server` storage. +- [ ] Does catbox accept uploads from our server? - [ ] Does MangaBaka accept our storage URLs, litterbox/catbox URLs, and `swebtoon-phinf.pstatic.net` (`direct`)? - [ ] Does MangaBaka copy images on submission or on approval? @@ -48,6 +43,9 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). - [ ] Does the MangaBaka cover form accept fields filled by a userscript? ## Someday +- [ ] Password gate (spec §4.11) — only if the open site gets abused or traffic grows too large. +- [ ] Cloudflare Workers entry (`server/worker.ts`, R2 binding). If added: Workers serve static + assets before the Worker unless `run_worker_first` is set — needed for the gate to cover them. - [ ] Bun CLI on the same core. - [ ] More sources: Tapas, Tappytoon, Lezhin, Manta. - [ ] Following Webtoons language editions of the same series. diff --git a/docs/specs/001-webtoons-covers.md b/docs/specs/001-webtoons-covers.md index 356029a..468f8b0 100644 --- a/docs/specs/001-webtoons-covers.md +++ b/docs/specs/001-webtoons-covers.md @@ -1,5 +1,5 @@ --- -status: draft +status: approved --- # 001 — Webtoons covers @@ -23,6 +23,10 @@ the runtime is Bun, and images go to a temporary host with a 30-day expiry. Revised again the same day: catbox.moe and litterbox are also supported as image hosts, the server can be password-gated, and the same server code runs on Cloudflare Workers or under Bun on our own machine. +Signed off by the user 2026-10-14, with two changes applied here: the server +is **self-hosted** (Bun), and the **password gate is deferred** until abuse or +traffic calls for it (§4.11 keeps the design). The Cloudflare entry is +deferred too; the handler stays web-standard so it can be added later. --- @@ -39,11 +43,10 @@ on Cloudflare Workers or under Bun on our own machine. 2. a **userscript** (Tampermonkey/Violentmonkey) on mangabaka.org, which needs no proxy because `GM_xmlhttpRequest` is not bound by CORS. A Bun CLI on the same core comes later. -- **One server, two deployments.** The server is a single `fetch(Request) → - Response` handler that runs on Cloudflare Workers or under `Bun.serve` on - our own machine (e.g. behind our own reverse proxy) (§4.10). -- **Lockable.** An optional password gates the whole site, the page and the API - alike (§4.11). +- **Self-hosted server.** The server is a single web-standard `fetch(Request) → + Response` handler run by `Bun.serve` on our own machine, behind our reverse + proxy (§4.10). Because it is web-standard, a Cloudflare Workers entry can be + added later without touching it. - **Choice of image host:** our own storage (30-day expiry), litterbox (up to 72h), catbox.moe (permanent), or no rehosting (§4.6). - **Find historical covers**, not just the current one. @@ -66,7 +69,10 @@ on Cloudflare Workers or under Bun on our own machine. - Running our own permanent image host. Our storage only needs to outlive the MangaBaka review, so it expires after 30 days; catbox.moe is the option for anyone who wants permanent links (§4.6). -- User accounts. The gate is one shared password, not per-user logins. +- Access control, for now. The site is open; the password gate (§4.11) is + designed but deferred until abuse or traffic calls for it. +- A Cloudflare Workers deployment, for now (§4.10). +- User accounts, ever: if a gate is added, it is one shared password. --- @@ -249,9 +255,9 @@ src/ web/ the static page (HTML + TS bundled by `bun build`) + login page server/ app.ts the portable fetch(Request) handler: gate, assets, /api/* - gate.ts password check, signed session cookie, login rate limit - storage.ts Storage interface: R2 binding or S3-compatible bucket - worker.ts Cloudflare Workers entry (R2 binding, static assets) + gate.ts (deferred) password check, signed session cookie, login rate limit + storage.ts Storage interface: S3-compatible bucket (R2 binding later) + worker.ts (deferred) Cloudflare Workers entry (R2 binding, static assets) bun.ts Bun.serve entry for self-hosting (S3 storage, assets from dist/) userscript/ entry + metadata banner, bundled by `bun build` into mbseed.user.js cli/ later: Bun CLI on the same core @@ -431,7 +437,7 @@ export interface ImageHost { | Host | Page | Userscript | Lifetime | |---|---|---|---| -| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same, with the gate password (§4.11) | 30 days (bucket lifecycle rule) | +| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (bucket lifecycle rule) | | `litterbox` (default otherwise) | multipart upload of the inspected bytes **directly from the browser** (CORS open), `time=72h` unless the user picks less | same, via `GM_xmlhttpRequest` | 1h / 12h / 24h / 72h | | `catbox` | `POST /api/upload { url, host: "catbox", userhash? }` — the server fetches the image and uploads it, because catbox has no CORS | **directly** via `GM_xmlhttpRequest`, no server needed | permanent | | `direct` | no upload: the normalised source URL | same | as long as Webtoons keeps it | @@ -522,7 +528,7 @@ language: series.language, type, note, content_rating, is_default }`. (`@connect webtoons.com`, `pstatic.net`, `web.archive.org`, `catbox.moe`, and our server's host). Same review cards as the page. - Rehosts to litterbox or catbox directly, or to our server's storage - (`/api/upload`, with the gate password; no storage keys in the script). With + (`/api/upload`; no storage keys in the script). With litterbox or catbox it works with **no server at all**. - Submits through the API when a token is set (stored with `GM_setValue`). - **Without a token**, on the cover form page it can fill the form fields @@ -533,20 +539,19 @@ language: series.language, type, note, content_rating, is_default }`. | Route | Behaviour | |---|---| -| `GET /login`, `POST /login`, `POST /logout` | The gate (§4.11). Always reachable. | | `GET /*` | Static assets of the page. | -| `GET /api/config` | `{ storage: boolean, gated: boolean }`, so the page knows which hosts to offer. | +| `GET /api/config` | `{ storage: boolean }`, so the page knows which hosts to offer. | | `GET /api/fetch?url=` | Fetches `url` only if its host is in the allowlist (union of `Source.hosts` plus `web.archive.org`), with a browser User-Agent; streams the response back. Anything else: 403. | | `POST /api/upload` | `{ url, host: "store" \| "catbox", userhash? }` (§4.6). `url` must pass the same allowlist. | The API routes reject other origins (except the userscript, which is not bound by CORS), cap response size at 20 MB, and are rate-limited per IP. -**Two deployments, one handler.** `app.ts` exports +**One handler; Bun now, Cloudflare later.** `app.ts` exports `handle(req: Request, env: ServerEnv): Promise` using only web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`): -| | Cloudflare (`worker.ts`) | Self-hosted (`bun.ts`) | +| | Cloudflare (`worker.ts`, deferred) | Self-hosted (`bun.ts`, **now**) | |---|---|---| | Run | `wrangler dev` / `wrangler deploy` | `bun run src/server/bun.ts` behind our reverse proxy | | Static assets | Workers static assets, with `run_worker_first = true` so the gate also covers assets (confirm the key when scaffolding) | read from the built `dist/web/` | @@ -554,9 +559,16 @@ web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`): | Password | `wrangler secret put MBSEED_PASSWORD` | `MBSEED_PASSWORD` env var | | Client IP for rate limits | `CF-Connecting-IP` | `server.requestIP(req)`, or `X-Forwarded-For` only when `MBSEED_TRUST_PROXY=1` | -Self-hosting also removes the Cloudflare-egress risk (§7). +Self-hosting also avoids the Cloudflare-egress risk. Our reverse proxy +terminates TLS; how `X-Forwarded-For` is trusted is set by `MBSEED_TRUST_PROXY`. -### 4.11 Access gate (`server/gate.ts`) +### 4.11 Access gate (`server/gate.ts`) — deferred + +**Not built in the first version.** Decided 2026-10-14: the self-hosted site +starts open, and we add this if it gets abused or traffic grows too large. +Until then the server's protections are the host allowlist, the 20 MB cap and +the per-IP rate limit. The design below is kept for when it's needed; until +then there is no `/login`, and `/api/config` has no `gated` field. An optional shared password locks the **whole** service: page, assets and API. @@ -583,15 +595,16 @@ An optional shared password locks the **whole** service: page, assets and API. - **Secrets.** The MangaBaka token stays in the user's browser (or userscript storage) and is only sent to `api.mangabaka.org`. The server's only secrets - are `MBSEED_PASSWORD` and, when self-hosted, the `MBSEED_S3_*` credentials - (on Cloudflare, R2 is a binding). The catbox userhash is forwarded, never + are the `MBSEED_S3_*` credentials (and `MBSEED_PASSWORD` once the gate + exists). The catbox userhash is forwarded, never stored. - **Errors.** The UI shows a per-stage status; Wayback failure is a warning, not a failure. -- **Cost.** On Cloudflare, free tiers cover expected use: a run is ~60–100 Worker - requests and ≤ ~6 MB stored for 30 days. Abuse of the proxy is the main cost - risk, hence the allowlist, the size cap, the per-IP rate limit and the - password gate. +- **Load.** A run is ~60–100 proxied requests and ≤ ~6 MB stored for 30 days. + Abuse of the open proxy is the main risk, hence the allowlist, the size cap + and the per-IP rate limit; the gate (§4.11) is the next step if needed. + (Earlier estimate for Cloudflare: $0 on free tiers; ~$5/month for a busy + public instance.) ## 6. Testing @@ -611,8 +624,8 @@ An optional shared password locks the **whole** service: page, assets and API. ## 7. Open questions and risks -1. **Cloudflare egress.** Webtoons or Wayback may block or throttle Cloudflare - IPs. The userscript and a self-hosted server don't have this risk. +1. **Server egress.** Webtoons or Wayback may throttle our server's IP under + load. (Cloudflare IPs would be riskier; the userscript uses the user's IP.) 2. **When MangaBaka copies images** (submission or approval) and whether it accepts our storage URLs, `litter.catbox.moe` / `files.catbox.moe` URLs, or `swebtoon-phinf.pstatic.net` (`direct`). If it copies only on approval, @@ -620,10 +633,11 @@ An optional shared password locks the **whole** service: page, assets and API. 3. **"Main cover" = `is_default: true`** is inferred from the API spec. 4. **Where users create a personal access token** in the MangaBaka UI. 5. **Whether the cover form accepts fields filled by a userscript.** -6. **Whether catbox accepts uploads from Cloudflare IPs** (catbox-via-server). -7. **Hosting details:** Cloudflare account and domains, or which of our own - servers and which S3-compatible bucket; whether the gate or the reverse - proxy's auth locks it. +6. **Whether catbox accepts uploads from our server** (catbox-via-server). +7. **Hosting details:** which of our servers and domain, and the storage + behind `server` — an S3-compatible bucket (MinIO, R2, B2…) as specified, or, + if simpler on our box, a local directory with a daily 30-day sweep (decide + in the plan; it would be a `Storage` implementation, nothing else changes). ## 8. Test data