From 7d7a82e291facf62c748dce5463e07a292ef51a6 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Wed, 14 Oct 2026 22:11:50 +0200 Subject: [PATCH] docs: store rehosted covers in a folder on the server Server storage is a plain folder (MBSEED_STORAGE_DIR) served at /covers/ with a daily sweep deleting files older than 30 days. S3/R2 remains a later Storage option. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 +-- docs/ARCHITECTURE.md | 4 +-- docs/DECISIONS.md | 12 +++++++++ docs/TODO.md | 7 ++--- docs/specs/001-webtoons-covers.md | 44 +++++++++++++++++++------------ 5 files changed, 47 insertions(+), 24 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ca76e8b..12cc17e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -61,10 +61,10 @@ Full picture: `docs/ARCHITECTURE.md` and the spec — read them before your firs ## Boundaries -Never commit `.env*`, `.dev.vars`, tokens, passwords, S3 keys or Cloudflare credentials. +Never commit `.env*`, `.dev.vars`, tokens, passwords or other credentials. Never force-push, rewrite pushed history, or commit directly to `main`. -Ask first: new runtime dependencies · deploying, or any server/bucket configuration · +Ask first: new runtime dependencies · deploying, or any server configuration · real uploads to litterbox/catbox · anything sent to MangaBaka with a real token · anything destructive (`rm -rf`, `git reset --hard`) · work the approved plan does not cover. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 863c5ef..3b9dc0e 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -13,7 +13,7 @@ self-hosted Bun server (primary), a userscript, and later a Bun CLI. Replace thi |---|---| | `src/core/` | Pure pipeline: sources, Wayback history, image inspection, dedupe, seed, MangaBaka client. No DOM, Bun, Node or `GM_*` APIs. | | `src/web/` | Static page UI. | -| `src/server/` | Web-standard `fetch` handler (`app.ts`) run by `bun.ts` (`Bun.serve`, behind our reverse proxy). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`. Gate and Workers entry deferred. | +| `src/server/` | Web-standard `fetch` handler (`app.ts`) run by `bun.ts` (`Bun.serve`, behind our reverse proxy). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`, `/covers/*` from the storage folder (30-day sweep). Gate and Workers entry deferred. | | `src/userscript/` | Userscript on mangabaka.org; network via `GM_xmlhttpRequest`. | | `src/cli/` | Later: Bun CLI. | | `test/fixtures/` | Saved real responses; tests never hit the network. | @@ -28,7 +28,7 @@ sha256) → dedupe against MangaBaka → seed → user reviews → rehost (expli | `api.mangabaka.org` | direct (CORS open) | Nothing works. | | Webtoons HTML / `swebtoon-phinf.pstatic.net` | server proxy or `GM_xmlhttpRequest` | Nothing works. | | Wayback (`web.archive.org`) | same | Warning; current covers only. | -| Server storage (S3-compatible bucket) | server | `server` host hidden; litterbox/catbox/direct still work. | +| Server storage (folder on our server) | server | `server` host hidden; litterbox/catbox/direct still work. | | litterbox / catbox | browser directly (litterbox), server or `GM_xmlhttpRequest` (catbox) | That host fails; others still work. | ## Invariants diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index 3da62ed..bef3280 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -16,6 +16,18 @@ the repo every session — measurably slower and more tokens). **Consequences:** Rule changes happen in one file; pointer files must stay thin. Plans are working artefacts: kept after merge they go stale and contradict the code, so the spec is the record. +## 2026-10-14 — Server storage is a plain folder on our server +**Status:** accepted (replaces the S3/R2 storage in the entries below) +**Context:** Self-hosted, a bucket is extra infrastructure. The user chose: "For now, store images in +a plain folder on the server." +**Decision:** `MBSEED_STORAGE_DIR` holds `.`; our server serves it at `/covers/…` +under `MBSEED_PUBLIC_URL`. `bun.ts` deletes files older than 30 days (by mtime) at startup and every +24 hours. Writes are temp file + rename. It sits behind the `Storage` interface. +**Alternatives:** S3-compatible bucket (MinIO, R2, B2) with a lifecycle rule — kept as a later option. +**Consequences:** The server must be publicly reachable for MangaBaka to fetch covers, and its +uptime now matters until each review is done. Folder access needs Bun APIs, so it lives in +`storage-dir.ts`, not in the web-standard `app.ts`. + ## 2026-10-14 — Spec signed off; self-hosted only; password gate deferred **Status:** accepted (narrows the entry below) **Context:** The user signed off spec 001 and said the app will run self-hosted, and that diff --git a/docs/TODO.md b/docs/TODO.md index f2ec0a0..b6dc83e 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -22,8 +22,8 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). registry, webtoons (urls, parse, index), history/wayback, images, covers, mangabaka client, hosting (server, litterbox, catbox, direct), seed. - [ ] `server/`: `app.ts` (assets, `/api/config`, `/api/fetch` allowlist, `/api/upload` store|catbox, - per-IP rate limit), storage, entry `bun.ts`. Decide in the plan: S3-compatible bucket or - local directory with a 30-day sweep (spec §7.7). + `/covers/*`, per-IP rate limit), `Storage` + folder storage with a daily 30-day sweep, + entry `bun.ts`. - [ ] `web/`: two-link form, review cards, host picker with lifetimes, rehost, copy, token flow (preview → confirm → save). - [ ] `userscript/`: panel on mangabaka.org series pages, `GM_xmlhttpRequest` transport, token in @@ -33,7 +33,7 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). - [ ] Update README, `docs/ARCHITECTURE.md` and `PRODUCT.md`/`DESIGN.md` to match reality. ## Verify with real services (once possible) -- [ ] Which of our servers and domain hosts it; which bucket (or local dir) backs `server` storage. +- [ ] Which of our servers and domain hosts it (`MBSEED_PUBLIC_URL`); where the covers folder lives. - [ ] Does catbox accept uploads from our server? - [ ] Does MangaBaka accept our storage URLs, litterbox/catbox URLs, and `swebtoon-phinf.pstatic.net` (`direct`)? @@ -43,6 +43,7 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md). - [ ] Does the MangaBaka cover form accept fields filled by a userscript? ## Someday +- [ ] S3-compatible bucket as a `Storage` option, if a single server's disk stops being enough. - [ ] Password gate (spec §4.11) — only if the open site gets abused or traffic grows too large. - [ ] Cloudflare Workers entry (`server/worker.ts`, R2 binding). If added: Workers serve static assets before the Worker unless `run_worker_first` is set — needed for the gate to cover them. diff --git a/docs/specs/001-webtoons-covers.md b/docs/specs/001-webtoons-covers.md index 468f8b0..face0ad 100644 --- a/docs/specs/001-webtoons-covers.md +++ b/docs/specs/001-webtoons-covers.md @@ -182,7 +182,7 @@ Verified on 2026-10-14. Treat them as assumptions that the fixtures in | Host | Browser CORS | Expiry | Verdict | |---|---|---|---| -| **Our storage** (R2 or any S3-compatible bucket) | n/a (behind our server) | 30-day lifecycle rule | **Default** when configured. | +| **Our storage** (a folder on our server) | n/a (served by our server) | 30 days (daily sweep) | **Default** when configured. | | **litterbox.catbox.moe** | **yes** (`access-control-allow-origin: *`) | `time` = `1h` / `12h` / `24h` / `72h` only | Supported; no account; 72h may be shorter than a review. | | **catbox.moe** | no | permanent | Supported via our server (page) or `GM_xmlhttpRequest` (userscript). | | 0x0.st | unverified | configurable | Not used. | @@ -247,7 +247,7 @@ src/ types.ts API types (subset of the OpenAPI spec we use) hosting/ host.ts ImageHost interface + registry - server.ts upload via our server → our storage (30 days) + server.ts upload via our server → its covers folder (30 days) litterbox.ts litterbox.catbox.moe, 1h–72h (direct; CORS open) catbox.ts catbox.moe, permanent (via server, or GM in the userscript) direct.ts no rehosting: submit source URLs as-is @@ -256,9 +256,10 @@ src/ server/ app.ts the portable fetch(Request) handler: gate, assets, /api/* gate.ts (deferred) password check, signed session cookie, login rate limit - storage.ts Storage interface: S3-compatible bucket (R2 binding later) + storage.ts Storage interface (put/get/sweep) + in-memory impl for tests + storage-dir.ts folder on disk (Bun file APIs; used only by bun.ts) worker.ts (deferred) Cloudflare Workers entry (R2 binding, static assets) - bun.ts Bun.serve entry for self-hosting (S3 storage, assets from dist/) + bun.ts Bun.serve entry for self-hosting (folder storage + daily sweep, assets from dist/) userscript/ entry + metadata banner, bundled by `bun build` into mbseed.user.js cli/ later: Bun CLI on the same core test/ @@ -437,15 +438,25 @@ export interface ImageHost { | Host | Page | Userscript | Lifetime | |---|---|---|---| -| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (bucket lifecycle rule) | +| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (daily sweep) | | `litterbox` (default otherwise) | multipart upload of the inspected bytes **directly from the browser** (CORS open), `time=72h` unless the user picks less | same, via `GM_xmlhttpRequest` | 1h / 12h / 24h / 72h | | `catbox` | `POST /api/upload { url, host: "catbox", userhash? }` — the server fetches the image and uploads it, because catbox has no CORS | **directly** via `GM_xmlhttpRequest`, no server needed | permanent | | `direct` | no upload: the normalised source URL | same | as long as Webtoons keeps it | -- **Server storage** stores at `covers/.` and returns the public - URL. On Cloudflare it is an R2 binding; self-hosted it is any S3-compatible - bucket (R2, B2, MinIO, AWS) via `aws4fetch`. Either way the bucket has a - 30-day lifecycle rule and is publicly readable (custom domain or `r2.dev`). +- **Server storage** is, for now, **a plain folder on our server** + (`MBSEED_STORAGE_DIR`). Files are named `.` and served by our + server at `GET /covers/.`; the returned URL is + `/covers/.`, so the server must be reachable + from the internet for MangaBaka to fetch it. + - Writes go to a temp file in the same folder, then `rename`, so a reader + never sees half a file. + - **Expiry:** `bun.ts` sweeps the folder at startup and every 24 hours, + deleting files whose modification time is older than 30 days. A re-upload + rewrites the file, which restarts its 30 days. + - `GET /covers/*` accepts only `^[0-9a-f]{64}\.(jpg|png|gif|webp)$` (no path + traversal) and sets `Content-Type` from the extension. + - Kept behind the `Storage` interface so an S3-compatible bucket or an R2 + binding can replace it later without touching `app.ts`. If the server has no storage configured, `GET /api/config` says so and the page hides the `server` host. - **litterbox 72h warning.** The UI warns that a review may take longer than @@ -541,6 +552,7 @@ language: series.language, type, note, content_rating, is_default }`. |---|---| | `GET /*` | Static assets of the page. | | `GET /api/config` | `{ storage: boolean }`, so the page knows which hosts to offer. | +| `GET /covers/.` | Serves a stored cover (§4.6). Public: MangaBaka fetches these. | | `GET /api/fetch?url=` | Fetches `url` only if its host is in the allowlist (union of `Source.hosts` plus `web.archive.org`), with a browser User-Agent; streams the response back. Anything else: 403. | | `POST /api/upload` | `{ url, host: "store" \| "catbox", userhash? }` (§4.6). `url` must pass the same allowlist. | @@ -555,7 +567,7 @@ web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`): |---|---|---| | Run | `wrangler dev` / `wrangler deploy` | `bun run src/server/bun.ts` behind our reverse proxy | | Static assets | Workers static assets, with `run_worker_first = true` so the gate also covers assets (confirm the key when scaffolding) | read from the built `dist/web/` | -| Storage | R2 binding | S3-compatible bucket via `MBSEED_S3_*` env vars, or none | +| Storage | R2 binding | folder `MBSEED_STORAGE_DIR` (public base URL `MBSEED_PUBLIC_URL`), or none | | Password | `wrangler secret put MBSEED_PASSWORD` | `MBSEED_PASSWORD` env var | | Client IP for rate limits | `CF-Connecting-IP` | `server.requestIP(req)`, or `X-Forwarded-For` only when `MBSEED_TRUST_PROXY=1` | @@ -594,9 +606,8 @@ An optional shared password locks the **whole** service: page, assets and API. ## 5. Cross-cutting concerns - **Secrets.** The MangaBaka token stays in the user's browser (or userscript - storage) and is only sent to `api.mangabaka.org`. The server's only secrets - are the `MBSEED_S3_*` credentials (and `MBSEED_PASSWORD` once the gate - exists). The catbox userhash is forwarded, never + storage) and is only sent to `api.mangabaka.org`. The server has no secrets + for now (`MBSEED_PASSWORD` once the gate exists). The catbox userhash is forwarded, never stored. - **Errors.** The UI shows a per-stage status; Wayback failure is a warning, not a failure. @@ -634,10 +645,9 @@ An optional shared password locks the **whole** service: page, assets and API. 4. **Where users create a personal access token** in the MangaBaka UI. 5. **Whether the cover form accepts fields filled by a userscript.** 6. **Whether catbox accepts uploads from our server** (catbox-via-server). -7. **Hosting details:** which of our servers and domain, and the storage - behind `server` — an S3-compatible bucket (MinIO, R2, B2…) as specified, or, - if simpler on our box, a local directory with a daily 30-day sweep (decide - in the plan; it would be a `Storage` implementation, nothing else changes). +7. **Hosting details:** which of our servers and domain (`MBSEED_PUBLIC_URL`), + and where the covers folder lives (`MBSEED_STORAGE_DIR`). Disk use is small: + at most ~6 MB per run, kept 30 days. ## 8. Test data