docs: store rehosted covers in a folder on the server
Server storage is a plain folder (MBSEED_STORAGE_DIR) served at /covers/ with a daily sweep deleting files older than 30 days. S3/R2 remains a later Storage option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
dd2d5a94bc
commit
7d7a82e291
5 files changed
+47
-24
No files matched your search
@@ -61,10 +61,10 @@ Full picture: `docs/ARCHITECTURE.md` and the spec — read them before your firs
|
||||
|
||||
## Boundaries
|
||||
|
||||
Never commit `.env*`, `.dev.vars`, tokens, passwords, S3 keys or Cloudflare credentials.
|
||||
Never commit `.env*`, `.dev.vars`, tokens, passwords or other credentials.
|
||||
Never force-push, rewrite pushed history, or commit directly to `main`.
|
||||
|
||||
Ask first: new runtime dependencies · deploying, or any server/bucket configuration ·
|
||||
Ask first: new runtime dependencies · deploying, or any server configuration ·
|
||||
real uploads to litterbox/catbox ·
|
||||
anything sent to MangaBaka with a real token · anything destructive (`rm -rf`, `git reset --hard`) ·
|
||||
work the approved plan does not cover.
|
||||
|
||||
@@ -13,7 +13,7 @@ self-hosted Bun server (primary), a userscript, and later a Bun CLI. Replace thi
|
||||
|---|---|
|
||||
| `src/core/` | Pure pipeline: sources, Wayback history, image inspection, dedupe, seed, MangaBaka client. No DOM, Bun, Node or `GM_*` APIs. |
|
||||
| `src/web/` | Static page UI. |
|
||||
| `src/server/` | Web-standard `fetch` handler (`app.ts`) run by `bun.ts` (`Bun.serve`, behind our reverse proxy). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`. Gate and Workers entry deferred. |
|
||||
| `src/server/` | Web-standard `fetch` handler (`app.ts`) run by `bun.ts` (`Bun.serve`, behind our reverse proxy). Static assets, `/api/fetch` allowlisted proxy, `/api/upload`, `/covers/*` from the storage folder (30-day sweep). Gate and Workers entry deferred. |
|
||||
| `src/userscript/` | Userscript on mangabaka.org; network via `GM_xmlhttpRequest`. |
|
||||
| `src/cli/` | Later: Bun CLI. |
|
||||
| `test/fixtures/` | Saved real responses; tests never hit the network. |
|
||||
@@ -28,7 +28,7 @@ sha256) → dedupe against MangaBaka → seed → user reviews → rehost (expli
|
||||
| `api.mangabaka.org` | direct (CORS open) | Nothing works. |
|
||||
| Webtoons HTML / `swebtoon-phinf.pstatic.net` | server proxy or `GM_xmlhttpRequest` | Nothing works. |
|
||||
| Wayback (`web.archive.org`) | same | Warning; current covers only. |
|
||||
| Server storage (S3-compatible bucket) | server | `server` host hidden; litterbox/catbox/direct still work. |
|
||||
| Server storage (folder on our server) | server | `server` host hidden; litterbox/catbox/direct still work. |
|
||||
| litterbox / catbox | browser directly (litterbox), server or `GM_xmlhttpRequest` (catbox) | That host fails; others still work. |
|
||||
|
||||
## Invariants
|
||||
|
||||
@@ -16,6 +16,18 @@ the repo every session — measurably slower and more tokens).
|
||||
**Consequences:** Rule changes happen in one file; pointer files must stay thin. Plans are working
|
||||
artefacts: kept after merge they go stale and contradict the code, so the spec is the record.
|
||||
|
||||
## 2026-10-14 — Server storage is a plain folder on our server
|
||||
**Status:** accepted (replaces the S3/R2 storage in the entries below)
|
||||
**Context:** Self-hosted, a bucket is extra infrastructure. The user chose: "For now, store images in
|
||||
a plain folder on the server."
|
||||
**Decision:** `MBSEED_STORAGE_DIR` holds `<sha256>.<ext>`; our server serves it at `/covers/…`
|
||||
under `MBSEED_PUBLIC_URL`. `bun.ts` deletes files older than 30 days (by mtime) at startup and every
|
||||
24 hours. Writes are temp file + rename. It sits behind the `Storage` interface.
|
||||
**Alternatives:** S3-compatible bucket (MinIO, R2, B2) with a lifecycle rule — kept as a later option.
|
||||
**Consequences:** The server must be publicly reachable for MangaBaka to fetch covers, and its
|
||||
uptime now matters until each review is done. Folder access needs Bun APIs, so it lives in
|
||||
`storage-dir.ts`, not in the web-standard `app.ts`.
|
||||
|
||||
## 2026-10-14 — Spec signed off; self-hosted only; password gate deferred
|
||||
**Status:** accepted (narrows the entry below)
|
||||
**Context:** The user signed off spec 001 and said the app will run self-hosted, and that
|
||||
|
||||
+4
-3
@@ -22,8 +22,8 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md).
|
||||
registry, webtoons (urls, parse, index), history/wayback, images, covers, mangabaka client,
|
||||
hosting (server, litterbox, catbox, direct), seed.
|
||||
- [ ] `server/`: `app.ts` (assets, `/api/config`, `/api/fetch` allowlist, `/api/upload` store|catbox,
|
||||
per-IP rate limit), storage, entry `bun.ts`. Decide in the plan: S3-compatible bucket or
|
||||
local directory with a 30-day sweep (spec §7.7).
|
||||
`/covers/*`, per-IP rate limit), `Storage` + folder storage with a daily 30-day sweep,
|
||||
entry `bun.ts`.
|
||||
- [ ] `web/`: two-link form, review cards, host picker with lifetimes, rehost, copy,
|
||||
token flow (preview → confirm → save).
|
||||
- [ ] `userscript/`: panel on mangabaka.org series pages, `GM_xmlhttpRequest` transport, token in
|
||||
@@ -33,7 +33,7 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md).
|
||||
- [ ] Update README, `docs/ARCHITECTURE.md` and `PRODUCT.md`/`DESIGN.md` to match reality.
|
||||
|
||||
## Verify with real services (once possible)
|
||||
- [ ] Which of our servers and domain hosts it; which bucket (or local dir) backs `server` storage.
|
||||
- [ ] Which of our servers and domain hosts it (`MBSEED_PUBLIC_URL`); where the covers folder lives.
|
||||
- [ ] Does catbox accept uploads from our server?
|
||||
- [ ] Does MangaBaka accept our storage URLs, litterbox/catbox URLs, and
|
||||
`swebtoon-phinf.pstatic.net` (`direct`)?
|
||||
@@ -43,6 +43,7 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md).
|
||||
- [ ] Does the MangaBaka cover form accept fields filled by a userscript?
|
||||
|
||||
## Someday
|
||||
- [ ] S3-compatible bucket as a `Storage` option, if a single server's disk stops being enough.
|
||||
- [ ] Password gate (spec §4.11) — only if the open site gets abused or traffic grows too large.
|
||||
- [ ] Cloudflare Workers entry (`server/worker.ts`, R2 binding). If added: Workers serve static
|
||||
assets before the Worker unless `run_worker_first` is set — needed for the gate to cover them.
|
||||
|
||||
@@ -182,7 +182,7 @@ Verified on 2026-10-14. Treat them as assumptions that the fixtures in
|
||||
|
||||
| Host | Browser CORS | Expiry | Verdict |
|
||||
|---|---|---|---|
|
||||
| **Our storage** (R2 or any S3-compatible bucket) | n/a (behind our server) | 30-day lifecycle rule | **Default** when configured. |
|
||||
| **Our storage** (a folder on our server) | n/a (served by our server) | 30 days (daily sweep) | **Default** when configured. |
|
||||
| **litterbox.catbox.moe** | **yes** (`access-control-allow-origin: *`) | `time` = `1h` / `12h` / `24h` / `72h` only | Supported; no account; 72h may be shorter than a review. |
|
||||
| **catbox.moe** | no | permanent | Supported via our server (page) or `GM_xmlhttpRequest` (userscript). |
|
||||
| 0x0.st | unverified | configurable | Not used. |
|
||||
@@ -247,7 +247,7 @@ src/
|
||||
types.ts API types (subset of the OpenAPI spec we use)
|
||||
hosting/
|
||||
host.ts ImageHost interface + registry
|
||||
server.ts upload via our server → our storage (30 days)
|
||||
server.ts upload via our server → its covers folder (30 days)
|
||||
litterbox.ts litterbox.catbox.moe, 1h–72h (direct; CORS open)
|
||||
catbox.ts catbox.moe, permanent (via server, or GM in the userscript)
|
||||
direct.ts no rehosting: submit source URLs as-is
|
||||
@@ -256,9 +256,10 @@ src/
|
||||
server/
|
||||
app.ts the portable fetch(Request) handler: gate, assets, /api/*
|
||||
gate.ts (deferred) password check, signed session cookie, login rate limit
|
||||
storage.ts Storage interface: S3-compatible bucket (R2 binding later)
|
||||
storage.ts Storage interface (put/get/sweep) + in-memory impl for tests
|
||||
storage-dir.ts folder on disk (Bun file APIs; used only by bun.ts)
|
||||
worker.ts (deferred) Cloudflare Workers entry (R2 binding, static assets)
|
||||
bun.ts Bun.serve entry for self-hosting (S3 storage, assets from dist/)
|
||||
bun.ts Bun.serve entry for self-hosting (folder storage + daily sweep, assets from dist/)
|
||||
userscript/ entry + metadata banner, bundled by `bun build` into mbseed.user.js
|
||||
cli/ later: Bun CLI on the same core
|
||||
test/
|
||||
@@ -437,15 +438,25 @@ export interface ImageHost {
|
||||
|
||||
| Host | Page | Userscript | Lifetime |
|
||||
|---|---|---|---|
|
||||
| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (bucket lifecycle rule) |
|
||||
| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (daily sweep) |
|
||||
| `litterbox` (default otherwise) | multipart upload of the inspected bytes **directly from the browser** (CORS open), `time=72h` unless the user picks less | same, via `GM_xmlhttpRequest` | 1h / 12h / 24h / 72h |
|
||||
| `catbox` | `POST /api/upload { url, host: "catbox", userhash? }` — the server fetches the image and uploads it, because catbox has no CORS | **directly** via `GM_xmlhttpRequest`, no server needed | permanent |
|
||||
| `direct` | no upload: the normalised source URL | same | as long as Webtoons keeps it |
|
||||
|
||||
- **Server storage** stores at `covers/<sha256>.<ext>` and returns the public
|
||||
URL. On Cloudflare it is an R2 binding; self-hosted it is any S3-compatible
|
||||
bucket (R2, B2, MinIO, AWS) via `aws4fetch`. Either way the bucket has a
|
||||
30-day lifecycle rule and is publicly readable (custom domain or `r2.dev`).
|
||||
- **Server storage** is, for now, **a plain folder on our server**
|
||||
(`MBSEED_STORAGE_DIR`). Files are named `<sha256>.<ext>` and served by our
|
||||
server at `GET /covers/<sha256>.<ext>`; the returned URL is
|
||||
`<MBSEED_PUBLIC_URL>/covers/<sha256>.<ext>`, so the server must be reachable
|
||||
from the internet for MangaBaka to fetch it.
|
||||
- Writes go to a temp file in the same folder, then `rename`, so a reader
|
||||
never sees half a file.
|
||||
- **Expiry:** `bun.ts` sweeps the folder at startup and every 24 hours,
|
||||
deleting files whose modification time is older than 30 days. A re-upload
|
||||
rewrites the file, which restarts its 30 days.
|
||||
- `GET /covers/*` accepts only `^[0-9a-f]{64}\.(jpg|png|gif|webp)$` (no path
|
||||
traversal) and sets `Content-Type` from the extension.
|
||||
- Kept behind the `Storage` interface so an S3-compatible bucket or an R2
|
||||
binding can replace it later without touching `app.ts`.
|
||||
If the server has no storage configured, `GET /api/config` says so and the
|
||||
page hides the `server` host.
|
||||
- **litterbox 72h warning.** The UI warns that a review may take longer than
|
||||
@@ -541,6 +552,7 @@ language: series.language, type, note, content_rating, is_default }`.
|
||||
|---|---|
|
||||
| `GET /*` | Static assets of the page. |
|
||||
| `GET /api/config` | `{ storage: boolean }`, so the page knows which hosts to offer. |
|
||||
| `GET /covers/<sha256>.<ext>` | Serves a stored cover (§4.6). Public: MangaBaka fetches these. |
|
||||
| `GET /api/fetch?url=` | Fetches `url` only if its host is in the allowlist (union of `Source.hosts` plus `web.archive.org`), with a browser User-Agent; streams the response back. Anything else: 403. |
|
||||
| `POST /api/upload` | `{ url, host: "store" \| "catbox", userhash? }` (§4.6). `url` must pass the same allowlist. |
|
||||
|
||||
@@ -555,7 +567,7 @@ web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`):
|
||||
|---|---|---|
|
||||
| Run | `wrangler dev` / `wrangler deploy` | `bun run src/server/bun.ts` behind our reverse proxy |
|
||||
| Static assets | Workers static assets, with `run_worker_first = true` so the gate also covers assets (confirm the key when scaffolding) | read from the built `dist/web/` |
|
||||
| Storage | R2 binding | S3-compatible bucket via `MBSEED_S3_*` env vars, or none |
|
||||
| Storage | R2 binding | folder `MBSEED_STORAGE_DIR` (public base URL `MBSEED_PUBLIC_URL`), or none |
|
||||
| Password | `wrangler secret put MBSEED_PASSWORD` | `MBSEED_PASSWORD` env var |
|
||||
| Client IP for rate limits | `CF-Connecting-IP` | `server.requestIP(req)`, or `X-Forwarded-For` only when `MBSEED_TRUST_PROXY=1` |
|
||||
|
||||
@@ -594,9 +606,8 @@ An optional shared password locks the **whole** service: page, assets and API.
|
||||
## 5. Cross-cutting concerns
|
||||
|
||||
- **Secrets.** The MangaBaka token stays in the user's browser (or userscript
|
||||
storage) and is only sent to `api.mangabaka.org`. The server's only secrets
|
||||
are the `MBSEED_S3_*` credentials (and `MBSEED_PASSWORD` once the gate
|
||||
exists). The catbox userhash is forwarded, never
|
||||
storage) and is only sent to `api.mangabaka.org`. The server has no secrets
|
||||
for now (`MBSEED_PASSWORD` once the gate exists). The catbox userhash is forwarded, never
|
||||
stored.
|
||||
- **Errors.** The UI shows a per-stage status; Wayback failure is a warning,
|
||||
not a failure.
|
||||
@@ -634,10 +645,9 @@ An optional shared password locks the **whole** service: page, assets and API.
|
||||
4. **Where users create a personal access token** in the MangaBaka UI.
|
||||
5. **Whether the cover form accepts fields filled by a userscript.**
|
||||
6. **Whether catbox accepts uploads from our server** (catbox-via-server).
|
||||
7. **Hosting details:** which of our servers and domain, and the storage
|
||||
behind `server` — an S3-compatible bucket (MinIO, R2, B2…) as specified, or,
|
||||
if simpler on our box, a local directory with a daily 30-day sweep (decide
|
||||
in the plan; it would be a `Storage` implementation, nothing else changes).
|
||||
7. **Hosting details:** which of our servers and domain (`MBSEED_PUBLIC_URL`),
|
||||
and where the covers folder lives (`MBSEED_STORAGE_DIR`). Disk use is small:
|
||||
at most ~6 MB per run, kept 30 days.
|
||||
|
||||
## 8. Test data
|
||||
|
||||
|
||||
Reference in new issue
Block a user