lucasw89/mangabaka-seeder
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dd2d5a94bc347f9cbf3bb70e27614488c4dc32a7
mangabaka-seeder/docs/specs/001-webtoons-covers.md
T
Lucas WintherandClaude Opus 5.5 dd2d5a94bc docs: approve spec 001; self-hosted Bun server, defer password gate 
The user signed off the spec. The server runs self-hosted under Bun;
the password gate and the Cloudflare Workers entry stay designed but
deferred until abuse or traffic calls for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 22:10:20 +02:00

674 lines
34 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
status: approved
---
# 001 — Webtoons covers
`mangabaka-seeder` gathers data about a series from an external site (a
"source") and turns it into an edit request ("submission") for
[MangaBaka](https://mangabaka.org). It is modelled on
[yambs](https://codeberg.org/derat/yambs), the MusicBrainz seeder: sources turn
into seeds, and an action decides what happens to them. The user always stays
in control of what is submitted.
The first feature is **covers**: given a MangaBaka series link and a Webtoons
series link, collect every cover the Webtoons series has used (current and
historical) and request them added to the MangaBaka series, with the current
poster as the default (main) cover for the edition's language.
This spec records the facts we verified about the external services, the
module boundaries and the extension points. Revised 2026-10-14: the user gives
both links directly, the product is a static web page first plus a userscript,
the runtime is Bun, and images go to a temporary host with a 30-day expiry.
Revised again the same day: catbox.moe and litterbox are also supported as
image hosts, the server can be password-gated, and the same server code runs
on Cloudflare Workers or under Bun on our own machine.
Signed off by the user 2026-10-14, with two changes applied here: the server
is **self-hosted** (Bun), and the **password gate is deferred** until abuse or
traffic calls for it (§4.11 keeps the design). The Cloudflare entry is
deferred too; the handler stays web-standard so it can be added later.
---
## 1. Goals and non-goals
**Goals**
- **Two links in, a reviewable cover submission out.** The user pastes a
MangaBaka series link and a Webtoons series link. There is no automatic
series matching.
- **Two front ends on one core:**
1. a **static web page** (the primary product), served by our small server,
which also proxies the sites browsers cannot read;
2. a **userscript** (Tampermonkey/Violentmonkey) on mangabaka.org, which needs
no proxy because `GM_xmlhttpRequest` is not bound by CORS.
A Bun CLI on the same core comes later.
- **Self-hosted server.** The server is a single web-standard `fetch(Request) →
Response` handler run by `Bun.serve` on our own machine, behind our reverse
proxy (§4.10). Because it is web-standard, a Cloudflare Workers entry can be
added later without touching it.
- **Choice of image host:** our own storage (30-day expiry), litterbox (up to
72h), catbox.moe (permanent), or no rehosting (§4.6).
- **Find historical covers**, not just the current one.
- **Work for everyone.** With a MangaBaka personal access token, submit through
the API. Without one, help the user enter the covers by hand.
- **New sources** (Tapas, Tappytoon, Lezhin, Manta, …) are one new module under
`src/core/sources/`, with no changes to the pipeline, the MangaBaka client or
the front ends.
- **Never change MangaBaka data silently.** API submissions always go through
preview, explicit confirmation and `save_mode: "review"`.
- **Never upload anything without explicit user action.** The user chooses the
content rating (default `safe`).
**Non-goals (for now)**
- Editing series fields other than images, such as titles, links or tags. The
submission layer is keyed by MangaBaka "resource" so this can come later.
- Following a Webtoons series across language editions. Each `title_no` belongs
to one language edition; run the tool once per edition link.
- Running our own permanent image host. Our storage only needs to outlive the
MangaBaka review, so it expires after 30 days; catbox.moe is the option for
anyone who wants permanent links (§4.6).
- Access control, for now. The site is open; the password gate (§4.11) is
designed but deferred until abuse or traffic calls for it.
- A Cloudflare Workers deployment, for now (§4.10).
- User accounts, ever: if a gate is added, it is one shared password.
---
## 2. Verified external facts
Verified on 2026-10-14. Treat them as assumptions that the fixtures in
`test/fixtures/` pin down.
### 2.1 Webtoons
| What | Where |
|---|---|
| Portrait poster (official series; 480×623 original) | `meta[property="og:image"]` |
| Square thumbnail | `meta[name="twitter:image"]` |
| Hero background (1920×321, official only) | `div.detail_bg` inline `style` → `background:url('…')` |
| Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) | `.detail_header span.thmb img[src]` |
| Title | `meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`) |
| Canonical URL | `link[rel="canonical"]` |
| `title_no` | `title_no` query parameter |
| Language | first path segment (`/en/`, `/fr/`, `/zh-hant/`, …) |
| Canvas vs official | `/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true` |
| Mature | inline JS `ageGradeNotice : true` |
- **Canvas series** have no portrait poster. Their `og:image` is a 1080×1080
square.
- **Image hosts.** `webtoon-phinf.pstatic.net` returns 403 without a Webtoons
`Referer`; `swebtoon-phinf.pstatic.net` serves the same path with no Referer.
- **Original size.** The `?type=…` query selects a resized variant. Removing it
returns the original. `crop540_540` crops posters, so always strip the query.
- **Old hosts.** Old snapshots reference `swebtoon.phinf.naver.net`, which
becomes `swebtoon-phinf.pstatic.net` with the same path.
- **No limits so far.** A browser User-Agent is enough. Age-gated titles still
serve full HTML, and no cookies are required. (Tested from a residential IP,
not from Cloudflare's; see §7.)
- **No usable JSON API.** The app API needs signed requests. We scrape HTML.
- **CORS:** neither the series HTML nor `swebtoon-phinf` images send
`Access-Control-Allow-Origin`. A page can show the images in `<img>` but
cannot read their bytes.
### 2.2 Historical covers (Wayback Machine)
- **Snapshot index.** The CDX API at
`https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original`
returns at most one snapshot per month.
- **Raw pages.** `https://web.archive.org/web/<ts>id_/<original>` returns the
page as it was archived, without Wayback's toolbar.
- **History exists.** Tower of God has about 2000 snapshots (2015–2026) that
reference five distinct cover images. All the older ones tested are **still
served live** from `swebtoon-phinf.pstatic.net`.
- **Fallback.** If an image is no longer live, it is often still at
`https://web.archive.org/web/<ts>id_/<image-url>`.
- **Not every image is a cover.** Some historical `og:image`s are landing-page
art, so candidates are classified by shape and reviewed by the user.
- **CORS:** no `Access-Control-Allow-Origin` seen on snapshots. CDX returned
503 during the check, so treat Wayback as unavailable from a plain page and
flaky in general.
### 2.3 MangaBaka
- **Endpoints.** The site is `https://mangabaka.org` and the API is
`https://api.mangabaka.org` (`api.mangabaka.dev` is deprecated). The OpenAPI
spec is at `https://mangabaka.org/api.json` (~700 KB), with docs at
`https://mangabaka.org/data/api`.
- **CORS is open.** `api.mangabaka.org` answers with
`access-control-allow-origin: *` and allows any header (including
`x-api-key`) and `POST`, on both public reads and
`/v0/my/submissions/…`. So the page and the userscript talk to MangaBaka
**directly**, never through our server.
- **Public reads.**
- `GET /v1/series/{id}` and `GET /v1/series/{id}/links` return the series
and its links. Links are canonicalised, e.g.
`https://www.webtoons.com/-/-/-/list?title_no=95`.
- `GET /v1/series/{id}/images` returns covers with `type`, `language`,
`index`, `note`, `content_rating`, `is_default` and `hashes.sha256`.
- Rate limits: about 180 requests per minute (search, which we no longer
use, is about 30).
- **Image `type` values:** `volume` (default), `volume_back`, `chapter`,
`season`, `banner`, `audiobook`, `other`. There is no "main" type; the main
cover is the one with `is_default: true`.
- **Submissions API.** Auth is `x-api-key: mb-…` (a personal access token).
All endpoints are v0 and marked internal.
- `GET /v0/my/submissions/series-images/{id}` returns the current image set
and a `version` token.
- `POST /v0/my/submissions/series-images/{id}/preview` is a dry run that
returns a diff.
- `POST /v0/my/submissions/series-images/{id}` saves, with body
`{ version, user_note, save_mode, data: { images: [...] } }`.
- `images` **replaces the whole set**, so it must contain the existing images.
- A stale `version` gives 409.
- Users without the contributor role and with fewer than 25 approved
submissions get 403.
- **Web form.** Without API access, users use
`https://mangabaka.org/data/submissions/series/{id}/images`. It cannot be
pre-filled with query parameters, and its SvelteKit CSRF check blocks
cross-site POSTs. (A userscript running *on* that page can still fill its
fields; see §4.9.)
- **Image URLs.** The spec describes the cover `url` as "Cover image URL. imgur
is rejected (aggressive rate limiting). Acts as the cover identity on save."
We don't know whether MangaBaka copies the image on submission or on
approval, so the hosted URL must survive the review (§4.6).
- **`content_rating`:** one of `safe`, `suggestive`, `erotica`,
`pornographic`. The user chooses it (§4.7).
- **Test series:** Tower of God is **series 9999**
(`https://mangabaka.org/manhwa/9999/Tower-of-God`; Webtoons `title_no=95`).
### 2.4 Image hosts
| Host | Browser CORS | Expiry | Verdict |
|---|---|---|---|
| **Our storage** (R2 or any S3-compatible bucket) | n/a (behind our server) | 30-day lifecycle rule | **Default** when configured. |
| **litterbox.catbox.moe** | **yes** (`access-control-allow-origin: *`) | `time` = `1h` / `12h` / `24h` / `72h` only | Supported; no account; 72h may be shorter than a review. |
| **catbox.moe** | no | permanent | Supported via our server (page) or `GM_xmlhttpRequest` (userscript). |
| 0x0.st | unverified | configurable | Not used. |
- **litterbox API:** `POST https://litterbox.catbox.moe/resources/internals/api.php`,
multipart `reqtype=fileupload`, `time=<1h|12h|24h|72h>`, `fileToUpload`;
the response body is the file URL.
- **catbox API:** `POST https://catbox.moe/user/api.php`, multipart
`reqtype=fileupload`, `fileToUpload`, optional `userhash` (attaches the
upload to a catbox account so it can be deleted later); the response body is
the file URL. Its `OPTIONS` preflight returns 412 with no CORS headers.
---
## 3. Pipeline
```
MangaBaka link ──▶ series id ──▶ GET /v1/series/{id}, /links, /images (direct; CORS open)
│ sanity check: a link has the same title_no
Webtoons link ──▶ source.resolve ───────┤
│ ▼
source.currentCovers + history (Wayback, via source.extractCovers)
│
▼ CoverCandidate[]
images.inspect (bytes → dimensions, sha256, liveness)
│
▼ InspectedCover[]
covers.dedupe (within candidates + against existing MangaBaka covers)
│
▼
seed.buildCoverSeed ──▶ CoverSeed (existing + new images, user's content rating)
│
▼ front end: review UI → (explicit action) → rehost → submit / fill form
```
Every network call goes through an injected `Transport` (§4.2), so every stage
runs unchanged in the server-backed page, the userscript, the CLI and tests.
---
## 4. Modules
```
src/
core/ pure TypeScript; no DOM, no Bun/Node APIs, no GM_*
model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
transport.ts Transport interface; per-host rate limit, retries, timeouts
pipeline.ts runs the stages above; the only place they are composed
links.ts parse the two input links (MangaBaka series id, source URL)
sources/
source.ts Source interface + helpers
registry.ts list of sources; findSource(url)
webtoons/
index.ts Webtoons Source implementation
parse.ts pure HTML → { series, covers } extraction
urls.ts URL parsing, canonicalisation, image URL normalisation
history/wayback.ts generic Wayback history provider (CDX + archived pages)
images.ts probe dimensions from header bytes, sha256 (WebCrypto), classify
covers.ts dedupe, merge observations, pick default
mangabaka/
client.ts typed API client (public reads + submissions)
types.ts API types (subset of the OpenAPI spec we use)
hosting/
host.ts ImageHost interface + registry
server.ts upload via our server → our storage (30 days)
litterbox.ts litterbox.catbox.moe, 1h–72h (direct; CORS open)
catbox.ts catbox.moe, permanent (via server, or GM in the userscript)
direct.ts no rehosting: submit source URLs as-is
seed/covers.ts CoverSeed builder: existing + new images → submission body
web/ the static page (HTML + TS bundled by `bun build`) + login page
server/
app.ts the portable fetch(Request) handler: gate, assets, /api/*
gate.ts (deferred) password check, signed session cookie, login rate limit
storage.ts Storage interface: S3-compatible bucket (R2 binding later)
worker.ts (deferred) Cloudflare Workers entry (R2 binding, static assets)
bun.ts Bun.serve entry for self-hosting (S3 storage, assets from dist/)
userscript/ entry + metadata banner, bundled by `bun build` into mbseed.user.js
cli/ later: Bun CLI on the same core
test/
fixtures/ saved HTML / CDX / JSON responses
*.test.ts bun test
```
Dependency direction: `web/`, `userscript/`, `server/` and `cli/` may import
`core/`; `core/` imports none of them. `server/` imports only the small part of
`core/` it needs (host allowlist, image sniffing). `server/app.ts` and
`server/gate.ts` use only web-standard APIs, so the same code runs in both
entries; `worker.ts` and `bun.ts` are thin adapters.
### 4.1 The `Source` interface (extension point)
```ts
export interface Source {
/** Stable id, e.g. "webtoons". Used in notes and output. */
readonly id: string;
/** Human-readable name, e.g. "WEBTOON". */
readonly name: string;
/** True if this source handles the URL. */
matches(url: URL): boolean;
/** Fetches the series page and returns normalised series info. */
resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;
/** Covers currently shown by the source. */
currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/**
* Optional: pure extraction of cover candidates from a series page's HTML.
* Sources that implement this get Wayback history for free.
*/
extractCovers?(html: string, pageUrl: URL): CoverCandidate[];
/** Optional source-specific history beyond Wayback (e.g. an API with old art). */
historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/** Normalises an image URL to its largest, hotlinkable form. */
normalizeImageUrl(url: string): string;
/** True if a MangaBaka series link points at this source series. */
isSameSeries(linkUrl: string, series: SourceSeries): boolean;
/** Hosts this source fetches from; the server proxy allowlist is built from these. */
readonly hosts: readonly string[];
}
export interface SourceSeries {
source: string; // Source.id
externalId: string; // e.g. Webtoons title_no "95"
title: string;
language: string; // MangaBaka language code, e.g. "en", "zh-Hant"
canonicalUrl: string;
archiveUrls: string[]; // URL prefixes to look up in Wayback
mature: boolean; // source flags the series as mature/age-gated (§4.7)
flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
}
export interface CoverCandidate {
url: string; // normalised
role: "poster" | "square" | "banner" | "foreground" | "other";
current: boolean; // shown on the live page now
seenAt?: string; // archive timestamp (YYYYMMDD…) if historical
archivedUrl?: string; // Wayback fallback for the image itself
}
```
**Adding a source** (e.g. Tapas): create `src/core/sources/tapas/` implementing
`Source` (HTML extraction in a pure `parse.ts` with fixture tests), add it to
`registry.ts`, and map its language codes to MangaBaka codes. Its `hosts` flow
into the server allowlist automatically.
### 4.2 Transport (how each front end reaches the network)
```ts
export interface Transport {
/** Fetch-like. Returns a standard Response. */
fetch(url: string, init?: { method?: string; headers?: Record<string, string>;
body?: BodyInit; signal?: AbortSignal }): Promise<Response>;
}
```
| Front end | MangaBaka API | Webtoons / Wayback / pstatic |
|---|---|---|
| Static page | `fetch` directly (CORS open) | `fetch("/api/fetch?url=…")` on our server (same origin; the session cookie rides along) |
| Userscript | `fetch` directly | `GM_xmlhttpRequest`, wrapped into a `Response` |
| CLI (later) | Bun `fetch` | Bun `fetch` |
| Tests | fake transport serving fixtures | fake transport serving fixtures |
Shared behaviour lives in `core/transport.ts` and wraps whichever transport is
injected:
- limits each host to 1 request per second, and Wayback to 1 every 1.5 s;
- times out after 20 s;
- retries twice with backoff on 429/5xx;
- never sends the MangaBaka token anywhere except `api.mangabaka.org`. In
particular the token never passes through our server.
### 4.3 History
`history/wayback.ts` knows nothing about any one source. For each prefix in
`series.archiveUrls` it:
1. queries CDX with `collapse=timestamp:6` (one snapshot per month, capped by a
history limit, default 60);
2. fetches each snapshot with the `id_` modifier;
3. calls `source.extractCovers(html, originalUrl)`;
4. normalises the image URLs and tags each candidate with `seenAt` and
`archivedUrl`.
Candidates that share a normalised URL are merged; the merged candidate records
the first and last time it was seen. Wayback failures (timeouts, 5xx, rate
limits) produce a warning and an empty history; they are never fatal. The user
can turn history off.
### 4.4 Image inspection and classification
`images.ts` downloads each candidate through the transport (live URL first,
then `archivedUrl`) and records `width`, `height`, `sha256` (WebCrypto
`crypto.subtle.digest`, available in browsers, Workers and Bun), `contentType`
and `live` status. Dimensions come from the image header bytes, so no DOM is
needed. Covers we've seen range from about 20 KB to 650 KB; downloads over
20 MB are refused. Candidates that cannot be downloaded at all are dropped with
a warning.
Classification by aspect ratio (h / w):
| Shape | Ratio | MangaBaka `type` |
|---|---|---|
| portrait | ≥ 1.15 | `volume` |
| square-ish | 0.85 – 1.15 | `other` |
| wide | < 0.5 | `banner` |
| otherwise | – | `other` |
### 4.5 Series and dedupe (`links.ts`, `covers.ts`)
- **Inputs.** `links.ts` accepts a MangaBaka series link or a bare id. Verified
forms: `https://mangabaka.org/9999` redirects (301) to the canonical
`https://mangabaka.org/manhwa/9999/Tower-of-God`; `/series/9999` is 404. So
take the first all-digits path segment. Plus a Webtoons series URL (`www.`
or `m.`).
- **Sanity check.** If none of the MangaBaka series' links passes
`source.isSameSeries` (same `title_no`), show a warning and require the user
to confirm. It is a warning, not an error: the link may simply be missing on
MangaBaka.
- **Duplicates.** Images with the same `sha256` are merged; the merged entry
keeps the live URL and the widest `seenAt` range.
- **Already on MangaBaka.** Candidates whose `sha256` matches an existing
MangaBaka image are dropped and shown as "already present".
- **Default selection.** The default is the current live poster, or the current
live square if there is none (canvas). The new default gets
`is_default: true` and every existing image gets `is_default: false`. The
reviewer can still reject that.
- **Notes.** Every new image gets a note, e.g. `"WEBTOON poster (current)"` or
`"WEBTOON poster, seen 2019-03 – 2021-01"`.
### 4.6 Image hosting
MangaBaka must be able to download each submitted image until a reviewer acts,
which may take weeks. The user picks the host per run; the UI shows each
host's expiry next to the button.
```ts
export interface ImageHost {
readonly id: "server" | "litterbox" | "catbox" | "direct";
/** Human-readable lifetime, shown in the UI ("30 days", "72 hours", "permanent"). */
lifetime(opts: HostOptions): string;
/** Rehosts one inspected image; returns a public URL. */
upload(image: { sourceUrl: string; bytes: Uint8Array; sha256: string; contentType: string },
ctx: HostContext): Promise<{ url: string }>;
}
```
| Host | Page | Userscript | Lifetime |
|---|---|---|---|
| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (bucket lifecycle rule) |
| `litterbox` (default otherwise) | multipart upload of the inspected bytes **directly from the browser** (CORS open), `time=72h` unless the user picks less | same, via `GM_xmlhttpRequest` | 1h / 12h / 24h / 72h |
| `catbox` | `POST /api/upload { url, host: "catbox", userhash? }` — the server fetches the image and uploads it, because catbox has no CORS | **directly** via `GM_xmlhttpRequest`, no server needed | permanent |
| `direct` | no upload: the normalised source URL | same | as long as Webtoons keeps it |
- **Server storage** stores at `covers/<sha256>.<ext>` and returns the public
URL. On Cloudflare it is an R2 binding; self-hosted it is any S3-compatible
bucket (R2, B2, MinIO, AWS) via `aws4fetch`. Either way the bucket has a
30-day lifecycle rule and is publicly readable (custom domain or `r2.dev`).
If the server has no storage configured, `GET /api/config` says so and the
page hides the `server` host.
- **litterbox 72h warning.** The UI warns that a review may take longer than
72 hours and the links will then be dead.
- **catbox userhash.** Optional, entered by the user, kept in the browser
(`localStorage` only if "remember" is ticked) or in `GM_setValue`. Through
the page it passes through our server, which forwards it to catbox only and
never logs or stores it.
Rules:
- **Upload only what we checked.** For hosts reached through our server, the
client sends the source URL, not bytes, so `/api/upload` can't be used to
host arbitrary files; the server returns the `sha256` of what it fetched and
the client rejects a mismatch. Hosts reached directly get the inspected bytes.
- **Re-uploads are cheap.** Server storage is keyed by hash, so rerunning
overwrites the same object (and restarts its 30 days). For litterbox and
catbox, the page keeps a per-session `sha256 → url` map and the userscript a
persistent one (`GM_setValue`), so the same image isn't uploaded twice.
- **No upload without an explicit user action.**
- **Failures are per image.** A failed upload is reported and the image left
out; if every upload fails, the action aborts.
- **Adding a host** is one file in `core/hosting/` plus one registry line.
### 4.7 Content rating
MangaBaka needs a `content_rating` per image. The tool can't judge it, so **the
user chooses it**, default `safe`: one selector for all new images, plus a
per-card override. If the source marks the series as mature (Webtoons
`ageGradeNotice : true`), the UI shows a warning to consider a rating above
`safe`. The default stays `safe`. Unattended runs (CLI `--yes`, later) must
pass a rating explicitly.
### 4.8 Seed (`seed/covers.ts`)
`buildCoverSeed(existing, newCovers, series, { contentRating })` returns:
```ts
interface CoverSeed {
seriesId: number;
userNote: string; // "Add WEBTOON covers from <canonicalUrl>"
images: SubmissionImage[]; // existing (unchanged except is_default) + new
added: InspectedCover[]; // for display
skipped: { cover: InspectedCover; reason: string }[];
}
```
Each new image becomes
`{ id: null, series_id, work_id: null, url, index: null, index_numeric: null,
language: series.language, type, note, content_rating, is_default }`.
`withHostedUrls(seed, urlMap)` swaps in the rehosted URLs after upload.
### 4.9 Front ends
**Static page** (`src/web/`, served by our server):
1. Two inputs: MangaBaka series link, Webtoons series link. Optional: "include
history" (on), history limit.
2. Runs the pipeline and shows one card per new cover (thumbnail, dimensions,
suggested type, language, note, content-rating selector, checkbox), plus
"already present" and skipped covers.
3. **Rehost selected** uploads to the chosen host (§4.6) and replaces the
shown URLs.
4. **Without a token:** "Copy all selected" and a link to
`https://mangabaka.org/data/submissions/series/{id}/images` for manual
entry.
5. **With a token:** the user pastes a personal access token (kept only in
memory unless they tick "remember on this device", then `localStorage`).
The page fetches the snapshot, shows the preview diff, and on confirmation
rehosts and saves with `save_mode: "review"`. 409: refetch, rebuild,
re-preview, ask again (once). 403: explain the 25-approved-submissions rule
and fall back to manual entry.
**Userscript** (`src/userscript/`, built to `mbseed.user.js`):
- `@match https://mangabaka.org/*`. On a series page it adds a "Seed covers"
panel; the MangaBaka series id comes from the page URL and the user pastes
the Webtoons link.
- Fetches Webtoons/Wayback/pstatic with `GM_xmlhttpRequest`
(`@connect webtoons.com`, `pstatic.net`, `web.archive.org`, `catbox.moe`,
and our server's host). Same review cards as the page.
- Rehosts to litterbox or catbox directly, or to our server's storage
(`/api/upload`; no storage keys in the script). With
litterbox or catbox it works with **no server at all**.
- Submits through the API when a token is set (stored with `GM_setValue`).
- **Without a token**, on the cover form page it can fill the form fields
for the selected covers. Whether the form accepts programmatically filled
fields is unverified (§7).
### 4.10 Server (`src/server/`)
| Route | Behaviour |
|---|---|
| `GET /*` | Static assets of the page. |
| `GET /api/config` | `{ storage: boolean }`, so the page knows which hosts to offer. |
| `GET /api/fetch?url=` | Fetches `url` only if its host is in the allowlist (union of `Source.hosts` plus `web.archive.org`), with a browser User-Agent; streams the response back. Anything else: 403. |
| `POST /api/upload` | `{ url, host: "store" \| "catbox", userhash? }` (§4.6). `url` must pass the same allowlist. |
The API routes reject other origins (except the userscript, which is not bound
by CORS), cap response size at 20 MB, and are rate-limited per IP.
**One handler; Bun now, Cloudflare later.** `app.ts` exports
`handle(req: Request, env: ServerEnv): Promise<Response>` using only
web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`):
| | Cloudflare (`worker.ts`, deferred) | Self-hosted (`bun.ts`, **now**) |
|---|---|---|
| Run | `wrangler dev` / `wrangler deploy` | `bun run src/server/bun.ts` behind our reverse proxy |
| Static assets | Workers static assets, with `run_worker_first = true` so the gate also covers assets (confirm the key when scaffolding) | read from the built `dist/web/` |
| Storage | R2 binding | S3-compatible bucket via `MBSEED_S3_*` env vars, or none |
| Password | `wrangler secret put MBSEED_PASSWORD` | `MBSEED_PASSWORD` env var |
| Client IP for rate limits | `CF-Connecting-IP` | `server.requestIP(req)`, or `X-Forwarded-For` only when `MBSEED_TRUST_PROXY=1` |
Self-hosting also avoids the Cloudflare-egress risk. Our reverse proxy
terminates TLS; how `X-Forwarded-For` is trusted is set by `MBSEED_TRUST_PROXY`.
### 4.11 Access gate (`server/gate.ts`) — deferred
**Not built in the first version.** Decided 2026-10-14: the self-hosted site
starts open, and we add this if it gets abused or traffic grows too large.
Until then the server's protections are the host allowlist, the 20 MB cap and
the per-IP rate limit. The design below is kept for when it's needed; until
then there is no `/login`, and `/api/config` has no `gated` field.
An optional shared password locks the **whole** service: page, assets and API.
- **Off by default.** If `MBSEED_PASSWORD` is unset, the gate is open and
`/api/config` reports `gated: false`. A self-hosted instance can instead rely
on the reverse proxy's own auth (e.g. basic auth or SSO) and leave it unset.
- **Login.** `GET /login` serves a minimal form. `POST /login` compares the
password in constant time, then sets `mbseed_session=<expiry>.<hmac>`
(`HttpOnly; Secure; SameSite=Strict; Path=/`, 30 days). The HMAC key is
derived from `MBSEED_PASSWORD`, so changing the password logs everyone out.
Login attempts are rate-limited per IP (5 per minute).
- **Every other route** requires a valid, unexpired session cookie: browser
page requests without one get a 302 to `/login`, and API requests get 401.
- **Userscript.** It can't rely on the cookie, so API calls may instead send
`Authorization: Bearer <password>` (constant-time compare). The userscript
stores the password with `GM_setValue`. It needs this only for
`/api/upload` with server storage or catbox-via-server; with litterbox or
direct catbox it never calls our server.
- **No accounts.** One password for the group; rotate it to revoke access.
---
## 5. Cross-cutting concerns
- **Secrets.** The MangaBaka token stays in the user's browser (or userscript
storage) and is only sent to `api.mangabaka.org`. The server's only secrets
are the `MBSEED_S3_*` credentials (and `MBSEED_PASSWORD` once the gate
exists). The catbox userhash is forwarded, never
stored.
- **Errors.** The UI shows a per-stage status; Wayback failure is a warning,
not a failure.
- **Load.** A run is ~60–100 proxied requests and ≤ ~6 MB stored for 30 days.
Abuse of the open proxy is the main risk, hence the allowlist, the size cap
and the per-IP rate limit; the gate (§4.11) is the next step if needed.
(Earlier estimate for Cloudflare: $0 on free tiers; ~$5/month for a busy
public instance.)
## 6. Testing
- **`bun test`** against fixtures saved from real responses: Webtoons pages
(official, canvas, mature, a 2015 Wayback snapshot), CDX output, MangaBaka
series/links/images/snapshot/preview JSON.
- Pure functions under test: link parsing, Webtoons parsing, URL normalisation,
classification, dedupe, default selection, seed building, content-rating
resolution, language mapping, the server allowlist, session cookie
signing/expiry.
- I/O (`pipeline`, `mangabaka/client`, `history/wayback`, every `hosting/*`,
`server/app.ts` routes incl. the gate: no cookie → 302/401, bad password,
expired cookie, Bearer header) with a fake transport and in-memory storage.
**No network in tests.**
- Manual smoke test: the page against Tower of God (MangaBaka `9999`, Webtoons
`title_no=95`), with "Rehost" and without submitting.
## 7. Open questions and risks
1. **Server egress.** Webtoons or Wayback may throttle our server's IP under
load. (Cloudflare IPs would be riskier; the userscript uses the user's IP.)
2. **When MangaBaka copies images** (submission or approval) and whether it
accepts our storage URLs, `litter.catbox.moe` / `files.catbox.moe` URLs, or
`swebtoon-phinf.pstatic.net` (`direct`). If it copies only on approval,
litterbox's 72h is likely too short.
3. **"Main cover" = `is_default: true`** is inferred from the API spec.
4. **Where users create a personal access token** in the MangaBaka UI.
5. **Whether the cover form accepts fields filled by a userscript.**
6. **Whether catbox accepts uploads from our server** (catbox-via-server).
7. **Hosting details:** which of our servers and domain, and the storage
behind `server` — an S3-compatible bucket (MinIO, R2, B2…) as specified, or,
if simpler on our box, a local directory with a daily 30-day sweep (decide
in the plan; it would be a `Storage` implementation, nothing else changes).
## 8. Test data
Tower of God historical `og:image`s:
```
2015 http://swebtoon.phinf.naver.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg
2017 http://swebtoon.phinf.naver.net/20151006_141/1444125046528RTfRM_JPEG/_EB93BBEB80AF_EBA38CEABCA5_E29587EABCB1_EB9384EB84B0_ipa.jpg
(twitter:image …/20151006_243/1444125042491PuNza_JPEG/…_mob.jpg)
2019 https://swebtoon-phinf.pstatic.net/20190318_291/1552868599909GoVLY_JPEG/10_EC8DB8EB84A4EC9DBC_ipad.jpg
(twitter:image …/20190318_26/1552868584759vl9A1_JPEG/11_EC8DB8EB84A4EC9DBC_mobile.jpg)
2021 https://swebtoon-phinf.pstatic.net/20210127_106/1611705089085OYcUB_JPEG/1Tower-of-God-Mobile-Landing-Page.jpg
2023+ https://swebtoon-phinf.pstatic.net/20230324_114/1679612146968KhLLz_JPEG/6TowerOfGod_landingpage_mobile.jpg
live https://swebtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg (poster, 480×623)
```
The 2014, 2015 and 2019 images are still live on `swebtoon-phinf.pstatic.net`;
the 2021 one is gone from Wayback's own copy but live on pstatic.
Other test inputs:
- canvas: `https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372`
- mature: title_no `6065` (filename contains spaces and `)`; URL-encode it)
- FR example: Violet Romance, `2Thumb_Poster_9782.jpg`.
## 9. References
- yambs (the model project, Go): `/home/user/yambs` on the maintainer's machine, with the same
sources → seed → actions shape.
- MangaBaka API docs: https://mangabaka.org/data/api; explorer: https://mangabaka.org/data/api/explorer;
support: https://mangabaka.org/discord.
- The maintainer is a regular MangaBaka contributor without API access (fewer than 25 approved
submissions as of 2026-10-14), so the manual path must be first-class.
Reference in new issue View Git Blame Copy Permalink