lucasw89/mangabaka-seeder
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
df7e1bcd42d43f197a9cb809f4f2410ff939bdf5
mangabaka-seeder/docs/specs/001-webtoons-covers.md
T
Lucas WintherandClaude Opus 5.5 3e927099c5 docs: amend spec 001 from tests against real Webtoons, Wayback and MangaBaka 
History granularity, honest Wayback User-Agent and 10 s pacing, tall
landing-art class, one default cover per series, paginated public
images. Ignore samples/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:03:15 +02:00

760 lines
40 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
status: approved
---
# 001 — Webtoons covers
`mangabaka-seeder` gathers data about a series from an external site (a
"source") and turns it into an edit request ("submission") for
[MangaBaka](https://mangabaka.org). It is modelled on
[yambs](https://codeberg.org/derat/yambs), the MusicBrainz seeder: sources turn
into seeds, and an action decides what happens to them. The user always stays
in control of what is submitted.
The first feature is **covers**: given a MangaBaka series link and a Webtoons
series link, collect every cover the Webtoons series has used (current and
historical) and request them added to the MangaBaka series, with the current
poster as the default (main) cover for the edition's language.
This spec records the facts we verified about the external services, the
module boundaries and the extension points. Revised 2026-10-14: the user gives
both links directly, the product is a static web page first plus a userscript,
the runtime is Bun, and images go to a temporary host with a 30-day expiry.
Revised again the same day: catbox.moe and litterbox are also supported as
image hosts, the server can be password-gated, and the same server code runs
on Cloudflare Workers or under Bun on our own machine.
Signed off by the user 2026-10-14, with two changes applied here: the server
is **self-hosted** (Bun), and the **password gate is deferred** until abuse or
traffic calls for it (§4.11 keeps the design). The Cloudflare entry is
deferred too; the handler stays web-standard so it can be added later.
Amended 2026-10-14 after testing against the real services (approved by the
user): history granularity instead of a snapshot limit, a "tall" landing-art
class, an honest User-Agent and 10 s pacing for Wayback, and one default cover
per series (§2.2, §2.3, §4.2–§4.5).
---
## 1. Goals and non-goals
**Goals**
- **Two links in, a reviewable cover submission out.** The user pastes a
MangaBaka series link and a Webtoons series link. There is no automatic
series matching.
- **Two front ends on one core:**
1. a **static web page** (the primary product), served by our small server,
which also proxies the sites browsers cannot read;
2. a **userscript** (Tampermonkey/Violentmonkey) on mangabaka.org, which needs
no proxy because `GM_xmlhttpRequest` is not bound by CORS.
A Bun CLI on the same core comes later.
- **Self-hosted server.** The server is a single web-standard `fetch(Request) →
Response` handler run by `Bun.serve` on our own machine, behind our reverse
proxy (§4.10). Because it is web-standard, a Cloudflare Workers entry can be
added later without touching it.
- **Choice of image host:** our own storage (30-day expiry), litterbox (up to
72h), catbox.moe (permanent), or no rehosting (§4.6).
- **Find historical covers**, not just the current one.
- **Work for everyone.** With a MangaBaka personal access token, submit through
the API. Without one, help the user enter the covers by hand.
- **New sources** (Tapas, Tappytoon, Lezhin, Manta, …) are one new module under
`src/core/sources/`, with no changes to the pipeline, the MangaBaka client or
the front ends.
- **Never change MangaBaka data silently.** API submissions always go through
preview, explicit confirmation and `save_mode: "review"`.
- **Never upload anything without explicit user action.** The user chooses the
content rating (default `safe`).
**Non-goals (for now)**
- Editing series fields other than images, such as titles, links or tags. The
submission layer is keyed by MangaBaka "resource" so this can come later.
- Following a Webtoons series across language editions. Each `title_no` belongs
to one language edition; run the tool once per edition link.
- Running our own permanent image host. Our storage only needs to outlive the
MangaBaka review, so it expires after 30 days; catbox.moe is the option for
anyone who wants permanent links (§4.6).
- Access control, for now. The site is open; the password gate (§4.11) is
designed but deferred until abuse or traffic calls for it.
- A Cloudflare Workers deployment, for now (§4.10).
- User accounts, ever: if a gate is added, it is one shared password.
---
## 2. Verified external facts
Verified on 2026-10-14. Treat them as assumptions that the fixtures in
`test/fixtures/` pin down.
### 2.1 Webtoons
| What | Where |
|---|---|
| Portrait poster (official series; 480×623 original) | `meta[property="og:image"]` |
| Square thumbnail | `meta[name="twitter:image"]` |
| Hero background (1920×321, official only) | `div.detail_bg` inline `style` → `background:url('…')` |
| Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) | `.detail_header span.thmb img[src]` |
| Title | `meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`) |
| Canonical URL | `link[rel="canonical"]` |
| `title_no` | `title_no` query parameter |
| Language | first path segment (`/en/`, `/fr/`, `/zh-hant/`, …) |
| Canvas vs official | `/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true` |
| Mature | inline JS `ageGradeNotice : true` |
- **Canvas series** have no portrait poster. Their `og:image` is a 1080×1080
square.
- **Image hosts.** `webtoon-phinf.pstatic.net` returns 403 without a Webtoons
`Referer`; `swebtoon-phinf.pstatic.net` serves the same path with no Referer.
- **Original size.** The `?type=…` query selects a resized variant. Removing it
returns the original. `crop540_540` crops posters, so always strip the query.
- **Old hosts.** Old snapshots reference `swebtoon.phinf.naver.net`, which
becomes `swebtoon-phinf.pstatic.net` with the same path.
- **No limits so far.** A browser User-Agent is enough. Age-gated titles still
serve full HTML, and no cookies are required. (Tested from a residential IP,
not from Cloudflare's; see §7.)
- **No usable JSON API.** The app API needs signed requests. We scrape HTML.
- **CORS:** neither the series HTML nor `swebtoon-phinf` images send
`Access-Control-Allow-Origin`. A page can show the images in `<img>` but
cannot read their bytes.
### 2.2 Historical covers (Wayback Machine)
- **Snapshot index.** The CDX API at
`https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original`
returns at most one snapshot per month.
- **Raw pages.** `https://web.archive.org/web/<ts>id_/<original>` returns the
page as it was archived, without Wayback's toolbar.
- **History exists.** Tower of God has about 2000 snapshots (2015–2026) that
reference five distinct cover images. All the older ones tested are **still
served live** from `swebtoon-phinf.pstatic.net`.
- **Fallback.** If an image is no longer live, it is often still at
`https://web.archive.org/web/<ts>id_/<image-url>`.
- **Not every image is a cover.** Some historical `og:image`s are landing-page
art, so candidates are classified by shape and reviewed by the user.
- **CORS:** no `Access-Control-Allow-Origin` seen on snapshots. CDX returned
503 during the check, so treat Wayback as unavailable from a plain page and
flaky in general.
- **Bot detection (tested 2026-10-14).** Requests sending a fake browser
User-Agent from a non-browser client were answered `429` with the body
"flagged … as suspected abusive bot traffic", even at one request per 10 s.
With an honest User-Agent (`mangabaka-seeder/<version> (+<repo URL>)`),
2 CDX lookups and 24 snapshots at one request per 10 s ran with no `429` at
all. So Wayback always gets the honest User-Agent; only Webtoons gets a
browser one.
- **Mobile pages.** `m.webtoons.com` has its own captures (35 months for Tower
of God, against 121 for `www.`); they fill gaps in the `www.` history.
- **What the history holds (Tower of God, every 6 months, 2015–2026):**
20 distinct images, all still live on `swebtoon-phinf.pstatic.net`. The
2015–2020 `og:image`s are 510×510 character art, the 2020-10 – 2026-03 ones
are 720×1230 landing-page art with an empty lower half, and only the current
one (480×623) is a cover with the logo. Hence the "tall" class in §4.4.
### 2.3 MangaBaka
- **Endpoints.** The site is `https://mangabaka.org` and the API is
`https://api.mangabaka.org` (`api.mangabaka.dev` is deprecated). The OpenAPI
spec is at `https://mangabaka.org/api.json` (~700 KB), with docs at
`https://mangabaka.org/data/api`.
- **CORS is open.** `api.mangabaka.org` answers with
`access-control-allow-origin: *` and allows any header (including
`x-api-key`) and `POST`, on both public reads and
`/v0/my/submissions/…`. So the page and the userscript talk to MangaBaka
**directly**, never through our server.
- **Public reads.**
- `GET /v1/series/{id}` and `GET /v1/series/{id}/links` return the series
and its links. Links are canonicalised, e.g.
`https://www.webtoons.com/-/-/-/list?title_no=95`.
- `GET /v1/series/{id}/images` returns covers with `id`, `type`, `language`,
`index`, `note`, `content_rating` and `hashes.sha256`, but **not**
`is_default`. It is **paginated** (`limit` ≤ 50; Tower of God has 31
images, `pagination.next` links the next page), so the client follows
`next` until it is `null`.
- Rate limits: about 180 requests per minute (search, which we no longer
use, is about 30).
- **Image `type` values:** `volume` (default), `volume_back`, `chapter`,
`season`, `banner`, `audiobook`, `other`. There is no "main" type; the main
cover is the one with `is_default: true`.
- **Submissions API.** Auth is `x-api-key: mb-…` (a personal access token).
All endpoints are v0 and marked internal.
- `GET /v0/my/submissions/series-images/{id}` returns the current image set
and a `version` token.
- `POST /v0/my/submissions/series-images/{id}/preview` is a dry run that
returns a diff.
- `POST /v0/my/submissions/series-images/{id}` saves, with body
`{ version, user_note, save_mode, data: { images: [...] } }`.
- `images` **replaces the whole set**, so it must contain the existing images.
- **Exactly one image in the set must have `is_default: true`** when the set
is non-empty (OpenAPI `series-images` schema). There is one main cover per
*series*, not per language.
- Snapshot: `{ data: { data: { images: [...] }, version } }`. Each image has
`id`, `series_id`, `work_id`, `url`, `index`, `index_numeric`, `language`,
`type`, `note`, `content_rating`, `is_default`; the keys `work_id`, `note`
and `index_numeric` are required even when `null`.
- Preview answers `{ data: { has_changes, changes: SubmissionDiff[] } }`.
- A stale `version` gives 409.
- Users without the contributor role and with fewer than 25 approved
submissions get 403.
- **Web form.** Without API access, users use
`https://mangabaka.org/data/submissions/series/{id}/images`. It cannot be
pre-filled with query parameters, and its SvelteKit CSRF check blocks
cross-site POSTs. (A userscript running *on* that page can still fill its
fields; see §4.9.)
- **Image URLs.** The spec describes the cover `url` as "Cover image URL. imgur
is rejected (aggressive rate limiting). Acts as the cover identity on save."
We don't know whether MangaBaka copies the image on submission or on
approval, so the hosted URL must survive the review (§4.6).
- **`content_rating`:** one of `safe`, `suggestive`, `erotica`,
`pornographic`. The user chooses it (§4.7).
- **Test series:** Tower of God is **series 9999**
(`https://mangabaka.org/manhwa/9999/Tower-of-God`; Webtoons `title_no=95`).
### 2.4 Image hosts
| Host | Browser CORS | Expiry | Verdict |
|---|---|---|---|
| **Our storage** (a folder on our server) | n/a (served by our server) | 30 days (daily sweep) | **Default** when configured. |
| **litterbox.catbox.moe** | **yes** (`access-control-allow-origin: *`) | `time` = `1h` / `12h` / `24h` / `72h` only | Supported; no account; 72h may be shorter than a review. |
| **catbox.moe** | no | permanent | Supported via our server (page) or `GM_xmlhttpRequest` (userscript). |
| 0x0.st | unverified | configurable | Not used. |
- **litterbox API:** `POST https://litterbox.catbox.moe/resources/internals/api.php`,
multipart `reqtype=fileupload`, `time=<1h|12h|24h|72h>`, `fileToUpload`;
the response body is the file URL.
- **catbox API:** `POST https://catbox.moe/user/api.php`, multipart
`reqtype=fileupload`, `fileToUpload`, optional `userhash` (attaches the
upload to a catbox account so it can be deleted later); the response body is
the file URL. Its `OPTIONS` preflight returns 412 with no CORS headers.
---
## 3. Pipeline
```
MangaBaka link ──▶ series id ──▶ GET /v1/series/{id}, /links, /images (direct; CORS open)
│ sanity check: a link has the same title_no
Webtoons link ──▶ source.resolve ───────┤
│ ▼
source.currentCovers + history (Wayback, via source.extractCovers)
│
▼ CoverCandidate[]
images.inspect (bytes → dimensions, sha256, liveness)
│
▼ InspectedCover[]
covers.dedupe (within candidates + against existing MangaBaka covers)
│
▼
seed.buildCoverSeed ──▶ CoverSeed (existing + new images, user's content rating)
│
▼ front end: review UI → (explicit action) → rehost → submit / fill form
```
Every network call goes through an injected `Transport` (§4.2), so every stage
runs unchanged in the server-backed page, the userscript, the CLI and tests.
---
## 4. Modules
```
src/
core/ pure TypeScript; no DOM, no Bun/Node APIs, no GM_*
model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
transport.ts Transport interface; per-host rate limit, retries, timeouts
pipeline.ts runs the stages above; the only place they are composed
links.ts parse the two input links (MangaBaka series id, source URL)
sources/
source.ts Source interface + helpers
registry.ts list of sources; findSource(url)
webtoons/
index.ts Webtoons Source implementation
parse.ts pure HTML → { series, covers } extraction
urls.ts URL parsing, canonicalisation, image URL normalisation
history/wayback.ts generic Wayback history provider (CDX + archived pages)
images.ts probe dimensions from header bytes, sha256 (WebCrypto), classify
covers.ts dedupe, merge observations, pick default
mangabaka/
client.ts typed API client (public reads + submissions)
types.ts API types (subset of the OpenAPI spec we use)
hosting/
host.ts ImageHost interface + registry
server.ts upload via our server → its covers folder (30 days)
litterbox.ts litterbox.catbox.moe, 1h–72h (direct; CORS open)
catbox.ts catbox.moe, permanent (via server, or GM in the userscript)
direct.ts no rehosting: submit source URLs as-is
seed/covers.ts CoverSeed builder: existing + new images → submission body
web/ the static page (HTML + TS bundled by `bun build`) + login page
server/
app.ts the portable fetch(Request) handler: gate, assets, /api/*
gate.ts (deferred) password check, signed session cookie, login rate limit
storage.ts Storage interface (put/get/sweep) + in-memory impl for tests
storage-dir.ts folder on disk (Bun file APIs; used only by bun.ts)
worker.ts (deferred) Cloudflare Workers entry (R2 binding, static assets)
bun.ts Bun.serve entry for self-hosting (folder storage + daily sweep, assets from dist/)
userscript/ entry + metadata banner, bundled by `bun build` into mbseed.user.js
cli/ later: Bun CLI on the same core
test/
fixtures/ saved HTML / CDX / JSON responses
*.test.ts bun test
```
Dependency direction: `web/`, `userscript/`, `server/` and `cli/` may import
`core/`; `core/` imports none of them. `server/` imports only the small part of
`core/` it needs (host allowlist, image sniffing). `server/app.ts` and
`server/gate.ts` use only web-standard APIs, so the same code runs in both
entries; `worker.ts` and `bun.ts` are thin adapters.
### 4.1 The `Source` interface (extension point)
```ts
export interface Source {
/** Stable id, e.g. "webtoons". Used in notes and output. */
readonly id: string;
/** Human-readable name, e.g. "WEBTOON". */
readonly name: string;
/** True if this source handles the URL. */
matches(url: URL): boolean;
/** Fetches the series page and returns normalised series info. */
resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;
/** Covers currently shown by the source. */
currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/**
* Optional: pure extraction of cover candidates from a series page's HTML.
* Sources that implement this get Wayback history for free.
*/
extractCovers?(html: string, pageUrl: URL): CoverCandidate[];
/** Optional source-specific history beyond Wayback (e.g. an API with old art). */
historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/** Normalises an image URL to its largest, hotlinkable form. */
normalizeImageUrl(url: string): string;
/** True if a MangaBaka series link points at this source series. */
isSameSeries(linkUrl: string, series: SourceSeries): boolean;
/** Hosts this source fetches from; the server proxy allowlist is built from these. */
readonly hosts: readonly string[];
}
export interface SourceSeries {
source: string; // Source.id
externalId: string; // e.g. Webtoons title_no "95"
title: string;
language: string; // MangaBaka language code, e.g. "en", "zh-Hant"
canonicalUrl: string;
archiveUrls: string[]; // URL prefixes to look up in Wayback
mature: boolean; // source flags the series as mature/age-gated (§4.7)
flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
}
export interface CoverCandidate {
url: string; // normalised
role: "poster" | "square" | "banner" | "foreground" | "other";
current: boolean; // shown on the live page now
seenAt?: string; // archive timestamp (YYYYMMDD…) if historical
archivedUrl?: string; // Wayback fallback for the image itself
}
```
**Adding a source** (e.g. Tapas): create `src/core/sources/tapas/` implementing
`Source` (HTML extraction in a pure `parse.ts` with fixture tests), add it to
`registry.ts`, and map its language codes to MangaBaka codes. Its `hosts` flow
into the server allowlist automatically.
### 4.2 Transport (how each front end reaches the network)
```ts
export interface Transport {
/** Fetch-like. Returns a standard Response. */
fetch(url: string, init?: { method?: string; headers?: Record<string, string>;
body?: BodyInit; signal?: AbortSignal }): Promise<Response>;
}
```
| Front end | MangaBaka API | Webtoons / Wayback / pstatic |
|---|---|---|
| Static page | `fetch` directly (CORS open) | `fetch("/api/fetch?url=…")` on our server (same origin; the session cookie rides along) |
| Userscript | `fetch` directly | `GM_xmlhttpRequest`, wrapped into a `Response` |
| CLI (later) | Bun `fetch` | Bun `fetch` |
| Tests | fake transport serving fixtures | fake transport serving fixtures |
Shared behaviour lives in `core/transport.ts` and wraps whichever transport is
injected:
- limits each host to 1 request per second, and Wayback to 1 every 10 s;
- times out after 20 s (uploads may ask for longer);
- retries twice with backoff on 429/5xx, honouring `Retry-After`, but **only
for `GET`/`HEAD`**: a retried `POST` could file a second edit request or
upload twice;
- sends Wayback the honest User-Agent (§2.2). (Browsers don't let a page set
`User-Agent`; the server proxy and the userscript set it per host.)
- never sends the MangaBaka token anywhere except `api.mangabaka.org`. In
particular the token never passes through our server.
### 4.3 History
`history/wayback.ts` knows nothing about any one source. For each prefix in
`series.archiveUrls` it:
1. queries CDX with `collapse=timestamp:6`, which lists every month that has a
capture (one cheap request per prefix), and drops rows whose page is not
the same series (`matchType=prefix` on `title_no=95` also matches
`title_no=950`);
2. splits the span from the first capture to now into buckets of the user's
**granularity** — every 1, 3, 6 (default) or 12 months — and keeps one
capture per bucket, preferring `www.` over `m.`;
3. fetches each chosen snapshot with the `id_` modifier;
4. calls `source.extractCovers(html, originalUrl)`;
5. normalises the image URLs and tags each candidate with `seenAt` and
`archivedUrl`.
Tower of God at 6 months is 2 CDX lookups and 24 snapshots, about 4–5 minutes
at one request per 10 s, so the UI shows progress and can cancel; a cancelled
run keeps what it found.
Candidates that share a normalised URL are merged; the merged candidate records
the first and last time it was seen. Wayback failures (timeouts, 5xx, rate
limits) produce a warning, never a fatal error. A snapshot that still fails
after its retries **stops the history** (Wayback is blocking or down; waiting
on each remaining snapshot would take minutes apiece) and keeps the covers
found so far. The user can turn history off.
### 4.4 Image inspection and classification
`images.ts` downloads each candidate through the transport (live URL first,
then `archivedUrl`) and records `width`, `height`, `sha256` (WebCrypto
`crypto.subtle.digest`, available in browsers, Workers and Bun), `contentType`
and `live` status. Dimensions come from the image header bytes, so no DOM is
needed. Covers we've seen range from about 20 KB to 650 KB; downloads over
20 MB are refused. Candidates that cannot be downloaded at all are dropped with
a warning.
Classification by aspect ratio (h / w):
| Shape | Ratio | MangaBaka `type` | Selected by default |
|---|---|---|---|
| tall (landing-page art) | > 1.6 | `other` | no — labelled "landing-page art" |
| portrait | 1.15 – 1.6 | `volume` | yes |
| square-ish | 0.85 – 1.15 | `other` | yes |
| wide | < 0.5 | `banner` | yes |
| otherwise | – | `other` | yes |
Real Webtoons posters are about 1.3 (480×623); landing-page art is 1.7
(720×1230) with an empty lower half (§2.2).
### 4.5 Series and dedupe (`links.ts`, `covers.ts`)
- **Inputs.** `links.ts` accepts a MangaBaka series link or a bare id. Verified
forms: `https://mangabaka.org/9999` redirects (301) to the canonical
`https://mangabaka.org/manhwa/9999/Tower-of-God`; `/series/9999` is 404. So
take the first all-digits path segment. Plus a Webtoons series URL (`www.`
or `m.`).
- **Sanity check.** If none of the MangaBaka series' links passes
`source.isSameSeries` (same `title_no`), show a warning and require the user
to confirm. It is a warning, not an error: the link may simply be missing on
MangaBaka.
- **Duplicates.** Images with the same `sha256` are merged; the merged entry
keeps the live URL and the widest `seenAt` range.
- **Already on MangaBaka.** Candidates whose `sha256` matches an existing
MangaBaka image are dropped and shown as "already present".
- **Default selection.** The suggested default is the current live poster, or
the current live square if there is none (canvas). MangaBaka has **one**
default per series (§2.3), so making a Webtoons edition's poster the default
replaces the series' main cover for every language. The UI therefore shows
a checkbox "Make this the series' main cover", on by default, with that
warning. When on, that image gets `is_default: true` and every other image
`false`; when the suggested image is already on MangaBaka, the existing
image is the one marked. When off, existing defaults are left alone. The
reviewer can still reject it.
- **Notes.** Every new image gets a note, e.g. `"WEBTOON poster (current)"` or
`"WEBTOON poster, seen 2019-03 – 2021-01"`.
### 4.6 Image hosting
MangaBaka must be able to download each submitted image until a reviewer acts,
which may take weeks. The user picks the host per run; the UI shows each
host's expiry next to the button.
```ts
export interface ImageHost {
readonly id: "server" | "litterbox" | "catbox" | "direct";
/** Human-readable lifetime, shown in the UI ("30 days", "72 hours", "permanent"). */
lifetime(opts: HostOptions): string;
/** Rehosts one inspected image; returns a public URL. */
upload(image: { sourceUrl: string; bytes: Uint8Array; sha256: string; contentType: string },
ctx: HostContext): Promise<{ url: string }>;
}
```
| Host | Page | Userscript | Lifetime |
|---|---|---|---|
| `server` (default when the server has storage) | `POST /api/upload { url, host: "store" }` | same | 30 days (daily sweep) |
| `litterbox` (default otherwise) | multipart upload of the inspected bytes **directly from the browser** (CORS open), `time=72h` unless the user picks less | same, via `GM_xmlhttpRequest` | 1h / 12h / 24h / 72h |
| `catbox` | `POST /api/upload { url, host: "catbox", userhash? }` — the server fetches the image and uploads it, because catbox has no CORS | **directly** via `GM_xmlhttpRequest`, no server needed | permanent |
| `direct` | no upload: the normalised source URL | same | as long as Webtoons keeps it |
- **Server storage** is, for now, **a plain folder on our server**
(`MBSEED_STORAGE_DIR`). Files are named `<sha256>.<ext>` and served by our
server at `GET /covers/<sha256>.<ext>`; the returned URL is
`<MBSEED_PUBLIC_URL>/covers/<sha256>.<ext>`, so the server must be reachable
from the internet for MangaBaka to fetch it.
- Writes go to a temp file in the same folder, then `rename`, so a reader
never sees half a file.
- **Expiry:** `bun.ts` sweeps the folder at startup and every 24 hours,
deleting files whose modification time is older than 30 days. A re-upload
rewrites the file, which restarts its 30 days.
- **Size cap.** The folder's total size is capped by
`MBSEED_STORAGE_MAX_BYTES` (default 1 GiB, about 170 worst-case runs per
30 days). `bun.ts` measures the folder at startup and keeps a running total
in memory, updated on every write and sweep; uploads are serialised so two
can't race past the cap. Re-uploading a file that already exists doesn't
count again.
- **When full, refuse; never evict.** An upload that would exceed the cap
gets `507 Insufficient Storage`, and nothing is deleted early: older files
may still be waiting for a MangaBaka reviewer. Space comes back as the
daily sweep expires files. The page then offers litterbox or catbox
instead.
- The per-image 20 MB cap still applies.
- `GET /covers/*` accepts only `^[0-9a-f]{64}\.(jpg|png|gif|webp)$` (no path
traversal) and sets `Content-Type` from the extension.
- Kept behind the `Storage` interface so an S3-compatible bucket or an R2
binding can replace it later without touching `app.ts`.
If the server has no storage configured, `GET /api/config` says so and the
page hides the `server` host.
- **litterbox 72h warning.** The UI warns that a review may take longer than
72 hours and the links will then be dead.
- **catbox userhash.** Optional, entered by the user, kept in the browser
(`localStorage` only if "remember" is ticked) or in `GM_setValue`. Through
the page it passes through our server, which forwards it to catbox only and
never logs or stores it.
Rules:
- **Upload only what we checked.** For hosts reached through our server, the
client sends the source URL, not bytes, so `/api/upload` can't be used to
host arbitrary files; the server returns the `sha256` of what it fetched and
the client rejects a mismatch. Hosts reached directly get the inspected bytes.
- **Re-uploads are cheap.** Server storage is keyed by hash, so rerunning
overwrites the same object (and restarts its 30 days). For litterbox and
catbox, the page keeps a per-session `sha256 → url` map and the userscript a
persistent one (`GM_setValue`), so the same image isn't uploaded twice.
- **No upload without an explicit user action.**
- **Failures are per image.** A failed upload is reported and the image left
out; if every upload fails, the action aborts.
- **Adding a host** is one file in `core/hosting/` plus one registry line.
### 4.7 Content rating
MangaBaka needs a `content_rating` per image. The tool can't judge it, so **the
user chooses it**, default `safe`: one selector for all new images, plus a
per-card override. If the source marks the series as mature (Webtoons
`ageGradeNotice : true`), the UI shows a warning to consider a rating above
`safe`. The default stays `safe`. Unattended runs (CLI `--yes`, later) must
pass a rating explicitly.
### 4.8 Seed (`seed/covers.ts`)
`buildCoverSeed(existing, newCovers, series, { contentRating })` returns:
```ts
interface CoverSeed {
seriesId: number;
userNote: string; // "Add WEBTOON covers from <canonicalUrl>"
images: SubmissionImage[]; // existing (unchanged except is_default) + new
added: InspectedCover[]; // for display
skipped: { cover: InspectedCover; reason: string }[];
}
```
Each new image becomes
`{ id: null, series_id, work_id: null, url, index: null, index_numeric: null,
language: series.language, type, note, content_rating, is_default }`.
`withHostedUrls(seed, urlMap)` swaps in the rehosted URLs after upload.
### 4.9 Front ends
**Static page** (`src/web/`, served by our server):
1. Two inputs: MangaBaka series link, Webtoons series link. Optional: "include
history" (on), granularity (every 1 / 3 / 6 / 12 months, default 6).
2. Runs the pipeline and shows one card per new cover (thumbnail, dimensions,
suggested type, language, note, content-rating selector, checkbox), plus
"already present" and skipped covers.
3. **Rehost selected** uploads to the chosen host (§4.6) and replaces the
shown URLs.
4. **Without a token:** "Copy all selected" and a link to
`https://mangabaka.org/data/submissions/series/{id}/images` for manual
entry.
5. **With a token:** the user pastes a personal access token (kept only in
memory unless they tick "remember on this device", then `localStorage`).
The page fetches the snapshot, shows the preview diff, and on confirmation
rehosts and saves with `save_mode: "review"`. 409: refetch, rebuild,
re-preview, ask again (once). 403: explain the 25-approved-submissions rule
and fall back to manual entry.
**Userscript** (`src/userscript/`, built to `mbseed.user.js`):
- `@match https://mangabaka.org/*`. On a series page it adds a "Seed covers"
panel; the MangaBaka series id comes from the page URL and the user pastes
the Webtoons link.
- Fetches Webtoons/Wayback/pstatic with `GM_xmlhttpRequest`
(`@connect webtoons.com`, `pstatic.net`, `web.archive.org`, `catbox.moe`,
and our server's host). Same review cards as the page.
- Rehosts to litterbox or catbox directly, or to our server's storage
(`/api/upload`; no storage keys in the script). With
litterbox or catbox it works with **no server at all**.
- Submits through the API when a token is set (stored with `GM_setValue`).
- **Without a token**, on the cover form page it can fill the form fields
for the selected covers. Whether the form accepts programmatically filled
fields is unverified (§7).
### 4.10 Server (`src/server/`)
| Route | Behaviour |
|---|---|
| `GET /*` | Static assets of the page. |
| `GET /api/config` | `{ storage: boolean, storageFull: boolean }`, so the page knows which hosts to offer (a full store is shown disabled, with the reason). |
| `GET /covers/<sha256>.<ext>` | Serves a stored cover (§4.6). Public: MangaBaka fetches these. |
| `GET /api/fetch?url=` | Fetches `url` only if its host is in the allowlist (union of `Source.hosts` plus `web.archive.org`), with a browser User-Agent for source hosts and the honest one for Wayback (§2.2); follows at most 5 redirects, each re-checked against the allowlist; returns the response. Anything else: 403. |
| `POST /api/upload` | `{ url, host: "store" \| "catbox", userhash? }` (§4.6). `url` must pass the same allowlist. |
The API routes reject other origins (except the userscript, which is not bound
by CORS), cap response size at 20 MB, and are rate-limited per IP.
**One handler; Bun now, Cloudflare later.** `app.ts` exports
`handle(req: Request, env: ServerEnv): Promise<Response>` using only
web-standard APIs (`fetch`, `Request`, `Response`, `crypto.subtle`):
| | Cloudflare (`worker.ts`, deferred) | Self-hosted (`bun.ts`, **now**) |
|---|---|---|
| Run | `wrangler dev` / `wrangler deploy` | `bun run src/server/bun.ts` behind our reverse proxy |
| Static assets | Workers static assets, with `run_worker_first = true` so the gate also covers assets (confirm the key when scaffolding) | read from the built `dist/web/` |
| Storage | R2 binding | folder `MBSEED_STORAGE_DIR` (public base URL `MBSEED_PUBLIC_URL`, cap `MBSEED_STORAGE_MAX_BYTES`), or none |
| Password | `wrangler secret put MBSEED_PASSWORD` | `MBSEED_PASSWORD` env var |
| Client IP for rate limits | `CF-Connecting-IP` | `server.requestIP(req)`, or `X-Forwarded-For` only when `MBSEED_TRUST_PROXY=1` |
Self-hosting also avoids the Cloudflare-egress risk. Our reverse proxy
terminates TLS; how `X-Forwarded-For` is trusted is set by `MBSEED_TRUST_PROXY`.
### 4.11 Access gate (`server/gate.ts`) — deferred
**Not built in the first version.** Decided 2026-10-14: the self-hosted site
starts open, and we add this if it gets abused or traffic grows too large.
Until then the server's protections are the host allowlist, the 20 MB cap and
the per-IP rate limit. The design below is kept for when it's needed; until
then there is no `/login`, and `/api/config` has no `gated` field.
An optional shared password locks the **whole** service: page, assets and API.
- **Off by default.** If `MBSEED_PASSWORD` is unset, the gate is open and
`/api/config` reports `gated: false`. A self-hosted instance can instead rely
on the reverse proxy's own auth (e.g. basic auth or SSO) and leave it unset.
- **Login.** `GET /login` serves a minimal form. `POST /login` compares the
password in constant time, then sets `mbseed_session=<expiry>.<hmac>`
(`HttpOnly; Secure; SameSite=Strict; Path=/`, 30 days). The HMAC key is
derived from `MBSEED_PASSWORD`, so changing the password logs everyone out.
Login attempts are rate-limited per IP (5 per minute).
- **Every other route** requires a valid, unexpired session cookie: browser
page requests without one get a 302 to `/login`, and API requests get 401.
- **Userscript.** It can't rely on the cookie, so API calls may instead send
`Authorization: Bearer <password>` (constant-time compare). The userscript
stores the password with `GM_setValue`. It needs this only for
`/api/upload` with server storage or catbox-via-server; with litterbox or
direct catbox it never calls our server.
- **No accounts.** One password for the group; rotate it to revoke access.
---
## 5. Cross-cutting concerns
- **Secrets.** The MangaBaka token stays in the user's browser (or userscript
storage) and is only sent to `api.mangabaka.org`. The server has no secrets
for now (`MBSEED_PASSWORD` once the gate exists). The catbox userhash is forwarded, never
stored.
- **Errors.** The UI shows a per-stage status; Wayback failure is a warning,
not a failure.
- **Load.** A run is ~60–100 proxied requests and ≤ ~6 MB stored for 30 days.
Abuse of the open proxy is the main risk, hence the allowlist, the size cap
and the per-IP rate limit; the gate (§4.11) is the next step if needed.
(Earlier estimate for Cloudflare: $0 on free tiers; ~$5/month for a busy
public instance.)
## 6. Testing
- **`bun test`** against fixtures saved from real responses: Webtoons pages
(official, canvas, mature, a 2015 Wayback snapshot), CDX output, MangaBaka
series/links/images/snapshot/preview JSON.
- Pure functions under test: link parsing, Webtoons parsing, URL normalisation,
classification, dedupe, default selection, seed building, content-rating
resolution, language mapping, the server allowlist, session cookie
signing/expiry.
- I/O (`pipeline`, `mangabaka/client`, `history/wayback`, every `hosting/*`,
`server/app.ts` routes incl. the gate: no cookie → 302/401, bad password,
expired cookie, Bearer header) with a fake transport and in-memory storage.
**No network in tests.**
- Manual smoke test: the page against Tower of God (MangaBaka `9999`, Webtoons
`title_no=95`), with "Rehost" and without submitting.
## 7. Open questions and risks
1. **Server egress.** Webtoons or Wayback may throttle our server's IP under
load. (Cloudflare IPs would be riskier; the userscript uses the user's IP.)
2. **When MangaBaka copies images** (submission or approval) and whether it
accepts our storage URLs, `litter.catbox.moe` / `files.catbox.moe` URLs, or
`swebtoon-phinf.pstatic.net` (`direct`). If it copies only on approval,
litterbox's 72h is likely too short.
3. **"Main cover" = `is_default: true`** is inferred from the API spec.
4. **Where users create a personal access token** in the MangaBaka UI.
5. **Whether the cover form accepts fields filled by a userscript.**
6. **Whether catbox accepts uploads from our server** (catbox-via-server).
7. **Whether MangaBaka's `hashes.sha256` is of the original bytes.** Dedupe
against existing covers relies on it; if MangaBaka re-encodes images, the
"already present" check misses and the reviewer sees duplicates.
8. **Hosting details:** which of our servers and domain (`MBSEED_PUBLIC_URL`),
where the covers folder lives (`MBSEED_STORAGE_DIR`), and whether the
1 GiB default cap suits that disk (~6 MB per run at most, kept 30 days).
## 8. Test data
Tower of God historical `og:image`s:
```
2015 http://swebtoon.phinf.naver.net/20140710_28/1404979632969uKBA8_JPEG/01_EC8BA0EC9D98ED8391.jpg
2017 http://swebtoon.phinf.naver.net/20151006_141/1444125046528RTfRM_JPEG/_EB93BBEB80AF_EBA38CEABCA5_E29587EABCB1_EB9384EB84B0_ipa.jpg
(twitter:image …/20151006_243/1444125042491PuNza_JPEG/…_mob.jpg)
2019 https://swebtoon-phinf.pstatic.net/20190318_291/1552868599909GoVLY_JPEG/10_EC8DB8EB84A4EC9DBC_ipad.jpg
(twitter:image …/20190318_26/1552868584759vl9A1_JPEG/11_EC8DB8EB84A4EC9DBC_mobile.jpg)
2021 https://swebtoon-phinf.pstatic.net/20210127_106/1611705089085OYcUB_JPEG/1Tower-of-God-Mobile-Landing-Page.jpg
2023+ https://swebtoon-phinf.pstatic.net/20230324_114/1679612146968KhLLz_JPEG/6TowerOfGod_landingpage_mobile.jpg
live https://swebtoon-phinf.pstatic.net/20250204_46/17386458444547o1b2_JPEG/95.jpg (poster, 480×623)
```
The 2014, 2015 and 2019 images are still live on `swebtoon-phinf.pstatic.net`;
the 2021 one is gone from Wayback's own copy but live on pstatic.
A 6-month sweep on 2026-10-14 found 20 distinct images (5 posters as `og:image`,
4 squares, 6 banners, 5 foregrounds), all live; see §2.2 for what they are.
2015 snapshots write `background:url(…)` without quotes and use the hosts
`swebtoon.phinf.naver.net` and `webtoon.phinf.naver.net`; both map to
`swebtoon-phinf.pstatic.net` with the same path. On the canvas example the
`og:image` and `twitter:image` URLs differ but the bytes are identical (merged
by `sha256`).
Other test inputs:
- canvas: `https://www.webtoons.com/en/canvas/crow-time/list?title_no=693372`
- mature: title_no `6065` (filename contains spaces and `)`; URL-encode it)
- FR example: Violet Romance, `2Thumb_Poster_9782.jpg`.
## 9. References
- yambs (the model project, Go): `/home/user/yambs` on the maintainer's machine, with the same
sources → seed → actions shape.
- MangaBaka API docs: https://mangabaka.org/data/api; explorer: https://mangabaka.org/data/api/explorer;
support: https://mangabaka.org/discord.
- The maintainer is a regular MangaBaka contributor without API access (fewer than 25 approved
submissions as of 2026-10-14), so the manual path must be first-class.
Reference in new issue View Git Blame Copy Permalink