lucasw89/mangabaka-seeder
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add README and architecture spec for cover seeder

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-10-14 04:53:30 +02:00
commit 9eca8f4f82
2 files changed
+533
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+140
View File
@@ -0,0 +1,140 @@
# mangabaka-seeder
`mbseed` is a command-line tool for seeding edits to the
[MangaBaka](https://mangabaka.org) manga/comics database, in the spirit of
[yambs](https://codeberg.org/derat/yambs) for MusicBrainz.
Today it does one thing well: **given a Webtoons series URL, it finds every
cover that series has used, current and historical, and requests them added to
the matching MangaBaka series**. The current poster becomes the main (default)
cover for the edition's language.
More sources (Tapas, Tappytoon, …) and more kinds of edits are planned; see
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for how they plug in.
## How it works
1. **Read the series.** It reads the Webtoons page: title, `title_no`, language,
and official or canvas.
2. **Find the MangaBaka series.** It searches MangaBaka by title and picks the
series whose Webtoons link has the same `title_no`.
3. **Collect covers.**
- **Current:** the portrait poster, the square thumbnail, and the banner and
hero art.
- **Historical:** the cover images in archived copies of the series page on
the [Wayback Machine](https://web.archive.org), up to one snapshot per
month.
4. **Check each image.**
- It downloads each image to get its size and hash.
- It classifies the image by shape: portrait → `volume`, square → `other`,
wide → `banner`.
- It drops duplicates and images MangaBaka already has.
5. **Build the submission.** The submission keeps all existing covers and adds
the new ones. The current poster is the default, or the square for canvas
series, which have no poster. Every new cover gets a note recording where
and when it was seen.
6. **Hand it to you.** It either submits through the MangaBaka API (with your
confirmation) or opens a local review page to help you enter it by hand.
Nothing reaches MangaBaka without your confirmation, and API submissions always
go to review (`save_mode: "review"`).
## Installation
Requires Node.js 22 or newer.
```sh
npm install
npm run build
npm link # puts `mbseed` on your PATH
```
## Usage
```
Usage: mbseed [options] <URL>
Options:
--action <print|serve|submit> What to do with the seed
(default: submit if MANGABAKA_TOKEN is set, else serve)
--series <id> MangaBaka series id (skips automatic matching)
--no-history Only use covers currently on the source page
--history-limit <n> Max archived snapshots to inspect (default 60)
--json With --action print: print the seed as JSON
--yes With --action submit: don't ask for confirmation
--addr <host:port> With --action serve: listen address (default localhost:8999)
--verbose Log every HTTP request
-h, --help Show help
```
Environment:
| Variable | Purpose |
|---|---|
| `MANGABAKA_TOKEN` | Personal access token (`mb-…`). Enables `--action submit`. |
| `MANGABAKA_API` | API base URL (default `https://api.mangabaka.org`). |
### Without API access (most users)
MangaBaka's submissions API is only open to contributors, or to users with at
least 25 approved submissions. Without a token, `mbseed` starts a local review
page:
```sh
mbseed https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95
# → Review page at http://localhost:8999
```
The page shows each new cover with its thumbnail, size, suggested type,
language and note. Tick the ones you want, copy their URLs, and use the link to
open MangaBaka's cover form for the series. The form can't be pre-filled
automatically.
### With API access
```sh
export MANGABAKA_TOKEN=mb-...
mbseed https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95
```
`mbseed` fetches the current cover set, shows MangaBaka's preview diff, asks
for confirmation and submits the request for review. If the API refuses access
(403), it falls back to the review page.
### Just looking
```sh
mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95
mbseed --action print --json <URL> > seed.json
```
## Supported sources
| Source | URLs | Current covers | Historical covers |
|---|---|---|---|
| WEBTOON (official & canvas) | `https://www.webtoons.com/<lang>/…/list?title_no=N`, `m.webtoons.com` | poster, square, banner, foreground | Wayback Machine |
Each Webtoons language edition has its own `title_no`. Run `mbseed` once per
edition URL; the covers are tagged with that edition's language.
## Notes and limitations
- **Image hosting.** Covers are submitted as URLs on Webtoons' image server
(`swebtoon-phinf.pstatic.net`), at their original size. We haven't confirmed
that MangaBaka accepts this host.
- **Some covers may only exist in the archive.** If an old cover is no longer
on Webtoons' server, its archive.org copy is used instead.
- **Not everything is a cover.** Some historical images are promotional or
landing-page art. Review the list before submitting.
- **Requests are rate-limited** to stay polite to Webtoons, the Wayback Machine
and MangaBaka.
## Development
```sh
npm test # vitest, fixture-based, no network
npm run typecheck
npm run dev -- --action print <URL> # run from source with tsx
```
To add a new source, see [docs/ARCHITECTURE.md § 4.1](docs/ARCHITECTURE.md#41-the-source-interface-extension-point).
docs/ARCHITECTURE.md
+393
View File
@@ -0,0 +1,393 @@
# Architecture
`mangabaka-seeder` is a command-line tool that gathers data about a series from
an external site (a "source") and turns it into an edit request ("submission")
for [MangaBaka](https://mangabaka.org). It is modeled on
[yambs](https://codeberg.org/derat/yambs), the MusicBrainz seeder: sources turn
into seeds, and an action decides what happens to them. The user always stays
in control of what is submitted.
The first feature is **covers**: given a Webtoons series URL, collect every
cover that series has used (current and historical) and request them added to
the matching MangaBaka series, with the current poster as the default (main)
cover for the edition's language.
This document is the design spec. It records the facts we verified about the
external services, the module boundaries, and the extension points.
---
## 1. Goals and non-goals
**Goals**
- One command: `mbseed <url>` produces a ready-to-review cover submission.
- Find historical covers, not just the current one.
- Work for everyone. With a MangaBaka API token, submit through the API. Without
one, fall back to a local review page that helps with manual entry.
- New sources (Tapas, Tappytoon, Lezhin, Manta, …) can be added as one new
module under `src/sources/`, with no changes to the pipeline, the MangaBaka
client or the actions.
- Never change MangaBaka data silently. API submissions always go through
preview, explicit confirmation and `save_mode: "review"`.
**Non-goals (for now)**
- Editing series fields other than images, such as titles, links or tags. The
submission layer is keyed by MangaBaka "resource" so this can come later.
- Following a Webtoons series across language editions. Each `title_no` belongs
to one language edition; run the tool once per edition URL.
- Re-hosting images. We submit URLs; MangaBaka downloads them.
---
## 2. Verified external facts
These were verified on 2026-10-14. Treat them as assumptions that the fixtures
in `test/fixtures/` pin down.
### 2.1 Webtoons
| What | Where |
|---|---|
| Portrait poster (official series; 480×623 original) | `meta[property="og:image"]` |
| Square thumbnail | `meta[name="twitter:image"]` |
| Hero background (1920×321, official only) | `div.detail_bg` inline `style` → `background:url('…')` |
| Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) | `.detail_header span.thmb img[src]` |
| Title | `meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`) |
| Canonical URL | `link[rel="canonical"]` |
| `title_no` | `title_no` query parameter |
| Language | first path segment (`/en/`, `/fr/`, `/zh-hant/`, …) |
| Canvas vs official | `/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true` |
- **Canvas series** have no portrait poster. Their `og:image` is a 1080×1080
square.
- **Image hosts.** `webtoon-phinf.pstatic.net` returns 403 without a Webtoons
`Referer`; `swebtoon-phinf.pstatic.net` serves the same path with no Referer.
- **Original size.** The `?type=…` query selects a resized variant. Removing it
returns the original. `crop540_540` crops posters, so always strip the query.
- **Old hosts.** Old snapshots reference `swebtoon.phinf.naver.net`, which
becomes `swebtoon-phinf.pstatic.net` with the same path.
- **No limits so far.** A browser User-Agent is enough. Age-gated titles still
serve full HTML, and no cookies are required.
- **No usable JSON API.** The app API needs signed requests. We scrape HTML.
### 2.2 Historical covers (Wayback Machine)
- **Snapshot index.** The CDX API at
`https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original`
returns at most one snapshot per month.
- **Raw pages.** `https://web.archive.org/web/<ts>id_/<original>` returns the
page as it was archived, without Wayback's toolbar.
- **History exists.** For Tower of God, snapshots from 2015 to 2025 reference
five distinct cover images. All the older ones tested are **still served live**
from `swebtoon-phinf.pstatic.net`.
- **Fallback.** If an image is no longer live, it is often still at
`https://web.archive.org/web/<ts>id_/<image-url>`.
- **Not every image is a cover.** Some historical `og:image`s are landing-page
art, so candidates are classified by shape and reviewed by the user.
### 2.3 MangaBaka
- **Endpoints.** The site is `https://mangabaka.org` and the API is
`https://api.mangabaka.org`. The OpenAPI spec is at
`https://mangabaka.org/api.json`, with docs at
`https://mangabaka.org/data/api`.
- **Public reads.**
- `GET /v2/series/match?q=<title>` and `GET /v2/series/search?q=…` find
series. They are limited to about 30 requests per minute.
- `GET /v1/series/{id}` and `GET /v1/series/{id}/links` return the series
and its links. Links are canonicalised, e.g.
`https://www.webtoons.com/-/-/-/list?title_no=95`.
- `GET /v1/series/{id}/images` returns covers with `type`, `language`,
`index`, `note`, `content_rating`, `is_default` and `hashes.sha256`.
- **Image `type` values:** `volume` (default), `volume_back`, `chapter`,
`season`, `banner`, `audiobook`, `other`. There is no "main" type; the main
cover is the one with `is_default: true`.
- **Submissions API.** Auth is `x-api-key: mb-…`. All endpoints are v0 and
marked internal.
- `GET /v0/my/submissions/series-images/{id}` returns the current image set
and a `version` token.
- `POST /v0/my/submissions/series-images/{id}/preview` is a dry run that
returns a diff.
- `POST /v0/my/submissions/series-images/{id}` saves, with body
`{ version, user_note, save_mode, data: { images: [...] } }`.
- `images` **replaces the whole set**, so it must contain the existing images.
- A stale `version` gives 409.
- Users without the contributor role and with fewer than 25 approved
submissions get 403.
- **Web form.** Without API access, users use
`https://mangabaka.org/data/submissions/series/{id}/images`. It cannot be
pre-filled with query parameters, and its SvelteKit CSRF check blocks
cross-site POSTs.
- **Open risk:** we have not confirmed that MangaBaka accepts images hosted on
`pstatic.net`. We do know it rejects imgur.
---
## 3. Pipeline
```
┌──────────────┐ SourceSeries ┌──────────────────┐
URL ────▶ │ source.resolve│ ───────────────▶ │ mangabaka.resolve │──▶ MangaBaka series id
└──────────────┘ └──────────────────┘
│
▼
source.currentCovers + history (Wayback, via source.extractCovers)
│
▼ CoverCandidate[]
images.inspect (download, dimensions, sha256, liveness)
│
▼ InspectedCover[]
covers.dedupe (within candidates + against existing MangaBaka covers)
│
▼
seed.buildCoverSeed ──▶ CoverSeed (existing images + new images)
│
▼
action: print | serve | submit
```
Each stage is a pure function or a small module with injected I/O (`fetch`),
so it can be tested with fixtures.
---
## 4. Modules
```
src/
cli.ts argument parsing (node:util parseArgs), wiring, exit codes
pipeline.ts runs the stages above; the only place they are composed
http.ts fetch wrapper: User-Agent, per-host rate limit, retries, timeouts
model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
sources/
source.ts Source interface + helpers
registry.ts list of sources; findSource(url)
webtoons/
index.ts Webtoons Source implementation
parse.ts pure HTML → { series, covers } extraction
urls.ts URL parsing, canonicalisation, image URL normalisation
history/
wayback.ts generic Wayback history provider (CDX + archived pages)
images.ts download + probe dimensions + sha256 + classify by shape
covers.ts dedupe, merge observations, pick default
mangabaka/
client.ts typed API client (public reads + submissions)
resolve.ts source series → MangaBaka series id
types.ts API types (subset of the OpenAPI spec we use)
seed/
covers.ts CoverSeed builder: existing + new images → submission body
actions/
print.ts human-readable or --json output
serve.ts local review page (node:http, single HTML page, no framework)
submit.ts snapshot → preview → confirm → save (review mode)
test/
fixtures/ saved HTML / CDX / JSON responses
*.test.ts vitest
```
### 4.1 The `Source` interface (extension point)
```ts
export interface Source {
/** Stable id, e.g. "webtoons". Used in notes and CLI output. */
readonly id: string;
/** Human-readable name, e.g. "WEBTOON". */
readonly name: string;
/** True if this source handles the URL. */
matches(url: URL): boolean;
/** Fetches the series page and returns normalised series info. */
resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;
/** Covers currently shown by the source. */
currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/**
* Optional: pure extraction of cover candidates from a series page's HTML.
* Sources that implement this get Wayback history for free. The history
* provider runs it on archived copies of `series.archiveUrls`.
*/
extractCovers?(html: string, pageUrl: URL): CoverCandidate[];
/** Optional source-specific history beyond Wayback (e.g. an API with old art). */
historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
/** Normalises an image URL to its largest, hotlinkable form. */
normalizeImageUrl(url: string): string;
/** True if a MangaBaka series link points at this source series. */
isSameSeries(linkUrl: string, series: SourceSeries): boolean;
}
export interface SourceSeries {
source: string; // Source.id
externalId: string; // e.g. Webtoons title_no "95"
title: string;
language: string; // MangaBaka language code, e.g. "en", "zh-Hant"
canonicalUrl: string;
archiveUrls: string[]; // URL prefixes to look up in Wayback
flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
}
export interface CoverCandidate {
url: string; // normalised
role: "poster" | "square" | "banner" | "foreground" | "other";
current: boolean; // shown on the live page now
seenAt?: string; // archive timestamp (YYYYMMDD…) if historical
archivedUrl?: string; // Wayback fallback for the image itself
}
```
**Adding a source** (e.g. Tapas) means:
1. Create `src/sources/tapas/` that implements `Source`. Put the HTML
extraction in a pure `parse.ts` and give it fixture tests.
2. Add it to the array in `src/sources/registry.ts`.
3. Map its language codes to MangaBaka codes.
The pipeline, Wayback history, image inspection, deduplication, MangaBaka
resolution and all actions work unchanged. A source with an API (Tappytoon, for
example) can skip `extractCovers` and implement `historicalCovers`, or neither.
### 4.2 History
`history/wayback.ts` knows nothing about any one source. For each prefix in
`series.archiveUrls` it:
1. queries CDX with `collapse=timestamp:6` (one snapshot per month, capped by
`--history-limit`, default 60);
2. fetches each snapshot with the `id_` modifier;
3. calls `source.extractCovers(html, originalUrl)`;
4. normalises the image URLs with `source.normalizeImageUrl` and tags each
candidate with `seenAt` and `archivedUrl`.
Candidates that share a normalised URL are merged, and the merged candidate
records the first and last time it was seen. Wayback failures (timeouts, 5xx,
rate limits) produce a warning and an empty history; they are never fatal.
`--no-history` skips this stage.
### 4.3 Image inspection and classification
`images.ts` downloads each candidate (live URL first, then `archivedUrl`) and
records its `width`, `height`, `sha256`, `bytes` and `live` status. Dimensions
come from the image header, read with the `image-size` package. Candidates that
cannot be downloaded at all are dropped with a warning.
Classification by aspect ratio (h / w):
| Shape | Ratio | MangaBaka `type` |
|---|---|---|
| portrait | ≥ 1.15 | `volume` |
| square-ish | 0.85 – 1.15 | `other` |
| wide | < 0.5 | `banner` |
| otherwise | – | `other` |
### 4.4 Dedupe and default selection (`covers.ts`)
- **Duplicates.** Images with the same `sha256` are merged. The merged entry
keeps the live URL and the widest `seenAt` range.
- **Already on MangaBaka.** Candidates whose `sha256` matches an existing
MangaBaka image are dropped and reported as "already present".
- **Default selection.**
- The **default** is the current live poster.
- If there is none (canvas series), it is the current live square.
- The new default gets `is_default: true`, and every existing image gets
`is_default: false`. This is the "main cover" for the language edition.
- The reviewer on MangaBaka can still reject that change.
- **Notes.** Every new image gets a note, e.g.
`"WEBTOON poster (current)"` or `"WEBTOON poster, seen 2019-03 – 2021-01"`.
### 4.5 MangaBaka series resolution (`mangabaka/resolve.ts`)
1. If `--series <id>` is given, use it, and warn if no link of that series
passes `source.isSameSeries`.
2. Otherwise call `/v2/series/match?q=<title>`, then `/v2/series/search`. For
each result, fetch `/v1/series/{id}/links` and keep the series whose link
passes `source.isSameSeries`. For Webtoons that means the same `title_no`.
3. If no series or several series match, stop with exit code 2 and print the
candidates (id, title, URL) so the user can rerun with `--series`.
### 4.6 Seed (`seed/covers.ts`)
`buildCoverSeed(existing, newCovers, series)` returns:
```ts
interface CoverSeed {
seriesId: number;
userNote: string; // "Add WEBTOON covers from <canonicalUrl>"
images: SubmissionImage[]; // existing (unchanged except is_default) + new
added: InspectedCover[]; // for display
skipped: { cover: InspectedCover; reason: string }[];
}
```
Each new image becomes:
`{ id: null, series_id, work_id: null, url, index: null, index_numeric: null,
language: series.language, type, note, content_rating: "safe", is_default }`.
### 4.7 Actions
| Action | Default when | Effect |
|---|---|---|
| `print` | `--action print` | Prints series, candidates, the planned seed, and skips. `--json` prints the seed as JSON. No writes. |
| `serve` | no `MANGABAKA_TOKEN` | Serves `http://localhost:8999` (see below). |
| `submit` | `MANGABAKA_TOKEN` set | Submits through the API (see below). |
**`serve`** shows a review page with one card per new cover: a thumbnail,
dimensions, type, language, note, a "copy URL" button and a checkbox. It also
has a "Copy all selected" button and a link to
`https://mangabaka.org/data/submissions/series/{id}/images`. The page is a
single HTML string rendered server-side with inline JS and no framework. It
binds to localhost only.
**`submit`**:
1. `GET` the snapshot (`version` plus current images) and rebuild the seed on
top of it.
2. `POST …/preview` and print the diff.
3. Ask "Submit for review? [y/N]". `--yes` skips this.
4. `POST` with `save_mode: "review"`.
5. Handle the outcome:
- **409:** fetch the snapshot again, rebuild, preview again and ask again,
at most once.
- **403:** explain the 25-approved-submissions requirement and fall back to
`serve`.
---
## 5. Cross-cutting concerns
- **HTTP.** All network access goes through `http.ts`. It:
- sends a browser User-Agent;
- limits each host to 1 request per second, and Wayback to 1 every 1.5 s;
- times out after 20 s;
- retries twice with backoff on 429/5xx;
- never sends the MangaBaka token to any other host.
- **Logging.** Progress goes to stderr and results to stdout, so
`print --json` can be piped. `--verbose` logs every request.
- **Exit codes.**
- 0: success, including "nothing new to add"
- 1: unexpected error
- 2: bad usage, an unsupported URL, or an ambiguous or missing series match
- 3: the user declined to submit.
- **Config.** `MANGABAKA_TOKEN` (env), `MANGABAKA_API` (env, default
`https://api.mangabaka.org`, for testing).
---
## 6. Testing
- **Unit tests with vitest**, run against fixtures saved from real responses:
- Webtoons pages (official, canvas, an archived 2015 snapshot) and CDX output
- MangaBaka match, links, images, snapshot and preview JSON.
- **Pure functions under test:**
- parsing, URL normalisation and canonicalisation
- classification, dedupe and default selection
- seed building and the language mapping.
- **I/O modules** (`pipeline`, `mangabaka/client`, `actions/submit`,
`history/wayback`) are tested with an injected fake `fetch`.
- **No network in tests.** A manual smoke test is documented in the README:
`mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95`.