From 9eca8f4f82e1b97140d9098948d2215c3adfa2d8 Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Wed, 14 Oct 2026 04:53:30 +0200 Subject: [PATCH] docs: add README and architecture spec for cover seeder Co-Authored-By: Claude Opus 5.5 --- README.md | 140 +++++++++++++++ docs/ARCHITECTURE.md | 393 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 533 insertions(+) create mode 100644 README.md create mode 100644 docs/ARCHITECTURE.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..fe5144e --- /dev/null +++ b/README.md @@ -0,0 +1,140 @@ +# mangabaka-seeder + +`mbseed` is a command-line tool for seeding edits to the +[MangaBaka](https://mangabaka.org) manga/comics database, in the spirit of +[yambs](https://codeberg.org/derat/yambs) for MusicBrainz. + +Today it does one thing well: **given a Webtoons series URL, it finds every +cover that series has used, current and historical, and requests them added to +the matching MangaBaka series**. The current poster becomes the main (default) +cover for the edition's language. + +More sources (Tapas, Tappytoon, …) and more kinds of edits are planned; see +[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for how they plug in. + +## How it works + +1. **Read the series.** It reads the Webtoons page: title, `title_no`, language, + and official or canvas. +2. **Find the MangaBaka series.** It searches MangaBaka by title and picks the + series whose Webtoons link has the same `title_no`. +3. **Collect covers.** + - **Current:** the portrait poster, the square thumbnail, and the banner and + hero art. + - **Historical:** the cover images in archived copies of the series page on + the [Wayback Machine](https://web.archive.org), up to one snapshot per + month. +4. **Check each image.** + - It downloads each image to get its size and hash. + - It classifies the image by shape: portrait → `volume`, square → `other`, + wide → `banner`. + - It drops duplicates and images MangaBaka already has. +5. **Build the submission.** The submission keeps all existing covers and adds + the new ones. The current poster is the default, or the square for canvas + series, which have no poster. Every new cover gets a note recording where + and when it was seen. +6. **Hand it to you.** It either submits through the MangaBaka API (with your + confirmation) or opens a local review page to help you enter it by hand. + +Nothing reaches MangaBaka without your confirmation, and API submissions always +go to review (`save_mode: "review"`). + +## Installation + +Requires Node.js 22 or newer. + +```sh +npm install +npm run build +npm link # puts `mbseed` on your PATH +``` + +## Usage + +``` +Usage: mbseed [options] + +Options: + --action What to do with the seed + (default: submit if MANGABAKA_TOKEN is set, else serve) + --series MangaBaka series id (skips automatic matching) + --no-history Only use covers currently on the source page + --history-limit Max archived snapshots to inspect (default 60) + --json With --action print: print the seed as JSON + --yes With --action submit: don't ask for confirmation + --addr With --action serve: listen address (default localhost:8999) + --verbose Log every HTTP request + -h, --help Show help +``` + +Environment: + +| Variable | Purpose | +|---|---| +| `MANGABAKA_TOKEN` | Personal access token (`mb-…`). Enables `--action submit`. | +| `MANGABAKA_API` | API base URL (default `https://api.mangabaka.org`). | + +### Without API access (most users) + +MangaBaka's submissions API is only open to contributors, or to users with at +least 25 approved submissions. Without a token, `mbseed` starts a local review +page: + +```sh +mbseed https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95 +# → Review page at http://localhost:8999 +``` + +The page shows each new cover with its thumbnail, size, suggested type, +language and note. Tick the ones you want, copy their URLs, and use the link to +open MangaBaka's cover form for the series. The form can't be pre-filled +automatically. + +### With API access + +```sh +export MANGABAKA_TOKEN=mb-... +mbseed https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95 +``` + +`mbseed` fetches the current cover set, shows MangaBaka's preview diff, asks +for confirmation and submits the request for review. If the API refuses access +(403), it falls back to the review page. + +### Just looking + +```sh +mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95 +mbseed --action print --json > seed.json +``` + +## Supported sources + +| Source | URLs | Current covers | Historical covers | +|---|---|---|---| +| WEBTOON (official & canvas) | `https://www.webtoons.com//…/list?title_no=N`, `m.webtoons.com` | poster, square, banner, foreground | Wayback Machine | + +Each Webtoons language edition has its own `title_no`. Run `mbseed` once per +edition URL; the covers are tagged with that edition's language. + +## Notes and limitations + +- **Image hosting.** Covers are submitted as URLs on Webtoons' image server + (`swebtoon-phinf.pstatic.net`), at their original size. We haven't confirmed + that MangaBaka accepts this host. +- **Some covers may only exist in the archive.** If an old cover is no longer + on Webtoons' server, its archive.org copy is used instead. +- **Not everything is a cover.** Some historical images are promotional or + landing-page art. Review the list before submitting. +- **Requests are rate-limited** to stay polite to Webtoons, the Wayback Machine + and MangaBaka. + +## Development + +```sh +npm test # vitest, fixture-based, no network +npm run typecheck +npm run dev -- --action print # run from source with tsx +``` + +To add a new source, see [docs/ARCHITECTURE.md § 4.1](docs/ARCHITECTURE.md#41-the-source-interface-extension-point). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..40e890d --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,393 @@ +# Architecture + +`mangabaka-seeder` is a command-line tool that gathers data about a series from +an external site (a "source") and turns it into an edit request ("submission") +for [MangaBaka](https://mangabaka.org). It is modeled on +[yambs](https://codeberg.org/derat/yambs), the MusicBrainz seeder: sources turn +into seeds, and an action decides what happens to them. The user always stays +in control of what is submitted. + +The first feature is **covers**: given a Webtoons series URL, collect every +cover that series has used (current and historical) and request them added to +the matching MangaBaka series, with the current poster as the default (main) +cover for the edition's language. + +This document is the design spec. It records the facts we verified about the +external services, the module boundaries, and the extension points. + +--- + +## 1. Goals and non-goals + +**Goals** + +- One command: `mbseed ` produces a ready-to-review cover submission. +- Find historical covers, not just the current one. +- Work for everyone. With a MangaBaka API token, submit through the API. Without + one, fall back to a local review page that helps with manual entry. +- New sources (Tapas, Tappytoon, Lezhin, Manta, …) can be added as one new + module under `src/sources/`, with no changes to the pipeline, the MangaBaka + client or the actions. +- Never change MangaBaka data silently. API submissions always go through + preview, explicit confirmation and `save_mode: "review"`. + +**Non-goals (for now)** + +- Editing series fields other than images, such as titles, links or tags. The + submission layer is keyed by MangaBaka "resource" so this can come later. +- Following a Webtoons series across language editions. Each `title_no` belongs + to one language edition; run the tool once per edition URL. +- Re-hosting images. We submit URLs; MangaBaka downloads them. + +--- + +## 2. Verified external facts + +These were verified on 2026-10-14. Treat them as assumptions that the fixtures +in `test/fixtures/` pin down. + +### 2.1 Webtoons + +| What | Where | +|---|---| +| Portrait poster (official series; 480×623 original) | `meta[property="og:image"]` | +| Square thumbnail | `meta[name="twitter:image"]` | +| Hero background (1920×321, official only) | `div.detail_bg` inline `style` → `background:url('…')` | +| Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) | `.detail_header span.thmb img[src]` | +| Title | `meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`) | +| Canonical URL | `link[rel="canonical"]` | +| `title_no` | `title_no` query parameter | +| Language | first path segment (`/en/`, `/fr/`, `/zh-hant/`, …) | +| Canvas vs official | `/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true` | + +- **Canvas series** have no portrait poster. Their `og:image` is a 1080×1080 + square. +- **Image hosts.** `webtoon-phinf.pstatic.net` returns 403 without a Webtoons + `Referer`; `swebtoon-phinf.pstatic.net` serves the same path with no Referer. +- **Original size.** The `?type=…` query selects a resized variant. Removing it + returns the original. `crop540_540` crops posters, so always strip the query. +- **Old hosts.** Old snapshots reference `swebtoon.phinf.naver.net`, which + becomes `swebtoon-phinf.pstatic.net` with the same path. +- **No limits so far.** A browser User-Agent is enough. Age-gated titles still + serve full HTML, and no cookies are required. +- **No usable JSON API.** The app API needs signed requests. We scrape HTML. + +### 2.2 Historical covers (Wayback Machine) + +- **Snapshot index.** The CDX API at + `https://web.archive.org/cdx/search/cdx?url=&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original` + returns at most one snapshot per month. +- **Raw pages.** `https://web.archive.org/web/id_/` returns the + page as it was archived, without Wayback's toolbar. +- **History exists.** For Tower of God, snapshots from 2015 to 2025 reference + five distinct cover images. All the older ones tested are **still served live** + from `swebtoon-phinf.pstatic.net`. +- **Fallback.** If an image is no longer live, it is often still at + `https://web.archive.org/web/id_/`. +- **Not every image is a cover.** Some historical `og:image`s are landing-page + art, so candidates are classified by shape and reviewed by the user. + +### 2.3 MangaBaka + +- **Endpoints.** The site is `https://mangabaka.org` and the API is + `https://api.mangabaka.org`. The OpenAPI spec is at + `https://mangabaka.org/api.json`, with docs at + `https://mangabaka.org/data/api`. +- **Public reads.** + - `GET /v2/series/match?q=` and `GET /v2/series/search?q=…` find + series. They are limited to about 30 requests per minute. + - `GET /v1/series/{id}` and `GET /v1/series/{id}/links` return the series + and its links. Links are canonicalised, e.g. + `https://www.webtoons.com/-/-/-/list?title_no=95`. + - `GET /v1/series/{id}/images` returns covers with `type`, `language`, + `index`, `note`, `content_rating`, `is_default` and `hashes.sha256`. +- **Image `type` values:** `volume` (default), `volume_back`, `chapter`, + `season`, `banner`, `audiobook`, `other`. There is no "main" type; the main + cover is the one with `is_default: true`. +- **Submissions API.** Auth is `x-api-key: mb-…`. All endpoints are v0 and + marked internal. + - `GET /v0/my/submissions/series-images/{id}` returns the current image set + and a `version` token. + - `POST /v0/my/submissions/series-images/{id}/preview` is a dry run that + returns a diff. + - `POST /v0/my/submissions/series-images/{id}` saves, with body + `{ version, user_note, save_mode, data: { images: [...] } }`. + - `images` **replaces the whole set**, so it must contain the existing images. + - A stale `version` gives 409. + - Users without the contributor role and with fewer than 25 approved + submissions get 403. +- **Web form.** Without API access, users use + `https://mangabaka.org/data/submissions/series/{id}/images`. It cannot be + pre-filled with query parameters, and its SvelteKit CSRF check blocks + cross-site POSTs. +- **Open risk:** we have not confirmed that MangaBaka accepts images hosted on + `pstatic.net`. We do know it rejects imgur. + +--- + +## 3. Pipeline + +``` + ┌──────────────┐ SourceSeries ┌──────────────────┐ + URL ────▶ │ source.resolve│ ───────────────▶ │ mangabaka.resolve │──▶ MangaBaka series id + └──────────────┘ └──────────────────┘ + │ + ▼ + source.currentCovers + history (Wayback, via source.extractCovers) + │ + ▼ CoverCandidate[] + images.inspect (download, dimensions, sha256, liveness) + │ + ▼ InspectedCover[] + covers.dedupe (within candidates + against existing MangaBaka covers) + │ + ▼ + seed.buildCoverSeed ──▶ CoverSeed (existing images + new images) + │ + ▼ + action: print | serve | submit +``` + +Each stage is a pure function or a small module with injected I/O (`fetch`), +so it can be tested with fixtures. + +--- + +## 4. Modules + +``` +src/ + cli.ts argument parsing (node:util parseArgs), wiring, exit codes + pipeline.ts runs the stages above; the only place they are composed + http.ts fetch wrapper: User-Agent, per-host rate limit, retries, timeouts + model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed) + sources/ + source.ts Source interface + helpers + registry.ts list of sources; findSource(url) + webtoons/ + index.ts Webtoons Source implementation + parse.ts pure HTML → { series, covers } extraction + urls.ts URL parsing, canonicalisation, image URL normalisation + history/ + wayback.ts generic Wayback history provider (CDX + archived pages) + images.ts download + probe dimensions + sha256 + classify by shape + covers.ts dedupe, merge observations, pick default + mangabaka/ + client.ts typed API client (public reads + submissions) + resolve.ts source series → MangaBaka series id + types.ts API types (subset of the OpenAPI spec we use) + seed/ + covers.ts CoverSeed builder: existing + new images → submission body + actions/ + print.ts human-readable or --json output + serve.ts local review page (node:http, single HTML page, no framework) + submit.ts snapshot → preview → confirm → save (review mode) +test/ + fixtures/ saved HTML / CDX / JSON responses + *.test.ts vitest +``` + +### 4.1 The `Source` interface (extension point) + +```ts +export interface Source { + /** Stable id, e.g. "webtoons". Used in notes and CLI output. */ + readonly id: string; + /** Human-readable name, e.g. "WEBTOON". */ + readonly name: string; + + /** True if this source handles the URL. */ + matches(url: URL): boolean; + + /** Fetches the series page and returns normalised series info. */ + resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>; + + /** Covers currently shown by the source. */ + currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>; + + /** + * Optional: pure extraction of cover candidates from a series page's HTML. + * Sources that implement this get Wayback history for free. The history + * provider runs it on archived copies of `series.archiveUrls`. + */ + extractCovers?(html: string, pageUrl: URL): CoverCandidate[]; + + /** Optional source-specific history beyond Wayback (e.g. an API with old art). */ + historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>; + + /** Normalises an image URL to its largest, hotlinkable form. */ + normalizeImageUrl(url: string): string; + + /** True if a MangaBaka series link points at this source series. */ + isSameSeries(linkUrl: string, series: SourceSeries): boolean; +} + +export interface SourceSeries { + source: string; // Source.id + externalId: string; // e.g. Webtoons title_no "95" + title: string; + language: string; // MangaBaka language code, e.g. "en", "zh-Hant" + canonicalUrl: string; + archiveUrls: string[]; // URL prefixes to look up in Wayback + flags: Record<string, boolean>; // source-specific, e.g. { canvas: true } +} + +export interface CoverCandidate { + url: string; // normalised + role: "poster" | "square" | "banner" | "foreground" | "other"; + current: boolean; // shown on the live page now + seenAt?: string; // archive timestamp (YYYYMMDD…) if historical + archivedUrl?: string; // Wayback fallback for the image itself +} +``` + +**Adding a source** (e.g. Tapas) means: + +1. Create `src/sources/tapas/` that implements `Source`. Put the HTML + extraction in a pure `parse.ts` and give it fixture tests. +2. Add it to the array in `src/sources/registry.ts`. +3. Map its language codes to MangaBaka codes. + +The pipeline, Wayback history, image inspection, deduplication, MangaBaka +resolution and all actions work unchanged. A source with an API (Tappytoon, for +example) can skip `extractCovers` and implement `historicalCovers`, or neither. + +### 4.2 History + +`history/wayback.ts` knows nothing about any one source. For each prefix in +`series.archiveUrls` it: + +1. queries CDX with `collapse=timestamp:6` (one snapshot per month, capped by + `--history-limit`, default 60); +2. fetches each snapshot with the `id_` modifier; +3. calls `source.extractCovers(html, originalUrl)`; +4. normalises the image URLs with `source.normalizeImageUrl` and tags each + candidate with `seenAt` and `archivedUrl`. + +Candidates that share a normalised URL are merged, and the merged candidate +records the first and last time it was seen. Wayback failures (timeouts, 5xx, +rate limits) produce a warning and an empty history; they are never fatal. +`--no-history` skips this stage. + +### 4.3 Image inspection and classification + +`images.ts` downloads each candidate (live URL first, then `archivedUrl`) and +records its `width`, `height`, `sha256`, `bytes` and `live` status. Dimensions +come from the image header, read with the `image-size` package. Candidates that +cannot be downloaded at all are dropped with a warning. + +Classification by aspect ratio (h / w): + +| Shape | Ratio | MangaBaka `type` | +|---|---|---| +| portrait | ≥ 1.15 | `volume` | +| square-ish | 0.85 – 1.15 | `other` | +| wide | < 0.5 | `banner` | +| otherwise | – | `other` | + +### 4.4 Dedupe and default selection (`covers.ts`) + +- **Duplicates.** Images with the same `sha256` are merged. The merged entry + keeps the live URL and the widest `seenAt` range. +- **Already on MangaBaka.** Candidates whose `sha256` matches an existing + MangaBaka image are dropped and reported as "already present". +- **Default selection.** + - The **default** is the current live poster. + - If there is none (canvas series), it is the current live square. + - The new default gets `is_default: true`, and every existing image gets + `is_default: false`. This is the "main cover" for the language edition. + - The reviewer on MangaBaka can still reject that change. +- **Notes.** Every new image gets a note, e.g. + `"WEBTOON poster (current)"` or `"WEBTOON poster, seen 2019-03 – 2021-01"`. + +### 4.5 MangaBaka series resolution (`mangabaka/resolve.ts`) + +1. If `--series <id>` is given, use it, and warn if no link of that series + passes `source.isSameSeries`. +2. Otherwise call `/v2/series/match?q=<title>`, then `/v2/series/search`. For + each result, fetch `/v1/series/{id}/links` and keep the series whose link + passes `source.isSameSeries`. For Webtoons that means the same `title_no`. +3. If no series or several series match, stop with exit code 2 and print the + candidates (id, title, URL) so the user can rerun with `--series`. + +### 4.6 Seed (`seed/covers.ts`) + +`buildCoverSeed(existing, newCovers, series)` returns: + +```ts +interface CoverSeed { + seriesId: number; + userNote: string; // "Add WEBTOON covers from <canonicalUrl>" + images: SubmissionImage[]; // existing (unchanged except is_default) + new + added: InspectedCover[]; // for display + skipped: { cover: InspectedCover; reason: string }[]; +} +``` + +Each new image becomes: +`{ id: null, series_id, work_id: null, url, index: null, index_numeric: null, +language: series.language, type, note, content_rating: "safe", is_default }`. + +### 4.7 Actions + +| Action | Default when | Effect | +|---|---|---| +| `print` | `--action print` | Prints series, candidates, the planned seed, and skips. `--json` prints the seed as JSON. No writes. | +| `serve` | no `MANGABAKA_TOKEN` | Serves `http://localhost:8999` (see below). | +| `submit` | `MANGABAKA_TOKEN` set | Submits through the API (see below). | + +**`serve`** shows a review page with one card per new cover: a thumbnail, +dimensions, type, language, note, a "copy URL" button and a checkbox. It also +has a "Copy all selected" button and a link to +`https://mangabaka.org/data/submissions/series/{id}/images`. The page is a +single HTML string rendered server-side with inline JS and no framework. It +binds to localhost only. + +**`submit`**: + +1. `GET` the snapshot (`version` plus current images) and rebuild the seed on + top of it. +2. `POST …/preview` and print the diff. +3. Ask "Submit for review? [y/N]". `--yes` skips this. +4. `POST` with `save_mode: "review"`. +5. Handle the outcome: + - **409:** fetch the snapshot again, rebuild, preview again and ask again, + at most once. + - **403:** explain the 25-approved-submissions requirement and fall back to + `serve`. + +--- + +## 5. Cross-cutting concerns + +- **HTTP.** All network access goes through `http.ts`. It: + - sends a browser User-Agent; + - limits each host to 1 request per second, and Wayback to 1 every 1.5 s; + - times out after 20 s; + - retries twice with backoff on 429/5xx; + - never sends the MangaBaka token to any other host. +- **Logging.** Progress goes to stderr and results to stdout, so + `print --json` can be piped. `--verbose` logs every request. +- **Exit codes.** + - 0: success, including "nothing new to add" + - 1: unexpected error + - 2: bad usage, an unsupported URL, or an ambiguous or missing series match + - 3: the user declined to submit. +- **Config.** `MANGABAKA_TOKEN` (env), `MANGABAKA_API` (env, default + `https://api.mangabaka.org`, for testing). + +--- + +## 6. Testing + +- **Unit tests with vitest**, run against fixtures saved from real responses: + - Webtoons pages (official, canvas, an archived 2015 snapshot) and CDX output + - MangaBaka match, links, images, snapshot and preview JSON. +- **Pure functions under test:** + - parsing, URL normalisation and canonicalisation + - classification, dedupe and default selection + - seed building and the language mapping. +- **I/O modules** (`pipeline`, `mangabaka/client`, `actions/submit`, + `history/wayback`) are tested with an injected fake `fetch`. +- **No network in tests.** A manual smoke test is documented in the README: + `mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95`.