394 lines
17 KiB
Markdown
394 lines
17 KiB
Markdown
# Architecture
|
||
|
||
`mangabaka-seeder` is a command-line tool that gathers data about a series from
|
||
an external site (a "source") and turns it into an edit request ("submission")
|
||
for [MangaBaka](https://mangabaka.org). It is modeled on
|
||
[yambs](https://codeberg.org/derat/yambs), the MusicBrainz seeder: sources turn
|
||
into seeds, and an action decides what happens to them. The user always stays
|
||
in control of what is submitted.
|
||
|
||
The first feature is **covers**: given a Webtoons series URL, collect every
|
||
cover that series has used (current and historical) and request them added to
|
||
the matching MangaBaka series, with the current poster as the default (main)
|
||
cover for the edition's language.
|
||
|
||
This document is the design spec. It records the facts we verified about the
|
||
external services, the module boundaries, and the extension points.
|
||
|
||
---
|
||
|
||
## 1. Goals and non-goals
|
||
|
||
**Goals**
|
||
|
||
- One command: `mbseed <url>` produces a ready-to-review cover submission.
|
||
- Find historical covers, not just the current one.
|
||
- Work for everyone. With a MangaBaka API token, submit through the API. Without
|
||
one, fall back to a local review page that helps with manual entry.
|
||
- New sources (Tapas, Tappytoon, Lezhin, Manta, …) can be added as one new
|
||
module under `src/sources/`, with no changes to the pipeline, the MangaBaka
|
||
client or the actions.
|
||
- Never change MangaBaka data silently. API submissions always go through
|
||
preview, explicit confirmation and `save_mode: "review"`.
|
||
|
||
**Non-goals (for now)**
|
||
|
||
- Editing series fields other than images, such as titles, links or tags. The
|
||
submission layer is keyed by MangaBaka "resource" so this can come later.
|
||
- Following a Webtoons series across language editions. Each `title_no` belongs
|
||
to one language edition; run the tool once per edition URL.
|
||
- Re-hosting images. We submit URLs; MangaBaka downloads them.
|
||
|
||
---
|
||
|
||
## 2. Verified external facts
|
||
|
||
These were verified on 2026-10-14. Treat them as assumptions that the fixtures
|
||
in `test/fixtures/` pin down.
|
||
|
||
### 2.1 Webtoons
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| Portrait poster (official series; 480×623 original) | `meta[property="og:image"]` |
|
||
| Square thumbnail | `meta[name="twitter:image"]` |
|
||
| Hero background (1920×321, official only) | `div.detail_bg` inline `style` → `background:url('…')` |
|
||
| Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) | `.detail_header span.thmb img[src]` |
|
||
| Title | `meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`) |
|
||
| Canonical URL | `link[rel="canonical"]` |
|
||
| `title_no` | `title_no` query parameter |
|
||
| Language | first path segment (`/en/`, `/fr/`, `/zh-hant/`, …) |
|
||
| Canvas vs official | `/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true` |
|
||
|
||
- **Canvas series** have no portrait poster. Their `og:image` is a 1080×1080
|
||
square.
|
||
- **Image hosts.** `webtoon-phinf.pstatic.net` returns 403 without a Webtoons
|
||
`Referer`; `swebtoon-phinf.pstatic.net` serves the same path with no Referer.
|
||
- **Original size.** The `?type=…` query selects a resized variant. Removing it
|
||
returns the original. `crop540_540` crops posters, so always strip the query.
|
||
- **Old hosts.** Old snapshots reference `swebtoon.phinf.naver.net`, which
|
||
becomes `swebtoon-phinf.pstatic.net` with the same path.
|
||
- **No limits so far.** A browser User-Agent is enough. Age-gated titles still
|
||
serve full HTML, and no cookies are required.
|
||
- **No usable JSON API.** The app API needs signed requests. We scrape HTML.
|
||
|
||
### 2.2 Historical covers (Wayback Machine)
|
||
|
||
- **Snapshot index.** The CDX API at
|
||
`https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original`
|
||
returns at most one snapshot per month.
|
||
- **Raw pages.** `https://web.archive.org/web/<ts>id_/<original>` returns the
|
||
page as it was archived, without Wayback's toolbar.
|
||
- **History exists.** For Tower of God, snapshots from 2015 to 2025 reference
|
||
five distinct cover images. All the older ones tested are **still served live**
|
||
from `swebtoon-phinf.pstatic.net`.
|
||
- **Fallback.** If an image is no longer live, it is often still at
|
||
`https://web.archive.org/web/<ts>id_/<image-url>`.
|
||
- **Not every image is a cover.** Some historical `og:image`s are landing-page
|
||
art, so candidates are classified by shape and reviewed by the user.
|
||
|
||
### 2.3 MangaBaka
|
||
|
||
- **Endpoints.** The site is `https://mangabaka.org` and the API is
|
||
`https://api.mangabaka.org`. The OpenAPI spec is at
|
||
`https://mangabaka.org/api.json`, with docs at
|
||
`https://mangabaka.org/data/api`.
|
||
- **Public reads.**
|
||
- `GET /v2/series/match?q=<title>` and `GET /v2/series/search?q=…` find
|
||
series. They are limited to about 30 requests per minute.
|
||
- `GET /v1/series/{id}` and `GET /v1/series/{id}/links` return the series
|
||
and its links. Links are canonicalised, e.g.
|
||
`https://www.webtoons.com/-/-/-/list?title_no=95`.
|
||
- `GET /v1/series/{id}/images` returns covers with `type`, `language`,
|
||
`index`, `note`, `content_rating`, `is_default` and `hashes.sha256`.
|
||
- **Image `type` values:** `volume` (default), `volume_back`, `chapter`,
|
||
`season`, `banner`, `audiobook`, `other`. There is no "main" type; the main
|
||
cover is the one with `is_default: true`.
|
||
- **Submissions API.** Auth is `x-api-key: mb-…`. All endpoints are v0 and
|
||
marked internal.
|
||
- `GET /v0/my/submissions/series-images/{id}` returns the current image set
|
||
and a `version` token.
|
||
- `POST /v0/my/submissions/series-images/{id}/preview` is a dry run that
|
||
returns a diff.
|
||
- `POST /v0/my/submissions/series-images/{id}` saves, with body
|
||
`{ version, user_note, save_mode, data: { images: [...] } }`.
|
||
- `images` **replaces the whole set**, so it must contain the existing images.
|
||
- A stale `version` gives 409.
|
||
- Users without the contributor role and with fewer than 25 approved
|
||
submissions get 403.
|
||
- **Web form.** Without API access, users use
|
||
`https://mangabaka.org/data/submissions/series/{id}/images`. It cannot be
|
||
pre-filled with query parameters, and its SvelteKit CSRF check blocks
|
||
cross-site POSTs.
|
||
- **Open risk:** we have not confirmed that MangaBaka accepts images hosted on
|
||
`pstatic.net`. We do know it rejects imgur.
|
||
|
||
---
|
||
|
||
## 3. Pipeline
|
||
|
||
```
|
||
┌──────────────┐ SourceSeries ┌──────────────────┐
|
||
URL ────▶ │ source.resolve│ ───────────────▶ │ mangabaka.resolve │──▶ MangaBaka series id
|
||
└──────────────┘ └──────────────────┘
|
||
│
|
||
▼
|
||
source.currentCovers + history (Wayback, via source.extractCovers)
|
||
│
|
||
▼ CoverCandidate[]
|
||
images.inspect (download, dimensions, sha256, liveness)
|
||
│
|
||
▼ InspectedCover[]
|
||
covers.dedupe (within candidates + against existing MangaBaka covers)
|
||
│
|
||
▼
|
||
seed.buildCoverSeed ──▶ CoverSeed (existing images + new images)
|
||
│
|
||
▼
|
||
action: print | serve | submit
|
||
```
|
||
|
||
Each stage is a pure function or a small module with injected I/O (`fetch`),
|
||
so it can be tested with fixtures.
|
||
|
||
---
|
||
|
||
## 4. Modules
|
||
|
||
```
|
||
src/
|
||
cli.ts argument parsing (node:util parseArgs), wiring, exit codes
|
||
pipeline.ts runs the stages above; the only place they are composed
|
||
http.ts fetch wrapper: User-Agent, per-host rate limit, retries, timeouts
|
||
model.ts shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
|
||
sources/
|
||
source.ts Source interface + helpers
|
||
registry.ts list of sources; findSource(url)
|
||
webtoons/
|
||
index.ts Webtoons Source implementation
|
||
parse.ts pure HTML → { series, covers } extraction
|
||
urls.ts URL parsing, canonicalisation, image URL normalisation
|
||
history/
|
||
wayback.ts generic Wayback history provider (CDX + archived pages)
|
||
images.ts download + probe dimensions + sha256 + classify by shape
|
||
covers.ts dedupe, merge observations, pick default
|
||
mangabaka/
|
||
client.ts typed API client (public reads + submissions)
|
||
resolve.ts source series → MangaBaka series id
|
||
types.ts API types (subset of the OpenAPI spec we use)
|
||
seed/
|
||
covers.ts CoverSeed builder: existing + new images → submission body
|
||
actions/
|
||
print.ts human-readable or --json output
|
||
serve.ts local review page (node:http, single HTML page, no framework)
|
||
submit.ts snapshot → preview → confirm → save (review mode)
|
||
test/
|
||
fixtures/ saved HTML / CDX / JSON responses
|
||
*.test.ts vitest
|
||
```
|
||
|
||
### 4.1 The `Source` interface (extension point)
|
||
|
||
```ts
|
||
export interface Source {
|
||
/** Stable id, e.g. "webtoons". Used in notes and CLI output. */
|
||
readonly id: string;
|
||
/** Human-readable name, e.g. "WEBTOON". */
|
||
readonly name: string;
|
||
|
||
/** True if this source handles the URL. */
|
||
matches(url: URL): boolean;
|
||
|
||
/** Fetches the series page and returns normalised series info. */
|
||
resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;
|
||
|
||
/** Covers currently shown by the source. */
|
||
currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
|
||
|
||
/**
|
||
* Optional: pure extraction of cover candidates from a series page's HTML.
|
||
* Sources that implement this get Wayback history for free. The history
|
||
* provider runs it on archived copies of `series.archiveUrls`.
|
||
*/
|
||
extractCovers?(html: string, pageUrl: URL): CoverCandidate[];
|
||
|
||
/** Optional source-specific history beyond Wayback (e.g. an API with old art). */
|
||
historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;
|
||
|
||
/** Normalises an image URL to its largest, hotlinkable form. */
|
||
normalizeImageUrl(url: string): string;
|
||
|
||
/** True if a MangaBaka series link points at this source series. */
|
||
isSameSeries(linkUrl: string, series: SourceSeries): boolean;
|
||
}
|
||
|
||
export interface SourceSeries {
|
||
source: string; // Source.id
|
||
externalId: string; // e.g. Webtoons title_no "95"
|
||
title: string;
|
||
language: string; // MangaBaka language code, e.g. "en", "zh-Hant"
|
||
canonicalUrl: string;
|
||
archiveUrls: string[]; // URL prefixes to look up in Wayback
|
||
flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
|
||
}
|
||
|
||
export interface CoverCandidate {
|
||
url: string; // normalised
|
||
role: "poster" | "square" | "banner" | "foreground" | "other";
|
||
current: boolean; // shown on the live page now
|
||
seenAt?: string; // archive timestamp (YYYYMMDD…) if historical
|
||
archivedUrl?: string; // Wayback fallback for the image itself
|
||
}
|
||
```
|
||
|
||
**Adding a source** (e.g. Tapas) means:
|
||
|
||
1. Create `src/sources/tapas/` that implements `Source`. Put the HTML
|
||
extraction in a pure `parse.ts` and give it fixture tests.
|
||
2. Add it to the array in `src/sources/registry.ts`.
|
||
3. Map its language codes to MangaBaka codes.
|
||
|
||
The pipeline, Wayback history, image inspection, deduplication, MangaBaka
|
||
resolution and all actions work unchanged. A source with an API (Tappytoon, for
|
||
example) can skip `extractCovers` and implement `historicalCovers`, or neither.
|
||
|
||
### 4.2 History
|
||
|
||
`history/wayback.ts` knows nothing about any one source. For each prefix in
|
||
`series.archiveUrls` it:
|
||
|
||
1. queries CDX with `collapse=timestamp:6` (one snapshot per month, capped by
|
||
`--history-limit`, default 60);
|
||
2. fetches each snapshot with the `id_` modifier;
|
||
3. calls `source.extractCovers(html, originalUrl)`;
|
||
4. normalises the image URLs with `source.normalizeImageUrl` and tags each
|
||
candidate with `seenAt` and `archivedUrl`.
|
||
|
||
Candidates that share a normalised URL are merged, and the merged candidate
|
||
records the first and last time it was seen. Wayback failures (timeouts, 5xx,
|
||
rate limits) produce a warning and an empty history; they are never fatal.
|
||
`--no-history` skips this stage.
|
||
|
||
### 4.3 Image inspection and classification
|
||
|
||
`images.ts` downloads each candidate (live URL first, then `archivedUrl`) and
|
||
records its `width`, `height`, `sha256`, `bytes` and `live` status. Dimensions
|
||
come from the image header, read with the `image-size` package. Candidates that
|
||
cannot be downloaded at all are dropped with a warning.
|
||
|
||
Classification by aspect ratio (h / w):
|
||
|
||
| Shape | Ratio | MangaBaka `type` |
|
||
|---|---|---|
|
||
| portrait | ≥ 1.15 | `volume` |
|
||
| square-ish | 0.85 – 1.15 | `other` |
|
||
| wide | < 0.5 | `banner` |
|
||
| otherwise | – | `other` |
|
||
|
||
### 4.4 Dedupe and default selection (`covers.ts`)
|
||
|
||
- **Duplicates.** Images with the same `sha256` are merged. The merged entry
|
||
keeps the live URL and the widest `seenAt` range.
|
||
- **Already on MangaBaka.** Candidates whose `sha256` matches an existing
|
||
MangaBaka image are dropped and reported as "already present".
|
||
- **Default selection.**
|
||
- The **default** is the current live poster.
|
||
- If there is none (canvas series), it is the current live square.
|
||
- The new default gets `is_default: true`, and every existing image gets
|
||
`is_default: false`. This is the "main cover" for the language edition.
|
||
- The reviewer on MangaBaka can still reject that change.
|
||
- **Notes.** Every new image gets a note, e.g.
|
||
`"WEBTOON poster (current)"` or `"WEBTOON poster, seen 2019-03 – 2021-01"`.
|
||
|
||
### 4.5 MangaBaka series resolution (`mangabaka/resolve.ts`)
|
||
|
||
1. If `--series <id>` is given, use it, and warn if no link of that series
|
||
passes `source.isSameSeries`.
|
||
2. Otherwise call `/v2/series/match?q=<title>`, then `/v2/series/search`. For
|
||
each result, fetch `/v1/series/{id}/links` and keep the series whose link
|
||
passes `source.isSameSeries`. For Webtoons that means the same `title_no`.
|
||
3. If no series or several series match, stop with exit code 2 and print the
|
||
candidates (id, title, URL) so the user can rerun with `--series`.
|
||
|
||
### 4.6 Seed (`seed/covers.ts`)
|
||
|
||
`buildCoverSeed(existing, newCovers, series)` returns:
|
||
|
||
```ts
|
||
interface CoverSeed {
|
||
seriesId: number;
|
||
userNote: string; // "Add WEBTOON covers from <canonicalUrl>"
|
||
images: SubmissionImage[]; // existing (unchanged except is_default) + new
|
||
added: InspectedCover[]; // for display
|
||
skipped: { cover: InspectedCover; reason: string }[];
|
||
}
|
||
```
|
||
|
||
Each new image becomes:
|
||
`{ id: null, series_id, work_id: null, url, index: null, index_numeric: null,
|
||
language: series.language, type, note, content_rating: "safe", is_default }`.
|
||
|
||
### 4.7 Actions
|
||
|
||
| Action | Default when | Effect |
|
||
|---|---|---|
|
||
| `print` | `--action print` | Prints series, candidates, the planned seed, and skips. `--json` prints the seed as JSON. No writes. |
|
||
| `serve` | no `MANGABAKA_TOKEN` | Serves `http://localhost:8999` (see below). |
|
||
| `submit` | `MANGABAKA_TOKEN` set | Submits through the API (see below). |
|
||
|
||
**`serve`** shows a review page with one card per new cover: a thumbnail,
|
||
dimensions, type, language, note, a "copy URL" button and a checkbox. It also
|
||
has a "Copy all selected" button and a link to
|
||
`https://mangabaka.org/data/submissions/series/{id}/images`. The page is a
|
||
single HTML string rendered server-side with inline JS and no framework. It
|
||
binds to localhost only.
|
||
|
||
**`submit`**:
|
||
|
||
1. `GET` the snapshot (`version` plus current images) and rebuild the seed on
|
||
top of it.
|
||
2. `POST …/preview` and print the diff.
|
||
3. Ask "Submit for review? [y/N]". `--yes` skips this.
|
||
4. `POST` with `save_mode: "review"`.
|
||
5. Handle the outcome:
|
||
- **409:** fetch the snapshot again, rebuild, preview again and ask again,
|
||
at most once.
|
||
- **403:** explain the 25-approved-submissions requirement and fall back to
|
||
`serve`.
|
||
|
||
---
|
||
|
||
## 5. Cross-cutting concerns
|
||
|
||
- **HTTP.** All network access goes through `http.ts`. It:
|
||
- sends a browser User-Agent;
|
||
- limits each host to 1 request per second, and Wayback to 1 every 1.5 s;
|
||
- times out after 20 s;
|
||
- retries twice with backoff on 429/5xx;
|
||
- never sends the MangaBaka token to any other host.
|
||
- **Logging.** Progress goes to stderr and results to stdout, so
|
||
`print --json` can be piped. `--verbose` logs every request.
|
||
- **Exit codes.**
|
||
- 0: success, including "nothing new to add"
|
||
- 1: unexpected error
|
||
- 2: bad usage, an unsupported URL, or an ambiguous or missing series match
|
||
- 3: the user declined to submit.
|
||
- **Config.** `MANGABAKA_TOKEN` (env), `MANGABAKA_API` (env, default
|
||
`https://api.mangabaka.org`, for testing).
|
||
|
||
---
|
||
|
||
## 6. Testing
|
||
|
||
- **Unit tests with vitest**, run against fixtures saved from real responses:
|
||
- Webtoons pages (official, canvas, an archived 2015 snapshot) and CDX output
|
||
- MangaBaka match, links, images, snapshot and preview JSON.
|
||
- **Pure functions under test:**
|
||
- parsing, URL normalisation and canonicalisation
|
||
- classification, dedupe and default selection
|
||
- seed building and the language mapping.
|
||
- **I/O modules** (`pipeline`, `mangabaka/client`, `actions/submit`,
|
||
`history/wayback`) are tested with an injected fake `fetch`.
|
||
- **No network in tests.** A manual smoke test is documented in the README:
|
||
`mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95`.
|