Files

T

Lucas WintherandClaude Opus 5.5 9eca8f4f82 docs: add README and architecture spec for cover seeder

Co-Authored-By: Claude Opus 5.5 <[email protected]>

2026-10-14 04:53:30 +02:00

17 KiB

Raw Blame History

Architecture

mangabaka-seeder is a command-line tool that gathers data about a series from an external site (a "source") and turns it into an edit request ("submission") for MangaBaka. It is modeled on yambs, the MusicBrainz seeder: sources turn into seeds, and an action decides what happens to them. The user always stays in control of what is submitted.

The first feature is covers: given a Webtoons series URL, collect every cover that series has used (current and historical) and request them added to the matching MangaBaka series, with the current poster as the default (main) cover for the edition's language.

This document is the design spec. It records the facts we verified about the external services, the module boundaries, and the extension points.

1. Goals and non-goals

Goals

One command: mbseed <url> produces a ready-to-review cover submission.
Find historical covers, not just the current one.
Work for everyone. With a MangaBaka API token, submit through the API. Without one, fall back to a local review page that helps with manual entry.
New sources (Tapas, Tappytoon, Lezhin, Manta, …) can be added as one new module under src/sources/, with no changes to the pipeline, the MangaBaka client or the actions.
Never change MangaBaka data silently. API submissions always go through preview, explicit confirmation and save_mode: "review".

Non-goals (for now)

Editing series fields other than images, such as titles, links or tags. The submission layer is keyed by MangaBaka "resource" so this can come later.
Following a Webtoons series across language editions. Each title_no belongs to one language edition; run the tool once per edition URL.
Re-hosting images. We submit URLs; MangaBaka downloads them.

2. Verified external facts

These were verified on 2026-10-14. Treat them as assumptions that the fixtures in test/fixtures/ pin down.

2.1 Webtoons

What	Where
Portrait poster (official series; 480×623 original)	`meta[property="og:image"]`
Square thumbnail	`meta[name="twitter:image"]`
Hero background (1920×321, official only)	`div.detail_bg` inline `style` → `background:url('…')`
Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square)	`.detail_header span.thmb img[src]`
Title	`meta[property="og:title"]` (fallback `h1.subj` / `h3.subj`)
Canonical URL	`link[rel="canonical"]`
`title_no`	`title_no` query parameter
Language	first path segment (`/en/`, `/fr/`, `/zh-hant/`, …)
Canvas vs official	`/canvas/` (or `/challenge/`) in the path; inline JS `isChallenge : true`

Canvas series have no portrait poster. Their og:image is a 1080×1080 square.
Image hosts. webtoon-phinf.pstatic.net returns 403 without a Webtoons Referer; swebtoon-phinf.pstatic.net serves the same path with no Referer.
Original size. The ?type=… query selects a resized variant. Removing it returns the original. crop540_540 crops posters, so always strip the query.
Old hosts. Old snapshots reference swebtoon.phinf.naver.net, which becomes swebtoon-phinf.pstatic.net with the same path.
No limits so far. A browser User-Agent is enough. Age-gated titles still serve full HTML, and no cookies are required.
No usable JSON API. The app API needs signed requests. We scrape HTML.

2.2 Historical covers (Wayback Machine)

Snapshot index. The CDX API at https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original returns at most one snapshot per month.
Raw pages. https://web.archive.org/web/<ts>id_/<original> returns the page as it was archived, without Wayback's toolbar.
History exists. For Tower of God, snapshots from 2015 to 2025 reference five distinct cover images. All the older ones tested are still served live from swebtoon-phinf.pstatic.net.
Fallback. If an image is no longer live, it is often still at https://web.archive.org/web/<ts>id_/<image-url>.
Not every image is a cover. Some historical og:images are landing-page art, so candidates are classified by shape and reviewed by the user.

2.3 MangaBaka

Endpoints. The site is https://mangabaka.org and the API is https://api.mangabaka.org. The OpenAPI spec is at https://mangabaka.org/api.json, with docs at https://mangabaka.org/data/api.
Public reads.
- GET /v2/series/match?q=<title> and GET /v2/series/search?q=… find series. They are limited to about 30 requests per minute.
- GET /v1/series/{id} and GET /v1/series/{id}/links return the series and its links. Links are canonicalised, e.g. https://www.webtoons.com/-/-/-/list?title_no=95.
- GET /v1/series/{id}/images returns covers with type, language, index, note, content_rating, is_default and hashes.sha256.
Image type values: volume (default), volume_back, chapter, season, banner, audiobook, other. There is no "main" type; the main cover is the one with is_default: true.
Submissions API. Auth is x-api-key: mb-…. All endpoints are v0 and marked internal.
- GET /v0/my/submissions/series-images/{id} returns the current image set and a version token.
- POST /v0/my/submissions/series-images/{id}/preview is a dry run that returns a diff.
- POST /v0/my/submissions/series-images/{id} saves, with body { version, user_note, save_mode, data: { images: [...] } }.
- images replaces the whole set, so it must contain the existing images.
- A stale version gives 409.
- Users without the contributor role and with fewer than 25 approved submissions get 403.
Web form. Without API access, users use https://mangabaka.org/data/submissions/series/{id}/images. It cannot be pre-filled with query parameters, and its SvelteKit CSRF check blocks cross-site POSTs.
Open risk: we have not confirmed that MangaBaka accepts images hosted on pstatic.net. We do know it rejects imgur.

3. Pipeline

           ┌──────────────┐   SourceSeries    ┌──────────────────┐
 URL ────▶ │ source.resolve│ ───────────────▶ │ mangabaka.resolve │──▶ MangaBaka series id
           └──────────────┘                   └──────────────────┘
                  │
                  ▼
     source.currentCovers  +  history (Wayback, via source.extractCovers)
                  │
                  ▼  CoverCandidate[]
           images.inspect  (download, dimensions, sha256, liveness)
                  │
                  ▼  InspectedCover[]
           covers.dedupe   (within candidates + against existing MangaBaka covers)
                  │
                  ▼
           seed.buildCoverSeed  ──▶ CoverSeed (existing images + new images)
                  │
                  ▼
           action: print | serve | submit

Each stage is a pure function or a small module with injected I/O (fetch), so it can be tested with fixtures.

4. Modules

src/
  cli.ts                  argument parsing (node:util parseArgs), wiring, exit codes
  pipeline.ts             runs the stages above; the only place they are composed
  http.ts                 fetch wrapper: User-Agent, per-host rate limit, retries, timeouts
  model.ts                shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
  sources/
    source.ts             Source interface + helpers
    registry.ts           list of sources; findSource(url)
    webtoons/
      index.ts            Webtoons Source implementation
      parse.ts            pure HTML → { series, covers } extraction
      urls.ts             URL parsing, canonicalisation, image URL normalisation
  history/
    wayback.ts            generic Wayback history provider (CDX + archived pages)
  images.ts               download + probe dimensions + sha256 + classify by shape
  covers.ts               dedupe, merge observations, pick default
  mangabaka/
    client.ts             typed API client (public reads + submissions)
    resolve.ts            source series → MangaBaka series id
    types.ts              API types (subset of the OpenAPI spec we use)
  seed/
    covers.ts             CoverSeed builder: existing + new images → submission body
  actions/
    print.ts              human-readable or --json output
    serve.ts              local review page (node:http, single HTML page, no framework)
    submit.ts             snapshot → preview → confirm → save (review mode)
test/
  fixtures/               saved HTML / CDX / JSON responses
  *.test.ts               vitest

4.1 The `Source` interface (extension point)

export interface Source {
  /** Stable id, e.g. "webtoons". Used in notes and CLI output. */
  readonly id: string;
  /** Human-readable name, e.g. "WEBTOON". */
  readonly name: string;

  /** True if this source handles the URL. */
  matches(url: URL): boolean;

  /** Fetches the series page and returns normalised series info. */
  resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;

  /** Covers currently shown by the source. */
  currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;

  /**
   * Optional: pure extraction of cover candidates from a series page's HTML.
   * Sources that implement this get Wayback history for free. The history
   * provider runs it on archived copies of `series.archiveUrls`.
   */
  extractCovers?(html: string, pageUrl: URL): CoverCandidate[];

  /** Optional source-specific history beyond Wayback (e.g. an API with old art). */
  historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;

  /** Normalises an image URL to its largest, hotlinkable form. */
  normalizeImageUrl(url: string): string;

  /** True if a MangaBaka series link points at this source series. */
  isSameSeries(linkUrl: string, series: SourceSeries): boolean;
}

export interface SourceSeries {
  source: string;          // Source.id
  externalId: string;      // e.g. Webtoons title_no "95"
  title: string;
  language: string;        // MangaBaka language code, e.g. "en", "zh-Hant"
  canonicalUrl: string;
  archiveUrls: string[];   // URL prefixes to look up in Wayback
  flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
}

export interface CoverCandidate {
  url: string;             // normalised
  role: "poster" | "square" | "banner" | "foreground" | "other";
  current: boolean;        // shown on the live page now
  seenAt?: string;         // archive timestamp (YYYYMMDD…) if historical
  archivedUrl?: string;    // Wayback fallback for the image itself
}

Adding a source (e.g. Tapas) means:

Create src/sources/tapas/ that implements Source. Put the HTML extraction in a pure parse.ts and give it fixture tests.
Add it to the array in src/sources/registry.ts.
Map its language codes to MangaBaka codes.

The pipeline, Wayback history, image inspection, deduplication, MangaBaka resolution and all actions work unchanged. A source with an API (Tappytoon, for example) can skip extractCovers and implement historicalCovers, or neither.

4.2 History

history/wayback.ts knows nothing about any one source. For each prefix in series.archiveUrls it:

queries CDX with collapse=timestamp:6 (one snapshot per month, capped by --history-limit, default 60);
fetches each snapshot with the id_ modifier;
calls source.extractCovers(html, originalUrl);
normalises the image URLs with source.normalizeImageUrl and tags each candidate with seenAt and archivedUrl.

Candidates that share a normalised URL are merged, and the merged candidate records the first and last time it was seen. Wayback failures (timeouts, 5xx, rate limits) produce a warning and an empty history; they are never fatal. --no-history skips this stage.

4.3 Image inspection and classification

images.ts downloads each candidate (live URL first, then archivedUrl) and records its width, height, sha256, bytes and live status. Dimensions come from the image header, read with the image-size package. Candidates that cannot be downloaded at all are dropped with a warning.

Classification by aspect ratio (h / w):

Shape	Ratio	MangaBaka `type`
portrait	≥ 1.15	`volume`
square-ish	0.85 – 1.15	`other`
wide	< 0.5	`banner`
otherwise	–	`other`

4.4 Dedupe and default selection (`covers.ts`)

Duplicates. Images with the same sha256 are merged. The merged entry keeps the live URL and the widest seenAt range.
Already on MangaBaka. Candidates whose sha256 matches an existing MangaBaka image are dropped and reported as "already present".
Default selection.
- The default is the current live poster.
- If there is none (canvas series), it is the current live square.
- The new default gets is_default: true, and every existing image gets is_default: false. This is the "main cover" for the language edition.
- The reviewer on MangaBaka can still reject that change.
Notes. Every new image gets a note, e.g. "WEBTOON poster (current)" or "WEBTOON poster, seen 2019-03 – 2021-01".

4.5 MangaBaka series resolution (`mangabaka/resolve.ts`)

If --series <id> is given, use it, and warn if no link of that series passes source.isSameSeries.
Otherwise call /v2/series/match?q=<title>, then /v2/series/search. For each result, fetch /v1/series/{id}/links and keep the series whose link passes source.isSameSeries. For Webtoons that means the same title_no.
If no series or several series match, stop with exit code 2 and print the candidates (id, title, URL) so the user can rerun with --series.

4.6 Seed (`seed/covers.ts`)

buildCoverSeed(existing, newCovers, series) returns:

interface CoverSeed {
  seriesId: number;
  userNote: string;  // "Add WEBTOON covers from <canonicalUrl>"
  images: SubmissionImage[]; // existing (unchanged except is_default) + new
  added: InspectedCover[];   // for display
  skipped: { cover: InspectedCover; reason: string }[];
}

Each new image becomes: { id: null, series_id, work_id: null, url, index: null, index_numeric: null, language: series.language, type, note, content_rating: "safe", is_default }.

4.7 Actions

Action	Default when	Effect
`print`	`--action print`	Prints series, candidates, the planned seed, and skips. `--json` prints the seed as JSON. No writes.
`serve`	no `MANGABAKA_TOKEN`	Serves `http://localhost:8999` (see below).
`submit`	`MANGABAKA_TOKEN` set	Submits through the API (see below).

serve shows a review page with one card per new cover: a thumbnail, dimensions, type, language, note, a "copy URL" button and a checkbox. It also has a "Copy all selected" button and a link to https://mangabaka.org/data/submissions/series/{id}/images. The page is a single HTML string rendered server-side with inline JS and no framework. It binds to localhost only.

submit:

GET the snapshot (version plus current images) and rebuild the seed on top of it.
POST …/preview and print the diff.
Ask "Submit for review? [y/N]". --yes skips this.
POST with save_mode: "review".
Handle the outcome:
- 409: fetch the snapshot again, rebuild, preview again and ask again, at most once.
- 403: explain the 25-approved-submissions requirement and fall back to serve.

5. Cross-cutting concerns

HTTP. All network access goes through http.ts. It:
- sends a browser User-Agent;
- limits each host to 1 request per second, and Wayback to 1 every 1.5 s;
- times out after 20 s;
- retries twice with backoff on 429/5xx;
- never sends the MangaBaka token to any other host.
Logging. Progress goes to stderr and results to stdout, so print --json can be piped. --verbose logs every request.
Exit codes.
- 0: success, including "nothing new to add"
- 1: unexpected error
- 2: bad usage, an unsupported URL, or an ambiguous or missing series match
- 3: the user declined to submit.
Config. MANGABAKA_TOKEN (env), MANGABAKA_API (env, default https://api.mangabaka.org, for testing).

6. Testing

Unit tests with vitest, run against fixtures saved from real responses:
- Webtoons pages (official, canvas, an archived 2015 snapshot) and CDX output
- MangaBaka match, links, images, snapshot and preview JSON.
Pure functions under test:
- parsing, URL normalisation and canonicalisation
- classification, dedupe and default selection
- seed building and the language mapping.
I/O modules (pipeline, mangabaka/client, actions/submit, history/wayback) are tested with an injected fake fetch.
No network in tests. A manual smoke test is documented in the README: mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95.

17 KiB Raw Blame History Unescape Escape