lucasw89/mangabaka-seeder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a9c049993a31ef55ca7c528913fcd25c617e0138
mangabaka-seeder/docs/ARCHITECTURE.md
T

23 KiB
Raw Blame History

Architecture

mangabaka-seeder is a command-line tool that gathers data about a series from an external site (a "source") and turns it into an edit request ("submission") for MangaBaka. It is modeled on yambs, the MusicBrainz seeder: sources turn into seeds, and an action decides what happens to them. The user always stays in control of what is submitted.

The first feature is covers: given a Webtoons series URL, collect every cover that series has used (current and historical) and request them added to the matching MangaBaka series, with the current poster as the default (main) cover for the edition's language.

This document is the design spec. It records the facts we verified about the external services, the module boundaries, and the extension points.

1. Goals and non-goals

Goals

  • One command: mbseed <url> produces a ready-to-review cover submission.
  • Find historical covers, not just the current one.
  • Work for everyone. With a MangaBaka API token, submit through the API. Without one, fall back to a local review page that helps with manual entry.
  • New sources (Tapas, Tappytoon, Lezhin, Manta, …) can be added as one new module under src/sources/, with no changes to the pipeline, the MangaBaka client or the actions.
  • Never change MangaBaka data silently. API submissions always go through preview, explicit confirmation and save_mode: "review".
  • Never upload anything without explicit user action. The user chooses the content rating (default safe).
  • Image hosts are pluggable too (catbox.moe by default, any S3-compatible bucket, or no rehosting).

Non-goals (for now)

  • Editing series fields other than images, such as titles, links or tags. The submission layer is keyed by MangaBaka "resource" so this can come later.
  • Following a Webtoons series across language editions. Each title_no belongs to one language edition; run the tool once per edition URL.
  • Hosting images on our own infrastructure by default. We rehost on a public image host (catbox.moe), or on a user-configured S3-compatible bucket.

2. Verified external facts

These were verified on 2026-10-14. Treat them as assumptions that the fixtures in test/fixtures/ pin down.

2.1 Webtoons

What Where
Portrait poster (official series; 480×623 original) meta[property="og:image"]
Square thumbnail meta[name="twitter:image"]
Hero background (1920×321, official only) div.detail_bg inline style → background:url('…')
Hero foreground (1200×240 PNG; on canvas series, the 1080×1080 square) .detail_header span.thmb img[src]
Title meta[property="og:title"] (fallback h1.subj / h3.subj)
Canonical URL link[rel="canonical"]
title_no title_no query parameter
Language first path segment (/en/, /fr/, /zh-hant/, …)
Canvas vs official /canvas/ (or /challenge/) in the path; inline JS isChallenge : true
  • Canvas series have no portrait poster. Their og:image is a 1080×1080 square.
  • Image hosts. webtoon-phinf.pstatic.net returns 403 without a Webtoons Referer; swebtoon-phinf.pstatic.net serves the same path with no Referer.
  • Original size. The ?type=… query selects a resized variant. Removing it returns the original. crop540_540 crops posters, so always strip the query.
  • Old hosts. Old snapshots reference swebtoon.phinf.naver.net, which becomes swebtoon-phinf.pstatic.net with the same path.
  • No limits so far. A browser User-Agent is enough. Age-gated titles still serve full HTML, and no cookies are required.
  • No usable JSON API. The app API needs signed requests. We scrape HTML.

2.2 Historical covers (Wayback Machine)

  • Snapshot index. The CDX API at https://web.archive.org/cdx/search/cdx?url=<page>&matchType=prefix&filter=statuscode:200&collapse=timestamp:6&fl=timestamp,original returns at most one snapshot per month.
  • Raw pages. https://web.archive.org/web/<ts>id_/<original> returns the page as it was archived, without Wayback's toolbar.
  • History exists. For Tower of God, snapshots from 2015 to 2025 reference five distinct cover images. All the older ones tested are still served live from swebtoon-phinf.pstatic.net.
  • Fallback. If an image is no longer live, it is often still at https://web.archive.org/web/<ts>id_/<image-url>.
  • Not every image is a cover. Some historical og:images are landing-page art, so candidates are classified by shape and reviewed by the user.

2.3 MangaBaka

  • Endpoints. The site is https://mangabaka.org and the API is https://api.mangabaka.org. The OpenAPI spec is at https://mangabaka.org/api.json, with docs at https://mangabaka.org/data/api.
  • Public reads.
    • GET /v2/series/match?q=<title> and GET /v2/series/search?q=… find series. They are limited to about 30 requests per minute.
    • GET /v1/series/{id} and GET /v1/series/{id}/links return the series and its links. Links are canonicalised, e.g. https://www.webtoons.com/-/-/-/list?title_no=95.
    • GET /v1/series/{id}/images returns covers with type, language, index, note, content_rating, is_default and hashes.sha256.
  • Image type values: volume (default), volume_back, chapter, season, banner, audiobook, other. There is no "main" type; the main cover is the one with is_default: true.
  • Submissions API. Auth is x-api-key: mb-…. All endpoints are v0 and marked internal.
    • GET /v0/my/submissions/series-images/{id} returns the current image set and a version token.
    • POST /v0/my/submissions/series-images/{id}/preview is a dry run that returns a diff.
    • POST /v0/my/submissions/series-images/{id} saves, with body { version, user_note, save_mode, data: { images: [...] } }.
    • images replaces the whole set, so it must contain the existing images.
    • A stale version gives 409.
    • Users without the contributor role and with fewer than 25 approved submissions get 403.
  • Web form. Without API access, users use https://mangabaka.org/data/submissions/series/{id}/images. It cannot be pre-filled with query parameters, and its SvelteKit CSRF check blocks cross-site POSTs.
  • Image URLs. The spec describes the cover url as "Cover image URL. imgur is rejected (aggressive rate limiting). Acts as the cover identity on save."
    • We don't know whether MangaBaka can fetch from pstatic.net (it is hotlink-protected on some hosts).
    • We don't know whether MangaBaka copies the image when the request is submitted or when a reviewer approves it.
    • So by default we rehost each new image on a host that keeps it (see §4.6).
  • content_rating: one of safe, suggestive, erotica, pornographic. The user chooses it (§4.7).

3. Pipeline

           ┌──────────────┐   SourceSeries    ┌──────────────────┐
 URL ────▶ │ source.resolve│ ───────────────▶ │ mangabaka.resolve │──▶ MangaBaka series id
           └──────────────┘                   └──────────────────┘
                  │
                  ▼
     source.currentCovers  +  history (Wayback, via source.extractCovers)
                  │
                  ▼  CoverCandidate[]
           images.inspect  (download, dimensions, sha256, liveness)
                  │
                  ▼  InspectedCover[]
           covers.dedupe   (within candidates + against existing MangaBaka covers)
                  │
                  ▼
           seed.buildCoverSeed  ──▶ CoverSeed (existing images + new images,
                  │                            content_rating chosen by the user)
                  ▼
           action: print | serve | submit
                  │
                  ▼  (only after explicit user action: serve button / submit confirm)
           imageHost.upload  ──▶ rehosted URLs replace source URLs in the seed

Each stage is a pure function or a small module with injected I/O (fetch), so it can be tested with fixtures.

4. Modules

src/
  cli.ts                  argument parsing (node:util parseArgs), wiring, exit codes
  pipeline.ts             runs the stages above; the only place they are composed
  http.ts                 fetch wrapper: User-Agent, per-host rate limit, retries, timeouts
  model.ts                shared types (SourceSeries, CoverCandidate, InspectedCover, CoverSeed)
  sources/
    source.ts             Source interface + helpers
    registry.ts           list of sources; findSource(url)
    webtoons/
      index.ts            Webtoons Source implementation
      parse.ts            pure HTML → { series, covers } extraction
      urls.ts             URL parsing, canonicalisation, image URL normalisation
  history/
    wayback.ts            generic Wayback history provider (CDX + archived pages)
  images.ts               download + probe dimensions + sha256 + classify by shape
  covers.ts               dedupe, merge observations, pick default
  mangabaka/
    client.ts             typed API client (public reads + submissions)
    resolve.ts            source series → MangaBaka series id
    types.ts              API types (subset of the OpenAPI spec we use)
  hosting/
    host.ts               ImageHost interface + upload cache (sha256 → url)
    registry.ts           list of hosts; selectHost(name, env)
    catbox.ts             catbox.moe (default)
    s3.ts                 S3-compatible bucket (R2, B2, MinIO, AWS) via aws4fetch
    direct.ts             no rehosting: submit source URLs as-is
  seed/
    covers.ts             CoverSeed builder: existing + new images → submission body
  actions/
    print.ts              human-readable or --json output
    serve.ts              local review page (node:http, single HTML page, no framework)
    submit.ts             snapshot → preview → confirm → save (review mode)
test/
  fixtures/               saved HTML / CDX / JSON responses
  *.test.ts               vitest

4.1 The Source interface (extension point)

export interface Source {
  /** Stable id, e.g. "webtoons". Used in notes and CLI output. */
  readonly id: string;
  /** Human-readable name, e.g. "WEBTOON". */
  readonly name: string;

  /** True if this source handles the URL. */
  matches(url: URL): boolean;

  /** Fetches the series page and returns normalised series info. */
  resolve(url: URL, ctx: SourceContext): Promise<SourceSeries>;

  /** Covers currently shown by the source. */
  currentCovers(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;

  /**
   * Optional: pure extraction of cover candidates from a series page's HTML.
   * Sources that implement this get Wayback history for free. The history
   * provider runs it on archived copies of `series.archiveUrls`.
   */
  extractCovers?(html: string, pageUrl: URL): CoverCandidate[];

  /** Optional source-specific history beyond Wayback (e.g. an API with old art). */
  historicalCovers?(series: SourceSeries, ctx: SourceContext): Promise<CoverCandidate[]>;

  /** Normalises an image URL to its largest, hotlinkable form. */
  normalizeImageUrl(url: string): string;

  /** True if a MangaBaka series link points at this source series. */
  isSameSeries(linkUrl: string, series: SourceSeries): boolean;
}

export interface SourceSeries {
  source: string;          // Source.id
  externalId: string;      // e.g. Webtoons title_no "95"
  title: string;
  language: string;        // MangaBaka language code, e.g. "en", "zh-Hant"
  canonicalUrl: string;
  archiveUrls: string[];   // URL prefixes to look up in Wayback
  mature: boolean;         // source flags the series as mature/age-gated (§4.7)
  flags: Record<string, boolean>; // source-specific, e.g. { canvas: true }
}

export interface CoverCandidate {
  url: string;             // normalised
  role: "poster" | "square" | "banner" | "foreground" | "other";
  current: boolean;        // shown on the live page now
  seenAt?: string;         // archive timestamp (YYYYMMDD…) if historical
  archivedUrl?: string;    // Wayback fallback for the image itself
}

Adding a source (e.g. Tapas) means:

  1. Create src/sources/tapas/ that implements Source. Put the HTML extraction in a pure parse.ts and give it fixture tests.
  2. Add it to the array in src/sources/registry.ts.
  3. Map its language codes to MangaBaka codes.

The pipeline, Wayback history, image inspection, deduplication, MangaBaka resolution and all actions work unchanged. A source with an API (Tappytoon, for example) can skip extractCovers and implement historicalCovers, or neither.

4.2 History

history/wayback.ts knows nothing about any one source. For each prefix in series.archiveUrls it:

  1. queries CDX with collapse=timestamp:6 (one snapshot per month, capped by --history-limit, default 60);
  2. fetches each snapshot with the id_ modifier;
  3. calls source.extractCovers(html, originalUrl);
  4. normalises the image URLs with source.normalizeImageUrl and tags each candidate with seenAt and archivedUrl.

Candidates that share a normalised URL are merged, and the merged candidate records the first and last time it was seen. Wayback failures (timeouts, 5xx, rate limits) produce a warning and an empty history; they are never fatal. --no-history skips this stage.

4.3 Image inspection and classification

images.ts downloads each candidate (live URL first, then archivedUrl) and records its width, height, sha256, contentType and live status. It keeps the bytes in memory for a later upload (§4.6). Covers we've seen range from about 20 KB to 650 KB. A few dozen candidates fit easily in memory, and downloads over 20 MB are refused as a safety cap. Dimensions come from the image header, read with the image-size package. Candidates that cannot be downloaded at all are dropped with a warning.

Classification by aspect ratio (h / w):

Shape Ratio MangaBaka type
portrait ≥ 1.15 volume
square-ish 0.85 – 1.15 other
wide < 0.5 banner
otherwise – other

4.4 Dedupe and default selection (covers.ts)

  • Duplicates. Images with the same sha256 are merged. The merged entry keeps the live URL and the widest seenAt range.
  • Already on MangaBaka. Candidates whose sha256 matches an existing MangaBaka image are dropped and reported as "already present".
  • Default selection.
    • The default is the current live poster.
    • If there is none (canvas series), it is the current live square.
    • The new default gets is_default: true, and every existing image gets is_default: false. This is the "main cover" for the language edition.
    • The reviewer on MangaBaka can still reject that change.
  • Notes. Every new image gets a note, e.g. "WEBTOON poster (current)" or "WEBTOON poster, seen 2019-03 – 2021-01".

4.5 MangaBaka series resolution (mangabaka/resolve.ts)

  1. If --series <id> is given, use it, and warn if no link of that series passes source.isSameSeries.
  2. Otherwise call /v2/series/match?q=<title>, then /v2/series/search. For each result, fetch /v1/series/{id}/links and keep the series whose link passes source.isSameSeries. For Webtoons that means the same title_no.
  3. If no series or several series match, stop with exit code 2 and print the candidates (id, title, URL) so the user can rerun with --series.

4.6 Image hosting (hosting/)

MangaBaka must be able to download every image we submit, and it may not do so until a reviewer approves the request. So new images are rehosted on a host that keeps them. This is a second extension point, shaped like sources:

export interface ImageHost {
  readonly id: string;               // "catbox" | "s3" | "direct" | …
  /** Throws a descriptive error if required config/env is missing. */
  checkConfig(env: NodeJS.ProcessEnv): void;
  /** Uploads the bytes and returns a public, durable URL. */
  upload(image: { bytes: Uint8Array; sha256: string; contentType: string;
                  filename: string }, ctx: HostContext): Promise<string>;
}
Host Selected by Notes
catbox (default) --image-host catbox Uploads to https://catbox.moe/user/api.php (reqtype=fileupload, multipart fileToUpload). Free, no account, files are permanent. Set CATBOX_USERHASH to attach uploads to your catbox account so you can delete them later.
s3 --image-host s3 Any S3-compatible bucket (Cloudflare R2, Backblaze B2, MinIO, AWS). Signed PUT with aws4fetch. Key is <source>/<externalId>/<sha256>.<ext>. Configured by MBSEED_S3_ENDPOINT, MBSEED_S3_BUCKET, MBSEED_S3_REGION (default auto), MBSEED_S3_ACCESS_KEY_ID, MBSEED_S3_SECRET_ACCESS_KEY and MBSEED_S3_PUBLIC_URL. The public URL is <PUBLIC_URL>/<key>.
direct --image-host direct No upload: submits the normalised source URL. Use it if MangaBaka turns out to fetch swebtoon-phinf.pstatic.net fine.

The upload rules:

  • Reuse earlier uploads. Uploads are cached in $XDG_CACHE_HOME/mangabaka-seeder/uploads.json, keyed by host id and sha256, so rerunning never uploads the same image twice.
  • Upload the bytes we checked. We send the bytes downloaded in §4.3, not a URL for the host to fetch. That way the host gets exactly the image whose hash we checked against MangaBaka.
  • No upload without an explicit user action. Images go up only when the user clicks Rehost selected in serve, or answers yes in submit. print never uploads.
  • Failures are per image. If an upload fails, that image is reported and left out of the submission, and the others continue. If every upload fails, the action aborts with exit code 1.
  • Adding a host (e.g. a future "ours" server) is one file in hosting/ plus one line in hosting/registry.ts.

4.7 Content rating

MangaBaka needs a content_rating per image. The tool can't judge this reliably, so the user chooses it, with safe as the default:

  • --content-rating <rating> sets it for all new images.
  • submit with no flag on a terminal asks Content rating for new covers [safe/suggestive/erotica/pornographic] (safe):, and Enter means safe.
  • submit --yes requires --content-rating explicitly, so an unattended run never assumes a rating; without it, exit code 2. On a non-interactive stdin without --yes, the same rule applies.
  • serve has a rating selector on each card, preset to the CLI value (default safe). Selected ratings are shown next to each copied URL.
  • print shows the rating it would use (--content-rating or safe).
  • Mature titles. If the source marks the series as mature (e.g. Webtoons ageGradeNotice : true), the prompt and the review page show a warning so the user considers a rating above safe. The default is still safe.

4.8 Seed (seed/covers.ts)

buildCoverSeed(existing, newCovers, series, { contentRating }) returns:

interface CoverSeed {
  seriesId: number;
  userNote: string;  // "Add WEBTOON covers from <canonicalUrl>"
  images: SubmissionImage[]; // existing (unchanged except is_default) + new
  added: InspectedCover[];   // for display
  skipped: { cover: InspectedCover; reason: string }[];
}

Each new image becomes: { id: null, series_id, work_id: null, url, index: null, index_numeric: null, language: series.language, type, note, content_rating, is_default }. url starts as the normalised source URL. withHostedUrls(seed, urlMap) swaps in the rehosted URLs once the upload has happened.

4.9 Actions

Action Default when Effect
print --action print Prints series, candidates, the planned seed, and skips. --json prints the seed as JSON. No uploads or writes.
serve no MANGABAKA_TOKEN Serves http://localhost:8999 (see below).
submit MANGABAKA_TOKEN set Submits through the API (see below).

serve shows a review page with one card per new cover. Each card has:

  • a thumbnail, its dimensions, the suggested type, language and note
  • a content-rating selector
  • a checkbox.

The page also has:

  • "Rehost selected", which uploads the selected images to the configured host and replaces the shown URLs with the hosted ones. With --image-host direct it is hidden.
  • "Copy all selected", which copies the URLs.
  • a link to https://mangabaka.org/data/submissions/series/{id}/images.

It is a single HTML string rendered server-side with inline JS and no framework. It binds to localhost only, and its single POST endpoint (/rehost) checks a random per-run token embedded in the page.

submit:

  1. Pick the content rating (§4.7).
  2. GET the snapshot (version plus current images) and rebuild the seed on top of it.
  3. POST …/preview with the source URLs and print the diff.
  4. Ask "Upload N images to and submit for review? [y/N]". --yes skips this.
  5. Rehost (§4.6), swap in the hosted URLs, and POST with save_mode: "review".
  6. Handle the outcome:
    • 409: fetch the snapshot again, rebuild, preview again and ask again, at most once. The upload cache prevents re-uploading.
    • 403: explain the 25-approved-submissions requirement and fall back to serve.

5. Cross-cutting concerns

  • HTTP. All network access goes through http.ts. It:
    • sends a browser User-Agent;
    • limits each host to 1 request per second, and Wayback to 1 every 1.5 s;
    • times out after 20 s;
    • retries twice with backoff on 429/5xx;
    • never sends credentials to any host other than the one they belong to.
  • Logging. Progress goes to stderr and results to stdout, so print --json can be piped. --verbose logs every request.
  • Exit codes.
    • 0: success, including "nothing new to add"
    • 1: unexpected error
    • 2: bad usage, an unsupported URL, or an ambiguous or missing series match
    • 3: the user declined to submit.
  • Config. All configuration comes from environment variables:
    • MANGABAKA_TOKEN
    • MANGABAKA_API (default https://api.mangabaka.org, for testing)
    • MBSEED_IMAGE_HOST, the default for --image-host
    • CATBOX_USERHASH
    • the MBSEED_S3_* variables (§4.6).
  • Secrets. The MangaBaka token is sent only to the MangaBaka API, and the S3 credentials only to the S3 endpoint.

6. Testing

  • Unit tests with vitest, run against fixtures saved from real responses:
    • Webtoons pages (official, canvas, an archived 2015 snapshot) and CDX output
    • MangaBaka match, links, images, snapshot and preview JSON.
  • Pure functions under test:
    • parsing, URL normalisation and canonicalisation
    • classification, dedupe and default selection
    • seed building, content-rating resolution and the language mapping
    • the upload cache.
  • I/O modules (pipeline, mangabaka/client, actions/submit, history/wayback, hosting/catbox, hosting/s3) are tested with an injected fake fetch.
  • No network in tests. A manual smoke test is documented in the README: mbseed --action print https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95.
Reference in New Issue View Git Blame Copy Permalink