Merge branch 'feat/kakaopage': KakaoPage source (spec 004), per-host request headers
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
@@ -9,7 +9,7 @@ MusicBrainz.
|
||||
> [the spec](docs/specs/001-webtoons-covers.md).
|
||||
|
||||
Today's goal is one thing done well: **given a MangaBaka series link and a
|
||||
WEBTOON or Tapas series link, find every cover that series has used, current
|
||||
WEBTOON, Tapas or KakaoPage series link, find every cover that series has used, current
|
||||
and historical, and request them added to the MangaBaka series**. The current
|
||||
poster becomes the main (default) cover for the edition's language.
|
||||
|
||||
@@ -82,11 +82,13 @@ least 25 approved submissions.
|
||||
| Source | Links | Current covers | Historical covers |
|
||||
|---|---|---|---|
|
||||
| WEBTOON (official & canvas) | `https://www.webtoons.com/<lang>/…/list?title_no=N`, `m.webtoons.com` | poster, square, banner, foreground | Wayback Machine |
|
||||
| KakaoPage (웹툰 & 웹소설) | `https://page.kakao.com/content/N` (or the old `…/home?seriesId=N`) | portrait cover (full size), landscape thumbnail | Wayback Machine |
|
||||
| Tapas (comics & novels) | `https://tapas.io/series/<name>` (or `…/info`, the series number, or an `/episode/N` link) | portrait cover (full size), banner | Wayback Machine |
|
||||
|
||||
Tapas is English only. Mature Tapas series are shown only to signed-in adults:
|
||||
the page can't read their current covers (it says so and still searches the
|
||||
archive), but the userscript can, when you are signed in to Tapas in that browser.
|
||||
Tapas is English only; KakaoPage is Korean. Mature Tapas series are shown only
|
||||
to signed-in adults: their current covers can't be read (the tool says so and
|
||||
still searches the archive). KakaoPage 15+ and 19+ titles are read normally and
|
||||
show the content-rating warning.
|
||||
|
||||
Each Webtoons language edition has its own `title_no`. Run the tool once per
|
||||
edition; the covers are tagged with that edition's language.
|
||||
|
||||
@@ -15,7 +15,7 @@ self-hosted Bun server (primary), a userscript, and later a Bun CLI.
|
||||
| `src/core/session.ts` | Review state shared by the page and the userscript: cards, main cover, rehosting (cards remember host, cache key and expiry), manual text, the download bundle, `restore` from history, token flow (preview → confirm → save for review). |
|
||||
| `src/core/zip.ts`, `download.ts` | Store-only ZIP writer (CRC-32, UTF-8 names); the download bundle: file names `NN_role_lang[_YYYY]_WxH.ext` and `covers.txt` (spec 002 §2). |
|
||||
| `src/core/saved.ts` | History model (spec 002 §3): `SavedSearch` (plain data, source by id), `toSaved`/`fromSaved` (drops expired links), `carryEdits` (search again), `pruneIds` (30 entries, 300 MB), `HistoryStore` interface and an in-memory one. |
|
||||
| `src/core/sources/` | `webtoons/` and `tapas/` (spec 003): each `urls.ts` (links, image URL normalisation, same-series), `parse.ts` (pure HTML parsing, also used on Wayback snapshots), `index.ts` (the `Source`); `registry.ts` lists them and builds the proxy and upload allowlists. |
|
||||
| `src/core/sources/` | `webtoons/`, `tapas/` (spec 003) and `kakaopage/` (spec 004; reads a JSON endpoint, not the page): each `urls.ts` (links, image URL normalisation, same-series), `parse.ts` (pure HTML parsing, also used on Wayback snapshots), `index.ts` (the `Source`); `registry.ts` lists them and builds the proxy and upload allowlists. |
|
||||
| `src/web/` | Built. The page ("Obi Shelf" design, see `DESIGN.md`): `template.ts` markup shared with the userscript panel, `app.ts` wiring (`mountApp`), `render.ts` shelf and timeline views, `view-model.ts` pure helpers, `transport.ts` (direct to MangaBaka/litterbox, everything else via `/api/fetch`), `styles.css` scoped for a shadow root, `save-file.ts` (Blob download, works in the panel's shadow root), `history-db.ts` (IndexedDB `HistoryStore`; null when storage is refused), `history-keeper.ts` (when the series on screen is saved: after a search, debounced edits, marks; flushes before a new search, an open and page hide; removal/clear stop tracking), `recent.ts` (History list and the start screen's recent shelf; the page only), `dark-reader.ts` (switches to the page's own dark theme and locks Dark Reader out when that extension is darkening the page). |
|
||||
| `src/server/` | Built. `app.ts`: web-standard handler (`createApp(env)`) — static assets, `/api/config`, `/api/fetch` allowlisted proxy (redirects re-checked, 20 MB cap), `/api/upload` (store or catbox; takes a source URL, never bytes), `/covers/*`. `storage.ts` (interface, in-memory) and `storage-dir.ts` (folder, size cap, 30-day sweep). `config.ts` (env, client IP), `rate-limit.ts`, `assets-path.ts`. `bun.ts`: thin `Bun.serve` entry. Gate and Workers entry deferred. |
|
||||
| `src/userscript/` | Built. `gm.ts`: `GM_xmlhttpRequest` transport (no proxy; honest User-Agent for Wayback) and GM storage. `main.ts`: a "Seed covers" launcher on MangaBaka series pages (follows client-side navigation) opening the page's app in a shadow-root panel; optional server URL in its settings. Built by `scripts/build-userscript.ts` into `dist/web/mbseed.user.js`, served at `/mbseed.user.js`. |
|
||||
@@ -61,6 +61,7 @@ requests) and smoke-tests it. Example deployment: `docker-compose.example.yml`.
|
||||
| Webtoons HTML / `swebtoon-phinf.pstatic.net` | server proxy or `GM_xmlhttpRequest` | Nothing works. |
|
||||
| Wayback (`web.archive.org`) | same | Warning; current covers only. |
|
||||
| Server storage (folder on our server) | server | `server` host hidden; litterbox/catbox/direct still work. |
|
||||
| KakaoPage (`bff-page.kakao.com`, images on `page-images.kakaoentcdn.com`) | server proxy or `GM_xmlhttpRequest`, with a `page.kakao.com` Referer (`requestHeadersFor`) | That source fails. |
|
||||
| Shynet (`analytics.lucaswinther.info`) | `<script>` on the page only | No page-view counts; nothing else changes. |
|
||||
| litterbox / catbox | browser directly (litterbox), server or `GM_xmlhttpRequest` (catbox) | That host fails; others still work. |
|
||||
|
||||
|
||||
@@ -16,6 +16,28 @@ the repo every session — measurably slower and more tokens).
|
||||
**Consequences:** Rule changes happen in one file; pointer files must stay thin. Plans are working
|
||||
artefacts: kept after merge they go stale and contradict the code, so the spec is the record.
|
||||
|
||||
## 2026-10-15 — KakaoPage source; per-host request headers (spec 004)
|
||||
**Status:** accepted
|
||||
**Context:** The user asked for KakaoPage next. Its pages render in the browser; the data comes from
|
||||
`bff-page.kakao.com/api/gateway/api/v1/content/overview`, which answers 403 without a `page.kakao.com` Referer.
|
||||
**Decision:** A third `Source` that reads that endpoint for current covers (`thumbnail` portrait, `land_thumbnail`
|
||||
landscape; originals via `filename=o1`) and archived server-rendered pages (both the old `/home?seriesId=` and
|
||||
`/content/` addresses) for history. `core/transport.ts` gains `requestHeadersFor(url)` (User-Agent as before, plus
|
||||
a Referer for `bff-page.kakao.com`); the server proxy and the userscript send it. `age_grade >= 15` sets `mature`
|
||||
(the rating warning). Kakao Webtoon (`webtoon.kakao.com`) is a separate site and not included.
|
||||
**Alternatives:** Rendering the page in a headless browser (heavy, server-only); reading the JS bundle's API
|
||||
calls at run time (brittle).
|
||||
**Consequences:** We depend on an internal API and its Referer check. A Referer is a browser-like header, as the
|
||||
Chrome User-Agent is for Webtoons.
|
||||
|
||||
## 2026-10-15 — Correction: the userscript does not send Tapas cookies
|
||||
**Status:** accepted (corrects "Tapas source (spec 003)" below)
|
||||
**Context:** The Tapas entry and spec 003 said a signed-in adult gets mature Tapas pages through the userscript.
|
||||
`gmTransport` uses `anonymous: true`, so no cookies are sent.
|
||||
**Decision:** The docs and the warning now say only the Wayback history helps for age-checked Tapas series.
|
||||
Sending cookies would need `anonymous: false` for tapas.io only; deferred (TODO).
|
||||
**Consequences:** None in code; the earlier claim was never true.
|
||||
|
||||
## 2026-10-15 — Page views via our Shynet instance
|
||||
**Status:** accepted
|
||||
**Context:** The user asked for analytics from their self-hosted Shynet.
|
||||
|
||||
+7
-1
@@ -49,7 +49,13 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md).
|
||||
find the series (check `ageGateTarget` there); a mature series pasted by slug keeps the slug as its id, so a
|
||||
MangaBaka link by number warns; an all-digit slug is read as an id; accept a quoted `pageMeta` id; archived
|
||||
history covers fall back to the 400×600 `_z` copy when the original is gone.
|
||||
- [ ] Verify with a signed-in, age-verified Tapas account (userscript): the mature info page parses.
|
||||
- [ ] Mature Tapas series in the userscript: send the user's tapas.io cookies (`anonymous: false` for tapas.io only), then check that a signed-in adult's info page parses.
|
||||
- [ ] Small KakaoPage fixes from the spec 004 review (deferred as minor): 2020 archive fallbacks point at the new-host
|
||||
`o1` URL instead of the `dn-img-page…th2` one the page used (only matters once a key is deleted); accept `-`/`_`
|
||||
in image keys; use a `Map` for the Referer table; zero-padded ids fail; translate `api_content_not_found`
|
||||
("KakaoPage has no series N") instead of showing the Korean message.
|
||||
- [ ] Check in Tampermonkey/Violentmonkey that the userscript's Referer reaches `bff-page.kakao.com` (only checked
|
||||
through the server proxy in Chrome).
|
||||
- [ ] Possible race (unconfirmed): pressing **Find covers** before the page has mounted (it awaits `/api/config`
|
||||
and IndexedDB first) may submit the form natively and reload. Seen once in a scripted run; not reproduced.
|
||||
|
||||
|
||||
@@ -27,8 +27,9 @@ Checked against *Villains Are Destined to Die* (comic, id 186414) and its novel
|
||||
- Lists use `_m` (180 px) of other images; `pc/…` images are signed promotions of other series. Neither is used.
|
||||
- Older pages use `d30womf5coomej.cloudfront.net/sa/…`; the same paths answer on `us-a.tapas.io`.
|
||||
- **Mature series:** without a signed-in, age-verified account, `/series/…/info` redirects to
|
||||
`/account/verify-age?from=https://tapas.io/series/<slug>/info`, a page with no series data. The userscript's
|
||||
`GM_xmlhttpRequest` sends the user's own tapas.io cookies, so a signed-in adult gets the real page there.
|
||||
`/account/verify-age?from=https://tapas.io/series/<slug>/info`, a page with no series data.
|
||||
*Corrected 2026-10-15 (spec 004 work):* the userscript's `GM_xmlhttpRequest` runs with `anonymous: true`, so it
|
||||
does **not** send the user's tapas.io cookies; the userscript sees the age check too.
|
||||
- **Language:** English only (MangaBaka `en`).
|
||||
- **Wayback:** `tapas.io/series/<slug>` captured about monthly since 2020; the prefix also matches
|
||||
`<slug>-novel` and other longer slugs.
|
||||
@@ -43,8 +44,8 @@ Checked against *Villains Are Destined to Die* (comic, id 186414) and its novel
|
||||
- **Current covers:** portrait (role `poster`, suffix removed) and banner (role `banner`). The poster is the
|
||||
suggested main cover as for Webtoons.
|
||||
- **Mature (age check):** the series is resolved from the `from=` link (slug; title from the slug), `mature: true`,
|
||||
no current covers, and a warning: Tapas shows this series only to signed-in adults; use the userscript while
|
||||
signed in, or tick the Wayback history. History still runs.
|
||||
no current covers, and a warning that Tapas shows it only to signed-in adults; tick the Wayback history.
|
||||
History still runs.
|
||||
- **Image URLs:** `normalizeImageUrl` maps both image hosts to `us-a.tapas.io`, drops `_<letters>` size suffixes
|
||||
after the UUID and any query.
|
||||
- **Same series:** a link is the same series when it is a tapas.io `/series/<slug>[/info][/]` with the same slug
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
status: approved
|
||||
---
|
||||
|
||||
# 004 — KakaoPage source
|
||||
|
||||
A third source (spec 001 §4.1): covers of a [KakaoPage](https://page.kakao.com) series, comics (웹툰) and
|
||||
novels (웹소설). Design approved by the user in chat on 2026-10-15. Kakao Webtoon (`webtoon.kakao.com`) is a
|
||||
different site and out of scope.
|
||||
|
||||
## 1. Verified facts (2026-10-15)
|
||||
|
||||
Checked against *악역의 엔딩은 죽음뿐* (Villains Are Destined to Die; comic 54845189, novel 52654302) and
|
||||
*울프 인더 하우스 [19세 완전판]* (63659962, 19+). Fixtures in `test/fixtures/kakaopage/` and
|
||||
`test/fixtures/wayback/kakaopage-*`.
|
||||
|
||||
- **Pages are rendered in the browser.** `page.kakao.com/content/<id>` is an 11 KB shell with a generic
|
||||
`og:image`; the data comes from
|
||||
`GET https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=<id>`.
|
||||
- **That endpoint needs a `page.kakao.com` Referer (or Origin)**; without one it answers 403. It answers 19+
|
||||
series without a login.
|
||||
- **Its `result.content`:** `series_id`, `title`, `category` (`웹툰` / `웹소설`), `age_grade` (0, 12, 15, 19),
|
||||
`lang` (`ko`), `thumbnail` (portrait key, e.g. 720×1030), `land_thumbnail` (landscape key, 540×370; may be null).
|
||||
An unknown id answers with an error (`상품 정보를 찾을 수 없습니다`, "product not found").
|
||||
- **Images:** `https://page-images.kakaoentcdn.com/download/resource?kid=<key>&filename=o1` is the original
|
||||
(`th2`, `th3` and `/dims/resize/…` are smaller). No Referer needed. Older pages use `dn-img-page.kakao.com` with
|
||||
the same query; old keys answer on the new host.
|
||||
- **Wayback:** `page.kakao.com/home?seriesId=<id>` (the old address, 2020–2022) and `page.kakao.com/content/<id>`
|
||||
(2022 on) are captured server-rendered. Every capture's `og:image` is this series' image (landscape in 2020–22,
|
||||
portrait in 2024). 2020 captures also hold `props.initialState.series.series.thumbnailUrl` and
|
||||
`landThumbnailUrl` in `__NEXT_DATA__`. Other images on the page (`sameSeriesList`, comments) are other series
|
||||
or users.
|
||||
|
||||
## 2. Behaviour
|
||||
|
||||
- **Links:** `page.kakao.com/content/<id>` (with or without a trailing path/query) and
|
||||
`page.kakao.com/home?seriesId=<id>`; `m.` hosts too.
|
||||
- **Series:** from the overview endpoint: `externalId` = id, `title`, `language` = `lang` (`ko`),
|
||||
`canonicalUrl` = `https://page.kakao.com/content/<id>`, `mature` = `age_grade >= 15` (the rating warning),
|
||||
`flags.novel`, `archiveUrls` = [`page.kakao.com/content/<id>`, `page.kakao.com/home?seriesId=<id>`].
|
||||
- **Current covers:** `thumbnail` → role `poster` (the suggested main cover), `land_thumbnail` → role `banner`.
|
||||
- **History:** from a capture: `og:image` (role `other`: portrait or landscape, the shape decides the type) and,
|
||||
when present, the series' own `thumbnailUrl` (`poster`) and `landThumbnailUrl` (`banner`).
|
||||
- **Same series:** a link is the same series when its id (`/content/<id>` or `seriesId=<id>`) equals ours exactly,
|
||||
so `seriesId=548451890` is another series.
|
||||
- **Image URLs:** both hosts → `page-images.kakaoentcdn.com`, `filename=o1`, the `kid` kept, any `/dims/…`
|
||||
dropped.
|
||||
- **Request headers per host:** `requestHeadersFor(url)` in `core/transport.ts` (User-Agent as before, plus
|
||||
`Referer: https://page.kakao.com/` for `bff-page.kakao.com`). The server proxy and the userscript send them.
|
||||
- **Hosts:** fetch `page.kakao.com`, `m.page.kakao.com`, `bff-page.kakao.com`, `page-images.kakaoentcdn.com`,
|
||||
`dn-img-page.kakao.com`; images the last two. The userscript `@connect`s `kakao.com` and `kakaoentcdn.com`.
|
||||
- **Chrome:** the link hint names "WEBTOON, Tapas or KakaoPage".
|
||||
|
||||
## 3. Testing
|
||||
|
||||
Offline unit tests on the captures: link parsing, URL normalisation, same-series, overview parsing (comic,
|
||||
novel, 19+, unknown id), archived pages (2020, 2022, 2024), Wayback rows through both prefixes, request headers.
|
||||
Then a real run in headless Chrome (current covers, then yearly history).
|
||||
|
||||
## 4. Risks
|
||||
|
||||
- The overview endpoint is an internal API: its shape or its Referer check can change.
|
||||
- Sending a `page.kakao.com` Referer is what KakaoPage's own page sends; like the browser User-Agent for
|
||||
Webtoons, it is a browser-like header, not an identity.
|
||||
@@ -1,13 +1,13 @@
|
||||
/** Tampermonkey/Violentmonkey metadata. @connect lists every host GM_xmlhttpRequest may reach. */
|
||||
export function userscriptBanner(version: string, serverUrl?: string): string {
|
||||
const connects = ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "web.archive.org", "catbox.moe"];
|
||||
const connects = ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "kakao.com", "kakaoentcdn.com", "web.archive.org", "catbox.moe"];
|
||||
if (serverUrl) connects.push(new URL(serverUrl).hostname);
|
||||
const lines = [
|
||||
"// ==UserScript==",
|
||||
"// @name MangaBaka Seeder",
|
||||
"// @namespace https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder",
|
||||
`// @version ${version}`,
|
||||
"// @description Find every cover of a WEBTOON or Tapas series and turn them into a MangaBaka cover edit request",
|
||||
"// @description Find every cover of a WEBTOON, Tapas or KakaoPage series and turn them into a MangaBaka cover edit request",
|
||||
"// @match https://mangabaka.org/*",
|
||||
"// @grant GM_xmlhttpRequest",
|
||||
"// @grant GM_getValue",
|
||||
|
||||
@@ -72,7 +72,7 @@ export async function runCovers(
|
||||
warn(`MangaBaka series ${seriesId} has no link to this ${source.name} series (${series.canonicalUrl}). Check that the two links belong together.`);
|
||||
}
|
||||
if (series.flags.ageGated) {
|
||||
warn(`${source.name} shows this series only to signed-in adults, so its current covers couldn't be read. Use the userscript while signed in there, or tick "Old covers from the Wayback Machine".`);
|
||||
warn(`${source.name} shows this series only to signed-in adults, so its current covers couldn't be read. Tick "Old covers from the Wayback Machine" for archived ones.`);
|
||||
}
|
||||
if (series.language === "unknown") warn(`Couldn't tell the language of ${series.canonicalUrl}; covers will be marked "unknown".`);
|
||||
const current = await source.currentCovers(series, ctx);
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import type { CoverCandidate, SourceSeries } from "../../model";
|
||||
import { SourceError, type Source, type SourceContext } from "../source";
|
||||
import { extractCovers, parseOverview, type Overview } from "./parse";
|
||||
import {
|
||||
API_HOST, IMAGE_HOSTS, PAGE_HOSTS, contentUrl, isKakaoPageUrl, isSameKakaoSeries, normalizeImageUrl, overviewUrl,
|
||||
parseKakaoPageUrl,
|
||||
} from "./urls";
|
||||
|
||||
/** Covers read by resolve(), so currentCovers() doesn't ask twice. */
|
||||
const currentBySeries = new WeakMap<SourceSeries, CoverCandidate[]>();
|
||||
|
||||
/** The page is rendered in the browser; its data endpoint has what we need (spec 004 §1). */
|
||||
async function fetchOverview(id: string, ctx: SourceContext): Promise<Overview> {
|
||||
const res = await ctx.transport.fetch(overviewUrl(id), { signal: ctx.signal });
|
||||
const body: unknown = await res.json().catch(() => null);
|
||||
if (!res.ok && !body) throw new SourceError(`KakaoPage answered HTTP ${res.status} for series ${id}.`);
|
||||
const overview = parseOverview(body);
|
||||
if (overview.id !== id) throw new SourceError(`KakaoPage sent series ${overview.id} when asked for ${id}.`);
|
||||
return overview;
|
||||
}
|
||||
|
||||
export const kakaopage: Source = {
|
||||
id: "kakaopage",
|
||||
name: "KakaoPage",
|
||||
hosts: [...PAGE_HOSTS, API_HOST, ...IMAGE_HOSTS],
|
||||
imageHosts: [...IMAGE_HOSTS],
|
||||
|
||||
matches: (url) => isKakaoPageUrl(url),
|
||||
|
||||
async resolve(url, ctx) {
|
||||
const id = parseKakaoPageUrl(url);
|
||||
const o = await fetchOverview(id, ctx);
|
||||
const series: SourceSeries = {
|
||||
source: "kakaopage",
|
||||
externalId: id,
|
||||
title: o.title,
|
||||
language: o.language,
|
||||
canonicalUrl: contentUrl(id),
|
||||
archiveUrls: [`page.kakao.com/content/${id}`, `page.kakao.com/home?seriesId=${id}`],
|
||||
// 15+ and 19+ titles: suggest a rating above "safe" (spec 004 §2).
|
||||
mature: o.ageGrade >= 15,
|
||||
flags: { novel: o.novel },
|
||||
};
|
||||
currentBySeries.set(series, o.covers.map((c) => ({ ...c, current: true })));
|
||||
return series;
|
||||
},
|
||||
|
||||
async currentCovers(series, ctx) {
|
||||
const cached = currentBySeries.get(series);
|
||||
if (cached) return cached;
|
||||
return (await fetchOverview(series.externalId, ctx)).covers.map((c) => ({ ...c, current: true }));
|
||||
},
|
||||
|
||||
extractCovers: (html, pageUrl) => extractCovers(html, pageUrl),
|
||||
normalizeImageUrl: (url) => normalizeImageUrl(url),
|
||||
isSameSeries: (linkUrl, series) => isSameKakaoSeries(linkUrl, series.externalId),
|
||||
};
|
||||
@@ -0,0 +1,89 @@
|
||||
import { metaContent } from "../../html";
|
||||
import type { CoverCandidate, CoverRole } from "../../model";
|
||||
import { SourceError } from "../source";
|
||||
import { normalizeImageUrl } from "./urls";
|
||||
|
||||
export interface Overview {
|
||||
id: string;
|
||||
title: string;
|
||||
language: string;
|
||||
ageGrade: number;
|
||||
novel: boolean;
|
||||
covers: CoverCandidate[];
|
||||
}
|
||||
|
||||
interface OverviewContent {
|
||||
series_id?: number;
|
||||
title?: string;
|
||||
lang?: string;
|
||||
age_grade?: number;
|
||||
category?: string;
|
||||
thumbnail?: string | null;
|
||||
land_thumbnail?: string | null;
|
||||
}
|
||||
|
||||
/** `GET …/content/overview?series_id=` (spec 004 §1). */
|
||||
export function parseOverview(body: unknown): Overview {
|
||||
const b = body as { result?: { content?: OverviewContent } | null; message?: string } | null;
|
||||
const c = typeof b === "object" && b !== null ? b.result?.content : undefined;
|
||||
if (!c?.series_id) {
|
||||
const why = typeof b === "object" && b !== null && typeof b.message === "string" ? `: ${b.message}` : "";
|
||||
throw new SourceError(`KakaoPage didn't return this series${why}`);
|
||||
}
|
||||
return {
|
||||
id: String(c.series_id),
|
||||
title: c.title?.trim() || `series ${c.series_id}`,
|
||||
language: c.lang || "ko",
|
||||
ageGrade: c.age_grade ?? 0,
|
||||
novel: c.category === "웹소설",
|
||||
covers: candidates([["poster", c.thumbnail], ["banner", c.land_thumbnail]]),
|
||||
};
|
||||
}
|
||||
|
||||
/** A Wayback capture: the series' own thumbnails (2020 pages) and the og:image (spec 004 §2). */
|
||||
export function extractCovers(html: string, pageUrl: URL): CoverCandidate[] {
|
||||
const series = nextData(html)?.props?.initialState?.series?.series;
|
||||
const og = metaContent(html, "og:image");
|
||||
return candidates(
|
||||
[["poster", series?.thumbnailUrl], ["banner", series?.landThumbnailUrl], ["other", og && !/\/shared\/ogImage\./.test(og) ? og : null]],
|
||||
pageUrl,
|
||||
);
|
||||
}
|
||||
|
||||
interface NextData {
|
||||
props?: { initialState?: { series?: { series?: { thumbnailUrl?: string; landThumbnailUrl?: string } } } };
|
||||
}
|
||||
|
||||
function nextData(html: string): NextData | null {
|
||||
const m = /<script id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/.exec(html);
|
||||
if (!m) return null;
|
||||
try {
|
||||
return JSON.parse(m[1]!) as NextData;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** The URL as the page wrote it (for the Wayback copy of the image); a bare key has none. */
|
||||
function absolute(raw: string): string | null {
|
||||
const s = raw.trim();
|
||||
if (!/^(https?:)?\/\//.test(s)) return null;
|
||||
try {
|
||||
return new URL(s.startsWith("//") ? `https:${s}` : s).toString();
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function candidates(found: [CoverRole, string | null | undefined][], base?: URL): CoverCandidate[] {
|
||||
const seen = new Set<string>();
|
||||
const out: CoverCandidate[] = [];
|
||||
for (const [role, raw] of found) {
|
||||
if (!raw?.trim()) continue;
|
||||
const url = normalizeImageUrl(raw, base);
|
||||
if (!url.startsWith("https://") || seen.has(url)) continue;
|
||||
seen.add(url);
|
||||
out.push({ url, role, current: false, originalUrl: absolute(raw) ?? url });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { LinkError } from "../../links";
|
||||
|
||||
export const PAGE_HOSTS: ReadonlySet<string> = new Set(["page.kakao.com", "m.page.kakao.com"]);
|
||||
export const API_HOST = "bff-page.kakao.com";
|
||||
export const IMAGE_HOST = "page-images.kakaoentcdn.com";
|
||||
/** Older pages name dn-img-page.kakao.com; the same keys answer on IMAGE_HOST (spec 004 §1). */
|
||||
export const IMAGE_HOSTS: ReadonlySet<string> = new Set([IMAGE_HOST, "dn-img-page.kakao.com"]);
|
||||
|
||||
const CONTENT_PATH = /^\/content\/(\d+)(?:\/|$)/;
|
||||
/** An image key as the API and old pages give it: "mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0". */
|
||||
const BARE_KEY = /^[A-Za-z0-9]+\/[A-Za-z0-9]+\/[A-Za-z0-9]+$/;
|
||||
|
||||
function idOf(url: URL): string | null {
|
||||
if (!PAGE_HOSTS.has(url.hostname)) return null;
|
||||
const content = CONTENT_PATH.exec(url.pathname)?.[1];
|
||||
if (content) return content;
|
||||
const home = url.pathname.replace(/\/+$/, "") === "/home" ? url.searchParams.get("seriesId") : null;
|
||||
return home && /^\d+$/.test(home) ? home : null;
|
||||
}
|
||||
|
||||
export const isKakaoPageUrl = (url: URL): boolean => idOf(url) !== null;
|
||||
|
||||
export function parseKakaoPageUrl(url: URL): string {
|
||||
const id = idOf(url);
|
||||
if (id) return id;
|
||||
if (!PAGE_HOSTS.has(url.hostname)) throw new LinkError(`Not a KakaoPage link: ${url.hostname}`);
|
||||
throw new LinkError("That KakaoPage link isn't a series. Use the series page (page.kakao.com/content/…).");
|
||||
}
|
||||
|
||||
export const overviewUrl = (id: string): string => `https://${API_HOST}/api/gateway/api/v1/content/overview?series_id=${id}`;
|
||||
export const contentUrl = (id: string): string => `https://page.kakao.com/content/${id}`;
|
||||
const originalFor = (kid: string) => `https://${IMAGE_HOST}/download/resource?kid=${kid}&filename=o1`;
|
||||
|
||||
/** The original image for a key or any of its URLs; other URLs pass through. */
|
||||
export function normalizeImageUrl(raw: string, base?: string | URL): string {
|
||||
const s = raw.trim();
|
||||
if (BARE_KEY.test(s)) return originalFor(s);
|
||||
let u: URL;
|
||||
try {
|
||||
u = new URL(s.startsWith("//") ? `https:${s}` : s, base);
|
||||
} catch {
|
||||
return s;
|
||||
}
|
||||
const kid = u.searchParams.get("kid");
|
||||
return IMAGE_HOSTS.has(u.hostname) && u.pathname === "/download/resource" && kid && BARE_KEY.test(kid) ? originalFor(kid) : u.toString();
|
||||
}
|
||||
|
||||
export function isSameKakaoSeries(linkUrl: string, id: string): boolean {
|
||||
try {
|
||||
return idOf(new URL(linkUrl)) === id;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -1,9 +1,10 @@
|
||||
import { WAYBACK_HOST } from "../transport";
|
||||
import type { Source } from "./source";
|
||||
import { kakaopage } from "./kakaopage";
|
||||
import { tapas } from "./tapas";
|
||||
import { webtoons } from "./webtoons";
|
||||
|
||||
export const SOURCES: readonly Source[] = [webtoons, tapas];
|
||||
export const SOURCES: readonly Source[] = [webtoons, tapas, kakaopage];
|
||||
|
||||
export function findSource(url: URL, sources: readonly Source[] = SOURCES): Source | undefined {
|
||||
return sources.find((s) => s.matches(url));
|
||||
|
||||
@@ -30,6 +30,15 @@ export function userAgentFor(url: string): string {
|
||||
return hostOf(url) === WAYBACK_HOST ? TOOL_USER_AGENT : BROWSER_USER_AGENT;
|
||||
}
|
||||
|
||||
/** Hosts that answer only with their own page as Referer, as a browser sends it (spec 004 §1). */
|
||||
const REFERERS: Readonly<Record<string, string>> = { "bff-page.kakao.com": "https://page.kakao.com/" };
|
||||
|
||||
/** Every request to a source or Wayback carries these: the User-Agent, and a Referer where a host needs one. */
|
||||
export function requestHeadersFor(url: string): Record<string, string> {
|
||||
const referer = REFERERS[hostOf(url)];
|
||||
return { "user-agent": userAgentFor(url), ...(referer ? { referer } : {}) };
|
||||
}
|
||||
|
||||
export function hostOf(url: string): string {
|
||||
try {
|
||||
return new URL(url).hostname;
|
||||
|
||||
+2
-2
@@ -2,7 +2,7 @@
|
||||
import { uploadToCatbox } from "../core/hosting/catbox";
|
||||
import { ImageError, MAX_IMAGE_BYTES, probeImage, readCapped, sha256Hex } from "../core/images";
|
||||
import { allowedFetchHosts, allowedImageHosts } from "../core/sources/registry";
|
||||
import { API_CALL_HEADER, TOOL_USER_AGENT, WAYBACK_HOST, userAgentFor, type Transport } from "../core/transport";
|
||||
import { API_CALL_HEADER, TOOL_USER_AGENT, WAYBACK_HOST, requestHeadersFor, type Transport } from "../core/transport";
|
||||
import { resolveAssetPath } from "./assets-path";
|
||||
import { RateLimiter } from "./rate-limit";
|
||||
import { CONTENT_TYPES, COVER_NAME, StorageFullError, type Storage } from "./storage";
|
||||
@@ -86,7 +86,7 @@ export function createApp(env: ServerEnv): (req: Request) => Promise<Response> {
|
||||
try {
|
||||
res = await upstream(url.toString(), {
|
||||
redirect: "manual",
|
||||
headers: { "user-agent": userAgentFor(url.toString()), accept: "*/*" },
|
||||
headers: { ...requestHeadersFor(url.toString()), accept: "*/*" },
|
||||
signal: AbortSignal.timeout(UPSTREAM_TIMEOUT_MS),
|
||||
});
|
||||
} catch (err) {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { KeyValueStore } from "../core/session";
|
||||
import { TransportError, userAgentFor, type Transport } from "../core/transport";
|
||||
import { TransportError, requestHeadersFor, type Transport } from "../core/transport";
|
||||
|
||||
export interface GmResponse {
|
||||
status: number;
|
||||
@@ -49,7 +49,7 @@ export function gmTransport(gm: GmXhr): Transport {
|
||||
const request = gm({
|
||||
method: (init.method ?? "GET").toUpperCase(),
|
||||
url,
|
||||
headers: { "user-agent": userAgentFor(url), ...init.headers },
|
||||
headers: { ...requestHeadersFor(url), ...init.headers },
|
||||
data: init.body ?? undefined,
|
||||
responseType: "arraybuffer",
|
||||
anonymous: true,
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ export const APP_HTML = /* html */ `
|
||||
<label class="mbseed-slip">
|
||||
<span class="mbseed-slip-label">Source series link</span>
|
||||
<input id="source-link" name="source" required autocomplete="off" spellcheck="false" placeholder="https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95">
|
||||
<span class="mbseed-slip-hint">The series page on the manga, manhwa or webcomic site. WEBTOON or Tapas for now.</span>
|
||||
<span class="mbseed-slip-hint">The series page on the manga, manhwa or webcomic site. WEBTOON, Tapas or KakaoPage for now.</span>
|
||||
</label>
|
||||
<div class="mbseed-settings">
|
||||
<label class="mbseed-check"><input id="history-on" type="checkbox"> Old covers from the Wayback Machine</label>
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"result":{"page_continue":{"first_read_product_id":63709590,"first_read_product_title":"울프 인 더 하우스 1화","first_read_is_free":true,"has_new_single":true,"has_unread_single":true},"display_ad":{"section_uid":"A15","banner_uid":"A15","trevi_uid":"","moment_uid":"DAN-9PaWqQNQpqtf5FFM"},"content":{"service_property":{"view_count":364548,"rating_count":3568,"bookmark_count":0,"highlighter_count":0,"rating_sum":35634,"comment_count":420,"preview":{"item":{"service_property":{},"operator_property":{},"asset_property":{},"uid":10525027,"product_id":63709590,"title":"울프 인 더 하우스 1화","thumbnail":"fg3Uw/hAdNQaSK14/b3yEf3a7xB6yICVVjkd5Hk","is_free":true,"state":"ST61","slide_type":"SD03","last_release_dt":"2024-03-18T18:01:47+09:00","size":3.0148,"page_count":33,"hidden":false,"free_change_dt":"1980-01-01T00:00:00+09:00","start_sale_dt":"2024-03-25T22:00:05+09:00","order_value":1,"version":"1.0"},"next_item":{"service_property":{},"operator_property":{},"asset_property":{},"uid":10525028,"product_id":63709592,"title":"울프 인 더 하우스 2화","thumbnail":"yffS8/hAd47i5f5Z/JwevO0PEbZFUa7hX7sRGEK","is_free":false,"state":"ST61","slide_type":"SD03","last_release_dt":"2024-03-18T18:01:47+09:00","size":2.9127,"page_count":33,"hidden":false,"free_change_dt":"1980-01-01T00:00:00+09:00","start_sale_dt":"2024-03-25T22:00:05+09:00","order_value":2,"version":"1.0"},"using_scroll":true}},"operator_property":{"is_manga":false},"asset_property":{},"series_id":63659962,"title":"울프 인더 하우스 [19세 완전판] ","thumbnail":"bn2lQs/hAd4XgsZ5Z/MtPVaiw3jf5BioxFC2tA91","category_uid":10,"category":"웹툰","sub_category_uid":119,"sub_category":"BL","is_all_free":false,"is_waitfree":true,"age_grade":19,"state":"ST61","on_issue":"N","series_type":"P","business_model":"T","authors":"박지연","description":"마지못해 사귀어왔던 애인과의 뻔한 이별. 후련한 마음으로 삶의 전환점을 찾던 민석은 대형견 백산이를 임시로 보호하게 된다. \n구질구질 구남친 대신 귀여운 멍멍이와 함께 자유를 만끽할 줄 알았는데... 생각보다 많이 엉뚱한 존재가 삶에 들어와버렸다!","free_slide_count":1,"last_slide_added_dt":"2024-03-25T22:00:05+09:00","start_sale_dt":"2024-03-25T22:00:05+09:00","waitfree_block_count":10,"waitfree_period_by_minute":1440,"sale_method":"TICKET","publisher_uid":33761,"on_sale_count":79,"talk_viewer_type":"TAP_SCROLL","first_product_id":63709590,"per_price_type":"S","translate_property":{"sub_category":{"th":"วาย","ko":"BL","en":"BL"},"category":{"th":"เว็บตูน","ko":"웹툰","en":"Comic"}},"lang":"ko","id":63659962}},"result_code":0,"response_time":"2026-10-15T21:15:32+09:00","message":"성공","message_key":"api_common_success"}
|
||||
@@ -0,0 +1 @@
|
||||
{"result_code":-205,"response_time":"2026-10-15T21:19:40+09:00","message":"상품 정보를 찾을 수 없습니다.","message_key":"api_content_not_found"}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,48 @@
|
||||
20220930003210 https://page.kakao.com/content/54845189
|
||||
20221008033151 https://page.kakao.com/content/54845189
|
||||
20221101090844 https://page.kakao.com/content/54845189
|
||||
20221205234140 https://page.kakao.com/content/54845189
|
||||
20230324163551 https://page.kakao.com/content/54845189
|
||||
20230531233552 https://page.kakao.com/content/54845189
|
||||
20231001142338 https://page.kakao.com/content/54845189
|
||||
20240226032102 https://page.kakao.com/content/54845189
|
||||
20240318073658 https://page.kakao.com/content/54845189
|
||||
20240523032432 https://page.kakao.com/content/54845189
|
||||
20240601102321 https://page.kakao.com/content/54845189
|
||||
20240701122409 https://page.kakao.com/content/54845189
|
||||
20240801000811 https://page.kakao.com/content/54845189
|
||||
20240901010746 https://page.kakao.com/content/54845189
|
||||
20241001105828 https://page.kakao.com/content/54845189
|
||||
20241101010508 https://page.kakao.com/content/54845189
|
||||
20241203053728 https://page.kakao.com/content/54845189
|
||||
20250314051917 https://page.kakao.com/content/54845189
|
||||
20250406172928 https://page.kakao.com/content/54845189
|
||||
20250530014613 https://page.kakao.com/content/54845189
|
||||
20250714081524 https://page.kakao.com/content/54845189
|
||||
20250905000510 https://page.kakao.com/content/54845189
|
||||
20251007180706 https://page.kakao.com/content/54845189
|
||||
20251105010252 https://page.kakao.com/content/54845189
|
||||
20260115194553 https://page.kakao.com/content/54845189
|
||||
20260219143912 https://page.kakao.com/content/54845189
|
||||
20260510090433 https://page.kakao.com/content/54845189
|
||||
20260815043603 https://page.kakao.com/content/54845189
|
||||
20250116002021 https://page.kakao.com/content/54845189%0A
|
||||
20260222073655 https://page.kakao.com/content/54845189/viewer/54846128
|
||||
20221112093759 https://page.kakao.com/content/54845189/viewer/54848086
|
||||
20260218104652 https://page.kakao.com/content/54845189/viewer/54848086
|
||||
20221112093801 https://page.kakao.com/content/54845189/viewer/54848086?utm_source=68413516352
|
||||
20260116045007 https://page.kakao.com/content/54845189/viewer/54872355
|
||||
20230316171512 https://page.kakao.com/content/54845189/viewer/60579636
|
||||
20230505142706 https://page.kakao.com/content/54845189/viewer/61824808
|
||||
20241220095541 https://page.kakao.com/content/54845189?utm_source=sh_clip&plpu=https%3A%2F%2Flink-page.kakao.com%2Fopen%2Fcontent%3Fseries_id%3D54845189%26utm_source%3Dsh_clip%26alfallback%3Dstore
|
||||
20260908183105 https://page.kakao.com/content/54845189?orderby=desc
|
||||
20221013131308 https://page.kakao.com/content/54845189?utm_source=sh_clip&pscme=kakaopage%3A%2F%2Fopen%2Fcontent%3Fseries_id%3D54845189%26utm_source%3Dsh_clip
|
||||
20250813010346 https://page.kakao.com/content/54845189?tab_type=about
|
||||
20260613024948 https://page.kakao.com/content/54845189?tab_type=about
|
||||
20241204083755 https://page.kakao.com/content/54845189?tab_type=notice
|
||||
20250813014455 https://page.kakao.com/content/54845189?tab_type=notice
|
||||
20260317095133 https://page.kakao.com/content/54845189?tab_type=notice
|
||||
20250812030043 https://page.kakao.com/content/54845189?tab_type=overview
|
||||
20260119153727 https://page.kakao.com/content/54845189?tab_type=overview
|
||||
20260224113711 https://page.kakao.com/content/54845189?tab_type=overview
|
||||
20240522172309 https://page.kakao.com/content/54845189?tab_type=product
|
||||
@@ -0,0 +1,17 @@
|
||||
20200928100812 https://page.kakao.com/home?seriesId=54845189
|
||||
20201001065424 https://page.kakao.com/home?seriesId=54845189
|
||||
20201101082642 https://page.kakao.com/home?seriesId=54845189
|
||||
20210402034928 https://page.kakao.com/home?seriesId=54845189
|
||||
20210523164837 https://page.kakao.com/home?seriesId=54845189
|
||||
20211210081733 https://page.kakao.com/home?seriesId=54845189
|
||||
20220506064341 https://page.kakao.com/home?seriesId=54845189
|
||||
20220608173932 https://page.kakao.com/home?seriesId=54845189
|
||||
20220902101346 https://page.kakao.com/home?seriesId=54845189
|
||||
20210906090332 https://page.kakao.com/home?seriesId=54845189&utm_source=tw&utm_medium=thread&utm_campaign=09&utm_id=kakao
|
||||
20210825154421 https://page.kakao.com/home?seriesId=54845189&utm_source=tw&utm_medium=thread&utm_campaign=endingdeath&utm_id=kakao
|
||||
20210903094133 https://page.kakao.com/home?seriesId=54845189&utm_source=tw&utm_medium=thread&utm_campaign=endingdeath&utm_id=kakao
|
||||
20210825185306 https://page.kakao.com/home?seriesId=54845189&utm_id=kakao
|
||||
20220331090657 https://page.kakao.com/home?seriesId=54845189&utm_source=%E3%85%81%E3%84%B4%E3%84%B7%E3%84%B9
|
||||
20211201094036 https://page.kakao.com/home?seriesId=54845189&utm_source=11
|
||||
20220331122250 https://page.kakao.com/home?seriesId=54845189&utm_source=23%E3%85%87
|
||||
20220601111259 https://page.kakao.com/home?seriesId=54845189&utm_source=5968
|
||||
+7
-1
@@ -26,6 +26,12 @@ describe("gmTransport", () => {
|
||||
expect(requests[0]!.headers!["user-agent"]).toMatch(/^Mozilla/);
|
||||
});
|
||||
|
||||
test("KakaoPage's data endpoint gets its page as Referer", async () => {
|
||||
const { gm, requests } = fakeGm((r) => r.onload({ status: 200, statusText: "OK", responseHeaders: "", response: new ArrayBuffer(0) }));
|
||||
await gmTransport(gm).fetch("https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=1");
|
||||
expect(requests[0]!.headers!.referer).toBe("https://page.kakao.com/");
|
||||
});
|
||||
|
||||
test("Wayback gets the honest User-Agent", async () => {
|
||||
const { gm, requests } = fakeGm((r) => r.onload({ status: 200, statusText: "OK", responseHeaders: "", response: new ArrayBuffer(0) }));
|
||||
await gmTransport(gm).fetch("https://web.archive.org/cdx/search/cdx?url=x");
|
||||
@@ -65,7 +71,7 @@ test("banner: matches mangabaka.org, grants GM APIs, connects to every host it n
|
||||
expect(banner).toContain("// @version 0.1.0\n");
|
||||
expect(banner).toContain("// @match https://mangabaka.org/*\n");
|
||||
for (const grant of ["GM_xmlhttpRequest", "GM_getValue", "GM_setValue", "GM_deleteValue"]) expect(banner).toContain(`// @grant ${grant}\n`);
|
||||
for (const host of ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "web.archive.org", "catbox.moe", "mbseed.example.org"]) {
|
||||
for (const host of ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "kakao.com", "kakaoentcdn.com", "web.archive.org", "catbox.moe", "mbseed.example.org"]) {
|
||||
expect(banner).toContain(`// @connect ${host}\n`);
|
||||
}
|
||||
expect(banner).toEndWith("// ==/UserScript==\n\n");
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { extractCovers, parseOverview } from "../src/core/sources/kakaopage/parse";
|
||||
import { fixtureJson, fixtureText } from "./helpers/fixtures";
|
||||
|
||||
const IMG = "https://page-images.kakaoentcdn.com/download/resource?kid=";
|
||||
|
||||
test("overview of a comic: title, language, age grade, portrait and landscape", async () => {
|
||||
const o = parseOverview(await fixtureJson("kakaopage/overview-54845189.json"));
|
||||
expect(o).toMatchObject({ id: "54845189", title: "악역의 엔딩은 죽음뿐", language: "ko", ageGrade: 0, novel: false });
|
||||
expect(o.covers.map((c) => [c.role, c.url])).toEqual([
|
||||
["poster", `${IMG}mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0&filename=o1`],
|
||||
["banner", `${IMG}RmobN/hzCtfKUqaD/6XgE5RjbBGMOn6thodamr0&filename=o1`],
|
||||
]);
|
||||
});
|
||||
|
||||
test("a novel, and a 19+ edition without a landscape image", async () => {
|
||||
expect(parseOverview(await fixtureJson("kakaopage/overview-52654302-novel.json"))).toMatchObject({ id: "52654302", novel: true });
|
||||
const adult = parseOverview(await fixtureJson("kakaopage/overview-63659962-19.json"));
|
||||
expect(adult).toMatchObject({ id: "63659962", ageGrade: 19 });
|
||||
expect(adult.covers.map((c) => c.role)).toEqual(["poster"]);
|
||||
});
|
||||
|
||||
test("an answer without content is an error naming the reason", () => {
|
||||
expect(() => parseOverview({ result: null, message: "상품 정보를 찾을 수 없습니다." })).toThrow(/상품 정보를 찾을 수 없습니다/);
|
||||
expect(() => parseOverview("nonsense")).toThrow(/KakaoPage/);
|
||||
});
|
||||
|
||||
test("2020 capture (old address): og:image plus the series' own thumbnails", async () => {
|
||||
const covers = extractCovers(await fixtureText("wayback/kakaopage-54845189-2020.html"), new URL("https://page.kakao.com/home?seriesId=54845189"));
|
||||
expect(covers.map((c) => [c.role, c.url])).toEqual([
|
||||
["poster", `${IMG}dNtPVq/hyCvfq21v6/b4EGtZDhGNKvzdBjWXg290&filename=o1`],
|
||||
["banner", `${IMG}bx4cET/hyATqBNkj9/kUeJCNRNNpYDYRBtpK1Oh1&filename=o1`],
|
||||
]);
|
||||
});
|
||||
|
||||
test("2022 and 2024 captures: the og:image only (other series on the page are ignored)", async () => {
|
||||
const at = (y: number) => fixtureText(`wayback/kakaopage-54845189-${y}.html`);
|
||||
const page = new URL("https://page.kakao.com/content/54845189");
|
||||
expect(extractCovers(await at(2022), page).map((c) => [c.role, c.url])).toEqual([["other", `${IMG}RmobN/hzCtfKUqaD/6XgE5RjbBGMOn6thodamr0&filename=o1`]]);
|
||||
const y2024 = extractCovers(await at(2024), page);
|
||||
expect(y2024.map((c) => [c.role, c.url])).toEqual([["other", `${IMG}dBnFgi/hzXap8fqJ4/Kzjixkeeek6kH1eOyRb490&filename=o1`]]);
|
||||
expect(y2024[0]!.originalUrl).toBe("https://page-images.kakaoentcdn.com/download/resource?kid=dBnFgi/hzXap8fqJ4/Kzjixkeeek6kH1eOyRb490&filename=o1");
|
||||
});
|
||||
@@ -0,0 +1,62 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { waybackHistory } from "../src/core/history/wayback";
|
||||
import { allowedFetchHosts, allowedImageHosts, findSource } from "../src/core/sources/registry";
|
||||
import { SourceError } from "../src/core/sources/source";
|
||||
import { kakaopage } from "../src/core/sources/kakaopage";
|
||||
import { FakeTransport, json, text } from "./helpers/fake-transport";
|
||||
import { fixtureJson, fixtureText } from "./helpers/fixtures";
|
||||
|
||||
const API = "https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=";
|
||||
const overview = (name: string) => async () => json(await fixtureJson(`kakaopage/${name}.json`));
|
||||
|
||||
test("registered; its hosts are allowlisted", () => {
|
||||
expect(findSource(new URL("https://page.kakao.com/content/54845189"))?.id).toBe("kakaopage");
|
||||
for (const host of ["page.kakao.com", "bff-page.kakao.com", "page-images.kakaoentcdn.com", "dn-img-page.kakao.com"]) expect(allowedFetchHosts().has(host)).toBe(true);
|
||||
for (const host of ["page-images.kakaoentcdn.com", "dn-img-page.kakao.com"]) expect(allowedImageHosts().has(host)).toBe(true);
|
||||
});
|
||||
|
||||
test("resolve reads the overview; currentCovers reuses it", async () => {
|
||||
const fake = new FakeTransport().on(`${API}54845189`, overview("overview-54845189"));
|
||||
const series = await kakaopage.resolve(new URL("https://page.kakao.com/home?seriesId=54845189"), { transport: fake });
|
||||
expect(series).toEqual({
|
||||
source: "kakaopage", externalId: "54845189", title: "악역의 엔딩은 죽음뿐", language: "ko",
|
||||
canonicalUrl: "https://page.kakao.com/content/54845189",
|
||||
archiveUrls: ["page.kakao.com/content/54845189", "page.kakao.com/home?seriesId=54845189"],
|
||||
mature: false, flags: { novel: false },
|
||||
});
|
||||
const covers = await kakaopage.currentCovers(series, { transport: fake });
|
||||
expect(fake.calls.length).toBe(1);
|
||||
expect(covers.map((c) => [c.role, c.current])).toEqual([["poster", true], ["banner", true]]);
|
||||
});
|
||||
|
||||
test("19+ is mature; a novel is flagged", async () => {
|
||||
const fake = new FakeTransport().on(`${API}63659962`, overview("overview-63659962-19")).on(`${API}52654302`, overview("overview-52654302-novel"));
|
||||
expect((await kakaopage.resolve(new URL("https://page.kakao.com/content/63659962"), { transport: fake })).mature).toBe(true);
|
||||
expect((await kakaopage.resolve(new URL("https://page.kakao.com/content/52654302"), { transport: fake })).flags.novel).toBe(true);
|
||||
});
|
||||
|
||||
test("an id KakaoPage doesn't know, and an HTTP error", async () => {
|
||||
const fake = new FakeTransport().on(`${API}53998542`, overview("overview-not-found")).on(`${API}1`, () => text("blocked", 403));
|
||||
await expect(kakaopage.resolve(new URL("https://page.kakao.com/content/53998542"), { transport: fake })).rejects.toThrow(/상품 정보를 찾을 수 없습니다/);
|
||||
await expect(kakaopage.resolve(new URL("https://page.kakao.com/content/1"), { transport: fake })).rejects.toBeInstanceOf(SourceError);
|
||||
});
|
||||
|
||||
test("Wayback history through both addresses keeps only this id", async () => {
|
||||
const fake = new FakeTransport().on(`${API}54845189`, overview("overview-54845189"))
|
||||
.on((u) => u.startsWith("https://web.archive.org/cdx/") && u.includes("home"), async () => text(await fixtureText("wayback/kakaopage-54845189-cdx-home.txt")))
|
||||
.on((u) => u.startsWith("https://web.archive.org/cdx/"), async () => text(await fixtureText("wayback/kakaopage-54845189-cdx-content.txt")))
|
||||
.on((u) => /\/web\/2020\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2020.html")))
|
||||
.on((u) => /\/web\/202[12]\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2022.html")))
|
||||
.on((u) => /\/web\/20\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2024.html")));
|
||||
const series = await kakaopage.resolve(new URL("https://page.kakao.com/content/54845189"), { transport: fake });
|
||||
const found = await waybackHistory(kakaopage, series, { every: 12, transport: fake, warn: () => {} });
|
||||
const snapshots = fake.urls().filter((u) => u.includes("id_/"));
|
||||
expect(snapshots.length).toBeGreaterThan(3);
|
||||
expect(snapshots.every((u) => /seriesId=54845189(&|$)|content\/54845189(\/|$|\?)/.test(u))).toBe(true);
|
||||
expect(new Set(found.map((c) => c.url))).toEqual(new Set([
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=dNtPVq/hyCvfq21v6/b4EGtZDhGNKvzdBjWXg290&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=bx4cET/hyATqBNkj9/kUeJCNRNNpYDYRBtpK1Oh1&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=RmobN/hzCtfKUqaD/6XgE5RjbBGMOn6thodamr0&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=dBnFgi/hzXap8fqJ4/Kzjixkeeek6kH1eOyRb490&filename=o1",
|
||||
]));
|
||||
});
|
||||
@@ -0,0 +1,49 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { LinkError } from "../src/core/links";
|
||||
import { isKakaoPageUrl, isSameKakaoSeries, normalizeImageUrl, overviewUrl, parseKakaoPageUrl } from "../src/core/sources/kakaopage/urls";
|
||||
|
||||
const u = (s: string) => new URL(s);
|
||||
|
||||
describe("links", () => {
|
||||
test("both addresses, with or without the mobile host and extras", () => {
|
||||
expect(parseKakaoPageUrl(u("https://page.kakao.com/content/54845189"))).toBe("54845189");
|
||||
expect(parseKakaoPageUrl(u("https://page.kakao.com/content/54845189/viewer/1?tab=episode"))).toBe("54845189");
|
||||
expect(parseKakaoPageUrl(u("https://m.page.kakao.com/home?seriesId=54845189&utm_source=tw"))).toBe("54845189");
|
||||
});
|
||||
|
||||
test("anything else is not a KakaoPage series", () => {
|
||||
for (const s of ["https://page.kakao.com/", "https://page.kakao.com/content/abc", "https://page.kakao.com/home", "https://webtoon.kakao.com/content/x/2383"]) {
|
||||
expect(isKakaoPageUrl(u(s))).toBe(false);
|
||||
expect(() => parseKakaoPageUrl(u(s))).toThrow(LinkError);
|
||||
}
|
||||
});
|
||||
|
||||
test("the data endpoint", () => {
|
||||
expect(overviewUrl("54845189")).toBe("https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=54845189");
|
||||
});
|
||||
});
|
||||
|
||||
describe("image URLs", () => {
|
||||
test("the original on page-images.kakaoentcdn.com", () => {
|
||||
const original = "https://page-images.kakaoentcdn.com/download/resource?kid=mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0&filename=o1";
|
||||
expect(normalizeImageUrl("https://page-images.kakaoentcdn.com/download/resource?kid=mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0&filename=o1/dims/resize/384")).toBe(original);
|
||||
expect(normalizeImageUrl("//dn-img-page.kakao.com/download/resource?kid=mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0&filename=th3", "https://page.kakao.com/content/1")).toBe(original);
|
||||
expect(normalizeImageUrl("http://dn-img-page.kakao.com/download/resource?kid=mW3j5/dJMcaacSrAK/WVoJJAYNlS9rayXUWi6AY0&filename=th2".replace("&", "&"))).toBe(original);
|
||||
});
|
||||
|
||||
test("a bare key becomes the original URL; other hosts are left alone", () => {
|
||||
expect(normalizeImageUrl("bx4cET/hyATqBNkj9/kUeJCNRNNpYDYRBtpK1Oh1"))
|
||||
.toBe("https://page-images.kakaoentcdn.com/download/resource?kid=bx4cET/hyATqBNkj9/kUeJCNRNNpYDYRBtpK1Oh1&filename=o1");
|
||||
expect(normalizeImageUrl("https://example.org/x.jpg?filename=th3")).toBe("https://example.org/x.jpg?filename=th3");
|
||||
});
|
||||
});
|
||||
|
||||
describe("same series", () => {
|
||||
test("the exact id, in either address", () => {
|
||||
expect(isSameKakaoSeries("https://page.kakao.com/content/54845189", "54845189")).toBe(true);
|
||||
expect(isSameKakaoSeries("https://page.kakao.com/home?seriesId=54845189&utm_source=tw", "54845189")).toBe(true);
|
||||
expect(isSameKakaoSeries("https://page.kakao.com/home?seriesId=548451890", "54845189")).toBe(false);
|
||||
expect(isSameKakaoSeries("https://page.kakao.com/content/548451890", "54845189")).toBe(false);
|
||||
expect(isSameKakaoSeries("nope", "54845189")).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -55,6 +55,13 @@ describe("/api/fetch", () => {
|
||||
expect(new Headers(calls[1]!.init!.headers).get("user-agent")).toBe(TOOL_USER_AGENT);
|
||||
});
|
||||
|
||||
test("KakaoPage's data endpoint is fetched with a page.kakao.com Referer", async () => {
|
||||
const API = "https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=54845189";
|
||||
const { call, calls } = setup({}, { [API]: () => Response.json({ result: {} }) });
|
||||
expect((await call(fetchPath(API))).status).toBe(200);
|
||||
expect(new Headers(calls[0]!.init!.headers).get("referer")).toBe("https://page.kakao.com/");
|
||||
});
|
||||
|
||||
test.each(["https://evil.example/x", "http://169.254.169.254/latest", "ftp://www.webtoons.com/x", "https://www.webtoons.com:8443/x", "not a url"])(
|
||||
"%p is refused without being fetched", async (url) => {
|
||||
const { call, calls } = setup();
|
||||
|
||||
@@ -102,3 +102,12 @@ test("Wayback gets the honest User-Agent, everything else a browser one", () =>
|
||||
expect(TOOL_USER_AGENT).toMatch(/^mangabaka-seeder\/\d+\.\d+\.\d+ \(\+https:\/\//);
|
||||
expect(userAgentFor("https://www.webtoons.com/en/x")).toMatch(/^Mozilla\/5\.0/);
|
||||
});
|
||||
|
||||
test("request headers per host: KakaoPage's data endpoint gets its page as Referer", async () => {
|
||||
const { requestHeadersFor } = await import("../src/core/transport");
|
||||
expect(requestHeadersFor("https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=1")).toEqual({
|
||||
"user-agent": expect.stringMatching(/^Mozilla/), referer: "https://page.kakao.com/",
|
||||
});
|
||||
expect(requestHeadersFor("https://www.webtoons.com/x")).toEqual({ "user-agent": expect.stringMatching(/^Mozilla/) });
|
||||
expect(requestHeadersFor("https://web.archive.org/cdx/search/cdx?url=x")).toEqual({ "user-agent": TOOL_USER_AGENT });
|
||||
});
|
||||
|
||||
@@ -68,8 +68,9 @@ test("registry finds the source and builds the proxy allowlist", () => {
|
||||
expect(findSource(new URL("https://www.webtoons.com/en/x/list?title_no=1"))).toBe(webtoons);
|
||||
expect(findSource(new URL("https://www.lezhin.com/en/comic/x"))).toBeUndefined();
|
||||
expect([...allowedFetchHosts()].sort()).toEqual(
|
||||
["d30womf5coomej.cloudfront.net", "m.tapas.io", "m.webtoons.com", "swebtoon-phinf.pstatic.net", "tapas.io", "us-a.tapas.io",
|
||||
"web.archive.org", "www.tapas.io", "www.webtoons.com"]);
|
||||
["bff-page.kakao.com", "d30womf5coomej.cloudfront.net", "dn-img-page.kakao.com", "m.page.kakao.com", "m.tapas.io", "m.webtoons.com",
|
||||
"page-images.kakaoentcdn.com", "page.kakao.com", "swebtoon-phinf.pstatic.net", "tapas.io", "us-a.tapas.io", "web.archive.org",
|
||||
"www.tapas.io", "www.webtoons.com"]);
|
||||
});
|
||||
|
||||
test("isSameSeries checks title_no", () => {
|
||||
|
||||
Reference in New Issue
Block a user