feat: KakaoPage source (웹툰 and 웹소설), registered; correct the Tapas cookie claim
KakaoPage is read from its data endpoint (current covers) and from archived server-rendered pages under both addresses (history). The userscript never sent Tapas cookies (anonymous: true): docs and the age-check warning now say so. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1d48796d05
commit
5b2ad867b3
@@ -9,7 +9,7 @@ MusicBrainz.
|
||||
> [the spec](docs/specs/001-webtoons-covers.md).
|
||||
|
||||
Today's goal is one thing done well: **given a MangaBaka series link and a
|
||||
WEBTOON or Tapas series link, find every cover that series has used, current
|
||||
WEBTOON, Tapas or KakaoPage series link, find every cover that series has used, current
|
||||
and historical, and request them added to the MangaBaka series**. The current
|
||||
poster becomes the main (default) cover for the edition's language.
|
||||
|
||||
@@ -82,11 +82,13 @@ least 25 approved submissions.
|
||||
| Source | Links | Current covers | Historical covers |
|
||||
|---|---|---|---|
|
||||
| WEBTOON (official & canvas) | `https://www.webtoons.com/<lang>/…/list?title_no=N`, `m.webtoons.com` | poster, square, banner, foreground | Wayback Machine |
|
||||
| KakaoPage (웹툰 & 웹소설) | `https://page.kakao.com/content/N` (or the old `…/home?seriesId=N`) | portrait cover (full size), landscape thumbnail | Wayback Machine |
|
||||
| Tapas (comics & novels) | `https://tapas.io/series/<name>` (or `…/info`, the series number, or an `/episode/N` link) | portrait cover (full size), banner | Wayback Machine |
|
||||
|
||||
Tapas is English only. Mature Tapas series are shown only to signed-in adults:
|
||||
the page can't read their current covers (it says so and still searches the
|
||||
archive), but the userscript can, when you are signed in to Tapas in that browser.
|
||||
Tapas is English only; KakaoPage is Korean. Mature Tapas series are shown only
|
||||
to signed-in adults: their current covers can't be read (the tool says so and
|
||||
still searches the archive). KakaoPage 15+ and 19+ titles are read normally and
|
||||
show the content-rating warning.
|
||||
|
||||
Each Webtoons language edition has its own `title_no`. Run the tool once per
|
||||
edition; the covers are tagged with that edition's language.
|
||||
|
||||
@@ -15,7 +15,7 @@ self-hosted Bun server (primary), a userscript, and later a Bun CLI.
|
||||
| `src/core/session.ts` | Review state shared by the page and the userscript: cards, main cover, rehosting (cards remember host, cache key and expiry), manual text, the download bundle, `restore` from history, token flow (preview → confirm → save for review). |
|
||||
| `src/core/zip.ts`, `download.ts` | Store-only ZIP writer (CRC-32, UTF-8 names); the download bundle: file names `NN_role_lang[_YYYY]_WxH.ext` and `covers.txt` (spec 002 §2). |
|
||||
| `src/core/saved.ts` | History model (spec 002 §3): `SavedSearch` (plain data, source by id), `toSaved`/`fromSaved` (drops expired links), `carryEdits` (search again), `pruneIds` (30 entries, 300 MB), `HistoryStore` interface and an in-memory one. |
|
||||
| `src/core/sources/` | `webtoons/` and `tapas/` (spec 003): each `urls.ts` (links, image URL normalisation, same-series), `parse.ts` (pure HTML parsing, also used on Wayback snapshots), `index.ts` (the `Source`); `registry.ts` lists them and builds the proxy and upload allowlists. |
|
||||
| `src/core/sources/` | `webtoons/`, `tapas/` (spec 003) and `kakaopage/` (spec 004; reads a JSON endpoint, not the page): each `urls.ts` (links, image URL normalisation, same-series), `parse.ts` (pure HTML parsing, also used on Wayback snapshots), `index.ts` (the `Source`); `registry.ts` lists them and builds the proxy and upload allowlists. |
|
||||
| `src/web/` | Built. The page ("Obi Shelf" design, see `DESIGN.md`): `template.ts` markup shared with the userscript panel, `app.ts` wiring (`mountApp`), `render.ts` shelf and timeline views, `view-model.ts` pure helpers, `transport.ts` (direct to MangaBaka/litterbox, everything else via `/api/fetch`), `styles.css` scoped for a shadow root, `save-file.ts` (Blob download, works in the panel's shadow root), `history-db.ts` (IndexedDB `HistoryStore`; null when storage is refused), `history-keeper.ts` (when the series on screen is saved: after a search, debounced edits, marks; flushes before a new search, an open and page hide; removal/clear stop tracking), `recent.ts` (History list and the start screen's recent shelf; the page only), `dark-reader.ts` (switches to the page's own dark theme and locks Dark Reader out when that extension is darkening the page). |
|
||||
| `src/server/` | Built. `app.ts`: web-standard handler (`createApp(env)`) — static assets, `/api/config`, `/api/fetch` allowlisted proxy (redirects re-checked, 20 MB cap), `/api/upload` (store or catbox; takes a source URL, never bytes), `/covers/*`. `storage.ts` (interface, in-memory) and `storage-dir.ts` (folder, size cap, 30-day sweep). `config.ts` (env, client IP), `rate-limit.ts`, `assets-path.ts`. `bun.ts`: thin `Bun.serve` entry. Gate and Workers entry deferred. |
|
||||
| `src/userscript/` | Built. `gm.ts`: `GM_xmlhttpRequest` transport (no proxy; honest User-Agent for Wayback) and GM storage. `main.ts`: a "Seed covers" launcher on MangaBaka series pages (follows client-side navigation) opening the page's app in a shadow-root panel; optional server URL in its settings. Built by `scripts/build-userscript.ts` into `dist/web/mbseed.user.js`, served at `/mbseed.user.js`. |
|
||||
@@ -61,6 +61,7 @@ requests) and smoke-tests it. Example deployment: `docker-compose.example.yml`.
|
||||
| Webtoons HTML / `swebtoon-phinf.pstatic.net` | server proxy or `GM_xmlhttpRequest` | Nothing works. |
|
||||
| Wayback (`web.archive.org`) | same | Warning; current covers only. |
|
||||
| Server storage (folder on our server) | server | `server` host hidden; litterbox/catbox/direct still work. |
|
||||
| KakaoPage (`bff-page.kakao.com`, images on `page-images.kakaoentcdn.com`) | server proxy or `GM_xmlhttpRequest`, with a `page.kakao.com` Referer (`requestHeadersFor`) | That source fails. |
|
||||
| Shynet (`analytics.lucaswinther.info`) | `<script>` on the page only | No page-view counts; nothing else changes. |
|
||||
| litterbox / catbox | browser directly (litterbox), server or `GM_xmlhttpRequest` (catbox) | That host fails; others still work. |
|
||||
|
||||
|
||||
@@ -16,6 +16,28 @@ the repo every session — measurably slower and more tokens).
|
||||
**Consequences:** Rule changes happen in one file; pointer files must stay thin. Plans are working
|
||||
artefacts: kept after merge they go stale and contradict the code, so the spec is the record.
|
||||
|
||||
## 2026-10-15 — KakaoPage source; per-host request headers (spec 004)
|
||||
**Status:** accepted
|
||||
**Context:** The user asked for KakaoPage next. Its pages render in the browser; the data comes from
|
||||
`bff-page.kakao.com/api/gateway/api/v1/content/overview`, which answers 403 without a `page.kakao.com` Referer.
|
||||
**Decision:** A third `Source` that reads that endpoint for current covers (`thumbnail` portrait, `land_thumbnail`
|
||||
landscape; originals via `filename=o1`) and archived server-rendered pages (both the old `/home?seriesId=` and
|
||||
`/content/` addresses) for history. `core/transport.ts` gains `requestHeadersFor(url)` (User-Agent as before, plus
|
||||
a Referer for `bff-page.kakao.com`); the server proxy and the userscript send it. `age_grade >= 15` sets `mature`
|
||||
(the rating warning). Kakao Webtoon (`webtoon.kakao.com`) is a separate site and not included.
|
||||
**Alternatives:** Rendering the page in a headless browser (heavy, server-only); reading the JS bundle's API
|
||||
calls at run time (brittle).
|
||||
**Consequences:** We depend on an internal API and its Referer check. A Referer is a browser-like header, as the
|
||||
Chrome User-Agent is for Webtoons.
|
||||
|
||||
## 2026-10-15 — Correction: the userscript does not send Tapas cookies
|
||||
**Status:** accepted (corrects "Tapas source (spec 003)" below)
|
||||
**Context:** The Tapas entry and spec 003 said a signed-in adult gets mature Tapas pages through the userscript.
|
||||
`gmTransport` uses `anonymous: true`, so no cookies are sent.
|
||||
**Decision:** The docs and the warning now say only the Wayback history helps for age-checked Tapas series.
|
||||
Sending cookies would need `anonymous: false` for tapas.io only; deferred (TODO).
|
||||
**Consequences:** None in code; the earlier claim was never true.
|
||||
|
||||
## 2026-10-15 — Page views via our Shynet instance
|
||||
**Status:** accepted
|
||||
**Context:** The user asked for analytics from their self-hosted Shynet.
|
||||
|
||||
+1
-1
@@ -49,7 +49,7 @@ Design: [specs/001-webtoons-covers.md](specs/001-webtoons-covers.md).
|
||||
find the series (check `ageGateTarget` there); a mature series pasted by slug keeps the slug as its id, so a
|
||||
MangaBaka link by number warns; an all-digit slug is read as an id; accept a quoted `pageMeta` id; archived
|
||||
history covers fall back to the 400×600 `_z` copy when the original is gone.
|
||||
- [ ] Verify with a signed-in, age-verified Tapas account (userscript): the mature info page parses.
|
||||
- [ ] Mature Tapas series in the userscript: send the user's tapas.io cookies (`anonymous: false` for tapas.io only), then check that a signed-in adult's info page parses.
|
||||
- [ ] Possible race (unconfirmed): pressing **Find covers** before the page has mounted (it awaits `/api/config`
|
||||
and IndexedDB first) may submit the form natively and reload. Seen once in a scripted run; not reproduced.
|
||||
|
||||
|
||||
@@ -27,8 +27,9 @@ Checked against *Villains Are Destined to Die* (comic, id 186414) and its novel
|
||||
- Lists use `_m` (180 px) of other images; `pc/…` images are signed promotions of other series. Neither is used.
|
||||
- Older pages use `d30womf5coomej.cloudfront.net/sa/…`; the same paths answer on `us-a.tapas.io`.
|
||||
- **Mature series:** without a signed-in, age-verified account, `/series/…/info` redirects to
|
||||
`/account/verify-age?from=https://tapas.io/series/<slug>/info`, a page with no series data. The userscript's
|
||||
`GM_xmlhttpRequest` sends the user's own tapas.io cookies, so a signed-in adult gets the real page there.
|
||||
`/account/verify-age?from=https://tapas.io/series/<slug>/info`, a page with no series data.
|
||||
*Corrected 2026-10-15 (spec 004 work):* the userscript's `GM_xmlhttpRequest` runs with `anonymous: true`, so it
|
||||
does **not** send the user's tapas.io cookies; the userscript sees the age check too.
|
||||
- **Language:** English only (MangaBaka `en`).
|
||||
- **Wayback:** `tapas.io/series/<slug>` captured about monthly since 2020; the prefix also matches
|
||||
`<slug>-novel` and other longer slugs.
|
||||
@@ -43,8 +44,8 @@ Checked against *Villains Are Destined to Die* (comic, id 186414) and its novel
|
||||
- **Current covers:** portrait (role `poster`, suffix removed) and banner (role `banner`). The poster is the
|
||||
suggested main cover as for Webtoons.
|
||||
- **Mature (age check):** the series is resolved from the `from=` link (slug; title from the slug), `mature: true`,
|
||||
no current covers, and a warning: Tapas shows this series only to signed-in adults; use the userscript while
|
||||
signed in, or tick the Wayback history. History still runs.
|
||||
no current covers, and a warning that Tapas shows it only to signed-in adults; tick the Wayback history.
|
||||
History still runs.
|
||||
- **Image URLs:** `normalizeImageUrl` maps both image hosts to `us-a.tapas.io`, drops `_<letters>` size suffixes
|
||||
after the UUID and any query.
|
||||
- **Same series:** a link is the same series when it is a tapas.io `/series/<slug>[/info][/]` with the same slug
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
/** Tampermonkey/Violentmonkey metadata. @connect lists every host GM_xmlhttpRequest may reach. */
|
||||
export function userscriptBanner(version: string, serverUrl?: string): string {
|
||||
const connects = ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "web.archive.org", "catbox.moe"];
|
||||
const connects = ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "kakao.com", "kakaoentcdn.com", "web.archive.org", "catbox.moe"];
|
||||
if (serverUrl) connects.push(new URL(serverUrl).hostname);
|
||||
const lines = [
|
||||
"// ==UserScript==",
|
||||
"// @name MangaBaka Seeder",
|
||||
"// @namespace https://gitea.lucaswinther.info/lucasw89/mangabaka-seeder",
|
||||
`// @version ${version}`,
|
||||
"// @description Find every cover of a WEBTOON or Tapas series and turn them into a MangaBaka cover edit request",
|
||||
"// @description Find every cover of a WEBTOON, Tapas or KakaoPage series and turn them into a MangaBaka cover edit request",
|
||||
"// @match https://mangabaka.org/*",
|
||||
"// @grant GM_xmlhttpRequest",
|
||||
"// @grant GM_getValue",
|
||||
|
||||
@@ -72,7 +72,7 @@ export async function runCovers(
|
||||
warn(`MangaBaka series ${seriesId} has no link to this ${source.name} series (${series.canonicalUrl}). Check that the two links belong together.`);
|
||||
}
|
||||
if (series.flags.ageGated) {
|
||||
warn(`${source.name} shows this series only to signed-in adults, so its current covers couldn't be read. Use the userscript while signed in there, or tick "Old covers from the Wayback Machine".`);
|
||||
warn(`${source.name} shows this series only to signed-in adults, so its current covers couldn't be read. Tick "Old covers from the Wayback Machine" for archived ones.`);
|
||||
}
|
||||
if (series.language === "unknown") warn(`Couldn't tell the language of ${series.canonicalUrl}; covers will be marked "unknown".`);
|
||||
const current = await source.currentCovers(series, ctx);
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import type { CoverCandidate, SourceSeries } from "../../model";
|
||||
import { SourceError, type Source, type SourceContext } from "../source";
|
||||
import { extractCovers, parseOverview, type Overview } from "./parse";
|
||||
import {
|
||||
API_HOST, IMAGE_HOSTS, PAGE_HOSTS, contentUrl, isKakaoPageUrl, isSameKakaoSeries, normalizeImageUrl, overviewUrl,
|
||||
parseKakaoPageUrl,
|
||||
} from "./urls";
|
||||
|
||||
/** Covers read by resolve(), so currentCovers() doesn't ask twice. */
|
||||
const currentBySeries = new WeakMap<SourceSeries, CoverCandidate[]>();
|
||||
|
||||
/** The page is rendered in the browser; its data endpoint has what we need (spec 004 §1). */
|
||||
async function fetchOverview(id: string, ctx: SourceContext): Promise<Overview> {
|
||||
const res = await ctx.transport.fetch(overviewUrl(id), { signal: ctx.signal });
|
||||
const body: unknown = await res.json().catch(() => null);
|
||||
if (!res.ok && !body) throw new SourceError(`KakaoPage answered HTTP ${res.status} for series ${id}.`);
|
||||
const overview = parseOverview(body);
|
||||
if (overview.id !== id) throw new SourceError(`KakaoPage sent series ${overview.id} when asked for ${id}.`);
|
||||
return overview;
|
||||
}
|
||||
|
||||
export const kakaopage: Source = {
|
||||
id: "kakaopage",
|
||||
name: "KakaoPage",
|
||||
hosts: [...PAGE_HOSTS, API_HOST, ...IMAGE_HOSTS],
|
||||
imageHosts: [...IMAGE_HOSTS],
|
||||
|
||||
matches: (url) => isKakaoPageUrl(url),
|
||||
|
||||
async resolve(url, ctx) {
|
||||
const id = parseKakaoPageUrl(url);
|
||||
const o = await fetchOverview(id, ctx);
|
||||
const series: SourceSeries = {
|
||||
source: "kakaopage",
|
||||
externalId: id,
|
||||
title: o.title,
|
||||
language: o.language,
|
||||
canonicalUrl: contentUrl(id),
|
||||
archiveUrls: [`page.kakao.com/content/${id}`, `page.kakao.com/home?seriesId=${id}`],
|
||||
// 15+ and 19+ titles: suggest a rating above "safe" (spec 004 §2).
|
||||
mature: o.ageGrade >= 15,
|
||||
flags: { novel: o.novel },
|
||||
};
|
||||
currentBySeries.set(series, o.covers.map((c) => ({ ...c, current: true })));
|
||||
return series;
|
||||
},
|
||||
|
||||
async currentCovers(series, ctx) {
|
||||
const cached = currentBySeries.get(series);
|
||||
if (cached) return cached;
|
||||
return (await fetchOverview(series.externalId, ctx)).covers.map((c) => ({ ...c, current: true }));
|
||||
},
|
||||
|
||||
extractCovers: (html, pageUrl) => extractCovers(html, pageUrl),
|
||||
normalizeImageUrl: (url) => normalizeImageUrl(url),
|
||||
isSameSeries: (linkUrl, series) => isSameKakaoSeries(linkUrl, series.externalId),
|
||||
};
|
||||
@@ -1,9 +1,10 @@
|
||||
import { WAYBACK_HOST } from "../transport";
|
||||
import type { Source } from "./source";
|
||||
import { kakaopage } from "./kakaopage";
|
||||
import { tapas } from "./tapas";
|
||||
import { webtoons } from "./webtoons";
|
||||
|
||||
export const SOURCES: readonly Source[] = [webtoons, tapas];
|
||||
export const SOURCES: readonly Source[] = [webtoons, tapas, kakaopage];
|
||||
|
||||
export function findSource(url: URL, sources: readonly Source[] = SOURCES): Source | undefined {
|
||||
return sources.find((s) => s.matches(url));
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@ export const APP_HTML = /* html */ `
|
||||
<label class="mbseed-slip">
|
||||
<span class="mbseed-slip-label">Source series link</span>
|
||||
<input id="source-link" name="source" required autocomplete="off" spellcheck="false" placeholder="https://www.webtoons.com/en/fantasy/tower-of-god/list?title_no=95">
|
||||
<span class="mbseed-slip-hint">The series page on the manga, manhwa or webcomic site. WEBTOON or Tapas for now.</span>
|
||||
<span class="mbseed-slip-hint">The series page on the manga, manhwa or webcomic site. WEBTOON, Tapas or KakaoPage for now.</span>
|
||||
</label>
|
||||
<div class="mbseed-settings">
|
||||
<label class="mbseed-check"><input id="history-on" type="checkbox"> Old covers from the Wayback Machine</label>
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
{"result_code":-205,"response_time":"2026-10-15T21:19:40+09:00","message":"상품 정보를 찾을 수 없습니다.","message_key":"api_content_not_found"}
|
||||
+1
-1
@@ -71,7 +71,7 @@ test("banner: matches mangabaka.org, grants GM APIs, connects to every host it n
|
||||
expect(banner).toContain("// @version 0.1.0\n");
|
||||
expect(banner).toContain("// @match https://mangabaka.org/*\n");
|
||||
for (const grant of ["GM_xmlhttpRequest", "GM_getValue", "GM_setValue", "GM_deleteValue"]) expect(banner).toContain(`// @grant ${grant}\n`);
|
||||
for (const host of ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "web.archive.org", "catbox.moe", "mbseed.example.org"]) {
|
||||
for (const host of ["api.mangabaka.org", "webtoons.com", "pstatic.net", "tapas.io", "d30womf5coomej.cloudfront.net", "kakao.com", "kakaoentcdn.com", "web.archive.org", "catbox.moe", "mbseed.example.org"]) {
|
||||
expect(banner).toContain(`// @connect ${host}\n`);
|
||||
}
|
||||
expect(banner).toEndWith("// ==/UserScript==\n\n");
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
import { expect, test } from "bun:test";
|
||||
import { waybackHistory } from "../src/core/history/wayback";
|
||||
import { allowedFetchHosts, allowedImageHosts, findSource } from "../src/core/sources/registry";
|
||||
import { SourceError } from "../src/core/sources/source";
|
||||
import { kakaopage } from "../src/core/sources/kakaopage";
|
||||
import { FakeTransport, json, text } from "./helpers/fake-transport";
|
||||
import { fixtureJson, fixtureText } from "./helpers/fixtures";
|
||||
|
||||
const API = "https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=";
|
||||
const overview = (name: string) => async () => json(await fixtureJson(`kakaopage/${name}.json`));
|
||||
|
||||
test("registered; its hosts are allowlisted", () => {
|
||||
expect(findSource(new URL("https://page.kakao.com/content/54845189"))?.id).toBe("kakaopage");
|
||||
for (const host of ["page.kakao.com", "bff-page.kakao.com", "page-images.kakaoentcdn.com", "dn-img-page.kakao.com"]) expect(allowedFetchHosts().has(host)).toBe(true);
|
||||
for (const host of ["page-images.kakaoentcdn.com", "dn-img-page.kakao.com"]) expect(allowedImageHosts().has(host)).toBe(true);
|
||||
});
|
||||
|
||||
test("resolve reads the overview; currentCovers reuses it", async () => {
|
||||
const fake = new FakeTransport().on(`${API}54845189`, overview("overview-54845189"));
|
||||
const series = await kakaopage.resolve(new URL("https://page.kakao.com/home?seriesId=54845189"), { transport: fake });
|
||||
expect(series).toEqual({
|
||||
source: "kakaopage", externalId: "54845189", title: "악역의 엔딩은 죽음뿐", language: "ko",
|
||||
canonicalUrl: "https://page.kakao.com/content/54845189",
|
||||
archiveUrls: ["page.kakao.com/content/54845189", "page.kakao.com/home?seriesId=54845189"],
|
||||
mature: false, flags: { novel: false },
|
||||
});
|
||||
const covers = await kakaopage.currentCovers(series, { transport: fake });
|
||||
expect(fake.calls.length).toBe(1);
|
||||
expect(covers.map((c) => [c.role, c.current])).toEqual([["poster", true], ["banner", true]]);
|
||||
});
|
||||
|
||||
test("19+ is mature; a novel is flagged", async () => {
|
||||
const fake = new FakeTransport().on(`${API}63659962`, overview("overview-63659962-19")).on(`${API}52654302`, overview("overview-52654302-novel"));
|
||||
expect((await kakaopage.resolve(new URL("https://page.kakao.com/content/63659962"), { transport: fake })).mature).toBe(true);
|
||||
expect((await kakaopage.resolve(new URL("https://page.kakao.com/content/52654302"), { transport: fake })).flags.novel).toBe(true);
|
||||
});
|
||||
|
||||
test("an id KakaoPage doesn't know, and an HTTP error", async () => {
|
||||
const fake = new FakeTransport().on(`${API}53998542`, overview("overview-not-found")).on(`${API}1`, () => text("blocked", 403));
|
||||
await expect(kakaopage.resolve(new URL("https://page.kakao.com/content/53998542"), { transport: fake })).rejects.toThrow(/상품 정보를 찾을 수 없습니다/);
|
||||
await expect(kakaopage.resolve(new URL("https://page.kakao.com/content/1"), { transport: fake })).rejects.toBeInstanceOf(SourceError);
|
||||
});
|
||||
|
||||
test("Wayback history through both addresses keeps only this id", async () => {
|
||||
const fake = new FakeTransport().on(`${API}54845189`, overview("overview-54845189"))
|
||||
.on((u) => u.startsWith("https://web.archive.org/cdx/") && u.includes("home"), async () => text(await fixtureText("wayback/kakaopage-54845189-cdx-home.txt")))
|
||||
.on((u) => u.startsWith("https://web.archive.org/cdx/"), async () => text(await fixtureText("wayback/kakaopage-54845189-cdx-content.txt")))
|
||||
.on((u) => /\/web\/2020\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2020.html")))
|
||||
.on((u) => /\/web\/202[12]\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2022.html")))
|
||||
.on((u) => /\/web\/20\d+id_\//.test(u), async () => text(await fixtureText("wayback/kakaopage-54845189-2024.html")));
|
||||
const series = await kakaopage.resolve(new URL("https://page.kakao.com/content/54845189"), { transport: fake });
|
||||
const found = await waybackHistory(kakaopage, series, { every: 12, transport: fake, warn: () => {} });
|
||||
const snapshots = fake.urls().filter((u) => u.includes("id_/"));
|
||||
expect(snapshots.length).toBeGreaterThan(3);
|
||||
expect(snapshots.every((u) => /seriesId=54845189(&|$)|content\/54845189(\/|$|\?)/.test(u))).toBe(true);
|
||||
expect(new Set(found.map((c) => c.url))).toEqual(new Set([
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=dNtPVq/hyCvfq21v6/b4EGtZDhGNKvzdBjWXg290&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=bx4cET/hyATqBNkj9/kUeJCNRNNpYDYRBtpK1Oh1&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=RmobN/hzCtfKUqaD/6XgE5RjbBGMOn6thodamr0&filename=o1",
|
||||
"https://page-images.kakaoentcdn.com/download/resource?kid=dBnFgi/hzXap8fqJ4/Kzjixkeeek6kH1eOyRb490&filename=o1",
|
||||
]));
|
||||
});
|
||||
@@ -55,6 +55,13 @@ describe("/api/fetch", () => {
|
||||
expect(new Headers(calls[1]!.init!.headers).get("user-agent")).toBe(TOOL_USER_AGENT);
|
||||
});
|
||||
|
||||
test("KakaoPage's data endpoint is fetched with a page.kakao.com Referer", async () => {
|
||||
const API = "https://bff-page.kakao.com/api/gateway/api/v1/content/overview?series_id=54845189";
|
||||
const { call, calls } = setup({}, { [API]: () => Response.json({ result: {} }) });
|
||||
expect((await call(fetchPath(API))).status).toBe(200);
|
||||
expect(new Headers(calls[0]!.init!.headers).get("referer")).toBe("https://page.kakao.com/");
|
||||
});
|
||||
|
||||
test.each(["https://evil.example/x", "http://169.254.169.254/latest", "ftp://www.webtoons.com/x", "https://www.webtoons.com:8443/x", "not a url"])(
|
||||
"%p is refused without being fetched", async (url) => {
|
||||
const { call, calls } = setup();
|
||||
|
||||
@@ -68,8 +68,9 @@ test("registry finds the source and builds the proxy allowlist", () => {
|
||||
expect(findSource(new URL("https://www.webtoons.com/en/x/list?title_no=1"))).toBe(webtoons);
|
||||
expect(findSource(new URL("https://www.lezhin.com/en/comic/x"))).toBeUndefined();
|
||||
expect([...allowedFetchHosts()].sort()).toEqual(
|
||||
["d30womf5coomej.cloudfront.net", "m.tapas.io", "m.webtoons.com", "swebtoon-phinf.pstatic.net", "tapas.io", "us-a.tapas.io",
|
||||
"web.archive.org", "www.tapas.io", "www.webtoons.com"]);
|
||||
["bff-page.kakao.com", "d30womf5coomej.cloudfront.net", "dn-img-page.kakao.com", "m.page.kakao.com", "m.tapas.io", "m.webtoons.com",
|
||||
"page-images.kakaoentcdn.com", "page.kakao.com", "swebtoon-phinf.pstatic.net", "tapas.io", "us-a.tapas.io", "web.archive.org",
|
||||
"www.tapas.io", "www.webtoons.com"]);
|
||||
});
|
||||
|
||||
test("isSameSeries checks title_no", () => {
|
||||
|
||||
Reference in New Issue
Block a user