The cut was marked with U+2026, which NFKC decomposes into three dots —
so re-sanitising a truncated string grew it by two characters and re-cut
it at a different word boundary. A title would quietly rewrite itself
every time the event was re-ingested, and the module promises
idempotency in its own docstring.
Append what normalisation would produce instead. The existing corpus
missed this because its only over-length entry has no space in its last
40%, so it happened to re-truncate to the identical string; the new test
uses prose.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Scraped text becomes React content, JSON on disk and eventually SQLite
rows, so it gets cleaned at one seam: the parse wrapper in toAdapter().
Every source passes through it, a source added tomorrow is covered
without its author doing anything, and no parser can opt out — parsers
stay pure readers of one site's markup.
Removes script/style/comment content and residual tags, decodes entities
to a fixed point so an encoded tag cannot resurrect in a later decoder,
NFKC-normalises, strips control, zero-width and bidi-override characters
(an RTL override visually spoofs a title), bounds each field to the cap
the schema already declares, and requires sourceUrl to be absolute
http(s).
Three things it will not do: touch a date, drop an event it could clean
instead, or repair in silence — every repair and drop is logged by
default. Event IDs are localStorage keys, so an ID is recomputed only
when a sanitised title actually changed and the ID was minted the
standard way; all seven fixtures pass through unrepaired and
byte-identical, which is the regression guard.
Also stops decodeEntities throwing RangeError on an out-of-range
numeric reference, which would have taken a whole source's events down.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
