Reading a page: cutting fields and the page by UTF-16 units can split an emoji, and Postulo refuses the capture #25

New issue

Closed

opened 2026-10-01 10:47:19 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:19 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-chromium at 7998de372, reading src/lib/parse.js and checking the cut against the core's JobPostingData at postulo 95348908d.

The extension cuts every text field, and the slimmed page, with String.prototype.slice, which counts UTF-16 code units. When the cut falls inside a character outside the Basic Multilingual Plane (an emoji, some CJK), the kept string ends in a lone high surrogate. JSON.stringify sends it as \ud83d, and the core's pydantic schemas refuse any string holding one, so the whole capture comes back refused with a message the person cannot relate to anything on screen.

Where:
- src/lib/parse.js:694, tidy(): value.trim().slice(0, LIMITS[field] ?? (field === "description" ? Infinity : MAX_FIELD))
- src/lib/parse.js:698, tidy(): tidied.description.slice(0, MAX_DESCRIPTION)
- src/lib/parse.js:663-664, pageMetadata(): title.slice(0, MAX_FIELD) and the site name
- src/lib/parse.js:862, slimPage(): html.slice(0, MAX_PAGE)
What happens: a posting title of 499 ASCII characters followed by U+1F680 comes out of tidy() 500 units long, its last unit 0xd83d (checked with node against the snapshot). On the core, JobPostingData.model_validate({"title": "Engineer \ud83d"}) raises string_unicode ("Input should be a valid string, unable to parse raw data as a unicode string"); a description does the same, and CaptureIn.html and CorrectionsIn are plain str fields validated the same way, so a page cut at MAX_PAGE inside a surrogate pair is refused too. The outbox marks the capture refused (src/lib/outbox.js:75-76) and the popup shows that sentence against the field or the page.
Why it matters: a capture of a long, emoji-rich posting (titles and descriptions on some boards carry them routinely) fails for a reason the person cannot see or fix; editing the visible text removes the broken unit only by luck. A page heavier than MAX_PAGE after slimming fails the same way whatever the person corrects.
Fix: add one helper that cuts on code points, for example one that drops the last kept unit when it is a high surrogate (0xd800–0xdbff), and use it at all four places. Counting code points also matches the core's max_length, which counts them, so a field the extension keeps is never longer than the core accepts.

Done when

test/parse.test.mjs has a case where tidy() cuts a title of 499 characters plus an emoji, and one where slimPage() cuts a page whose MAX_PAGE boundary falls inside a surrogate pair, both asserting that the result contains no lone surrogate (/[\ud800-\udbff](?![\udc00-\udfff])/ does not match).

Found on 2026-09-29 in a read-only review of postulo-chromium at 7998de372, reading `src/lib/parse.js` and checking the cut against the core's `JobPostingData` at postulo 95348908d. The extension cuts every text field, and the slimmed page, with `String.prototype.slice`, which counts UTF-16 code units. When the cut falls inside a character outside the Basic Multilingual Plane (an emoji, some CJK), the kept string ends in a lone high surrogate. `JSON.stringify` sends it as `\ud83d`, and the core's pydantic schemas refuse any string holding one, so the whole capture comes back refused with a message the person cannot relate to anything on screen. - **Where:** - `src/lib/parse.js:694`, `tidy()`: `value.trim().slice(0, LIMITS[field] ?? (field === "description" ? Infinity : MAX_FIELD))` - `src/lib/parse.js:698`, `tidy()`: `tidied.description.slice(0, MAX_DESCRIPTION)` - `src/lib/parse.js:663-664`, `pageMetadata()`: `title.slice(0, MAX_FIELD)` and the site name - `src/lib/parse.js:862`, `slimPage()`: `html.slice(0, MAX_PAGE)` - **What happens:** a posting title of 499 ASCII characters followed by U+1F680 comes out of `tidy()` 500 units long, its last unit `0xd83d` (checked with node against the snapshot). On the core, `JobPostingData.model_validate({"title": "Engineer \ud83d"})` raises `string_unicode` ("Input should be a valid string, unable to parse raw data as a unicode string"); a description does the same, and `CaptureIn.html` and `CorrectionsIn` are plain `str` fields validated the same way, so a page cut at `MAX_PAGE` inside a surrogate pair is refused too. The outbox marks the capture `refused` (`src/lib/outbox.js:75-76`) and the popup shows that sentence against the field or the page. - **Why it matters:** a capture of a long, emoji-rich posting (titles and descriptions on some boards carry them routinely) fails for a reason the person cannot see or fix; editing the visible text removes the broken unit only by luck. A page heavier than `MAX_PAGE` after slimming fails the same way whatever the person corrects. - **Fix:** add one helper that cuts on code points, for example one that drops the last kept unit when it is a high surrogate (`0xd800`–`0xdbff`), and use it at all four places. Counting code points also matches the core's `max_length`, which counts them, so a field the extension keeps is never longer than the core accepts. ## Done when - `test/parse.test.mjs` has a case where `tidy()` cuts a title of 499 characters plus an emoji, and one where `slimPage()` cuts a page whose `MAX_PAGE` boundary falls inside a surrogate pair, both asserting that the result contains no lone surrogate (`/[\ud800-\udbff](?![\udc00-\udfff])/` does not match).

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:19 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:47:19 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 15:20:38 +00:00

Cut text on whole characters, never through half of an emoji

tiagoagueda closed this issue

2026-10-04 15:20:38 +00:00

tiagoagueda referenced this issue