schema.org source: one over-long field or a NaN or infinite salary throws away the whole posting #592

New issue

Closed

opened 2026-10-01 10:48:07 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:07 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the schema.org source and feeding it out-of-range JSON-LD values.

Checked again on 2026-10-01 against main at 0a5cad587: still present. plugins/builtin/ and plugins/base.py:44-46 have not changed; the salary and currency fields the review form checks against are now at jobs/models.py:900-912.

The schema.org source copies what a posting states into JobPostingData without the bounds that model enforces, and _decimal lets a non-finite number through to a comparison that raises. One bad value therefore makes the source raise, parse_page logs and skips it, and the capture falls back to page metadata, losing every field the posting stated correctly. The page-metadata source already cuts its fields to 500 characters; this one does not. Its currency is cut to three characters but never checked, so a misspelt one reaches the review form, where the person has to correct it.

Where:
- plugins/builtin/__init__.py:313-328 _from_posting: passes title, _text(posting.get("hiringOrganization")) and location unbounded, while plugins/base.py:44-46 declares each with max_length=MAX_FIELD_CHARS (500), which raises rather than cuts.
- plugins/builtin/__init__.py:130-137 _decimal: Decimal("NaN") parses, then amount > 0, outside the try, raises InvalidOperation. Infinity (and a JSON 1e400, which json.loads reads as infinity) passes the comparison and is refused by pydantic as not finite. Nothing applies patterns.LARGEST (patterns.py:158), which the text reader uses (patterns.py:524).
- plugins/builtin/__init__.py:195 _one_salary: str(...).strip()[:3] with no check against CURRENCY_CODES, unlike patterns.py:463-472.
- The board recipes have the same gap: boards/linkedin.py:50-59 and boards/greenhouse.py:47-63 return text_of(...) unbounded, and BoardSource.parse (__init__.py:434) builds JobPostingData from it.
What happens: checked with the snapshot's code. A JSON-LD JobPosting whose title, company name or jobLocation address runs past 500 characters raises ValidationError ("String should have at most 500 characters"); a value of "NaN" or NaN raises InvalidOperation; "Infinity" or 1e400 raises ValidationError ("Input should be a finite number"). In each case registry.parse_page answered with page-metadata, and the location Lisbon, PT and the posted date the JSON-LD stated came back empty. A currency of "US Dollar" came back as US and "$" as $; a value of 1e30 came back as 1E+30. Those three pass the capture and fail the review form, against currency_code and max_digits=12 (jobs/models.py:897-909).
Why it matters: the schema.org source exists so that a board's own statement wins over guessing. A single field the board wrote carelessly, such as a location string listing every office or a templated salary that rendered NaN, should cost that field, not the posting. On a board page with a recipe, a long recipe field fails the board source the same way.
Fix: in _from_posting, cut title, company_name and location to MAX_FIELD_CHARS as PageMetadataSource._read does (__init__.py:558-562), and do the same in BoardSource.parse before building JobPostingData. In _decimal, return None unless amount.is_finite() and 0 < amount < patterns.LARGEST, with the comparison inside the try. In _one_salary, keep the stated currency only when it is an ISO code in CURRENCY_CODES or one of patterns.SYMBOLS (through patterns._code, made public); otherwise leave it empty, as the text reader does for a bare $. Sharing patterns' currency check rather than copying it keeps the two readers from drifting.

Done when

Tests in tests/test_capture.py feed SchemaOrgSource.parse a JSON-LD posting with a 600-character title, with a NaN, an Infinity and a 1e30 salary, and with currencies "US Dollar" and "$", and assert that each returns a posting with the other fields intact, the long field cut to 500 characters, no salary for the three bad amounts, and an empty currency for both. They fail today.
A test in the same file gives BoardSource a LinkedIn page whose location element holds over 500 characters and asserts that the board source still answers.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the schema.org source and feeding it out-of-range JSON-LD values. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `plugins/builtin/` and `plugins/base.py:44-46` have not changed; the salary and currency fields the review form checks against are now at `jobs/models.py:900-912`. The schema.org source copies what a posting states into `JobPostingData` without the bounds that model enforces, and `_decimal` lets a non-finite number through to a comparison that raises. One bad value therefore makes the source raise, `parse_page` logs and skips it, and the capture falls back to page metadata, losing every field the posting stated correctly. The page-metadata source already cuts its fields to 500 characters; this one does not. Its currency is cut to three characters but never checked, so a misspelt one reaches the review form, where the person has to correct it. - **Where:** - `plugins/builtin/__init__.py:313-328` `_from_posting`: passes `title`, `_text(posting.get("hiringOrganization"))` and `location` unbounded, while `plugins/base.py:44-46` declares each with `max_length=MAX_FIELD_CHARS` (500), which raises rather than cuts. - `plugins/builtin/__init__.py:130-137` `_decimal`: `Decimal("NaN")` parses, then `amount > 0`, outside the `try`, raises `InvalidOperation`. `Infinity` (and a JSON `1e400`, which `json.loads` reads as infinity) passes the comparison and is refused by pydantic as not finite. Nothing applies `patterns.LARGEST` (`patterns.py:158`), which the text reader uses (`patterns.py:524`). - `plugins/builtin/__init__.py:195` `_one_salary`: `str(...).strip()[:3]` with no check against `CURRENCY_CODES`, unlike `patterns.py:463-472`. - The board recipes have the same gap: `boards/linkedin.py:50-59` and `boards/greenhouse.py:47-63` return `text_of(...)` unbounded, and `BoardSource.parse` (`__init__.py:434`) builds `JobPostingData` from it. - **What happens:** checked with the snapshot's code. A JSON-LD `JobPosting` whose title, company name or `jobLocation` address runs past 500 characters raises `ValidationError` ("String should have at most 500 characters"); a `value` of `"NaN"` or `NaN` raises `InvalidOperation`; `"Infinity"` or `1e400` raises `ValidationError` ("Input should be a finite number"). In each case `registry.parse_page` answered with `page-metadata`, and the location `Lisbon, PT` and the posted date the JSON-LD stated came back empty. A currency of `"US Dollar"` came back as `US` and `"$"` as `$`; a value of `1e30` came back as `1E+30`. Those three pass the capture and fail the review form, against `currency_code` and `max_digits=12` (`jobs/models.py:897-909`). - **Why it matters:** the schema.org source exists so that a board's own statement wins over guessing. A single field the board wrote carelessly, such as a location string listing every office or a templated salary that rendered `NaN`, should cost that field, not the posting. On a board page with a recipe, a long recipe field fails the board source the same way. - **Fix:** in `_from_posting`, cut `title`, `company_name` and `location` to `MAX_FIELD_CHARS` as `PageMetadataSource._read` does (`__init__.py:558-562`), and do the same in `BoardSource.parse` before building `JobPostingData`. In `_decimal`, return `None` unless `amount.is_finite()` and `0 < amount < patterns.LARGEST`, with the comparison inside the `try`. In `_one_salary`, keep the stated currency only when it is an ISO code in `CURRENCY_CODES` or one of `patterns.SYMBOLS` (through `patterns._code`, made public); otherwise leave it empty, as the text reader does for a bare `$`. Sharing `patterns`' currency check rather than copying it keeps the two readers from drifting. ## Done when - Tests in `tests/test_capture.py` feed `SchemaOrgSource.parse` a JSON-LD posting with a 600-character title, with a `NaN`, an `Infinity` and a `1e30` salary, and with currencies `"US Dollar"` and `"$"`, and assert that each returns a posting with the other fields intact, the long field cut to 500 characters, no salary for the three bad amounts, and an empty currency for both. They fail today. - A test in the same file gives `BoardSource` a LinkedIn page whose location element holds over 500 characters and asserts that the board source still answers.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:07 +00:00

tiagoagueda added the

tier

bug

labels

2026-10-01 10:48:07 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 05:28:45 +00:00

Keep one careless posting field from costing the whole posting

tiagoagueda closed this issue

2026-10-04 05:28:45 +00:00