schema.org source: a microdata or RDFa JobPosting inside a page-level item (a WebPage body, mainEntity) is never read #589

New issue

Closed

opened 2026-10-01 10:48:06 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:06 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the microdata and RDFa readers and running the schema.org source on wrapped postings.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

The microdata and RDFa readers return only the outermost items on a page, and the schema.org source looks for a JobPosting only among those. Many site themes put itemscope itemtype="https://schema.org/WebPage" on <body>, so on such a site the posting is not outermost and is never seen: it is not read as a property of the WebPage either, because it carries no itemprop. A posting attached as itemprop="mainEntity" is read, but only as a nested value that _postings never looks inside. #176 added microdata and RDFa and taught JSON-LD to follow mainEntity; the markup spellings did not get the same.

Related to #176, which added these readers and claimed mainEntity is followed; that holds for JSON-LD only.

Where: plugins/builtin/htmlutil.py:499-505 (_outermost drops every item that has an item above it), :508-524 (extract_microdata) and :527-544 (extract_rdfa); :476-493 (_read_item does not descend into a child that opens its own item, and keeps it only when it is a property); plugins/builtin/__init__.py:256-264 (_postings checks only the top-level objects' @type) and :362 (SchemaOrgSource.parse); compare htmlutil.py:396-413 (_flatten, which follows mainEntity, @graph, itemListElement and item, but only for JSON-LD). The extension does the same: postulo-chromium src/lib/parse.js:192-203 (outermost, microdata) and :212 (rdfa).
What happens:
- <body itemscope itemtype="https://schema.org/WebPage"> around the same JobPosting <div> that is read correctly under a plain <body>: extract_microdata returns [{'@type': 'WebPage'}] and SchemaOrgSource().parse returns None.
- The posting marked itemprop="mainEntity" inside that body: extract_microdata returns [{'@type': 'WebPage', 'mainEntity': {'@type': 'JobPosting', ...}}], and parse still returns None.
- RDFa <body vocab="https://schema.org/" typeof="WebPage"> around <div typeof="JobPosting">: extract_rdfa returns [{'@type': 'WebPage'}], and parse returns None.
  In each case the capture falls through to page-metadata, which loses the company, location, salary, dates and employment type the page stated in its markup.
Why it matters: the boards #176 set out to read (older recruitment systems and public-sector sites publishing only itemprop markup) are often built on themes that wrap the whole page in a WebPage item, so the structured data is there and is not used.
Fix: per the microdata specification, treat every element with itemscope and no itemprop as a top-level item whatever its ancestors are, and do the same for RDFa typeof without property. Then pass the microdata and RDFa objects through _flatten (extended to look inside nested property values such as mainEntity, hasPart and about), so a posting held inside another item is found exactly as it is in JSON-LD. Make the same change in postulo-chromium's parse.js so the two readers stay identical.

Done when

Two fixtures in tests/fixtures/postings/, a WebPage-wrapped microdata posting and a mainEntity one (plus an RDFa variant), read by tests/test_source_corpus.py with their expected title and company; today they read as None.
The same fixtures pass in postulo-chromium's test/corpus.test.mjs.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the microdata and RDFa readers and running the schema.org source on wrapped postings. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The microdata and RDFa readers return only the outermost items on a page, and the schema.org source looks for a `JobPosting` only among those. Many site themes put `itemscope itemtype="https://schema.org/WebPage"` on `<body>`, so on such a site the posting is not outermost and is never seen: it is not read as a property of the WebPage either, because it carries no `itemprop`. A posting attached as `itemprop="mainEntity"` is read, but only as a nested value that `_postings` never looks inside. #176 added microdata and RDFa and taught JSON-LD to follow `mainEntity`; the markup spellings did not get the same. Related to #176, which added these readers and claimed `mainEntity` is followed; that holds for JSON-LD only. - **Where:** `plugins/builtin/htmlutil.py:499-505` (`_outermost` drops every item that has an item above it), `:508-524` (`extract_microdata`) and `:527-544` (`extract_rdfa`); `:476-493` (`_read_item` does not descend into a child that opens its own item, and keeps it only when it is a property); `plugins/builtin/__init__.py:256-264` (`_postings` checks only the top-level objects' `@type`) and `:362` (`SchemaOrgSource.parse`); compare `htmlutil.py:396-413` (`_flatten`, which follows `mainEntity`, `@graph`, `itemListElement` and `item`, but only for JSON-LD). The extension does the same: `postulo-chromium` `src/lib/parse.js:192-203` (`outermost`, `microdata`) and `:212` (`rdfa`). - **What happens:** - `<body itemscope itemtype="https://schema.org/WebPage">` around the same `JobPosting` `<div>` that is read correctly under a plain `<body>`: `extract_microdata` returns `[{'@type': 'WebPage'}]` and `SchemaOrgSource().parse` returns `None`. - The posting marked `itemprop="mainEntity"` inside that body: `extract_microdata` returns `[{'@type': 'WebPage', 'mainEntity': {'@type': 'JobPosting', ...}}]`, and `parse` still returns `None`. - RDFa `<body vocab="https://schema.org/" typeof="WebPage">` around `<div typeof="JobPosting">`: `extract_rdfa` returns `[{'@type': 'WebPage'}]`, and `parse` returns `None`. In each case the capture falls through to `page-metadata`, which loses the company, location, salary, dates and employment type the page stated in its markup. - **Why it matters:** the boards #176 set out to read (older recruitment systems and public-sector sites publishing only `itemprop` markup) are often built on themes that wrap the whole page in a `WebPage` item, so the structured data is there and is not used. - **Fix:** per the microdata specification, treat every element with `itemscope` and no `itemprop` as a top-level item whatever its ancestors are, and do the same for RDFa `typeof` without `property`. Then pass the microdata and RDFa objects through `_flatten` (extended to look inside nested property values such as `mainEntity`, `hasPart` and `about`), so a posting held inside another item is found exactly as it is in JSON-LD. Make the same change in `postulo-chromium`'s `parse.js` so the two readers stay identical. ## Done when - Two fixtures in `tests/fixtures/postings/`, a WebPage-wrapped microdata posting and a `mainEntity` one (plus an RDFa variant), read by `tests/test_source_corpus.py` with their expected title and company; today they read as `None`. - The same fixtures pass in `postulo-chromium`'s `test/corpus.test.mjs`.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:06 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:06 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:01 +00:00

Read a microdata or RDFa posting inside a page-level item

tiagoagueda closed this issue

2026-10-04 10:05:01 +00:00