Reading a page: omitted end tags (li, p, td, dd, option) nest each element in the last, and past ~1000 every source crashes #587

New issue

Closed

opened 2026-10-01 10:48:05 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:05 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the page tree builder and running the built-in sources over pages that leave out optional end tags.

Checked again on 2026-10-01 against main at 0a5cad587: still present. plugins/builtin/ has not changed; the 422 answer is now at api/api.py:620.

_TreeBuilder opens every non-void start tag and closes an element only when its own end tag arrives. HTML lets a page leave out </li>, </p>, </dt>, </dd>, </td>, </th>, </tr> and </option>, and a browser closes them for it. Here each such element becomes a child of the one before, so a list of N items written without </li> is a chain N levels deep. Every walk over the tree is recursive, so a deep enough chain raises RecursionError in all three built-in sources and in the remembered-places reader; a shallower one still has the wrong shape, and label places such as <dt>Local<dd>Lisboa are not found.

Where: plugins/builtin/htmlutil.py:140-148 (_TreeBuilder.handle_starttag: if tag not in VOID_TAGS: self._open.append(element), with no implied end tags) and :158-162 (handle_endtag); the recursive walks Element.iter (htmlutil.py:106-111), text_of's walk (:223-235), _text_length (:282-287), _read_item (:476-493) and hints._elements (plugins/builtin/hints.py:171-180); the sibling tests in hints._next_element (:183-191) and hints._labelled (:231-240).
What happens:
1. A page whose body holds a list of about 1,000 <li> (or <option>, or <p>) items with the end tags left out, which is valid HTML, is captured. With 1,200 items, registry.parse_page logs Capture source 'schema.org' failed ... RecursionError and the same for page-metadata, and returns None, even when the page's <head> carries a valid JSON-LD JobPosting, because extract_jsonld finds the script through root.iter(). The capture answers 422 "Nothing resembling a job posting was found there." (api/api.py:618, jobs/slow.py:66). registry.page_places already fails at 990 items, so nothing is learned from the page either.
2. Below that depth the tree is still wrong. hints.places on <dl><dt>Local</dt><dd>Lisboa</dd><dt>Vencimento</dt><dd>1000</dd></dl> gives the label places local and vencimento; on the same markup with the end tags left out it gives none, because each <dd> is a child of its <dt> rather than its next sibling. A table written <tr><td>Local<td>Lisboa loses its label places the same way.
Why it matters: captures of valid pages fail outright or lose the remembered places a person's corrections taught. The module docstring (htmlutil.py:11-14) promises that the browser extension, which reads the same page through a real DOM (postulo-chromium src/lib/parse.js), walks the same tree; for these pages it does not, so a page read in the browser and the same page fetched by the server give different results.
Fix: teach _TreeBuilder the implied end tags that matter: a start tag of li closes an open li up to the nearest ul/ol; dt/dd close an open dt/dd up to the nearest dl; a block start tag (p, div, ul, ol, table, the headings and the other block tags) closes an open p; td/th close an open td/th, tr closes an open tr (and its cell), and option/optgroup close an open option. Separately, make Element.iter, text_of, _text_length, _read_item and hints._elements iterative over an explicit stack, so that no page shape can exhaust the recursion limit.

Done when

tests/test_source_corpus.py (or a new tests/test_htmlutil.py) parses <ul><li>a<li>b</ul>, <dl><dt>x<dd>y</dl>, <table><tr><td>x<td>y<tr><td>z</table> and <p>a<p>b into the sibling shape a browser builds, and asserts it.
A test in tests/test_capture.py feeds SchemaOrgSource a page with a JSON-LD posting and 5,000 unclosed <li> and gets the posting back; the same page through registry.page_places returns places instead of logging an error.
A test in tests/test_remembered_places.py finds the local label place in <dl><dt>Local<dd>Lisboa</dl>.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the page tree builder and running the built-in sources over pages that leave out optional end tags. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `plugins/builtin/` has not changed; the 422 answer is now at `api/api.py:620`. `_TreeBuilder` opens every non-void start tag and closes an element only when its own end tag arrives. HTML lets a page leave out `</li>`, `</p>`, `</dt>`, `</dd>`, `</td>`, `</th>`, `</tr>` and `</option>`, and a browser closes them for it. Here each such element becomes a child of the one before, so a list of N items written without `</li>` is a chain N levels deep. Every walk over the tree is recursive, so a deep enough chain raises `RecursionError` in all three built-in sources and in the remembered-places reader; a shallower one still has the wrong shape, and label places such as `<dt>Local<dd>Lisboa` are not found. - **Where:** `plugins/builtin/htmlutil.py:140-148` (`_TreeBuilder.handle_starttag`: `if tag not in VOID_TAGS: self._open.append(element)`, with no implied end tags) and `:158-162` (`handle_endtag`); the recursive walks `Element.iter` (`htmlutil.py:106-111`), `text_of`'s `walk` (`:223-235`), `_text_length` (`:282-287`), `_read_item` (`:476-493`) and `hints._elements` (`plugins/builtin/hints.py:171-180`); the sibling tests in `hints._next_element` (`:183-191`) and `hints._labelled` (`:231-240`). - **What happens:** 1. A page whose body holds a list of about 1,000 `<li>` (or `<option>`, or `<p>`) items with the end tags left out, which is valid HTML, is captured. With 1,200 items, `registry.parse_page` logs `Capture source 'schema.org' failed ... RecursionError` and the same for `page-metadata`, and returns `None`, even when the page's `<head>` carries a valid JSON-LD `JobPosting`, because `extract_jsonld` finds the script through `root.iter()`. The capture answers 422 "Nothing resembling a job posting was found there." (`api/api.py:618`, `jobs/slow.py:66`). `registry.page_places` already fails at 990 items, so nothing is learned from the page either. 2. Below that depth the tree is still wrong. `hints.places` on `<dl><dt>Local</dt><dd>Lisboa</dd><dt>Vencimento</dt><dd>1000</dd></dl>` gives the label places `local` and `vencimento`; on the same markup with the end tags left out it gives none, because each `<dd>` is a child of its `<dt>` rather than its next sibling. A table written `<tr><td>Local<td>Lisboa` loses its label places the same way. - **Why it matters:** captures of valid pages fail outright or lose the remembered places a person's corrections taught. The module docstring (`htmlutil.py:11-14`) promises that the browser extension, which reads the same page through a real DOM (`postulo-chromium` `src/lib/parse.js`), walks the same tree; for these pages it does not, so a page read in the browser and the same page fetched by the server give different results. - **Fix:** teach `_TreeBuilder` the implied end tags that matter: a start tag of `li` closes an open `li` up to the nearest `ul`/`ol`; `dt`/`dd` close an open `dt`/`dd` up to the nearest `dl`; a block start tag (`p`, `div`, `ul`, `ol`, `table`, the headings and the other block tags) closes an open `p`; `td`/`th` close an open `td`/`th`, `tr` closes an open `tr` (and its cell), and `option`/`optgroup` close an open `option`. Separately, make `Element.iter`, `text_of`, `_text_length`, `_read_item` and `hints._elements` iterative over an explicit stack, so that no page shape can exhaust the recursion limit. ## Done when - `tests/test_source_corpus.py` (or a new `tests/test_htmlutil.py`) parses `<ul><li>a<li>b</ul>`, `<dl><dt>x<dd>y</dl>`, `<table><tr><td>x<td>y<tr><td>z</table>` and `<p>a<p>b` into the sibling shape a browser builds, and asserts it. - A test in `tests/test_capture.py` feeds `SchemaOrgSource` a page with a JSON-LD posting and 5,000 unclosed `<li>` and gets the posting back; the same page through `registry.page_places` returns places instead of logging an error. - A test in `tests/test_remembered_places.py` finds the `local` label place in `<dl><dt>Local<dd>Lisboa</dl>`.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:05 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:05 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:01 +00:00

Close the elements a page leaves its end tags off

tiagoagueda closed this issue

2026-10-04 10:05:01 +00:00

tiagoagueda referenced this issue from a commit