Capturing a page: without JSON-LD the whole HTML is parsed seven or eight times, and once more for its places #591

New issue

Closed

opened 2026-10-01 10:48:07 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:07 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the built-in sources and timing registry.parse_page on a generated 1.65 MB page.

Checked again on 2026-10-01 against main at 0a5cad587: still present. plugins/builtin/ has not changed; the API's _read now reads the page at api/api.py:618.

Every helper the built-in sources use takes the page as a string and builds its own tree with parse_html, a pure-Python HTMLParser walk. A page that carries JSON-LD is parsed once, but a page without it, which is every LinkedIn page and every page the fallback reads, is parsed seven times by parse_page, eight on a board with a recipe, and once more when its places are listed. On a large page each parse costs over half a second of CPU inside a synchronous gunicorn worker, and the browser extension sends the same results page forty times in one gesture.

Where:
- plugins/builtin/__init__.py:355-366 SchemaOrgSource.parse: extract_jsonld(html), then extract_microdata(html) and extract_rdfa(html), each a parse.
- plugins/builtin/__init__.py:529-569 PageMetadataSource._read: extract_meta(html), heading_title(html), main_text(html) and page_language(html), each a parse.
- plugins/builtin/__init__.py:410-433 BoardSource.parse: its own parse_html(html), then _fill_from(SchemaOrgSource, …) and _fill_from(PageMetadataSource, …) with the string, which repeat all of the above.
- plugins/builtin/hints.py:432 fill and hints.py:549 places: one parse each.
- plugins/builtin/htmlutil.py:333 main_text, :371 heading_title, :423 extract_jsonld, :510 extract_microdata, :534 extract_rdfa, :552 extract_meta, :571 page_language: where each parse happens. language_of(root) at :574 already takes a tree.
- Callers: api/api.py:616 (_read, synchronous in POST /captures and /captures/preview), jobs/slow.py:64, and jobs/remembered.py:276, :311, :330 (registry.page_places).
What happens: measured with the snapshot's code on a 1.65 MB page of 40 cards, one parse_html took 0.61 s. registry.parse_page took 5.58 s with 7 parses when the page had no JSON-LD (schema.org's three, then page-metadata's four), 6.48 s with 8 parses on a linkedin.com address (the board's own parse, then the same seven), and 0.62 s with 2 parses when it had JSON-LD. registry.page_places added 1.21 s with one more. Each remembered place handed to a source adds one more through hints.fill. The extension's batch capture (#177) stores the results page once and sends it whole with each posting (postulo-chromium/src/lib/store.js:76-101, src/lib/outbox.js:55-60), so forty postings from a LinkedIn results page of that size cost about 40 × 7.7 s, five minutes of CPU, of which seven parses in eight are repeats.
Why it matters: POST /captures reads the page inside the request, and the image runs three synchronous gunicorn workers (docker/Dockerfile:362). The outbox sends one capture at a time, so one worker does nothing else for minutes, and a person with two browsers (or a preview and a send) takes two of three. The same repetition makes every fallback capture several times slower than it needs to be.
Fix: parse once per read. Give extract_jsonld, extract_microdata, extract_rdfa, extract_meta, heading_title and main_text variants that take an Element, as language_of does, and keep the string versions as thin wrappers for the tests that call them (tests/test_capture.py:16). Have SchemaOrgSource.parse, PageMetadataSource.parse and BoardSource.parse build the tree once and hand it on (for instance through a private _parse(url, root, html, hints)), and let hints.fill and hints.places accept the tree. The board's _fill_from should call that tree-taking method rather than parse(url, html). None of the helpers mutates the tree, which is what makes sharing it safe; a test should keep it so.

Done when

A test in tests/test_capture.py counts calls to parse_html and asserts that registry.parse_page on a page without JSON-LD, and on a board address, parses the page once, and that hints.fill with places handed to it adds none.
The existing source and remembered-place tests pass unchanged.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the built-in sources and timing `registry.parse_page` on a generated 1.65 MB page. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `plugins/builtin/` has not changed; the API's `_read` now reads the page at `api/api.py:618`. Every helper the built-in sources use takes the page as a string and builds its own tree with `parse_html`, a pure-Python `HTMLParser` walk. A page that carries JSON-LD is parsed once, but a page without it, which is every LinkedIn page and every page the fallback reads, is parsed seven times by `parse_page`, eight on a board with a recipe, and once more when its places are listed. On a large page each parse costs over half a second of CPU inside a synchronous gunicorn worker, and the browser extension sends the same results page forty times in one gesture. - **Where:** - `plugins/builtin/__init__.py:355-366` `SchemaOrgSource.parse`: `extract_jsonld(html)`, then `extract_microdata(html)` and `extract_rdfa(html)`, each a parse. - `plugins/builtin/__init__.py:529-569` `PageMetadataSource._read`: `extract_meta(html)`, `heading_title(html)`, `main_text(html)` and `page_language(html)`, each a parse. - `plugins/builtin/__init__.py:410-433` `BoardSource.parse`: its own `parse_html(html)`, then `_fill_from(SchemaOrgSource, …)` and `_fill_from(PageMetadataSource, …)` with the string, which repeat all of the above. - `plugins/builtin/hints.py:432` `fill` and `hints.py:549` `places`: one parse each. - `plugins/builtin/htmlutil.py:333` `main_text`, `:371` `heading_title`, `:423` `extract_jsonld`, `:510` `extract_microdata`, `:534` `extract_rdfa`, `:552` `extract_meta`, `:571` `page_language`: where each parse happens. `language_of(root)` at `:574` already takes a tree. - Callers: `api/api.py:616` (`_read`, synchronous in `POST /captures` and `/captures/preview`), `jobs/slow.py:64`, and `jobs/remembered.py:276`, `:311`, `:330` (`registry.page_places`). - **What happens:** measured with the snapshot's code on a 1.65 MB page of 40 cards, one `parse_html` took 0.61 s. `registry.parse_page` took 5.58 s with 7 parses when the page had no JSON-LD (schema.org's three, then page-metadata's four), 6.48 s with 8 parses on a `linkedin.com` address (the board's own parse, then the same seven), and 0.62 s with 2 parses when it had JSON-LD. `registry.page_places` added 1.21 s with one more. Each remembered place handed to a source adds one more through `hints.fill`. The extension's batch capture (#177) stores the results page once and sends it whole with each posting (`postulo-chromium/src/lib/store.js:76-101`, `src/lib/outbox.js:55-60`), so forty postings from a LinkedIn results page of that size cost about 40 × 7.7 s, five minutes of CPU, of which seven parses in eight are repeats. - **Why it matters:** `POST /captures` reads the page inside the request, and the image runs three synchronous gunicorn workers (`docker/Dockerfile:362`). The outbox sends one capture at a time, so one worker does nothing else for minutes, and a person with two browsers (or a preview and a send) takes two of three. The same repetition makes every fallback capture several times slower than it needs to be. - **Fix:** parse once per read. Give `extract_jsonld`, `extract_microdata`, `extract_rdfa`, `extract_meta`, `heading_title` and `main_text` variants that take an `Element`, as `language_of` does, and keep the string versions as thin wrappers for the tests that call them (`tests/test_capture.py:16`). Have `SchemaOrgSource.parse`, `PageMetadataSource.parse` and `BoardSource.parse` build the tree once and hand it on (for instance through a private `_parse(url, root, html, hints)`), and let `hints.fill` and `hints.places` accept the tree. The board's `_fill_from` should call that tree-taking method rather than `parse(url, html)`. None of the helpers mutates the tree, which is what makes sharing it safe; a test should keep it so. ## Done when - A test in `tests/test_capture.py` counts calls to `parse_html` and asserts that `registry.parse_page` on a page without JSON-LD, and on a board address, parses the page once, and that `hints.fill` with places handed to it adds none. - The existing source and remembered-place tests pass unchanged.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:07 +00:00

tiagoagueda added the

enhancement

label

2026-10-01 10:48:07 +00:00

tiagoagueda referenced this issue from a commit

2026-10-06 06:21:08 +00:00

Parse a captured page once, however many sources read it

tiagoagueda commented

2026-10-06 09:32:16 +00:00

Author