Captures: when a waiting capture's kept page expires, what its review would learn from goes with it #524

New issue

Closed

opened 2026-10-01 10:47:23 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:23 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how a capture's kept page is removed.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at jobs/pages.py:800-821 (expire_unconfirmed, the loop at 818-819) and jobs/pages.py:692-699 (forget); jobs/remembered.py keeps its line numbers and now reads the kept source through jobs/kept.py, where #248 moved read_source.

Where a capture's source was kept, after_capture stores no page places on the capture, because the review reads them from the source. pages.forget therefore calls remembered.before_the_source_goes(page) before a source is deleted, so a capture still waiting keeps what its review learns from (#267). The scheduled expiry deletes the same rows with a bare page.delete() and skips that step, although it expires pending captures on purpose. A capture reviewed after its page expired teaches nothing about its site.

Where: jobs/pages.py:828-849 (expire_unconfirmed, the loop at 846-847); jobs/pages.py:720-727 (forget, the only caller of before_the_source_goes); jobs/remembered.py:270 (after_capture: wants_places = bool(host) and not (page is not None and page.source)); jobs/remembered.py:296-316 (before_the_source_goes); jobs/remembered.py:319-331 (_records).
What happens: POSTULO_CAPTURE_PAGE_KEEP_DAYS is 30 by default (config/settings/base.py:600). A person keeping page sources captures a page from a site Postulo reads poorly and leaves the capture waiting. On day 31 send_due_reminders calls expire_unconfirmed, which deletes the CapturedPage row and, through jobs.signals, its file. The person then reviews the capture and corrects the company name. _records finds no kept page and no learning["places"], so _learn gets an empty list and registry.learned_place finds nothing: no FieldHint is stored. The same capture reviewed a day earlier, or whose page the person had thrown away by hand, would have taught the site.
Why it matters: the two ways a kept source goes behave differently, and the one that runs unattended is the one that loses the lesson. Long-waiting captures are exactly the backlog a person reviews in bulk later.
Fix: in expire_unconfirmed, call remembered.before_the_source_goes(page) before page.delete() for each page that has a source (it already returns early for a capture that is not PENDING), or route expiry through pages.forget(page). Use select_related("capture") on stale so the capture is not fetched once per page.

Done when

A test in tests/test_remembered_on_review.py, beside test_a_kept_source_deleted_before_review_leaves_the_lesson_behind, that keeps sources, captures a page, ages the capture past POSTULO_CAPTURE_PAGE_KEEP_DAYS, runs pages.expire_unconfirmed(), and asserts that learning["places"] is set and that a review correcting the company name then remembers its place. It fails today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how a capture's kept page is removed. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `jobs/pages.py:800-821` (`expire_unconfirmed`, the loop at 818-819) and `jobs/pages.py:692-699` (`forget`); `jobs/remembered.py` keeps its line numbers and now reads the kept source through `jobs/kept.py`, where #248 moved `read_source`. Where a capture's source was kept, `after_capture` stores no page places on the capture, because the review reads them from the source. `pages.forget` therefore calls `remembered.before_the_source_goes(page)` before a source is deleted, so a capture still waiting keeps what its review learns from (#267). The scheduled expiry deletes the same rows with a bare `page.delete()` and skips that step, although it expires pending captures on purpose. A capture reviewed after its page expired teaches nothing about its site. - **Where:** `jobs/pages.py:828-849` (`expire_unconfirmed`, the loop at 846-847); `jobs/pages.py:720-727` (`forget`, the only caller of `before_the_source_goes`); `jobs/remembered.py:270` (`after_capture`: `wants_places = bool(host) and not (page is not None and page.source)`); `jobs/remembered.py:296-316` (`before_the_source_goes`); `jobs/remembered.py:319-331` (`_records`). - **What happens:** `POSTULO_CAPTURE_PAGE_KEEP_DAYS` is 30 by default (`config/settings/base.py:600`). A person keeping page sources captures a page from a site Postulo reads poorly and leaves the capture waiting. On day 31 `send_due_reminders` calls `expire_unconfirmed`, which deletes the `CapturedPage` row and, through `jobs.signals`, its file. The person then reviews the capture and corrects the company name. `_records` finds no kept page and no `learning["places"]`, so `_learn` gets an empty list and `registry.learned_place` finds nothing: no `FieldHint` is stored. The same capture reviewed a day earlier, or whose page the person had thrown away by hand, would have taught the site. - **Why it matters:** the two ways a kept source goes behave differently, and the one that runs unattended is the one that loses the lesson. Long-waiting captures are exactly the backlog a person reviews in bulk later. - **Fix:** in `expire_unconfirmed`, call `remembered.before_the_source_goes(page)` before `page.delete()` for each page that has a source (it already returns early for a capture that is not `PENDING`), or route expiry through `pages.forget(page)`. Use `select_related("capture")` on `stale` so the capture is not fetched once per page. ## Done when - A test in `tests/test_remembered_on_review.py`, beside `test_a_kept_source_deleted_before_review_leaves_the_lesson_behind`, that keeps sources, captures a page, ages the capture past `POSTULO_CAPTURE_PAGE_KEEP_DAYS`, runs `pages.expire_unconfirmed()`, and asserts that `learning["places"]` is set and that a review correcting the company name then remembers its place. It fails today.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:23 +00:00

tiagoagueda added the

tier

bug

labels

2026-10-01 10:47:23 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:43 +00:00

A kept page follows its capture, then the listing it became, and goes when that listing goes #664

tiagoagueda referenced this issue from a commit

2026-10-04 07:56:40 +00:00