Duplicate check: /captures/known misses listings beyond the 200 newest at a path, such as Indeed's /viewjob?jk= #556

New issue

Closed

opened 2026-10-01 10:47:37 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:37 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by reading jobs/known.py and running known() against 250 listings at one Indeed path.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at api/api.py:754-804 (known_captures); jobs/known.py, jobs/capture_views.py and core/addresses.py have not moved.

known() narrows in SQL with url__icontains=_needle(url), where the needle is the path alone, then decides with same_url, which keeps the query string, in Python, over at most LIMIT = 200 of the newest rows. On boards that carry the job id in the query (Indeed's /viewjob?jk=…, LinkedIn's …/jobs/search/?currentJobId=…) every listing from that board matches the needle, so once a person has more than 200 of them the older ones are never compared, and the endpoint answers that a posting already held has never been seen: the duplicate #178 was written to prevent comes back. The same shape also makes each answer read up to 200 whole listings and 200 whole captures.

Where: jobs/known.py:23-25 (LIMIT), jobs/known.py:43-46 _needle, jobs/known.py:57-78 known; callers api/api.py:752-800 known_captures (up to 100 postings per call, one known() each), jobs/capture_views.py:107 (the capture form) and jobs/capture_views.py:253-266 (the review screen); core/addresses.py:66-84 same_url.
What happens: with 250 listings at https://www.indeed.com/viewjob?jk=<id>, known() finds the newest (1 match) and nothing for any of the 50 oldest (0 matches), checked on the snapshot. For cost, the other reviewer's experiment with 300 Indeed listings (5 KB descriptions) and 150 waiting captures: a /captures/known call for 40 new postings made 133 queries and took 1.7 s, reading every listing's description and every capture's data to answer "none known". The browser extension calls it each time its popup opens on a results page.
Why it matters: the duplicate warning silently stops working for exactly the people who use a board most, and the popup gets slower as they do.
Fix: store the normalised address. Add an indexed url_key column holding same_url(url) to JobPosting and Capture, set on save, with a data migration for existing rows, and answer with filter(url_key=key) (or url_key__in=keys for the whole API call): one indexed query, no cap and no comparison in Python. Until then, at least include the query string in the needle when there is one, and load only the fields the answer shows (.only(...), or .defer("description") on listings and .defer("data", "learning") on captures).

Related to #178 (closed), which introduced the check.

Done when

A test in tests/test_known.py creates 250 listings at https://www.indeed.com/viewjob?jk=<n> and asserts that known() finds the oldest one; it fails today.
A test there asserts that a /captures/known call for 40 postings runs a number of queries independent of how many listings the account holds.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by reading `jobs/known.py` and running `known()` against 250 listings at one Indeed path. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `api/api.py:754-804` (`known_captures`); `jobs/known.py`, `jobs/capture_views.py` and `core/addresses.py` have not moved. `known()` narrows in SQL with `url__icontains=_needle(url)`, where the needle is the path alone, then decides with `same_url`, which keeps the query string, in Python, over at most `LIMIT = 200` of the newest rows. On boards that carry the job id in the query (Indeed's `/viewjob?jk=…`, LinkedIn's `…/jobs/search/?currentJobId=…`) every listing from that board matches the needle, so once a person has more than 200 of them the older ones are never compared, and the endpoint answers that a posting already held has never been seen: the duplicate #178 was written to prevent comes back. The same shape also makes each answer read up to 200 whole listings and 200 whole captures. - **Where:** `jobs/known.py:23-25` (`LIMIT`), `jobs/known.py:43-46` `_needle`, `jobs/known.py:57-78` `known`; callers `api/api.py:752-800` `known_captures` (up to 100 postings per call, one `known()` each), `jobs/capture_views.py:107` (the capture form) and `jobs/capture_views.py:253-266` (the review screen); `core/addresses.py:66-84` `same_url`. - **What happens:** with 250 listings at `https://www.indeed.com/viewjob?jk=<id>`, `known()` finds the newest (1 match) and nothing for any of the 50 oldest (0 matches), checked on the snapshot. For cost, the other reviewer's experiment with 300 Indeed listings (5 KB descriptions) and 150 waiting captures: a `/captures/known` call for 40 new postings made 133 queries and took 1.7 s, reading every listing's description and every capture's `data` to answer "none known". The browser extension calls it each time its popup opens on a results page. - **Why it matters:** the duplicate warning silently stops working for exactly the people who use a board most, and the popup gets slower as they do. - **Fix:** store the normalised address. Add an indexed `url_key` column holding `same_url(url)` to `JobPosting` and `Capture`, set on save, with a data migration for existing rows, and answer with `filter(url_key=key)` (or `url_key__in=keys` for the whole API call): one indexed query, no cap and no comparison in Python. Until then, at least include the query string in the needle when there is one, and load only the fields the answer shows (`.only(...)`, or `.defer("description")` on listings and `.defer("data", "learning")` on captures). Related to #178 (closed), which introduced the check. ## Done when - A test in `tests/test_known.py` creates 250 listings at `https://www.indeed.com/viewjob?jk=<n>` and asserts that `known()` finds the oldest one; it fails today. - A test there asserts that a `/captures/known` call for 40 postings runs a number of queries independent of how many listings the account holds.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:37 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:47:37 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:05:00 +00:00

Look addresses up by a stored key in the duplicate check

tiagoagueda closed this issue

2026-10-04 10:05:00 +00:00

tiagoagueda referenced this issue from a commit