Company page: the duplicate check strips the legal form off every other company's name on each view #553

New issue

Closed

opened 2026-10-01 10:47:36 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:47:36 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by reading the duplicate check and timing it over 2,000 companies.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

duplicates.for_company loads the name and website of every other company the person has and runs bare_name() on each: a Unicode fold, then a linear scan of the 116 trailing and 14 leading legal forms, repeated until none matches. It runs on every company page view, so the page's cost is linear in the number of companies, at about 80 µs per company. The module assumes a person's companies are counted "in dozens", but captures, CSV imports and applications each create companies, and a long search reaches hundreds.

Where:
- jobs/duplicates.py:222-257 (for_company: for pk, other_name, other_website in others: with bare_name(other_name) per row, :241-243)
- jobs/duplicates.py:139-172 (bare_name: strip() walks every form in _TRAILING, then _LEADING)
- jobs/duplicates.py:17 (the "counts in dozens" assumption)
- jobs/views.py:317-320 (CompanyDetailView.get_context_data calls it on every view)
What happens: with 2,000 companies, for_company takes about 224 ms per company page; bare_name alone is about 160 ms for 2,000 plain names and 380 ms for names carrying three forms (… GmbH & Co. KG). At 300 companies it adds about 30 ms. contacts_with_any, which is dict-based, takes 24 ms for 30 contacts over 2,000.
Why it matters: the company page is opened constantly, and its cost grows with the record for a check whose answer changes only when a company is renamed.
Fix: the cheap step is to index the forms by their last (and, for leading forms, first) word, so strip() looks up only the forms that can match instead of scanning all of them; memoising bare_name with functools.lru_cache helps a long-running process as well. The complete fix stores the bare name and the website domain in indexed columns set on save, with a data migration that recomputes them when LEGAL_FORMS changes, and asks the database: filter(Q(bare_name=…) | Q(domain=…)).exclude(pk=…).

Done when

A test in tests/test_duplicates.py times bare_name over 2,000 names and asserts it stays under a generous budget (for example 20 ms), and the existing matching tests still pass; or, with stored columns, a test asserts for_company makes a fixed number of queries and no per-row Python comparison.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by reading the duplicate check and timing it over 2,000 companies. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `duplicates.for_company` loads the name and website of every other company the person has and runs `bare_name()` on each: a Unicode fold, then a linear scan of the 116 trailing and 14 leading legal forms, repeated until none matches. It runs on every company page view, so the page's cost is linear in the number of companies, at about 80 µs per company. The module assumes a person's companies are counted "in dozens", but captures, CSV imports and applications each create companies, and a long search reaches hundreds. - **Where:** - `jobs/duplicates.py:222-257` (`for_company`: `for pk, other_name, other_website in others:` with `bare_name(other_name)` per row, `:241-243`) - `jobs/duplicates.py:139-172` (`bare_name`: `strip()` walks every form in `_TRAILING`, then `_LEADING`) - `jobs/duplicates.py:17` (the "counts in dozens" assumption) - `jobs/views.py:317-320` (`CompanyDetailView.get_context_data` calls it on every view) - **What happens:** with 2,000 companies, `for_company` takes about 224 ms per company page; `bare_name` alone is about 160 ms for 2,000 plain names and 380 ms for names carrying three forms (*… GmbH & Co. KG*). At 300 companies it adds about 30 ms. `contacts_with_any`, which is dict-based, takes 24 ms for 30 contacts over 2,000. - **Why it matters:** the company page is opened constantly, and its cost grows with the record for a check whose answer changes only when a company is renamed. - **Fix:** the cheap step is to index the forms by their last (and, for leading forms, first) word, so `strip()` looks up only the forms that can match instead of scanning all of them; memoising `bare_name` with `functools.lru_cache` helps a long-running process as well. The complete fix stores the bare name and the website domain in indexed columns set on save, with a data migration that recomputes them when `LEGAL_FORMS` changes, and asks the database: `filter(Q(bare_name=…) | Q(domain=…)).exclude(pk=…)`. ## Done when - A test in `tests/test_duplicates.py` times `bare_name` over 2,000 names and asserts it stays under a generous budget (for example 20 ms), and the existing matching tests still pass; or, with stored columns, a test asserts `for_company` makes a fixed number of queries and no per-row Python comparison.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:36 +00:00

tiagoagueda added the

tier

enhancement

labels

2026-10-01 10:47:36 +00:00

tiagoagueda referenced this issue from a commit

2026-10-05 23:54:30 +00:00

Look legal forms up by their edge word and cache bare_name

tiagoagueda commented

2026-10-06 09:32:03 +00:00