Matching: domains are cut to their last two labels, so any .co.uk sender matches a .co.uk company #19

New issue

Closed

opened 2026-10-01 10:48:00 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:00 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running registrable and match with hosts under two-label public suffixes.

registrable() takes the last two labels of a host as the organisation. Under a two-label public suffix (co.uk, com.br, com.pl, com.au, co.jp, ac.uk, org.uk and the like) those two labels are the suffix itself, so the sender's domain and every company website under the same suffix compare equal. The result is reported as matched by domain, the second strongest signal after a recorded contact.

Where:
- src/postulo_imap/matching.py:56-60, registrable: ".".join(parts[-2:]); its docstring says the last two labels are "enough to tell one organisation from another".
- src/postulo_imap/matching.py:117-126, match, step 3: domain = registrable(mail.domain), then domain not in PUBLIC_DOMAINS, then the comparison with domain_of(application.posting.company.website).
- src/postulo_imap/matching.py:35: "yahoo.co.uk" in PUBLIC_DOMAINS. The check is made on the reduced value, which for that address is co.uk, so this entry can never match.
- tests/test_imap.py:391-394, test_a_domain_is_reduced_to_the_organisation: only blackmesa.test.
What happens: run on the snapshot with one open application at a company whose website is https://www.acme.co.uk:
- registrable("careers.acme.co.uk") → co.uk; registrable("mail.yahoo.com.br") → com.br; "co.uk" in PUBLIC_DOMAINS → False.
- Mail from recruiter@otherfirm.co.uk → matched to Acme, how domain.
- Mail from jane.doe@yahoo.co.uk → matched to Acme, how domain.
- With a second open application at another .co.uk company, mail from hr@acme.co.uk → no match: the genuine one is discarded as ambiguous (_only, src/postulo_imap/matching.py:71-76), and the weaker name search decides.
The same reduction also equates everything under a hosting suffix (domain_of("https://acme.github.io") → github.io). A website typed with non-ASCII letters is compared as typed (domain_of("https://münchen.example") → münchen.example) while a sender's domain arrives in its xn-- form, so those never match.
Why it matters: a person with one open application in the United Kingdom, Brazil, Poland (.com.pl), Australia or Japan gets every recognised message from any other organisation under that suffix, private yahoo.co.uk addresses included, attached to that application with Matched by: domain. Accepting the suggestion writes the event, and for a rejection or an offer the status, onto the wrong application. The README says "public mail providers prove nothing and are ignored" (README.md:50-52).
Fix:
- Reduce hosts with the public suffix list. If a dependency is added (tldextract, publicsuffixlist), use only its bundled snapshot: tldextract downloads the list on first use unless it is built with suffix_list_urls=(), and a plugin must not make an outbound request of its own. A short built-in table of the common two-label suffixes is the alternative if no dependency is wanted.
- Test the full sender domain, and its reduced form, against PUBLIC_DOMAINS before comparing.
- Compare hosts in one form: IDNA-encode the website's host before reducing it.

Done when

tests/test_imap.py has tests that registrable("careers.acme.co.uk") is acme.co.uk and registrable("vagas.acme.com.br") is acme.com.br; that mail from another .co.uk organisation and from a yahoo.co.uk address is not matched by domain to an application at a .co.uk company; and that with two applications at different .co.uk companies, mail from one of them is matched to it by domain. All fail today.

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running `registrable` and `match` with hosts under two-label public suffixes. `registrable()` takes the last two labels of a host as the organisation. Under a two-label public suffix (`co.uk`, `com.br`, `com.pl`, `com.au`, `co.jp`, `ac.uk`, `org.uk` and the like) those two labels are the suffix itself, so the sender's domain and every company website under the same suffix compare equal. The result is reported as matched by *domain*, the second strongest signal after a recorded contact. - **Where:** - `src/postulo_imap/matching.py:56-60`, `registrable`: `".".join(parts[-2:])`; its docstring says the last two labels are "enough to tell one organisation from another". - `src/postulo_imap/matching.py:117-126`, `match`, step 3: `domain = registrable(mail.domain)`, then `domain not in PUBLIC_DOMAINS`, then the comparison with `domain_of(application.posting.company.website)`. - `src/postulo_imap/matching.py:35`: `"yahoo.co.uk"` in `PUBLIC_DOMAINS`. The check is made on the reduced value, which for that address is `co.uk`, so this entry can never match. - `tests/test_imap.py:391-394`, `test_a_domain_is_reduced_to_the_organisation`: only `blackmesa.test`. - **What happens:** run on the snapshot with one open application at a company whose website is `https://www.acme.co.uk`: - `registrable("careers.acme.co.uk")` → `co.uk`; `registrable("mail.yahoo.com.br")` → `com.br`; `"co.uk" in PUBLIC_DOMAINS` → `False`. - Mail from `recruiter@otherfirm.co.uk` → matched to Acme, how `domain`. - Mail from `jane.doe@yahoo.co.uk` → matched to Acme, how `domain`. - With a second open application at another `.co.uk` company, mail from `hr@acme.co.uk` → no match: the genuine one is discarded as ambiguous (`_only`, `src/postulo_imap/matching.py:71-76`), and the weaker name search decides. The same reduction also equates everything under a hosting suffix (`domain_of("https://acme.github.io")` → `github.io`). A website typed with non-ASCII letters is compared as typed (`domain_of("https://münchen.example")` → `münchen.example`) while a sender's domain arrives in its `xn--` form, so those never match. - **Why it matters:** a person with one open application in the United Kingdom, Brazil, Poland (`.com.pl`), Australia or Japan gets every recognised message from any other organisation under that suffix, private `yahoo.co.uk` addresses included, attached to that application with *Matched by: domain*. Accepting the suggestion writes the event, and for a rejection or an offer the status, onto the wrong application. The README says "public mail providers prove nothing and are ignored" (`README.md:50-52`). - **Fix:** - Reduce hosts with the public suffix list. If a dependency is added (`tldextract`, `publicsuffixlist`), use only its bundled snapshot: `tldextract` downloads the list on first use unless it is built with `suffix_list_urls=()`, and a plugin must not make an outbound request of its own. A short built-in table of the common two-label suffixes is the alternative if no dependency is wanted. - Test the full sender domain, and its reduced form, against `PUBLIC_DOMAINS` before comparing. - Compare hosts in one form: IDNA-encode the website's host before reducing it. ## Done when - `tests/test_imap.py` has tests that `registrable("careers.acme.co.uk")` is `acme.co.uk` and `registrable("vagas.acme.com.br")` is `acme.com.br`; that mail from another `.co.uk` organisation and from a `yahoo.co.uk` address is not matched by domain to an application at a `.co.uk` company; and that with two applications at different `.co.uk` companies, mail from one of them is matched to it by domain. All fail today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:00 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:00 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:00:13 +00:00

Fix the review findings: reading, classifying, matching, the form and what a refusal does

tiagoagueda referenced this issue

2026-10-19 07:00:17 +00:00

Fix the open bug and security issues (#8-#22, #24) #25