Classifying: a line wrap, an HTML entity, an inline tag or a typographic apostrophe hides a phrase #17

New issue

Closed

opened 2026-10-01 10:47:59 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:47:59 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running messages through parse and then classify(mail.haystack).

The rule patterns, and the company and role search in the matching, are written with single spaces, ASCII apostrophes and accented letters. The text they run against is never normalised: it keeps the line breaks of a wrapped body, the doubled spaces left where tags were, undecoded HTML entities, non-breaking spaces and typographic apostrophes. A message that misses for one of these reasons is counted as skipped and, in the default mode, flagged $Postulo, so it is never looked at again and nothing tells the person.

Where:
- src/postulo_imap/mailbox.py:93-111, plain_text: keeps line breaks; \n{3,} at line 110 does not see CRLF blank lines either.
- src/postulo_imap/mailbox.py:128-131, _strip_html: replaces each tag with a space, decodes no entity, and keeps the text of <head> and <title>.
- src/postulo_imap/mailbox.py:78-80, Mail.haystack: f"{self.subject}\n{self.text}".lower() and nothing more.
- src/postulo_imap/classifying.py:118-125, classify: searches the text as given.
- src/postulo_imap/matching.py:146-150, _mentions: the same for a company name or a role title of two words.
- src/postulo_imap/sync.py:183-186 and src/postulo_imap/sync.py:214-215: no verdict means skipped, and skipped is flagged all the same.
What happens: run on the snapshot; each is a whole message built with EmailMessage or as raw bytes:
- Plain text wrapped at the column, "…we regret to⏎inform you…" (LF or CRLF, format=flowed included) → None.
- <p>We <b>regret</b> to inform you</p> → text 'We regret to inform you' → None.
- We regret to inform you, and the same with U+00A0 characters → None.
- n'a pas été retenue → None.
- "Votre candidature n’a pas été retenue." with U+2019 → None; the same sentence with an ASCII apostrophe → rejected (fr).
- <div>Thank you for</div><div>your application</div> → None.
- An acknowledgement whose <head> holds <title>Job offer newsletter</title> → offer (en): the title's text is part of the haystack.
- _mentions("we at black\nmesa are", "Black Mesa") and the same with two spaces → False.
Why it matters: these are ordinary messages, not edge cases: recruiting systems mostly send HTML only, with entity-encoded accents, non-breaking spaces and inline markup; plain-text mail is wrapped at 72 to 78 columns; French mail composed in a word processor uses U+2019. Each one is a rejection, an invitation or an acknowledgement that never becomes a suggestion, while the connection reports a clean run. The tests only use single-line bodies and one HTML paragraph with no markup inside the sentence (tests/test_imap.py:242-245).
Fix: build the text the rules see separately from the excerpt that is stored. For the haystack: decode entities (html.unescape, or an html.parser.HTMLParser with convert_charrefs), drop head together with script and style, apply NFKC, map U+2019 and U+02BC to ', and collapse every run of whitespace, CR and LF included, to one space. Keep line breaks only in the excerpt. _mentions then runs on the same normalised text, with the needle normalised the same way.

Done when

tests/test_imap.py has a parametrised test with one message per case above (wrapped plain text with LF and with CRLF, inline bold,  , U+00A0, numeric and named entities, U+2019, a phrase split across two divs) that asserts the kind the unbroken sentence gives; every case fails today.
A test that an acknowledgement with offer words in its <title> is acknowledged, and one that match finds a two-word company name split by a line break in the body. Both fail today.

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running messages through `parse` and then `classify(mail.haystack)`. The rule patterns, and the company and role search in the matching, are written with single spaces, ASCII apostrophes and accented letters. The text they run against is never normalised: it keeps the line breaks of a wrapped body, the doubled spaces left where tags were, undecoded HTML entities, non-breaking spaces and typographic apostrophes. A message that misses for one of these reasons is counted as skipped and, in the default mode, flagged `$Postulo`, so it is never looked at again and nothing tells the person. - **Where:** - `src/postulo_imap/mailbox.py:93-111`, `plain_text`: keeps line breaks; `\n{3,}` at line 110 does not see CRLF blank lines either. - `src/postulo_imap/mailbox.py:128-131`, `_strip_html`: replaces each tag with a space, decodes no entity, and keeps the text of `<head>` and `<title>`. - `src/postulo_imap/mailbox.py:78-80`, `Mail.haystack`: `f"{self.subject}\n{self.text}".lower()` and nothing more. - `src/postulo_imap/classifying.py:118-125`, `classify`: searches the text as given. - `src/postulo_imap/matching.py:146-150`, `_mentions`: the same for a company name or a role title of two words. - `src/postulo_imap/sync.py:183-186` and `src/postulo_imap/sync.py:214-215`: no verdict means skipped, and skipped is flagged all the same. - **What happens:** run on the snapshot; each is a whole message built with `EmailMessage` or as raw bytes: - Plain text wrapped at the column, "…we regret to⏎inform you…" (LF or CRLF, `format=flowed` included) → `None`. - `<p>We <b>regret</b> to inform you</p>` → text `'We regret to inform you'` → `None`. - `We regret to inform you`, and the same with U+00A0 characters → `None`. - `n'a pas été retenue` → `None`. - "Votre candidature n’a pas été retenue." with U+2019 → `None`; the same sentence with an ASCII apostrophe → `rejected` (fr). - `<div>Thank you for</div><div>your application</div>` → `None`. - An acknowledgement whose `<head>` holds `<title>Job offer newsletter</title>` → `offer` (en): the title's text is part of the haystack. - `_mentions("we at black\nmesa are", "Black Mesa")` and the same with two spaces → `False`. - **Why it matters:** these are ordinary messages, not edge cases: recruiting systems mostly send HTML only, with entity-encoded accents, non-breaking spaces and inline markup; plain-text mail is wrapped at 72 to 78 columns; French mail composed in a word processor uses U+2019. Each one is a rejection, an invitation or an acknowledgement that never becomes a suggestion, while the connection reports a clean run. The tests only use single-line bodies and one HTML paragraph with no markup inside the sentence (`tests/test_imap.py:242-245`). - **Fix:** build the text the rules see separately from the excerpt that is stored. For the haystack: decode entities (`html.unescape`, or an `html.parser.HTMLParser` with `convert_charrefs`), drop `head` together with `script` and `style`, apply NFKC, map U+2019 and U+02BC to `'`, and collapse every run of whitespace, CR and LF included, to one space. Keep line breaks only in the excerpt. `_mentions` then runs on the same normalised text, with the needle normalised the same way. ## Done when - `tests/test_imap.py` has a parametrised test with one message per case above (wrapped plain text with LF and with CRLF, inline bold, ` `, U+00A0, numeric and named entities, U+2019, a phrase split across two `div`s) that asserts the kind the unbroken sentence gives; every case fails today. - A test that an acknowledgement with offer words in its `<title>` is `acknowledged`, and one that `match` finds a two-word company name split by a line break in the body. Both fail today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:59 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:47:59 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:00:13 +00:00

Fix the review findings: reading, classifying, matching, the form and what a refusal does

tiagoagueda referenced this issue

2026-10-19 07:00:17 +00:00

Fix the open bug and security issues (#8-#22, #24) #25