Reading a message: a From or Subject sent as raw UTF-8 loses the sender and garbles the subject #21

New issue

Closed

opened 2026-10-01 10:48:00 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:00 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running parse on messages whose headers carry unencoded bytes outside ASCII.

parse reads headers with the standard library's default (compat32) policy. For a header that contains a byte outside ASCII which is not wrapped in an RFC 2047 encoded word, that policy returns an email.header.Header object, not a str. email.utils.parseaddr answers ('', '') for anything that is not a string, so the sender and the sender's name come out empty, and _decoded turns the same kind of Subject into replacement characters instead of decoding it. Internationalised mail (RFC 6532) and a good many in-house systems send headers this way.

Where:
- src/postulo_imap/mailbox.py:135-136, parse: message = email.message_from_bytes(raw) and email.utils.parseaddr(message.get("From", "")).
- src/postulo_imap/mailbox.py:83-90, _decoded: str(make_header(decode_header(value))) on a Header whose charset is unknown yields U+FFFD for every such byte.
- src/postulo_imap/mailbox.py:150-153: sender, sender_name and subject are built from those two.
What happens: run on the snapshot with From: Müller GmbH <jobs@mueller.test> and Subject: Ihre Bewerbung bei Müller, both as raw UTF-8:
- mail.sender is '', mail.sender_name is '', mail.domain is '', and mail.subject is 'Ihre Bewerbung bei M��ller'. With the same headers in Latin-1 the result is the same with one replacement character.
- type(message.get("From")).__name__ is Header; parseaddr of it is ('', '').
- With an empty sender, neither the contact step nor the domain step of match runs (src/postulo_imap/matching.py:105 and :117-118), so the two strongest signals after the thread are lost. The company name, now M��ller, can no longer be found in the subject either, which is half of what the last step searches (mailbox.py:79-80).
- The suggestion shows From: <> and stores the garbled subject as its summary.
- A non-ASCII byte in the address-only form (From: jobs@mueller.test, raw UTF-8 subject) keeps the sender and still garbles the subject. Headers written as RFC 2047 encoded words are read correctly; tests/test_imap.py:242-245 covers only that form.
Why it matters: the message is still classified from its body, so a suggestion is filed, but it arrives unmatched or matched on weaker evidence, with a subject the person cannot read and no sender to recognise it by. Nothing says why. The same Header object is what makes parsedate_to_datetime raise on a Date header with such a byte, which is reported separately in this review; one way of reading headers fixes both.
Fix: read every header through one helper that always returns text. The raw value is available from message.raw_items() as a string in which each undecodable byte is a surrogate escape: encode it back with "ascii", "surrogateescape", decode the bytes as UTF-8 and fall back to Latin-1, then hand the result to parseaddr, _decoded and parsedate_to_datetime. Tried on the snapshot: this gives ('Müller GmbH', 'jobs@mueller.test') and the right subject for both encodings. Switching to policy=email.policy.default on its own is not enough: message["From"].addresses[0].display_name came back as 'M\udcc3\udcbcller GmbH', lone surrogates that cannot be encoded as UTF-8, so storing or rendering the suggestion would fail (reasoned, not run), which trades a garbled name for a failed pass.

Done when

tests/test_imap.py has a test in which parse of a message with From: Müller GmbH <jobs@mueller.test> and Subject: Ihre Bewerbung bei Müller, encoded once as UTF-8 and once as Latin-1 with no encoded words, returns sender == "jobs@mueller.test", sender_name == "Müller GmbH" and the subject unchanged, and no string of the Mail contains U+FFFD or a surrogate. It fails today on all three fields.

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, running `parse` on messages whose headers carry unencoded bytes outside ASCII. `parse` reads headers with the standard library's default (compat32) policy. For a header that contains a byte outside ASCII which is not wrapped in an RFC 2047 encoded word, that policy returns an `email.header.Header` object, not a `str`. `email.utils.parseaddr` answers `('', '')` for anything that is not a string, so the sender and the sender's name come out empty, and `_decoded` turns the same kind of `Subject` into replacement characters instead of decoding it. Internationalised mail (RFC 6532) and a good many in-house systems send headers this way. - **Where:** - `src/postulo_imap/mailbox.py:135-136`, `parse`: `message = email.message_from_bytes(raw)` and `email.utils.parseaddr(message.get("From", ""))`. - `src/postulo_imap/mailbox.py:83-90`, `_decoded`: `str(make_header(decode_header(value)))` on a `Header` whose charset is unknown yields U+FFFD for every such byte. - `src/postulo_imap/mailbox.py:150-153`: `sender`, `sender_name` and `subject` are built from those two. - **What happens:** run on the snapshot with `From: Müller GmbH <jobs@mueller.test>` and `Subject: Ihre Bewerbung bei Müller`, both as raw UTF-8: - `mail.sender` is `''`, `mail.sender_name` is `''`, `mail.domain` is `''`, and `mail.subject` is `'Ihre Bewerbung bei M��ller'`. With the same headers in Latin-1 the result is the same with one replacement character. - `type(message.get("From")).__name__` is `Header`; `parseaddr` of it is `('', '')`. - With an empty sender, neither the contact step nor the domain step of `match` runs (`src/postulo_imap/matching.py:105` and `:117-118`), so the two strongest signals after the thread are lost. The company name, now `M��ller`, can no longer be found in the subject either, which is half of what the last step searches (`mailbox.py:79-80`). - The suggestion shows *From: <>* and stores the garbled subject as its summary. - A non-ASCII byte in the address-only form (`From: jobs@mueller.test`, raw UTF-8 subject) keeps the sender and still garbles the subject. Headers written as RFC 2047 encoded words are read correctly; `tests/test_imap.py:242-245` covers only that form. - **Why it matters:** the message is still classified from its body, so a suggestion is filed, but it arrives unmatched or matched on weaker evidence, with a subject the person cannot read and no sender to recognise it by. Nothing says why. The same `Header` object is what makes `parsedate_to_datetime` raise on a `Date` header with such a byte, which is reported separately in this review; one way of reading headers fixes both. - **Fix:** read every header through one helper that always returns text. The raw value is available from `message.raw_items()` as a string in which each undecodable byte is a surrogate escape: encode it back with `"ascii", "surrogateescape"`, decode the bytes as UTF-8 and fall back to Latin-1, then hand the result to `parseaddr`, `_decoded` and `parsedate_to_datetime`. Tried on the snapshot: this gives `('Müller GmbH', 'jobs@mueller.test')` and the right subject for both encodings. Switching to `policy=email.policy.default` on its own is not enough: `message["From"].addresses[0].display_name` came back as `'M\udcc3\udcbcller GmbH'`, lone surrogates that cannot be encoded as UTF-8, so storing or rendering the suggestion would fail (reasoned, not run), which trades a garbled name for a failed pass. ## Done when - `tests/test_imap.py` has a test in which `parse` of a message with `From: Müller GmbH <jobs@mueller.test>` and `Subject: Ihre Bewerbung bei Müller`, encoded once as UTF-8 and once as Latin-1 with no encoded words, returns `sender == "jobs@mueller.test"`, `sender_name == "Müller GmbH"` and the subject unchanged, and no string of the `Mail` contains U+FFFD or a surrogate. It fails today on all three fields.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:00 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:00 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:00:13 +00:00

Fix the review findings: reading, classifying, matching, the form and what a refusal does

tiagoagueda referenced this issue

2026-10-19 07:00:17 +00:00

Fix the open bug and security issues (#8-#22, #24) #25