Sync: mail with no Message-ID is keyed on its UID, which repeats across folders, mailboxes and servers #11

New issue

Closed

opened 2026-10-01 10:47:56 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:47:56 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, following the identifier a suggestion is keyed on and running two passes over two folders with the plugin's fake mailbox.

A suggestion is filed once per (owner, source, external_id). For a message without a Message-ID header the plugin uses uid-<number>, the message's IMAP UID. A UID is unique only inside one folder under one UIDVALIDITY value, and the key carries neither the connection, the folder nor that value. A later message that happens to have the same number is taken for the earlier one: no suggestion is filed, it is counted as skipped, and it is flagged as read.

Where:
- src/postulo_imap/mailbox.py:150, parse: message_id=_decoded(message.get("Message-ID")) or f"uid-{uid}".
- src/postulo_imap/sync.py:188-191: suggest(connection.owner, source=self.name, external_id=mail.message_id, …), with source always "imap".
- In postulo, applications/suggestions.py:56-61: Suggestion.objects.filter(owner=owner, source=source, external_id=external_id[:250]).first() returns the existing row and False, whether that suggestion is waiting, accepted or declined.
- src/postulo_imap/sync.py:211-215: created false counts as skipped, and the uid is flagged with the rest.
- UIDVALIDITY is not read anywhere in the plugin.
What happens: run on the snapshot. An acknowledgement with no Message-ID is uid 1 in Jobs; the pass files a suggestion with external_id uid-1. The connection is then pointed at another folder, where a rejection with no Message-ID is also uid 1. The second pass reports pulled == 0, skipped == 1, the only suggestion is still the acknowledgement, and the rejection carries $Postulo, so it is never looked at again. The same happens:
- between two mailbox connections of one person, since both use the source imap;
- when the person changes the folder or the server of a connection, for example after moving to another mail provider, where numbering starts again at 1;
- when the server rebuilds the folder and issues a new UIDVALIDITY.
Why it matters: a recognised message is dropped without a trace, and what the queue shows for that number is a different message. Mail without the header is uncommon, since most servers add one on submission, but web forms and scripts that hand mail straight to a delivery agent do send it, and automatic mail is what this plugin exists to read.
Fix: derive the fallback from the message, not from where it sits: for example "sha256-" + hashlib.sha256(raw).hexdigest() of the raw bytes parse already has, which is stable across passes, folders and servers and cannot be mistaken for a real identifier or for a threading reference. If only part of the message is fetched in future, hash the header block. Suggestions already filed under uid-N keep their key; that only means one such message could be suggested a second time after the change.

Done when

tests/test_imap.py has a test in which two different messages without a Message-ID have the same uid in two folders, and a pass over each files two suggestions; today the second pass files none.
A test that parse(1, raw).message_id == parse(7, raw).message_id for a message without the header, and that it differs for a different message.

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, following the identifier a suggestion is keyed on and running two passes over two folders with the plugin's fake mailbox. A suggestion is filed once per `(owner, source, external_id)`. For a message without a `Message-ID` header the plugin uses `uid-<number>`, the message's IMAP UID. A UID is unique only inside one folder under one UIDVALIDITY value, and the key carries neither the connection, the folder nor that value. A later message that happens to have the same number is taken for the earlier one: no suggestion is filed, it is counted as skipped, and it is flagged as read. - **Where:** - `src/postulo_imap/mailbox.py:150`, `parse`: `message_id=_decoded(message.get("Message-ID")) or f"uid-{uid}"`. - `src/postulo_imap/sync.py:188-191`: `suggest(connection.owner, source=self.name, external_id=mail.message_id, …)`, with `source` always `"imap"`. - In postulo, `applications/suggestions.py:56-61`: `Suggestion.objects.filter(owner=owner, source=source, external_id=external_id[:250]).first()` returns the existing row and `False`, whether that suggestion is waiting, accepted or declined. - `src/postulo_imap/sync.py:211-215`: `created` false counts as skipped, and the uid is flagged with the rest. - UIDVALIDITY is not read anywhere in the plugin. - **What happens:** run on the snapshot. An acknowledgement with no `Message-ID` is uid 1 in *Jobs*; the pass files a suggestion with `external_id` `uid-1`. The connection is then pointed at another folder, where a rejection with no `Message-ID` is also uid 1. The second pass reports `pulled == 0, skipped == 1`, the only suggestion is still the acknowledgement, and the rejection carries `$Postulo`, so it is never looked at again. The same happens: - between two mailbox connections of one person, since both use the source `imap`; - when the person changes the folder or the server of a connection, for example after moving to another mail provider, where numbering starts again at 1; - when the server rebuilds the folder and issues a new UIDVALIDITY. - **Why it matters:** a recognised message is dropped without a trace, and what the queue shows for that number is a different message. Mail without the header is uncommon, since most servers add one on submission, but web forms and scripts that hand mail straight to a delivery agent do send it, and automatic mail is what this plugin exists to read. - **Fix:** derive the fallback from the message, not from where it sits: for example `"sha256-" + hashlib.sha256(raw).hexdigest()` of the raw bytes `parse` already has, which is stable across passes, folders and servers and cannot be mistaken for a real identifier or for a threading reference. If only part of the message is fetched in future, hash the header block. Suggestions already filed under `uid-N` keep their key; that only means one such message could be suggested a second time after the change. ## Done when - `tests/test_imap.py` has a test in which two different messages without a `Message-ID` have the same uid in two folders, and a pass over each files two suggestions; today the second pass files none. - A test that `parse(1, raw).message_id == parse(7, raw).message_id` for a message without the header, and that it differs for a different message.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:56 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:47:56 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:00:13 +00:00

Fix the review findings: reading, classifying, matching, the form and what a refusal does

tiagoagueda referenced this issue

2026-10-19 07:00:17 +00:00

Fix the open bug and security issues (#8-#22, #24) #25