Sync: every message of a pass is fetched whole, attachments included, in one command with no size limit #20

New issue

Closed

opened 2026-10-01 10:48:00 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:00 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, reading how a pass fetches its messages and measuring Mailbox.fetch and parse on a message with an attachment.

A pass asks the server for the complete source of every message the search returned, in a single FETCH, and holds all of it in memory at once in order to keep at most 2,000 characters of each. Nothing bounds a message's size or the number of messages, so the memory a pass needs is set by whatever is in the folder. In Leave the mailbox untouched mode the same download is repeated on every pass.

Where:
- src/postulo_imap/mailbox.py:195-208, Mailbox.fetch: answer = self.client.fetch(uids, ["BODY.PEEK[]"]), with uids being every result of the search (src/postulo_imap/sync.py:171-175). IMAPClient 3.1.0 joins all the ids into one command and returns one dictionary, so the whole answer exists before the first message is parsed.
- src/postulo_imap/mailbox.py:134-157, parse: email.message_from_bytes(raw) builds a second, parsed copy of each message, attachments included.
- src/postulo_imap/mailbox.py:43-45 and :111: EXCERPT_CHARS = 2000 is applied at the very end of plain_text, after everything was downloaded and parsed.
- src/postulo_imap/mailbox.py:185-188, unread_uids: without the keyword the search is SINCE <date> alone, so Leave the mailbox untouched (and Move it to a subfolder whenever the move fails) returns the whole look-back window on every pass.
- src/postulo_imap/mailbox.py:3-7 says "What is read is deliberately small", and README.md:73-74 "no attachments". That is true of what is stored, not of what is transferred and held.
What happens:
- A first pass over a Jobs folder holding thirty days of recruiter mail with PDF attachments makes the process allocate the sum of all message sizes for the answer, plus the parser's working memory per message. Measured on the snapshot: parsing one 6.75 MB message (a 5 MB attachment after base64) peaked at about 52 MB of allocations, and 52 characters of it were kept. With a stand-in client, fetch passed twenty uids to the server in one call with ['BODY.PEEK[]'].
- The work runs inline: in the scheduler's pass (plugins/syncing.py:110-137 in postulo) and in the web worker that serves Sync now (plugins/views.py:455). On a small self-hosted machine or a container with a memory limit, the process is killed. sync_connection writes synced_at only after plugin.sync returns (plugins/syncing.py:86), so a killed pass leaves the connection due and the next pass starts the same download again; sync connections that come after it in the pass do not run.
- In untouched mode the cost does not end with the first pass: every message inside the window is downloaded and parsed again as often as every 15 minutes, to find through suggest() that it was already filed.
- Nothing is processed until the single answer is complete, so a connection that drops or stalls past TIMEOUT (mailbox.py:51) anywhere in the transfer loses the entire pass, and the next one starts from the beginning.
Why it matters: the size of somebody's mail decides whether the scheduler survives, and whoever sends mail that the person's filter files into the folder chooses that size. The plugin needs the headers and the start of the text; it pays for every attachment.
Fix:
- Fetch in batches of 20 to 50 uids, and classify, suggest and flag each batch before fetching the next, so memory is bounded by a batch and a failure loses one batch.
- Ask for RFC822.SIZE together with the headers first. Above a limit (1 MB, say) fetch BODY.PEEK[HEADER] and a bounded partial of the text, BODY.PEEK[TEXT]<0.65536>, and parse the two joined. Checked on the snapshot: the headers plus the first 64 KiB of the body of the message above give parse the same subject, identifier and text as the whole message. Choosing the first text part from BODYSTRUCTURE is the more exact alternative.
- In untouched mode fetch BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)] first and skip every message whose identifier already has a suggestion for this owner and source.
- Say in the README what is transferred, once it is true.

Done when

tests/test_imap.py has a test whose fake client records each fetch call: a folder of 120 messages is never asked for in one call of more than the batch size, and a message whose RFC822.SIZE is above the limit is never asked for with BODY.PEEK[]. Both fail today (one call, all uids, BODY.PEEK[]).
tests/test_imap.py has a test in which a second pass in untouched mode asks for no message body of a message that already has a suggestion; today it fetches all of them.

Found on 2026-09-29 in a read-only review of postulo-imap at cc09c6381, reading how a pass fetches its messages and measuring `Mailbox.fetch` and `parse` on a message with an attachment. A pass asks the server for the complete source of every message the search returned, in a single `FETCH`, and holds all of it in memory at once in order to keep at most 2,000 characters of each. Nothing bounds a message's size or the number of messages, so the memory a pass needs is set by whatever is in the folder. In *Leave the mailbox untouched* mode the same download is repeated on every pass. - **Where:** - `src/postulo_imap/mailbox.py:195-208`, `Mailbox.fetch`: `answer = self.client.fetch(uids, ["BODY.PEEK[]"])`, with `uids` being every result of the search (`src/postulo_imap/sync.py:171-175`). IMAPClient 3.1.0 joins all the ids into one command and returns one dictionary, so the whole answer exists before the first message is parsed. - `src/postulo_imap/mailbox.py:134-157`, `parse`: `email.message_from_bytes(raw)` builds a second, parsed copy of each message, attachments included. - `src/postulo_imap/mailbox.py:43-45` and `:111`: `EXCERPT_CHARS = 2000` is applied at the very end of `plain_text`, after everything was downloaded and parsed. - `src/postulo_imap/mailbox.py:185-188`, `unread_uids`: without the keyword the search is `SINCE <date>` alone, so *Leave the mailbox untouched* (and *Move it to a subfolder* whenever the move fails) returns the whole look-back window on every pass. - `src/postulo_imap/mailbox.py:3-7` says "What is read is deliberately small", and `README.md:73-74` "no attachments". That is true of what is stored, not of what is transferred and held. - **What happens:** - A first pass over a *Jobs* folder holding thirty days of recruiter mail with PDF attachments makes the process allocate the sum of all message sizes for the answer, plus the parser's working memory per message. Measured on the snapshot: parsing one 6.75 MB message (a 5 MB attachment after base64) peaked at about 52 MB of allocations, and 52 characters of it were kept. With a stand-in client, `fetch` passed twenty uids to the server in one call with `['BODY.PEEK[]']`. - The work runs inline: in the scheduler's pass (`plugins/syncing.py:110-137` in postulo) and in the web worker that serves *Sync now* (`plugins/views.py:455`). On a small self-hosted machine or a container with a memory limit, the process is killed. `sync_connection` writes `synced_at` only after `plugin.sync` returns (`plugins/syncing.py:86`), so a killed pass leaves the connection due and the next pass starts the same download again; sync connections that come after it in the pass do not run. - In untouched mode the cost does not end with the first pass: every message inside the window is downloaded and parsed again as often as every 15 minutes, to find through `suggest()` that it was already filed. - Nothing is processed until the single answer is complete, so a connection that drops or stalls past `TIMEOUT` (`mailbox.py:51`) anywhere in the transfer loses the entire pass, and the next one starts from the beginning. - **Why it matters:** the size of somebody's mail decides whether the scheduler survives, and whoever sends mail that the person's filter files into the folder chooses that size. The plugin needs the headers and the start of the text; it pays for every attachment. - **Fix:** - Fetch in batches of 20 to 50 uids, and classify, suggest and flag each batch before fetching the next, so memory is bounded by a batch and a failure loses one batch. - Ask for `RFC822.SIZE` together with the headers first. Above a limit (1 MB, say) fetch `BODY.PEEK[HEADER]` and a bounded partial of the text, `BODY.PEEK[TEXT]<0.65536>`, and parse the two joined. Checked on the snapshot: the headers plus the first 64 KiB of the body of the message above give `parse` the same subject, identifier and text as the whole message. Choosing the first text part from `BODYSTRUCTURE` is the more exact alternative. - In untouched mode fetch `BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)]` first and skip every message whose identifier already has a suggestion for this owner and source. - Say in the README what is transferred, once it is true. ## Done when - `tests/test_imap.py` has a test whose fake client records each `fetch` call: a folder of 120 messages is never asked for in one call of more than the batch size, and a message whose `RFC822.SIZE` is above the limit is never asked for with `BODY.PEEK[]`. Both fail today (one call, all uids, `BODY.PEEK[]`). - `tests/test_imap.py` has a test in which a second pass in untouched mode asks for no message body of a message that already has a suggestion; today it fetches all of them.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:00 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:00 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:00:13 +00:00

Fix the review findings: reading, classifying, matching, the form and what a refusal does

tiagoagueda referenced this issue

2026-10-19 07:00:17 +00:00

Fix the open bug and security issues (#8-#22, #24) #25