Plugin surface: a sync cannot keep any state between runs, so postulo-imap reads unrecognised mail again on every pass #1165

New issue

Closed

opened 2026-10-31 03:04:27 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-31 03:04:27 +00:00

Owner

What exists today

The scout read the stale local checkout; on the forge's main (7880562) postulo-imap#20 already narrowed this:

src/postulo_imap/sync.py:266-279: with mark = nothing, or when the folder's PERMANENTFLAGS refuse new keywords (mailbox.py:344-348), use_keyword is false and unread_uids (mailbox.py:350-367) searches SINCE <today - days> alone, so every pass gets the whole look-back (30 days by default, up to 3,650).
sync.py:288-293: each batch of 25 is surveyed (RFC822.SIZE and the Message-ID header, cheap) and _known (sync.py:358-373) drops the messages that already have a suggestion. That covers recognised mail only.
A message the rules do not recognise gets no suggestion (sync.py:377-379 returns "skipped"), so _known never knows it, and Mailbox.read (mailbox.py:400-420) downloads it again, whole up to 1 MB, on every pass. In a Jobs folder most messages are of that kind: job alerts, newsletters, acknowledgements the rules do not match. The same is true of a message that could not be parsed.
The test double in tests/test_imap.py:133-139 answers SELECT with EXISTS and PERMANENTFLAGS only; test_leaving_the_mailbox_untouched (line 712) checks a recognised message, which is the case already handled.

The protocol supports the fix everywhere: RFC 3501 §2.3.1.1 (and RFC 9051 §2.3.1.1) makes UIDs strictly ascending within a folder for as long as its UIDVALIDITY is unchanged, and SELECT returns UIDVALIDITY and UIDNEXT (§6.3.1, §7.1). IMAPClient 3.1.0 hands both back in select_folder's answer (imapclient.py:816-817). The core has nowhere for a plugin to keep them: Connection (src/postulo/plugins/models.py:283-305) has config, secrets and last_report, SyncReport (plugins/base.py:1028-1057) has counters and notes, and no plugin writes to config today.

What would change

Core (one issue on postulo, shared with PR4 and PR5): a small per-connection state a sync plugin reads at the start of a run and returns at the end, for example SyncReport.state: dict | None, saved by _run_connection into a new Connection.sync_state JSONField only when the run did not fail, and dropped when the connection's configuration (server, folder, account) is edited. It goes on the plugin surface (plugins/api.py) and in postulo.plugins.testing's surface check. Bounded in size (a few kB) so it cannot become a store.

postulo-imap (one issue): in the modes that do not flag, remember {folder, uidvalidity, last_uid}. At the next pass, if the folder and UIDVALIDITY match, search UID <last_uid+1>:* SINCE <date> and drop any UID not above last_uid from the answer (RFC 3501 §6.4.8: n:* always includes the highest UID even when it is below n). If UIDVALIDITY changed, read the look-back as today. Record last_uid only for batches that were read to the end, so a pass cut short resumes where it stopped. The keyword mode is unchanged. The test double gains UIDVALIDITY/UIDNEXT in its SELECT answer and a UID range in search.

A side effect worth stating in the README: lengthening the look-back no longer reaches back for mail already passed over; a changed days should reset the state, which the core's "drop on configuration edit" gives for free.

CONDSTORE/QRESYNC (RFC 7162) is not needed: the plugin never revisits a message, so new UIDs are all it asks.

What it saves or protects

medium: in Leave the mailbox untouched mode, or on a folder that refuses keywords, a pass reads only what arrived since the last one; today it downloads the whole source of every unrecognised message of the look-back each time (for example 280 newsletters of 50 kB is about 14 MB per pass, 336 MB a day at the hourly default; an estimate, not measured)

Accessibility

none

Decided by the maintainer (2026-10-31)

per-connection state on the plugin surface: one bounded JSON field written only after a run that did not fail

Decisions

Per-connection state on the plugin surface? Recommended: yes, one bounded JSON field written only after a run that did not fail; PR4 (DAV discovery) and PR5 (DAV sync tokens) need the same thing.
Split into: a core issue (sync state on the surface) and a postulo-imap issue (UIDVALIDITY and last UID), the second waiting on the first.

Found by the efficiency and reliability audit of 2026-10-30 (finding PR2; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today The scout read the stale local checkout; on the forge's `main` (`7880562`) postulo-imap#20 already narrowed this: - `src/postulo_imap/sync.py:266-279`: with `mark = nothing`, or when the folder's `PERMANENTFLAGS` refuse new keywords (`mailbox.py:344-348`), `use_keyword` is false and `unread_uids` (`mailbox.py:350-367`) searches `SINCE <today - days>` alone, so every pass gets the whole look-back (30 days by default, up to 3,650). - `sync.py:288-293`: each batch of 25 is surveyed (`RFC822.SIZE` and the `Message-ID` header, cheap) and `_known` (`sync.py:358-373`) drops the messages that already have a suggestion. That covers recognised mail only. - A message the rules do not recognise gets no suggestion (`sync.py:377-379` returns `"skipped"`), so `_known` never knows it, and `Mailbox.read` (`mailbox.py:400-420`) downloads it again, whole up to 1 MB, on every pass. In a *Jobs* folder most messages are of that kind: job alerts, newsletters, acknowledgements the rules do not match. The same is true of a message that could not be parsed. - The test double in `tests/test_imap.py:133-139` answers `SELECT` with `EXISTS` and `PERMANENTFLAGS` only; `test_leaving_the_mailbox_untouched` (line 712) checks a recognised message, which is the case already handled. The protocol supports the fix everywhere: RFC 3501 §2.3.1.1 (and RFC 9051 §2.3.1.1) makes UIDs strictly ascending within a folder for as long as its `UIDVALIDITY` is unchanged, and `SELECT` returns `UIDVALIDITY` and `UIDNEXT` (§6.3.1, §7.1). IMAPClient 3.1.0 hands both back in `select_folder`'s answer (`imapclient.py:816-817`). The core has nowhere for a plugin to keep them: `Connection` (`src/postulo/plugins/models.py:283-305`) has `config`, secrets and `last_report`, `SyncReport` (`plugins/base.py:1028-1057`) has counters and notes, and no plugin writes to `config` today. ## What would change Core (one issue on postulo, shared with PR4 and PR5): a small per-connection state a sync plugin reads at the start of a run and returns at the end, for example `SyncReport.state: dict | None`, saved by `_run_connection` into a new `Connection.sync_state` JSONField only when the run did not fail, and dropped when the connection's configuration (server, folder, account) is edited. It goes on the plugin surface (`plugins/api.py`) and in `postulo.plugins.testing`'s surface check. Bounded in size (a few kB) so it cannot become a store. postulo-imap (one issue): in the modes that do not flag, remember `{folder, uidvalidity, last_uid}`. At the next pass, if the folder and `UIDVALIDITY` match, search `UID <last_uid+1>:* SINCE <date>` and drop any UID not above `last_uid` from the answer (RFC 3501 §6.4.8: `n:*` always includes the highest UID even when it is below `n`). If `UIDVALIDITY` changed, read the look-back as today. Record `last_uid` only for batches that were read to the end, so a pass cut short resumes where it stopped. The keyword mode is unchanged. The test double gains `UIDVALIDITY`/`UIDNEXT` in its `SELECT` answer and a `UID` range in `search`. A side effect worth stating in the README: lengthening the look-back no longer reaches back for mail already passed over; a changed `days` should reset the state, which the core's "drop on configuration edit" gives for free. CONDSTORE/QRESYNC (RFC 7162) is not needed: the plugin never revisits a message, so new UIDs are all it asks. ## What it saves or protects medium: in *Leave the mailbox untouched* mode, or on a folder that refuses keywords, a pass reads only what arrived since the last one; today it downloads the whole source of every unrecognised message of the look-back each time (for example 280 newsletters of 50 kB is about 14 MB per pass, 336 MB a day at the hourly default; an estimate, not measured) ## Accessibility none ## Decided by the maintainer (2026-10-31) per-connection state on the plugin surface: one bounded JSON field written only after a run that did not fail ## Decisions - **Per-connection state on the plugin surface?** Recommended: yes, one bounded JSON field written only after a run that did not fail; PR4 (DAV discovery) and PR5 (DAV sync tokens) need the same thing. - **Split into:** a core issue (sync state on the surface) and a postulo-imap issue (UIDVALIDITY and last UID), the second waiting on the first. --- Found by the efficiency and reliability audit of 2026-10-30 (finding PR2; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:27 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:27 +00:00

tiagoagueda referenced this issue from Postulo/postulo-imap

2026-10-31 03:04:36 +00:00

Remember the folder's UIDVALIDITY and the last UID read, once the core keeps a sync's state #34

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00