Sync: each changed or unclaimed card and event is fetched with a GET of its own, and in read-only mode every foreign event is fetched on every run #45

New issue

Closed

opened 2026-10-31 03:04:29 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:29 +00:00

Owner

What exists today

On the forge's main (83710e9):

Listing: Server.etags (src/postulo_dav/dav.py:347-375) is a Depth 1 PROPFIND for getetag and resourcetype, once per half per run (sync.py:681, sync.py:968). It is the full membership, about 150 to 300 bytes per resource, read to LISTING_LIMIT (2 MB).
Reading: Server.get (dav.py:377-381) is one GET per resource. Contacts read every card no link claims (sync.py:700-712) and every linked card whose ETag moved (sync.py:827). Interviews read every unlinked event whenever wanted is true (sync.py:993-1007), and every linked event whose ETag moved (sync.py:1100).
wanted (sync.py:995-1001) is true whenever an interview of the last 90 days has no twin. In read-only mode nothing is ever pushed, so it stays true for as long as such an interview exists, and every event on the calendar that is not Postulo's is fetched again on every run, only to be counted as foreign at sync.py:1184-1192. Cards that are never adopted (no FN, an over-long href or UID, sync.py:935-941) are likewise fetched on every run.
The test double (tests/test_dav.py:45-120, an httpx.MockTransport handler) answers PROPFIND, GET, PUT, DELETE, MKCOL and MKCALENDAR; it has no REPORT.

Protocol: addressbook-multiget (RFC 6352 §8.7) and calendar-multiget (RFC 4791 §7.9) are mandatory for every CardDAV and CalDAV server, so every server the README names (Nextcloud, Radicale, Baïkal, SOGo, Fastmail, iCloud) answers them; one REPORT names many hrefs and returns each one's getetag and address-data or calendar-data in a multistatus. RFC 6578 sync-collection is optional, advertised in DAV:supported-report-set; SabreDAV (Nextcloud, Baïkal), Radicale, Fastmail, iCloud and Google implement it. CTag (getctag) is a non-standard Apple property that RFC 6578's DAV:sync-token replaces.

What would change

Multiget (this issue). A Server.multiget(collection, hrefs, kind) that sends one REPORT per batch of hrefs (sized so the answer stays under LISTING_LIMIT, e.g. 50 cards or 50 events), parses the multistatus with the existing defusedxml reader, holds every returned href to the collection exactly as etags() does (#13, #33), and returns href → Resource. The read() closures of both halves prefetch every href they are about to need in one call before the loops, and keep get() as the fallback for a resource the multistatus answered with 404 or omitted. The fake server learns REPORT.
In read-only mode the foreign events are still read on every run, but as a few REPORTs rather than hundreds of GETs; skipping the unchanged ones would need stored state (item 2).
Sync tokens (later, optional). With the core's per-connection state (PR2), keep the DAV:sync-token of each collection and ask sync-collection for the delta; fall back to the full listing on valid-sync-token (403/409) or when the report is not supported. This turns the listing too into a delta, but the listing is already small; the GETs were the cost.

Library: the plugin speaks DAV through the core's guarded httpx client by design (#13, #33), so caldav/vobject clients are not a fit; the multiget body is a few lines of XML beside the existing _ETAGS.

What it saves or protects

medium: the GETs of a run become one REPORT per 50 to 100 resources (a first run against a 300-card address book goes from about 302 requests to about 8); a read-only connection on a shared calendar of 500 events stops making 500 GETs an hour

Accessibility

none

Decided by the maintainer (2026-10-31)

DAV multiget now (batches of 50, split on size refusal); sync tokens later, once PR2 exists

Decisions

Sync tokens? Recommended: not now. Multiget removes almost all of the cost with no stored state and no fallback path; revisit once the core has per-connection state for PR2.
Batch size. Recommended: 50 resources per REPORT, read to its own limit of 50 × RESOURCE_LIMIT-sized answers capped at a few MB; a batch whose answer is refused for size (cards with photos) is split in two and asked again.

Found by the efficiency and reliability audit of 2026-10-30 (finding PR5; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today On the forge's `main` (`83710e9`): - Listing: `Server.etags` (`src/postulo_dav/dav.py:347-375`) is a Depth 1 `PROPFIND` for `getetag` and `resourcetype`, once per half per run (`sync.py:681`, `sync.py:968`). It is the full membership, about 150 to 300 bytes per resource, read to `LISTING_LIMIT` (2 MB). - Reading: `Server.get` (`dav.py:377-381`) is one `GET` per resource. Contacts read every card no link claims (`sync.py:700-712`) and every linked card whose ETag moved (`sync.py:827`). Interviews read every unlinked event whenever `wanted` is true (`sync.py:993-1007`), and every linked event whose ETag moved (`sync.py:1100`). - `wanted` (`sync.py:995-1001`) is true whenever an interview of the last 90 days has no twin. In read-only mode nothing is ever pushed, so it stays true for as long as such an interview exists, and every event on the calendar that is not Postulo's is fetched again on every run, only to be counted as foreign at `sync.py:1184-1192`. Cards that are never adopted (no `FN`, an over-long href or UID, `sync.py:935-941`) are likewise fetched on every run. - The test double (`tests/test_dav.py:45-120`, an `httpx.MockTransport` handler) answers `PROPFIND`, `GET`, `PUT`, `DELETE`, `MKCOL` and `MKCALENDAR`; it has no `REPORT`. Protocol: `addressbook-multiget` (RFC 6352 §8.7) and `calendar-multiget` (RFC 4791 §7.9) are mandatory for every CardDAV and CalDAV server, so every server the README names (Nextcloud, Radicale, Baïkal, SOGo, Fastmail, iCloud) answers them; one `REPORT` names many hrefs and returns each one's `getetag` and `address-data` or `calendar-data` in a multistatus. RFC 6578 `sync-collection` is optional, advertised in `DAV:supported-report-set`; SabreDAV (Nextcloud, Baïkal), Radicale, Fastmail, iCloud and Google implement it. CTag (`getctag`) is a non-standard Apple property that RFC 6578's `DAV:sync-token` replaces. ## What would change 1. **Multiget (this issue).** A `Server.multiget(collection, hrefs, kind)` that sends one `REPORT` per batch of hrefs (sized so the answer stays under `LISTING_LIMIT`, e.g. 50 cards or 50 events), parses the multistatus with the existing `defusedxml` reader, holds every returned href to the collection exactly as `etags()` does (#13, #33), and returns `href → Resource`. The `read()` closures of both halves prefetch every href they are about to need in one call before the loops, and keep `get()` as the fallback for a resource the multistatus answered with 404 or omitted. The fake server learns `REPORT`. In read-only mode the foreign events are still read on every run, but as a few `REPORT`s rather than hundreds of GETs; skipping the unchanged ones would need stored state (item 2). 2. **Sync tokens (later, optional).** With the core's per-connection state (PR2), keep the `DAV:sync-token` of each collection and ask `sync-collection` for the delta; fall back to the full listing on `valid-sync-token` (403/409) or when the report is not supported. This turns the listing too into a delta, but the listing is already small; the GETs were the cost. Library: the plugin speaks DAV through the core's guarded `httpx` client by design (#13, #33), so `caldav`/`vobject` clients are not a fit; the multiget body is a few lines of XML beside the existing `_ETAGS`. ## What it saves or protects medium: the GETs of a run become one `REPORT` per 50 to 100 resources (a first run against a 300-card address book goes from about 302 requests to about 8); a read-only connection on a shared calendar of 500 events stops making 500 GETs an hour ## Accessibility none ## Decided by the maintainer (2026-10-31) DAV multiget now (batches of 50, split on size refusal); sync tokens later, once PR2 exists ## Decisions - **Sync tokens?** Recommended: not now. Multiget removes almost all of the cost with no stored state and no fallback path; revisit once the core has per-connection state for PR2. - **Batch size.** Recommended: 50 resources per `REPORT`, read to its own limit of 50 × `RESOURCE_LIMIT`-sized answers capped at a few MB; a batch whose answer is refused for size (cards with photos) is split in two and asked again. --- Found by the efficiency and reliability audit of 2026-10-30 (finding PR5; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:29 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:29 +00:00

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 23:01:26 +00:00

Read and write fewer times per sync #51

tiagoagueda referenced this issue from a commit

2026-11-11 23:01:27 +00:00