Send each verified instance backup to a WebDAV folder, as a backup destination in this package #6

New issue

Open

opened 2026-09-30 20:30:01 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-30 20:30:01 +00:00

Owner

What the plugin should do

A third entry point in the group postulo/postulo#325 creates, set up by staff in Server settings → Backups, not per person: folder URL, user name, password, and the sync's https:// rule in validate().
Room first. DAV:quota-available-bytes, asked by name (RFC 4331 §3), below the archive's size fails before a byte is sent, saying how much is needed. Nextcloud's -1, -2 and -3 (not computed, unknown, unlimited) mean go ahead; a 507 mid-upload is reported as a full folder.
Upload without overwriting. The archive is streamed, not read into memory, under its own name (today postulo-backup-<stamp>.tar.gz), with If-None-Match: *. On a retry, a 412 over a file of the archive's size means the reply was lost, and counts as sent.
Large archives on Nextcloud use chunked upload v2: MKCOL /remote.php/dav/uploads/<user>/<id>, chunks numbered 1 to 10000, of 5 MB to 5 GB, then MOVE <id>/.file to the target with Overwrite: F (RFC 4918 §10.6), a Destination header on every step and OC-Total-Length for an early refusal. The server drops unfinished uploads after 24 hours; a retry starts a new one. Elsewhere, one PUT. None of this was tried against a real Nextcloud.
Confirm. The uploaded file must show the archive's size before it counts as sent; its href and ETag are the reference handed back.
Remote retention, through the kind's list and delete: only archives this destination put in the folder (by name) are listed or deleted, never other files, so one folder serves one instance; each DELETE carries If-Match with the recorded ETag. How many to keep is the core's setting.
Scope. Every request stays under the chosen folder (plus Nextcloud's uploads area while chunking), and hrefs the server returns are checked first; #13 owns that check for the whole client.
Test reads the folder and its free space and writes nothing (postulo/postulo-dav#2), so the first backup proves write permission.
Bytes and plain folders in src/postulo_dav/dav.py are shared with #5; whichever lands first does them.

The credential is kept on the instance and opens a folder of archives holding everyone's data. Whoever takes over the instance holds it, and one that can delete for retention can delete every off-site archive. The README recommends one confined to the backup folder and used for nothing else: a Hetzner Storage Box sub-account (only its own sub-directory, over WebDAV) or a dedicated Nextcloud account. A Nextcloud app password cannot be narrowed to a folder, a Fastmail one only by protocol.

Encryption. The archive holds everyone's data in clear; only the stored secrets are encrypted, under a key it leaves out. It should be encrypted before it leaves the machine, decided with postulo/postulo#242 §3 in postulo/postulo#325 so every destination gets the same answer. This plugin sends what it is handed and encrypts nothing itself.

Decisions

Where it lives. Recommended: in postulo-dav, as the list frames it, reusing the client, though staff rather than a person configure it; otherwise a package of its own.
When to chunk. Recommended: only on Nextcloud (a folder under /remote.php/dav/files/) and above a threshold set when it is built; plain PUT elsewhere.
How to confirm. Size and ETag (recommended), or reading the archive back and hashing it, a whole download every run.

Needs from the core

postulo/postulo#325: the kind, instance-level settings and secrets, the call after an archive verifies, retries, remote retention, and the outcome on the Backups page. Nothing here runs before it lands, beyond the shared client work; until then, the wiki's cron plus restic, borg or age carry archives off the machine.

New strings go through uv run postulo-messages extract, fr-fr and pt-pt filled on the commit (the test gates them), the 23 EU catalogues before a tag (uv run pytest -m release). The kind's names are new on the surface, so the core pin moves (pyproject.toml's dev group pins commit 1d6cdd7e0) and tests/test_surface.py is extended.

Done when

With the destination set, a scheduled backup ends as a confirmed archive in the folder and the Backups page says so; a folder without room receives nothing, and says how much is needed.
A retry after a lost reply leaves one archive; with keep-N, the folder holds at most N of Postulo's archives, older ones go only after a new one is confirmed, and no other file is touched.
The in-memory DAV server covers plain PUT, the chunked path, a full folder and a listing href outside the folder.
The README carries the credential advice.

From the list of DAV plugin improvements of 2026-09-30.

The list of DAV improvements gives the storage component a second use: "and/or backup postulo data (on the server/admin side)". This package does nothing with backups, and the core would not call it if it did. `write_backup` (`src/postulo/core/backup.py:325` in the core) writes one verified `.tar.gz` to a local directory, `POSTULO_BACKUP_DIR` by default, and calls nothing afterwards. Connections are per person (`Connection.owner` is required), and the only instance-level plugin configuration is the transports' columns on `SiteSettings`. postulo/postulo#242 (open, 0.6.0) plans the *Backups* section with no remote destination. So this is the plugin half of postulo/postulo#325, and waits for it. ## What the plugin should do - **A third entry point** in the group postulo/postulo#325 creates, set up by staff in *Server settings → Backups*, not per person: folder URL, user name, password, and the sync's `https://` rule in `validate()`. - **Room first.** `DAV:quota-available-bytes`, asked by name (RFC 4331 §3), below the archive's size fails before a byte is sent, saying how much is needed. Nextcloud's -1, -2 and -3 (not computed, unknown, unlimited) mean go ahead; a 507 mid-upload is reported as a full folder. - **Upload without overwriting.** The archive is streamed, not read into memory, under its own name (today `postulo-backup-<stamp>.tar.gz`), with `If-None-Match: *`. On a retry, a 412 over a file of the archive's size means the reply was lost, and counts as sent. - **Large archives on Nextcloud** use chunked upload v2: `MKCOL /remote.php/dav/uploads/<user>/<id>`, chunks numbered 1 to 10000, of 5 MB to 5 GB, then `MOVE <id>/.file` to the target with `Overwrite: F` (RFC 4918 §10.6), a `Destination` header on every step and `OC-Total-Length` for an early refusal. The server drops unfinished uploads after 24 hours; a retry starts a new one. Elsewhere, one `PUT`. None of this was tried against a real Nextcloud. - **Confirm.** The uploaded file must show the archive's size before it counts as sent; its href and ETag are the reference handed back. - **Remote retention**, through the kind's list and delete: only archives this destination put in the folder (by name) are listed or deleted, never other files, so one folder serves one instance; each `DELETE` carries `If-Match` with the recorded ETag. How many to keep is the core's setting. - **Scope.** Every request stays under the chosen folder (plus Nextcloud's uploads area while chunking), and hrefs the server returns are checked first; #13 owns that check for the whole client. - **Test** reads the folder and its free space and writes nothing (postulo/postulo-dav#2), so the first backup proves write permission. - Bytes and plain folders in `src/postulo_dav/dav.py` are shared with #5; whichever lands first does them. **The credential** is kept on the instance and opens a folder of archives holding everyone's data. Whoever takes over the instance holds it, and one that can delete for retention can delete every off-site archive. The README recommends one confined to the backup folder and used for nothing else: a Hetzner Storage Box sub-account (only its own sub-directory, over WebDAV) or a dedicated Nextcloud account. A Nextcloud app password cannot be narrowed to a folder, a Fastmail one only by protocol. **Encryption.** The archive holds everyone's data in clear; only the stored secrets are encrypted, under a key it leaves out. It should be encrypted before it leaves the machine, decided with postulo/postulo#242 §3 in postulo/postulo#325 so every destination gets the same answer. This plugin sends what it is handed and encrypts nothing itself. ## Decisions - **Where it lives.** Recommended: in postulo-dav, as the list frames it, reusing the client, though staff rather than a person configure it; otherwise a package of its own. - **When to chunk.** Recommended: only on Nextcloud (a folder under `/remote.php/dav/files/`) and above a threshold set when it is built; plain `PUT` elsewhere. - **How to confirm.** Size and ETag (recommended), or reading the archive back and hashing it, a whole download every run. ## Needs from the core postulo/postulo#325: the kind, instance-level settings and secrets, the call after an archive verifies, retries, remote retention, and the outcome on the *Backups* page. Nothing here runs before it lands, beyond the shared client work; until then, the wiki's cron plus restic, borg or age carry archives off the machine. New strings go through `uv run postulo-messages extract`, fr-fr and pt-pt filled on the commit (the test gates them), the 23 EU catalogues before a tag (`uv run pytest -m release`). The kind's names are new on the surface, so the core pin moves (`pyproject.toml`'s dev group pins commit 1d6cdd7e0) and `tests/test_surface.py` is extended. ## Done when - With the destination set, a scheduled backup ends as a confirmed archive in the folder and the *Backups* page says so; a folder without room receives nothing, and says how much is needed. - A retry after a lost reply leaves one archive; with keep-N, the folder holds at most N of Postulo's archives, older ones go only after a new one is confirmed, and no other file is touched. - The in-memory DAV server covers plain `PUT`, the chunked path, a full folder and a listing href outside the folder. - The README carries the credential advice. From the list of DAV plugin improvements of 2026-09-30.

tiagoagueda added the

security

enhancement

labels

2026-09-30 20:30:02 +00:00

tiagoagueda referenced this issue

2026-09-30 20:30:04 +00:00

Choose each address book, calendar, task list and folder from the server by href, with a mode each, and hold every request to them #13

tiagoagueda referenced this issue from Postulo/postulo

2026-09-30 20:30:04 +00:00

Add a backup-destination plugin kind that receives each verified archive, set up by staff under Server settings → Backups #325

tiagoagueda referenced this issue