Add a backup-destination plugin kind that receives each verified archive, set up by staff under Server settings → Backups #325

New issue

Closed

opened 2026-09-30 20:30:04 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-30 20:30:04 +00:00

Owner

The DAV improvements ask the plugin to "backup postulo data (on the server/admin side)" (postulo/postulo-dav#6), and the core gives a plugin no way to. write_backup (src/postulo/core/backup.py:325-402) writes and verifies one .tar.gz in POSTULO_BACKUP_DIR, and calls nothing afterwards. No plugin kind is about backups: StorePlugin.put takes one person's document, the connected kinds hang off a Connection, whose owner is required, and the only instance-level plugin configuration is the transports' columns on SiteSettings (src/postulo/core/models.py:1004-1021). #242 (open, 0.6.0) plans the Backups section of Server settings (list, back up now, download, delete, schedule, retention, metrics) but names no destination, and leaves passphrase encryption undecided (§3). Its dependencies (#219, #220, #221, #234) are closed; this lands with it or after.

Proposal

1. The kind

A backup-destination kind with its own entry-point group (say postulo.backups) and a small protocol on postulo.plugins.api: the usual config_fields(), validate() and test(), plus
- send(archive, manifest, config), returning a reference (an id of the plugin's choosing and a label) and raising on failure;
- list(config), the references of the archives it holds there;
- delete(reference, config).
archive is the verified archive as it leaves (encrypted, §4), read in chunks, with its name and size; manifest is plain values, for naming and logging, never written beside the archive in clear.
Like a transport it is the operator's plumbing: not a connected kind, and not under the per-person plugin policy (#95; UNGOVERNED_KINDS in src/postulo/plugins/policy.py).
A kind touches registry.GROUPS (which the installer reads, src/postulo/plugins/installing.py:120-122), LABELS and TONES in src/postulo/plugins/kinds.py, and the table of kinds in Writing a plugin.

2. When it runs

After an archive verifies, from the schedule or Back up now, a send to the destination is queued. The scheduler sends and retries it with a growing wait, as it sends document copies (src/postulo/documents/archiving.py: a claim, a lease, a cap on attempts), not as an errand, which never retries (#247). An upload can take far longer than a document copy, so it must neither delay the next pass's reminders nor outlive its own claim. Remote retention runs only after a send is confirmed, keeps the newest N (its own count, defaulting to the local one; or none deleted, for a credential that cannot delete), and deletes only what list() returned. A failed send never deletes or rotates the local archive.

3. What staff see

Staff choose the destination in the Backups section, on a form drawn from its FieldSpecs as a transport's is (src/postulo/core/server_forms.py:560); secrets are encrypted under the same key as connections' and transports', masked, and kept when left blank. Each archive on the list says where it went: sent (when, and the remote name), waiting, or failed in the plugin's words. A failed send shows on Overview as a failed run does (#242 §5). Metrics beside #242's, such as postulo_backup_offsite_last_success_timestamp_seconds and postulo_backup_offsite_failures_total.

4. Encryption

The archive holds everyone's data in clear; only the stored secrets (connections', transports', the mail password) are encrypted, under a key it leaves out. #242 §3 must be settled, here or there, before any archive leaves the machine; its never-stored passphrase cannot serve a scheduled send. An age recipient can: only the public key is kept on the instance, the identity stays with the operator, and the wiki already names age. Restoring then starts with decrypting by hand.

5. Out of scope

Restoring from a destination: download the archive by hand and restore as today (#242 §6).

Decisions

Where the settings live. (a) A trio on SiteSettings like the transports' (plugin name, JSON configuration, encrypted secrets); (b) a staff-owned Connection; (c) a small staff-only table. Recommended: (a), for one destination. (b) is an OwnedModel, deleted with its owner's account (on_delete=CASCADE), so an administrator leaving would silently stop off-site backups. (c) if several destinations are wanted later.
Encryption. Recommended: the core encrypts before any destination sees the archive, to an age recipient (a new dependency in the image), and refuses a destination without one in the first version.
manage.py backup. Recommended: an archive written into POSTULO_BACKUP_DIR queues a send, with --no-send to skip; one written elsewhere does not.

Security

Staff only: every new URL answers 404 to anyone else. A log line for each send and each remote delete (archive, destination, outcome), and one naming who changed the destination. A destination's requests go through api.client(), so the destination policy applies.

Needed by

postulo/postulo-dav#6.

Done when

A verified archive reaches the configured destination, retried on failure, with its outcome on the Backups list and in the metrics.
Remote retention deletes only after a confirmed send, and only what the destination listed; a failing destination leaves the local archive where it is.
The new names on postulo.plugins.api each have their line in Writing a plugin (tests/test_wiki_surface.py, which also wants the new group named), and are covered by the published postulo.plugins.testing check and tests/test_plugin_surface.py.
The new pages are in the page-coverage walk (tests/test_page_coverage.py) and the isolation sweep (tests/security/test_isolation_sweep.py); the metrics are on Health, metrics and logs.
New strings are in the working catalogues (en, fr-fr, pt-pt, pt-br as drafts), and Backups and your data gains a section.

From the list of DAV plugin improvements of 2026-09-30.

The DAV improvements ask the plugin to "backup postulo data (on the server/admin side)" (postulo/postulo-dav#6), and the core gives a plugin no way to. `write_backup` (`src/postulo/core/backup.py:325-402`) writes and verifies one `.tar.gz` in `POSTULO_BACKUP_DIR`, and calls nothing afterwards. No plugin kind is about backups: `StorePlugin.put` takes one person's document, the connected kinds hang off a `Connection`, whose owner is required, and the only instance-level plugin configuration is the transports' columns on `SiteSettings` (`src/postulo/core/models.py:1004-1021`). #242 (open, 0.6.0) plans the *Backups* section of *Server settings* (list, back up now, download, delete, schedule, retention, metrics) but names no destination, and leaves passphrase encryption undecided (§3). Its dependencies (#219, #220, #221, #234) are closed; this lands with it or after. ## Proposal ### 1. The kind - A backup-destination kind with its own entry-point group (say `postulo.backups`) and a small protocol on `postulo.plugins.api`: the usual `config_fields()`, `validate()` and `test()`, plus - `send(archive, manifest, config)`, returning a reference (an id of the plugin's choosing and a label) and raising on failure; - `list(config)`, the references of the archives it holds there; - `delete(reference, config)`. `archive` is the verified archive as it leaves (encrypted, §4), read in chunks, with its name and size; `manifest` is plain values, for naming and logging, never written beside the archive in clear. - Like a transport it is the operator's plumbing: not a connected kind, and not under the per-person plugin policy (#95; `UNGOVERNED_KINDS` in `src/postulo/plugins/policy.py`). - A kind touches `registry.GROUPS` (which the installer reads, `src/postulo/plugins/installing.py:120-122`), `LABELS` and `TONES` in `src/postulo/plugins/kinds.py`, and the table of kinds in *Writing a plugin*. ### 2. When it runs After an archive verifies, from the schedule or *Back up now*, a send to the destination is queued. The scheduler sends and retries it with a growing wait, as it sends document copies (`src/postulo/documents/archiving.py`: a claim, a lease, a cap on attempts), not as an errand, which never retries (#247). An upload can take far longer than a document copy, so it must neither delay the next pass's reminders nor outlive its own claim. Remote retention runs only after a send is confirmed, keeps the newest *N* (its own count, defaulting to the local one; or none deleted, for a credential that cannot delete), and deletes only what `list()` returned. A failed send never deletes or rotates the local archive. ### 3. What staff see Staff choose the destination in the *Backups* section, on a form drawn from its `FieldSpec`s as a transport's is (`src/postulo/core/server_forms.py:560`); secrets are encrypted under the same key as connections' and transports', masked, and kept when left blank. Each archive on the list says where it went: sent (when, and the remote name), waiting, or failed in the plugin's words. A failed send shows on *Overview* as a failed run does (#242 §5). Metrics beside #242's, such as `postulo_backup_offsite_last_success_timestamp_seconds` and `postulo_backup_offsite_failures_total`. ### 4. Encryption The archive holds everyone's data in clear; only the stored secrets (connections', transports', the mail password) are encrypted, under a key it leaves out. #242 §3 must be settled, here or there, before any archive leaves the machine; its never-stored passphrase cannot serve a scheduled send. An age recipient can: only the public key is kept on the instance, the identity stays with the operator, and the wiki already names age. Restoring then starts with decrypting by hand. ### 5. Out of scope Restoring from a destination: download the archive by hand and restore as today (#242 §6). ## Decisions - **Where the settings live.** (a) A trio on `SiteSettings` like the transports' (plugin name, JSON configuration, encrypted secrets); (b) a staff-owned `Connection`; (c) a small staff-only table. Recommended: (a), for one destination. (b) is an `OwnedModel`, deleted with its owner's account (`on_delete=CASCADE`), so an administrator leaving would silently stop off-site backups. (c) if several destinations are wanted later. - **Encryption.** Recommended: the core encrypts before any destination sees the archive, to an age recipient (a new dependency in the image), and refuses a destination without one in the first version. - **`manage.py backup`.** Recommended: an archive written into `POSTULO_BACKUP_DIR` queues a send, with `--no-send` to skip; one written elsewhere does not. ## Security Staff only: every new URL answers 404 to anyone else. A log line for each send and each remote delete (archive, destination, outcome), and one naming who changed the destination. A destination's requests go through `api.client()`, so the destination policy applies. ## Needed by postulo/postulo-dav#6. ## Done when - A verified archive reaches the configured destination, retried on failure, with its outcome on the *Backups* list and in the metrics. - Remote retention deletes only after a confirmed send, and only what the destination listed; a failing destination leaves the local archive where it is. - The new names on `postulo.plugins.api` each have their line in *Writing a plugin* (`tests/test_wiki_surface.py`, which also wants the new group named), and are covered by the published `postulo.plugins.testing` check and `tests/test_plugin_surface.py`. - The new pages are in the page-coverage walk (`tests/test_page_coverage.py`) and the isolation sweep (`tests/security/test_isolation_sweep.py`); the metrics are on *Health, metrics and logs*. - New strings are in the working catalogues (en, fr-fr, pt-pt, pt-br as drafts), and *Backups and your data* gains a section. From the list of DAV plugin improvements of 2026-09-30.

tiagoagueda added the

security

enhancement

labels

2026-09-30 20:30:04 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:42 +00:00

Uploaded documents: no orphans, no corrupted files, and one boundary for where the bytes live #663

tiagoagueda referenced this issue

2026-10-19 10:56:10 +00:00

SEC2 index: security review of 2026-10-16 (86 issues across the core, plugins and extensions) #992

tiagoagueda added this to the 0.5.0 milestone