Backups: a backup killed mid-way blocks scheduled backups for 6 hours and Back up now for 24 #1131

New issue

Closed

opened 2026-10-31 03:04:14 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:14 +00:00

Owner

What exists today

core/backups.py:53-55 sets LOCK_SECONDS = 6 * 3600, and take (core/backups.py:245-261) holds LOCK_KEY with cache.add for that long, releasing it in a finally. A worker killed mid-backup (container restart, out-of-memory killer, deploy) never runs the finally, so:

run_scheduled (core/backups.py:401-413) raises Busy for up to 6 hours and records "Another backup was already running." A daily slot that falls inside those hours is lost.
The change the scout missed: Back up now is blocked for longer. core/views_backups.py:94-100 _a_backup_is_running also refuses while any backup errand is WAITING or WORKING and was created in the last 24 hours. The killed errand stays WORKING until errands.fail_interrupted (core/errands.py:193, default hours=24, called from send_due_reminders.py:214) marks it failed after 24 hours. Until then the button says "A backup is already running. Wait for it to finish." (core/views_backups.py:173-175), with no way to clear it from the interface.

What would change

A short lease that the running backup renews, rather than one long enough for the slowest backup:

LOCK_SECONDS of about 10 minutes, renewed by the backup as it goes: core/backup.py write_backup walks the media and plugin trees in _add_tree (called at lines 555-557), which is the natural place to call a renew every minute or every N files. With R2's lease table that is UPDATE ... WHERE key AND token; with the cache as today it is cache.touch(LOCK_KEY, LOCK_SECONDS).
_a_backup_is_running stops trusting the errand row on its own: it asks the lease, and treats a backup errand as running only while the lease is held. A dead backup then frees both the schedule and the button within one lease period.
Optionally, fail_interrupted for the backup kind uses the same short window, so the dead errand's page says it was interrupted within minutes, not a day.

What it saves or protects

small: a worker restart during a backup no longer costs the next scheduled backup or a day of Back up now

Accessibility

none

Decisions

Renewal or a liveness check on the holder (host and PID)? Recommended: renewal. Containers do not share a PID space, and a renewal needs nothing but the lease.
Order with R2. Recommended: after R2, using its renew; it can land alone with cache.touch if R2 waits.

Found by the efficiency and reliability audit of 2026-10-30 (finding R3; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `core/backups.py:53-55` sets `LOCK_SECONDS = 6 * 3600`, and `take` (`core/backups.py:245-261`) holds `LOCK_KEY` with `cache.add` for that long, releasing it in a `finally`. A worker killed mid-backup (container restart, out-of-memory killer, deploy) never runs the `finally`, so: - `run_scheduled` (`core/backups.py:401-413`) raises `Busy` for up to 6 hours and records "Another backup was already running." A daily slot that falls inside those hours is lost. - The change the scout missed: *Back up now* is blocked for longer. `core/views_backups.py:94-100` `_a_backup_is_running` also refuses while any `backup` errand is `WAITING` or `WORKING` and was created in the last 24 hours. The killed errand stays `WORKING` until `errands.fail_interrupted` (`core/errands.py:193`, default `hours=24`, called from `send_due_reminders.py:214`) marks it failed after 24 hours. Until then the button says "A backup is already running. Wait for it to finish." (`core/views_backups.py:173-175`), with no way to clear it from the interface. ## What would change A short lease that the running backup renews, rather than one long enough for the slowest backup: - `LOCK_SECONDS` of about 10 minutes, renewed by the backup as it goes: `core/backup.py` `write_backup` walks the media and plugin trees in `_add_tree` (called at lines 555-557), which is the natural place to call a `renew` every minute or every N files. With R2's lease table that is `UPDATE ... WHERE key AND token`; with the cache as today it is `cache.touch(LOCK_KEY, LOCK_SECONDS)`. - `_a_backup_is_running` stops trusting the errand row on its own: it asks the lease, and treats a `backup` errand as running only while the lease is held. A dead backup then frees both the schedule and the button within one lease period. - Optionally, `fail_interrupted` for the `backup` kind uses the same short window, so the dead errand's page says it was interrupted within minutes, not a day. ## What it saves or protects small: a worker restart during a backup no longer costs the next scheduled backup or a day of *Back up now* ## Accessibility none ## Decisions - **Renewal or a liveness check on the holder (host and PID)?** Recommended: renewal. Containers do not share a PID space, and a renewal needs nothing but the lease. - **Order with R2.** Recommended: after R2, using its `renew`; it can land alone with `cache.touch` if R2 waits. --- Found by the efficiency and reliability audit of 2026-10-30 (finding R3; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:14 +00:00

tiagoagueda added the

performance

bug

labels

2026-10-31 03:04:14 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a commit

2026-11-10 13:37:55 +00:00