Backups → Restoring in a container: the worker keeps running through a restore, and the scheduler stays stopped after it #541

New issue

Closed

opened 2026-10-01 10:47:30 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:30 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d and the wiki at de1f52467, following the container restore recipe against docker/compose.yml.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

The container recipe for a restore stops postulo and scheduler, but not worker, so a background worker can go on reading and writing the database that restore is overwriting. The closing up -d names no profile, so Compose starts only the web service: a scheduler or worker that was running before the restore is left stopped, and stays stopped across reboots.

Where: postulo.wiki/Backups-and-your-data.md:195-202 (the "In a container" block); docker/compose.yml:45-46 (scheduler, profiles: ["scheduler"]) and docker/compose.yml:88-89 (worker, profiles: ["worker"]).
What happens:
1. An operator runs the scheduler and the worker, as docker/compose.yml:43 and :87 tell them to (--profile scheduler up -d, --profile worker up -d).
2. They follow the recipe: docker compose -f docker/compose.yml stop postulo scheduler. The worker (manage.py work, a pass every few seconds) is not named and keeps its connection to /app/data/postulo.sqlite3.
3. restore overwrites the database through SQLite's backup API while the worker is using it. The page's own rule, at line 164-167, is that "nothing else may be using it while the command runs". If the worker's WAL files happen to be present, restore refuses and the page's next suggestion is --force.
4. docker compose -f docker/compose.yml up -d starts only services without a profile. scheduler and worker were stopped by hand, so restart: unless-stopped does not bring them back after a reboot either.
Why it matters: a restore that runs with a live writer is the corruption the page warns about; and after a clean restore, due reminders, notifier deliveries and the queued slow work (captures, logos, PDFs, exports) silently stop until somebody notices the missing heartbeat. With POSTULO_BACKGROUND_WORK=true, queued work piles up with nobody emptying it — the "button that does nothing" docker/compose.yml:86 warns about.
Fix: stop every service, whatever its profile, before the restore: docker compose -f docker/compose.yml --profile scheduler --profile worker stop. After it, start the same set the operator runs, for example docker compose -f docker/compose.yml --profile scheduler --profile worker up -d, with a sentence saying to drop the profiles they do not use (the worker must only run together with POSTULO_BACKGROUND_WORK=true); or recommend COMPOSE_PROFILES in .env so that plain up -d and stop cover them. Add the worker to the sentence at line 166 ("a web worker or the scheduler"), and give the PostgreSQL stack (docker/compose.postgres.yml, scheduler profile only) the same treatment.

Related to #234 (closed), which added this recipe to give a container a safe restore.

Done when

The container restore block in Backups-and-your-data.md stops every service docker/compose.yml defines, worker included, before run ... restore, and its closing command starts the scheduler and worker again for an operator who runs them.
Followed on a stack started with --profile scheduler --profile worker, docker compose -f docker/compose.yml ps shows all three services running after the restore, and none of them was up while restore ran.

Found on 2026-09-29 in a read-only review of postulo at 95348908d and the wiki at de1f52467, following the container restore recipe against `docker/compose.yml`. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The container recipe for a restore stops `postulo` and `scheduler`, but not `worker`, so a background worker can go on reading and writing the database that `restore` is overwriting. The closing `up -d` names no profile, so Compose starts only the web service: a scheduler or worker that was running before the restore is left stopped, and stays stopped across reboots. - **Where:** `postulo.wiki/Backups-and-your-data.md:195-202` (the "In a container" block); `docker/compose.yml:45-46` (`scheduler`, `profiles: ["scheduler"]`) and `docker/compose.yml:88-89` (`worker`, `profiles: ["worker"]`). - **What happens:** 1. An operator runs the scheduler and the worker, as `docker/compose.yml:43` and `:87` tell them to (`--profile scheduler up -d`, `--profile worker up -d`). 2. They follow the recipe: `docker compose -f docker/compose.yml stop postulo scheduler`. The worker (`manage.py work`, a pass every few seconds) is not named and keeps its connection to `/app/data/postulo.sqlite3`. 3. `restore` overwrites the database through SQLite's backup API while the worker is using it. The page's own rule, at line 164-167, is that "nothing else may be using it while the command runs". If the worker's WAL files happen to be present, `restore` refuses and the page's next suggestion is `--force`. 4. `docker compose -f docker/compose.yml up -d` starts only services without a profile. `scheduler` and `worker` were stopped by hand, so `restart: unless-stopped` does not bring them back after a reboot either. - **Why it matters:** a restore that runs with a live writer is the corruption the page warns about; and after a clean restore, due reminders, notifier deliveries and the queued slow work (captures, logos, PDFs, exports) silently stop until somebody notices the missing heartbeat. With `POSTULO_BACKGROUND_WORK=true`, queued work piles up with nobody emptying it — the "button that does nothing" `docker/compose.yml:86` warns about. - **Fix:** stop every service, whatever its profile, before the restore: `docker compose -f docker/compose.yml --profile scheduler --profile worker stop`. After it, start the same set the operator runs, for example `docker compose -f docker/compose.yml --profile scheduler --profile worker up -d`, with a sentence saying to drop the profiles they do not use (the worker must only run together with `POSTULO_BACKGROUND_WORK=true`); or recommend `COMPOSE_PROFILES` in `.env` so that plain `up -d` and `stop` cover them. Add the worker to the sentence at line 166 ("a web worker or the scheduler"), and give the PostgreSQL stack (`docker/compose.postgres.yml`, scheduler profile only) the same treatment. Related to #234 (closed), which added this recipe to give a container a safe restore. ## Done when - The container restore block in `Backups-and-your-data.md` stops every service `docker/compose.yml` defines, `worker` included, before `run ... restore`, and its closing command starts the scheduler and worker again for an operator who runs them. - Followed on a stack started with `--profile scheduler --profile worker`, `docker compose -f docker/compose.yml ps` shows all three services running after the restore, and none of them was up while `restore` ran.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:30 +00:00

tiagoagueda added the

documentation

bug

labels

2026-10-01 10:47:30 +00:00

tiagoagueda referenced this issue

2026-10-03 01:59:13 +00:00

Docker Compose at home: one deployment page for a homelab, and the drift between the compose files and the wiki #704

tiagoagueda referenced this issue from a commit

2026-10-04 01:59:02 +00:00