The scheduler can send twice, dies on one error, is always "unhealthy", and nothing notices when it stops #221

New issue

Closed

opened 2026-09-15 21:23:23 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-15 21:23:23 +00:00

Owner

The scheduler (send_due_reminders --loop) is the weakest part of operating Postulo. Found in the 2026-09-15 code audit.

Sending

Send first, stamp later. send_due_reminders.py:40-58 and applications/quiet.py:86-91 notify, then stamp notified_at row by row, with no claim, no transaction and no lock between runs. Cron plus --loop (both offered in compose.yml), or a restart mid-pass, resends everything notified but not yet stamped.
Store copies can be sent twice. documents/archiving.py:149-170 does not claim pending_copies rows, so overlapping runs or Send now can put the same copy twice, although the wiki says "nothing is ever sent twice".
One error ends the run. handle() (:82-102) has no try/except around each pass and never calls close_old_connections(). A SQLite lock timeout, a dropped PostgreSQL connection or a reminder whose posting has no company kills the process, and the restart re-runs the whole entrypoint.
Syncs block reminders. run_syncs() runs inline (plugins/syncing.py:109-117), so a slow sync holds up everything after it.

Operating it

Always unhealthy. The image-wide HEALTHCHECK curls :8000/healthz (Dockerfile:252-253); the scheduler serves nothing, and compose.yml does not override the check.
Unsafe start order. The scheduler sets POSTULO_SKIP_MIGRATE=1 but depends_on: [postulo] has no condition: service_healthy, so new code can start against an unmigrated schema. It also runs plugins sync at the same moment as the web container, against a shared record with no locking.
Nothing notices it has stopped. core/metrics.py says it answers "is the scheduler still going round", but there is no last-pass timestamp. postulo_pending{kind="reminders"} counts every undone reminder, due or not.
Stale plugin set. Plugins installed or disabled from the web never reach the scheduler; see the plugin lifecycle issue (#228).

Proposal

Claim before sending: a conditional UPDATE … SET notified_at=now WHERE pk=… AND notified_at IS NULL, sending only if a row changed. The same for store copies (select_for_update(skip_locked=True) or a sending state).
A lease per pass (cache.add with a timeout, or a database row) so two schedulers never overlap.
try/except Exception with logger.exception per item and per pass, plus close_old_connections() each iteration.
Give syncs a time budget, or move them to tasks.
A heartbeat each pass. Export postulo_scheduler_last_pass_timestamp_seconds, postulo_overdue{kind="reminders"} (due, not notified, older than 15 minutes) and postulo_failures{kind="syncs"}. Document a sample alert.
In compose: a scheduler healthcheck that reads the heartbeat, depends_on: postulo: condition: service_healthy, and POSTULO_SKIP_PLUGIN_SYNC=1 on the scheduler.

The scheduler (`send_due_reminders --loop`) is the weakest part of operating Postulo. Found in the 2026-09-15 code audit. ## Sending - **Send first, stamp later.** `send_due_reminders.py:40-58` and `applications/quiet.py:86-91` notify, then stamp `notified_at` row by row, with no claim, no transaction and no lock between runs. Cron plus `--loop` (both offered in `compose.yml`), or a restart mid-pass, resends everything notified but not yet stamped. - **Store copies can be sent twice.** `documents/archiving.py:149-170` does not claim `pending_copies` rows, so overlapping runs or *Send now* can `put` the same copy twice, although the wiki says "nothing is ever sent twice". - **One error ends the run.** `handle()` (`:82-102`) has no try/except around each pass and never calls `close_old_connections()`. A SQLite lock timeout, a dropped PostgreSQL connection or a reminder whose posting has no company kills the process, and the restart re-runs the whole entrypoint. - **Syncs block reminders.** `run_syncs()` runs inline (`plugins/syncing.py:109-117`), so a slow sync holds up everything after it. ## Operating it - **Always unhealthy.** The image-wide `HEALTHCHECK` curls `:8000/healthz` (`Dockerfile:252-253`); the scheduler serves nothing, and `compose.yml` does not override the check. - **Unsafe start order.** The scheduler sets `POSTULO_SKIP_MIGRATE=1` but `depends_on: [postulo]` has no `condition: service_healthy`, so new code can start against an unmigrated schema. It also runs `plugins sync` at the same moment as the web container, against a shared record with no locking. - **Nothing notices it has stopped.** `core/metrics.py` says it answers "is the scheduler still going round", but there is no last-pass timestamp. `postulo_pending{kind="reminders"}` counts every undone reminder, due or not. - **Stale plugin set.** Plugins installed or disabled from the web never reach the scheduler; see the plugin lifecycle issue (#228). ## Proposal - Claim before sending: a conditional `UPDATE … SET notified_at=now WHERE pk=… AND notified_at IS NULL`, sending only if a row changed. The same for store copies (`select_for_update(skip_locked=True)` or a *sending* state). - A lease per pass (`cache.add` with a timeout, or a database row) so two schedulers never overlap. - `try/except Exception` with `logger.exception` per item and per pass, plus `close_old_connections()` each iteration. - Give syncs a time budget, or move them to tasks. - A heartbeat each pass. Export `postulo_scheduler_last_pass_timestamp_seconds`, `postulo_overdue{kind="reminders"}` (due, not notified, older than 15 minutes) and `postulo_failures{kind="syncs"}`. Document a sample alert. - In compose: a scheduler healthcheck that reads the heartbeat, `depends_on: postulo: condition: service_healthy`, and `POSTULO_SKIP_PLUGIN_SYNC=1` on the scheduler.

tiagoagueda added the

bug

observability

labels

2026-09-15 21:23:23 +00:00

tiagoagueda referenced this issue

2026-09-15 21:24:46 +00:00

The PostgreSQL deployment is broken end to end: no driver or pg_dump in the image, no scheduler, never tested #219

tiagoagueda referenced this issue

2026-09-15 21:24:49 +00:00

Deadlines on the calendar, and reminders that can be edited, postponed and deleted #238

tiagoagueda added this to the 0.4.0 milestone