Notifications: a send that fails at a notifier is lost, while a webhook delivery is retried for ten hours #1163

New issue

Closed

opened 2026-10-31 03:04:27 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:27 +00:00

Owner

What exists today

Core, src/postulo/notifications/service.py:77-118 _deliver: each notifier's send() is called once. ConnectionUnusable retires the connection (lines 94-105); any other exception is logged and written on the connection with record_test(False, …) (lines 106-111), and the notification is gone. Callers do not retry either: send_due_reminders.py:61-63 stamps notified_at with a conditional UPDATE before announcing, so a reminder whose only notifier failed is never announced again; applications/quiet.py:113 and jobs/closing.py:116 claim the same way.

The webhook notifier is the exception: it only queues a row (delivers_later), and notifications/webhooks.py:35-36, 122-140 retries it with a doubling delay from five minutes, eight attempts (about ten hours). Notification.key (send_due_reminders.py:93-97) is documented as stable "however many times one of them retries (#229)", so the payload is already shaped for retries; only the webhook uses that.

postulo-apprise on the forge's main (0cd0aa1), src/postulo_apprise/notifier.py:520-530 send: raises when apprise.notify returns false, which Apprise does when any one of the connection's services refused. So the core sees one failure per connection, not per service.

The scout's second point, "URLs re-parsed per send" (notifier.py:444-487, _problems then build): each send parses the field about three times and resolves each server's host for check_destination. That is deliberate, since the instance's destination policy and the name's addresses may have changed since the form was saved (postulo-apprise#10, #12, #15), and it costs milliseconds per send against sends that happen a few times a day. Refuted as a cost; not filed.

What would change

Core: a failed send() (an ordinary exception, not ConnectionUnusable) queues a retry row for that connection and notification, reusing the webhook's schedule and claim logic (FIRST_RETRY, doubling, MAX_ATTEMPTS, the conditional UPDATE lease), delivered by the scheduler pass that already delivers webhooks. The row keeps the notification fields the webhook row keeps today, so it falls under the same export, erasure and pruning rules that PST-023 put on webhook rows. Retrying stops when the connection is switched off or retired, as for webhooks (#243). record_test(False) stays, so the connection still shows the failure at once.

Generalising the webhook outbox into one "pending delivery" table for every notifier is the cleaner shape; a second table beside it is the smaller change. Either way no library is needed; django_tasks (already in use) could carry the delay, but the webhook path already chose rows plus the scheduler, and one mechanism is easier to reason about.

A side benefit: notifications sent from a request (jobs/sharing.py:342) wait today for a slow notifier (Apprise's own deadline is 45 s, notifier.py:71); with a queue, the request could queue and return. That is a separate decision and not part of this issue.

What it saves or protects

medium (reliability): a reminder announced while Telegram, ntfy, the mail server or a push service is briefly unreachable still arrives, minutes later, instead of never

Accessibility

none

Decided by the maintainer (2026-10-31)

retry failed notifier sends in core on the webhook schedule; duplicates on partial Apprise failure accepted and documented

Decisions

Retry failed notifier sends? Recommended: yes, in the core, with the webhook's schedule.
Partial failure on a connection with several Apprise services. Recommended: accept a possible duplicate on the services that took it, and say so in the plugin's README; sending per service and remembering which succeeded would need per-delivery state in the plugin, for little gain.
Split into: one core issue (the retry queue). The URL re-parsing is not filed.

Found by the efficiency and reliability audit of 2026-10-30 (finding PR8; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today Core, `src/postulo/notifications/service.py:77-118` `_deliver`: each notifier's `send()` is called once. `ConnectionUnusable` retires the connection (lines 94-105); any other exception is logged and written on the connection with `record_test(False, …)` (lines 106-111), and the notification is gone. Callers do not retry either: `send_due_reminders.py:61-63` stamps `notified_at` with a conditional `UPDATE` before announcing, so a reminder whose only notifier failed is never announced again; `applications/quiet.py:113` and `jobs/closing.py:116` claim the same way. The webhook notifier is the exception: it only queues a row (`delivers_later`), and `notifications/webhooks.py:35-36, 122-140` retries it with a doubling delay from five minutes, eight attempts (about ten hours). `Notification.key` (`send_due_reminders.py:93-97`) is documented as stable "however many times one of them retries (#229)", so the payload is already shaped for retries; only the webhook uses that. postulo-apprise on the forge's `main` (`0cd0aa1`), `src/postulo_apprise/notifier.py:520-530` `send`: raises when `apprise.notify` returns false, which Apprise does when any one of the connection's services refused. So the core sees one failure per connection, not per service. The scout's second point, "URLs re-parsed per send" (`notifier.py:444-487`, `_problems` then `build`): each send parses the field about three times and resolves each server's host for `check_destination`. That is deliberate, since the instance's destination policy and the name's addresses may have changed since the form was saved (postulo-apprise#10, #12, #15), and it costs milliseconds per send against sends that happen a few times a day. Refuted as a cost; not filed. ## What would change Core: a failed `send()` (an ordinary exception, not `ConnectionUnusable`) queues a retry row for that connection and notification, reusing the webhook's schedule and claim logic (`FIRST_RETRY`, doubling, `MAX_ATTEMPTS`, the conditional `UPDATE` lease), delivered by the scheduler pass that already delivers webhooks. The row keeps the notification fields the webhook row keeps today, so it falls under the same export, erasure and pruning rules that PST-023 put on webhook rows. Retrying stops when the connection is switched off or retired, as for webhooks (#243). `record_test(False)` stays, so the connection still shows the failure at once. Generalising the webhook outbox into one "pending delivery" table for every notifier is the cleaner shape; a second table beside it is the smaller change. Either way no library is needed; `django_tasks` (already in use) could carry the delay, but the webhook path already chose rows plus the scheduler, and one mechanism is easier to reason about. A side benefit: notifications sent from a request (`jobs/sharing.py:342`) wait today for a slow notifier (Apprise's own deadline is 45 s, `notifier.py:71`); with a queue, the request could queue and return. That is a separate decision and not part of this issue. ## What it saves or protects medium (reliability): a reminder announced while Telegram, ntfy, the mail server or a push service is briefly unreachable still arrives, minutes later, instead of never ## Accessibility none ## Decided by the maintainer (2026-10-31) retry failed notifier sends in core on the webhook schedule; duplicates on partial Apprise failure accepted and documented ## Decisions - **Retry failed notifier sends?** Recommended: yes, in the core, with the webhook's schedule. - **Partial failure on a connection with several Apprise services.** Recommended: accept a possible duplicate on the services that took it, and say so in the plugin's README; sending per service and remembering which succeeded would need per-delivery state in the plugin, for little gain. - **Split into:** one core issue (the retry queue). The URL re-parsing is not filed. --- Found by the efficiency and reliability audit of 2026-10-30 (finding PR8; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:27 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:27 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 10:38:54 +00:00