Webhooks: a delivery whose hostname briefly fails to resolve is given up for good, blamed on the private-address rule #549

New issue

Closed

opened 2026-10-01 10:47:33 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:33 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the webhook deliverer and the destination check, and calling check_destination with a resolver that raises EAI_AGAIN.

Related to #240.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/http.py:36-46 (check_destination) and, since #248 moved the address check out of plugins/fetching.py, at plugins/public_addresses.py:31-36 (_addresses_for) and plugins/public_addresses.py:58-60 (public_addresses_for), which is where the fix's Unresolvable belongs; notifications/webhooks.py:124-129 is where it was. The same call repeated on today's code raised the same DestinationRefused with the private-address sentence.

deliver() treats every DestinationRefused from the pre-flight check as final, and the check turns a failed DNS lookup into a DestinationRefused. A resolver hiccup therefore ends the delivery after one attempt, instead of the eight attempts with back-off every other transient failure gets, and the recorded error tells the person to ask for private addresses to be allowed.

Where: notifications/webhooks.py:124-129 (deliver: except DestinationRefused as refused: return fail(str(refused), final=True)); plugins/http.py:34-44 (check_destination wraps every UnsafeURL); plugins/fetching.py:98-103 (_addresses_for raises UnsafeURL("That hostname could not be resolved.") on any socket.gaierror, temporary ones included) and plugins/fetching.py:125-127.
What happens: with the default policy (POSTULO_CONNECTIONS_ALLOW_PRIVATE off), getaddrinfo raising EAI_AGAIN makes check_destination raise DestinationRefused with the message "That hostname could not be resolved. Connections may only reach private or local addresses when the operator sets POSTULO_CONNECTIONS_ALLOW_PRIVATE=true." deliver records the row as GIVEN_UP with next_attempt_at=None after its first attempt. On an instance that allows private destinations the check returns early, the same outage surfaces inside plugin.post as an ordinary exception, and the row is retried. So the operator's address policy decides whether a DNS blip loses notifications.
Why it matters: a short resolver outage on the instance, or at the receiver's DNS provider, during a scheduler pass gives up every due delivery for that receiver, and those events never reach the person's automation. The history then says the address is private, which sends them looking in the wrong place.
Fix: keep "could not be resolved" apart from "is private", as core/destinations.py already does with Unresolvable and Refused: raise a distinct Unresolvable(UnsafeURL) from _addresses_for and public_addresses_for, let check_destination re-raise it without the private-address sentence, and in deliver treat it as a transient failure (fail(...) without final). Keep final=True for a real private-address refusal.

Done when

A test in tests/test_webhooks.py patches socket.getaddrinfo to raise gaierror(EAI_AGAIN), calls deliver on a pending row, and asserts the row is FAILED with a later next_attempt_at and an error that does not mention POSTULO_CONNECTIONS_ALLOW_PRIVATE.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the webhook deliverer and the destination check, and calling `check_destination` with a resolver that raises `EAI_AGAIN`. Related to #240. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/http.py:36-46` (`check_destination`) and, since #248 moved the address check out of `plugins/fetching.py`, at `plugins/public_addresses.py:31-36` (`_addresses_for`) and `plugins/public_addresses.py:58-60` (`public_addresses_for`), which is where the fix's `Unresolvable` belongs; `notifications/webhooks.py:124-129` is where it was. The same call repeated on today's code raised the same `DestinationRefused` with the private-address sentence. `deliver()` treats every `DestinationRefused` from the pre-flight check as final, and the check turns a failed DNS lookup into a `DestinationRefused`. A resolver hiccup therefore ends the delivery after one attempt, instead of the eight attempts with back-off every other transient failure gets, and the recorded error tells the person to ask for private addresses to be allowed. - **Where:** `notifications/webhooks.py:124-129` (`deliver`: `except DestinationRefused as refused: return fail(str(refused), final=True)`); `plugins/http.py:34-44` (`check_destination` wraps every `UnsafeURL`); `plugins/fetching.py:98-103` (`_addresses_for` raises `UnsafeURL("That hostname could not be resolved.")` on any `socket.gaierror`, temporary ones included) and `plugins/fetching.py:125-127`. - **What happens:** with the default policy (`POSTULO_CONNECTIONS_ALLOW_PRIVATE` off), `getaddrinfo` raising `EAI_AGAIN` makes `check_destination` raise `DestinationRefused` with the message "That hostname could not be resolved. Connections may only reach private or local addresses when the operator sets POSTULO_CONNECTIONS_ALLOW_PRIVATE=true." `deliver` records the row as `GIVEN_UP` with `next_attempt_at=None` after its first attempt. On an instance that allows private destinations the check returns early, the same outage surfaces inside `plugin.post` as an ordinary exception, and the row is retried. So the operator's address policy decides whether a DNS blip loses notifications. - **Why it matters:** a short resolver outage on the instance, or at the receiver's DNS provider, during a scheduler pass gives up every due delivery for that receiver, and those events never reach the person's automation. The history then says the address is private, which sends them looking in the wrong place. - **Fix:** keep "could not be resolved" apart from "is private", as `core/destinations.py` already does with `Unresolvable` and `Refused`: raise a distinct `Unresolvable(UnsafeURL)` from `_addresses_for` and `public_addresses_for`, let `check_destination` re-raise it without the private-address sentence, and in `deliver` treat it as a transient failure (`fail(...)` without `final`). Keep `final=True` for a real private-address refusal. ## Done when - A test in `tests/test_webhooks.py` patches `socket.getaddrinfo` to raise `gaierror(EAI_AGAIN)`, calls `deliver` on a pending row, and asserts the row is `FAILED` with a later `next_attempt_at` and an error that does not mention `POSTULO_CONNECTIONS_ALLOW_PRIVATE`.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:33 +00:00

tiagoagueda added the

bug

tier

labels

2026-10-01 10:47:33 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 02:35:51 +00:00

Retry a webhook whose hostname briefly fails to resolve

tiagoagueda closed this issue

2026-10-04 02:35:51 +00:00