Capture: a refusal from the pinning hook escapes fetch_page as a 500 instead of a capture error #607

New issue

Closed

opened 2026-10-01 10:48:21 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:21 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the capture fetch path and its callers, and settled by running fetch_page with a resolver that answers the first lookup and fails the second.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/http.py:116-136 (_public_only, raising at line 133), plugins/fetching.py:179-185 (the loop of fetch_page, with the redirect check at line 192) and api/api.py:609-616; jobs/slow.py and core/errands.py have not moved. The address check itself moved to plugins/public_addresses.py (#248), which fetching re-exports, and the same run ends in the same DestinationRefused.

fetch_page resolves the address once in validate_public_url, and the request hook of public_only_client resolves it again on every request to pin the connection (the fix for #58 and #215). When that second lookup fails, or answers with an address that is not public, the hook raises DestinationRefused. That class is a plain Exception: it is neither a CaptureError nor an httpx.HTTPError, so nothing between the hook and the person catches it. The address is still refused, so the security property holds; what is lost is the explanation, and the API answers 500.

Where: plugins/http.py:114-134 (_public_only, which raises DestinationRefused(str(exc)) at line 131); plugins/fetching.py:224-230 (fetch_page, whose loop catches only httpx.HTTPError around client.get); api/api.py:607-613 (except CaptureError around fetch_page); jobs/slow.py:52-60 (fetch_a_capture, the same except CaptureError); core/errands.py:139-149 (the errand runner's catch-all).
What happens: a capture URL whose name resolves for the first lookup and then fails to resolve (a transient resolver error) or answers with a private address on the next one. Running fetch_page("https://example.org/job") with socket.getaddrinfo answering a public address once and then raising gaierror ends in postulo.plugins.http.DestinationRefused: That hostname could not be resolved. rather than in an UnsafeURL. Through POST /api/v1/captures that is an unhandled exception and a 500; through the capture errand it is logged with a traceback as an unexpected failure, and the person reads "Something went wrong. The server log has the details." instead of "That hostname could not be resolved." or "That address is on a private or local network".
Why it matters: the message the hook carries is already the translated, person-facing sentence from public_addresses_for, and it is thrown away. An operator sees an error-level traceback for what is an expected refusal, and an API client (the browser extensions) gets a server error with no detail it can show. The same applies to a redirect hop whose target changes its answer between the check at line 237 and the request that follows.
Fix: in fetch_page, catch the hook's refusal beside httpx.HTTPError and turn it into the capture error it stands for:
```
except http.DestinationRefused as exc:
    raise UnsafeURL(str(exc)) from exc
```
Keep DestinationRefused as it is for the other users of public_only_client (resume/links.py:74, jobs/logos.py:122), which already catch it by name.

Done when

A test in tests/test_capture.py patches public_addresses.socket.getaddrinfo to answer a public address on the first call and raise gaierror on the next, stubs robots_allow to allow, and asserts that fetch_page raises fetching.UnsafeURL with "could not be resolved"; it fails today with DestinationRefused.
An API test posts a capture under the same patch and asserts a 422 problem document rather than a 500.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the capture fetch path and its callers, and settled by running `fetch_page` with a resolver that answers the first lookup and fails the second. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/http.py:116-136` (`_public_only`, raising at line 133), `plugins/fetching.py:179-185` (the loop of `fetch_page`, with the redirect check at line 192) and `api/api.py:609-616`; `jobs/slow.py` and `core/errands.py` have not moved. The address check itself moved to `plugins/public_addresses.py` (#248), which `fetching` re-exports, and the same run ends in the same `DestinationRefused`. `fetch_page` resolves the address once in `validate_public_url`, and the request hook of `public_only_client` resolves it again on every request to pin the connection (the fix for #58 and #215). When that second lookup fails, or answers with an address that is not public, the hook raises `DestinationRefused`. That class is a plain `Exception`: it is neither a `CaptureError` nor an `httpx.HTTPError`, so nothing between the hook and the person catches it. The address is still refused, so the security property holds; what is lost is the explanation, and the API answers 500. - **Where:** `plugins/http.py:114-134` (`_public_only`, which raises `DestinationRefused(str(exc))` at line 131); `plugins/fetching.py:224-230` (`fetch_page`, whose loop catches only `httpx.HTTPError` around `client.get`); `api/api.py:607-613` (`except CaptureError` around `fetch_page`); `jobs/slow.py:52-60` (`fetch_a_capture`, the same `except CaptureError`); `core/errands.py:139-149` (the errand runner's catch-all). - **What happens:** a capture URL whose name resolves for the first lookup and then fails to resolve (a transient resolver error) or answers with a private address on the next one. Running `fetch_page("https://example.org/job")` with `socket.getaddrinfo` answering a public address once and then raising `gaierror` ends in `postulo.plugins.http.DestinationRefused: That hostname could not be resolved.` rather than in an `UnsafeURL`. Through `POST /api/v1/captures` that is an unhandled exception and a 500; through the capture errand it is logged with a traceback as an unexpected failure, and the person reads "Something went wrong. The server log has the details." instead of "That hostname could not be resolved." or "That address is on a private or local network". - **Why it matters:** the message the hook carries is already the translated, person-facing sentence from `public_addresses_for`, and it is thrown away. An operator sees an error-level traceback for what is an expected refusal, and an API client (the browser extensions) gets a server error with no `detail` it can show. The same applies to a redirect hop whose target changes its answer between the check at line 237 and the request that follows. - **Fix:** in `fetch_page`, catch the hook's refusal beside `httpx.HTTPError` and turn it into the capture error it stands for: ```python except http.DestinationRefused as exc: raise UnsafeURL(str(exc)) from exc ``` Keep `DestinationRefused` as it is for the other users of `public_only_client` (`resume/links.py:74`, `jobs/logos.py:122`), which already catch it by name. ## Done when - A test in `tests/test_capture.py` patches `public_addresses.socket.getaddrinfo` to answer a public address on the first call and raise `gaierror` on the next, stubs `robots_allow` to allow, and asserts that `fetch_page` raises `fetching.UnsafeURL` with "could not be resolved"; it fails today with `DestinationRefused`. - An API test posts a capture under the same patch and asserts a 422 problem document rather than a 500.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:21 +00:00

tiagoagueda added the

tier

bug

labels

2026-10-01 10:48:21 +00:00

tiagoagueda referenced this issue from a commit

2026-10-01 13:47:40 +00:00

Harden fetching a page or a logo, and the username an archive brings

tiagoagueda closed this issue

2026-10-01 13:47:40 +00:00