Outbound requests: pinning rewrites the URL to the address, so relative and same-host redirects break #363

New issue

Closed

opened 2026-10-01 10:46:09 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:46:09 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the request hooks in plugins/http.py and following redirects through them with an httpx.MockTransport.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/http.py:101-113 (_pin), :57-80 (_guard) and :116-136 (_public_only), plugins/fetching.py:192 and :206 (fetch_page), and api/api.py:614; the other locations are unchanged.

_pin connects to the approved address by rewriting the request's URL to it, keeping the name only in the Host header and the sni_hostname extension. httpx builds every redirect from that rewritten request, so the address leaks into whatever comes next. A capture whose page answers with a relative redirect fails or lands on the wrong site, and a connection that is redirected to its own hostname loses its credentials. Neither happens when POSTULO_CONNECTIONS_ALLOW_PRIVATE is on, because then the connection client pins nothing.

Where: plugins/http.py:99-111 (_pin: request.url = request.url.copy_with(host=str(address))), called from _guard (:55-78) and _public_only (:114-134); plugins/fetching.py:237 (fetch_page: current = validate_public_url(str(response.next_request.url))) and :251 (FetchedPage(url=current, ...)).
What happens:
1. Capture. fetch_page("https://jobs.example.org/jobs/1"), answered with 301 Location: /jobs/1-engineer, makes a second request to https://93.184.216.34/jobs/1-engineer with Host: 93.184.216.34 and no SNI name, because next_request.url was resolved against the rewritten URL and client.get(current) builds a fresh request from it. Over HTTPS the certificate is checked against the number and fails; over HTTP the server's default virtual host answers. When it does succeed, FetchedPage.url, and so Capture.url (jobs/slow.py:60 and :74, api/api.py:612), is the address rather than the listing's link. Slug and trailing-slash redirects of this kind are common on job boards.
2. Connections. Through http.client(auth=("me", "pw")), PROPFIND https://cloud.example.org/.well-known/carddav answered with an absolute 301 to https://cloud.example.org/remote.php/dav/ is followed without Authorization, because httpx compares the new URL with the rewritten one, sees another origin and strips the credentials; the server answers 401. With private destinations allowed the same exchange answers 207. postulo-dav's discovery (src/postulo_dav/dav.py:126-137) starts from exactly this path.
Why it matters: captures of ordinary public pages fail, or store the wrong address, and DAV, Paperless or webhook connections to a server that redirects to itself fail authentication only on instances that keep the default, safer setting.
Fix: pin below the request rather than by rewriting it: a custom httpx transport (or httpcore network backend) that keeps the URL, Host, SNI and redirects on the name and opens the socket to the approved address. Short of that, in fetch_page resolve Location against current with urljoin instead of reading next_request.url, and in _guard/_public_only restore the name on the response's request before httpx builds the redirect.

Done when

tests/security/test_outbound.py has a capture test with a relative 301 that expects the second request to carry the hostname in Host and SNI and FetchedPage.url to be the hostname URL, and a client(auth=...) test with an absolute same-host redirect that expects Authorization on the second request.

Related to #215 (closed), whose pinning in the connection client brought the second case with it.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the request hooks in `plugins/http.py` and following redirects through them with an `httpx.MockTransport`. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/http.py:101-113` (`_pin`), `:57-80` (`_guard`) and `:116-136` (`_public_only`), `plugins/fetching.py:192` and `:206` (`fetch_page`), and `api/api.py:614`; the other locations are unchanged. `_pin` connects to the approved address by rewriting the request's URL to it, keeping the name only in the `Host` header and the `sni_hostname` extension. httpx builds every redirect from that rewritten request, so the address leaks into whatever comes next. A capture whose page answers with a relative redirect fails or lands on the wrong site, and a connection that is redirected to its own hostname loses its credentials. Neither happens when `POSTULO_CONNECTIONS_ALLOW_PRIVATE` is on, because then the connection client pins nothing. - **Where:** `plugins/http.py:99-111` (`_pin`: `request.url = request.url.copy_with(host=str(address))`), called from `_guard` (`:55-78`) and `_public_only` (`:114-134`); `plugins/fetching.py:237` (`fetch_page`: `current = validate_public_url(str(response.next_request.url))`) and `:251` (`FetchedPage(url=current, ...)`). - **What happens:** 1. *Capture.* `fetch_page("https://jobs.example.org/jobs/1")`, answered with `301 Location: /jobs/1-engineer`, makes a second request to `https://93.184.216.34/jobs/1-engineer` with `Host: 93.184.216.34` and no SNI name, because `next_request.url` was resolved against the rewritten URL and `client.get(current)` builds a fresh request from it. Over HTTPS the certificate is checked against the number and fails; over HTTP the server's default virtual host answers. When it does succeed, `FetchedPage.url`, and so `Capture.url` (`jobs/slow.py:60` and `:74`, `api/api.py:612`), is the address rather than the listing's link. Slug and trailing-slash redirects of this kind are common on job boards. 2. *Connections.* Through `http.client(auth=("me", "pw"))`, `PROPFIND https://cloud.example.org/.well-known/carddav` answered with an absolute `301` to `https://cloud.example.org/remote.php/dav/` is followed without `Authorization`, because httpx compares the new URL with the rewritten one, sees another origin and strips the credentials; the server answers 401. With private destinations allowed the same exchange answers 207. `postulo-dav`'s discovery (`src/postulo_dav/dav.py:126-137`) starts from exactly this path. - **Why it matters:** captures of ordinary public pages fail, or store the wrong address, and DAV, Paperless or webhook connections to a server that redirects to itself fail authentication only on instances that keep the default, safer setting. - **Fix:** pin below the request rather than by rewriting it: a custom httpx transport (or httpcore network backend) that keeps the URL, `Host`, SNI and redirects on the name and opens the socket to the approved address. Short of that, in `fetch_page` resolve `Location` against `current` with `urljoin` instead of reading `next_request.url`, and in `_guard`/`_public_only` restore the name on the response's request before httpx builds the redirect. ## Done when - `tests/security/test_outbound.py` has a capture test with a relative `301` that expects the second request to carry the hostname in `Host` and SNI and `FetchedPage.url` to be the hostname URL, and a `client(auth=...)` test with an absolute same-host redirect that expects `Authorization` on the second request. Related to #215 (closed), whose pinning in the connection client brought the second case with it.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:09 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:46:09 +00:00

tiagoagueda referenced this issue from a commit

2026-10-01 13:47:40 +00:00

Harden fetching a page or a logo, and the username an archive brings

tiagoagueda commented

2026-10-01 13:47:56 +00:00

Author

Owner

Half of this landed with #321, in 317a3fa1e.

Fixed, the capture half: a relative redirect, and a same-host one, is now joined to the address that was asked for rather than to the pinned IP. The next hop goes out with the right Host and SNI, and the capture keeps the site's address. tests/security/test_outbound.py::test_a_redirect_is_followed_by_name_and_its_body_is_never_read holds it with a 301. Logo redirects are followed by hand in the same way.
Not fixed, the connection half: http.client(auth=…) on an absolute same-host 301 still drops Authorization. The #321 fixer probed it and got 401, with the second request sent to https://93.184.216.34/remote.php/dav/ and no Authorization. That needs the pin moved below the request, and no test covers it yet.

So this issue stays open for the connection half.

Half of this landed with #321, in `317a3fa1e`. - **Fixed, the capture half:** a relative redirect, and a same-host one, is now joined to the address that was asked for rather than to the pinned IP. The next hop goes out with the right `Host` and SNI, and the capture keeps the site's address. `tests/security/test_outbound.py::test_a_redirect_is_followed_by_name_and_its_body_is_never_read` holds it with a 301. Logo redirects are followed by hand in the same way. - **Not fixed, the connection half:** `http.client(auth=…)` on an absolute same-host 301 still drops `Authorization`. The #321 fixer probed it and got 401, with the second request sent to `https://93.184.216.34/remote.php/dav/` and no `Authorization`. That needs the pin moved below the request, and no test covers it yet. So this issue stays open for the connection half.

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:56 +00:00

Keep the site's name on a pinned request once its answer is in

tiagoagueda closed this issue

2026-10-04 10:04:56 +00:00