Checking a career link: the fallback GET reads the whole response into memory, with no size cap #619

New issue

Closed

opened 2026-10-01 10:48:26 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:26 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the link check's requests.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/http.py:157-167 (public_only_client) and plugins/http.py:139-145 (_build); resume/links.py and resume/views.py are where they were.

Check it answers and Check them all send HEAD, and when the answer is 401, 403, 405, 501 or any 5xx they ask again with client.get(url). That is a non-streaming request, so httpx reads the entire body into memory before returning, and nothing in the public-only client limits its size. Only the status code is used; the body is never needed.

Where: resume/links.py:67-73 (_ask: if response.status_code in (401, 403, 405, 501) or response.status_code >= 500: response = client.get(url)), with the client from plugins/http.py:155-165 (public_only_client), built by :137-143 (_build) with a timeout and no size limit. Reached from resume/views.py:364-380 (LinkCheckView, one link or every link).
What happens: a person's link points at a large file on a host that refuses HEAD (405), or at a pre-signed object-storage address, which is signed for GET and answers HEAD with 403. Pressing Check makes the web worker download and hold the whole file. The 8-second TIMEOUT applies to each read, not to the whole download, so a steady stream is never cut off.
Who can trigger it: any signed-in account holder, with a link of their own; it needs no hostile intent, only a large file behind a host that refuses HEAD.
Why it matters: on the small hosts Postulo is meant for, one worker holding a multi-gigabyte body can exhaust memory for every account on the instance. Related to #321, whose third point describes the same buffering in fetching._read_capped and logos.download but not this call.
Fix: replace the fallback with with client.stream("GET", url) as response: and read nothing (or at most a few kilobytes) before closing, since only the status is wanted. If #321's fix adds a capped-read helper to plugins/http.py, use it here as well.

Done when

A test in tests/test_links_and_letters.py answers HEAD with 405 through a mock transport and GET with a body that never ends, and shows that the check returns the status without reading more than a small bound.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the link check's requests. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/http.py:157-167` (`public_only_client`) and `plugins/http.py:139-145` (`_build`); `resume/links.py` and `resume/views.py` are where they were. *Check it answers* and *Check them all* send `HEAD`, and when the answer is 401, 403, 405, 501 or any 5xx they ask again with `client.get(url)`. That is a non-streaming request, so `httpx` reads the entire body into memory before returning, and nothing in the public-only client limits its size. Only the status code is used; the body is never needed. - **Where:** `resume/links.py:67-73` (`_ask`: `if response.status_code in (401, 403, 405, 501) or response.status_code >= 500: response = client.get(url)`), with the client from `plugins/http.py:155-165` (`public_only_client`), built by `:137-143` (`_build`) with a timeout and no size limit. Reached from `resume/views.py:364-380` (`LinkCheckView`, one link or every link). - **What happens:** a person's link points at a large file on a host that refuses `HEAD` (405), or at a pre-signed object-storage address, which is signed for `GET` and answers `HEAD` with 403. Pressing *Check* makes the web worker download and hold the whole file. The 8-second `TIMEOUT` applies to each read, not to the whole download, so a steady stream is never cut off. - **Who can trigger it:** any signed-in account holder, with a link of their own; it needs no hostile intent, only a large file behind a host that refuses `HEAD`. - **Why it matters:** on the small hosts Postulo is meant for, one worker holding a multi-gigabyte body can exhaust memory for every account on the instance. Related to #321, whose third point describes the same buffering in `fetching._read_capped` and `logos.download` but not this call. - **Fix:** replace the fallback with `with client.stream("GET", url) as response:` and read nothing (or at most a few kilobytes) before closing, since only the status is wanted. If #321's fix adds a capped-read helper to `plugins/http.py`, use it here as well. ## Done when - A test in `tests/test_links_and_letters.py` answers `HEAD` with 405 through a mock transport and `GET` with a body that never ends, and shows that the check returns the status without reading more than a small bound.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:26 +00:00

tiagoagueda added the

security

bug

tier

labels

2026-10-01 10:48:26 +00:00

tiagoagueda commented

2026-10-01 11:03:09 +00:00

Author

Owner

Evidence from the review of #321 on 2026-09-30, which measured this and found the same pattern in two more places.

Measured here: a server answering the fallback GET with 20 MB had all 20,971,520 bytes read. resume/links.py also leaves follow_redirects on, so each redirect's body is read whole too. It sends httpx's default Accept-Encoding, which includes br: a 316-byte brotli answer was held as 200 MB.
The same call shape elsewhere: accounts/avatars.py (the Gravatar fetch: client.get, then response.content) and core/updates.py::_fetch (client.get, then .json()). Both go to fixed hosts over HTTPS, so they matter less, but they have no cap either.

#321 adds http.read_body(response, limit, deadline) for the capture, logo and robots.txt fetches. It streams, stops at a size limit, bounds decompression, and holds the whole download to a deadline. Using it here, with Accept-Encoding: gzip, deflate and follow_redirects=False, would close all three.

Evidence from the review of #321 on 2026-09-30, which measured this and found the same pattern in two more places. - **Measured here:** a server answering the fallback GET with 20 MB had all 20,971,520 bytes read. `resume/links.py` also leaves `follow_redirects` on, so each redirect's body is read whole too. It sends httpx's default `Accept-Encoding`, which includes `br`: a 316-byte brotli answer was held as 200 MB. - **The same call shape elsewhere:** `accounts/avatars.py` (the Gravatar fetch: `client.get`, then `response.content`) and `core/updates.py::_fetch` (`client.get`, then `.json()`). Both go to fixed hosts over HTTPS, so they matter less, but they have no cap either. #321 adds `http.read_body(response, limit, deadline)` for the capture, logo and robots.txt fetches. It streams, stops at a size limit, bounds decompression, and holds the whole download to a deadline. Using it here, with `Accept-Encoding: gzip, deflate` and `follow_redirects=False`, would close all three.

tiagoagueda referenced this issue from a commit

2026-10-03 15:37:05 +00:00

Stream the link check's fallback GET and close it unread

tiagoagueda closed this issue

2026-10-03 15:37:05 +00:00