The capture throttle guards the form and not the API, which fetches at twenty times the rate #194

New issue

Closed

opened 2026-09-12 18:59:32 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-12 18:59:32 +00:00

Owner

Observation

throttle.capture -- POSTULO_CAPTURE_RATE, 30 an hour per account -- is called in exactly
one place: jobs/capture_views.py:88, the web form. The comment there says why it is tight:

Capture is the one surface that makes this server talk to somebody else's, and nothing
bounded how fast an account could ask it to (#112).

POST /api/v1/captures makes the same outbound fetch (_read() calls fetch_page when
no html is supplied) and is bounded by nothing but POSTULO_API_RATE: 600 an hour, per
token. So the thing the tight limit exists for -- this server fetching addresses somebody
else chose -- is available at twenty times the rate through the API, and a token handed to
a misbehaving client is the one holder of that allowance. The form is the less capable
surface and the more limited one.

What it is not

Not a reason to put 30/h on every API capture. A capture that arrives with its HTML
fetches nothing -- that is the whole point of the browser extension sending the page it
sees -- and #177 sends forty of them from one results page in one gesture. The form counts
those too ("the parse is still work somebody asked for"), which is defensible for a form and
would cap #177 at thirty. The limit is about the fetch.

What fixing it is

Apply throttle.capture in _read() on the branch that fetches -- payload.html empty --
and leave captures that bring their page under the API rate. Two lines and a test: a token
that asks the server to fetch a thirty-first page in an hour gets the same refusal the
form gives (429, with the sentence throttle.TooOften carries), and a token that sends
pages with their HTML does not. POST /captures/preview fetches too, and should count.

Worth writing where POSTULO_CAPTURE_RATE is documented (Configuration.md) that it now
means fetches, on either surface.

Classification

Security, tier/2: the limit that exists to stop this server being used to fetch other
people's addresses is not applied on the surface most able to do it.

## Observation `throttle.capture` -- `POSTULO_CAPTURE_RATE`, 30 an hour per account -- is called in exactly one place: `jobs/capture_views.py:88`, the web form. The comment there says why it is tight: > Capture is the one surface that makes this server talk to somebody else's, and nothing > bounded how fast an account could ask it to (#112). `POST /api/v1/captures` makes the same outbound fetch (`_read()` calls `fetch_page` when no `html` is supplied) and is bounded by nothing but `POSTULO_API_RATE`: **600 an hour, per token**. So the thing the tight limit exists for -- this server fetching addresses somebody else chose -- is available at twenty times the rate through the API, and a token handed to a misbehaving client is the one holder of that allowance. The form is the *less* capable surface and the more limited one. ## What it is not Not a reason to put 30/h on every API capture. A capture that arrives **with its HTML** fetches nothing -- that is the whole point of the browser extension sending the page it sees -- and #177 sends forty of them from one results page in one gesture. The form counts those too ("the parse is still work somebody asked for"), which is defensible for a form and would cap #177 at thirty. The limit is about the fetch. ## What fixing it is Apply `throttle.capture` in `_read()` on the branch that fetches -- `payload.html` empty -- and leave captures that bring their page under the API rate. Two lines and a test: a token that asks the server to fetch a thirty-first page in an hour gets the same refusal the form gives (`429`, with the sentence `throttle.TooOften` carries), and a token that sends pages with their HTML does not. `POST /captures/preview` fetches too, and should count. Worth writing where `POSTULO_CAPTURE_RATE` is documented (`Configuration.md`) that it now means *fetches*, on either surface. ## Classification Security, tier/2: the limit that exists to stop this server being used to fetch other people's addresses is not applied on the surface most able to do it.

tiagoagueda added the

security

bug

tier

labels

2026-09-12 18:59:32 +00:00

tiagoagueda referenced this issue from a commit

2026-09-12 19:16:34 +00:00

Announce several captures from one page once, not once each

tiagoagueda referenced this issue

2026-09-12 19:17:12 +00:00

A results page holds forty listings and Postulo can capture one of them #177