Capture: \"The site's robots.txt is honoured\" is shown even when the instance is set to ignore robots.txt #630

New issue

Closed

opened 2026-10-01 10:48:31 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:31 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, comparing the capture page's help text with the fetching code.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/fetching.py:110-111 (the early return in robots_allow) and core/site.py:131-135 (capture_ignore_robots); the template line and CaptureCreateView._render have not moved.

Under How this works the capture form tells everybody that the site's robots.txt is honoured. Since robots.txt became a Server settings switch (and POSTULO_CAPTURE_IGNORE_ROBOTS), an instance can turn that off, and then robots_allow answers yes without fetching anything. The page then says the opposite of what the instance does.

Where: templates/jobs/capture_form.html:85 ({% translate "The site's robots.txt is honoured." %}, unconditional); plugins/fetching.py:154-155 (if site.capture_ignore_robots(): return True); the policy in core/site.py:154-158; CaptureCreateView._render in jobs/capture_views.py:87-91 passes only form, sources and known.
What happens: on an instance where the administrator chose to ignore robots.txt, a person captures a page from a site that asks automated clients to stay away, having been told by the page that Postulo would not fetch it.
Why it matters: the statement is about what the server does on the person's behalf towards third-party sites, and it should be true on the instance that shows it. The Server settings page already reports the effective value correctly (templates/server/capture.html:25-29). The neighbouring line about private addresses is accurate, since capture always uses public_only_client.
Fix: pass site.capture_ignore_robots() into the capture form's context and choose between two sentences: the current one, and one saying this instance has been set not to consult robots.txt.

Done when

A test in tests/test_capture.py sets the stored policy to ignore robots.txt, renders the capture form and asserts the "honoured" sentence is absent and the other present; it fails today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, comparing the capture page's help text with the fetching code. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/fetching.py:110-111` (the early return in `robots_allow`) and `core/site.py:131-135` (`capture_ignore_robots`); the template line and `CaptureCreateView._render` have not moved. Under *How this works* the capture form tells everybody that the site's robots.txt is honoured. Since robots.txt became a Server settings switch (and `POSTULO_CAPTURE_IGNORE_ROBOTS`), an instance can turn that off, and then `robots_allow` answers yes without fetching anything. The page then says the opposite of what the instance does. - **Where:** `templates/jobs/capture_form.html:85` (`{% translate "The site's robots.txt is honoured." %}`, unconditional); `plugins/fetching.py:154-155` (`if site.capture_ignore_robots(): return True`); the policy in `core/site.py:154-158`; `CaptureCreateView._render` in `jobs/capture_views.py:87-91` passes only `form`, `sources` and `known`. - **What happens:** on an instance where the administrator chose to ignore robots.txt, a person captures a page from a site that asks automated clients to stay away, having been told by the page that Postulo would not fetch it. - **Why it matters:** the statement is about what the server does on the person's behalf towards third-party sites, and it should be true on the instance that shows it. The Server settings page already reports the effective value correctly (`templates/server/capture.html:25-29`). The neighbouring line about private addresses is accurate, since capture always uses `public_only_client`. - **Fix:** pass `site.capture_ignore_robots()` into the capture form's context and choose between two sentences: the current one, and one saying this instance has been set not to consult robots.txt. ## Done when - A test in `tests/test_capture.py` sets the stored policy to ignore robots.txt, renders the capture form and asserts the "honoured" sentence is absent and the other present; it fails today.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:31 +00:00

tiagoagueda added the

interface

bug

tier

labels

2026-10-01 10:48:31 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 07:56:40 +00:00

The capture form states what this instance does about robots.txt

tiagoagueda closed this issue