Capture: robots.txt is read only on the first host, and a robots.txt that redirects counts as allowing everything #364

New issue

Closed

opened 2026-10-01 10:46:09 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:09 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the page-fetching code and running robots_allow against a mock transport.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/fetching.py:100-136 (robots_allow) and :163-194 (fetch_page, with the robots check at :171).

fetch_page asks robots_allow once, for the URL it was given, with the client it opened with follow_redirects=False. A robots.txt that answers with a redirect (http to https, the bare domain to www., both very common) therefore comes back as a 301 or 302, and every status other than 200 means "allowed". And when the posting URL itself redirects to another host, that host's robots.txt is never read. The module docstring says "robots.txt is honoured", and the Server settings page offers "Ignore robots.txt when capturing" as though it were.

Where: plugins/fetching.py:144-180 (robots_allow: if response.status_code != 200: return True); plugins/fetching.py:208-239 (fetch_page: the client is opened with follow_redirects=False, and the robots check at :216 sits before the for _hop in range(MAX_REDIRECTS + 1) loop, which changes current to another host without checking again).
What happens: checked on the snapshot: with https://example.org/robots.txt answering 301 Location: https://www.example.org/robots.txt, and that file disallowing /careers, robots_allow("https://example.org/careers/1", client=…) returns True after one request; the redirect target is never fetched. In the same way, a posting at https://short.example/123 that redirects to https://jobs.example.com/careers/123 is fetched without jobs.example.com/robots.txt being consulted.
Why it matters: the instance says it honours robots.txt, and an administrator who leaves "Ignore robots.txt" off is relying on that; on many real sites the check never reads the file at all. RFC 9309 §2.3.1.2 asks a client to follow at least five redirects for robots.txt. The robots_allow docstring also says a missing, unreachable or unparseable file means yes "which is what the standard says"; RFC 9309 §2.3.1.4 says the opposite for a server error (5xx): assume complete disallow. Either follow it or stop citing the standard for that case.
Fix: in robots_allow, follow up to MAX_REDIRECTS redirects for robots.txt by hand through the same guarded client (each hop goes through the public-only hook already), and parse the final 200. In fetch_page, when a redirect changes the scheme, host or port, call robots_allow again for the new URL before requesting it. Correct the docstring's claim about the standard, or treat 5xx as disallow.

Done when

tests/test_capture.py has a test where robots.txt redirects to a file that disallows the path and robots_allow returns False, and a test where the posting redirects to a second host whose robots.txt disallows it and fetch_page raises RobotsDisallowed.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the page-fetching code and running `robots_allow` against a mock transport. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/fetching.py:100-136` (`robots_allow`) and `:163-194` (`fetch_page`, with the robots check at `:171`). `fetch_page` asks `robots_allow` once, for the URL it was given, with the client it opened with `follow_redirects=False`. A `robots.txt` that answers with a redirect (http to https, the bare domain to `www.`, both very common) therefore comes back as a 301 or 302, and every status other than 200 means "allowed". And when the posting URL itself redirects to another host, that host's `robots.txt` is never read. The module docstring says "`robots.txt` is honoured", and the Server settings page offers "Ignore robots.txt when capturing" as though it were. - **Where:** `plugins/fetching.py:144-180` (`robots_allow`: `if response.status_code != 200: return True`); `plugins/fetching.py:208-239` (`fetch_page`: the client is opened with `follow_redirects=False`, and the robots check at `:216` sits before the `for _hop in range(MAX_REDIRECTS + 1)` loop, which changes `current` to another host without checking again). - **What happens:** checked on the snapshot: with `https://example.org/robots.txt` answering `301 Location: https://www.example.org/robots.txt`, and that file disallowing `/careers`, `robots_allow("https://example.org/careers/1", client=…)` returns `True` after one request; the redirect target is never fetched. In the same way, a posting at `https://short.example/123` that redirects to `https://jobs.example.com/careers/123` is fetched without `jobs.example.com/robots.txt` being consulted. - **Why it matters:** the instance says it honours `robots.txt`, and an administrator who leaves "Ignore robots.txt" off is relying on that; on many real sites the check never reads the file at all. RFC 9309 §2.3.1.2 asks a client to follow at least five redirects for `robots.txt`. The `robots_allow` docstring also says a missing, unreachable or unparseable file means yes "which is what the standard says"; RFC 9309 §2.3.1.4 says the opposite for a server error (5xx): assume complete disallow. Either follow it or stop citing the standard for that case. - **Fix:** in `robots_allow`, follow up to `MAX_REDIRECTS` redirects for `robots.txt` by hand through the same guarded client (each hop goes through the public-only hook already), and parse the final 200. In `fetch_page`, when a redirect changes the scheme, host or port, call `robots_allow` again for the new URL before requesting it. Correct the docstring's claim about the standard, or treat 5xx as disallow. ## Done when - `tests/test_capture.py` has a test where `robots.txt` redirects to a file that disallows the path and `robots_allow` returns `False`, and a test where the posting redirects to a second host whose `robots.txt` disallows it and `fetch_page` raises `RobotsDisallowed`.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:46:09 +00:00

tiagoagueda added the

tier

bug

labels

2026-10-01 10:46:09 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 09:26:54 +00:00

Follow a redirected robots.txt and ask each redirect target's own robots.txt

tiagoagueda closed this issue

2026-10-04 09:26:54 +00:00