Postulo/postulo

Watch

Fork 0

Outbound requests: the documents say capture is the only one, and the container fetches GeoNames unasked from a fixed path #366

New issue

Closed

opened 2026-10-01 10:46:10 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:46:10 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d and postulo.wiki at de1f52467, comparing what the documents say leaves the instance with the callers of postulo.plugins.http and core/destinations.py, the container entrypoint and the provisioning commands.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at docs/PLAN.md:256 (the No telemetry line), plugins/fetching.py:163 (the capture fetch) and :123 (its robots.txt), and core/mail.py:180 and :240 (the two destinations.approve calls, the second in GuardedBackend); the entrypoint, the two provisioning commands and the other locations have not moved.

SECURITY.md, the README, docs/PLAN.md and the wiki promise that Postulo makes no outbound request except a capture a person triggers. The core makes about a dozen kinds of outbound request, several to addresses a person typed, and the container downloads GeoNames' tables at first start with nobody asking, through a client of its own rather than the guarded one. The entrypoint also tests a fixed path for those tables, so an operator who moves POSTULO_GEOLOCATIONS_DIR gets the download again on every start. An operator who writes egress rules or a privacy notice from these documents gets both wrong.

1. The documents name capture as the only outbound request

Where:
- SECURITY.md:75-76: "Postulo makes no outbound network calls except URL captures you explicitly trigger."
- SECURITY.md:32-33: "Nothing is fetched from the network unless a person asked for it".
- README.md:100 and docs/PLAN.md:243, the same sentence in other words.
- Hardening.md:192-194 in the wiki: capture "is the only thing here that makes your server issue an outbound request to an address somebody else supplied".
- Configuration.md:654 in the wiki, the POSTULO_CAPTURE_RATE row, which says the same.
- Installing-Postulo.md:368-369 in the wiki: "Postulo makes no request on your behalf unless you say so".
- Configuration.md:104-105 in the wiki is the one place that mentions the download at first start.
What happens: besides capture (plugins/fetching.py:208, and its robots.txt at :167), the core
- checks the links of a CV (resume/links.py:68);
- fetches a logo from a typed address and through Find logo (jobs/logos.py:120);
- delivers webhooks (plugins/webhook/__init__.py:241) and web push (plugins/browser/webpush.py:222);
- dials the instance's mail server and each person's own (core/mail.py:190, and GuardedBackend at :250, which plugins/own_mail/__init__.py:192 builds for a person's server);
- exchanges and refreshes OAuth tokens (plugins/consent.py:238);
- reads plugin catalogues and downloads plugin releases (plugins/catalogue.py:228, :296);
- fetches a Gravatar picture for a person who opted in, which sends a hash of their address (accounts/avatars.py:119);
- asks the update source once a day when POSTULO_UPDATE_CHECK is on (core/updates.py:51);
- talks to the single sign-on provider when one is configured;
- and, with no person behind it, downloads GeoNames at first start and reinstalls plugin packages on plugins sync after an upgrade (docker/entrypoint.sh:19, :26-30).
Link checks, logo addresses, webhook receivers and own-mail servers are all addresses a person typed, so capture is not the only request to "an address somebody else supplied" either.
Why it matters: these are the documents an operator reads to decide what the instance may reach. Firewalling on their word breaks webhooks, push and mail sign-in; a privacy notice written from them understates what reaches third parties; and an instance on a monitored or metered network contacts download.geonames.org with no explanation anywhere but one wiki paragraph. Hardening already has a section, "Where the server is allowed to dial" (Hardening.md:407), that describes the guards but not the list.
Fix: replace the SECURITY.md bullet, the README line and the PLAN.md line with "no telemetry, and outbound requests only for the features listed in Hardening", and reword SECURITY.md:32-33 to name what is fetched without a person asking (GeoNames at first start, plugin packages on sync after an upgrade, deliveries to connections a person set up, the opt-in update check). Add to Hardening → Where the server is allowed to dial a short table of every outbound destination: what triggers it, whose address it is, which guard applies, and which setting turns it off or bounds it. Correct Hardening.md:192-194 and the POSTULO_CAPTURE_RATE row so they no longer call capture the only fetch of a chosen address (a separate finding of this review covers link checks and logos spending no allowance), and the sentence in Installing Postulo.

2. The GeoNames download at start is unasked and skips the guarded client

Where: docker/entrypoint.sh:22-30; jobs/management/commands/fetch_geonames.py:55-61, handle; jobs/management/commands/fetch_esco.py:136, the same pattern; docs/THREAT-MODEL.md:74-77, rule 5.
What happens: on the first start the entrypoint finds no city table and runs manage.py fetch_geonames, which makes two requests through httpx.Client(..., follow_redirects=True): no address check, no pinning, redirects followed wherever they lead. The block has no switch of the kind migrations and plugin sync have (POSTULO_SKIP_MIGRATE, POSTULO_SKIP_PLUGIN_SYNC), so when the web container's fetch fails, the scheduler and the worker each try again at their own start. Rule 5 of the threat model asks every new outbound request to go through postulo.plugins.http.client(); fetch_geonames and fetch_esco are the two that do not, and the first is the one that runs with no person behind it.
Why it matters: the addresses are fixed, so the exposure is small, but the one automatic fetch is also one of the two exempt from the rule every other request obeys, and an operator cannot turn it off.
Fix: build the client in both commands with postulo.plugins.http.public_only_client() (it passes follow_redirects and headers through), or name operator-run provisioning commands as an exception in rule 5. Either make the start-up fetch opt-in or give it a switch (for example POSTULO_SKIP_GEONAMES=1, documented in .env.example beside how to provide the two files by hand), and say which in the documents of section 1.

3. The entrypoint tests a fixed path, not `POSTULO_GEOLOCATIONS_DIR`

Where: docker/entrypoint.sh:26-27 tests and creates /app/data/geonames; jobs/places.py:37-39 reads POSTULO_GEOLOCATIONS_DIR, which docker/Dockerfile:155 sets and Configuration.md:100 in the wiki documents as configurable; jobs/management/commands/fetch_geonames.py:137-140, _write, which does not create the directory it writes into.
What happens: an operator sets POSTULO_GEOLOCATIONS_DIR=/app/data/maps. places.DATA_DIR follows it and the entrypoint does not: every start of every container finds no file at the fixed path, creates /app/data/geonames, and downloads several megabytes again. If the new directory does not exist, _write fails on the missing parent after the download, the map stays empty, and the same happens at the next start.
Fix: test "${POSTULO_GEOLOCATIONS_DIR:-/app/data/geonames}/geonames-cities1000.txt" and mkdir -p the same directory; have _write create target.parent.

Done when

A test in tests/test_fetch_geonames.py fails while the command builds an httpx.Client of its own and passes once it uses the guarded client (or rule 5 names the exception).
A test in tests/test_shell_scripts.py fails while docker/entrypoint.sh names /app/data/geonames without POSTULO_GEOLOCATIONS_DIR, and a test in tests/test_fetch_geonames.py fails while _write cannot write into a directory that does not exist yet.
SECURITY.md, README.md, docs/PLAN.md, Hardening, Configuration and Installing Postulo no longer say capture is the only outbound request or that nothing is fetched unasked; Hardening lists every outbound destination the core makes, and the start-up download is named with how to turn it off.

Found on 2026-09-29 in a read-only review of postulo at 95348908d and postulo.wiki at de1f52467, comparing what the documents say leaves the instance with the callers of `postulo.plugins.http` and `core/destinations.py`, the container entrypoint and the provisioning commands. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `docs/PLAN.md:256` (the *No telemetry* line), `plugins/fetching.py:163` (the capture fetch) and `:123` (its `robots.txt`), and `core/mail.py:180` and `:240` (the two `destinations.approve` calls, the second in `GuardedBackend`); the entrypoint, the two provisioning commands and the other locations have not moved. `SECURITY.md`, the README, `docs/PLAN.md` and the wiki promise that Postulo makes no outbound request except a capture a person triggers. The core makes about a dozen kinds of outbound request, several to addresses a person typed, and the container downloads GeoNames' tables at first start with nobody asking, through a client of its own rather than the guarded one. The entrypoint also tests a fixed path for those tables, so an operator who moves `POSTULO_GEOLOCATIONS_DIR` gets the download again on every start. An operator who writes egress rules or a privacy notice from these documents gets both wrong. ## 1. The documents name capture as the only outbound request - **Where:** - `SECURITY.md:75-76`: "Postulo makes no outbound network calls except URL captures you explicitly trigger." - `SECURITY.md:32-33`: "Nothing is fetched from the network unless a person asked for it". - `README.md:100` and `docs/PLAN.md:243`, the same sentence in other words. - `Hardening.md:192-194` in the wiki: capture "is the only thing here that makes *your* server issue an outbound request to an address somebody else supplied". - `Configuration.md:654` in the wiki, the `POSTULO_CAPTURE_RATE` row, which says the same. - `Installing-Postulo.md:368-369` in the wiki: "Postulo makes no request on your behalf unless you say so". - `Configuration.md:104-105` in the wiki is the one place that mentions the download at first start. - **What happens:** besides capture (`plugins/fetching.py:208`, and its `robots.txt` at `:167`), the core - checks the links of a CV (`resume/links.py:68`); - fetches a logo from a typed address and through *Find logo* (`jobs/logos.py:120`); - delivers webhooks (`plugins/webhook/__init__.py:241`) and web push (`plugins/browser/webpush.py:222`); - dials the instance's mail server and each person's own (`core/mail.py:190`, and `GuardedBackend` at `:250`, which `plugins/own_mail/__init__.py:192` builds for a person's server); - exchanges and refreshes OAuth tokens (`plugins/consent.py:238`); - reads plugin catalogues and downloads plugin releases (`plugins/catalogue.py:228`, `:296`); - fetches a Gravatar picture for a person who opted in, which sends a hash of their address (`accounts/avatars.py:119`); - asks the update source once a day when `POSTULO_UPDATE_CHECK` is on (`core/updates.py:51`); - talks to the single sign-on provider when one is configured; - and, with no person behind it, downloads GeoNames at first start and reinstalls plugin packages on `plugins sync` after an upgrade (`docker/entrypoint.sh:19`, `:26-30`). Link checks, logo addresses, webhook receivers and own-mail servers are all addresses a person typed, so capture is not the only request to "an address somebody else supplied" either. - **Why it matters:** these are the documents an operator reads to decide what the instance may reach. Firewalling on their word breaks webhooks, push and mail sign-in; a privacy notice written from them understates what reaches third parties; and an instance on a monitored or metered network contacts `download.geonames.org` with no explanation anywhere but one wiki paragraph. *Hardening* already has a section, "Where the server is allowed to dial" (`Hardening.md:407`), that describes the guards but not the list. - **Fix:** replace the `SECURITY.md` bullet, the README line and the `PLAN.md` line with "no telemetry, and outbound requests only for the features listed in *Hardening*", and reword `SECURITY.md:32-33` to name what is fetched without a person asking (GeoNames at first start, plugin packages on sync after an upgrade, deliveries to connections a person set up, the opt-in update check). Add to *Hardening → Where the server is allowed to dial* a short table of every outbound destination: what triggers it, whose address it is, which guard applies, and which setting turns it off or bounds it. Correct `Hardening.md:192-194` and the `POSTULO_CAPTURE_RATE` row so they no longer call capture the only fetch of a chosen address (a separate finding of this review covers link checks and logos spending no allowance), and the sentence in *Installing Postulo*. ## 2. The GeoNames download at start is unasked and skips the guarded client - **Where:** `docker/entrypoint.sh:22-30`; `jobs/management/commands/fetch_geonames.py:55-61`, `handle`; `jobs/management/commands/fetch_esco.py:136`, the same pattern; `docs/THREAT-MODEL.md:74-77`, rule 5. - **What happens:** on the first start the entrypoint finds no city table and runs `manage.py fetch_geonames`, which makes two requests through `httpx.Client(..., follow_redirects=True)`: no address check, no pinning, redirects followed wherever they lead. The block has no switch of the kind migrations and plugin sync have (`POSTULO_SKIP_MIGRATE`, `POSTULO_SKIP_PLUGIN_SYNC`), so when the web container's fetch fails, the scheduler and the worker each try again at their own start. Rule 5 of the threat model asks every new outbound request to go through `postulo.plugins.http.client()`; `fetch_geonames` and `fetch_esco` are the two that do not, and the first is the one that runs with no person behind it. - **Why it matters:** the addresses are fixed, so the exposure is small, but the one automatic fetch is also one of the two exempt from the rule every other request obeys, and an operator cannot turn it off. - **Fix:** build the client in both commands with `postulo.plugins.http.public_only_client()` (it passes `follow_redirects` and `headers` through), or name operator-run provisioning commands as an exception in rule 5. Either make the start-up fetch opt-in or give it a switch (for example `POSTULO_SKIP_GEONAMES=1`, documented in `.env.example` beside how to provide the two files by hand), and say which in the documents of section 1. ## 3. The entrypoint tests a fixed path, not `POSTULO_GEOLOCATIONS_DIR` - **Where:** `docker/entrypoint.sh:26-27` tests and creates `/app/data/geonames`; `jobs/places.py:37-39` reads `POSTULO_GEOLOCATIONS_DIR`, which `docker/Dockerfile:155` sets and `Configuration.md:100` in the wiki documents as configurable; `jobs/management/commands/fetch_geonames.py:137-140`, `_write`, which does not create the directory it writes into. - **What happens:** an operator sets `POSTULO_GEOLOCATIONS_DIR=/app/data/maps`. `places.DATA_DIR` follows it and the entrypoint does not: every start of every container finds no file at the fixed path, creates `/app/data/geonames`, and downloads several megabytes again. If the new directory does not exist, `_write` fails on the missing parent after the download, the map stays empty, and the same happens at the next start. - **Fix:** test `"${POSTULO_GEOLOCATIONS_DIR:-/app/data/geonames}/geonames-cities1000.txt"` and `mkdir -p` the same directory; have `_write` create `target.parent`. ## Done when - A test in `tests/test_fetch_geonames.py` fails while the command builds an `httpx.Client` of its own and passes once it uses the guarded client (or rule 5 names the exception). - A test in `tests/test_shell_scripts.py` fails while `docker/entrypoint.sh` names `/app/data/geonames` without `POSTULO_GEOLOCATIONS_DIR`, and a test in `tests/test_fetch_geonames.py` fails while `_write` cannot write into a directory that does not exist yet. - `SECURITY.md`, `README.md`, `docs/PLAN.md`, *Hardening*, *Configuration* and *Installing Postulo* no longer say capture is the only outbound request or that nothing is fetched unasked; *Hardening* lists every outbound destination the core makes, and the start-up download is named with how to turn it off.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:46:10 +00:00

tiagoagueda added the

documentation

tier

bug

labels

2026-10-01 10:46:10 +00:00

tiagoagueda referenced this issue

2026-10-03 01:58:50 +00:00

Read a European Digital Credential: keep the file untouched, take what it states, and say plainly whether its seal was checked #670

tiagoagueda referenced this issue

2026-10-03 01:58:52 +00:00

A company's name in each language, taken from Wikidata, with a typed name only where Wikidata has none #673

tiagoagueda referenced this issue

2026-10-03 01:59:02 +00:00

Publications: suggest an entry from a DOI or from your ORCID record, only when asked, through the guarded client #688

tiagoagueda referenced this issue

2026-10-03 01:59:08 +00:00

Your career: citations, read as how often your published work has been cited, and when it was counted #697

tiagoagueda referenced this issue

2026-10-03 01:59:10 +00:00

Maps on OpenStreetMap data: choose the engine, the precision and where the tiles come from #700

tiagoagueda referenced this issue

2026-10-03 01:59:11 +00:00

Place a company on the map from its Wikidata coordinates #701

tiagoagueda referenced this issue

2026-10-03 01:59:12 +00:00

Place a postal address on the map: companies, contacts and the person's own, each only when asked #702

tiagoagueda referenced this issue

2026-10-03 01:59:13 +00:00

Docker Compose at home: one deployment page for a homelab, and the drift between the compose files and the wiki #704

tiagoagueda commented

2026-10-03 01:59:28 +00:00

Author

Owner

The maintainer's list of 2026-10-02 leads to new outbound lookups that the inventory here should name once they exist. None is built yet.

Each is meant to go through the guarded client (postulo.plugins.http, rule 5 at docs/THREAT-MODEL.md:74-77) with size, time and address limits, and each is opt-in, run by a button press and never in the background:

#673: a company's names in each language, from the Wikidata item the person recorded.
#675, phase 2: a company's identifiers compared with its Wikidata item.
#688: a publication suggested from a DOI or an ORCID.
#700: geocoding an address, only if the maintainer chooses it.

From the list of improvements of 2026-10-02.

The maintainer's list of 2026-10-02 leads to new outbound lookups that the inventory here should name once they exist. None is built yet. Each is meant to go through the guarded client (`postulo.plugins.http`, rule 5 at `docs/THREAT-MODEL.md:74-77`) with size, time and address limits, and each is opt-in, run by a button press and never in the background: - #673: a company's names in each language, from the Wikidata item the person recorded. - #675, phase 2: a company's identifiers compared with its Wikidata item. - #688: a publication suggested from a DOI or an ORCID. - #700: geocoding an address, only if the maintainer chooses it. From the list of improvements of 2026-10-02.

tiagoagueda referenced this issue from a commit

2026-10-04 09:26:54 +00:00

Build the GeoNames download on the guarded client, follow POSTULO_GEOLOCATIONS_DIR, and stop calling capture the only outbound request

tiagoagueda closed this issue

2026-10-04 09:26:54 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 09:26:55 +00:00

Refresh the catalogues' source line references after the last bunch of fixes