Plugins pages: every render fetches every catalogue index and signature over the network #602

New issue

Closed

opened 2026-10-01 10:48:11 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:11 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by following where installing.status() gets its provenance and counting the calls in a test.

Checked again on 2026-10-01 against main at 0a5cad587: still present. status() moved from plugins/installing.py to plugins/provenance.py:170-230 (#248), so the code is now at plugins/provenance.py:212-213 (digests = signed_digests() if entries else {}), with the callers at plugins/policy.py:274, accounts/settings_views.py:243 and :251, core/server_views.py:961 and plugins/management/commands/plugins.py:55 (plugins list); signed_digests() (plugins/provenance.py:105-128) and plugins/catalogue.py have not moved.

When at least one plugin is installed and at least one repository is configured, every view of a person's Settings → Plugins and of Server settings → Plugins downloads and verifies every configured catalogue, inside the request. A page that should cost a file read costs two HTTPS round trips per repository, and it breaks the promise in plugins/catalogue.py:22-23: "An index is fetched when an administrator opens the page and asks, and never in the background: Postulo makes no request nobody asked for."

Where: plugins/installing.py:1301-1302 (status(): digests = provenance.signed_digests() if entries else {}); plugins/provenance.py:105-128 (signed_digests() calls catalogue.fetch_all() at :120); plugins/catalogue.py:220-244 (fetch, fetch_all: the index and its .sig, one after the other, per repository, with the 10-second default timeout of http.client()); callers plugins/policy.py:283 (overview()), accounts/settings_views.py:241 (GET) and :249 (POST), core/server_views.py:960 (Server → Plugins).
What happens: with one plugin installed and one repository configured, each signed-in person who opens Settings → Plugins makes Postulo fetch the index (up to 2 MB) and its signature from the repository's host. Saving the page does it twice: once in post() through overview(internal=True), and again on the redirected GET. A test that replaced catalogue.fetch_all with a counter saw one call per overview() and one per GET of /settings/plugins/. With a slow or unreachable host each render waits up to the timeouts for each repository in turn and holds a gunicorn worker for that long.
Why it matters: the page a person visits to switch their own plugins is slowed, or stalled, by a third-party host they have never heard of; the repository host learns each time somebody on the instance opens their plugin settings; and the work is wasted, because the answer only changes when the record or a repository changes. For a person who is not an administrator the digests are not even shown, since the row lookup misses (see the issue about marks.get(plugin.name)).
Fix: stop fetching while rendering. Keep the verified digests from the last explicit Refresh or catalogue install (core/server_views.py _refresh, catalogue.install) in the cache, keyed by repository and its key, and have signed_digests() read only that; with nothing cached the answer stays uploaded, as it already is when a repository cannot be reached. Better still, decide an upload's provenance once at install time and store it on the record line, so status() needs no catalogue at all.

Related to #231 (closed), whose point 6 asked for provenance "cached per record mtime": the record is cached, but the digests it is compared with are still fetched over the network on every render.

Done when

A test in tests/test_plugin_provenance.py that installs a record line, configures a repository, replaces catalogue.fetch_all with a counter, and asserts that GET and POST of settings:plugins and GET of server:plugins call it zero times; and one that a Refresh followed by a page view still labels a matching upload as Official or Custom.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, by following where `installing.status()` gets its provenance and counting the calls in a test. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `status()` moved from `plugins/installing.py` to `plugins/provenance.py:170-230` (#248), so the code is now at `plugins/provenance.py:212-213` (`digests = signed_digests() if entries else {}`), with the callers at `plugins/policy.py:274`, `accounts/settings_views.py:243` and `:251`, `core/server_views.py:961` and `plugins/management/commands/plugins.py:55` (`plugins list`); `signed_digests()` (`plugins/provenance.py:105-128`) and `plugins/catalogue.py` have not moved. When at least one plugin is installed and at least one repository is configured, every view of a person's *Settings → Plugins* and of *Server settings → Plugins* downloads and verifies every configured catalogue, inside the request. A page that should cost a file read costs two HTTPS round trips per repository, and it breaks the promise in `plugins/catalogue.py:22-23`: "An index is fetched when an administrator opens the page and asks, and never in the background: Postulo makes no request nobody asked for." - **Where:** `plugins/installing.py:1301-1302` (`status()`: `digests = provenance.signed_digests() if entries else {}`); `plugins/provenance.py:105-128` (`signed_digests()` calls `catalogue.fetch_all()` at :120); `plugins/catalogue.py:220-244` (`fetch`, `fetch_all`: the index and its `.sig`, one after the other, per repository, with the 10-second default timeout of `http.client()`); callers `plugins/policy.py:283` (`overview()`), `accounts/settings_views.py:241` (GET) and `:249` (POST), `core/server_views.py:960` (Server → Plugins). - **What happens:** with one plugin installed and one repository configured, each signed-in person who opens *Settings → Plugins* makes Postulo fetch the index (up to 2 MB) and its signature from the repository's host. Saving the page does it twice: once in `post()` through `overview(internal=True)`, and again on the redirected GET. A test that replaced `catalogue.fetch_all` with a counter saw one call per `overview()` and one per GET of `/settings/plugins/`. With a slow or unreachable host each render waits up to the timeouts for each repository in turn and holds a gunicorn worker for that long. - **Why it matters:** the page a person visits to switch their own plugins is slowed, or stalled, by a third-party host they have never heard of; the repository host learns each time somebody on the instance opens their plugin settings; and the work is wasted, because the answer only changes when the record or a repository changes. For a person who is not an administrator the digests are not even shown, since the row lookup misses (see the issue about `marks.get(plugin.name)`). - **Fix:** stop fetching while rendering. Keep the verified digests from the last explicit *Refresh* or catalogue install (`core/server_views.py` `_refresh`, `catalogue.install`) in the cache, keyed by repository and its key, and have `signed_digests()` read only that; with nothing cached the answer stays *uploaded*, as it already is when a repository cannot be reached. Better still, decide an upload's provenance once at install time and store it on the record line, so `status()` needs no catalogue at all. Related to #231 (closed), whose point 6 asked for provenance "cached per record mtime": the record is cached, but the digests it is compared with are still fetched over the network on every render. ## Done when - A test in `tests/test_plugin_provenance.py` that installs a record line, configures a repository, replaces `catalogue.fetch_all` with a counter, and asserts that GET and POST of `settings:plugins` and GET of `server:plugins` call it zero times; and one that a *Refresh* followed by a page view still labels a matching upload as *Official* or *Custom*.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:11 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:11 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:58 +00:00

Stop fetching the catalogues to draw the Plugins pages

tiagoagueda closed this issue

2026-10-04 10:04:58 +00:00

tiagoagueda referenced this issue from a commit