Plugin upgrade and rollback: the old code keeps running until a restart, though the page says none is needed #599

New issue

Closed

opened 2026-10-01 10:48:10 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:48:10 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how the registry loads plugins and checking EntryPoint.load() against a module rewritten on disk.

Related to #228.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at plugins/installing.py:1072-1088 (_forget_metadata_cache), core/server_views.py:1395-1402 and plugins/views.py:113-114; plugins/registry.py:190-194 and :237-271 and plugins/management/commands/plugins.py:92-95 have not moved. catch_up now takes activate and record_stamp from plugins/record.py (#248), and nothing in src touches sys.modules.

Every process loads a plugin with entry_point.load(), which imports its module. After an upgrade or a plugins rollback, catch_up() and _forget_metadata_cache() rebuild the registry, but the plugin's modules are still in sys.modules, so load() hands back the old classes. Nothing in src evicts them. The upload confirmation and the management command nevertheless say the new version is in use everywhere and that a restart would add nothing.

Where: plugins/registry.py:190-194 (_load_third_party: plugin = entry_point.load()()), plugins/registry.py:237-271 (catch_up, which clears the cache and the metadata finder but not sys.modules), plugins/installing.py:1246-1262 (_forget_metadata_cache), core/server_views.py:1394-1401 ("every part of this instance is using it already. A restart would add nothing"), plugins/management/commands/plugins.py:92-95 ("It is in use everywhere already"), and the comment at plugins/views.py:112-113 ("reinstalling restarts the process").
What happens: an administrator uploads 1.1 of a notifier to fix a bug. Every web worker and the scheduler keep running 1.0's top-level module until they are restarted or retired by gunicorn's --max-requests; the scheduler is never retired. Any submodule the plugin imports for the first time after the upgrade (apprise imports lazily) comes from 1.1's files, so a process can run a mixture of both versions. Rolling back a broken upgrade with plugins rollback likewise leaves the broken code loaded. Checked: an EntryPoint for probe_mod:P loaded version 1.0; after the file was rewritten as 2.0 and importlib.invalidate_caches() called, load() still returned 1.0.
Why it matters: the administrator is told a fix is live when it is not, and may see failures that match neither version.
Fix: either evict the plugin's modules before the registry rebuilds (when the record stamp shows a plugin's version changed or it was rolled back, drop the top-level packages listed in its dist-info RECORD, or found through importlib.metadata.packages_distributions(), from sys.modules), or tell the truth: when the record already had a line for that name, say a restart is needed, and keep the "no restart" wording for first installs.

Done when

tests/test_plugin_rollback.py has a test that installs version 1.0 of a wheel, loads its source through registry.plugins("source"), installs 2.0, and asserts that the registry's instance reports 2.0 (or, with the second fix, that the confirmation for an upgrade says a restart is needed).

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how the registry loads plugins and checking `EntryPoint.load()` against a module rewritten on disk. Related to #228. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `plugins/installing.py:1072-1088` (`_forget_metadata_cache`), `core/server_views.py:1395-1402` and `plugins/views.py:113-114`; `plugins/registry.py:190-194` and `:237-271` and `plugins/management/commands/plugins.py:92-95` have not moved. `catch_up` now takes `activate` and `record_stamp` from `plugins/record.py` (#248), and nothing in `src` touches `sys.modules`. Every process loads a plugin with `entry_point.load()`, which imports its module. After an upgrade or a `plugins rollback`, `catch_up()` and `_forget_metadata_cache()` rebuild the registry, but the plugin's modules are still in `sys.modules`, so `load()` hands back the old classes. Nothing in `src` evicts them. The upload confirmation and the management command nevertheless say the new version is in use everywhere and that a restart would add nothing. - **Where:** `plugins/registry.py:190-194` (`_load_third_party`: `plugin = entry_point.load()()`), `plugins/registry.py:237-271` (`catch_up`, which clears the cache and the metadata finder but not `sys.modules`), `plugins/installing.py:1246-1262` (`_forget_metadata_cache`), `core/server_views.py:1394-1401` ("every part of this instance is using it already. A restart would add nothing"), `plugins/management/commands/plugins.py:92-95` ("It is in use everywhere already"), and the comment at `plugins/views.py:112-113` ("reinstalling restarts the process"). - **What happens:** an administrator uploads 1.1 of a notifier to fix a bug. Every web worker and the scheduler keep running 1.0's top-level module until they are restarted or retired by gunicorn's `--max-requests`; the scheduler is never retired. Any submodule the plugin imports for the first time after the upgrade (apprise imports lazily) comes from 1.1's files, so a process can run a mixture of both versions. Rolling back a broken upgrade with `plugins rollback` likewise leaves the broken code loaded. Checked: an `EntryPoint` for `probe_mod:P` loaded version 1.0; after the file was rewritten as 2.0 and `importlib.invalidate_caches()` called, `load()` still returned 1.0. - **Why it matters:** the administrator is told a fix is live when it is not, and may see failures that match neither version. - **Fix:** either evict the plugin's modules before the registry rebuilds (when the record stamp shows a plugin's version changed or it was rolled back, drop the top-level packages listed in its dist-info `RECORD`, or found through `importlib.metadata.packages_distributions()`, from `sys.modules`), or tell the truth: when the record already had a line for that name, say a restart is needed, and keep the "no restart" wording for first installs. ## Done when - `tests/test_plugin_rollback.py` has a test that installs version 1.0 of a wheel, loads its source through `registry.plugins("source")`, installs 2.0, and asserts that the registry's instance reports 2.0 (or, with the second fix, that the confirmation for an upgrade says a restart is needed).

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:10 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:10 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:57 +00:00

Say a restart is needed after a plugin upgrade or rollback

tiagoagueda closed this issue

2026-10-04 10:04:57 +00:00