Background worker: keeps the first policy row it reads until it restarts, so Server settings changes never reach it #473

New issue

Closed

opened 2026-10-01 10:46:57 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:57 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following where the memoised policy row is cleared.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at core/memo.py:25-54 (the two memos, forget_current with its docstring, and forget_decisions, moved there by #248 and still handed out by core/site.py and plugins/policy.py), core/site.py:110-121 (current) and plugins/policy.py:146-155 (_answers). The callers of forget_current are now core/middleware.py:118, core/models.py:1082 and notifications/management/commands/send_due_reminders.py:184, and the readers named below are at jobs/pages.py:155-156 and plugins/fetching.py:110.

site.current() memoises the SiteSettings row in an asgiref Local, and plugins.policy memoises its plugin decisions the same way. Both are cleared at the start of each request, when the row is saved in the same process, and at the top of each scheduler pass. The manage.py work process added by #247 is a fourth long-lived process and clears neither, so the first errand that reads the policy fixes it for every later errand the worker runs, until the container restarts.

Where: core/site.py:117-144 (current, forget_current, whose docstring names three places the row can change and misses the worker); core/errands.py:116-151 (perform); core/management/commands/work.py:60-81 (the loop around django_tasks_db's Worker, which runs every task in the same thread); plugins/policy.py:141-153 (_memo, forget_decisions). The only callers of forget_current are core/middleware.py:118, core/models.py:1081 and notifications/management/commands/send_due_reminders.py:182.
What happens: with POSTULO_BACKGROUND_WORK=true and the worker running, an administrator switches off keeping the source or rendering of captured pages, switches off ignoring robots.txt, or changes plugin availability. Captures fetched by the worker keep reading the old answers from site.capture_keep_source() / capture_keep_rendering() (jobs/pages.py:150-155) and site.capture_ignore_robots() (plugins/fetching.py:154), and notification delivery keeps the old plugin decisions. The row the web container saved is a different process's memo; nothing in the worker drops its own.
Why it matters: the instance's half of the capture-keeping switch is described as final (jobs/pages.py), yet after an administrator says no the worker goes on storing copies of other people's pages, and goes on ignoring robots.txt, for as long as it runs. The same instance without a worker obeys at once, so the behaviour depends on how it was deployed.
Fix: call site.forget_current() and policy.forget_decisions() at the top of errands.perform(), which covers the worker and the inline path alike; add the worker to forget_current's docstring.

Done when

tests/test_errands.py runs two errands.perform() calls outside a request, with SiteSettings.objects.filter(pk=1).update(capture_keep_source=False) between them, and asserts that a handler reading site.capture_keep_source() sees the new value in the second.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, following where the memoised policy row is cleared. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `core/memo.py:25-54` (the two memos, `forget_current` with its docstring, and `forget_decisions`, moved there by #248 and still handed out by `core/site.py` and `plugins/policy.py`), `core/site.py:110-121` (`current`) and `plugins/policy.py:146-155` (`_answers`). The callers of `forget_current` are now `core/middleware.py:118`, `core/models.py:1082` and `notifications/management/commands/send_due_reminders.py:184`, and the readers named below are at `jobs/pages.py:155-156` and `plugins/fetching.py:110`. `site.current()` memoises the `SiteSettings` row in an asgiref `Local`, and `plugins.policy` memoises its plugin decisions the same way. Both are cleared at the start of each request, when the row is saved in the same process, and at the top of each scheduler pass. The `manage.py work` process added by #247 is a fourth long-lived process and clears neither, so the first errand that reads the policy fixes it for every later errand the worker runs, until the container restarts. - **Where:** `core/site.py:117-144` (`current`, `forget_current`, whose docstring names three places the row can change and misses the worker); `core/errands.py:116-151` (`perform`); `core/management/commands/work.py:60-81` (the loop around `django_tasks_db`'s `Worker`, which runs every task in the same thread); `plugins/policy.py:141-153` (`_memo`, `forget_decisions`). The only callers of `forget_current` are `core/middleware.py:118`, `core/models.py:1081` and `notifications/management/commands/send_due_reminders.py:182`. - **What happens:** with `POSTULO_BACKGROUND_WORK=true` and the worker running, an administrator switches off keeping the source or rendering of captured pages, switches off ignoring `robots.txt`, or changes plugin availability. Captures fetched by the worker keep reading the old answers from `site.capture_keep_source()` / `capture_keep_rendering()` (`jobs/pages.py:150-155`) and `site.capture_ignore_robots()` (`plugins/fetching.py:154`), and notification delivery keeps the old plugin decisions. The row the web container saved is a different process's memo; nothing in the worker drops its own. - **Why it matters:** the instance's half of the capture-keeping switch is described as final (`jobs/pages.py`), yet after an administrator says no the worker goes on storing copies of other people's pages, and goes on ignoring `robots.txt`, for as long as it runs. The same instance without a worker obeys at once, so the behaviour depends on how it was deployed. - **Fix:** call `site.forget_current()` and `policy.forget_decisions()` at the top of `errands.perform()`, which covers the worker and the inline path alike; add the worker to `forget_current`'s docstring. ## Done when - `tests/test_errands.py` runs two `errands.perform()` calls outside a request, with `SiteSettings.objects.filter(pk=1).update(capture_keep_source=False)` between them, and asserts that a handler reading `site.capture_keep_source()` sees the new value in the second.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:57 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:46:57 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:58 +00:00

Test that each errand reads the instance settings afresh

tiagoagueda closed this issue

2026-10-04 10:04:58 +00:00