manage.py work: a stop signal during a task is lost, and an interrupted errand shows as working for a week #474

New issue

Closed

opened 2026-10-01 10:46:57 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:57 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the worker loop against django_tasks_db 0.13.0's Worker.

Related to #247.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

manage.py work wraps django_tasks_db's Worker in its own while True loop. When SIGTERM arrives while a task is running, upstream's Worker.shutdown only sets running = False and lets the task finish; run() then returns. The loop does not look at that: it writes the heartbeat, sleeps, builds a fresh Worker with running = True and installs its signal handlers again. The graceful stop is forgotten, and the container is killed with SIGKILL at the end of its grace period, often in the middle of the next task. Nothing then moves that errand on.

Where: core/management/commands/work.py:60-81 (Command.handle); core/errands.py:133 (perform sets WORKING before the handler runs); core/errands.py:169-179 (forget_old deletes by age only); upstream django_tasks_db/management/commands/db_worker.py, Worker.shutdown and Worker.run.
What happens: an operator runs docker compose restart worker, or upgrades the stack, while a PDF render, capture or export is in progress. The entrypoint execs the command, so Python receives SIGTERM; the current task finishes, and the worker then carries on taking work until Docker sends SIGKILL. A task running at that moment leaves its errand in WORKING and its DBTaskResult in RUNNING, neither of which anything revisits. The person's page polls every few seconds and says "You can leave this page. The work carries on without you." until forget_old removes the errand seven days later. Any other end of the worker mid-task (an out-of-memory kill during a Chromium render) leaves the same state.
Why it matters: a restart that should take seconds always waits out the grace period and can cut work off half way, and the person is told the work is carrying on when it never will; they have no way to learn that it failed or to try again from that page.
Fix: in work.py, stop the loop when the finished worker's running is false (if not worker.running: return after worker.run()), or install the command's own handler that sets a flag the loop checks before building the next Worker. Separately, have the scheduler mark errands still WAITING or WORKING past a generous deadline as FAILED, with a sentence saying the work was interrupted, so the page stops polling.

Done when

A test in tests/test_slow_work.py makes a handler send the process's own shutdown handler a SIGTERM while it runs, and asserts that call_command("work") returns after that task instead of starting another pass.
A test in tests/test_errands.py ages a WORKING errand past the deadline, runs the scheduler's pass, and asserts that it is FAILED with a readable message.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the worker loop against `django_tasks_db` 0.13.0's `Worker`. Related to #247. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. `manage.py work` wraps `django_tasks_db`'s `Worker` in its own `while True` loop. When SIGTERM arrives while a task is running, upstream's `Worker.shutdown` only sets `running = False` and lets the task finish; `run()` then returns. The loop does not look at that: it writes the heartbeat, sleeps, builds a fresh `Worker` with `running = True` and installs its signal handlers again. The graceful stop is forgotten, and the container is killed with SIGKILL at the end of its grace period, often in the middle of the next task. Nothing then moves that errand on. - **Where:** `core/management/commands/work.py:60-81` (`Command.handle`); `core/errands.py:133` (`perform` sets `WORKING` before the handler runs); `core/errands.py:169-179` (`forget_old` deletes by age only); upstream `django_tasks_db/management/commands/db_worker.py`, `Worker.shutdown` and `Worker.run`. - **What happens:** an operator runs `docker compose restart worker`, or upgrades the stack, while a PDF render, capture or export is in progress. The entrypoint `exec`s the command, so Python receives SIGTERM; the current task finishes, and the worker then carries on taking work until Docker sends SIGKILL. A task running at that moment leaves its errand in `WORKING` and its `DBTaskResult` in `RUNNING`, neither of which anything revisits. The person's page polls every few seconds and says "You can leave this page. The work carries on without you." until `forget_old` removes the errand seven days later. Any other end of the worker mid-task (an out-of-memory kill during a Chromium render) leaves the same state. - **Why it matters:** a restart that should take seconds always waits out the grace period and can cut work off half way, and the person is told the work is carrying on when it never will; they have no way to learn that it failed or to try again from that page. - **Fix:** in `work.py`, stop the loop when the finished worker's `running` is false (`if not worker.running: return` after `worker.run()`), or install the command's own handler that sets a flag the loop checks before building the next `Worker`. Separately, have the scheduler mark errands still `WAITING` or `WORKING` past a generous deadline as `FAILED`, with a sentence saying the work was interrupted, so the page stops polling. ## Done when - A test in `tests/test_slow_work.py` makes a handler send the process's own shutdown handler a SIGTERM while it runs, and asserts that `call_command("work")` returns after that task instead of starting another pass. - A test in `tests/test_errands.py` ages a `WORKING` errand past the deadline, runs the scheduler's pass, and asserts that it is `FAILED` with a readable message.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:57 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:46:57 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 10:04:58 +00:00

Stop the worker after a stop signal and fail errands it never finished

tiagoagueda closed this issue

2026-10-04 10:04:58 +00:00

tiagoagueda referenced this issue