Document stores: removing a connection leaves its unsent copies failing for good, on every Send now #511

New issue

Closed

opened 2026-10-01 10:47:12 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:12 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the copy scheduler and the connection delete view, and confirmed by calling send_now() on the snapshot with a failed copy whose connection is gone.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at documents/models.py:843-850 (DocumentCopy.connection) and plugins/views.py:464-476 (ConnectionDeleteView); documents/archiving.py, core/metrics.py and core/importer.py have not moved.

DocumentCopy.connection is SET_NULL, so when a person removes a store connection, every copy that had not yet arrived stays behind with nothing to send it to. Nothing clears or closes those rows. Each Send to stores now resets their attempts and fails them again, and the scheduler picks them up again after each press. The document keeps a permanent "failed" line naming the old connection, and the operator's failure gauge counts it for good.

Where: documents/models.py:815-822 (DocumentCopy.connection, on_delete=models.SET_NULL), plugins/views.py:463-475 (ConnectionDeleteView, which does nothing about copies), documents/archiving.py:224-242 (send_now: every copy that is not SENT is retried, with copy.attempts = 0), documents/archiving.py:123-124 (send_copy fails a copy with no connection: "The connection is gone or switched off."), documents/archiving.py:173-189 (pending_copies deliberately keeps copies whose connection row is gone), core/metrics.py:249-250 (postulo_failures{kind="document_copies"} counts every FAILED copy).
What happens: a person's Paperless copy of a document failed (the store was down). They remove the connection and add it again — the removal page says "You can connect it again later" — or replace it with another store. schedule_copies gives the document a new copy for the new connection, but the old copy has connection=NULL and status failed. Every Send to stores now on that document then answers "%(sent)d copies sent, %(failed)d failed", or "The copy could not be sent; the document says why" once the new store already has the file, and the document shows a failed copy for a connection that no longer exists. Reproduced on the snapshot: three calls of send_now() on such a document each returned (0, 1), and the copy ended failed, attempts 1, due again five minutes later in pending_copies(). With no press, the scheduler dials it until MAX_ATTEMPTS and it then stays failed indefinitely.
Why it matters: the person cannot tell a real failure from a leftover one and has no way to clear it short of deleting the document; the messages after Send now are wrong; and an operator alerting on postulo_failures sees a failure that can never resolve.
Fix: when a connection is deleted, delete its copies that are not SENT (a pre_delete receiver on Connection, so the admin and account deletion are covered too, not only ConnectionDeleteView). Keep SENT copies, which are the references #130 and the importer (core/importer.py:81-100) mean to keep with no connection behind them. For rows already orphaned, a data migration deletes copies with connection IS NULL and a status other than sent; send_now and pending_copies should then also exclude connection__isnull=True rows that are not SENT, so an orphan can never be retried.

Done when

A test in tests/test_stores.py removes a store connection that has a failed and a sent copy, and asserts that the failed copy is gone, the sent copy remains with connection empty, and send_now() on the document afterwards returns (0, 0).

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the copy scheduler and the connection delete view, and confirmed by calling `send_now()` on the snapshot with a failed copy whose connection is gone. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `documents/models.py:843-850` (`DocumentCopy.connection`) and `plugins/views.py:464-476` (`ConnectionDeleteView`); `documents/archiving.py`, `core/metrics.py` and `core/importer.py` have not moved. `DocumentCopy.connection` is `SET_NULL`, so when a person removes a store connection, every copy that had not yet arrived stays behind with nothing to send it to. Nothing clears or closes those rows. Each *Send to stores now* resets their attempts and fails them again, and the scheduler picks them up again after each press. The document keeps a permanent "failed" line naming the old connection, and the operator's failure gauge counts it for good. - **Where:** `documents/models.py:815-822` (`DocumentCopy.connection`, `on_delete=models.SET_NULL`), `plugins/views.py:463-475` (`ConnectionDeleteView`, which does nothing about copies), `documents/archiving.py:224-242` (`send_now`: every copy that is not `SENT` is retried, with `copy.attempts = 0`), `documents/archiving.py:123-124` (`send_copy` fails a copy with no connection: "The connection is gone or switched off."), `documents/archiving.py:173-189` (`pending_copies` deliberately keeps copies whose connection row is gone), `core/metrics.py:249-250` (`postulo_failures{kind="document_copies"}` counts every `FAILED` copy). - **What happens:** a person's Paperless copy of a document failed (the store was down). They remove the connection and add it again — the removal page says "You can connect it again later" — or replace it with another store. `schedule_copies` gives the document a new copy for the new connection, but the old copy has `connection=NULL` and status `failed`. Every *Send to stores now* on that document then answers "%(sent)d copies sent, %(failed)d failed", or "The copy could not be sent; the document says why" once the new store already has the file, and the document shows a failed copy for a connection that no longer exists. Reproduced on the snapshot: three calls of `send_now()` on such a document each returned `(0, 1)`, and the copy ended `failed`, `attempts 1`, due again five minutes later in `pending_copies()`. With no press, the scheduler dials it until `MAX_ATTEMPTS` and it then stays `failed` indefinitely. - **Why it matters:** the person cannot tell a real failure from a leftover one and has no way to clear it short of deleting the document; the messages after *Send now* are wrong; and an operator alerting on `postulo_failures` sees a failure that can never resolve. - **Fix:** when a connection is deleted, delete its copies that are not `SENT` (a `pre_delete` receiver on `Connection`, so the admin and account deletion are covered too, not only `ConnectionDeleteView`). Keep `SENT` copies, which are the references #130 and the importer (`core/importer.py:81-100`) mean to keep with no connection behind them. For rows already orphaned, a data migration deletes copies with `connection IS NULL` and a status other than `sent`; `send_now` and `pending_copies` should then also exclude `connection__isnull=True` rows that are not `SENT`, so an orphan can never be retried. ## Done when - A test in `tests/test_stores.py` removes a store connection that has a failed and a sent copy, and asserts that the failed copy is gone, the sent copy remains with `connection` empty, and `send_now()` on the document afterwards returns `(0, 0)`.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:12 +00:00

tiagoagueda added the

tier

bug

labels

2026-10-01 10:47:12 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 07:56:40 +00:00

Drop a removed connection's unsent copies instead of failing them for good

tiagoagueda closed this issue

2026-10-04 07:56:40 +00:00