Store: each document repeats three or four name lookups, a slow consumption uploads the file again, and the scheduler waits up to 20 s per document

tiagoagueda commented

2026-10-31 03:04:32 +00:00

Owner

What exists today

postulo-paperless on the forge's main (04268aa), src/postulo_paperless/store.py:

put (lines 449-489) opens a new Archive per document and calls ensure (lines 263-274) for the document type, the correspondent and the tag, and for the custom field after the upload: three or four GET …?name__iexact= requests per document, none remembered between documents, though the answers are the same for a whole batch.
wait_for (lines 302-331) polls the task once a second for up to WAIT_SECONDS = 20 (line 57), with time.sleep in the calling thread. When Paperless takes longer (OCR of a scan), it raises StillConsuming; the core records a failure and retries the copy 5, 10, 20 … minutes later (documents/archiving.py:37-39), and the retry uploads the whole file again (line 466). That is correct, since Paperless reports the checksum duplicate and the plugin reads it as success (lines 316-322), but it is a second upload and a second consumption run just to learn the id.

Core, src/postulo/documents/archiving.py:263-273 send_pending takes up to BATCH = 50 copies per pass with no time budget, and the scheduler calls it before webhooks and syncs (notifications/management/commands/send_due_reminders.py:203-206). Webhooks have BUDGET = 60.0 (notifications/webhooks.py:44) and syncs run_syncs(budget=…); copies have none, so 50 documents at up to 20 s of polling each (plus uploads up to 60 s) can hold the pass for a quarter of an hour. Send now (documents/views.py:936) runs the same wait inside a web request.

Reminders are announced before copies, so they are not starved; webhooks, syncs and the rest of the pass are delayed.

What would change

postulo-paperless:

Ask before uploading. Compute the file's MD5 (Paperless's own document checksum) and look for it first, GET /api/documents/?checksum__iexact=<md5>&page_size=1; when found, return that id without uploading. A retry after StillConsuming then costs one small request, and a document already archived by hand is not uploaded at all. (Check the filter on the oldest Paperless-ngx version the README supports; fall back to uploading if it is refused.)
Remember the lookups. A small cache of ensure answers keyed by the Paperless address and the kind and name, kept for the length of a scheduler pass (a module-level dictionary with a few minutes' expiry), dropped for an entry whose id Paperless then refuses with 400.
Wait less. With item 1 a short wait (5 s) loses nothing: the next attempt finds the document by checksum.

Core: give archiving.send_pending the same time budget the webhooks have (stop starting copies once 60 s have gone, leave the rest to the next pass).

What it saves or protects

medium: a backlog of 50 documents no longer holds the scheduler pass for up to about 17 minutes of polling before webhooks and syncs run; a document whose OCR outlasts the wait is not uploaded a second time; about 150 lookup requests per 50 documents become a few

Accessibility

none

Decisions

Split into: a postulo-paperless issue (checksum lookup, lookup cache, shorter wait) and a core issue (a budget for send_pending).

Found by the efficiency and reliability audit of 2026-10-30 (finding PR9; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today postulo-paperless on the forge's `main` (`04268aa`), `src/postulo_paperless/store.py`: - `put` (lines 449-489) opens a new `Archive` per document and calls `ensure` (lines 263-274) for the document type, the correspondent and the tag, and for the custom field after the upload: three or four `GET …?name__iexact=` requests per document, none remembered between documents, though the answers are the same for a whole batch. - `wait_for` (lines 302-331) polls the task once a second for up to `WAIT_SECONDS = 20` (line 57), with `time.sleep` in the calling thread. When Paperless takes longer (OCR of a scan), it raises `StillConsuming`; the core records a failure and retries the copy 5, 10, 20 … minutes later (`documents/archiving.py:37-39`), and the retry uploads the whole file again (line 466). That is correct, since Paperless reports the checksum duplicate and the plugin reads it as success (lines 316-322), but it is a second upload and a second consumption run just to learn the id. Core, `src/postulo/documents/archiving.py:263-273` `send_pending` takes up to `BATCH = 50` copies per pass with no time budget, and the scheduler calls it before webhooks and syncs (`notifications/management/commands/send_due_reminders.py:203-206`). Webhooks have `BUDGET = 60.0` (`notifications/webhooks.py:44`) and syncs `run_syncs(budget=…)`; copies have none, so 50 documents at up to 20 s of polling each (plus uploads up to 60 s) can hold the pass for a quarter of an hour. *Send now* (`documents/views.py:936`) runs the same wait inside a web request. Reminders are announced before copies, so they are not starved; webhooks, syncs and the rest of the pass are delayed. ## What would change postulo-paperless: 1. **Ask before uploading.** Compute the file's MD5 (Paperless's own document checksum) and look for it first, `GET /api/documents/?checksum__iexact=<md5>&page_size=1`; when found, return that id without uploading. A retry after `StillConsuming` then costs one small request, and a document already archived by hand is not uploaded at all. (Check the filter on the oldest Paperless-ngx version the README supports; fall back to uploading if it is refused.) 2. **Remember the lookups.** A small cache of `ensure` answers keyed by the Paperless address and the kind and name, kept for the length of a scheduler pass (a module-level dictionary with a few minutes' expiry), dropped for an entry whose id Paperless then refuses with 400. 3. **Wait less.** With item 1 a short wait (5 s) loses nothing: the next attempt finds the document by checksum. Core: give `archiving.send_pending` the same time budget the webhooks have (stop starting copies once 60 s have gone, leave the rest to the next pass). ## What it saves or protects medium: a backlog of 50 documents no longer holds the scheduler pass for up to about 17 minutes of polling before webhooks and syncs run; a document whose OCR outlasts the wait is not uploaded a second time; about 150 lookup requests per 50 documents become a few ## Accessibility none ## Decisions - **Split into:** a postulo-paperless issue (checksum lookup, lookup cache, shorter wait) and a core issue (a budget for `send_pending`). --- Found by the efficiency and reliability audit of 2026-10-30 (finding PR9; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:32 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:32 +00:00

tiagoagueda referenced this issue from Postulo/postulo

2026-10-31 03:04:37 +00:00

Stores: sending document copies has no time budget, so a slow store holds the scheduler pass before webhooks and syncs #1172

tiagoagueda commented

2026-10-31 03:05:04 +00:00