Store: a file type Paperless does not consume is uploaded six times and ends as a bare "HTTP 400

tiagoagueda commented

2026-10-01 10:48:19 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, running _explain against the error bodies Django REST Framework produces and reading Paperless-ngx's PostDocumentSerializer.validate_document.

When Paperless refuses a file because of its type, the plugin neither says why nor stops offering it. The reason is dropped from the message, and the refusal is treated as a failure to retry, so the whole file is uploaded again on every attempt.

1. The reason is dropped

Where: src/postulo_paperless/store.py:88-95 (_explain).
What happens: _explain reads only a top-level detail key: detail = body.get("detail", "") if isinstance(body, dict) else "". Django REST Framework uses detail for authentication and permission errors. A validation refusal is a mapping of field name to a list of messages, and Paperless's refusal of a file type is one of those: validate_document raises serializers.ValidationError(_("File type %(type)s not supported") % ...), which reaches the client as {"document": ["File type ... not supported"]}. Run against the snapshot, _explain answers 'HTTP 400' for that body, for a body with several field errors, and for a list body.
Why it matters: the copy shows failed: PaperlessError: HTTP 400, with nothing saying the file type is the cause.

2. The refusal is retried instead of declined

Where: src/postulo_paperless/store.py:149-157 (Archive.post_document) and :311-334 (PaperlessStore.put), which has no path that returns None.
What happens: the store contract lets put return None to say not for me (plugins/base.py:784-787 in the core), which send_copy records as not accepted with no further attempt (documents/archiving.py:155-160). This plugin raises instead, so the core retries on the growing wait. Postulo accepts uploads of pdf, doc, docx, odt, rtf, txt, png, jpg and jpeg (documents/models.py:458-460), up to 20 MB (documents/forms.py:26); a Paperless without Tika consumes no office formats. A .docx reference is therefore sent in full six times over about two and a half hours, each attempt preceded by the look-ups that create the document type and correspondent, and ends as failed.
Why it matters: bandwidth and Paperless's time spent on a request that cannot succeed, and a failed badge for something that is not a failure and that the person cannot fix by trying again.

3. The test passes on a body Paperless does not send

Where: tests/test_paperless.py:68-69 and :318-321.
What happens: test_a_refusal_is_explained passes because the fake answers httpx.Response(400, json={"detail": self.reject_with}). Paperless puts that sentence under document, in a list.
Fix: make _explain flatten field errors: for a mapping without detail, join each key's messages; accept a list body; keep the 120-character cut for anything long. In post_document, treat a 400 whose errors are on the document field as a refusal of the file and have put return None, so the copy is not accepted once. Change the fake to answer {"document": [...]}.

Done when

test_a_refusal_is_explained in tests/test_paperless.py uses a fake that answers {"document": ["File type ... not supported"]}; with that body it fails today.
A test in tests/test_paperless.py shows that such a refusal makes put return None, and, through send_pending(), that the copy ends as CopyStatus.DECLINED after one upload.
A test shows that _explain names the field and message for a 400 on another field.

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, running `_explain` against the error bodies Django REST Framework produces and reading Paperless-ngx's `PostDocumentSerializer.validate_document`. When Paperless refuses a file because of its type, the plugin neither says why nor stops offering it. The reason is dropped from the message, and the refusal is treated as a failure to retry, so the whole file is uploaded again on every attempt. ## 1. The reason is dropped - **Where:** `src/postulo_paperless/store.py:88-95` (`_explain`). - **What happens:** `_explain` reads only a top-level `detail` key: `detail = body.get("detail", "") if isinstance(body, dict) else ""`. Django REST Framework uses `detail` for authentication and permission errors. A validation refusal is a mapping of field name to a list of messages, and Paperless's refusal of a file type is one of those: `validate_document` raises `serializers.ValidationError(_("File type %(type)s not supported") % ...)`, which reaches the client as `{"document": ["File type ... not supported"]}`. Run against the snapshot, `_explain` answers `'HTTP 400'` for that body, for a body with several field errors, and for a list body. - **Why it matters:** the copy shows *failed: PaperlessError: HTTP 400*, with nothing saying the file type is the cause. ## 2. The refusal is retried instead of declined - **Where:** `src/postulo_paperless/store.py:149-157` (`Archive.post_document`) and `:311-334` (`PaperlessStore.put`), which has no path that returns `None`. - **What happens:** the store contract lets `put` return `None` to say *not for me* (`plugins/base.py:784-787` in the core), which `send_copy` records as *not accepted* with no further attempt (`documents/archiving.py:155-160`). This plugin raises instead, so the core retries on the growing wait. Postulo accepts uploads of `pdf`, `doc`, `docx`, `odt`, `rtf`, `txt`, `png`, `jpg` and `jpeg` (`documents/models.py:458-460`), up to 20 MB (`documents/forms.py:26`); a Paperless without Tika consumes no office formats. A `.docx` reference is therefore sent in full six times over about two and a half hours, each attempt preceded by the look-ups that create the document type and correspondent, and ends as failed. - **Why it matters:** bandwidth and Paperless's time spent on a request that cannot succeed, and a *failed* badge for something that is not a failure and that the person cannot fix by trying again. ## 3. The test passes on a body Paperless does not send - **Where:** `tests/test_paperless.py:68-69` and `:318-321`. - **What happens:** `test_a_refusal_is_explained` passes because the fake answers `httpx.Response(400, json={"detail": self.reject_with})`. Paperless puts that sentence under `document`, in a list. - **Fix:** make `_explain` flatten field errors: for a mapping without `detail`, join each key's messages; accept a list body; keep the 120-character cut for anything long. In `post_document`, treat a 400 whose errors are on the `document` field as a refusal of the file and have `put` return `None`, so the copy is *not accepted* once. Change the fake to answer `{"document": [...]}`. ## Done when - `test_a_refusal_is_explained` in `tests/test_paperless.py` uses a fake that answers `{"document": ["File type ... not supported"]}`; with that body it fails today. - A test in `tests/test_paperless.py` shows that such a refusal makes `put` return `None`, and, through `send_pending()`, that the copy ends as `CopyStatus.DECLINED` after one upload. - A test shows that `_explain` names the field and message for a 400 on another field.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:48:19 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:19 +00:00

tiagoagueda referenced this issue

2026-10-17 10:39:53 +00:00

SEC2-061: Store reads every Paperless answer whole with no deadline, so a server the account holder names can hold the shared scheduler #16

tiagoagueda commented

2026-10-17 10:40:32 +00:00

Author

Owner

From the 2026-10-16 security review: since this issue rewrites _explain, please fold one more change into it.

_explain (store.py:88-95) copies the server's detail uncapped, or response.text[:120], with newlines kept:

detail = body.get("detail", "") if isinstance(body, dict) else ""
...
detail = response.text[:120]

The core caps the copy's last_error at 500 (archiving.py:134) and the page renders it escaped, but archiving.py:188-190 also sends the exception to logger.exception(...), and the console formatter does not fold newlines (core/logs.py:146-152). So whatever answers as the Paperless server can write lines of its choosing into the operator's log beside real entries, against the core's own rule that "a log line is one place a newline from a stranger must never land" (core/logs.py:88-90). Informational: the server is one the account holder configured.

Fix, for every branch of the new _explain, including the DRF field errors this issue adds: detail = " ".join(str(detail).split())[:120].

Done when a test whose answer carries a detail with a newline (and one with 1 000 characters) gets a one-line message of at most 120 characters of detail.

From the 2026-10-16 security review: since this issue rewrites `_explain`, please fold one more change into it. `_explain` (`store.py:88-95`) copies the server's `detail` uncapped, or `response.text[:120]`, with newlines kept: ```python detail = body.get("detail", "") if isinstance(body, dict) else "" ... detail = response.text[:120] ``` The core caps the copy's `last_error` at 500 (`archiving.py:134`) and the page renders it escaped, but `archiving.py:188-190` also sends the exception to `logger.exception(...)`, and the console formatter does not fold newlines (`core/logs.py:146-152`). So whatever answers as the Paperless server can write lines of its choosing into the operator's log beside real entries, against the core's own rule that "a log line is one place a newline from a stranger must never land" (`core/logs.py:88-90`). Informational: the server is one the account holder configured. Fix, for every branch of the new `_explain`, including the DRF field errors this issue adds: `detail = " ".join(str(detail).split())[:120]`. Done when a test whose answer carries a `detail` with a newline (and one with 1 000 characters) gets a one-line message of at most 120 characters of detail.

tiagoagueda referenced this issue from Postulo/postulo

2026-10-17 10:41:23 +00:00

SEC2 index: security review of 2026-10-16 (86 issues across the core, plugins and extensions) #992

tiagoagueda referenced this issue from a commit

2026-10-19 07:05:31 +00:00

Read Paperless's answers within a deadline and a cap, retire a dead token, decline unconsumable files, and stop mis-reading the duplicate's id

tiagoagueda referenced this issue

2026-10-19 07:05:58 +00:00

Security review fixes: bounded answers, dead token, declined files, duplicate id, repinned lock #18

tiagoagueda commented

2026-10-19 09:52:11 +00:00

Author