Store: with POSTULO_PUBLIC_URL unset the application link is a bare path, Paperless refuses it, and an archived document ends as failed #12

New issue

Closed

opened 2026-10-01 10:48:18 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:18 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, following metadata.application_url from the core into the custom field and replaying put against a mock transport.

put writes metadata.application_url into a Paperless custom field it creates with data_type="url", guarded only by the value being non-empty. The core builds that value with no request to hand, so on an instance where POSTULO_PUBLIC_URL is not set, which is the default, it is a bare path such as /applications/7/. Paperless-ngx validates URL fields and refuses a value with no scheme (uri_validator in src/documents/validators.py, called from CustomFieldInstanceSerializer.validate, read at v2.18.4). The refusal arrives after the document has been consumed, and it raises out of put.

Where: src/postulo_paperless/store.py:327-329 (PaperlessStore.put: if field_name and metadata.application_url:, then ensure("custom_fields", field_name, data_type="url") and set_custom_field); :198-213 (Archive.set_custom_field, which raises PaperlessError on any answer but 200); :88-95 (_explain, which reads only detail). In the core: documents/stores.py:77 (application_url = absolute_url(application.get_absolute_url()), no request), notifications/base.py:164-165 (return f"{base}{path}" if base else path) and config/settings/base.py:530 (POSTULO_PUBLIC_URL defaults to empty).
What happens: an operator installs the plugin and leaves POSTULO_PUBLIC_URL unset; the wiki documents it as optional. A person connects Paperless and keeps the default custom field name, "Postulo". For every document that belongs to an application:
1. the file is uploaded and consumed, so Paperless holds it;
2. the PATCH that writes the link is answered 400, and put raises PaperlessError: HTTP 400, with no reason, because Paperless's validation errors carry no detail key;
3. the core records the copy as failed, keeps no ExternalRef, and retries: each of the six attempts uploads the whole file again, is told it is a duplicate, and fails at the same PATCH.
Checked on the snapshot with a transport that answers the PATCH as Paperless does: two attempts gave two uploads, one document in the archive and PaperlessError('HTTP 400') both times. Documents with no application have an empty application_url and are archived normally, which makes the failure look random. The tests only ever use https://postulo.example/applications/7/.
Why it matters: on a default instance every CV and letter sent with an application is shown as failed although the archive holds it, is uploaded six times, and never gets its link into Paperless.
Fix:
- Write the field only when the value is an absolute address: urlsplit(value).scheme in ("http", "https") and a host.
- Treat the link as decoration. Once wait_for has returned an id the document is archived, so a failure to create or write the custom field should be logged and the ExternalRef still returned; an archived document is then never shown as failed and never uploaded again for it.
- Let _explain show Paperless's field errors when there is no detail, so that a 400 says why.
- Say in the README that the link needs POSTULO_PUBLIC_URL. The core could also leave application_url empty rather than hand a store a path that leads nowhere from another machine.

Done when

A test in tests/test_paperless.py shows put with application_url="/applications/7/" returning the document's reference without sending a PATCH.
A test in tests/test_paperless.py, with the fake answering the PATCH with 400, shows put still returning the reference, so that the copy is recorded as sent and not offered again.

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, following `metadata.application_url` from the core into the custom field and replaying `put` against a mock transport. `put` writes `metadata.application_url` into a Paperless custom field it creates with `data_type="url"`, guarded only by the value being non-empty. The core builds that value with no request to hand, so on an instance where `POSTULO_PUBLIC_URL` is not set, which is the default, it is a bare path such as `/applications/7/`. Paperless-ngx validates URL fields and refuses a value with no scheme (`uri_validator` in `src/documents/validators.py`, called from `CustomFieldInstanceSerializer.validate`, read at v2.18.4). The refusal arrives after the document has been consumed, and it raises out of `put`. - **Where:** `src/postulo_paperless/store.py:327-329` (`PaperlessStore.put`: `if field_name and metadata.application_url:`, then `ensure("custom_fields", field_name, data_type="url")` and `set_custom_field`); `:198-213` (`Archive.set_custom_field`, which raises `PaperlessError` on any answer but 200); `:88-95` (`_explain`, which reads only `detail`). In the core: `documents/stores.py:77` (`application_url = absolute_url(application.get_absolute_url())`, no request), `notifications/base.py:164-165` (`return f"{base}{path}" if base else path`) and `config/settings/base.py:530` (`POSTULO_PUBLIC_URL` defaults to empty). - **What happens:** an operator installs the plugin and leaves `POSTULO_PUBLIC_URL` unset; the wiki documents it as optional. A person connects Paperless and keeps the default custom field name, "Postulo". For every document that belongs to an application: 1. the file is uploaded and consumed, so Paperless holds it; 2. the `PATCH` that writes the link is answered 400, and `put` raises `PaperlessError: HTTP 400`, with no reason, because Paperless's validation errors carry no `detail` key; 3. the core records the copy as *failed*, keeps no `ExternalRef`, and retries: each of the six attempts uploads the whole file again, is told it is a duplicate, and fails at the same `PATCH`. Checked on the snapshot with a transport that answers the `PATCH` as Paperless does: two attempts gave two uploads, one document in the archive and `PaperlessError('HTTP 400')` both times. Documents with no application have an empty `application_url` and are archived normally, which makes the failure look random. The tests only ever use `https://postulo.example/applications/7/`. - **Why it matters:** on a default instance every CV and letter sent with an application is shown as *failed* although the archive holds it, is uploaded six times, and never gets its link into Paperless. - **Fix:** - Write the field only when the value is an absolute address: `urlsplit(value).scheme in ("http", "https")` and a host. - Treat the link as decoration. Once `wait_for` has returned an id the document is archived, so a failure to create or write the custom field should be logged and the `ExternalRef` still returned; an archived document is then never shown as failed and never uploaded again for it. - Let `_explain` show Paperless's field errors when there is no `detail`, so that a 400 says why. - Say in the README that the link needs `POSTULO_PUBLIC_URL`. The core could also leave `application_url` empty rather than hand a store a path that leads nowhere from another machine. ## Done when - A test in `tests/test_paperless.py` shows `put` with `application_url="/applications/7/"` returning the document's reference without sending a `PATCH`. - A test in `tests/test_paperless.py`, with the fake answering the `PATCH` with 400, shows `put` still returning the reference, so that the copy is recorded as sent and not offered again.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:18 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:18 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:05:31 +00:00

Read Paperless's answers within a deadline and a cap, retire a dead token, decline unconsumable files, and stop mis-reading the duplicate's id

tiagoagueda referenced this issue

2026-10-19 07:05:58 +00:00

Security review fixes: bounded answers, dead token, declined files, duplicate id, repinned lock #18