Store: a retried document is matched to the wrong Paperless document when its title holds a '#' and a number #11

New issue

Closed

opened 2026-10-01 10:48:18 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:48:18 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, reading how the duplicate answer is parsed and replaying it against a mock transport that answers in Paperless-ngx's own words.

When Paperless refuses a file it already holds, the plugin reads the existing document's id out of the task's message with _DUPLICATE_ID = re.compile(r"#(\d+)") and .search(), which takes the first match. Paperless-ngx 2.x writes that message as <file name>: Not consuming <file name>: It is a duplicate of <title> (#<id>). (src/documents/consumer.py, pre_check_duplicate and _fail, read at v2.18.4), so the title comes before the id. The title is the one this plugin composed from the kind, the company and the job title, or an upload's own title, and any #<digits> in it wins. The plugin then treats that number as the document. The retry the README calls safe is not safe for such a title.

Where: src/postulo_paperless/store.py:56 (_DUPLICATE_ID); :182-187 (Archive.wait_for, the FAILURE branch: match = _DUPLICATE_ID.search(result) and return int(match.group(1))); :177-179 (the same search in the SUCCESS branch); :326-329 (PaperlessStore.put, which writes the application's link into the document that id names) and :330-334 (the ExternalRef built from it). tests/test_paperless.py:82-83 (the fake's message, the same shape) and :296 (the only duplicate test, whose title holds no #).
What happens: a CV is sent for a posting titled "Engineer (Req #48213)". Paperless takes longer than WAIT_SECONDS to consume it, put raises StillConsuming, and the scheduler offers the file again. Paperless answers that it is a duplicate of "CV — Company — Engineer (Req #48213) — date (#57)". Checked on the snapshot:
- With "Req #48213", the plugin asks for document 48213, Paperless answers 404, and put raises PaperlessError: HTTP 404. Every one of the six attempts uploads the file, is told it is a duplicate and fails the same way, so the copy ends failed although the archive holds it.
- With "Req #1" and a document 1 in the archive, put returned ref.id == "1": document 1, an unrelated document, had its Postulo link replaced with this application's address, and Postulo recorded and linked document 1 as the copy.
- With the custom field left blank there is no request to fail, and the wrong id is stored without any sign.
- The same answer arrives without a retry when one file is held by two documents in Postulo; the title in the message is then whatever the existing document is called in Paperless, which its owner may have edited.
Why it matters: the link from a document in Postulo to its copy points at another document or at none, and a field on an unrelated document in the person's archive is overwritten. The plugin's docstring and the README both rest the retry on this parse ("a document is never archived twice, however many times it is offered").
Fix: take the id from the end of the message, not the start: match the closing (#<digits>), allowing the full stop and the "Note: existing document is in the trash." sentence that may follow it, or use the last match. Apply the same to the SUCCESS branch. Worth knowing while there: on Paperless-ngx's development branch, as read on 2026-10-01, pre_check_duplicate refuses a duplicate only when CONSUMER_DELETE_DUPLICATES is set and otherwise consumes the file again, so the premise that a second offer is refused needs checking against the version that ships it.

Done when

A test in tests/test_paperless.py offers the same file twice with a role of "Engineer (Req #1)" while document 1 exists, and shows the second put returning the id of the document Paperless named, with document 1's custom fields untouched.
A test in tests/test_paperless.py does the same for an upload whose own title holds # and a number, with the fake's message in Paperless's full form (file name twice, a trailing full stop).

Found on 2026-09-29 in a read-only review of postulo-paperless at 973969bb7, reading how the duplicate answer is parsed and replaying it against a mock transport that answers in Paperless-ngx's own words. When Paperless refuses a file it already holds, the plugin reads the existing document's id out of the task's message with `_DUPLICATE_ID = re.compile(r"#(\d+)")` and `.search()`, which takes the first match. Paperless-ngx 2.x writes that message as `<file name>: Not consuming <file name>: It is a duplicate of <title> (#<id>).` (`src/documents/consumer.py`, `pre_check_duplicate` and `_fail`, read at v2.18.4), so the title comes before the id. The title is the one this plugin composed from the kind, the company and the job title, or an upload's own title, and any `#<digits>` in it wins. The plugin then treats that number as the document. The retry the README calls safe is not safe for such a title. - **Where:** `src/postulo_paperless/store.py:56` (`_DUPLICATE_ID`); `:182-187` (`Archive.wait_for`, the `FAILURE` branch: `match = _DUPLICATE_ID.search(result)` and `return int(match.group(1))`); `:177-179` (the same search in the `SUCCESS` branch); `:326-329` (`PaperlessStore.put`, which writes the application's link into the document that id names) and `:330-334` (the `ExternalRef` built from it). `tests/test_paperless.py:82-83` (the fake's message, the same shape) and `:296` (the only duplicate test, whose title holds no `#`). - **What happens:** a CV is sent for a posting titled "Engineer (Req #48213)". Paperless takes longer than `WAIT_SECONDS` to consume it, `put` raises `StillConsuming`, and the scheduler offers the file again. Paperless answers that it is a duplicate of "CV — Company — Engineer (Req #48213) — date (#57)". Checked on the snapshot: - With "Req #48213", the plugin asks for document 48213, Paperless answers 404, and `put` raises `PaperlessError: HTTP 404`. Every one of the six attempts uploads the file, is told it is a duplicate and fails the same way, so the copy ends *failed* although the archive holds it. - With "Req #1" and a document 1 in the archive, `put` returned `ref.id == "1"`: document 1, an unrelated document, had its Postulo link replaced with this application's address, and Postulo recorded and linked document 1 as the copy. - With the custom field left blank there is no request to fail, and the wrong id is stored without any sign. - The same answer arrives without a retry when one file is held by two documents in Postulo; the title in the message is then whatever the existing document is called in Paperless, which its owner may have edited. - **Why it matters:** the link from a document in Postulo to its copy points at another document or at none, and a field on an unrelated document in the person's archive is overwritten. The plugin's docstring and the README both rest the retry on this parse ("a document is never archived twice, however many times it is offered"). - **Fix:** take the id from the end of the message, not the start: match the closing `(#<digits>)`, allowing the full stop and the "Note: existing document is in the trash." sentence that may follow it, or use the last match. Apply the same to the `SUCCESS` branch. Worth knowing while there: on Paperless-ngx's development branch, as read on 2026-10-01, `pre_check_duplicate` refuses a duplicate only when `CONSUMER_DELETE_DUPLICATES` is set and otherwise consumes the file again, so the premise that a second offer is refused needs checking against the version that ships it. ## Done when - A test in `tests/test_paperless.py` offers the same file twice with a role of "Engineer (Req #1)" while document 1 exists, and shows the second `put` returning the id of the document Paperless named, with document 1's custom fields untouched. - A test in `tests/test_paperless.py` does the same for an upload whose own title holds `#` and a number, with the fake's message in Paperless's full form (file name twice, a trailing full stop).

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:48:18 +00:00

tiagoagueda added the

bug

label

2026-10-01 10:48:18 +00:00

tiagoagueda referenced this issue from a commit

2026-10-19 07:05:31 +00:00

Read Paperless's answers within a deadline and a cap, retire a dead token, decline unconsumable files, and stop mis-reading the duplicate's id

tiagoagueda referenced this issue

2026-10-19 07:05:58 +00:00

Security review fixes: bounded answers, dead token, declined files, duplicate id, repinned lock #18