Ask for a document by its checksum, remember name lookups, and wait less #30

Merged

tiagoagueda merged 1 commit from 27-store-lookups into main

2026-11-12 06:39:50 +00:00

tiagoagueda commented

2026-11-11 22:58:41 +00:00

Owner

Closes #27

Before uploading, the store asks Paperless for a document with the file's checksum (from 1.16; an older Paperless that ignores the filter falls back to the upload); name lookups are remembered for five minutes and forgotten when Paperless refuses an id; the wait for a consumption goes from 20 s to 5 s, matching core #1172's budget. Tests count requests and uploads against the test double, including an older-Paperless mode.

🤖 Generated with Claude Code

Closes #27 Before uploading, the store asks Paperless for a document with the file's checksum (from 1.16; an older Paperless that ignores the filter falls back to the upload); name lookups are remembered for five minutes and forgotten when Paperless refuses an id; the wait for a consumption goes from 20 s to 5 s, matching core #1172's budget. Tests count requests and uploads against the test double, including an older-Paperless mode. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

tiagoagueda added 1 commit

2026-11-11 22:58:41 +00:00

Ask for a document by its checksum, remember name lookups, and wait less

CI / test (pull_request) Successful in 2m49s

Details

CI / Audit the locked dependencies (pull_request) Failing after 2m30s

Details

cc7852fa24

Each document asked Paperless for its document type, correspondent, tag and custom field
by name, three or four requests that a batch for one company answered the same way every
time. A consumption slower than the 20 s wait raised StillConsuming, and the retry minutes
later uploaded the whole file again only to read the duplicate refusal for its id.

put now computes the file's MD5, which is the checksum Paperless keeps of an original, and
asks GET documents/?checksum__iexact=...&page_size=1 first. Paperless-ngx before 1.16 does
not know that filter and ignores it, answering with every document, so a match counts only
when exactly one comes back and documents/<id>/metadata/ names the same original_checksum;
anything else, or a refusal, falls back to the upload and its duplicate answer, as before. A
retry after a slow consumption, or a document archived by hand, now costs two small
requests and no upload.

ensure remembers each id for five minutes in a module-level map keyed by the address, a
digest of the token, the table and the name, which outlives the Archive each document
gets. A 400 from post_document forgets the ids it names (or all this document took from
memory), and one from the custom-field PATCH forgets that field, so a correspondent
deleted in Paperless is looked up again on the next attempt.

With the retry that cheap, the wait drops from 20 s to 5 s. This agrees with the core's
half, postulo/postulo#1172 (07bf8d907, on main): send_pending stops starting copies after
60 s but never cuts one off, so a copy's own wait is what a pass can overrun by.

The tests count requests to the in-memory Paperless: one lookup per name across a batch
of five documents, a lookup again only once the memory has expired, a refused id
forgotten and asked for again, one upload across a slow first attempt and its retry, none
for a document Paperless already holds, and the upload kept for a Paperless that ignores
or refuses the filter. Six of them fail on the old store.

Closes #27

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

tiagoagueda force-pushed 27-store-lookups from cc7852fa24

CI / test (pull_request) Successful in 2m49s

Details

CI / Audit the locked dependencies (pull_request) Failing after 2m30s

Details

to 3aaae8137b

CI / Audit the locked dependencies (pull_request) Successful in 2m56s

Details

CI / test (pull_request) Successful in 3m52s

Details

2026-11-12 05:06:41 +00:00

Compare