Scheduler: the upload scrub rereads and rehashes 25 files every five minutes however recently they were checked #1118

New issue

Closed

opened 2026-10-31 03:04:08 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:08 +00:00

Owner

What exists today

src/postulo/notifications/management/commands/send_due_reminders.py:230 calls scrub.verify_uploads(limit=SCRUB_SLICE) on every scheduler pass; SCRUB_SLICE = 25 (line 32) and the pass runs every 300 s by default (line 127), 288 passes a day.

src/postulo/documents/scrub.py:47-69 takes the 25 uploads with the oldest verified_at (line 55, no age filter), reads each file whole through integrity.damage_of (documents/integrity.py:176-196, a SHA-256 over the file), and writes damage, verified_at and size with one UPDATE per row (line 62). The selection is "least recently checked first" with no "and not checked in the last N days", so on an instance with fewer than 7,200 uploads every file is hashed several times a day: with 100 uploads, each one is read 72 times a day. Uploads are capped at 20 MB (documents/forms.py:29, per #663), so the worst case is 500 MB read per pass.

#663 (closed) asked for "a fixed number of rows each time, oldest-checked first"; it did not ask for a minimum age, so this is a refinement of that design, not a reversal. No open issue covers it (searched "scrub", "verify uploads", "media sweep").

What would change

In verify_uploads, when limit is given (the scheduled slice), select only rows with verified_at null or older than a new setting POSTULO_SCRUB_EVERY_DAYS (default 7):

rows.filter(Q(verified_at__isnull=True) | Q(verified_at__lt=now - dt.timedelta(days=…)))

manage.py verify_media (no limit) keeps reading everything. The slice size stays 25, which still reaches a backlog of 7,200 files a day. The wiki page that describes the scrub gains the setting. An index on verified_at (candidate Q7) makes the filter cheap; both can be one PR.

No library: one queryset filter and one setting.

What it saves or protects

medium: 7,200 file reads and hashes and 7,200 single-row UPDATEs a day become one read per upload per chosen interval (for 100 uploads checked weekly, about 14 a day)

Accessibility

none

Decisions

How often should each upload be reread? Recommended: weekly by default, configurable; corruption on disk is rare and a weekly check still reports it within days.

Found by the efficiency and reliability audit of 2026-10-30 (finding B2; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/notifications/management/commands/send_due_reminders.py:230` calls `scrub.verify_uploads(limit=SCRUB_SLICE)` on every scheduler pass; `SCRUB_SLICE = 25` (line 32) and the pass runs every 300 s by default (line 127), 288 passes a day. `src/postulo/documents/scrub.py:47-69` takes the 25 uploads with the oldest `verified_at` (line 55, no age filter), reads each file whole through `integrity.damage_of` (`documents/integrity.py:176-196`, a SHA-256 over the file), and writes `damage`, `verified_at` and `size` with one `UPDATE` per row (line 62). The selection is "least recently checked first" with no "and not checked in the last N days", so on an instance with fewer than 7,200 uploads every file is hashed several times a day: with 100 uploads, each one is read 72 times a day. Uploads are capped at 20 MB (`documents/forms.py:29`, per #663), so the worst case is 500 MB read per pass. #663 (closed) asked for "a fixed number of rows each time, oldest-checked first"; it did not ask for a minimum age, so this is a refinement of that design, not a reversal. No open issue covers it (searched "scrub", "verify uploads", "media sweep"). ## What would change In `verify_uploads`, when `limit` is given (the scheduled slice), select only rows with `verified_at` null or older than a new setting `POSTULO_SCRUB_EVERY_DAYS` (default 7): `rows.filter(Q(verified_at__isnull=True) | Q(verified_at__lt=now - dt.timedelta(days=…)))` `manage.py verify_media` (no `limit`) keeps reading everything. The slice size stays 25, which still reaches a backlog of 7,200 files a day. The wiki page that describes the scrub gains the setting. An index on `verified_at` (candidate Q7) makes the filter cheap; both can be one PR. No library: one queryset filter and one setting. ## What it saves or protects medium: 7,200 file reads and hashes and 7,200 single-row UPDATEs a day become one read per upload per chosen interval (for 100 uploads checked weekly, about 14 a day) ## Accessibility none ## Decisions - **How often should each upload be reread?** Recommended: weekly by default, configurable; corruption on disk is rare and a weekly check still reports it within days. --- Found by the efficiency and reliability audit of 2026-10-30 (finding B2; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:08 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:08 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 10:38:54 +00:00