Scheduler: the orphan sweep walks the whole media tree every five minutes for files that must be a day old to qualify #1119

New issue

Closed

opened 2026-10-31 03:04:09 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:09 +00:00

Owner

What exists today

src/postulo/notifications/management/commands/send_due_reminders.py:231 calls scrub.sweep() on every pass (every 300 s). src/postulo/documents/scrub.py:112-148 lists the strays through media.unreferenced (core/media.py:41-55), which first loads every file name every row names (referenced_names, lines 22-38: one values_list query per FileField on every model) and then walks the media root with filestore.listing (documents/filestore.py:95-106): listdir per directory and, per file, get_modified_time and size, two stat calls. When more than SWEEP_SMALL (10) old strays are found, sweep walks the tree a second time just to count the files (scrub.py:129).

A file qualifies only when it is older than the grace period, POSTULO_MEDIA_SWEEP_GRACE_HOURS, default 24 (config/settings/base.py:733). Running the sweep every five minutes therefore finds, at best, a file that became eligible in the last five minutes of a 24-hour wait; once an hour would delete every orphan at most an hour later than today.

Measured: filestore.listing() over a scratch media root of 2,000 one-byte files in 20 directories, under the test settings with MEDIA_ROOT overridden, took 299 ms on this machine (NTFS; a Linux ext4 or overlay volume is likely several times faster). Cost scales linearly with the number of kept files.

Correction to the scout: the second walk happens only when more than 10 aged strays exist, which on a healthy instance is rare, so the steady cost is one walk per pass, not two.

What would change

Run the sweep at most once per hour (or per grace / 24): remember the last sweep in the cache (cache.add("postulo:media-sweep", 1, 3600) returns False while the hour is not over) or in the heartbeat-style file the scheduler already keeps. The pass skips it otherwise.
In sweep, count the kept files during the same walk instead of walking again: make unreferenced return both the strays and the total, or collect listing() once into a list and derive both.

No library needed.

What it saves or protects

small: one full media walk (about 300 ms per 2,000 files measured on this Windows machine) plus one query per file field, 288 times a day, cut to once an hour or less

Accessibility

none

Decisions

How often should the sweep run? Recommended: hourly; with a 24-hour grace that changes nothing anyone can see.

Found by the efficiency and reliability audit of 2026-10-30 (finding B3; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `src/postulo/notifications/management/commands/send_due_reminders.py:231` calls `scrub.sweep()` on every pass (every 300 s). `src/postulo/documents/scrub.py:112-148` lists the strays through `media.unreferenced` (`core/media.py:41-55`), which first loads every file name every row names (`referenced_names`, lines 22-38: one `values_list` query per `FileField` on every model) and then walks the media root with `filestore.listing` (`documents/filestore.py:95-106`): `listdir` per directory and, per file, `get_modified_time` and `size`, two `stat` calls. When more than `SWEEP_SMALL` (10) old strays are found, `sweep` walks the tree a second time just to count the files (`scrub.py:129`). A file qualifies only when it is older than the grace period, `POSTULO_MEDIA_SWEEP_GRACE_HOURS`, default 24 (`config/settings/base.py:733`). Running the sweep every five minutes therefore finds, at best, a file that became eligible in the last five minutes of a 24-hour wait; once an hour would delete every orphan at most an hour later than today. Measured: `filestore.listing()` over a scratch media root of 2,000 one-byte files in 20 directories, under the test settings with `MEDIA_ROOT` overridden, took 299 ms on this machine (NTFS; a Linux ext4 or overlay volume is likely several times faster). Cost scales linearly with the number of kept files. Correction to the scout: the second walk happens only when more than 10 aged strays exist, which on a healthy instance is rare, so the steady cost is one walk per pass, not two. ## What would change 1. Run the sweep at most once per hour (or per `grace / 24`): remember the last sweep in the cache (`cache.add("postulo:media-sweep", 1, 3600)` returns False while the hour is not over) or in the heartbeat-style file the scheduler already keeps. The pass skips it otherwise. 2. In `sweep`, count the kept files during the same walk instead of walking again: make `unreferenced` return both the strays and the total, or collect `listing()` once into a list and derive both. No library needed. ## What it saves or protects small: one full media walk (about 300 ms per 2,000 files measured on this Windows machine) plus one query per file field, 288 times a day, cut to once an hour or less ## Accessibility none ## Decisions - **How often should the sweep run?** Recommended: hourly; with a 24-hour grace that changes nothing anyone can see. --- Found by the efficiency and reliability audit of 2026-10-30 (finding B3; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:09 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:09 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 10:38:54 +00:00