Server settings → Overview: every visit walks the whole media directory to count files and bytes #488

New issue

Closed

opened 2026-10-01 10:47:03 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-10-01 10:47:03 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the Overview view and timing its directory walk.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

The Overview page, where /server/ redirects, counts the files under MEDIA_ROOT and adds up their sizes on every request. It does so with rglob("*") plus an is_file() and a stat() for each entry, across every account's uploads, rendered documents and kept capture pages, synchronously and with nothing cached. The cost grows with the instance's whole history and is paid on each visit.

Where: core/server_views.py:65-73, _directory_size; called from OverviewView.get_context_data at :134.
What happens: on an instance with years of uploads and kept captures, or with media on a network volume, opening Server settings takes as long as the walk and holds a worker for that time. As a measure, 20,000 small files in 100 folders took 1.4 s to walk in the snapshot's interpreter on a local Windows disk; a network mount is slower for each stat().
Why it matters: the first page of Server settings slows down as the instance ages, for a figure that changes slowly and that nobody needs to the second.
Fix: cache the pair for a few minutes (the page already shows stored answers, such as updates.status()), or have the scheduler's pass compute and store it and let the page show what was stored with its age; alternatively sum the sizes the database already records for the files it owns and walk the directory only on request.

Done when

A test in tests/test_server_settings.py renders the Overview twice with _directory_size patched to count its calls and finds it called at most once; it fails today with two.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the Overview view and timing its directory walk. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The Overview page, where `/server/` redirects, counts the files under `MEDIA_ROOT` and adds up their sizes on every request. It does so with `rglob("*")` plus an `is_file()` and a `stat()` for each entry, across every account's uploads, rendered documents and kept capture pages, synchronously and with nothing cached. The cost grows with the instance's whole history and is paid on each visit. - **Where:** `core/server_views.py:65-73`, `_directory_size`; called from `OverviewView.get_context_data` at `:134`. - **What happens:** on an instance with years of uploads and kept captures, or with media on a network volume, opening Server settings takes as long as the walk and holds a worker for that time. As a measure, 20,000 small files in 100 folders took 1.4 s to walk in the snapshot's interpreter on a local Windows disk; a network mount is slower for each `stat()`. - **Why it matters:** the first page of Server settings slows down as the instance ages, for a figure that changes slowly and that nobody needs to the second. - **Fix:** cache the pair for a few minutes (the page already shows stored answers, such as `updates.status()`), or have the scheduler's pass compute and store it and let the page show what was stored with its age; alternatively sum the sizes the database already records for the files it owns and walk the directory only on request. ## Done when - A test in `tests/test_server_settings.py` renders the Overview twice with `_directory_size` patched to count its calls and finds it called at most once; it fails today with two.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:47:03 +00:00

tiagoagueda added the

enhancement

tier

labels

2026-10-01 10:47:03 +00:00

tiagoagueda referenced this issue from a commit

2026-10-05 23:05:04 +00:00

Cache the media file count and size on the server Overview for five minutes

tiagoagueda commented

2026-10-06 09:31:59 +00:00