Postulo/postulo

Watch

Fork 0

Caches: expired entries of the database cache and the rate-limit file cache are never removed until 200,000 pile up, and every write counts or lists them all #1167

New issue

Closed

opened 2026-10-31 03:04:28 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:28 +00:00

Owner

This issue holds 2 findings of the audit that share one fix or the same files.

D1: Cache: expired rows of the database cache are never removed until it holds 200,000, while every write counts the whole table

What exists today

src/postulo/config/settings/base.py:212-223: the default cache is Django's DatabaseCache on table postulo_cache, with MAX_ENTRIES raised to 200,000 (POSTULO_CACHE_MAX_ENTRIES) so that rate-limit and lockout counters cannot be culled by key order (#755).

Django 6.1's backend (.venv/Lib/site-packages/django/core/cache/backends/db.py):

_base_set (line 113) runs SELECT COUNT(*) on the table on every set, add and touch-style write (line 121) and calls _cull only when the count exceeds MAX_ENTRIES (line 131).
_cull (line 258) is the only place that deletes expired rows in bulk (DELETE … WHERE expires < now, line 265).
get_many deletes an expired row only when that same key is read again (lines 92 and 226).

So a key that is never asked for again stays until the table passes 200,000 rows. Postulo writes such keys all the time: allauth's rate limits keep a history per address or account in the default cache (allauth/core/internal/ratelimit.py:67, :211, plus a lock key per hit at :39); the dashboard figures are cached under a fingerprint that changes whenever the person's data changes (applications/analytics.py:435-440, 24-hour TTL), so every edit leaves the previous key behind; backup check results are keyed by file name, size and mtime (core/backups.py:102-108, 30 days). Leases (core/scheduler.py:103, plugins/syncing.py:100, core/backups.py:245) are deleted by their owners and are not the problem.

Measured: a scratch SQLite table shaped like postulo_cache with 200,000 rows of 200-byte values is 55 MB, and SELECT COUNT(*) on it takes about 60 ms on this machine (three runs: 65, 58, 62 ms). Every cache write on a long-lived instance pays a share of that, inside the request's transaction.

What would change

Delete expired rows on a schedule: one statement, DELETE FROM postulo_cache WHERE expires < now, run from the scheduler pass and from core/slow.py:tidy_up_if_due (the hourly request-time tidy, so an instance without a scheduler is covered too), only when the default cache is the database backend. Django has no public method for it; the statement is the same one _cull issues, and can be written with the backend's own model (cache.cache_model_class) or the connection's quote_name exactly as _cull does.

Deleting only rows already past their expiry cannot remove a live counter, so #755's protection is unchanged. The COUNT(*) per write stays (it is Django's), but on a table of live rows only it costs microseconds. A test: an expired row and a live one, a pass, the expired one is gone and the live one and its value remain.

What it saves or protects

medium: the cache table stays at its live size instead of growing towards 200,000 rows (about 55 MB in a scratch SQLite copy), and the COUNT(*) every cache write runs stays cheap (measured 60 ms at 200,000 rows)

Accessibility

none

Decisions

Hourly or every pass? Recommended: hourly, through the same once-an-hour key tidy_up_if_due uses; the scheduler pass may call it every time since the add gate makes it cheap.

D2: Rate limits: the file cache keeps a file per counter per window for ever and lists the whole directory on every write

What exists today

src/postulo/config/settings/base.py:231-236: when the default cache is the database one, Postulo's own allowances use Django's FileBasedCache in data/ratelimit with MAX_ENTRIES 200,000 (#756).

src/postulo/core/throttle.py:114-117 names a counter postulo:rate:<action>:<who>:<window>, the window being int(now) // rate.seconds. Every default rate is per hour (base.py:496-530), so each action and caller gets a new key, and therefore a new file, every hour. consume (lines 120-152) does add (which is has_key plus set), then incr (a get plus set) and touch on later hits.

Django 6.1's backend (.venv/Lib/site-packages/django/core/cache/backends/filebased.py):

set calls _cull() every time (line 51); _cull globs the whole directory (_list_cache_files, lines 161-169) and returns early unless the count reaches MAX_ENTRIES (lines 107-110).
An expired file is deleted only when that same key is opened again (_is_expired, lines 147-159). A past window's key is never built again, so its file is never opened again.
At 200,000 files _cull deletes a random third (line 114), live counters included.

So every hour of API use by a token leaves two files (api and api-account, throttle.py:221-224), every public endpoint caller one (endpoint, line 285), and so on, none of them ever removed until the directory reaches 200,000 files, when a random third, possibly including live counters, goes.

Measured: glob("*.djcache") over a scratch directory of 20,000 empty cache-named files took 75 to 82 ms on this Windows machine (three runs); that is the cost added to every set once the directory has grown that far. ext4 would be faster, the growth is the same.

What would change

Add a purge of expired files to the hourly tidy proposed in D1 (scheduler pass and core/slow.py:tidy_up_if_due), only when caches["ratelimit"] is a FileBasedCache: for each file the backend lists, open it and let the backend's own expiry check decide (_is_expired deletes the file when it is past its expiry). Those two methods are private but have been stable in Django for over a decade; the alternative is reading the leading pickled expiry by hand, which duplicates them. A file whose counter is still live is untouched, so no allowance is reset.

No library replaces this short of moving the counters to Redis, which stays the operator's choice (POSTULO_CACHE_URL).

What it saves or protects

medium: the data/ratelimit directory stays at the live counters (dozens of files) instead of growing by one file per action, caller and hour; each counted request stops paying a directory listing that grows with it (measured 80 ms at 20,000 files on this machine)

Accessibility

none

Decisions

One issue with D1 or two? Recommended: one PR, "Cache: delete expired entries hourly", covering both backends, since they share the trigger and the test shape; split into: D1 (database cache rows) and D2 (rate-limit files) only if the tracker wants one issue per backend.

Found by the efficiency and reliability audit of 2026-10-30 (finding D1, D2; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

This issue holds 2 findings of the audit that share one fix or the same files. ## D1: Cache: expired rows of the database cache are never removed until it holds 200,000, while every write counts the whole table ### What exists today `src/postulo/config/settings/base.py:212-223`: the default cache is Django's `DatabaseCache` on table `postulo_cache`, with `MAX_ENTRIES` raised to 200,000 (`POSTULO_CACHE_MAX_ENTRIES`) so that rate-limit and lockout counters cannot be culled by key order (#755). Django 6.1's backend (`.venv/Lib/site-packages/django/core/cache/backends/db.py`): - `_base_set` (line 113) runs `SELECT COUNT(*)` on the table on every `set`, `add` and `touch`-style write (line 121) and calls `_cull` only when the count exceeds `MAX_ENTRIES` (line 131). - `_cull` (line 258) is the only place that deletes expired rows in bulk (`DELETE … WHERE expires < now`, line 265). - `get_many` deletes an expired row only when that same key is read again (lines 92 and 226). So a key that is never asked for again stays until the table passes 200,000 rows. Postulo writes such keys all the time: allauth's rate limits keep a history per address or account in the default cache (`allauth/core/internal/ratelimit.py:67`, `:211`, plus a lock key per hit at `:39`); the dashboard figures are cached under a fingerprint that changes whenever the person's data changes (`applications/analytics.py:435-440`, 24-hour TTL), so every edit leaves the previous key behind; backup check results are keyed by file name, size and mtime (`core/backups.py:102-108`, 30 days). Leases (`core/scheduler.py:103`, `plugins/syncing.py:100`, `core/backups.py:245`) are deleted by their owners and are not the problem. Measured: a scratch SQLite table shaped like `postulo_cache` with 200,000 rows of 200-byte values is 55 MB, and `SELECT COUNT(*)` on it takes about 60 ms on this machine (three runs: 65, 58, 62 ms). Every cache write on a long-lived instance pays a share of that, inside the request's transaction. ### What would change Delete expired rows on a schedule: one statement, `DELETE FROM postulo_cache WHERE expires < now`, run from the scheduler pass and from `core/slow.py:tidy_up_if_due` (the hourly request-time tidy, so an instance without a scheduler is covered too), only when the default cache is the database backend. Django has no public method for it; the statement is the same one `_cull` issues, and can be written with the backend's own model (`cache.cache_model_class`) or the connection's `quote_name` exactly as `_cull` does. Deleting only rows already past their expiry cannot remove a live counter, so #755's protection is unchanged. The `COUNT(*)` per write stays (it is Django's), but on a table of live rows only it costs microseconds. A test: an expired row and a live one, a pass, the expired one is gone and the live one and its value remain. ### What it saves or protects medium: the cache table stays at its live size instead of growing towards 200,000 rows (about 55 MB in a scratch SQLite copy), and the `COUNT(*)` every cache write runs stays cheap (measured 60 ms at 200,000 rows) ### Accessibility none ### Decisions - **Hourly or every pass?** Recommended: hourly, through the same once-an-hour key `tidy_up_if_due` uses; the scheduler pass may call it every time since the `add` gate makes it cheap. ## D2: Rate limits: the file cache keeps a file per counter per window for ever and lists the whole directory on every write ### What exists today `src/postulo/config/settings/base.py:231-236`: when the default cache is the database one, Postulo's own allowances use Django's `FileBasedCache` in `data/ratelimit` with `MAX_ENTRIES` 200,000 (#756). `src/postulo/core/throttle.py:114-117` names a counter `postulo:rate:<action>:<who>:<window>`, the window being `int(now) // rate.seconds`. Every default rate is per hour (`base.py:496-530`), so each action and caller gets a new key, and therefore a new file, every hour. `consume` (lines 120-152) does `add` (which is `has_key` plus `set`), then `incr` (a `get` plus `set`) and `touch` on later hits. Django 6.1's backend (`.venv/Lib/site-packages/django/core/cache/backends/filebased.py`): - `set` calls `_cull()` every time (line 51); `_cull` globs the whole directory (`_list_cache_files`, lines 161-169) and returns early unless the count reaches `MAX_ENTRIES` (lines 107-110). - An expired file is deleted only when that same key is opened again (`_is_expired`, lines 147-159). A past window's key is never built again, so its file is never opened again. - At 200,000 files `_cull` deletes a random third (line 114), live counters included. So every hour of API use by a token leaves two files (`api` and `api-account`, `throttle.py:221-224`), every public endpoint caller one (`endpoint`, line 285), and so on, none of them ever removed until the directory reaches 200,000 files, when a random third, possibly including live counters, goes. Measured: `glob("*.djcache")` over a scratch directory of 20,000 empty cache-named files took 75 to 82 ms on this Windows machine (three runs); that is the cost added to every `set` once the directory has grown that far. ext4 would be faster, the growth is the same. ### What would change Add a purge of expired files to the hourly tidy proposed in D1 (scheduler pass and `core/slow.py:tidy_up_if_due`), only when `caches["ratelimit"]` is a `FileBasedCache`: for each file the backend lists, open it and let the backend's own expiry check decide (`_is_expired` deletes the file when it is past its expiry). Those two methods are private but have been stable in Django for over a decade; the alternative is reading the leading pickled expiry by hand, which duplicates them. A file whose counter is still live is untouched, so no allowance is reset. No library replaces this short of moving the counters to Redis, which stays the operator's choice (`POSTULO_CACHE_URL`). ### What it saves or protects medium: the `data/ratelimit` directory stays at the live counters (dozens of files) instead of growing by one file per action, caller and hour; each counted request stops paying a directory listing that grows with it (measured 80 ms at 20,000 files on this machine) ### Accessibility none ### Decisions - **One issue with D1 or two?** Recommended: one PR, "Cache: delete expired entries hourly", covering both backends, since they share the trigger and the test shape; split into: D1 (database cache rows) and D2 (rate-limit files) only if the tracker wants one issue per backend. --- Found by the efficiency and reliability audit of 2026-10-30 (finding D1, D2; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:28 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:28 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a pull request that will close it,

2026-11-11 10:38:54 +00:00