Container: gunicorn loads the app separately in each of three workers and reloads the data caches every 500 requests #1111

New issue

Open

opened 2026-10-31 03:04:04 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:04 +00:00

Owner

What exists today

docker/Dockerfile:379 sets ENV GUNICORN_CMD_ARGS="--timeout 120 --max-requests 500 --max-requests-jitter 50" and :387-392 runs gunicorn postulo.config.wsgi:application --bind 0.0.0.0:8000 --workers 3 ... (gunicorn 26.2.0 in uv.lock). There is no --preload, no gc.freeze(), and no --worker-tmp-dir.

So each of the three workers imports Django, every app and every plugin, and builds the registries on its own. Measured with tracemalloc from import postulo.config.wsgi to a resolved URLconf under the test settings (audit/m_scripts/boot.py): 88.2 MB traced. Then each worker builds the lazy caches of M1 (about 57 MB, 5.6 s), M2 (62 MB, 2.8 s; the two share Babel's cache, together about 64 MB) and M3 (14.6 MB, 0.2 s) on first use, and --max-requests 500 throws them away and builds them again at every refill. On a Raspberry Pi the 9 s of first-use CPU is paid by whoever makes the first request that touches them.

What changed from the candidate:

--preload alone shares only what is built before the fork. The M1 to M3 caches are lazy (lru_cache on first call), so they would still be built per worker unless the fix also warms them in the master; and fixing M1, M2 (#1086) and M3 removes most of what is worth sharing. Order: those three first, then this.
CPython reference counting dirties shared pages, so the saving from --preload is partial unless gc.freeze() is called before the fork (the documented pattern, Python 3.7+).
--worker-tmp-dir /dev/shm matters only where /tmp is on disk: docker/compose.hardened.yml:25-26 already mounts /tmp as tmpfs, docker/compose.yml does not. The heartbeat file on an SD card is a small write every few seconds per worker, a reliability point on a Pi rather than a memory one. Check gunicorn 26's notes for whether its heartbeat still uses a temporary file.
Preloading is safe with the current code as read: no ready() in */apps.py queries the database or starts threads (plugins/public_addresses.py:50 creates its ThreadPoolExecutor at import, which starts no thread until the first submit), and plugins installed through the interface are picked up per worker through the registry stamp (plugins/registry.py:310), not by a restart.

What would change

A small postulo/config/gunicorn.py (gunicorn's Python config file) loaded by the CMD with -c, so the defaults live in one reviewed file while GUNICORN_CMD_ARGS still overrides any flag:

preload_app = True;
worker_tmp_dir = "/dev/shm";
a when_ready (or the end of the preload) that calls gc.freeze();
optionally, after M1 to M3 are fixed, a warm-up of what is left of the data caches in the master.

Keep --workers 3, --timeout 120, --max-requests 500 --max-requests-jitter 50 as they are. The entrypoint's database and plugin sync steps already run before gunicorn starts, so nothing there changes. CONN_MAX_AGE (candidate R4) is a separate setting and a separate issue.

What it saves or protects

medium: about 88 MB of imported code and registries (traced at boot) shared across three workers instead of built three times, and the M1 to M3 caches (about 75 MB, 9 s of CPU on first use) not rebuilt at every worker refill

Accessibility

none

Decided by the maintainer (2026-10-31)

yes, after the memory fixes (M1, M3, #1086): gunicorn config file with preload_app, gc.freeze(), worker_tmp_dir /dev/shm, documented with how to turn each off

Decisions

Change the defaults for every operator? Recommended: yes, after M1, M2 (#1086) and M3 land; --preload with gc.freeze() and worker_tmp_dir = "/dev/shm" in a gunicorn config file, documented in the operator page with how to turn each off.
Split? Recommended: one issue; the three settings are one config file and one review.

Found by the efficiency and reliability audit of 2026-10-30 (finding M5; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `docker/Dockerfile:379` sets `ENV GUNICORN_CMD_ARGS="--timeout 120 --max-requests 500 --max-requests-jitter 50"` and `:387-392` runs `gunicorn postulo.config.wsgi:application --bind 0.0.0.0:8000 --workers 3 ...` (gunicorn 26.2.0 in `uv.lock`). There is no `--preload`, no `gc.freeze()`, and no `--worker-tmp-dir`. So each of the three workers imports Django, every app and every plugin, and builds the registries on its own. Measured with `tracemalloc` from `import postulo.config.wsgi` to a resolved URLconf under the test settings (`audit/m_scripts/boot.py`): 88.2 MB traced. Then each worker builds the lazy caches of M1 (about 57 MB, 5.6 s), M2 (62 MB, 2.8 s; the two share Babel's cache, together about 64 MB) and M3 (14.6 MB, 0.2 s) on first use, and `--max-requests 500` throws them away and builds them again at every refill. On a Raspberry Pi the 9 s of first-use CPU is paid by whoever makes the first request that touches them. What changed from the candidate: - `--preload` alone shares only what is built before the fork. The M1 to M3 caches are lazy (`lru_cache` on first call), so they would still be built per worker unless the fix also warms them in the master; and fixing M1, M2 (#1086) and M3 removes most of what is worth sharing. Order: those three first, then this. - CPython reference counting dirties shared pages, so the saving from `--preload` is partial unless `gc.freeze()` is called before the fork (the documented pattern, Python 3.7+). - `--worker-tmp-dir /dev/shm` matters only where `/tmp` is on disk: `docker/compose.hardened.yml:25-26` already mounts `/tmp` as tmpfs, `docker/compose.yml` does not. The heartbeat file on an SD card is a small write every few seconds per worker, a reliability point on a Pi rather than a memory one. Check gunicorn 26's notes for whether its heartbeat still uses a temporary file. - Preloading is safe with the current code as read: no `ready()` in `*/apps.py` queries the database or starts threads (`plugins/public_addresses.py:50` creates its `ThreadPoolExecutor` at import, which starts no thread until the first `submit`), and plugins installed through the interface are picked up per worker through the registry stamp (`plugins/registry.py:310`), not by a restart. ## What would change A small `postulo/config/gunicorn.py` (gunicorn's Python config file) loaded by the CMD with `-c`, so the defaults live in one reviewed file while `GUNICORN_CMD_ARGS` still overrides any flag: - `preload_app = True`; - `worker_tmp_dir = "/dev/shm"`; - a `when_ready` (or the end of the preload) that calls `gc.freeze()`; - optionally, after M1 to M3 are fixed, a warm-up of what is left of the data caches in the master. Keep `--workers 3`, `--timeout 120`, `--max-requests 500 --max-requests-jitter 50` as they are. The entrypoint's database and plugin sync steps already run before gunicorn starts, so nothing there changes. `CONN_MAX_AGE` (candidate R4) is a separate setting and a separate issue. ## What it saves or protects medium: about 88 MB of imported code and registries (traced at boot) shared across three workers instead of built three times, and the M1 to M3 caches (about 75 MB, 9 s of CPU on first use) not rebuilt at every worker refill ## Accessibility none ## Decided by the maintainer (2026-10-31) yes, after the memory fixes (M1, M3, #1086): gunicorn config file with preload_app, gc.freeze(), worker_tmp_dir /dev/shm, documented with how to turn each off ## Decisions - **Change the defaults for every operator?** Recommended: yes, after M1, M2 (#1086) and M3 land; `--preload` with `gc.freeze()` and `worker_tmp_dir = "/dev/shm"` in a gunicorn config file, documented in the operator page with how to turn each off. - **Split?** Recommended: one issue; the three settings are one config file and one review. --- Found by the efficiency and reliability audit of 2026-10-30 (finding M5; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 1.0.0 milestone

2026-10-31 03:04:04 +00:00

tiagoagueda added the

performance

enhancement

labels

2026-10-31 03:04:04 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:40 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108