Test settings: they still read the developer's .env, so the suite differs from one machine to the next #414

New issue

Closed

opened 2026-10-01 10:46:31 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:31 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how the test settings are built and running them, and two security tests, with a few POSTULO_* variables set.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

Related to #76 and #117 (both closed), which fixed particular tests that depended on a developer's .env. This issue is about the cause they left in place: the test settings themselves are still computed from that file.

config/settings/test.py calls itself "independent of the developer's .env", but it starts with from .base import *, and config/settings/base.py:23 reads REPO_DIR / ".env" into the environment and computes every POSTULO_* setting from it at import. The autouse fixture _no_inherited_environment clears those variables from os.environ before each test, which is after the settings exist, so it changes only what site.overridden_by reports. test.py pins some settings (secret key, database, mail, media, static) and leaves the rest to whatever the file says: the cache, background work, the log handler, POSTULO_METRICS_ENABLED, POSTULO_ADMIN_URL, POSTULO_TRUSTED_PROXIES, POSTULO_CONNECTIONS_ALLOW_PRIVATE, the rate limits and the email-code limits.

Where:
- config/settings/base.py:22-23 (environ.Env.read_env(REPO_DIR / ".env"), unconditional), :171-173 (CACHES from POSTULO_CACHE_URL), :347 (POSTULO_METRICS_ENABLED), :383 (POSTULO_ADMIN_URL), :637 (POSTULO_BACKGROUND_WORK), :739 and :782-806 (the file handler exists only when POSTULO_LOG_DIR is non-empty).
- config/settings/test.py:1 (the docstring), :7 (from .base import *) and :39-40 (LOGGING["handlers"]["file"]["filename"] = ...).
- tests/conftest.py:68-82 (_no_inherited_environment).
- tests/security/test_metrics.py:37-40 and tests/security/test_admin_exposure.py:68-71, two tests that assert a default.
What happens: with the variables set as a .env would set them (read_env uses setdefault, so a line in the file and a variable in the environment have the same effect), the test settings carry them:
- POSTULO_METRICS_ENABLED=true and POSTULO_ADMIN_URL=back-office make test_off_by_default_and_saying_nothing and test_the_default_mounts_no_admin_at_all fail (assert True is False, assert 'back-office/' == ''). Without the variables both pass.
- POSTULO_CACHE_URL=redis://localhost:6379/1 (the example in .env.example:99) gives the suite RedisCache at that address. Tests call cache.clear() (tests/security/test_rate_limits.py:39, tests/security/test_candidate_file.py:56), so a run empties the developer's own cache database, and a run with Redis stopped fails in unrelated places.
- POSTULO_BACKGROUND_WORK=true makes errands.send enqueue instead of working inline, so every test that expects an errand to be done finds it waiting.
- POSTULO_LOG_DIR= (empty, to turn the file log off) makes importing the test settings raise KeyError: 'file' at test.py:40, so the suite cannot start.
Why it matters: the same checkout runs a different suite on different machines. A developer who keeps the settings of a live instance in .env sees security tests fail locally that pass in CI. The reverse also holds: a .env can make a test pass locally that fails in CI, or hide a regression in the default of a security setting. #76 and #117 each fixed a symptom (the subprocess reader in tests/security/conftest.py, which stubs read_env for exactly this reason, and the per-test environment scrub); the next setting read at import will repeat the problem.
Fix: stop reading .env for the test settings. Either move the read_env call out of base.py into dev.py and prod.py, or guard it so that it is skipped when DJANGO_SETTINGS_MODULE is the test module (or when test.py has set a flag before the import). pytest-django then builds the test settings from the process environment alone. Because a variable exported in the shell still reaches base.py, also pin in test.py what decides behaviour: CACHES = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}} (or the database cache on the test database), POSTULO_BACKGROUND_WORK = False, and build the file handler unconditionally rather than indexing into one that base.py may have left out. Keep the docstring of test.py true.

Done when

A test in tests/test_settings_isolation.py (new) loads postulo.config.settings.test in a subprocess with a temporary .env setting POSTULO_ADMIN_URL and POSTULO_METRICS_ENABLED, and asserts that both keep their defaults. It fails today.
The same file imports the test settings in a subprocess with POSTULO_LOG_DIR=, POSTULO_CACHE_URL=redis://localhost:6379/1 and POSTULO_BACKGROUND_WORK=true in the environment, and asserts that the import succeeds, the cache is not Redis and POSTULO_BACKGROUND_WORK is false. It fails today with KeyError: 'file'.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading how the test settings are built and running them, and two security tests, with a few `POSTULO_*` variables set. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. Related to #76 and #117 (both closed), which fixed particular tests that depended on a developer's `.env`. This issue is about the cause they left in place: the test settings themselves are still computed from that file. `config/settings/test.py` calls itself "independent of the developer's .env", but it starts with `from .base import *`, and `config/settings/base.py:23` reads `REPO_DIR / ".env"` into the environment and computes every `POSTULO_*` setting from it at import. The autouse fixture `_no_inherited_environment` clears those variables from `os.environ` before each test, which is after the settings exist, so it changes only what `site.overridden_by` reports. `test.py` pins some settings (secret key, database, mail, media, static) and leaves the rest to whatever the file says: the cache, background work, the log handler, `POSTULO_METRICS_ENABLED`, `POSTULO_ADMIN_URL`, `POSTULO_TRUSTED_PROXIES`, `POSTULO_CONNECTIONS_ALLOW_PRIVATE`, the rate limits and the email-code limits. - **Where:** - `config/settings/base.py:22-23` (`environ.Env.read_env(REPO_DIR / ".env")`, unconditional), `:171-173` (`CACHES` from `POSTULO_CACHE_URL`), `:347` (`POSTULO_METRICS_ENABLED`), `:383` (`POSTULO_ADMIN_URL`), `:637` (`POSTULO_BACKGROUND_WORK`), `:739` and `:782-806` (the `file` handler exists only when `POSTULO_LOG_DIR` is non-empty). - `config/settings/test.py:1` (the docstring), `:7` (`from .base import *`) and `:39-40` (`LOGGING["handlers"]["file"]["filename"] = ...`). - `tests/conftest.py:68-82` (`_no_inherited_environment`). - `tests/security/test_metrics.py:37-40` and `tests/security/test_admin_exposure.py:68-71`, two tests that assert a default. - **What happens:** with the variables set as a `.env` would set them (`read_env` uses `setdefault`, so a line in the file and a variable in the environment have the same effect), the test settings carry them: - `POSTULO_METRICS_ENABLED=true` and `POSTULO_ADMIN_URL=back-office` make `test_off_by_default_and_saying_nothing` and `test_the_default_mounts_no_admin_at_all` fail (`assert True is False`, `assert 'back-office/' == ''`). Without the variables both pass. - `POSTULO_CACHE_URL=redis://localhost:6379/1` (the example in `.env.example:99`) gives the suite `RedisCache` at that address. Tests call `cache.clear()` (`tests/security/test_rate_limits.py:39`, `tests/security/test_candidate_file.py:56`), so a run empties the developer's own cache database, and a run with Redis stopped fails in unrelated places. - `POSTULO_BACKGROUND_WORK=true` makes `errands.send` enqueue instead of working inline, so every test that expects an errand to be done finds it waiting. - `POSTULO_LOG_DIR=` (empty, to turn the file log off) makes importing the test settings raise `KeyError: 'file'` at `test.py:40`, so the suite cannot start. - **Why it matters:** the same checkout runs a different suite on different machines. A developer who keeps the settings of a live instance in `.env` sees security tests fail locally that pass in CI. The reverse also holds: a `.env` can make a test pass locally that fails in CI, or hide a regression in the default of a security setting. #76 and #117 each fixed a symptom (the subprocess reader in `tests/security/conftest.py`, which stubs `read_env` for exactly this reason, and the per-test environment scrub); the next setting read at import will repeat the problem. - **Fix:** stop reading `.env` for the test settings. Either move the `read_env` call out of `base.py` into `dev.py` and `prod.py`, or guard it so that it is skipped when `DJANGO_SETTINGS_MODULE` is the test module (or when `test.py` has set a flag before the import). pytest-django then builds the test settings from the process environment alone. Because a variable exported in the shell still reaches `base.py`, also pin in `test.py` what decides behaviour: `CACHES = {"default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}}` (or the database cache on the test database), `POSTULO_BACKGROUND_WORK = False`, and build the `file` handler unconditionally rather than indexing into one that `base.py` may have left out. Keep the docstring of `test.py` true. ## Done when - A test in `tests/test_settings_isolation.py` (new) loads `postulo.config.settings.test` in a subprocess with a temporary `.env` setting `POSTULO_ADMIN_URL` and `POSTULO_METRICS_ENABLED`, and asserts that both keep their defaults. It fails today. - The same file imports the test settings in a subprocess with `POSTULO_LOG_DIR=`, `POSTULO_CACHE_URL=redis://localhost:6379/1` and `POSTULO_BACKGROUND_WORK=true` in the environment, and asserts that the import succeeds, the cache is not Redis and `POSTULO_BACKGROUND_WORK` is false. It fails today with `KeyError: 'file'`.

tiagoagueda added this to the 0.6.0 milestone

2026-10-01 10:46:31 +00:00

tiagoagueda added the

bug

tier

labels

2026-10-01 10:46:31 +00:00

tiagoagueda referenced this issue from a commit

2026-10-04 09:26:54 +00:00

Build the test settings from the process environment, never from .env

tiagoagueda closed this issue

2026-10-04 09:26:54 +00:00