CI has tested nothing for a fortnight: a collection error only a developer's .env hides #117

New issue

Closed

opened 2026-09-08 07:42:03 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-08 07:42:03 +00:00

Owner

Observation

Every CI run since run 128 has failed, on main, on every push — fourteen consecutive commits. The failure is in test (3.12), test (3.13), test (3.14) and browser; styles and security pass throughout.

ERROR collecting tests/security/test_health_redirect.py
tests/security/test_health_redirect.py:35: in <module>
    from postulo.config.settings.prod import SECURE_REDIRECT_EXEMPT
src/postulo/config/settings/prod.py:10: in ImproperlyConfigured
E   POSTULO_SECRET_KEY must be set.
!!!!! Interrupted: 1 error during collection !!!!!

Why it went unnoticed for a fortnight

It passes on a developer's machine. config/settings/prod.py refuses to import without POSTULO_SECRET_KEY, which is correct of it. The repository's .env — gitignored, and belonging to whoever is developing here — supplies one, so the module imports locally and the suite is green. CI has no .env, so the import raises.

It is a collection error, so it aborts the whole run. Not one failing test: no tests run at all. test fell from ~100s to ~35s and browser from 172s to ~64s, which is the shape of a suite that never started.

And the two jobs that pass are the two that never run pytest. styles builds the stylesheet, security audits dependencies. So the signal was "CI is red" without any indication that nothing was being tested.

The same class of bug was found and fixed during #84 — tests/security/test_health_redirect.py also depended on the developer's .env supplying POSTULO_TIME_ZONE — and an autouse fixture now clears the ENV_OVERRIDES variables for every test. It could not have caught this one: POSTULO_SECRET_KEY is not one of those variables, and a collection error happens before any fixture runs.

The irony worth recording

tests/security/test_requests.py, in the same directory, had already solved this exactly right, and says so in a comment:

The repository's .env belongs to whoever is developing here. What is under test is what the module itself says, so the file is taken out of the picture rather than trusted to be absent.

It reads the production settings in a subprocess with every POSTULO_* variable stripped. The file next to it imported the module directly.

The fix

One way to read production, in tests/security/conftest.py, used by both files: a session-scoped fixture running that subprocess. test_health_redirect.py takes the exemption list from the fixture instead of importing prod at module scope, which keeps the property it wanted — the list is read from what ships, never retyped — without needing a secret key to collect.

Reproduced by moving .env aside: collection aborts with CI's exact error before the change, and 1595 unit tests and 33 browser tests pass after it.

Worth doing separately

Nothing in CI says "the suite did not run" differently from "the suite failed". A collection error and a failing assertion look the same from the outside, and this one hid for a fortnight behind a red mark that everybody had stopped reading.

Classification

Bug. Tier 2: no user-visible defect, and the project's stated assurance — a suite that runs on every push — was not true for a fortnight.

## Observation Every CI run since **run 128** has failed, on `main`, on every push — fourteen consecutive commits. The failure is in `test (3.12)`, `test (3.13)`, `test (3.14)` and `browser`; `styles` and `security` pass throughout. ``` ERROR collecting tests/security/test_health_redirect.py tests/security/test_health_redirect.py:35: in <module> from postulo.config.settings.prod import SECURE_REDIRECT_EXEMPT src/postulo/config/settings/prod.py:10: in ImproperlyConfigured E POSTULO_SECRET_KEY must be set. !!!!! Interrupted: 1 error during collection !!!!! ``` ## Why it went unnoticed for a fortnight **It passes on a developer's machine.** `config/settings/prod.py` refuses to import without `POSTULO_SECRET_KEY`, which is correct of it. The repository's `.env` — gitignored, and belonging to whoever is developing here — supplies one, so the module imports locally and the suite is green. CI has no `.env`, so the import raises. **It is a collection error, so it aborts the whole run.** Not one failing test: no tests run at all. `test` fell from ~100s to ~35s and `browser` from 172s to ~64s, which is the shape of a suite that never started. **And the two jobs that pass are the two that never run pytest.** `styles` builds the stylesheet, `security` audits dependencies. So the signal was "CI is red" without any indication that *nothing was being tested*. The same class of bug was found and fixed during #84 — `tests/security/test_health_redirect.py` also depended on the developer's `.env` supplying `POSTULO_TIME_ZONE` — and an autouse fixture now clears the `ENV_OVERRIDES` variables for every test. It could not have caught this one: `POSTULO_SECRET_KEY` is not one of those variables, and a collection error happens before any fixture runs. ## The irony worth recording `tests/security/test_requests.py`, in the same directory, had already solved this exactly right, and says so in a comment: > The repository's .env belongs to whoever is developing here. What is under test is what the module itself says, so the file is taken out of the picture rather than trusted to be absent. It reads the production settings in a subprocess with every `POSTULO_*` variable stripped. The file next to it imported the module directly. ## The fix One way to read production, in `tests/security/conftest.py`, used by both files: a session-scoped fixture running that subprocess. `test_health_redirect.py` takes the exemption list from the fixture instead of importing `prod` at module scope, which keeps the property it wanted — the list is read from what ships, never retyped — without needing a secret key to collect. Reproduced by moving `.env` aside: collection aborts with CI's exact error before the change, and 1595 unit tests and 33 browser tests pass after it. ## Worth doing separately Nothing in CI says "the suite did not run" differently from "the suite failed". A collection error and a failing assertion look the same from the outside, and this one hid for a fortnight behind a red mark that everybody had stopped reading. ## Classification Bug. Tier 2: no user-visible defect, and the project's stated assurance — a suite that runs on every push — was not true for a fortnight.

tiagoagueda added this to the 0.3.0 milestone

2026-09-08 07:42:03 +00:00

tiagoagueda added the

bug

tier

labels

2026-09-08 07:42:03 +00:00

tiagoagueda referenced this issue from a commit

2026-09-08 07:42:46 +00:00

Stop the test suite depending on a file CI does not have

tiagoagueda closed this issue

2026-09-08 07:42:46 +00:00