The health check cannot fail in production: SSL redirect answers it before the view does #82

New issue

Closed

opened 2026-09-07 12:42:54 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-09-07 12:42:54 +00:00

Owner

Observation

Found while deploying 0.2.0 behind traefik. With POSTULO_SSL_REDIRECT on — which is the
production default — the container's health check stops checking anything.

docker/Dockerfile:

HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
    CMD curl -fsS http://127.0.0.1:8000/healthz || exit 1

prod.py:

SECURE_SSL_REDIRECT = env.bool("POSTULO_SSL_REDIRECT", default=True)

There is no SECURE_REDIRECT_EXEMPT, so SecurityMiddleware answers that plain-HTTP
request with a 301 to https://127.0.0.1:8000/healthz — before any view runs, and before
anything touches the database.

And curl -f only fails on 4xx and 5xx. Verified rather than assumed:

$ curl -fsS http://127.0.0.1:8899/healthz   # a server that only ever 301s
$ echo $?
0

Empty body, exit 0. The check passes.

What that means

The health check reports healthy whenever SecurityMiddleware is loaded, which is always.
It would report healthy with the database gone, the migrations unapplied, every view
raising — anything the healthz view was written to detect:

def healthz(request):
    try:
        connection.ensure_connection()
    except Exception:
        return JsonResponse({"status": "error", "database": "unavailable"}, status=503)

That 503 is unreachable in production. So is the restart that restart: unless-stopped plus
a failing health check would have produced. Every production deployment of the shipped
image has a liveness probe that cannot fail.

It is invisible for the usual reason: a health check that always passes looks exactly like a
healthy service.

The fix

# The liveness probe is the one thing that must answer over plain HTTP: it is curl inside
# the container talking to 127.0.0.1, where there is no TLS to redirect to.
SECURE_REDIRECT_EXEMPT = [r"^healthz$"]

With a test that asserts /healthz returns 200 under production settings with the redirect
on, and that everything else still redirects — the second half matters, since an exemption
pattern that is too broad would be worse than the bug.

Worth checking /metrics at the same time: a Prometheus scraper reaching it over plain HTTP
inside the network would be redirected too, and a scraper follows redirects less politely
than a browser does.

Workaround in the meantime

Set POSTULO_SSL_REDIRECT=false and let the reverse proxy redirect at its entrypoint, which
is what traefik does anyway. That is what the ragnar deployment does, with a comment saying
why, and it is a perfectly good production posture — but it should be a choice rather than
the only configuration in which the health check works.

Classification

Bug. Not a security hole: the redirect still happens for real traffic, and nothing is
exposed. What is lost is the ability to notice that the application has stopped working.

## Observation Found while deploying 0.2.0 behind traefik. **With `POSTULO_SSL_REDIRECT` on — which is the production default — the container's health check stops checking anything.** `docker/Dockerfile`: ```dockerfile HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \ CMD curl -fsS http://127.0.0.1:8000/healthz || exit 1 ``` `prod.py`: ```python SECURE_SSL_REDIRECT = env.bool("POSTULO_SSL_REDIRECT", default=True) ``` There is no `SECURE_REDIRECT_EXEMPT`, so `SecurityMiddleware` answers that plain-HTTP request with a 301 to `https://127.0.0.1:8000/healthz` — **before any view runs, and before anything touches the database**. And `curl -f` only fails on 4xx and 5xx. Verified rather than assumed: ``` $ curl -fsS http://127.0.0.1:8899/healthz # a server that only ever 301s $ echo $? 0 ``` Empty body, exit 0. The check passes. ## What that means The health check reports healthy whenever `SecurityMiddleware` is loaded, which is always. It would report healthy with the database gone, the migrations unapplied, every view raising — anything the `healthz` view was written to detect: ```python def healthz(request): try: connection.ensure_connection() except Exception: return JsonResponse({"status": "error", "database": "unavailable"}, status=503) ``` That 503 is unreachable in production. So is the restart that `restart: unless-stopped` plus a failing health check would have produced. **Every production deployment of the shipped image has a liveness probe that cannot fail.** It is invisible for the usual reason: a health check that always passes looks exactly like a healthy service. ## The fix ```python # The liveness probe is the one thing that must answer over plain HTTP: it is curl inside # the container talking to 127.0.0.1, where there is no TLS to redirect to. SECURE_REDIRECT_EXEMPT = [r"^healthz$"] ``` With a test that asserts `/healthz` returns 200 under production settings with the redirect on, and that everything else still redirects — the second half matters, since an exemption pattern that is too broad would be worse than the bug. Worth checking `/metrics` at the same time: a Prometheus scraper reaching it over plain HTTP inside the network would be redirected too, and a scraper follows redirects less politely than a browser does. ## Workaround in the meantime Set `POSTULO_SSL_REDIRECT=false` and let the reverse proxy redirect at its entrypoint, which is what traefik does anyway. That is what the ragnar deployment does, with a comment saying why, and it is a perfectly good production posture — but it should be a choice rather than the only configuration in which the health check works. ## Classification Bug. Not a security hole: the redirect still happens for real traffic, and nothing is exposed. What is lost is the ability to notice that the application has stopped working.

tiagoagueda added this to the 0.3.0 milestone

2026-09-07 12:42:54 +00:00

tiagoagueda added the

bug

tier

observability

labels

2026-09-07 12:42:54 +00:00

tiagoagueda referenced this issue from a commit

2026-09-07 15:10:46 +00:00

Let the health check fail

tiagoagueda closed this issue