Image health check: answered 400 unless POSTULO_ALLOWED_HOSTS keeps 127.0.0.1, so scheduler and worker never start #580

New issue

Closed

opened 2026-10-01 10:47:48 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:47:48 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the image's health check against the settings and confirming it with the test client.

Checked again on 2026-10-01 against main at 0a5cad587: still present.

The image's HEALTHCHECK asks http://127.0.0.1:8000/healthz, which sends Host: 127.0.0.1:8000. CommonMiddleware validates the host before any view runs, so on an instance whose POSTULO_ALLOWED_HOSTS names only its public host, as the install docs show, every probe is refused with 400 and the container is marked unhealthy while it serves pages normally. The scheduler and worker services wait for a healthy web container, so they never start.

Where: docker/Dockerfile:334-335 (the HEALTHCHECK); config/settings/base.py:29 (ALLOWED_HOSTS from the environment, default localhost,127.0.0.1); config/settings/prod.py:55 exempts healthz from the HTTPS redirect only; docker/compose.yml:74-79 and :118-123, docker/compose.postgres.yml:76-83 (depends_on: postulo: condition: service_healthy); the wiki's Installing-Postulo.md:8 and :193 (POSTULO_ALLOWED_HOSTS=postulo.example.org) and Troubleshooting.md:199-201; scripts/check-image.sh:33, which always runs with localhost,127.0.0.1.
What happens:
1. An operator replaces localhost,127.0.0.1 in .env with their domain, as the README and wiki instruct.
2. The web container answers through the proxy, but every 30 seconds the probe gets 400 Bad Request and a DisallowedHost error with its traceback is logged; Docker marks the container unhealthy.
3. docker compose --profile scheduler up -d and --profile worker up -d wait on service_healthy and never start, so reminders, gone-quiet notices, store copies, syncs and queued background work never run.
4. Troubleshooting says an unhealthy container means /healthz "answered 503 or did not answer at all", which sends the operator to the database.
  With the test client and ALLOWED_HOSTS=["postulo.example.org"], GET /healthz with Host: 127.0.0.1:8000 returns 400; with 127.0.0.1 added it returns 200.
Why it matters: a healthy instance reports itself broken, the background services silently never run, and the probe can no longer tell a real failure from this one.
Fix: make the probe independent of the operator's host list. Either add a small middleware placed first that answers GET /healthz from a loopback REMOTE_ADDR before host validation, or have the production settings always admit localhost and send -H 'Host: localhost' from the HEALTHCHECK. Say in Health, metrics and logs which host the probe uses, and correct the Troubleshooting sentence. Related to #82 (closed), which fixed the HTTPS-redirect half of the same probe but not the host check.

Done when

A test in tests/test_image_build.py (or a new tests/core/test_healthz.py) requests /healthz exactly as the HEALTHCHECK does, under override_settings(ALLOWED_HOSTS=["postulo.example.org"]), and expects 200; it gets 400 today.
scripts/check-image.sh gains a run with POSTULO_ALLOWED_HOSTS=postulo.example.org that waits for Docker's own health status to become healthy.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the image's health check against the settings and confirming it with the test client. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The image's `HEALTHCHECK` asks `http://127.0.0.1:8000/healthz`, which sends `Host: 127.0.0.1:8000`. `CommonMiddleware` validates the host before any view runs, so on an instance whose `POSTULO_ALLOWED_HOSTS` names only its public host, as the install docs show, every probe is refused with 400 and the container is marked unhealthy while it serves pages normally. The scheduler and worker services wait for a healthy web container, so they never start. - **Where:** `docker/Dockerfile:334-335` (the `HEALTHCHECK`); `config/settings/base.py:29` (`ALLOWED_HOSTS` from the environment, default `localhost,127.0.0.1`); `config/settings/prod.py:55` exempts `healthz` from the HTTPS redirect only; `docker/compose.yml:74-79` and `:118-123`, `docker/compose.postgres.yml:76-83` (`depends_on: postulo: condition: service_healthy`); the wiki's `Installing-Postulo.md:8` and `:193` (`POSTULO_ALLOWED_HOSTS=postulo.example.org`) and `Troubleshooting.md:199-201`; `scripts/check-image.sh:33`, which always runs with `localhost,127.0.0.1`. - **What happens:** 1. An operator replaces `localhost,127.0.0.1` in `.env` with their domain, as the README and wiki instruct. 2. The web container answers through the proxy, but every 30 seconds the probe gets `400 Bad Request` and a `DisallowedHost` error with its traceback is logged; Docker marks the container unhealthy. 3. `docker compose --profile scheduler up -d` and `--profile worker up -d` wait on `service_healthy` and never start, so reminders, gone-quiet notices, store copies, syncs and queued background work never run. 4. *Troubleshooting* says an unhealthy container means `/healthz` "answered 503 or did not answer at all", which sends the operator to the database. With the test client and `ALLOWED_HOSTS=["postulo.example.org"]`, `GET /healthz` with `Host: 127.0.0.1:8000` returns 400; with `127.0.0.1` added it returns 200. - **Why it matters:** a healthy instance reports itself broken, the background services silently never run, and the probe can no longer tell a real failure from this one. - **Fix:** make the probe independent of the operator's host list. Either add a small middleware placed first that answers `GET /healthz` from a loopback `REMOTE_ADDR` before host validation, or have the production settings always admit `localhost` and send `-H 'Host: localhost'` from the `HEALTHCHECK`. Say in *Health, metrics and logs* which host the probe uses, and correct the *Troubleshooting* sentence. Related to #82 (closed), which fixed the HTTPS-redirect half of the same probe but not the host check. ## Done when - A test in `tests/test_image_build.py` (or a new `tests/core/test_healthz.py`) requests `/healthz` exactly as the `HEALTHCHECK` does, under `override_settings(ALLOWED_HOSTS=["postulo.example.org"])`, and expects 200; it gets 400 today. - `scripts/check-image.sh` gains a run with `POSTULO_ALLOWED_HOSTS=postulo.example.org` that waits for Docker's own health status to become `healthy`.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:47:48 +00:00

tiagoagueda added the

labels

2026-10-01 10:47:48 +00:00

tiagoagueda referenced this issue from a commit

2026-10-01 22:40:45 +00:00

Answer the container's own health check whatever the host list holds