Docker image: fetch_esco writes into the root-owned source tree, so the documented install step fails #411

New issue

Closed

opened 2026-10-01 10:46:30 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-01 10:46:30 +00:00

Owner

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the image build against the install guide's ESCO step.

Checked again on 2026-10-01 against main at 0a5cad587: still present. The code is now at jobs/models.py:974 (JobPosting.save); jobs/esco.py, jobs/places.py, fetch_esco.py, docker/Dockerfile and docker/entrypoint.sh have not changed.

The ESCO classification is read from, and written to, a directory beside the code, and nothing lets an operator move it. In the image that directory belongs to root while the container runs as postulo, so the command the install guide tells every Docker operator to run cannot write its files. GeoNames met the same problem and was given POSTULO_GEOLOCATIONS_DIR on the data volume; ESCO was not.

Where: jobs/esco.py:84 (DATA_DIR = Path(__file__).resolve().parent / "data", no environment override, unlike jobs/places.py:37-39); jobs/management/commands/fetch_esco.py:162-172 (Command.handle writes esco-<revision>.json.tmp and esco-skills-<revision>.zip into DATA_DIR); docker/Dockerfile:266-267 (COPY --from=build /app/src ./src, no --chown, so root-owned); docker/Dockerfile:322-329 (only /app/data is given to postulo, then USER postulo); docker/entrypoint.sh:22-30 (fetches GeoNames only); wiki Installing-Postulo.md:10, :94, :321-324.
What happens: an operator follows "The short version" and runs docker compose -f docker/compose.yml exec postulo python manage.py fetch_esco --revision 1.2.1. The command runs as postulo, harvests the whole classification (the guide says about half a minute), then fails with a PermissionError writing into /app/src/postulo/jobs/data. From then on no listing title gets an ISCO-08 code (JobPosting.save, jobs/models.py:971), the title box offers no unit groups and the skill box offers nothing. An operator who works around it with exec -u root writes into the container layer: the files are gone at the next docker compose pull && up -d, and the worker and scheduler containers of docker/compose.yml, which run the same image separately, never have them, so a listing saved by the worker (a capture) gets no code.
Why it matters: the recommended deployment cannot turn on #266's classification by following its own guide, and the image breaks the rule its Dockerfile states (docker/Dockerfile:271-272, "the running container should not need to write to its own image") and places.py:36 repeats.
Fix: read POSTULO_ESCO_DIR in esco.DATA_DIR (or one variable for both reference datasets), defaulting to the package directory for a source install; set it to /app/data/esco in the Dockerfile's ENV block beside POSTULO_GEOLOCATIONS_DIR, and create the directory with the other /app/data ones. Optionally fetch it in the entrypoint when absent, with a failure logged and not fatal, as for GeoNames. Then correct Installing-Postulo.md: the files live on the data volume and survive a pull, so the "run the line again" advice at :97-98 and :321-324 goes.

Done when

tests/test_esco.py has a test that sets POSTULO_ESCO_DIR, reloads postulo.jobs.esco and finds DATA_DIR there; it fails today.
tests/test_image_build.py has a test that the Dockerfile sets POSTULO_ESCO_DIR under /app/data; it fails today.

Found on 2026-09-29 in a read-only review of postulo at 95348908d, reading the image build against the install guide's ESCO step. Checked again on 2026-10-01 against `main` at `0a5cad587`: still present. The code is now at `jobs/models.py:974` (`JobPosting.save`); `jobs/esco.py`, `jobs/places.py`, `fetch_esco.py`, `docker/Dockerfile` and `docker/entrypoint.sh` have not changed. The ESCO classification is read from, and written to, a directory beside the code, and nothing lets an operator move it. In the image that directory belongs to root while the container runs as `postulo`, so the command the install guide tells every Docker operator to run cannot write its files. GeoNames met the same problem and was given `POSTULO_GEOLOCATIONS_DIR` on the data volume; ESCO was not. - **Where:** `jobs/esco.py:84` (`DATA_DIR = Path(__file__).resolve().parent / "data"`, no environment override, unlike `jobs/places.py:37-39`); `jobs/management/commands/fetch_esco.py:162-172` (`Command.handle` writes `esco-<revision>.json.tmp` and `esco-skills-<revision>.zip` into `DATA_DIR`); `docker/Dockerfile:266-267` (`COPY --from=build /app/src ./src`, no `--chown`, so root-owned); `docker/Dockerfile:322-329` (only `/app/data` is given to `postulo`, then `USER postulo`); `docker/entrypoint.sh:22-30` (fetches GeoNames only); wiki `Installing-Postulo.md:10`, `:94`, `:321-324`. - **What happens:** an operator follows "The short version" and runs `docker compose -f docker/compose.yml exec postulo python manage.py fetch_esco --revision 1.2.1`. The command runs as `postulo`, harvests the whole classification (the guide says about half a minute), then fails with a `PermissionError` writing into `/app/src/postulo/jobs/data`. From then on no listing title gets an ISCO-08 code (`JobPosting.save`, `jobs/models.py:971`), the title box offers no unit groups and the skill box offers nothing. An operator who works around it with `exec -u root` writes into the container layer: the files are gone at the next `docker compose pull && up -d`, and the `worker` and `scheduler` containers of `docker/compose.yml`, which run the same image separately, never have them, so a listing saved by the worker (a capture) gets no code. - **Why it matters:** the recommended deployment cannot turn on #266's classification by following its own guide, and the image breaks the rule its Dockerfile states (`docker/Dockerfile:271-272`, "the running container should not need to write to its own image") and `places.py:36` repeats. - **Fix:** read `POSTULO_ESCO_DIR` in `esco.DATA_DIR` (or one variable for both reference datasets), defaulting to the package directory for a source install; set it to `/app/data/esco` in the Dockerfile's `ENV` block beside `POSTULO_GEOLOCATIONS_DIR`, and create the directory with the other `/app/data` ones. Optionally fetch it in the entrypoint when absent, with a failure logged and not fatal, as for GeoNames. Then correct `Installing-Postulo.md`: the files live on the data volume and survive a pull, so the "run the line again" advice at `:97-98` and `:321-324` goes. ## Done when - `tests/test_esco.py` has a test that sets `POSTULO_ESCO_DIR`, reloads `postulo.jobs.esco` and finds `DATA_DIR` there; it fails today. - `tests/test_image_build.py` has a test that the Dockerfile sets `POSTULO_ESCO_DIR` under `/app/data`; it fails today.

tiagoagueda added this to the 0.5.0 milestone

2026-10-01 10:46:30 +00:00

tiagoagueda added the

documentation

bug

tier

labels

2026-10-01 10:46:30 +00:00

tiagoagueda referenced this issue

2026-10-03 01:59:13 +00:00

Docker Compose at home: one deployment page for a homelab, and the drift between the compose files and the wiki #704

tiagoagueda referenced this issue from a commit