Image: .dockerignore patterns match only at the root, so local builds send and ship the downloaded datasets and every pycache #1141

New issue

Closed

opened 2026-10-31 03:04:18 +00:00 by tiagoagueda · 0 comments

tiagoagueda commented

2026-10-31 03:04:18 +00:00

Owner

What exists today

.dockerignore:6 (data), :12 (__pycache__) and :13 (*.py[cod]) are matched relative to the context root, as Docker's .dockerignore rules say (Go filepath.Match per path, ** needed for any depth). So they exclude ./data and a root __pycache__ only.

Measured in the working tree: src/postulo/jobs/data is 71 MB, of which only the licence notes and nace-2.1.json (177 KB) are tracked; the rest is esco-1.2.1.json (6.4 MB), esco-skills-1.2.1.zip (4.7 MB), geonames-cities1000.txt (32 MB), geonames-countryinfo.txt and geonames-cities1000.sqlite (30.5 MB), written by fetch_esco and fetch_geonames. The __pycache__ directories under src total about 12.5 MB (du).

Correction to the candidate: the cost is not only the context sent. docker/Dockerfile:92 (COPY src ./src) and :293 (COPY --from=build /app/src ./src) carry those files into the runtime image, where nothing reads them (the image reads GeoNames and ESCO from /app/data/geonames and /app/data/esco, :160-161). docker compose -f docker/compose.yml -f docker/compose.build.yml up --build and scripts/check-image.sh both build from the working tree, so a source install made on a machine that has run the app ships 71 MB of dead data. The styles stage's COPY src at :33 copies them as well. CI checks out a clean tree, so the published images are not affected.

Side finding: geonames-cities1000.sqlite (src/postulo/jobs/places.py:55, LOOKUP_FILE) is not in .gitignore; git status shows it untracked (??). .gitignore:48-55 lists the ESCO files and geonames-*.txt but not geonames-*.sqlite (nor geonames-postcodes.sqlite, places.py:62), so a careless git add -A would commit 30 MB.

What would change

.dockerignore: **/__pycache__, **/*.py[cod], and the dataset files by path: src/postulo/jobs/data/esco-* and src/postulo/jobs/data/geonames-* (the tracked nace-2.1.json and licence notes stay). **/node_modules and **/.pytest_cache while there.
.gitignore: add /src/postulo/jobs/data/geonames-*.sqlite.
A test in tests/test_image_build.py that every dataset pattern under src/ in .gitignore has a matching .dockerignore line would keep the two together.

What it saves or protects

small-medium: on a developer machine with the datasets fetched, 71 MB of GeoNames and ESCO files plus about 12.5 MB of __pycache__ under src go into the build context and, through COPY src, into the local image; CI images are unaffected

Accessibility

none

Decisions

One issue for both ignore files? Recommended: yes, one small pull request; the .gitignore line is the same class of mistake.

Found by the efficiency and reliability audit of 2026-10-30 (finding I5; index postulo/postulo#1108). Checked against main at daee30b15 (2026-10-30).

## What exists today `.dockerignore:6` (`data`), `:12` (`__pycache__`) and `:13` (`*.py[cod]`) are matched relative to the context root, as Docker's `.dockerignore` rules say (Go `filepath.Match` per path, `**` needed for any depth). So they exclude `./data` and a root `__pycache__` only. Measured in the working tree: `src/postulo/jobs/data` is 71 MB, of which only the licence notes and `nace-2.1.json` (177 KB) are tracked; the rest is `esco-1.2.1.json` (6.4 MB), `esco-skills-1.2.1.zip` (4.7 MB), `geonames-cities1000.txt` (32 MB), `geonames-countryinfo.txt` and `geonames-cities1000.sqlite` (30.5 MB), written by `fetch_esco` and `fetch_geonames`. The `__pycache__` directories under `src` total about 12.5 MB (`du`). Correction to the candidate: the cost is not only the context sent. `docker/Dockerfile:92` (`COPY src ./src`) and `:293` (`COPY --from=build /app/src ./src`) carry those files into the runtime image, where nothing reads them (the image reads GeoNames and ESCO from `/app/data/geonames` and `/app/data/esco`, `:160-161`). `docker compose -f docker/compose.yml -f docker/compose.build.yml up --build` and `scripts/check-image.sh` both build from the working tree, so a source install made on a machine that has run the app ships 71 MB of dead data. The `styles` stage's `COPY src` at `:33` copies them as well. CI checks out a clean tree, so the published images are not affected. Side finding: `geonames-cities1000.sqlite` (`src/postulo/jobs/places.py:55`, `LOOKUP_FILE`) is not in `.gitignore`; `git status` shows it untracked (`??`). `.gitignore:48-55` lists the ESCO files and `geonames-*.txt` but not `geonames-*.sqlite` (nor `geonames-postcodes.sqlite`, `places.py:62`), so a careless `git add -A` would commit 30 MB. ## What would change - `.dockerignore`: `**/__pycache__`, `**/*.py[cod]`, and the dataset files by path: `src/postulo/jobs/data/esco-*` and `src/postulo/jobs/data/geonames-*` (the tracked `nace-2.1.json` and licence notes stay). `**/node_modules` and `**/.pytest_cache` while there. - `.gitignore`: add `/src/postulo/jobs/data/geonames-*.sqlite`. - A test in `tests/test_image_build.py` that every dataset pattern under `src/` in `.gitignore` has a matching `.dockerignore` line would keep the two together. ## What it saves or protects small-medium: on a developer machine with the datasets fetched, 71 MB of GeoNames and ESCO files plus about 12.5 MB of `__pycache__` under `src` go into the build context and, through `COPY src`, into the local image; CI images are unaffected ## Accessibility none ## Decisions - **One issue for both ignore files?** Recommended: yes, one small pull request; the `.gitignore` line is the same class of mistake. --- Found by the efficiency and reliability audit of 2026-10-30 (finding I5; index postulo/postulo#1108). Checked against `main` at `daee30b15` (2026-10-30).

tiagoagueda added this to the 0.6.0 milestone

2026-10-31 03:04:18 +00:00

tiagoagueda added the

performance

bug

labels

2026-10-31 03:04:18 +00:00

tiagoagueda referenced this issue

2026-10-31 03:04:41 +00:00

Efficiency and reliability audit of 2026-10-30: index #1108

tiagoagueda referenced this issue from a commit

2026-11-10 11:31:09 +00:00