The image never takes Debian's updates: the layer that upgrades is served from the builder's cache #300

New issue

Closed

opened 2026-09-29 06:32:49 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-09-29 06:32:49 +00:00

Owner

The dev image for 420b9d102 failed its scan gate (run 585): Grype found OpenSSL
3.0.20-1~deb12u2 in the image with 3.0.22-1~deb12u1 available, three CVEs at High and
one at Critical. Nothing in that push touched the image. The fix had been in Debian's
security archive for days, and the image did not have it because the layer that takes
Debian's updates never ran.

What the logs say

docker/Dockerfile installs the runtime's packages in one layer,
apt-get update && apt-get upgrade -y && apt-get install …, and #155 put the upgrade
there on purpose: the base image keeps what it was built with, so the upgrade is the only
thing that reaches a package like libssl3, which arrives with Python.

A layer is served from the builder's cache for as long as its text and the layer under it
are the same. Neither changes when Debian publishes an update. On the runner:

Run	Builder	The apt layer
582, `e143fc80d`, 25 September	`default`, the one that is scanned	`CACHED`
582	`postulo-dev`, the one that is pushed, amd64	`CACHED`
582	`postulo-dev`, arm64	`CACHED`
585, `420b9d102`, 28 September	`default`	`CACHED`

Run 582's full report already listed libssl3 3.0.20-1~deb12u2, fixed in
3.0.22-1~deb12u1, at Medium, which is under the gate. Between the two runs the findings
were rescored, and the gate did its job.

So the sentence in the Dockerfile, "between an image that drifts towards being patched and
one that reliably stays unpatched, this takes the first", has not been true on any builder
that keeps its cache. The image stays at the day its apt layer was last built, and the
published dev images, both architectures, have been carrying packages with a fix
available.

Two builders are involved, each with a cache of its own: the daemon's, whose image is
scanned, and the docker-container builder buildx pushes from. A fix that refreshes one
of them leaves a scanned image and a published image that differ.

What to do

A build argument, declared in the runtime stage above the apt layer, whose value is part
of that layer's cache key. A new value makes the layer run again, on whichever builder.
Both workflows work out one moment per run and hand it to both of their builds, so the
image that is scanned and the image that is published asked Debian the same question.
--pull beside it, so the base image is asked for as well whatever the builder's
default is.
scripts/scan-image.sh does the same when it builds the image itself.
Without the argument the build behaves as it does today, and the layer says so in the
build log.

Not --no-cache: it would rebuild the stylesheet and the Python environment on every push,
under emulation for arm64, to refresh one layer. Not a date as the value: an update
published after the day's first build would fail the gate until midnight, which is a gate
failing for a reason nobody can act on.

Done when

The apt layer runs on every build a workflow makes, in both builders, and the log says
when Debian was asked.
tests/test_image_build.py holds the argument above the layer and holds every build in
the workflows and the scan script to passing it.
Hardening on the wiki says what a cached build means for somebody building their own
image.
The dev image for the commit that lands this is green.

The dev image for `420b9d102` failed its scan gate (run 585): Grype found OpenSSL `3.0.20-1~deb12u2` in the image with `3.0.22-1~deb12u1` available, three CVEs at High and one at Critical. Nothing in that push touched the image. The fix had been in Debian's security archive for days, and the image did not have it because **the layer that takes Debian's updates never ran**. ## What the logs say `docker/Dockerfile` installs the runtime's packages in one layer, `apt-get update && apt-get upgrade -y && apt-get install …`, and #155 put the upgrade there on purpose: the base image keeps what it was built with, so the upgrade is the only thing that reaches a package like `libssl3`, which arrives with Python. A layer is served from the builder's cache for as long as its text and the layer under it are the same. Neither changes when Debian publishes an update. On the runner: | Run | Builder | The apt layer | | --- | --- | --- | | 582, `e143fc80d`, 25 September | `default`, the one that is scanned | `CACHED` | | 582 | `postulo-dev`, the one that is pushed, amd64 | `CACHED` | | 582 | `postulo-dev`, arm64 | `CACHED` | | 585, `420b9d102`, 28 September | `default` | `CACHED` | Run 582's full report already listed `libssl3 3.0.20-1~deb12u2`, fixed in `3.0.22-1~deb12u1`, at Medium, which is under the gate. Between the two runs the findings were rescored, and the gate did its job. So the sentence in the Dockerfile, "between an image that drifts towards being patched and one that reliably stays unpatched, this takes the first", has not been true on any builder that keeps its cache. The image stays at the day its apt layer was last built, and the published dev images, both architectures, have been carrying packages with a fix available. Two builders are involved, each with a cache of its own: the daemon's, whose image is scanned, and the `docker-container` builder buildx pushes from. A fix that refreshes one of them leaves a scanned image and a published image that differ. ## What to do - A build argument, declared in the runtime stage above the apt layer, whose value is part of that layer's cache key. A new value makes the layer run again, on whichever builder. - Both workflows work out one moment per run and hand it to both of their builds, so the image that is scanned and the image that is published asked Debian the same question. `--pull` beside it, so the base image is asked for as well whatever the builder's default is. - `scripts/scan-image.sh` does the same when it builds the image itself. - Without the argument the build behaves as it does today, and the layer says so in the build log. Not `--no-cache`: it would rebuild the stylesheet and the Python environment on every push, under emulation for arm64, to refresh one layer. Not a date as the value: an update published after the day's first build would fail the gate until midnight, which is a gate failing for a reason nobody can act on. ## Done when - The apt layer runs on every build a workflow makes, in both builders, and the log says when Debian was asked. - `tests/test_image_build.py` holds the argument above the layer and holds every build in the workflows and the scan script to passing it. - *Hardening* on the wiki says what a cached build means for somebody building their own image. - The dev image for the commit that lands this is green.

tiagoagueda added this to the 0.5.0 milestone

2026-09-29 06:32:49 +00:00

tiagoagueda added the

security

bug

labels

2026-09-29 06:32:49 +00:00

tiagoagueda referenced this issue from a commit

2026-09-29 06:39:48 +00:00

Make the layer that takes Debian's updates run on every build

tiagoagueda closed this issue

2026-09-29 06:39:48 +00:00

tiagoagueda commented

2026-09-29 06:58:27 +00:00

Author

Owner

Landed as 05781f881, and run 587 is the first one built with it. What its log shows:

Builder	Architecture	The apt layer	OpenSSL
`default`, the image that is scanned	amd64	ran, 36 s	`3.0.20-1~deb12u2` to `3.0.22-1~deb12u1`
`postulo-dev`, the image that is pushed	amd64	ran, 62 s	the same
`postulo-dev`	arm64, emulated	ran, 186 s	the same

All three printed Debian's packages as of: 2026-09-29T06:39:57Z, the one moment the run
read from the clock. The gate's verdict was "Nothing fixable at HIGH,CRITICAL or above",
and the image was published as 0.3.0-dev.05781f8. The whole job took eleven minutes,
against twelve for run 582, so running the layer costs less than the variation between
runs.

The seven other jobs on that commit are green as well.

Two things the logs showed that the issue did not know when it was filed:

The workflows' comments said the multi-architecture build reused the layers of the one
that was scanned. It never did: the two builders share nothing. The comments are
corrected, and the two builds are tied together by the moment they are given.
The 0.3.0 release image (run 443) was pushed by a builder that had never built before,
so its packages were the ones of its day. Its scan read the daemon's cached layer, which
was not the image that was pushed. A release built from now on is scanned and pushed from
the same moment.

Landed as `05781f881`, and run 587 is the first one built with it. What its log shows: | Builder | Architecture | The apt layer | OpenSSL | | --- | --- | --- | --- | | `default`, the image that is scanned | amd64 | ran, 36 s | `3.0.20-1~deb12u2` to `3.0.22-1~deb12u1` | | `postulo-dev`, the image that is pushed | amd64 | ran, 62 s | the same | | `postulo-dev` | arm64, emulated | ran, 186 s | the same | All three printed `Debian's packages as of: 2026-09-29T06:39:57Z`, the one moment the run read from the clock. The gate's verdict was "Nothing fixable at HIGH,CRITICAL or above", and the image was published as `0.3.0-dev.05781f8`. The whole job took eleven minutes, against twelve for run 582, so running the layer costs less than the variation between runs. The seven other jobs on that commit are green as well. Two things the logs showed that the issue did not know when it was filed: - The workflows' comments said the multi-architecture build reused the layers of the one that was scanned. It never did: the two builders share nothing. The comments are corrected, and the two builds are tied together by the moment they are given. - The 0.3.0 release image (run 443) was pushed by a builder that had never built before, so its packages were the ones of its day. Its scan read the daemon's cached layer, which was not the image that was pushed. A release built from now on is scanned and pushed from the same moment.

tiagoagueda referenced this issue

2026-10-31 03:04:18 +00:00

Dev image: Debian's apt layer runs again on every push to main, not once a day #1139