Userspace configuration is not captured as code #8

Closed
opened 2026-08-27 23:22:24 +00:00 by tiagoagueda · 1 comment
Owner

The running system is the product of a long interactive session and cannot currently be
rebuilt without repeating it from memory. Hand-made state includes:

  • units: ir-protocols.service, bt-public-addr.service (+ /usr/local/sbin/bt-public-addr.sh)
  • firmware placed by hand: /lib/firmware/brcm/BCM4335C0.hcd (note the rename — see
    firmware/README.md)
  • network: 10-any-eth.network with ActivationPolicy=always-up, 20-wlan0.network,
    ifupdown neutralised
  • systemd drop-ins: watchdog (10-watchdog.conf), journald size cap
  • packages: bluez, ir-keytable, v4l-utils, gpiod, avahi, timesyncd, u-boot-tools, dtc, tmux

Done when

  • a provision.sh (or Ansible role) in this repo reproduces all of the above on a fresh
    Debian rootfs
  • it is verified by running it against the rescue SD rather than trusting it
The running system is the product of a long interactive session and cannot currently be rebuilt without repeating it from memory. Hand-made state includes: - units: `ir-protocols.service`, `bt-public-addr.service` (+ `/usr/local/sbin/bt-public-addr.sh`) - firmware placed by hand: `/lib/firmware/brcm/BCM4335C0.hcd` (note the rename — see `firmware/README.md`) - network: `10-any-eth.network` with `ActivationPolicy=always-up`, `20-wlan0.network`, ifupdown neutralised - systemd drop-ins: watchdog (`10-watchdog.conf`), journald size cap - packages: bluez, ir-keytable, v4l-utils, gpiod, avahi, timesyncd, u-boot-tools, dtc, tmux **Done when** - [x] a `provision.sh` (or Ansible role) in this repo reproduces all of the above on a fresh Debian rootfs - [x] it is verified by running it against the rescue SD rather than trusting it
Author
Owner

Done — tools/provision/, and testing it from scratch found three real defects

tools/provision/ holds a files/ tree mirroring the filesystem, packages.txt, and an idempotent provision.sh. A second run printing nothing but = is the point: that is how drift shows up.

On the two done-when boxes

Box 1 is met literally. It was run against a fresh debootstrap --variant=minbase trixie armhf tree, twice, and the result compared against the live board: 0 files differ, 0 packages installed that the board does not have, draco identical, all four firmware blobs identical.

Box 2 I did not do as written, and the substitution is better. It asked for verification against the rescue SD. That card is now a full clone of the eMMC, so running there would report = on every line and prove nothing at all. A bare rootfs is the harder test, and it earned its keep immediately.

What testing from scratch caught that the on-board run could not

The on-board run reported 16/16 green. It was still wrong three times over, because passing against a system that is already correct only proves the script recognises that system — not that it can build one.

  1. e2fsprogs, cron, nftables, systemd-timesyncd were missing. The root filter excluded everything Debian installs at priority important/standard, but --variant=minbase installs neither. Without e2fsprogs the entire fsck policy from #14 is inert on a rebuilt system.

  2. Recommends were on while the board was built without them, pulling in 23 packages it does not have — including a whole evolution-data-server stack behind bluez-obexd.

  3. Turning recommends off then dropped two packages that fail silently:

    • libubootenv-tool provides fw_printenv/fw_setenv, which boot-mark-good calls. Without it the U-Boot bootcount is never cleared, so BOOTCOUNT_LIMIT falls back to uImage.known-good after three perfectly good boots.
    • wireless-regdb provides regulatory.db, which the initramfs hook copies. Without it cfg80211 never leaves the world domain — the exact bug fixed on the rescue card earlier today, reintroduced by rebuilding from this script.

    Neither produces an error. The board would boot, look healthy, and be quietly wrong.

Three decisions worth recording

packages.txt is 48 names, not 237. apt-mark showmanual reports 237 including bash and coreutils. Subtracting Debian's priority set leaves 163, still mostly dependency noise. Keeping only the roots of the dependency graph leaves 28; the rest are the minbase gaps and the needed recommends, both marked in the file. That filter also dropped gcc-arm-linux-gnueabihf and its toolchain, installed at some point on a board that is itself ARM — provisioning will not put them back.

The NVRAM is not installed verbatim. firmware/nvram_ap6335.txt is the vendor original with ccode=0. The script applies CCODE at install time, so a local edit is visible rather than hidden inside a file labelled "vendor".

rdreg is compiled, not committed — it was a 490 KB static ARM binary; the source is in usb3-re/rdreg.c.

The gap this exposed, left open on purpose

This board has no working module tree. /lib/modules holds directories for two older kernels and none for the running one, and deploy-kernel.sh never runs make modules_install, so modprobe cannot work at all. Nothing has noticed because everything needed is built in — which is also why CONFIG_ZRAM went in as =y rather than =m.

provision.sh prints this at the end of every run rather than burying it. It deserves its own issue: no provisioning script can honestly claim to rebuild this system until module installation is part of the deploy.

## Done — `tools/provision/`, and testing it from scratch found three real defects `tools/provision/` holds a `files/` tree mirroring the filesystem, `packages.txt`, and an idempotent `provision.sh`. A second run printing nothing but `=` is the point: that is how drift shows up. ### On the two done-when boxes **Box 1 is met literally.** It was run against a fresh `debootstrap --variant=minbase trixie armhf` tree, twice, and the result compared against the live board: **0 files differ**, **0 packages installed that the board does not have**, `draco` identical, all four firmware blobs identical. **Box 2 I did not do as written, and the substitution is better.** It asked for verification against the rescue SD. That card is now a *full clone* of the eMMC, so running there would report `=` on every line and prove nothing at all. A bare rootfs is the harder test, and it earned its keep immediately. ### What testing from scratch caught that the on-board run could not The on-board run reported **16/16 green**. It was still wrong three times over, because passing against a system that is already correct only proves the script *recognises* that system — not that it can *build* one. 1. **`e2fsprogs`, `cron`, `nftables`, `systemd-timesyncd` were missing.** The root filter excluded everything Debian installs at priority *important*/*standard*, but `--variant=minbase` installs neither. Without `e2fsprogs` the entire fsck policy from #14 is inert on a rebuilt system. 2. **Recommends were on** while the board was built without them, pulling in 23 packages it does not have — including a whole `evolution-data-server` stack behind `bluez-obexd`. 3. Turning recommends off then dropped two packages that **fail silently**: - `libubootenv-tool` provides `fw_printenv`/`fw_setenv`, which `boot-mark-good` calls. Without it the U-Boot bootcount is never cleared, so `BOOTCOUNT_LIMIT` falls back to `uImage.known-good` after three perfectly good boots. - `wireless-regdb` provides `regulatory.db`, which the initramfs hook copies. Without it cfg80211 never leaves the world domain — the exact bug fixed on the rescue card earlier today, reintroduced by rebuilding from this script. Neither produces an error. The board would boot, look healthy, and be quietly wrong. ### Three decisions worth recording **`packages.txt` is 48 names, not 237.** `apt-mark showmanual` reports 237 including `bash` and `coreutils`. Subtracting Debian's priority set leaves 163, still mostly dependency noise. Keeping only the **roots of the dependency graph** leaves 28; the rest are the minbase gaps and the needed recommends, both marked in the file. That filter also dropped `gcc-arm-linux-gnueabihf` and its toolchain, installed at some point on a board that is itself ARM — provisioning will not put them back. **The NVRAM is not installed verbatim.** `firmware/nvram_ap6335.txt` is the vendor original with `ccode=0`. The script applies `CCODE` at install time, so a local edit is visible rather than hidden inside a file labelled "vendor". **`rdreg` is compiled, not committed** — it was a 490 KB static ARM binary; the source is in `usb3-re/rdreg.c`. ### The gap this exposed, left open on purpose **This board has no working module tree.** `/lib/modules` holds directories for two older kernels and none for the running one, and `deploy-kernel.sh` never runs `make modules_install`, so `modprobe` cannot work at all. Nothing has noticed because everything needed is built in — which is also why `CONFIG_ZRAM` went in as `=y` rather than `=m`. `provision.sh` prints this at the end of every run rather than burying it. It deserves its own issue: no provisioning script can honestly claim to rebuild this system until module installation is part of the deploy.
Sign in to join this conversation.
No description provided.