Ship pstore contents off the board on boot #21

New issue

Closed

opened 2026-08-27 23:22:27 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-08-27 23:22:27 +00:00

Owner

ramoops is proven to survive a reset, and systemd-pstore archives records into
/var/lib/systemd/pstore/. But they stay on the board, and the ring is small — a second crash
overwrites the evidence from the first.

Note the gotcha: /sys/fs/pstore looks empty after boot precisely because systemd has moved
the records; look in /var/lib/systemd/pstore/.

Done when

crash records are copied off the board automatically on boot
retention is long enough to compare two failures

`ramoops` is proven to survive a reset, and `systemd-pstore` archives records into `/var/lib/systemd/pstore/`. But they stay on the board, and the ring is small — a second crash overwrites the evidence from the first. Note the gotcha: `/sys/fs/pstore` looks *empty* after boot precisely because systemd has moved the records; look in `/var/lib/systemd/pstore/`. **Done when** - [x] crash records are copied off the board automatically on boot - [x] retention is long enough to compare two failures

tiagoagueda added the

reliability

P3-normal

labels

2026-08-27 23:22:27 +00:00

tiagoagueda referenced this issue

2026-08-29 15:40:47 +00:00

sun9i-a80 has no GPU device tree node #57

tiagoagueda referenced this issue from a commit

2026-08-29 15:56:09 +00:00

Ship crash records off the board on boot (#21)

tiagoagueda commented

2026-08-29 15:56:35 +00:00

Author

Owner

Done — records leave the board on boot, over a key that can do one thing

draco-pstore-ship runs at boot, after systemd-pstore has moved records out of /sys/fs/pstore, and copies the archive to the build host. Verified on a real boot rather than by hand:

Starting draco-pstore-ship.service - Ship ramoops crash records off the board...
shipping 5 record(s) to <build host>
stored 5 file(s), 288K, in a80/pstore/a80-debian/20260829-154809
Finished draco-pstore-ship.service.

The archive now holds three timestamped deliveries, so "compare two failures" is comparing two directories.

The transport decision

The board is the less trusted device here: it runs experimental kernels, it is the thing that crashes, and it is the thing most likely to be in a bad state. Giving it a general-purpose SSH key to the build host to solve a logging problem would be the wrong trade.

So the authorized_keys entry carries restrict plus a forced command — no pty, no forwarding, no agent, and nothing but the receiver. Verified by attacking it rather than assuming:

$ ssh -i /etc/ssh/draco-pstore-key <host> "id; cat /etc/shadow"
  -> the receiver ran instead; no id, no shadow

Records are never deleted locally — having them in two places is the point — and a checksum of names, sizes and mtimes means a board that reboots often does not fill the archive with identical copies.

Three bugs the testing found, all mine

The receiver silently failed every shipment. It passed --no-absolute-names, which GNU tar does not have, and sent tar's stderr to /dev/null. So the flag error was invisible and every delivery reported "tar failed" with no way to see why. Same shape as the || true mistake earlier today: hiding a diagnostic to keep output tidy costs more than it saves. The flag is gone — tar's default already strips leading slashes and refuses .. — and stderr is no longer discarded.

A path traversal, in the one place that explicitly handles untrusted input. The label taken from SSH_ORIGINAL_COMMAND went through a character blocklist that allowed dots, so a client sending ".." would have produced $DEST/../$STAMP and written outside the archive entirely. Filtering a blocklist and hoping is the wrong shape; it now matches a whitelist pattern and rejects everything else. Attempting ".." lands in pstore/unknown/.

The build host's login banner was being logged line by line as though it were output from the shipment. ssh -q.

A note for whoever reads this next

The gotcha recorded in the issue is worth repeating because it is genuinely easy to get wrong: /sys/fs/pstore looks empty after boot precisely because systemd has already moved the records. Looking there and concluding nothing was captured is the mistake. /var/lib/systemd/pstore/ is where they are, and now also a80/pstore/<host>/<timestamp>/ on the build host.

Committed in d49cb6b; the receiver lives at tools/pstore-receive with its one-time install instructions in the header.

Done when

crash records are copied off the board automatically on boot
retention is long enough to compare two failures

## Done — records leave the board on boot, over a key that can do one thing `draco-pstore-ship` runs at boot, after `systemd-pstore` has moved records out of `/sys/fs/pstore`, and copies the archive to the build host. Verified on a real boot rather than by hand: ``` Starting draco-pstore-ship.service - Ship ramoops crash records off the board... shipping 5 record(s) to <build host> stored 5 file(s), 288K, in a80/pstore/a80-debian/20260829-154809 Finished draco-pstore-ship.service. ``` The archive now holds three timestamped deliveries, so "compare two failures" is comparing two directories. ### The transport decision The board is the **less trusted** device here: it runs experimental kernels, it is the thing that crashes, and it is the thing most likely to be in a bad state. Giving it a general-purpose SSH key to the build host to solve a *logging* problem would be the wrong trade. So the `authorized_keys` entry carries `restrict` plus a forced command — no pty, no forwarding, no agent, and nothing but the receiver. Verified by attacking it rather than assuming: ``` $ ssh -i /etc/ssh/draco-pstore-key <host> "id; cat /etc/shadow" -> the receiver ran instead; no id, no shadow ``` Records are never deleted locally — having them in two places is the point — and a checksum of names, sizes and mtimes means a board that reboots often does not fill the archive with identical copies. ### Three bugs the testing found, all mine **The receiver silently failed every shipment.** It passed `--no-absolute-names`, which GNU tar does not have, *and* sent tar's stderr to `/dev/null`. So the flag error was invisible and every delivery reported "tar failed" with no way to see why. Same shape as the `|| true` mistake earlier today: hiding a diagnostic to keep output tidy costs more than it saves. The flag is gone — tar's default already strips leading slashes and refuses `..` — and stderr is no longer discarded. **A path traversal, in the one place that explicitly handles untrusted input.** The label taken from `SSH_ORIGINAL_COMMAND` went through a character *blocklist* that allowed dots, so a client sending `".."` would have produced `$DEST/../$STAMP` and written outside the archive entirely. Filtering a blocklist and hoping is the wrong shape; it now matches a whitelist pattern and rejects everything else. Attempting `".."` lands in `pstore/unknown/`. **The build host's login banner** was being logged line by line as though it were output from the shipment. `ssh -q`. ### A note for whoever reads this next The gotcha recorded in the issue is worth repeating because it is genuinely easy to get wrong: **`/sys/fs/pstore` looks empty after boot precisely because systemd has already moved the records.** Looking there and concluding nothing was captured is the mistake. `/var/lib/systemd/pstore/` is where they are, and now also `a80/pstore/<host>/<timestamp>/` on the build host. Committed in `d49cb6b`; the receiver lives at `tools/pstore-receive` with its one-time install instructions in the header. **Done when** - [x] crash records are copied off the board automatically on boot - [x] retention is long enough to compare two failures

tiagoagueda closed this issue

2026-08-29 15:56:36 +00:00