Board sometimes hangs in SPL and needs a physical power cycle #3

New issue

Open

opened 2026-08-27 23:22:23 +00:00 by tiagoagueda · 1 comment

tiagoagueda commented

2026-08-27 23:22:23 +00:00

Owner

On 2026-08-27 the board failed to come back from an ordinary systemctl reboot. The
serial console shows SPL starting, initialising DRAM, then emitting two garbage characters
where the next message should be:

DRAM: 3584 MiB
Trying to boot from MMC2      <- all 68 other boots that day
DRAM: 3584 MiB
SS                            <- this one, then silence

SS is not a prefix of anything SPL prints, so the SoC glitched rather than failing down a
code path. It stayed dead — no network, ARP INCOMPLETE on both interfaces — until it was
physically power-cycled, after which it booted cleanly and has not repeated.

This board has prior history of non-deterministic boot failures (see the regulator comments
in the board DTS).

Why this is the issue that decides "production". A host that occasionally does not return
from a reboot is not a production host, and with no remote power control a single occurrence
ends the service until someone walks over.

Done when

the failure rate is measured, not guessed — loop power-cycles against the serial capture
and count (needs remote power control first)
a cause is identified, or the rate is shown low enough to accept and is written down

On 2026-08-27 the board failed to come back from an ordinary `systemctl reboot`. The serial console shows SPL starting, initialising DRAM, then emitting two garbage characters where the next message should be: ``` DRAM: 3584 MiB Trying to boot from MMC2 <- all 68 other boots that day DRAM: 3584 MiB SS <- this one, then silence ``` `SS` is not a prefix of anything SPL prints, so the SoC glitched rather than failing down a code path. It stayed dead — no network, ARP `INCOMPLETE` on both interfaces — until it was physically power-cycled, after which it booted cleanly and has not repeated. This board has prior history of non-deterministic boot failures (see the regulator comments in the board DTS). **Why this is the issue that decides "production".** A host that occasionally does not return from a reboot is not a production host, and with no remote power control a single occurrence ends the service until someone walks over. **Done when** - [ ] the failure rate is measured, not guessed — loop power-cycles against the serial capture and count (needs remote power control first) - [ ] a cause is identified, or the rate is shown low enough to accept and is written down

tiagoagueda added the

reliability

P1-critical

labels

2026-08-27 23:22:23 +00:00

tiagoagueda commented

2026-08-28 20:10:12 +00:00

Author

Owner

Partly mitigated 2026-08-28, not closed. #37 arms a watchdog in SPL, which is the
stage this issue is about and which nothing else covered - the hardware watchdog under Linux,
the extlinux fallback and the rescue card all assume the SoC reached U-Boot proper.

Three register writes on R_WDT as the first thing sunxi_board_init() does, before DRAM init,
which is the point the observed hang had already passed. Handoff verified: U-Boot proper
adopts the same instance through the WDT uclass, and Linux stops it in sunxi_wdt_probe().

It stays open because the mitigation is unproven. There is no way to induce an SPL hang on
demand - the observed event was one boot in 68 - so the disassembly and repeated clean boots
are all the evidence there is. See 50-uboot-resilience.md.

**Partly mitigated 2026-08-28, not closed.** #37 arms a watchdog in SPL, which is the stage this issue is about and which nothing else covered - the hardware watchdog under Linux, the extlinux fallback and the rescue card all assume the SoC reached U-Boot proper. Three register writes on R_WDT as the first thing `sunxi_board_init()` does, before DRAM init, which is the point the observed hang had already passed. Handoff verified: U-Boot proper adopts the same instance through the WDT uclass, and Linux stops it in `sunxi_wdt_probe()`. **It stays open because the mitigation is unproven.** There is no way to induce an SPL hang on demand - the observed event was one boot in 68 - so the disassembly and repeated clean boots are all the evidence there is. See `50-uboot-resilience.md`.

tiagoagueda referenced this issue from a commit

2026-08-29 14:22:15 +00:00

Firewall and unattended security updates (#15)

tiagoagueda referenced this issue

2026-08-29 14:22:15 +00:00

No unattended security updates and no firewall ruleset #15

tiagoagueda referenced this issue

2026-08-29 15:09:04 +00:00

AR100 voltage service dies if the coprocessor is started too early #47

tiagoagueda referenced this issue

2026-08-30 14:26:49 +00:00

Investigate FEL as a recovery path - the wiki says A80 supports it #73