Jetson A/B OTA: reboot loop after power interruption before update commit

General Discussions
, , , ,
1

Hello! We’re investigating cases in which a device can become stuck in a reboot loop after power is interrupted during or shortly after an OTA update finishes, before Mender has commited the update and acknowledged it as successful. The affected devices are deployed remotely, so we are not present when power is interrupted. In these cases, the user may unplug the device while an update is finishing or shortly after it reboots.

Setup

  • NVIDIA Jetson/Tegra platform with bootloader and rootfs A/B redundancy.
  • Mender client 4.0.7, using a rootfs-image Update Module and a custom Tegra/Mender integration.
  • UEFI boot with L4TLauncher; firmware versions observed include 36.4.3 and 36.4.4.
  • Persistent Mender state under /data/mender.

Our observation

Serial output on affected device showed a firmware-level reset and the device never reached Linux:

Switching the bootchain. Resetting the system in 2 seconds.

In UEFI, we saw:

OS chain A status: Unbootable
OS chain B status: Normal
L4T Boot Mode: ExtLinux

Manually changing OS chain A to Normal allowed the device to boot. After recovery, nvbootctrl showed rootfs A and bootloader A active, while bootloader B was Unbootable and the capsule update status was 2.
We did not capture the bootloader and rootfs status tables simultaneously before recovery, so we cannot establish the exact sequence that led to those states. A subsequent OTA succeeded, and the device later booted from B with both bootloader slots reporting Normal.

In a separate power-interruption test, the device rebooted once and then reached Linux. We later observed a pending upgrade with both rootfs slots Normal:

Current rootfs slot: A
Active rootfs slot: A
slot: 0, retry_count: 0, status: normal
slot: 1, retry_count: 1, status: normal

upgrade_available=1
/var/lib/mender/boot_attempted exists

The previous boot’s journal was not persistent, so we could not determine which component requested the extra reboot. This test did not reproduce the persistent firmware-level loop.

Integration behavior we’re investigating

We found that the nv_update_verifier.sh at our pinned commit uses a persistent boot_attempted marker.

If upgrade_available is not 0 and the marker matches the current bootloader slot, the script removes the marker and forces a reboot. If upgrade_available=0, it calls nvbootctrl verify.

We’re wondering how this is intended to behave when a device loses power after booting the update but before Mender commits it. We have not established that the verifier’s forced-reboot branch ran during the persistent loop. In the captured loop, the resets occurred in firmware before Linux started.

We also found that our fw_setenv implementation stores upgrade_available as a file-backed marker. We’re still investigating how the rootfs-image rollback path coordinates that marker with NVIDIA’s slot selection and verification.

Questions

  1. Is this a known failure mode for power loss between the OTA reboot and Mender commit on a Jetson A/B integration?
  2. How should the integration coordinate Mender’s upgrade_available state, Jetson boot retries, and nvbootctrl verify so that an interrupted trial boot does not exhaust the last known-good fallback?
  3. Is the verifier’s forced reboot on a repeated boot into the same slot expected after power loss? What is supposed to select and acknowledge the known-good slot if the update cannot be committed?
  4. What slot/status transitions and logs would you recommend capturing to distinguish an expected rollback from a boot-control integration issue?

We can try to reproduce power interruptions on a test device and collect serial logs and slot states. We’d appreciate guidance on the expected recovery behavior and any existing tests or fixes for this case.

We’d appreciate any comment and would like to know if someone has experienced this before too. Thanks in advance!!