Mender dashboard says update failed because of upgrade_available

General Discussions
1

Hi,

I was able to integrate Mender onto a Digi CC6 (I will post the integration guide in a few days). However, after performing an artifact update, I can confirm that Mender switched to the B partition and that the change in the update did actually occur.

However, the backend complains that the update failed because the upgrade_available is expected to set to true, but the u-boot environment variable isn’t present. This is the error log from the backend:

2025-02-11 22:17:48 +0000 UTC info: Running Mender client version: 3.5.3
2025-02-11 22:17:48 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2025-02-11 22:17:48 +0000 UTC info: No public key was provided for authenticating the artifact
2025-02-11 22:17:49 +0000 UTC info: Opening device "/dev/mmcblk0p2" for writing
2025-02-11 22:17:49 +0000 UTC info: Native sector size of block device /dev/mmcblk0p2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2025-02-11 22:19:47 +0000 UTC info: All bytes were successfully written to the new partition
2025-02-11 22:19:47 +0000 UTC info: The optimized block-device writer wrote a total of 969 frames, where 50 frames did need to be rewritten (i.e., skipped)
2025-02-11 22:19:47 +0000 UTC info: Wrote 1015021568/1015021568 bytes to the inactive partition
2025-02-11 22:19:47 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2025-02-11 22:19:47 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2025-02-11 22:19:47 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2025-02-11 22:19:47 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2025-02-11 22:19:47 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2025-02-11 22:19:47 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2025-02-11 22:19:47 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2025-02-11 22:19:47 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2025-02-11 22:19:47 +0000 UTC info: Rebooting device(s)
2025-02-11 22:19:47 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p3
2025-02-11 22:19:48 +0000 UTC info: Daemon terminated with SIGTERM
2025-02-11 22:20:59 +0000 UTC info: Running Mender client version: 3.5.3
2025-02-11 22:20:59 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2025-02-11 22:20:59 +0000 UTC error: transient error: Reboot to the new update failed. Expected "upgrade_available" flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back
2025-02-11 22:20:59 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2025-02-11 22:20:59 +0000 UTC info: Performing rollback
2025-02-11 22:20:59 +0000 UTC info: No update available, so no rollback needed.
2025-02-11 22:20:59 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2025-02-11 22:20:59 +0000 UTC info: Rebooting device(s) after rollback
2025-02-11 22:20:59 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p2
2025-02-11 22:20:59 +0000 UTC info: Daemon terminated with SIGTERM
2025-02-11 22:21:53 +0000 UTC info: Running Mender client version: 3.5.3
2025-02-11 22:21:53 +0000 UTC info: Mender shut down in state: rollback-reboot
2025-02-11 22:21:53 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2025-02-11 22:21:53 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2025-02-11 22:21:53 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2025-02-11 22:21:53 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2025-02-11 22:21:53 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2025-02-11 22:21:53 +0000 UTC info: Device unauthorized; attempting reauthorization
2025-02-11 22:21:54 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2025-02-11 22:21:54 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2025-02-11 22:21:54 +0000 UTC info: Local proxy started
2025-02-11 22:21:54 +0000 UTC info: Reauthorization successful

What’s interesting is that it looks like Mender sets the upgrade_available u-boot variable to 1 before restarting (I confirmed by viewing the environment variable in u-boot). But then the second time it reboots, the environment variable goes away and I see the above error in the backend.

2

Hi @mabembedded,

Although it looks like the backend, it’s actually the client that is detecting this and “complaining” about it. The backend just shows the client logs verbatim, you can check that with journalctl -u mender-client on the device.

In all situations where I have seen similar things happen, there actually were two reboots instead of one. Are you by chance rolling out an update which does some first boot style initializations, like setting a hostname, and then immediately reboots before the Mender Client gets started again? Looking at the timestamps, this is very possible, there is a one minute gap at 22:19.

So to diagnose it, my suggestion is to watch the serial terminal when the update is being completed into reboot. If there are two, then you need to find out why and fix it.

Greetz,
Josef