Mender OTA Update successful but reported by Enterprise server as a failure

General Discussions
1

Hello,
My issue is very similar to an issue raised last year (Mender deployment error Expected \"upgrade_available\" flag to be true but it was false ).

I’m using a Mender Enterprise server to deploy an over the air update.
The update is identified as being successful, because the server shows the artifact name has been updated and the ‘mount’ command shows the rootfs partition change. However the mender server incorrectly reports the deployment as a failure,
Below is the contents of the Deployment log:-

2021-07-28 04:00:56 +0000 UTC info: Running Mender client version: 2.6.0
2021-07-28 04:00:56 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2021-07-28 04:00:56 +0000 UTC info: No public key was provided for authenticating the artifact
2021-07-28 04:00:56 +0000 UTC info: Opening device “/dev/sda2” for writing
2021-07-28 04:00:56 +0000 UTC info: Native sector size of block device /dev/sda2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2021-07-28 04:08:11 +0000 UTC info: All bytes were successfully written to the new partition
2021-07-28 04:08:11 +0000 UTC info: The optimized block-device writer wrote a total of 1585 frames, where 0 frames did need to be rewritten (i.e., skipped)
2021-07-28 04:08:12 +0000 UTC info: Wrote 1660944384/1660944384 bytes to the inactive partition
2021-07-28 04:08:12 +0000 UTC info: State transition: update-store [Download_Enter] → update-after-store [Download_Leave]
2021-07-28 04:08:12 +0000 UTC info: State transition: update-after-store [Download_Leave] → update-install [ArtifactInstall]
2021-07-28 04:08:12 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2021-07-28 04:08:13 +0000 UTC info: State transition: update-install [ArtifactInstall] → reboot [ArtifactReboot_Enter]
2021-07-28 04:08:13 +0000 UTC info: Rebooting device(s)
2021-07-28 04:08:13 +0000 UTC info: Mender rebooting from active partition: /dev/sda3
2021-07-28 04:08:13 +0000 UTC info: Daemon terminated with SIGTERM
2021-07-28 04:10:46 +0000 UTC info: Running Mender client version: 2.6.0
2021-07-28 04:10:46 +0000 UTC info: State transition: init [none] → after-reboot [ArtifactReboot_Leave]
2021-07-28 04:10:46 +0000 UTC error: transient error: Reboot to the new update failed. Expected “upgrade_available” flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back
2021-07-28 04:10:46 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] → rollback [ArtifactRollback]
2021-07-28 04:10:46 +0000 UTC info: Performing rollback
2021-07-28 04:10:46 +0000 UTC info: No update available, so no rollback needed.
2021-07-28 04:10:46 +0000 UTC info: State transition: rollback [ArtifactRollback] → rollback-reboot [ArtifactRollbackReboot_Enter]
2021-07-28 04:10:46 +0000 UTC info: Rebooting device(s) after rollback
2021-07-28 04:10:46 +0000 UTC info: Mender rebooting from inactive partition: /dev/sda2
2021-07-28 04:10:47 +0000 UTC info: Daemon terminated with SIGTERM
2021-07-28 04:10:47 +0000 UTC info: State transition: rollback-reboot [ArtifactRollbackReboot_Enter] → verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2021-07-28 04:13:20 +0000 UTC info: Running Mender client version: 2.6.0
2021-07-28 04:13:20 +0000 UTC info: Mender shut down in state: rollback-reboot
2021-07-28 04:13:20 +0000 UTC info: State transition: init [none] → verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2021-07-28 04:13:20 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] → after-rollback-reboot [ArtifactRollbackReboot_Leave]
2021-07-28 04:13:20 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] → update-error [ArtifactFailure]
2021-07-28 04:13:20 +0000 UTC info: State transition: update-error [ArtifactFailure] → cleanup [Error]
2021-07-28 04:13:20 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

2

Unfortunately it seems text files are not authorized. Therefore please advise if you require other files how best to receive them.

3

Hi @GrahamTooms,

As the other post stated, I believe it is important to take a look at the logs. And journalctl -u mender-client is a good place to start with.

If the output is not too big, I would recommend copying the content here. However, if it is too big, for the sake of sharing the file, maybe you can add the extension yml, so the system will allow you to upload it.

Regards,
Luis

4

Hi Luis,

Thanks for your response. Attached are the following files:

  • journalctl_20210728_1400.txt.yml : Output from ‘journalctl -f -u mender-client’, piped to output file, up to point of reset.
  • deployment_20210728_1400.txt.yml : The resulting log file available from the server deployments (I think same as pasted in my original post).
  • messages_20210728_1400.txt.yml: journald log file after the reset (forced by the OTA update).
    Note the messages files is not stored in persistent storage, hence the information in this log file is post reset only.
    deployment_20210728_1400.txt.yml (3.4 KB)
    journalctl_20210728_1400.txt.yml (4.6 KB)
    messages_20210728_1400.txt.yml (18.3 KB)
5

I have the same problem.
I believe the problem is because Mender is configured to use ttyS0, and som other tty connections. What these are used for, I haven’t been able to find out, but in my case after I removed ttyS0 from from:
MENDER_GRUB_KERNEL_BOOT_ARGS="console=tty0,115200n8 console=ttyO0,115200n8 console=ttyAMA0,115200n8"
Then the update fails with the responds
2021-07-28 04:10:46 +0000 UTC error: transient error: Reboot to the new update failed. Expected “upgrade_available” flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back

6

Got it to work.

Seems to be an problem with creating artifacts using the snapshot method, since creating artifacts over ssh works fine.