I believe I integrated mender with our Yocto-based toradex verdin-imx8mplus SoM (there was a weird scenario with his SoM, see Trying to integration mender with Toradex verdin-imx8mp - #2 by whsplk).
When I try updating via hosted mendor, mender would rollback the update after booting into the new partition. Checking the logs, it mentions "transient error: The deployment was aborted from the server","timestamp":"2022-03-02T01:03:01
before rolling back.
At first I thought it might be our custom yocto recipes that might be affecting this issue, but the rollback still persists after stripping our application layer.
Not sure how else to debug this. If anyone from mender or experience with mender can comment, that woud be great!
Thanks
Full logs:
2022-03-02 01:13:25 +0000 UTC info: Running Mender client version: 3.2.1
2022-03-02 01:13:25 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2022-03-02 01:13:25 +0000 UTC info: No public key was provided for authenticating the artifact
2022-03-02 01:13:25 +0000 UTC info: Opening device "/dev/mmcblk2p3" for writing
2022-03-02 01:13:25 +0000 UTC info: Native sector size of block device /dev/mmcblk2p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
2022-03-02 01:14:19 +0000 UTC info: All bytes were successfully written to the new partition
2022-03-02 01:14:19 +0000 UTC info: The optimized block-device writer wrote a total of 1953 frames, where 15 frames did need to be rewritten (i.e., skipped)
2022-03-02 01:14:20 +0000 UTC info: Wrote 2046820352/2046820352 bytes to the inactive partition
2022-03-02 01:14:20 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2022-03-02 01:14:20 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2022-03-02 01:14:20 +0000 UTC info: Validating the Update Info: https://s3.amazonaws.com/hosted-mender-artifacts/621bf49f19dc3d571d2dd3cb/bafab3e5-3ea1-4a1a-a1f4-9aed81c3adc1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAQWI25QR6NDALMYE2%2F20220302%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220302T011420Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=a3c9880ae0b5206d1b1a72f8004a0a6f72400820b2012e4a0abfd0a2ce3443f8 [name: minimal-3; devices: [verdin-imx8mp]]
2022-03-02 01:14:20 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2022-03-02 01:14:20 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2022-03-02 01:14:20 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2022-03-02 01:14:20 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2022-03-02 01:14:20 +0000 UTC error: Error receiving scheduled update data: failed to check update info on the server. Response: &{429 Too Many Requests 429 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[0] Content-Type:[application/json; charset=utf-8] Date:[Wed, 02 Mar 2022 01:14:20 GMT] X-Men-Requestid:[4007ee82-418a-48e0-8b17-4de9e3d5453e]] 0x40002504c0 0 [] false false map[] 0x40001ded00 <nil>}
2022-03-02 01:14:20 +0000 UTC error: Update control map check failed: transient error: failed to check update info on the server. Response: &{429 Too Many Requests 429 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[0] Content-Type:[application/json; charset=utf-8] Date:[Wed, 02 Mar 2022 01:14:20 GMT] X-Men-Requestid:[4007ee82-418a-48e0-8b17-4de9e3d5453e]] 0x40002504c0 0 [] false false map[] 0x40001ded00 <nil>}, retrying...
2022-03-02 01:14:20 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control-retry-refresh-maps [none]
2022-03-02 01:14:20 +0000 UTC info: Wait 1m0s before next update control map fetch/update attempt
2022-03-02 01:15:20 +0000 UTC info: State transition: mender-update-control-retry-refresh-maps [none] -> mender-update-control-refresh-maps [none]
2022-03-02 01:15:20 +0000 UTC info: Validating the Update Info: https://s3.amazonaws.com/hosted-mender-artifacts/621bf49f19dc3d571d2dd3cb/bafab3e5-3ea1-4a1a-a1f4-9aed81c3adc1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAQWI25QR6NDALMYE2%2F20220302%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220302T011520Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=6eaed3b5ebc56fcde56ac1f22012bcf56a867854fceb9c1c597391494cafe44b [name: minimal-3; devices: [verdin-imx8mp]]
2022-03-02 01:15:21 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2022-03-02 01:15:21 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2022-03-02 01:15:21 +0000 UTC info: Rebooting device(s)
2022-03-02 01:15:21 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk2p2
2022-03-02 01:15:21 +0000 UTC info: Daemon terminated with SIGTERM
2022-03-02 01:17:35 +0000 UTC info: Running Mender client version: 3.2.1
2022-03-02 01:17:35 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2022-03-02 01:17:35 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2022-03-02 01:17:35 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> mender-update-control-refresh-maps [none]
2022-03-02 01:17:35 +0000 UTC info: Device unauthorized; attempting reauthorization
2022-03-02 01:17:35 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2022-03-02 01:17:36 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2022-03-02 01:17:36 +0000 UTC info: Local proxy started
2022-03-02 01:17:36 +0000 UTC info: Reauthorization successful
2022-03-02 01:17:36 +0000 UTC error: transient error: The deployment was aborted from the server
2022-03-02 01:17:36 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> rollback [ArtifactRollback]
2022-03-02 01:17:36 +0000 UTC info: Performing rollback
2022-03-02 01:17:36 +0000 UTC info: Rolling back to the inactive partition (2).
2022-03-02 01:17:36 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2022-03-02 01:17:36 +0000 UTC info: Rebooting device(s) after rollback
2022-03-02 01:17:36 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk2p3
2022-03-02 01:17:36 +0000 UTC info: Daemon terminated with SIGTERM
2022-03-02 01:19:50 +0000 UTC info: Running Mender client version: 3.2.1
2022-03-02 01:19:50 +0000 UTC info: Mender shut down in state: rollback-reboot
2022-03-02 01:19:50 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2022-03-02 01:19:50 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2022-03-02 01:19:50 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2022-03-02 01:19:50 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2022-03-02 01:19:50 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2022-03-02 01:19:50 +0000 UTC info: Device unauthorized; attempting reauthorization
2022-03-02 01:19:50 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2022-03-02 01:19:51 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2022-03-02 01:19:51 +0000 UTC info: Local proxy started
2022-03-02 01:19:51 +0000 UTC info: Reauthorization successful