Hi there,
I’m currently using a hosted mender instance (https://eu.hosted.mender.io) to provision a Raspberry Pi 4, which currently has a semi “silver” image on it, and we’re looking to send over the “golden” image through an artifact deployment. It gets pretty far through the process until it reboots, where it’ll boot into the new install partition with full network access, and functions just fine, however the Mender server will not recognise this and stay on “Rebooting”.
If after a while we manually reboot the device, it’ll reboot just fine and display the tty login screen before immediately rebooting again and dropping us back to the original rootfs it started with, displaying a “Fail” on the Mender server dashboard.
This is the deployment log the Mender dashboard gives us after it “fails”:
2023-09-01 15:35:42 +0000 UTC info: Running Mender client version: 3.5.1
2023-09-01 15:35:42 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2023-09-01 15:35:42 +0000 UTC info: No public key was provided for authenticating the artifact
2023-09-01 15:35:42 +0000 UTC info: Opening device "/dev/mmcblk0p3" for writing
2023-09-01 15:35:42 +0000 UTC info: Native sector size of block device /dev/mmcblk0p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
2023-09-01 15:38:28 +0000 UTC info: All bytes were successfully written to the new partition
2023-09-01 15:38:28 +0000 UTC info: The optimized block-device writer wrote a total of 2036 frames, where 2022 frames did need to be rewritten (i.e., skipped)
2023-09-01 15:38:29 +0000 UTC info: Wrote 2134751232/2134751232 bytes to the inactive partition
2023-09-01 15:38:29 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2023-09-01 15:38:29 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2023-09-01 15:38:29 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-09-01 15:38:29 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2023-09-01 15:38:29 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 3
2023-09-01 15:38:29 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2023-09-01 15:38:29 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-09-01 15:38:29 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2023-09-01 15:38:29 +0000 UTC info: Rebooting device(s)
2023-09-01 15:38:29 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p2
2023-09-01 15:38:29 +0000 UTC error: error forwarding from client to backend: websocket: close 1006 (abnormal closure): unexpected EOF
2023-09-01 15:38:29 +0000 UTC warning: error while sending close message: write tcp 127.0.0.1:38085->127.0.0.1:45180: use of closed network connection
2023-09-01 15:38:29 +0000 UTC info: Daemon terminated with SIGTERM
2023-09-01 15:39:11 +0000 UTC info: Running Mender client version: 3.5.1
2023-09-01 15:39:11 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2023-09-01 15:39:11 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2023-09-01 15:39:11 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> mender-update-control-refresh-maps [none]
2023-09-01 15:39:11 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-09-01 15:39:11 +0000 UTC info: State transition: mender-update-control [none] -> update-commit [ArtifactCommit_Enter]
2023-09-01 15:39:11 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-09-01 15:39:11 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-09-01 15:39:12 +0000 UTC error: Failed to authorize with "https://hosted.mender.io": authentication request rejected server error message: Unauthorized
2023-09-01 15:39:12 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-09-01 15:39:12 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-09-01 15:39:12 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-09-01 15:39:12 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-09-01 15:39:12 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-09-01 15:39:12 +0000 UTC warning: UpdatePollIntervalSeconds is not defined
2023-09-01 15:39:12 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:39:12 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:41:15 +0000 UTC info: Forced wake-up from sleep
2023-09-01 15:41:15 +0000 UTC error: Cannot check update or update inventory while in update-commit state
2023-09-01 15:41:15 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2023-09-01 15:41:15 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-09-01 15:41:15 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-09-01 15:41:16 +0000 UTC error: Failed to authorize with "https://hosted.mender.io": authentication request rejected server error message: Unauthorized
2023-09-01 15:41:16 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-09-01 15:41:16 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-09-01 15:41:16 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-09-01 15:41:16 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-09-01 15:41:16 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-09-01 15:41:16 +0000 UTC warning: UpdatePollIntervalSeconds is not defined
2023-09-01 15:41:16 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:41:16 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:45:33 +0000 UTC info: Forced wake-up from sleep
2023-09-01 15:45:33 +0000 UTC error: Cannot check update or update inventory while in update-commit state
2023-09-01 15:45:33 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2023-09-01 15:45:33 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-09-01 15:45:33 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-09-01 15:45:33 +0000 UTC error: Failed to authorize with "https://hosted.mender.io": authentication request rejected server error message: Unauthorized
2023-09-01 15:45:33 +0000 UTC warning: Reauthorization failed with error: transient error: authorization request failed
2023-09-01 15:45:33 +0000 UTC error: Failed to report status: transient error: authorization request failed
2023-09-01 15:45:33 +0000 UTC error: error reporting update status: reporting status failed: transient error: authorization request failed
2023-09-01 15:45:33 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: transient error: authorization request failed
2023-09-01 15:45:33 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2023-09-01 15:45:33 +0000 UTC warning: UpdatePollIntervalSeconds is not defined
2023-09-01 15:45:33 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:45:33 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2023-09-01 15:47:24 +0000 UTC info: Daemon terminated with SIGTERM
2023-09-01 15:48:02 +0000 UTC info: Running Mender client version: 3.5.1
2023-09-01 15:48:02 +0000 UTC error: Mender shut down in state: update-commit
2023-09-01 15:48:02 +0000 UTC info: State transition: init [none] -> rollback [ArtifactRollback]
2023-09-01 15:48:02 +0000 UTC info: Performing rollback
2023-09-01 15:48:02 +0000 UTC info: No update available, so no rollback needed.
2023-09-01 15:48:02 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2023-09-01 15:48:02 +0000 UTC info: Rebooting device(s) after rollback
2023-09-01 15:48:02 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p2
2023-09-01 15:48:03 +0000 UTC info: Daemon terminated with SIGTERM
2023-09-01 15:48:43 +0000 UTC info: Running Mender client version: 3.5.1
2023-09-01 15:48:43 +0000 UTC info: Mender shut down in state: rollback-reboot
2023-09-01 15:48:43 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-09-01 15:48:43 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-09-01 15:48:43 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2023-09-01 15:48:43 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2023-09-01 15:48:43 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2023-09-01 15:48:43 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-09-01 15:48:43 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/eth0
2023-09-01 15:48:44 +0000 UTC info: successfully received new authorization data from server https://eu.hosted.mender.io
2023-09-01 15:48:44 +0000 UTC info: Local proxy started
2023-09-01 15:48:44 +0000 UTC info: Reauthorization successful
Is anyone able to provide some guidance? Both images are barely touched, only having internet configs and the mender clients installed. The silver image is based on Raspian 22/01/28, and the artifact we’re attempting to deploy through Mender is based on Raspian 22/04/07.