Mender update and reboot OK with rpi3b+ but black screen/rollback with rpi3

I have a bunch of devices using mender with either rpi3b+ or rpi3 hardware.
All my inventory is using mender client 2.4.1 + raspbian Stretch

I just made a new release using mender client 3.1 with an upgrade to raspbian Buster (i also tried raspbian Bullseye).

With rpi3b+ : the upgrade went fine
BUT with all rpi3 : i get a black screen just after reboot and after an hour, there is a rollback

In mender logs, i get the following logs

2021-11-10 22:27:43 +0000 UTC info: Running Mender client version: 2.4.1
2021-11-10 22:27:43 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2021-11-10 22:27:43 +0000 UTC info: No public key was provided for authenticating the artifact
2021-11-10 22:27:44 +0000 UTC info: Opening device "/dev/mmcblk0p2" for writing
2021-11-10 22:27:44 +0000 UTC info: Native sector size of block device /dev/mmcblk0p2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2021-11-10 22:45:09 +0000 UTC info: All bytes were successfully written to the new partition
2021-11-10 22:45:09 +0000 UTC info: The optimized block-device writer wrote a total of 5110 frames, where 3808 frames did need to be rewritten (i.e., skipped)
2021-11-10 22:45:10 +0000 UTC info: Wrote 5357291520/5357291520 bytes to the inactive partition
2021-11-10 22:45:32 +0000 UTC error: Got unexpected HTTP status when reporting status: 500
2021-11-10 22:45:33 +0000 UTC error: error reporting update status: (request_id: ): reporting status failed, bad status 500 server error message: failed to parse server response: EOF
2021-11-10 22:45:33 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2021-11-10 22:45:33 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2021-11-10 22:45:33 +0000 UTC error: Got unexpected HTTP status when reporting status: 503
2021-11-10 22:45:33 +0000 UTC error: statescript: Can not send start status to server: (request_id: ): reporting status failed, bad status 503 server error message: failed to parse server response: invalid character 'S' looking for beginning of value
2021-11-10 22:45:33 +0000 UTC error: Got unexpected HTTP status when reporting status: 503
2021-11-10 22:45:33 +0000 UTC error: statescript: Can not send finished status to server: (request_id: ): reporting status failed, bad status 503 server error message: failed to parse server response: invalid character 'S' looking for beginning of value
2021-11-10 22:45:33 +0000 UTC error: Got unexpected HTTP status when reporting status: 503
2021-11-10 22:45:33 +0000 UTC error: error reporting update status: (request_id: ): reporting status failed, bad status 503 server error message: failed to parse server response: invalid character 'S' looking for beginning of value
2021-11-10 22:45:33 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2021-11-10 22:45:33 +0000 UTC info: Probing the Bootloader environment for which separator to use
2021-11-10 22:45:33 +0000 UTC info: Writing map[mender_uboot_separator:1] to the U-Boot environment, using separator:  
2021-11-10 22:45:33 +0000 UTC info: Writing map[mender_uboot_separator:] to the U-Boot environment, using separator:  
2021-11-10 22:45:33 +0000 UTC info: Writing map[upgrade_available:1 bootcount:0 mender_boot_part:2 mender_boot_part_hex:2] to the U-Boot environment, using separator:  
2021-11-10 22:45:33 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2021-11-10 22:45:33 +0000 UTC error: Got unexpected HTTP status when reporting status: 503
2021-11-10 22:45:33 +0000 UTC error: error reporting update status: (request_id: ): reporting status failed, bad status 503 server error message: failed to parse server response: invalid character 'S' looking for beginning of value
2021-11-10 22:45:33 +0000 UTC info: Rebooting device(s)
2021-11-10 22:45:33 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p3
2021-10-29 16:54:12 +0000 UTC info: Running Mender client version: 3.1.0
2021-10-29 16:54:12 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2021-10-29 16:54:12 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> after-reboot [ArtifactReboot_Leave]
2021-10-29 16:54:12 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> mender-update-control [none]
2021-10-29 16:54:12 +0000 UTC info: State transition: mender-update-control [none] -> update-commit [ArtifactCommit_Enter]
2021-10-29 16:54:12 +0000 UTC error: Failed to report status: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 16:54:12 +0000 UTC error: error reporting update status: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 16:54:12 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 16:54:12 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2021-10-29 17:24:12 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2021-10-29 17:24:12 +0000 UTC error: Failed to report status: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:24:12 +0000 UTC error: error reporting update status: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:24:12 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:24:12 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2021-10-29 17:54:12 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2021-10-29 17:54:12 +0000 UTC error: Failed to report status: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:54:12 +0000 UTC error: error reporting update status: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:54:12 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 17:54:12 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2021-10-29 18:24:12 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> update-commit [ArtifactCommit_Enter]
2021-10-29 18:24:12 +0000 UTC error: Failed to report status: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 18:24:12 +0000 UTC error: error reporting update status: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 18:24:12 +0000 UTC error: Failed to send status report to server: transient error: reporting status failed: Put "https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status": dial tcp: lookup mender.mapado.net on 1.1.1.1:53: dial udp 1.1.1.1:53: connect: network is unreachable
2021-10-29 18:24:12 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] -> update-pre-commit-status-report-retry [ArtifactCommit_Enter]
2021-10-29 18:24:12 +0000 UTC error: transient error: Tried sending status report maximum number of times.
2021-10-29 18:24:12 +0000 UTC info: State transition: update-pre-commit-status-report-retry [ArtifactCommit_Enter] -> rollback [ArtifactRollback]
2021-10-29 18:24:12 +0000 UTC info: Performing rollback
2021-10-29 18:24:12 +0000 UTC info: Rolling back to the inactive partition (3).
2021-10-29 18:24:13 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2021-10-29 18:24:13 +0000 UTC info: Rebooting device(s) after rollback
2021-10-29 18:24:13 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p2
2021-10-29 18:24:13 +0000 UTC info: Daemon terminated with SIGTERM
2020-12-14 13:53:58 +0000 UTC info: Running Mender client version: 2.4.1
2020-12-14 13:53:58 +0000 UTC info: Mender shut down in state: rollback-reboot
2020-12-14 13:53:58 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2020-12-14 13:53:58 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2020-12-14 13:53:58 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2020-12-14 13:53:58 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2020-12-14 13:53:58 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2020-12-14 13:53:59 +0000 UTC error: Failed to report status: Put https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status: not a valid certificate, openssl verify rc: 9 server cert file: /etc/mender/server.crt
2020-12-14 13:53:59 +0000 UTC error: error reporting update status: reporting status failed: Put https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status: not a valid certificate, openssl verify rc: 9 server cert file: /etc/mender/server.crt
2020-12-14 13:53:59 +0000 UTC error: Failed to send status to server: transient error: reporting status failed: Put https://mender.mapado.net/api/devices/v1/deployments/device/deployments/36340e01-a562-4744-8783-d676e48545be/status: not a valid certificate, openssl verify rc: 9 server cert file: /etc/mender/server.crt
2020-12-14 13:53:59 +0000 UTC info: State transition: update-status-report [none] -> update-retry-report [none]
2021-11-11 00:47:05 +0000 UTC info: State transition: update-retry-report [none] -> update-status-report [none]

Do anybody have something of an idea to help me fix this ?

looks like from your error log that you have no networking at the point it ran, which also possibly means your internal clock has not been synced and was set to the past causing the not a valid certificate yet openssl error…

thanks, i’m going to follow this insight