Unable to switch A/B partition between Ubuntu 18 and Ubuntu 20 for OTA

We tired to deploy a OTA update with Ubuntu 20 to a mender client that is currently running Ubuntu 18, the deployment it self seems to be ok as the client writes the artifact into the inactive partition, however once it reboots, it won’t be able to boot into the image with following error

transient error: Reboot to the new update failed. Expected “upgrade_available” flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back

Any suggestion of what to do?

Full log

2023-06-09 13:53:50 +0000 UTC info: Running Mender client version: 3.5.0
2023-06-09 13:53:50 +0000 UTC error: Mender shut down in state: update-fetch
2023-06-09 13:53:50 +0000 UTC info: State transition: init [none] -> idle [Idle]
2023-06-09 13:57:23 +0000 UTC info: Running Mender client version: 3.5.0
2023-06-09 13:57:24 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2023-06-09 13:57:24 +0000 UTC info: No public key was provided for authenticating the artifact
2023-06-09 13:57:24 +0000 UTC info: Opening device "/dev/sda2" for writing
2023-06-09 13:57:24 +0000 UTC info: Native sector size of block device /dev/sda2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2023-06-09 14:06:30 +0000 UTC info: All bytes were successfully written to the new partition
2023-06-09 14:06:30 +0000 UTC info: The optimized block-device writer wrote a total of 24305 frames, where 9543 frames did need to be rewritten (i.e., skipped)
2023-06-09 14:06:32 +0000 UTC info: Wrote 25484591104/25484591104 bytes to the inactive partition
2023-06-09 14:06:32 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2023-06-09 14:06:32 +0000 UTC info: State transition: update-after-store [Download_Leave] -> mender-update-control-refresh-maps [none]
2023-06-09 14:06:32 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-06-09 14:06:32 +0000 UTC info: State transition: mender-update-control [none] -> update-install [ArtifactInstall]
2023-06-09 14:06:32 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2023-06-09 14:06:33 +0000 UTC info: State transition: update-install [ArtifactInstall] -> mender-update-control-refresh-maps [none]
2023-06-09 14:06:33 +0000 UTC info: State transition: mender-update-control-refresh-maps [none] -> mender-update-control [none]
2023-06-09 14:06:33 +0000 UTC info: State transition: mender-update-control [none] -> reboot [ArtifactReboot_Enter]
2023-06-09 14:06:33 +0000 UTC info: Rebooting device(s)
2023-06-09 14:06:33 +0000 UTC info: Mender rebooting from active partition: /dev/sda3
2023-06-09 14:06:33 +0000 UTC info: Daemon terminated with SIGTERM
2023-06-09 14:07:33 +0000 UTC info: Running Mender client version: 3.5.0
2023-06-09 14:07:33 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2023-06-09 14:07:33 +0000 UTC error: transient error: Reboot to the new update failed. Expected "upgrade_available" flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back
2023-06-09 14:07:33 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2023-06-09 14:07:33 +0000 UTC info: Executing script: ArtifactRollback_Enter_01_Inform_User
2023-06-09 14:07:33 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-06-09 14:07:34 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/enp1s0
2023-06-09 14:07:34 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2023-06-09 14:07:34 +0000 UTC info: Local proxy started
2023-06-09 14:07:34 +0000 UTC info: Reauthorization successful
2023-06-09 14:07:37 +0000 UTC info: Performing rollback
2023-06-09 14:07:37 +0000 UTC info: No update available, so no rollback needed.
2023-06-09 14:07:37 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2023-06-09 14:07:37 +0000 UTC info: Rebooting device(s) after rollback
2023-06-09 14:07:37 +0000 UTC info: Mender rebooting from inactive partition: /dev/sda3
2023-06-09 14:07:39 +0000 UTC info: Daemon terminated with SIGTERM
2023-06-09 14:08:10 +0000 UTC info: Running Mender client version: 3.5.0
2023-06-09 14:08:10 +0000 UTC info: Mender shut down in state: rollback-reboot
2023-06-09 14:08:10 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-06-09 14:08:10 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2023-06-09 14:08:10 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2023-06-09 14:08:11 +0000 UTC info: Executing script: ArtifactFailure_Enter_01_Inform_User
2023-06-09 14:08:11 +0000 UTC info: Device unauthorized; attempting reauthorization
2023-06-09 14:08:11 +0000 UTC info: Output (stderr) from command "/usr/share/mender/identity/mender-device-identity": using interface /sys/class/net/enp1s0
2023-06-09 14:08:11 +0000 UTC info: successfully received new authorization data from server https://hosted.mender.io
2023-06-09 14:08:11 +0000 UTC info: Local proxy started
2023-06-09 14:08:11 +0000 UTC info: Reauthorization successful
2023-06-09 14:08:15 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2023-06-09 14:08:15 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

It seems to be caused by grub unable to load in the right kernel when switching partition, the Ubuntu 18 partition are using 5.3gke kernel, while the Ubuntu 20 partition is looking for 5.4 kernel, which doesn’t exists on the boot partition as the system are previously mender converted from Ubuntu 18.

Is it even possible to do what we are want, updating to mender converted Ubuntu 18 to 20 over the air??