Upgrading Mender client 2.6 to 4.0

General Discussions
1

Hello. I’m trying to update mender 2.6.1 to 4.0.4 with no success. Could I ask, please, if it can be done in theory? Thanks.

2025-01-03 09:31:16 +0000 UTC info: Running Mender client version: 2.6.1
2025-01-03 09:31:20 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2025-01-03 09:31:21 +0000 UTC info: No public key was provided for authenticating the artifact
2025-01-03 09:31:21 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2025-01-03 09:31:21 +0000 UTC info: Opening device "/dev/mmcblk0p2" for writing
2025-01-03 09:31:21 +0000 UTC info: Native sector size of block device /dev/mmcblk0p2 is 512 bytes. Mender will write in chunks of 1048576 bytes
2025-01-03 09:35:43 +0000 UTC info: All bytes were successfully written to the new partition
2025-01-03 09:35:43 +0000 UTC info: The optimized block-device writer wrote a total of 1745 frames, where 1177 frames did need to be rewritten (i.e., skipped)
2025-01-03 09:35:43 +0000 UTC info: Wrote 1828716544/1828716544 bytes to the inactive partition
2025-01-03 09:35:44 +0000 UTC info: State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]
2025-01-03 09:35:44 +0000 UTC info: State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]
2025-01-03 09:35:45 +0000 UTC info: Enabling partition with new image installed to be a boot candidate: 2
2025-01-03 09:35:46 +0000 UTC info: State transition: update-install [ArtifactInstall] -> reboot [ArtifactReboot_Enter]
2025-01-03 09:35:46 +0000 UTC info: Rebooting device(s)
2025-01-03 09:35:46 +0000 UTC info: Mender rebooting from active partition: /dev/mmcblk0p3
2025-01-03 09:35:47 +0000 UTC info: Daemon terminated with SIGTERM
2025-01-03 09:38:29 +0000 UTC info: Running Mender client version: 2.6.1
2025-01-03 09:38:29 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2025-01-03 09:38:30 +0000 UTC info: State transition: init [none] -> after-reboot [ArtifactReboot_Leave]
2025-01-03 09:38:31 +0000 UTC error: transient error: Reboot to the new update failed. Expected "upgrade_available" flag to be true but it was false. Either the switch to the new partition was unsuccessful, or the bootloader rolled back
2025-01-03 09:38:31 +0000 UTC info: State transition: after-reboot [ArtifactReboot_Leave] -> rollback [ArtifactRollback]
2025-01-03 09:38:31 +0000 UTC info: Performing rollback
2025-01-03 09:38:31 +0000 UTC info: No update available, so no rollback needed.
2025-01-03 09:38:31 +0000 UTC info: State transition: rollback [ArtifactRollback] -> rollback-reboot [ArtifactRollbackReboot_Enter]
2025-01-03 09:38:32 +0000 UTC info: Rebooting device(s) after rollback
2025-01-03 09:38:32 +0000 UTC info: Mender rebooting from inactive partition: /dev/mmcblk0p3
2025-01-03 09:38:32 +0000 UTC info: Daemon terminated with SIGTERM
2025-01-03 09:41:01 +0000 UTC info: Running Mender client version: 2.6.1
2025-01-03 09:41:01 +0000 UTC info: Mender shut down in state: rollback-reboot
2025-01-03 09:41:01 +0000 UTC info: Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available
2025-01-03 09:41:01 +0000 UTC info: State transition: init [none] -> verify-rollback-reboot [ArtifactRollbackReboot_Leave]
2025-01-03 09:41:01 +0000 UTC info: State transition: verify-rollback-reboot [ArtifactRollbackReboot_Leave] -> after-rollback-reboot [ArtifactRollbackReboot_Leave]
2025-01-03 09:41:02 +0000 UTC info: State transition: after-rollback-reboot [ArtifactRollbackReboot_Leave] -> update-error [ArtifactFailure]
2025-01-03 09:41:03 +0000 UTC info: State transition: update-error [ArtifactFailure] -> cleanup [Error]
2025-01-03 09:41:03 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]
2

Hi @Vladimir,

(You seem to be having a different issue so maybe is best on a separate thread?)

Yes, in theory it is possible. But there are some caveats.

First the log that you are copying in your message is from the device (presumably from journalctl), so we cannot see what the Mender Client 4 attempted to do. Can you provide us the deployment logs from the server? There it should be possible to see that the partition with Client 4 booted, then failed somehow, and triggered the rollback into the old partition with Client 2.

But just guessing a bit by looking at your logs, I think that the issue might be that your update image (your new partition) does not have the rootfs-image update module.

Before Mender Client 4.0, the update modules were optional because the client came with a built-in module (the rootfs-image one). Since 4.0, the client does not come with any built-in, but installs some modules by default (like the rootfs-image one). This means that /usr/share/mender/modules/v3 must exist and the rootfs-image module should be installed there in the new partition if you want to complete a rootfs update.

Lluis

3

Hello @lluiscampos. The log I posted above is the deployment failure log I’ve copied from mender web-interface. Also, I think that new Mender 4.0.4 image contains /usr/share/mender/modules/v3/rootfs-image file. I create sdimg file alongside with mender update file for RPi4, and rootfs-image exists in sdimg. Not sure if it is possible to check in the .mender file directly. Mender 2.6.1 image doesn’t have /usr/share/modules directory. Thanks.

4

@Vladimir Interesting. If the log is from the server, it means that the client in the new partition never got started (or it failed early before even attempting to continue the update).

Are you creating your images with yocto or mender-convert? Can you try first updating from 2.6.1 to 3.x?

You can check the .mender file with mender-artifact. For example mender-artifact cat my-update.mender:/usr/share/modules/v3/rootfs-image

5

@lluiscampos I create images with yocto. The old image (Mender 2.6.1) is dunfell 3.1 yocto branch, the new one (Mender 4.0.4) is scarthgap 5.0 yocto branch. I checked existense of rootfs-image by mender-artifact, it is present, thanks. Yes, I will create Mender 3.x image and try. Thanks.

6

So you are also upgrading through two LTSs of yocto: dunfell → (kirkstone, skipped) → scarthgap. This I didn’t expect.

Note that yocto releases are not guaranteed to be upgradable; in Mender we have detected several times breakages wrt bootloader integration or other parts and had amended or created workarounds.

I would tackle one problem at a time. Trying the following combinations should point to where the problem may be:

  1. Upgrade dunfell Mender 2.6 to dunfell Mender 3 → it most certainly works
  2. Upgrade dunfell Mender 2.6 to kirkstone Mender 3 → if it fails, we know that that the issue is dunfell to kirkstone
  3. Upgrade dunfell Mender 2.6 to kirkstone Mender 4 → if it fails, we know that that the issue is Mender 2 to Mender 4
  4. Upgrade dunfell Mender 2.6 to scarthgap Mender 4 → if it fails, we know that that the issue is dunfell to scarthgap

Hope it helps