# OTA Update failed with "No match between boot and root partitions" **URL:** \ **Category:** General Discussions\ **Tags:** yocto, raspberry-pi-3, system-update\ **Created:** [December 4, 2024, 2:09pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370 "2024-12-04T14:09:41Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![NicoLueg](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/nicolueg/32/2364_2.png) [@NicoLueg](https://hub.mender.io/u/NicoLueg)\ **Post date:** [December 4, 2024, 2:09pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/1 "2024-12-04T14:09:41Z")
Hello guys! We use Mender for a few months but are still at the start of our journey. We have like 20 productive devices at our customers’ locations and introduced Mender to be able to update the device without having access to the device in case anything happens. Now we have the case that something broke up and we need to update the devices. We implemented the update in our custom layer (only some changes in the Python code) and built the image with the same configuration as before. When we tried to create a deployment we received this log: ```auto 2024-12-04 09:43:41 +0000 UTC info: Running Mender client version: 3.5.3 2024-12-04 09:43:41 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter] 2024-12-04 09:43:41 +0000 UTC info: No public key was provided for authenticating the artifact 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read environment, using default 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read default environment from file 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read environment, using default 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read default environment from file 2024-12-04 09:43:42 +0000 UTC error: Artifact install failed: Payload: can not install Payload: core-image-minimal-raspberrypi3-64.ext4: No match between boot and root partitions.: exit status 243 2024-12-04 09:43:42 +0000 UTC info: State transition: update-store [Download_Enter] -> cleanup [Error] 2024-12-04 09:43:42 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none] ``` We also get the error “No match between …” on the devices when we run `mender show-artifact`. We tried to solve this by reading some very old discussions here but we aren’t able to solve this as we don’t really understand what the error is telling us. Can someone please assist with that? ## Details of our integration We integrated mender in an existing Yocto project that consists of the following layer: ```auto poky/meta poky/meta-poky poky/meta-yocto-bsp meta-raspberrypi meta-rust meta-openembedded/meta-oe meta-openembedded/meta-filesystems meta-openembedded/meta-networking meta-openembedded/meta-python meta-virtualization meta-mender/meta-mender-core meta-our-custom layer (only has some services that run python scripts) ``` This is the relevant part of our configuration: ```conf CONF_VERSION = "2" MACHINE ?= "raspberrypi3-64" # Mender configurations MENDER_ARTIFACT_NAME = "release-1.1" INHERIT += "mender-full" MENDER_SERVER_URL="https://eu.hosted.mender.io/" MENDER_TENANT_TOKEN = "" MENDER_FEATURES_ENABLE:append = " mender-uboot mender-image-sd" MENDER_FEATURES_DISABLE:append = " mender-grub mender-image-uefi" MENDER_STORAGE_DEVICE = "/dev/mmcblk0" MENDER_BOOT_PART_SIZE_MB = "128" MENDER_STORAGE_TOTAL_SIZE_MB = "30448" ``` We have these partitions on our devices: | Device | Boot | Start | End | Sectors | Size | Id | Type | | --- | --- | --- | --- | --- | --- | --- | --- | | /dev/mmcblk0p1 | \* | 49152 | 311295 | 262144 | 128M | c | W95 FAT32 (LBA) | | /dev/mmcblk0p2 | | 311296 | 31195135 | 30883840 | 14.7G | 83 | Linux | | /dev/mmcblk0p3 | | 31195136 | 62078975 | 30883840 | 14.7G | 83 | Linux | | /dev/mmcblk0p4 | | 62078976 | 124735487 | 62656512 | 29.9G | 83 | Linux | Thank you for any help … ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [December 4, 2024, 2:37pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/2 "2024-12-04T14:37:26Z")
Hi @NicoLueg, This is the source of your problem: > [@NicoLueg](#): > > ```auto > 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read environment, using default > 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read default environment from file > 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read environment, using default > 2024-12-04 09:43:42 +0000 UTC info: Output (stderr) from command "fw_printenv": Cannot read default environment from file > > ``` `fw_printenv` can’t read the `u-boot` environment and returns the default. The default probably does not contain a proper root partition setting, hence the Mender Client can’t understand the current state and bails out. As for the reasons why your `u-boot` environment is defective or non-readable, I cannot comment. Do you have a local reproducer? If so, what is the output if `fw_printenv`, and what are its configuration settings? Additionally, you might want to `hexdump` the environment memory addresses to see what’s actually there, and/or inspect the `u-boot` build log via serial console. Greetz, Josef ---
**Author:** ![je-s](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/je-s/32/2362_2.png) [@je-s](https://hub.mender.io/u/je-s)\ **Post date:** [December 4, 2024, 11:15pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/3 "2024-12-04T23:15:49Z")
Hey @TheYoctoJester, working together with @NicoLueg here. Your explanation sounds reasonable, but we’re not sure why this occurs, despite the image working correctly when burning it to a storage manually without trying to do an update. `fw_printenv` outputs: > Configuration file wrong or corrupted After building, we have the following in the build directory: `fw_env.config`: ```auto /boot/uboot.env 0x0000 0x4000 ``` `fw_env_config.default`: ```auto /dev/mmcblk0 0x800000 0x20000 /dev/mmcblk0 0x1000000 0x20000 ``` The latter one (the default) also reflects the contents in `/data/u-boot/fw_env.config` and `/etc/fw_env.config`on the actual device with the _prior_ version which we want to update. So it seems like the device is falling back to that default. Additional information: Despite that we’re creating the Mender Artifact with: ```auto mender-artifact-linux write rootfs-image -t $DEVICE_TYPE -n rootfs-1.1 -o rootfs-1.1.mender -f yocto_deploy_result/raspberrypi3-64/core-image-minimal-raspberrypi3-64.sdimg --software-version 1.1 ``` We also tried to create an artifact with the `ext4` instead of the `sdimd` output of the Bitbake process, and also with the Mender artifact created by Bitbake itself in the course of building. All three variants spawned the same error. Let us know when you need anything else. 🙂 Thank you very much so far. Best regards JE ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [December 5, 2024, 6:33am UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/4 "2024-12-05T06:33:32Z")
Hi @je-s, so this is definitely incorrect: > [@je-s](#): > > Despite that we’re creating the Mender Artifact with: > > ```auto > mender-artifact-linux write rootfs-image -t $DEVICE_TYPE -n rootfs-1.1 -o rootfs-1.1.mender -f yocto_deploy_result/raspberrypi3-64/core-image-minimal-raspberrypi3-64.sdimg --software-version 1.1 > > ``` Reason: you’re putting a full sd card image into an Artifact. The Client doesn’t care, to it it’s just a binary blob, and will try to put it into the partition (which will probably fail due to size), but you don’t even get there, as the partitions preconditions are checked before starting the download. If you want to create the Artifact manually, then you need to use the `.ext4` (or whatever you configured it to) filesystem as payload. I would suggest to also inspect the actual command which is run if the Artifact is created through `bitbake` for comparison. After looking at your layers, I also think I’ve found the problem: you don’t use `meta-mender-raspberrypi`, which was in `meta-mender` up to and including `kirkstone`, and in `meta-mender-community` since `scarthgap`. That one brings the `u-boot` configuration and patches required for the Raspberry Pi. So if you really did not use it in the build, that’s the problem. And I’m pretty sure that the build will never have passed a test update then. If not mentioning it was just an oversight, no straight idea. So here’s a couple of pointers to dig: - if you take a fresh SD card, flash it with the `.sdimg`, boot it and then run `fw_printenv`, what do you get? - in such a freshly built image, are there also the two diverging `fw_env.config` and `fw_env_config.default`? - has the update process worked at some point in the past, with exactly that build? - is this a standard RPi3, or some variant? Greetz, Josef ---
**Author:** ![je-s](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/je-s/32/2362_2.png) [@je-s](https://hub.mender.io/u/je-s)\ **Post date:** [December 5, 2024, 2:34pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/5 "2024-12-05T14:34:28Z")
Hey @TheYoctoJester, our usual way was to create the artifact with the `.ext4` image, or we used the one provided by Bitbake. Regarding the `sdimg` it makes sense that it’s the wrong way to create an artifact anyways. But the error was the same in all three variants we’ve tested. That was more or less just one of the failed trials by doing trial & error. 🙂 We indeed didn’t include this layer. Trying to see what happens when we integrate that now. But since the installation seems to lack the correct configuration nevertheless, I guess that’ll still not help when trying to update the OS in that way. Do you maybe see any way to patch this problem remotely and fix that issue up, so that we’re able to introduce an OS update? Probably my understanding of the issue isn’t deep enough, but I would suppose that if we manage to patch the `fw_env.config` remotely -via an application update maybe-, we should be able to make the update work , since it seems that it’s only missing that file. Or do you think we’re stuck with that and just need to roll out new SD cards? Regarding your other questions: - With a fresh SD card we still have the same output for `fw_printenv` - The two files are also still diverging. Going to see what changes when `meta-mender-raspberry` is integrated. - We thought it did, but it was an application update that worked. We never did an successful OS Update before unfortunately. - It’s an RPi3 B+. The + changed some Chipsets in comparison to the non-plus version, but it’s officially supported and running that didn’t spawn any further problems. Best regards JE ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [December 5, 2024, 3:09pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/6 "2024-12-05T15:09:13Z")
Hi @je-s, If the layer is not there then the other questions are not relevant anymore, because that explains your problem fully. Unfortunately it’s not just the `fw_env.config`, but actually the `u-boot` binary itself on the boot partition that you need to update, at least. Not sure if other things might be affected too, but at least from a first glance it _might_ be good enough. Please note that changing that is definitely a single point of failure and might brick the devices. If the devices are easily accessible, rolling out fresh SD cards might be a more time efficient solution, as you don’t need to construct the application update artifacts and test the process. Greetz, Josef ---
**Author:** ![je-s](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/je-s/32/2362_2.png) [@je-s](https://hub.mender.io/u/je-s)\ **Post date:** [December 5, 2024, 3:28pm UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/7 "2024-12-05T15:28:46Z")
Hey @TheYoctoJester, thank you very much, we’re tracing that path then. 🙂 In case we’ve found a solution to make it work remotely so that the mender-client consumes the updates, I’ll give an update here. Cause since application updates work like they should, we have the option to inject files into the file system, which might do the trick, in case we’re able to update u-boot and the minimal configurations required. Thanks for your support so far! BR JE ---
**Author:** ![je-s](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/je-s/32/2362_2.png) [@je-s](https://hub.mender.io/u/je-s)\ **Post date:** [December 6, 2024, 1:45am UTC](https://hub.mender.io/t/ota-update-failed-with-no-match-between-boot-and-root-partitions/7370/8 "2024-12-06T01:45:23Z")
Update: - We could validate that the missing `meta-mender-raspberry` layer was indeed the source of the problem. After adding that it’s no problem to issue OS updates anymore. Works without any trouble. Also, when hexdumping the respective address space, you can clearly see the correct configuration now, whereas that was missing completely before the patch. Still wondering why it even booted in the first place, seems there was a fallback bootloader in place which handled that. - In regards of fixing already deployed devices remotely we don’t have a solution yet. Theoretically it’s possible to update the bootloader remotely, but practically it’s hard to reproduce the exact configurations needed, so that we end up with a useful update package which would maybe allow us to fix up broken devices. _Update:_ We skipped the latter part, since supplying new storage media once was ultimately faster. In theory it should be possible to patch the bootloader remotely via a .deb or .rpm package and do all the modifications required, but for that you’d need a pretty exact configuration. The process was too tedious, which is why we skipped on that and just made a fresh and clean hardware-deployment now. 🙂