Error in U-Boot setup: Boot and Root partitions don't match

Dear community,

I recently had to upgrade my entire build to Yocto 2.7 (warrior). I’m aware that Mender doesn’t officially support this latest version yet, but was hopeful that I might still get basic standalone updating working for my first batch of devices after my image built successfuly.

I’m using:

When I try to install an update using:

root@raspberrypi0-wifi:~# mender -install http://192.168.1.121:5000/core-image-base-raspberrypi0-wifi--20190725204233.mender

The process results in the following error:

Failed to read the current active partition: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) ‘mender_setup’ is not run by the U-Boot boot script: exit status 1

full output
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf module=config
INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config
ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) 'mender_setup' is not run by the U-Boot boot script: exit status 1 module=main
INFO[0000] Performing remote update from: [http://192.168.1.121:5000/core-image-base-raspberrypi0-wifi--20190725204233.mender]. module=standalone
Installing Artifact of size 98060800...
INFO[0000] no public key was provided for authenticating the artifact module=installer
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available module=modules
ERRO[0000] Download failed: Payload: can not install Payload: core-image-base-raspberrypi0-wifi.ext4: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) 'mender_setup' is not run by the U-Boot boot script: exit status 1 module=standalone
ERRO[0000] Payload: can not install Payload: core-image-base-raspberrypi0-wifi.ext4: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) 'mender_setup' is not run by the U-Boot boot script: exit status 1 module=main

Here’s the contents of the /etc/fw_env.config file:

/dev/mmcblk0 0x400000 0x4000
/dev/mmcblk0 0x800000 0x4000
fw_printenv output
altbootcmd=run mender_altbootcmd; run bootcmd
bootlimit=1
bootcount=0
upgrade_available=0
mender_boot_part=2
mender_boot_part_hex=2
mender_uboot_boot=mmc 0:1
mender_uboot_if=mmc
mender_uboot_dev=0
mender_boot_kernel_type=bootm
mender_kernel_name=uImage
mender_dtb_name=bcm2710-rpi-cm3.dtb
mender_pre_setup_commands=
mender_post_setup_commands=
mender_check_saveenv_canary=1
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_comm
ands; fi; setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk0p2; else setenv mender_boot_part_
name /dev/mmcblk0p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_
part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_po
st_setup_commands; fi
mender_altbootcmd=if test ${mender_boot_part} = 2; then setenv mender_boot_part 3; setenv mender_boot_part_hex 3; else setenv mender_boot_part 2; setenv mender_boot_part_hex 2; fi; s
etenv upgrade_available 0; saveenv; run mender_setup
mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi
bootcmd=run distro_bootcmd
bootdelay=2
baudrate=115200
preboot=usb start
loadaddr=0x00200000
arch=arm
cpu=arm1176
board=rpi
board_name=rpi
vendor=raspberrypi
soc=bcm283x
dhcpuboot=usb start; dhcp u-boot.uimg; bootm
stdin=serial,usbkbd
stdout=serial,vidconsole
stderr=serial,vidconsole
fdt_high=ffffffff
initrd_high=ffffffff
kernel_addr_r=0x00080000
scriptaddr=0x02400000
pxefile_addr_r=0x02500000
fdt_addr_r=0x02600000
ramdisk_addr_r=0x02700000
mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi
boot_net_usb_start=usb start
usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi
boot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} 
efi/boot/bootarm.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
efi_dtb_prefixes=/ /dtb/ /dtb/current/
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; for prefix in ${efi_dtb_prefixes};
 do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootarm.efi
; then echo Found EFI removable media binary efi/boot/bootarm.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
boot_prefixes=/ /boot/
boot_scripts=boot.scr.uimg boot.scr
boot_script_dhcp=boot.scr.uimg
boot_targets=mmc0 mmc1 usb0 pxe dhcp 
boot_syslinux_conf=extlinux/extlinux.conf
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIP
T FAILED: continuing...; fi
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run bo
ot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_
for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devn
um}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done
bootcmd_mmc0=setenv devnum 0; run mmc_boot
bootcmd_mmc1=setenv devnum 1; run mmc_boot
bootcmd_usb0=setenv devnum 0; run usb_boot
bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi
bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; t
hen setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00010:UNDI:003000;setenv 
bootp_arch 0xa;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_
addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done

I configure meta-mender as follows:

#
# Mender
#
# These are all provided by the meta-mender-core and meta-mender-raspberry layers
#
MENDER_FEATURES_ENABLE += "\
  mender-uboot \
  mender-image \
  mender-image-sd \
  mender-install \
"

# Memory card storage
MENDER_STORAGE_DEVICE = "/dev/mmcblk0"

# Memory card labeled 8GiB
MENDER_STORAGE_TOTAL_SIZE_MB = "7555"

# These are simply to align with how the "stock" RPi machines are
# configured.
MENDER_PARTITION_ALIGNMENT = "4194304"
MENDER_BOOT_PART_SIZE_MB = "40"
MENDER_DATA_PART_SIZE_MB = "512"

# rpi-base.inc removes these as they are normally installed on to the
# vfat boot partition. To be able to update the Linux kernel Mender
# uses an image that resides on the root file system and below line
# ensures that they are installed to /boot
IMAGE_INSTALL_append += "kernel-image kernel-devicetree"

# The artifact name is the Bitcoin block number at the point of the build
# https://blockstream.info/block/00000000000000000013680dae0234163c1b7af585990e694b621381d02b47da
MENDER_ARTIFACT_NAME = "582010"

# Mender will build an image called `sdimg` which shall be used instead
# of `rpi-sdimg`.
IMAGE_FSTYPES_remove += "rpi-sdimg"

# Use the same type here as specified in ARTIFACTIMG_FSTYPE to avoid

# building an unneeded image file.
SDIMG_ROOTFS_TYPE = "ext4"
ARTIFACTIMG_FSTYPE = "ext4"

And set RPI_USE_U_BOOT = "1".

The full configuration can be found in meta-candy/conf/distro/candy.conf.

I’ve tried to figure out where exactly the cause for this problem is, but failed in finding out. Now I don’t know if I should pursue this path further or revert to thud and try to resolve the other unrelated issues that made me upgrade in the first place.

I’d greatly appreciate any hints on how this could be resolved and feedback if there’s an easy fix at all. Thank you! :raised_hands:

David

Hi David,

Welcome to Mender Hub. We definitely appreciate hearing from users.

It seems that the environment is mismatched between UBoot and Linux. On my current RpI3 build, from thud, the fw_env.config file is identical to yours so the issue is something else.

I suggest you start by dumping the environment from UBoot and from Linux and comparing them.

1 Like

Hi Drew, thank you!

Here is the diff between the uboot and linux variables: davidknezic/linux.txt.

I made some more tests and experiments trying to narrow down the problem. I’ve learnt from Mender’s source code that it queries a specific variable during the check that fails for me:

root@raspberrypi0-wifi:~# fw_printenv mender_saveenv_canary
Warning: Bad CRC, using default environment
## Error: "mender_saveenv_canary" not defined

Interestingly, running that command returns an odd CRC error and reports that the value is missing, which can also be seen in the diff above.

I then tried to set the variable manually, to see if the updating process would work. After setting the variable, the CRC error disappeared and I ran an update.

root@raspberrypi0-wifi:~# mender -data /data/mender/ -config /data/mender/mender.conf -install http://192.168.1.121:5000/core-image-base-raspberrypi0-wifi--20190726162603.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf  module=config
INFO[0000] Loaded configuration file: /data/mender/mender.conf  module=config
WARN[0000] No server URL(s) specified in mender configuration.  module=config
WARN[0000] Server entry 1 has no associated server URL.  module=config
INFO[0000] Mender running on partition: /dev/mmcblk0p2   module=main
INFO[0000] Performing remote update from: [http://192.168.1.121:5000/core-image-base-raspberrypi0-wifi--20190726162603.mender].  module=standalone
Installing Artifact of size 99718144...
INFO[0000] no public key was provided for authenticating the artifact  module=installer
INFO[0000] Update Module path "/usr/share/mender/modules/v3" could not be opened (open /usr/share/mender/modules/v3: no such file or directory). Update modules will not be available  module=modules
INFO[0000] native sector size of block device /dev/mmcblk0p3 is 512, we will write in chunks of 1048576  module=dual_rootfs_device
INFO[0000] opening device /dev/mmcblk0p3 for writing     module=block_device
INFO[0000] partition /dev/mmcblk0p3 size: 3661627392     module=block_device
................................   1% 1024 KiB
................................   2% 2048 KiB
(21 minutes later)
...INFO[1254] wrote 3661627392/3661627392 bytes of update to device /dev/mmcblk0p3  module=dual_rootfs_device
                              100% 97381 KiB
INFO[1255] Enabling partition with new image installed to be a boot candidate: 3  module=dual_rootfs_device
Use -commit to update, or -rollback to roll back the update.
At least one payload requested a reboot of the device it updated.

It seemed to work.

root@raspberrypi0-wifi:~# fw_printenv mender_boot_part
mender_boot_part=3

root@raspberrypi0-wifi:~# fw_printenv bootcount
bootcount=0

root@raspberrypi0-wifi:~# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=1 cma=256M  smsc95xx.macaddr=B8:27:EB:14:4F:98 vc_mem.mem_base=0x1fa00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=ttyS0,115200  rootfstype=ext4 rootwait   logo.nologo  root=/dev/mmcblk0p2

reboot

However, after a reboot the device booted from the previous partition /dev/mmcblk0p2 again.

root@raspberrypi0-wifi:~# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=1 cma=256M  smsc95xx.macaddr=B8:27:EB:14:4F:98 vc_mem.mem_base=0x1fa00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=ttyS0,115200  rootfstype=ext4 rootwait   logo.nologo  root=/dev/mmcblk0p2

I rebooted and entered the uboot console after that to look into the environment. It still references the old partition mender_boot_part=2. So I thought let’s intervene manually again and see if I could even boot from the updated partition:

U-Boot> setenv mender_boot_part 3
U-Boot> setenv mender_boot_part_hex 3
U-Boot> run mender_setup              
U-Boot> run bootcmd

With that manual intervention it boots from the correct partition /dev/mmcblk0p3.

root@raspberrypi0-wifi:~# cat /proc/cmdline 
coherent_pool=1M 8250.nr_uarts=1 cma=256M  smsc95xx.macaddr=B8:27:EB:14:4F:98 vc_mem.mem_base=0x1fa00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 console=ttyS0,115200  rootfstype=ext4 rootwait   logo.nologo  root=/dev/mmcblk0p3

So it seems that the uboot and linux environments really aren’t in sync which causes all the trouble. Do you have an idea how I could try to fix this?

Hi David,

The CRC error is normal when the environment block has not been initialized. Basically, UBoot has a default setup of environment variables that will be used in this case.

I just took a look in the thud branch and that indicates that auto-configure doesn’t work for RpI due to it’s use of a boot script instead of the standard Mender environment additions. https://github.com/mendersoftware/meta-mender/blob/352cd81fe4dd6e44ba0819da1d8e189109e805e6/meta-mender-raspberrypi/recipes-bsp/u-boot/u-boot-raspberrypi.inc#L5

I think you need to disable auto-configuration and see if you can get it to build.

Drew

Hi Drew,

Okay. It already seems like auto-configure is disabled:

bitbake -e u-boot | grep '^MENDER_UBOOT_AUTO_CONFIGURE='
MENDER_UBOOT_AUTO_CONFIGURE="0"

Interestingly through, I’d assume that 0001-CONFIGS-rpi-enable-mender-requirements.patch shouldn’t be applied then, but I just discovered this in the logs:

Applying patch 0001-CONFIGS-rpi-enable-mender-requirements.patch
patching file configs/rpi_0_w_defconfig
Hunk #1 succeeded at 38 (offset 13 lines).
patching file configs/rpi_2_defconfig
Hunk #1 succeeded at 38 (offset 6 lines).
patching file configs/rpi_3_32b_defconfig
Hunk #1 succeeded at 41 (offset 7 lines).
patching file configs/rpi_3_defconfig
Hunk #1 succeeded at 41 (offset 7 lines).
patching file configs/rpi_defconfig
Hunk #1 succeeded at 38 (offset 6 lines).
patching file include/configs/rpi.h
Hunk #1 succeeded at 75 with fuzz 2 (offset -16 lines).
The context lines in the patches can be updated with devtool:
devtool modify u-boot
devtool finish --force-patch-refresh u-boot <layer_path>
Don’t forget to review changes done by devtool!
WARNING: u-boot-1_2019.01-r0 do_patch: QA Issue: Patch log indicates that patches do not apply cleanly. [patch-fuzz]
WARNING: u-boot-1_2019.01-r0 do_provide_mender_defines: Found more than one dtb specified in KERNEL_DEVICETREE. Only one should be specified. Choosing the last one.

So I added an u-boot_%.bbappend file with MENDER_UBOOT_AUTO_CONFIGURE = "0" in it and I assume that now auto-configure is disabled. At least some variables are gone from the u-boot environment.

The error still persists. Does that now mean I need to do the patching myself according to this guide?
https://docs.mender.io/2.0/devices/yocto-project/bootloader-support/u-boot/manual-u-boot-integration#u-boot-features

Oh well, it seems I misunderstood the meta-mender-raspberry u-boot append. The patch of course is supposed to be applied when MENDER_UBOOT_AUTO_CONFIGURE = "0". There is slight fuzz according to the logs, but the patched files look correct, the configs and the include file.

I notice that this CRC warning always comes up, when in user space. Even after several restarts and manual uboot saveenv. I wonder if it is possible that the fw_env.config doesn’t point to the correct env address. I can see the mender_saveenv_canary variable set to 1 in the uboot shell after the first boot, but never see it in user space because it always pulls the default env.

Is it possible that this is the issue and if so, is there a way to find out where the env is saved exactly?

I suppose that is likely the case but it’s odd that the version in Thud has the same fw_env.config without exhibiting this issue.

I would look at regenerating the patches in question so they apply cleanly; there are some cases where that doesn’t work properly.

As for finding the environment location, the only way I know to do it is to dig through the configured UBoot sources.

Drew

You can check the U-boot configuration in Yocto with the following:

cat tmp/work/raspberrypi3-poky-linux-gnueabi/u-boot/1_2019.01-r0/git/include/config_mender_defines.h 

Note that you might need to run:

bitbake -c cleansstate u-boot
bitbake -c deploy u-boot

For the file to appear (it might get cleaned up by Yocto).

I did a quick test run, trying to reproduce your environment and the variables do turn out correct to me (matching what is in /etc/fw_env.config),

/* Shell variables */
#define MENDER_BOOT_PART_NUMBER 1
#define MENDER_BOOT_PART_NUMBER_HEX 1
#define MENDER_ROOTFS_PART_A_NUMBER 2
#define MENDER_ROOTFS_PART_A_NUMBER_HEX 2
#define MENDER_ROOTFS_PART_B_NUMBER 3
#define MENDER_ROOTFS_PART_B_NUMBER_HEX 3
#define MENDER_UBOOT_STORAGE_INTERFACE "mmc"
#define MENDER_UBOOT_STORAGE_DEVICE 0

/* BB variables. */
#define MENDER_STORAGE_DEVICE "/dev/mmcblk0"
#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 0x400000
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 0x800000
#define MENDER_ROOTFS_PART_A_NAME "/dev/mmcblk0p2"
#define MENDER_ROOTFS_PART_B_NAME "/dev/mmcblk0p3"

/* For sanity checks. */
#define MENDER_BOOTENV_SIZE 0x4000

Currently on the road though so not able to test anything.

Had a chance to test things out on warrior, and I have something working here for Raspberry Pi,

Will be merged in the official meta-mender once official warrior support is added.

1 Like

Mirza, thank you! Your rebased uboot patch made it work on warrior. I was able to successfully update my candy dispenser firmware. :yum:

The changes even work with the master branches of poky, oe and the raspberrypi layer, with a slight modification of the e2fsprogs csum patch: https://github.com/the-lightning-land/meta-mender/commit/7d626ac04cb84ecd891448f3678533a5c281ed9c