Artifact update error: Failed to read the current active partition: No match between boot and root partitions

First of all: Mender is quite new to me. I am trying to set up a system for firmware update of several devices (FriendlyELEC NanoPi R1) running Ubuntu 16.04, but I’m not able to deploy any artifacts to the devices. I would really appreceate if anybodyu could help me!

The Mender version of this distribution is created using ‘mender-convert’ based on info on https://docs.mender.io/system-updates-debian-family/convert-a-mender-debian-image

The deplyment log from hosted.mender.io reads:

2020-11-10 13:04:04 +0000 UTC info: Running Mender client version: 2.3.0
2020-11-10 13:04:05 +0000 UTC info: State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]
2020-11-10 13:04:05 +0000 UTC info: No public key was provided for authenticating the artifact
2020-11-10 13:04:05 +0000 UTC error: Artifact install failed: Payload: can not install Payload: root_part_0.0.1.ext4.gz: No match between boot and root partitions.: exit status 1
2020-11-10 13:04:05 +0000 UTC info: State transition: update-store [Download_Enter] -> cleanup [Error]
2020-11-10 13:04:05 +0000 UTC info: State transition: cleanup [Error] -> update-status-report [none]

The error seems to be No match between boot and root partitions.

Checking the device gives me a hint thtat something is wrong:

$ sudo mender show-artifact

ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: exit status 1
xxx_release_1

Checking fw_printenv also indicates something wrong with the boot (bad CRC):

$ sudo fw_printenv
    Warning: Bad CRC, using default environment
    bootcmd=bootp; setenv bootargs root=/dev/nfs nfsroot=${serverip}:${rootpath} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}::off; bootm
    bootdelay=1
    baudrate=115200
    fdt_high=0xffffffff
    initrd_high=0xffffffff
    rootdev=0
    rootpart=2
    bootpart=1
    kernel=zImage
    loadaddr=0x71080000
    dtb_name=s5p4418-nanopi2-rev01.dtb
    dtb_addr=0x7A000000
    initrd_name=ramdisk.img
    initrd_addr=0x79000000
    initrd_size=0x600000
    load_dtb=ext4load mmc ${rootdev}:${bootpart} ${dtb_addr} ${dtb_name}; run dtb_reserve
    load_kernel=ext4load mmc ${rootdev}:${bootpart} ${loadaddr} ${kernel}
    load_initrd=ext4load mmc ${rootdev}:${bootpart} ${initrd_addr} ${initrd_name}; setenv initrd_size 0x${filesize}
    mmcboot=run load_kernel; run load_initrd; run load_dtb; bootz ${loadaddr} ${initrd_addr}:${initrd_size} ${dtb_addr}
    bootcmd=run mmcboot
    splashimage=0x78000000
    splashfile=logo.bmp
    splashpos=m,m
    fb_addr=
    dtb_reserve=if test -n "$dtb_addr"; then fdt addr $dtb_addr; fi

I’ve also read somewhere that mender_boot_part environment variable is
unset or wrong. That may be the case, but currently I have no idea on how to fix it.

My problem is that I don’t have too much knowledge about uboot and boot configuration, so I am afraidthat the issue easily can be fixed in the configuration. I’ve tried to read as much as possible, both documentation on mender and looking for help in the Mender Hub, but i cannot find anything that fixes this problem.

Details:

  • Core: Ubuntu 16.04 Xenial
  • Hardware: FriendlyElec NanoPi R1
  • Disk image: nanopi-r1_sd_friendlycore-xenial_4.14_armhf_20191219.img
  • /mender-convert/configs/NanoPi_R1_config’:
    MENDER_DEVICE_TYPE="NanoPi_R1"
    MENDER_GRUB_EFI_INTEGRATION=n
    MENDER_KERNEL_IMAGETYPE="zImage"
    MENDER_PARTITION_ALIGNMENT="8388608"
    MENDER_STORAGE_TOTAL_SIZE_MB="7456"

Commands to produce Mender image:

$ INPUT_DISK_IMAGE=$(ls nanopi-r1_sd_friendlycore-xenial_4.14_armhf_20191219.img)
$ export MENDER_ARTIFACT_NAME="xxx_release_1"
$ sudo ./scripts/bootstrap-rootfs-overlay-hosted-server.sh --output-dir ${PWD}/rootfs_overlay --tenant-token "<xxx>"
$ sudo -E ./mender-convert --disk-image $INPUT_DISK_IMAGE --config configs/nanopi_r1_config --overlay rootfs_overlay/

Commands to flash device (booted from external SD-card)

$ gzip -c -k -d nanopi-r1_sd_friendlycore-xenial_4.14_armhf_20191219-NanoPi_R1-mender.img.gz | dd of=/dev/mmcblk1 bs=10M status=progress conv=fdatasync 

Hi @johan, welcome to Mender hub and thanks for the detailed report.

The error logs do indeed indicate that the mender_boot_part bootloader environment variables are unset. For full image updates, this is obviously needed so Mender knows which partition it should be writing to.

The CRC error message is normal with U-Boot when the environment area is uninitialized. In this case U-boot defaults to the “default environment”. However with proper Mender U-Boot integration this should not appear as we will have updated the environment at boot time.

My suspicion is that mender-convert is trying to use Grub based integration which is our default now. Basically, U-boot is the first stage bootloader which then launches Grub as an EFI app. Grub then does the Mender required processing. However, that requires some functionality in U-Boot which may not be enabled in the Ubuntu NanoPi image you are using here.

I know that @tranchitella did some work with Ubuntu on the RpI and maybe he has some suggestions. Otherwise, @mirzak do you have any suggestions how this board may be supportable?

Drew

So to me it looks like you disabled the GRUB integration with:

MENDER_GRUB_EFI_INTEGRATION=n

The purpose of above, is essentially that you do not need do anything with U-Boot as the Mender integration is in GRUB. But for this to work there are certain prerequisites, the primary being that the existing U-Boot binary is using something called distro_bootcmd, which is an “boot probe” of sorts, which should find the GRUB binary and load it.

But based on the output you pasted from fw_printenv I would say that your device does not use distro_bootcmd so I think it is correct to do MENDER_GRUB_EFI_INTEGRATION=n.

The fallback then is that you need to integrate a set of patches in your U-Boot and recompile it, and then provide a U-Boot binary as part of the “mender-convert” to replace the existing one. This gets a bit more involved and can be cumbersome if you do not have past experience with this, but I will share some links that will hopefully be useful and that enable you to progress further.

The only thing I can do is point you at references, and documentation. Would recommend the following links:

Many and all over the place :slight_smile:, but hopefully they can provide some additional insights that enable you to move forward.

Thanks a lot for the quick response. I really cannot remember anymore why I disabled GRUB, but I presume that the build failed or that the device did not boot the image… I’ll look into the other tips you gave me. Thanks!

I forgot to include the /uboot/boot.cmd and /uboot/uEnv.txt from the Mender image I created using mender-convert. Don’t know if it helps much, but here they are:

Note: I manually removed the overlayfs (mmcblk0p3) and recompiled to boot.scr prior to the conversion to simplify the template image and fix another issue.

/uboot/boot.cmd

# Recompile with: mkimage -C none -A arm -T script -d boot.cmd boot.scr
# CPU=H3
# OS=friendlycore/ubuntu-oled/ubuntu-wifiap/openwrt/debian/debian-nas...

echo "running boot.scr"
setenv load_addr 0x44000000
setenv fix_addr 0x44500000
fatload mmc 0 ${load_addr} uEnv.txt
env import -t ${load_addr} ${filesize}

fatload mmc 0 ${kernel_addr} ${kernel}
fatload mmc 0 ${ramdisk_addr} ${ramdisk}
setenv ramdisk_size ${filesize}

fatload mmc 0 ${dtb_addr} sun8i-${cpu}-${board}.dtb
fdt addr ${dtb_addr}

# merge overlay
fdt resize 65536
overlay search
for i in ${overlays}; do
    if fatload mmc 0 ${load_addr} overlays/sun8i-h3-${i}.dtbo; then
        echo "applying overlay ${i}..."
        fdt apply ${load_addr}
    fi
done
fatload mmc 0 ${fix_addr} overlays/sun8i-h3-fixup.scr
source ${fix_addr}

# setup XR819 MAC address
if test $board = nanopi-duo; then fdt set xr819 local-mac-address ${wifi_mac_node}; fi

# setup boot_device
fdt set mmc${boot_mmc} boot_device <1>

#setenv overlayfs data=/dev/mmcblk0p3
#setenv hdmi_res drm_kms_helper.edid_firmware=HDMI-A-1:edid/1280x720.bin video=HDMI-A-1:1280x720@60
setenv pmdown snd-soc-core.pmdown_time=3600000

setenv bootargs "console=${debug_port} earlyprintk
root=/dev/mmcblk0p2 rootfstype=ext4 rw rootwait fsck.repair=${fsck.repair}
panic=10 fbcon=${fbcon} ${hdmi_res} ${overlayfs} ${pmdown}"

bootz ${kernel_addr} ${ramdisk_addr}:${ramdisk_size} ${dtb_addr}

/uboot/uEnv.txt

#################################################################
# General options
#################################################################
ramdisk=rootfs.cpio.gz
kernel=zImage
fix_addr 0x44500000
kernel_addr=0x46000000
ramdisk_addr=0x47000000
dtb_addr=0x48000000

debug_port=ttyS0,115200
fsck.repair=yes

# 0 = /dev/fb0, First framebuffer
# 1 = /dev/fb1, Second framebuffer
# README: linux/Documentation/fb/fbcon.txt
fbcon=map:0

Update: I finally made it work. Took a lot of time to dig into all the details, but I know a bit more about U-Boot and Mender now… Thanks @drewmoseley @mirzak and for helping me finding the right information.

My path to success to get Mender to run on Nano Pi R1 (not all details are included):

1 Build U-boot for Mender

I concluded that I had to build the U-Boot image from original source and Mender patches. When we started looking at Mender as a system, we did not realize that this was necessary as the system only was tested using Raspberry devices.

Steps to build U-Boot

  • Read the following posts:
  • Forked U-boot for NanoPi R1 (link)
  • Installed necessary toolchain and cross compiler from http://download.friendlyarm.com/nanopineo
  • Modified/patched the forked U-Boot source to become Mender U-Boot
  • Added the following lines to configs/nanopi_h3_defconfig:
    • CONFIG_ENV_IS_IN_MMC=y
    • CONFIG_USE_BOOTARGS=y
    • CONFIG_BOOTARGS="console=ttyS0,115200 panic=10 rootfstype=ext4 rw rootwait fsck.repair=yes"
  • Remove the following lines from configs/nanopi_h3_defconfig:
    • CONFIG_ENV_OFFSET ...
  • Added the following lines to include/configs/sun8i.h (Note: should be in include/friendlyelec/boardtype.h, but this file is not included early enough…):
    • #define CONFIG_BOOTCOUNT_LIMIT
    • #define CONFIG_BOOTCOUNT_ENV
    • #define CONFIG_SYS_REDUNDAND_ENVIRONMENT(undocumented?)
  • Removed/fixed the following lines from Kconfig
    • ENV_OFFSET: default 0x88000 if ARCH_SUNXI (should be 0x400000)
    • ENV_SIZE: default 0x20000 if ARCH_SUNXI (should be 0x4000)
  • Fixed the MENDER_LOAD_KERNEL_AND_FDT by adding the following lines after loading fdt. This was a bit tricky, but necessary to load the kernel from correct partition and avoiding re-enumerating all the mmc’s). Also note that all boot files are lodead from /boot/ at root partition. I am not even sure if this is the correct way to do it, but it works…:
# define MENDER_LOAD_KERNEL_AND_FDT                                        \
    "if test \"${fdt_addr_r}\" != \"\"; then "                             \
    "load ${mender_uboot_root} ${fdt_addr_r} /boot/${mender_dtb_name}; " \
    "fdt addr ${fdt_addr_r}; "                                             \
    "fdt set mmc${boot_mmc} boot_device <1>; "                             \
    "fi; "                                                                 \
    "load ${mender_uboot_root} ${kernel_addr_r} /boot/${mender_kernel_name}; "
#endif
  • Compiled U-Boot and fw_printenv
$ export PATH=/opt/FriendlyARM/toolchain/4.9.3/bin:$PATH
$ make nanopi_h3_defconfig ARCH=arm CROSS_COMPILE=arm-linux-
$ make ARCH=arm CROSS_COMPILE=arm-linux-
$ make envtools ARCH=arm CROSS_COMPILE=arm-linux-

The files fw_env.config, fw_printenv and u-boot-sunxi-with-spl.bin is then used as a bases (togehter with the base image) and fed into mender-convert to create the final complete image.

2 Create image using mender-convert

The trick to get the final image was to use basic mender-convert for Ubuntu/Debian and convert the image by injecting the U-Boot image directly into using dd. In addition, all boot files were placed in a folder called /boot/ at the root-fs. Finally fw_printenv/fw_setenv was installed together with the configuration file.

  • Created a special configuration file (configs/nanopi_r1_config) for Nano Pi R1 to be used in mender-convert:
MENDER_DEVICE_TYPE="NanoPi_R1"
MENDER_GRUB_EFI_INTEGRATION=n
MENDER_KERNEL_IMAGETYPE="zImage"
MENDER_IGNORE_MISSING_EFI_STUB=1

# 8MB alignment
MENDER_PARTITION_ALIGNMENT="8388608"
MENDER_STORAGE_TOTAL_SIZE_MB="7456"
MENDER_COMPRESS_DISK_IMAGE="gzip"

function platform_modify() {
  log_info "Copying boot-files to /boot/"
  run_and_log_cmd "sudo cp -R work/boot/* work/rootfs/boot/"
  log_info "Installing fw_printenv and fw_setenv"
  run_and_log_cmd "sudo install -m 755 u-boot-files/fw_printenv work/rootfs/sbin/fw_printenv"
  run_and_log_cmd "sudo ln -fs /sbin/fw_printenv work/rootfs/sbin/fw_setenv"
  run_and_log_cmd "sudo ln -fs /sbin/fw_printenv work/rootfs/usr/bin/fw_printenv"
  run_and_log_cmd "sudo ln -fs /sbin/fw_printenv work/rootfs/usr/bin/fw_setenv"
  run_and_log_cmd "sudo ln -fs /sbin/fw_printenv work/rootfs/usr/local/bin/fw_printenv"
  run_and_log_cmd "sudo ln -fs /sbin/fw_printenv work/rootfs/usr/local/bin/fw_setenv"
  run_and_log_cmd "sudo cp u-boot-files/fw_env.config work/rootfs/etc"
}

function platform_package() {
  log_info "Writing pre-compiled U-Boot image to ${img_path}"
  run_and_log_cmd "sudo dd if=u-boot-files/u-boot-sunxi-with-spl.bin of=${img_path} bs=1024 seek=8 conv=notrunc"
}

  • Put files from U-Boot in a folder:
    mender-convert/u-boot-files/fw_env.config
    mender-convert/u-boot-files/fw_printenv
    mender-convert/u-boot-files/u-boot-sunxi-with-spl.bin

  • Run mender-convert

This image can finally be dd’d directly to the device…

3 Final notes

Final testing remains. I cannot guarantie that this will actually work. Remaining tasks:

  • Test and verify that device is bootable from both partitions (p2 and p3)
    • Seems to be working by manually setting mender_boot_part and mender_boot_part_hex
  • Test that and ferify device appears on Mender management server
  • Test and verify that device can be updated with artifacts

Edit: Realized that i could place all boot files (kernel and *.dtb) in the folder /boot/ at root. Changed /u-boot/ -> /boot/. Also fixed some typos.

2 Likes

Hello, @mirzak. I am working hard to get this ting to work. It is so close…

A new update:

Starting to look good, but Artifact upload (root-fs) does not work as expected, but this time it there is something strange going on…

  1. The Artifact downloads, and the device reboots
  2. Reboot fails as the /dev/mmcblk0p2 does not contain any kernel (or anything, relly). The partition is bad.
  3. U-Boot hangs because it ignores errors on kernel-load

I tried to reboot manually (power on/off), and the device reboots again and again, still no kernal can be found. Entering U-Boot console reviles that mender_boot_part and mender_boot_part_hex are missing! Is this correct? Running mender_altbootcmd manually from U-Boot console makes the device boot the device, but failover fails…

Regarding the partition used to create the Artifact: I am successfully able to mount and browse it. It is a simple partition dump fed into mender-artifact tool

Questions:

  1. How can mender destroy the secondary root partition (/dev/mmcblk0p3) from a “accepted” Artifact (a bit open, maybe)?
  2. The initial boot failure is caused by the bad partition update, but shouldn’t Mender try to boot back into the working partition /dev/mmcblk0p2 automatically?
  3. Do I have to take extra measures to avoid hanging kernel by checking in all loads in the bootcmd
  4. Is altbootcmd ever run automatically by mender?

Thanks!

Hi @johan indeed mender_boot_part and mender_boot_part_hex must be present in the uboot environment. I thought it would error out if those were not set though.

Can you install your artifact in standalone mode and provide the command line output? If not, can you at least leave journalctl -u mender-client --follow running while you do the OTA deployment and provide the output from that?

Also, the output of printenv from the U-Boot prompt and fw_printenv from the Linux prompt might show something interesting.

Drew

Hi @johan,

First of all, thank you for a fantastic summary of your progress. This will be very useful to our community and thank you for your contributions.

Now to your problems,

Entering U-Boot console reviles that mender_boot_part and mender_boot_part_hex are missing! Is this correct?

This is not correct. I think you are experiencing a bug/problem in the Mender client in combination with older U-Boot versions (pre 2018.xx). Based on the link you shared you are using 2017.xx U-Boot.

More details can be found here, [MEN-3970] - Mender and CFEngine (by Northern.tech) Jira

We have fixed this in 2.3.1 and 2.4.1 version of the Mender client, so if you could update I think this wíll solve the problem with mender_boot_part and mender_boot_part_hex disappearing.

  1. How can mender destroy the secondary root partition ( /dev/mmcblk0p3 ) from a “accepted” Artifact (a bit open, maybe)?

Not sure, but first suspection might be that your partitions might overlap or similar. Can you share output of:

  • fdisk - l /dev/mmcblk0
  • cat /etc/fw_env.config
  1. The initial boot failure is caused by the bad partition update, but shouldn’t Mender try to boot back into the working partition /dev/mmcblk0p2 automatically?

Yes, and depends. What is your bootcmd in U-Boot? It should be something like this:

run mender_setup;
# your boot stuff here
run mender_try_to_recover;

The mender_try_to_recover should trigger a rollback if executed.

  1. Do I have to take extra measures to avoid hanging kernel by checking in all loads in the bootcmd

I believe the same mechanism as explained at previous step should cover this.

  1. Is altbootcmd ever run automatically by mender?

Yes, or not by Mender per-se, this is a U-Boot feature,

#define CONFIG_BOOTCOUNT_LIMIT
#define CONFIG_BOOTCOUNT_ENV

Assuming that you have defined above as part of the Mender integration, it will call altbootcmd if bootcount >= bootlimit. bootlimit=1 by default.

Thanks again @drewmoseley and @mirzak for the response! I really appreciate all help I can get. It helps me getting forward on this journey!

Disk partition issue (fixed):

This was due to the fact that I used a compressed (gz) image in the command: mender-artifact write rootfs-image -f root-part.ext4.gz .... I simply did a hexdump of /dev/mmcblk0p2 and discovered the first 3 bytes were 1F 8B 08 (gzip file signature). Using a uncompressed .ext4 file to the mender-artifact worked correctly. Reading the documentation, it is still unclear to me if .ext4.gz is possible or not and why there was no error message or warning…

Missing mender_boot_part and mender_boot_part_hex

This is a bit more tricky. https://tracker.mender.io/browse/MEN-3970 is probably the cause of this, yes. Tried the following commands to verify:

You are correct that using ‘=’ when using fw_setenv does not work:

$ sudo fw_printenv x
## Error: "x" not defined
$ sudo fw_setenv x=123
$ sudo fw_printenv x
## Error: "x" not defined
$ sudo fw_setenv x 123
$ sudo fw_printenv x
x=123
$

As I see it, this leaves me with two options:

  1. Upgrade U-Boot to latest
  2. Upgrade mender-client to latest

Option 1. Upgrading U-Boot:

This sounds like a lot of work, as this is a patch to a forked and modified 2017-version of the original U-Boot. This probably requires manual merging of a few files…

Option 2. Specifying mender-client version

Upgrading mender-client seemed the easiest at first, but then i discovered that mender-client 4.1 requires libssl1.1. Unfortunately, this is not available on Ubuntu 16.04. This leves me with mender-client 3.1, using libssl1.0. Looking at the documentation for mender-convert, I probably will have to specify MENDER_CLIENT_VERSION="2.3.1"

Looking at mender-convert-modify script, this will result in doing the following wget:

$ wget -Nq https://downloads.mender.io/2.3.1/dist-packages/debian/armhf/mender-client_2.3.1-1_armhf.deb -P work/mender-deb

I will try option 2 and report back…

Success!

FriendlyELEC NanoPi R1 is now Mender-capable. I am able to update rootfs with Mender Artifact, and also verified automatic rollback with faulty Artifact upload.

https://tracker.mender.io/browse/MEN-3970 was the cause of the failure, but was fixed in 2.3.1 (and 2.4.1). As mentioned in previous post, I need to use 2.3.1 because of Ubuntu 16.04. and libssl1.0

Final fixes that made everything work:

  • Specified mender-client version in mender-convert: MENDER_CLIENT_VERSION="2.3.1"
  • Some fixes to MENDER_LOAD_KERNEL_AND_FDT to ensure reboot (mender_try_to_recover) on dtb/kernal load failure.

The tinal MENDER_LOAD_KERNEL_AND_FDT looks like this:

# define MENDER_LOAD_KERNEL_AND_FDT                                                         \
    "if test \"${fdt_addr_r}\" != \"\"; then \n"                                            \
    "    if load ${mender_uboot_root} ${fdt_addr_r} /boot/${mender_dtb_name}; then \n"      \
    "        fdt addr ${fdt_addr_r}; \n"                                                    \
    "        fdt set mmc${boot_mmc} boot_device <1>; \n"                                    \
    "    else \n"                                                                           \
    "        echo \"Unable to load /boot/${mender_dtb_name} from ${mender_uboot_root}\" \n" \
    "        run mender_try_to_recover; \n"                                                 \
    "    fi; \n"                                                                            \
    "else \n"                                                                               \
    "    run mender_try_to_recover; \n"                                                     \
    "fi; \n"                                                                                \
    "if load ${mender_uboot_root} ${kernel_addr_r} /boot/${mender_kernel_name}; then; \n"   \
    "else \n"                                                                               \
    "    echo \"Unable to load /boot/${mender_kernel_name} from ${mender_uboot_root}\" \n"  \
    "    run mender_try_to_recover; \n"                                                     \
    "    reset; \n"                                                                         \
    "fi; \n"
#endif
1 Like

Glad to hear everything worked out @johan

Would gladly take contributions to integrate this in our upstream tools if you are interested. We also have a section for sharing such integrations:

https://hub.mender.io/c/board-integrations

Yes, was thinking about this. I am open to questions now that I got things to work. I haven’t had the time to do a clean write-up, but might do that when I have passed some deadlines at work…