Error during delta update

General Discussions
1

Hi all
I have created mender patched os using mender-convert tool from raspbian os. I have successfully done rootfs update. Delta artifact have been created but during deployment, I am getting error: “Artifact Payload type ‘mender-binary-delta’ is not supported by this Mender Client. Make sure the Update Module is installed on the device”
To create delta artifact I used, Create a Delta update Artifact | Mender documentation

Here’s the log:

2021-08-27 11:32:24 +0000 UTC info: Running Mender client version: 3.0.0
2021-08-27 11:32:24 +0000 UTC warning: RetryPollIntervalSeconds is not defined
2021-08-27 11:32:26 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2021-08-27 11:32:26 +0000 UTC info: No public key was provided for authenticating the artifact
2021-08-27 11:32:26 +0000 UTC error: Fetching Artifact headers failed: installer: failed to read Artifact: readHeaderV3: handleHeaderReads: Artifact Payload type ‘mender-binary-delta’ is not supported by this Mender Client. Make sure the Update Module is installed on the device
2021-08-27 11:32:26 +0000 UTC info: State transition: update-store [Download_Enter] → cleanup [Error]
2021-08-27 11:32:26 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

I am not using Yocto in entire process. Where I went wrong?

Thanks

2

I am using the same client which is used for system update. Is there any other client available which is required for delta update?

3

It means the mender-binary-delta module is not installed on the device.

Mender-binary-delta can’t ordinarily be used with an image from mender-convert. It doesn’t mean it can’t work, but we don’t support it out of the box, and you need to take some care and make sure the image is prepared for running in read-only mode.

I would strongly recommend moving to a Yocto based build if you are planning to use mender-binary-delta. Then install mender-binary-delta following these instructions.

4

Why delta image is so huge?

To create delta file, I have used mender-convert tool to create rootfs mender update file from base raspbian os. After that, I have added a couple of small packages such as nmap and 1-2 small packages and created rootfs mender update file. Now using mender-binary-delta generator, I created delta file between both rootfs mender update files.
My question is why delta image size is 1.23GB for such a small change in rootfs update file? The entire rootfs for base image is 1,54,42,48,832 bytes and updated image is 1,58,58,98,496 bytes!

please respond!

5

mender-client have started the delta image download after placeing mender-binary-delta file at location
/usr/share/mender/modules/v3/, as mention in your documentation.
It got stuck at 69% and didn’t moved further.
I am now working towards yocto build with ur documentation

6

How to get mender-binary-delta generator for different architectures such as RISCV? Is source code available for porting?
Please respond

7

You will need to provide some logs if we are going to have a chance to help you. Can you for example post the client logs from journalctl -u mender-client?

Mender-binary-delta is closed source, and RISCV is not available at the moment, but this is something we could consider building if there is customer interest. What are your needs in this area?

If you are not comfortable discussing it in an open forum, feel free to contact us using our contact form instead.

8

For RISCV, we are looking for seamless OTA updates, including application, rootfs and delta update. We are also working towards porting mender for RISCV and also testing mender on pi.
For RISCV whether mender-binary-delta would be provided by mender directly?

9

HI
I am getting following error while building mender via yocto. I have tried to change the values of variables MENDER_RESERVED_SPACE_BOOTLOADER_DATA and MENDER_PARTITION_ALIGNMENT frim local.conf but still it does not get resolved. Please check

ERROR: u-boot-1_2020.01-r0 do_provide_mender_defines: BOOTENV_SIZE (0x4000) is too big to fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA (12288) with proper alignment. Please either: 1. Increase MENDER_RESERVED_SPACE_BOOTLOADER_DATA manually and make sure it is an even multiple of MENDER_PARTITION_ALIGNMENT (2048). -or- 2. Decrease BOOTENV_SIZE in the U-Boot recipe so that it can fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA. Please see Yocto Project build | Mender documentation for more information.
ERROR: u-boot-1_2020.01-r0 do_provide_mender_defines: Execution of ‘/home/abhishek/Documents/mender-raspberrypi/build/tmp/work/raspberrypi4-poky-linux-gnueabi/u-boot/1_2020.01-r0/temp/run.do_provide_mender_defines.12324’ failed with exit code 1:

  • true
  • [ ! -e /home/abhishek/Documents/mender-raspberrypi/build/tmp/work/raspberrypi4-poky-linux-gnueabi/u-boot/1_2020.01-r0/fw_env.config.default ]
  • [ 0 -eq 0 ]
  • expr 12288 % ( 2048 * 2 )
  • [ 0 -ne 0 ]
  • [ 32768 -gt 12288 ]
  • bbfatal BOOTENV_SIZE (0x4000) is too big to fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA (12288) with proper alignment. Please either: 1. Increase MENDER_RESERVED_SPACE_BOOTLOADER_DATA manually and make sure it is an even multiple of MENDER_PARTITION_ALIGNMENT (2048). -or- 2. Decrease BOOTENV_SIZE in the U-Boot recipe so that it can fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA. Please see Yocto Project build | Mender documentation for more information.
  • [ -p /home/abhishek/Documents/mender-raspberrypi/build/tmp/work/raspberrypi4-poky-linux-gnueabi/u-boot/1_2020.01-r0/temp/fifo.12324 ]
  • printf %b\0 bbfatal BOOTENV_SIZE (0x4000) is too big to fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA (12288) with proper alignment. Please either: 1. Increase MENDER_RESERVED_SPACE_BOOTLOADER_DATA manually and make sure it is an even multiple of MENDER_PARTITION_ALIGNMENT (2048). -or- 2. Decrease BOOTENV_SIZE in the U-Boot recipe so that it can fit two copies inside MENDER_RESERVED_SPACE_BOOTLOADER_DATA. Please see Yocto Project build | Mender documentation for more information.
  • exit 1
  • bb_exit_handler
  • ret=1
  • echo WARNING: exit code 1 from a shell command.
    WARNING: exit code 1 from a shell command.
  • exit 1

ERROR: Logfile of failure stored in: /home/abhishek/Documents/mender-raspberrypi/build/tmp/work/raspberrypi4-poky-linux-gnueabi/u-boot/1_2020.01-r0/temp/log.do_provide_mender_defines.12324
ERROR: Task (/home/abhishek/Documents/mender-raspberrypi/src/poky/meta/recipes-bsp/u-boot/u-boot_2020.01.bb:do_provide_mender_defines) failed with exit code ‘1’

10

With your current settings for MENDER_PARTITION_ALIGNMENT and BOOTENV_SIZE, MENDER_RESERVED_SPACE_BOOTLOADER_DATAneeds to be at least 32768. Try setting it to this.

11

Yes, Mender would build this from our closed source tree and offer it like the other platforms in binary form. But this needs to be negotiated first with one of our sales representatives, since it will be a feature request.

12

I have removed these variables from local.conf and added MENDER_BOOT_PART_SIZE_MB = “40”.This resolved the issue. Thanks

13

Hi
Even after the image is built successfully, the image is not not booting in pi. Why? After booting via HDMI (cannot ssh! ), I am getting recovery.elf error. Image attached

Screenshot_20210901-172149498
Screenshot_20210901-172149498974×557 130 KB

These are the extra additions in my local.conf file for mender

ENDER_ARTIFACT_NAME = “release-v.3.0”
INHERIT += “mender-full”
MACHINE = “raspberrypi4”
RPI_USE_U_BOOT = “1”
KERNEL_DEVICETREE = “bcm2711-rpi-4-b.dtb”
#MENDER_PARTITION_ALIGNMENT = “2048”
#MENDER_RESERVED_SPACE_BOOTLOADER_DATA = “12288”
MENDER_BOOT_PART_SIZE_MB = “40”
IMAGE_INSTALL_append = " kernel-image kernel-devicetree mender-binary-delta"
IMAGE_FSTYPES_remove += " rpi-sdimg"
TARGET_IMAGE=“core-image-base”
IMAGE_FEATURES_append = " read-only-rootfs"
LICENSE_FLAGS_WHITELIST_append = " commercial_mender-binary-delta"
FILESEXTRAPATHS_prepend_pn-mender-binary-delta := “${HOME}/mender-binary-delta-1.2.1/:”

MENDER_SERVER_URL = “https://hosted.mender.io”
MENDER_TENANT_TOKEN =xxxxxxxxxxxxxxxxx

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = “systemd”
DISTRO_FEATURES_BACKFILL_CONSIDERED = “sysvinit”
VIRTUAL-RUNTIME_initscripts = “”
IMAGE_FSTYPES = “ext4”

14

The local.conf settings look correct to me. Have you also included the meta-raspberrypi and meta-mender-raspberrypi layers?

15

Yes. All layers are added as required

bitbake-layers add-layer ${BASE}/src/meta-openembedded/meta-oe
bitbake-layers add-layer ${BASE}/src/meta-openembedded/meta-python
bitbake-layers add-layer ${BASE}/src/meta-openembedded/meta-multimedia
bitbake-layers add-layer ${BASE}/src/meta-openembedded/meta-networking
bitbake-layers add-layer ${BASE}/src/meta-raspberrypi
bitbake-layers add-layer ${BASE}/src/meta-mender/meta-mender-core
bitbake-layers add-layer ${BASE}/src/meta-mender/meta-mender-raspberrypi

16

What should be my next step now?

17

Can you paste your complete local.conf and bblayers.conf? I’ll try to test it on one of our build machines.

18

Sure

local.conf file

# This sets the default machine to be qemux86-64 if no other machine is selected:
MACHINE ??= "raspberrypi4"


DISTRO ?= "poky"

PACKAGE_CLASSES ?= "package_rpm"


# We default to enabling the debugging tweaks.
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"


USER_CLASSES ?= "buildstats image-mklibs image-prelink"


# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"


BB_DISKMON_DIRS ??= "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K \
    ABORT,/tmp,10M,1K"


CONF_VERSION = "1"

###################################################
###
### Configuration added by yocto-mender-rpi4.sh
###
###################################################
MENDER_ARTIFACT_NAME = "release-v.3.0"
INHERIT += "mender-full"
MACHINE = "raspberrypi4"
RPI_USE_U_BOOT = "1"
KERNEL_DEVICETREE = "bcm2711-rpi-4-b.dtb"
#MENDER_PARTITION_ALIGNMENT = "2048"
#MENDER_RESERVED_SPACE_BOOTLOADER_DATA = "12288"
MENDER_BOOT_PART_SIZE_MB = "40"
IMAGE_INSTALL_append = " kernel-image kernel-devicetree mender-binary-delta"
IMAGE_FSTYPES_remove += " rpi-sdimg"
TARGET_IMAGE="core-image-base"


IMAGE_FEATURES_append = " read-only-rootfs"
LICENSE_FLAGS_WHITELIST_append = " commercial_mender-binary-delta"
FILESEXTRAPATHS_prepend_pn-mender-binary-delta := "${HOME}/mender-binary-delta-1.2.1/:"
# Build for Hosted Mender
# To get your tenant token, log in to https://hosted.mender.io,
# click your email at the top right and then "My organization".
# Remember to remove the meta-mender-demo layer (if you have added it).
# We recommend Mender 1.4.0 and Yocto Project's rocko or later for Hosted Mender.
#
MENDER_SERVER_URL = "https://hosted.mender.io"
MENDER_TENANT_TOKEN = XXXXXXXXXXXXXXXXXXXXXXXXX

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
IMAGE_FSTYPES = "ext4"
19

bblayer.conf file

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  /home/abhishek/Documents/mender-raspberrypi/src/poky/meta \
  /home/abhishek/Documents/mender-raspberrypi/src/poky/meta-poky \
  /home/abhishek/Documents/mender-raspberrypi/src/poky/meta-yocto-bsp \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-openembedded/meta-oe \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-openembedded/meta-python \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-openembedded/meta-multimedia \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-openembedded/meta-networking \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-raspberrypi \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-mender/meta-mender-core \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-mender/meta-mender-raspberrypi \
  /home/abhishek/Documents/mender-raspberrypi/src/meta-mender/meta-mender-commercial \
  "
20

FYI @abhishek I edited your comments above to use the code markdown, so that @kacf or others can easily read and copy-paste!

1 Like