Robust delta update rootfs

Update Modules
1

Description

An Update Module to install binary delta rootfs updates on a device.

This Update module is available with a commercial Mender plan. Please refer to the Mender.io product page for more information about commercial plans, and sign up for a free trial for Mender Professional here.

Example use-cases:

  • Perform a rootfs update using far less bandwidth than a traditional full-image update

Specification

Specification
Module name mender-binary-delta
Supports rollback yes
Requires reboot yes
Artifact generation script yes
Full operating system updater yes
Source code Closed source
Maintainer Northern.tech

Prerequisites

  • An existing Yocto environment with a Mender board integration working to be able to deploy “rootfs-image” updates
  • Ability to use read only root filesystem in your build (how to enable it is shown below, Delta updates will not work without it)
  • A Mender Professional or Mender Enterprise account. Sign up for a free trial for Mender Professional.
  • mender-artifact version 3.4.0 or newer on your development machine. You can download the pre-built binary from here
  • Mender client version 2.3.0 or newer

You can enable Mender client version 2.x.x in Yocto by adding the following in your local.conf:

PREFERRED_VERSION_pn-mender = "2.%"
PREFERRED_VERSION_pn-mender-artifact = "3.%"
PREFERRED_VERSION_pn-mender-artifact-native = "3.%"
  • mender-binary-delta requires “read-only” rootfs

If you do not already have “read-only” mode on your rootfs images you can enable this with adding the following in e.g local.conf

IMAGE_FEATURES_append = " read-only-rootfs"

It is a good idea to enable this first and test that your device is still operating as intended, before proceeding in testing mender-binary-delta.

Download mender-binary-delta

NB: The following steps are for Kirkstone and newer. Please see Pre-kirkstone section for required changes to these steps.

Change directory to $HOME:

cd ${HOME}

The latest version of mender-binary-delta is 1.5.0. If you are using hosted Mender (Mender Professional or hosted Mender Enterprise), download the mender-binary-delta archive with the following command:

HOSTED_MENDER_EMAIL="myusername@example.com"
curl -u $HOSTED_MENDER_EMAIL -O https://downloads.customer.mender.io/content/hosted/mender-binary-delta/1.5.0/mender-binary-delta-1.5.0.tar.xz

On the other hand, if you are using on-premise Mender Enterprise, download using the following command:

MENDER_ENTERPRISE_EMAIL="myusername@example.com"
curl -u $MENDER_ENTERPRISE_EMAIL -O https://downloads.customer.mender.io/content/on-prem/mender-binary-delta/1.5.0/mender-binary-delta-1.5.0.tar.xz

If you need to access older version of mender-binary-delta, see Archived versions below.

Unpack mender-binary-delta-generator

The archive mender-binary-delta-1.5.0.tar.xz contains the binaries needed to generate and apply deltas.

Unpack the mender-binary-delta-1.5.0.tar.xz in your home directory:

tar -xvf mender-binary-delta-1.5.0.tar.xz

The content of the archive should be:

.
├── aarch64
│   ├── mender-binary-delta
│   └── mender-binary-delta-generator
├── arm
│   ├── mender-binary-delta
│   └── mender-binary-delta-generator
└── x86_64
    ├── mender-binary-delta
    └── mender-binary-delta-generator

As you might have guessed, we have prepared binaries for the armhf and x86_64 architectures. If your platform architecture does not match the provided binaries, please contact us and we should be able to compile binaries for your target.

The mender-binary-delta binary is intended to be installed on the device on which you intend to perform delta updates on. However, the Yocto integration works directly with the .tar.xz file, so we don’t need the binary here.

The mender-binary-delta-generator is intended to be used on your development machine to generate deltas between two Mender Artifacts. You can use the x86_64 mender-binary-delta-generator even if your device is armhf, the generator tool is architecture agnostic.

We only need the generator, so copy it to /usr/bin:

sudo cp mender-binary-delta-1.5.0/x86_64/mender-binary-delta-generator /usr/bin

Then delete the rest of the unpacked files:

rm -rf mender-binary-delta-1.5.0

Integrating mender-binary-delta to an existing Yocto environment

It is assumed that you already have a Yocto Project environment configured using meta-mender and that you are in the build directory.

Add meta-mender-commercial layer to your Yocto environment:

bitbake-layers add-layer ../sources/meta-mender/meta-mender-commercial

Add the following your local.conf to include mender-binary-delta in your build:

cat <<EOF >> conf/local.conf
# Customizations for Mender delta-update support

# This will not impact your image if you already have this enabled
IMAGE_FEATURES:append = " read-only-rootfs"

IMAGE_INSTALL:append = " mender-binary-delta"
LICENSE_FLAGS_ACCEPTED:append = " commercial_mender-yocto-layer-license"
SRC_URI:pn-mender-binary-delta = "file://${HOME}/mender-binary-delta-1.5.0.tar.xz"

EOF

Preparing images and artifacts

For the purpose of this tutorial you can set the MENDER_ARTIFACT_NAME to the following:

MENDER_ARTIFACT_NAME = "release-v.1.0"

Set helper variables (update with the appropriate values used in your environment):

TARGET_IMAGE="core-image-base"
MACHINE="raspberrypi4"

Build an image (target image might differ in your environment):

bitbake ${TARGET_IMAGE}

Use the output of the build to provision your device with e.g with the provided disk image (e.g .sdimg) file. The result should be that you have a device that has booted and has the following file installed:

# ls /usr/share/mender/modules/v3/mender-binary-delta -alh
-rwxr-xr-x    1 root     root       98.2K Aug 19 08:52 /usr/share/mender/modules/v3/mender-binary-delta

Please verify this before proceeding.

Save the generated Mender Artifact for later (will be used to generate a delta):

cp tmp/deploy/images/${MACHINE}/${TARGET_IMAGE}-${MACHINE}.mender ${HOME}/release-v.1.0.mender

Generate a second Mender Artifact which can be used to test deploying delta updates. Start of with making a change to the image content, such as adding an ssh server (for diagnostics purposes):

cat <<EOF >> conf/local.conf

IMAGE_INSTALL_append = " openssh"

EOF

Update the MENDER_ARTIFACT_NAME to the following:

MENDER_ARTIFACT_NAME = "release-v.2.0"

Build the new image (target image might differ in your environment):

bitbake ${TARGET_IMAGE}

Save the generated Mender Artifact for later (will be used to generate a delta):

cp tmp/deploy/images/${MACHINE}/${TARGET_IMAGE}-${MACHINE}.mender ${HOME}/release-v.2.0.mender

Generating delta Artifacts

This tutorial assumes that Mender Enterprise is being used. If not, then dependencies need to be tracked manually, which is not recommended. See the troubleshooting section for ways to deal with this.

Generate delta for v1.0 -> v2.0 update path:

mender-binary-delta-generator -o v2.0-deltafrom-v1.0.mender release-v.1.0.mender release-v.2.0.mender

Generate delta for v2.0 -> v1.0 update path:

mender-binary-delta-generator -o v1.0-deltafrom-v2.0.mender release-v.2.0.mender release-v.1.0.mender

Using the delta Mender Artifacts

We should have now have the Mender Artifacts ready and you can upload them to the Mender server. The uploaded files should be:

  • release-v.1.0.mender (rootfs-image)
  • release-v.2.0.mender (rootfs-image)
  • v1.0-deltafrom-v2.0.mender (mender-binary-delta)
  • v2.0-deltafrom-v1.0.mender (mender-binary-delta)

After having uploaded these four Artifacts, you should have two Releases in the Mender UI. This is because the rootfs-image Artifact and the corresponding mender-binary-delta Artifact are referring to the same software.

If you have followed this tutorial your provisioned device should be running release-v1.0 which was provisioned using a disk image (e.g sdimg).

Deploy delta updates

Start off with deploying release-v.2.0.mender to the device, and once that is completed you can deploy release-v.1.0.mender. Note that at this point there is no distinction between deploying a full rootfs-image Artifact and a mender-binary-delta Artifact. Mender keeps track of dependencies automatically and selects the correct one. More formally, it selects the smallest Artifact which satisfies the dependencies, but will select the bigger rootfs-image Artifact if the device has an unknown delta base.

If you want to verify that the mender-binary-delta Artifact is indeed being used, delete the rootfs-image from the one of the Releases in the UI, and then deploy that release.

Pre-kirkstone Yocto branches

If you are using a Yocto branch older than kirkstone, such as dunfell or older, you need slightly altered steps to use mender-binary-delta. Two things are required:

  1. The .tar.xz file must be unpacked before it can be used, so make sure you keep it around, and don’t delete the unpacked files.

  2. The build configuration is a little different. See the next section for details.

Set up the Yocto environment

The environment is almost the same as for kirkstone, but you need to make these changes to the configuration:

  1. Remove the SRC_URI line that refers to the downloaded .tar.xz file.

  2. Add a line like below, and make sure the path points to where the .tar.xz file was unpacked:

    FILESEXTRAPATHS_prepend_pn-mender-binary-delta := "${HOME}/mender-binary-delta-1.5.0/:"

    NOTE! The trailing : in FILESEXTRAPATHS_prepend_pn-mender-binary-delta is intentional and important to have in place.

  3. The Yocto Project has renamed the variable LICENSE_FLAGS_WHITELIST to LICENSE_FLAGS_ACCEPTED in kirkstone and later branches, so make sure to use the former variant with dunfell and older branches.

  4. The Yocto Project has also changed the override syntax so, you need to make sure the configuration is using the old syntax with dunfell and older branches. More information about this change can be found in the Yocto Project Manual.

Troubleshooting

Error message: Artifact dependency "rootfs_image_checksum" not satisfied by currently installed artifact

This message is usually because of one of these reasons:

  • You are using mender-binary-delta version 1.1.1 or earlier together with Mender Server 2.6.0 or later, or together with mender-artifact 3.5.0 or later. For this to work correctly you need to upgrade mender-binary-delta to 1.2.0 or later.

  • You are using a non-Enterprise version of Mender, and there are several mender-binary-delta and/or rootfs-image Artifacts with the same name (grouped under the same Release in the UI). In non-Enterprise versions of Mender, there is no automatic dependency tracking, so each Artifact needs to have a different name, even if the only difference between two Artifacts is the delta base.

  • You are trying to deploy a mender-binary-delta Artifact onto a device which does not have the delta base checksum that the Artifact was made for. If this is the case, you need to make a new mender-binary-delta Artifact based on the Artifact which is currently installed on the device, and install that instead.

  • It can also be an indication that the very first rootfs-image Artifact has not been deployed yet. If so, you need to install a rootfs-image Artifact to the device first, and make sure that the next mender-binary-delta Artifact uses this Artifact as a base. This only needs to be done once for each device. This requirement may be lifted in the future.

Deployment finishes with “No artifact” without updating the device

This usually happens for one of three reasons:

  1. The device has never received a rootfs-image Artifact update.

  2. The Release which is being deployed only has mender-binary-delta Artifacts uploaded, and none of them match the delta base on the device.

  3. The client is version 2.2 or older, and therefore it doesn’t know how to ask for a delta Artifact.

These are the solutions for each case:

  1. Upload the corresponding rootfs-image Artifact, which will be used in the deployment instead of the mender-binary-delta Artifact. This only needs to be done once for each device. This requirement may be lifted in the future.

  2. Upload a mender-binary-delta Artifact using the currently installed Artifact of the device as a base. Alternatively, solution number 1 is a valid solution for case number 2 as well, but it will use more bandwidth.

  3. Solution number 3 is the same as for the error message type_info depends values not yet supported, described below.

Error message: type_info depends values not yet supported

This can happen if the client is too old to support the dependency tracking information present in mender-binary-delta 1.1 and later. If you encounter this you can specify the --make-transitional argument when making the delta Artifact using mender-binary-delta-generator. The resulting Artifact will not have any dependency information associated with it, and you must give it a different name from the original Artifact it was made from, using the -n argument. For example:

./mender-binary-delta-generator -o v2.0-deltafrom-v1.0.mender --make-transitional -n v2.0-deltafrom-v1.0 release-v.1.0.mender release-v.2.0.mender

Note that if you use this, the original rootfs-image Artifact must have been created with the --no-checksum-provide argument (without this you would not have been able to install the original rootfs-image Artifact either). See the bullet point about --no-checksum-provide in the changelog for mender-artifact 3.3.0 for more information.

Error messages when trying to use mender-binary-delta with UBIFS

UBIFS support requires at least mender-binary-delta version 1.5.0. In addition, either mender-flash or ubiupdatevol need to be installed on the device, and the FlashTool configuration setting must be set to one of them. On Yocto, the you can use the MENDER_FLASH_TOOL variables to control this (defaults to mender-flash).

Archived versions

Older versions of mender-binary-delta are listed here. Use the links together with the download instructions described earlier to get them.

Hosted Mender On-prem Mender Enterprise
1.4.1 1.4.1
1.4.0 1.4.0
1.3.0 1.3.0
1.2.1 1.2.1
1.2.0 1.2.0
1.1.1 1.1.1
1.1.0 1.1.0
1.0.1 1.0.1
1.0.0 1.0.0

For information about what has changed in each release, see the Mender changelogs section and look for “mender-binary-delta”.

5 Likes