Mender from scratch

This post is primarily for education purposes to provide you with detailed insights in how Mender is installed on a system and can be useful when porting Mender to a new build system.

The official integrations of Mender are meta-mender (Yocto/OE-core) and Debian family.

Buildroot is community supported.

This will be a quite technical blog post, and a certain knowledge of embedded builds is assumed.

As Mender is an evolving project, this post might get out date from time to time but we will do our best to keep it up to date.

Partition layout

In order to create the proper image for the memory card using Mender, we require a particular partition layout, and the build system should make sure this is followed when creating the image.

It is possible to use other layouts, but the four main pieces, the boot partition, the two rootfs partitions, and the persistent data partition, all need to be there. Some aspects of the configuration may also change, such as references to the individual partitions.

Boot partition is optional if your specific board does not require it

The exact expected partition layout is:

  • Partition 1: Boot partition, mounted on /uboot (optional)
  • Partition 2: Rootfs A partition, mounted on /
  • Partition 3: Rootfs B partition, inactive, mounted on /
  • Partition 4: Data partition (persistent data), mounted on /data

Partition layout

NOTE This post makes assumptions where to mount certain partitions and this is mainly for consistency. You should be able to adopt this to your needs.

The Mender program

Compiling Mender

Compiling or cross-compiling Mender is covered in our official documentation,

Configuring Mender

Mender comes with several configuration files that are important for it to function. This should appear in the rootfs filesystem in the locations specified below.

/etc/mender/mender.conf

An example of this file is pasted below,

{
    "InventoryPollIntervalSeconds": 5,
    "RetryPollIntervalSeconds": 30,
    "RootfsPartA": "/dev/mmcblk0p2",
    "RootfsPartB": "/dev/mmcblk0p3",
    "ServerCertificate": "/etc/mender/server.crt",
    "ServerURL": "https://hosted.mender.io",
    "TenantToken": "",
    "UpdatePollIntervalSeconds": 5

The configuration options are documented in our official documentation,

It is also an good idea to read trough the recommendations for choosing polling intervals,

/etc/mender/server.crt

This is only needed if you are using self-signed certificates or using a CA that is not included in the ca-certificates bundle.

We provide a demo certificate for usage with the Mender demo environment. Do not use this for production, as the private key is freely available in our source code repositories.

If you are setting up an Mender on-premise server, a server.crt file will be generated if you follow the Certification and Keys section.

/usr/share/mender/identity/mender-device-identity

This script is called by the Mender client to generate a unique device identity.

This script shall exit with non 0 status code on errors. In this case the agent
will discard any output the script may have produced.

This script shall output identity data in key=value format, one entry per line. Example

$ ./mender-device-identity
mac=de:ad:ca:fe:00:01
cpuid=1112233

Multiple values are ok, but the important part is that these values do not change over time as that would result in a new device identity.

An example script that outputs the MAC address of the first valid network interface can be found in the the Mender client source.

Common stable sources for device identity:

  • MAC address of a specific network interface (though some devices have randomized MAC which makes it unsuitable as an source for device identity)
  • Custom serial number which is provisioned during production
  • CID register of eMMC

/usr/share/mender/inventory/*

The inventory scripts are used to collect data to send to the Mender server, the data will be sent periodically and how often depends on the UpdatePollIntervalSeconds configuration option. You can have multiple scripts in the /usr/share/mender/inventory/ directory.

The Mender server only stores this data without further processing. This data can later be used to query devices using the server API based on inventory information.

The Mender server GUI will display the inventory data, e.g

This script shall output inventory data in key=value format, one entry per line. Example:

$ ./mender-inventory-hostinfo 
cpu_model=ARMv7 Processor rev 1 (v7l)
mem_total_kB=102847

Example inventory scripts can be found in the Mender client source

Installing Mender

Mender should be installed into /usr/bin on the target device or some other location that is typically part of PATH.

There are some additional files that need to be configured or added to get a fully functional system with Mender.

/etc/fstab

It is important that the root filesystem is mounted with a similar entry to below,

/dev/root            /                    auto       defaults              1  1

Above is needed because we will be switching between two different parts that act as the root filesystem and then the path to the device can not be hard-coded. /dev/root is created based on the root= Linux kernel argument.

In addition to common entries such as the root filesystem, the /etc/fstab file needs to contain the following entries:

/dev/mmcblk0p1   /uboot               auto       defaults,sync,auto    0  0
/dev/mmcblk0p5   /data                auto       defaults,auto         0  0

The two partitions need to match the partitions in your layout that contain the U-Boot environment file and the ext3/ext4 data partition, respectively. See the official partition layout for more information.

Note that if you are not using a dedicated boot partition, the /uboot entry may not be necessary.

/etc/mender/artifact_info

In /etc/mender/artifact_info there should be a single line:

artifact_name=<artifact_name>

<artifact_name> is name of the image or update that. This is what the device will report that it is running, and different images must have different names

Example:

$ cat /etc/mender/artifact_info 
artifact_name=release-1

/var/lib/mender/mender -> /data/mender

In this post we have assumed that the we mount the fourth partition which is the persistent data partition to /data.

The default directory where Mender will store state is /var/lib/mender, but to be able to store state across updates you typically create a symlink, /var/lib/mender -> /data/mender, making sure that anything that is written to /var/lib/mender is persistent across updates.

/var/lib/mender/device_type

NOTE! This assumes that the symlink /var/lib/mender/mender -> /data/mender is in place so this path could also be represented as /data/mender/device_type.

In /var/lib/mender/device_type there should be a single line:

device_type=<device_type>

<device_type> should be a string that uniquely identifies a group of devices.

systemd

There is an example systemd service for Mender which can be found our Yocto layer which you typically install under /lib/systemd/system and to enable it you need to run:

 systemctl enable meneder && systemctl start mender

SysVinit

We do not provide an official SysVinit script for Mender but there is one in the Buildroot source code.

GRUB

More information coming soon…

GRUB does not require any patches to the actual source code and most of the integration is in the grub.cfg file. You can take a look at this repository for more information,

U-Boot

Now we’re getting into the advanced stuff! U-Boot is one of the trickiest parts to integrate in the build. We need to patch U-Boot to support Mender’s automatic partition selection and rollback features, as well as provide some auxiliary files.

NOTE! We have an U-boot fork for Raspberry Pi boards where all the necessary patches have been applied and is a good reference as well. This U-boot fork is also used by our mender-convert tool

Denx, the company behind U-boot

U-Boot configuration

U-Boot needs to store its state, which is typically referred to as U-boot “environment”, to persistent storage during upgrade procedures. To accomplish this you need to add,

CONFIG_ENV_IS_IN_MMC=y

to your board defconfig file. Note if you are using device with RAW NAND then the equivalent configuration is,

CONFIG_ENV_IS_IN_NAND=y

You need to enable the following features in a board specific header file,

 #define CONFIG_BOOTCOUNT_LIMIT
 #define CONFIG_BOOTCOUNT_ENV

Above features are utilized to perform an automatic roll-back, a simplified explanation is that it will call alt_bootcmd if bootcount exceeds bootlimit.

Any occurrences of CONFIG_ENV_OFFSET should be removed from board specific header or defconfig files, these will be defined based on values in config_mender_defines.h which is described below.

Any occurrences *_ENV_INTERFACE, *_ENV_DEVICE, *_ENV_PART and *_ENV_DEVICE_AND_PART should be removed from board specific header or defconfig files. These will be defined based on config_mender_defines.h which is described below.

config_mender_defines.h

config_mender_defines.h is a special file used by Mender during compilation to provide configuration values to U-Boot supplied by the build system. This file is auto-generated during a Yocto build, but we need to generate it ourselves for our third party build system.

Below I have put an example of what this looks like when it’s filled out (this example is from Raspberry Pi 3 integration):

#ifndef HEADER_CONFIG_MENDER_DEFINES_H
#define HEADER_CONFIG_MENDER_DEFINES_H

/* Shell variables */
#define MENDER_BOOT_PART_NUMBER 1
#define MENDER_BOOT_PART_NUMBER_HEX 1
#define MENDER_ROOTFS_PART_A_NUMBER 2
#define MENDER_ROOTFS_PART_A_NUMBER_HEX 2
#define MENDER_ROOTFS_PART_B_NUMBER 3
#define MENDER_ROOTFS_PART_B_NUMBER_HEX 3
#define MENDER_UBOOT_STORAGE_INTERFACE "mmc"
#define MENDER_UBOOT_STORAGE_DEVICE 0

/* BB variables. */
#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 0x400000
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 0x800000

/* For sanity checks. */
#define MENDER_BOOTENV_SIZE 0x4000

#define MENDER_UBOOT_PRE_SETUP_COMMANDS ""
#define MENDER_UBOOT_POST_SETUP_COMMANDS ""

The six first entries are simply partition numbers, and the two entries containing UBOOT describe how to refer to the storage device using U-Boot’s syntax.

In this case we refer to the MultiMediaCard interface, or MMC, and we select the first (0’th) such slot. The MENDER_STORAGE_DEVICE_BASE is the string which the Linux kernel uses to refer to the MMC storage, excluding the partition number.

The MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET variables define where to store the U-boot environment on the storage medium. Take note of these values as they represent an address of on the storage medium, meaning that the first partition should be after these addresses, otherwise you can end up with conflicts. In the example presented the environment is aligned to 4MB, meaning that the first partition should start at a 12MB offset.

The MENDER_BOOTENV_SIZE is the size of the U-Boot environment and should match CONFIG_ENV_SIZE. This might feel like an extra unnecessary step and it is in this context, but our Yocto integration uses these variables to perform sanity checks and since we will re-use the same patches that are in Yocto we need to add this.

The PRE_SETUP_COMMANDS/POST_SETUP_COMMANDS are also residue from Yocto, but they need to be defined or we will get an build error.

Patches

There are a couple of board independent patches that need to be applied to U-boot, and they are:

  • 0002-Generic-boot-code-for-Mender.patch
  • 0003-Integration-of-Mender-boot-code-into-U-Boot.patch
  • 0006-env-Kconfig-Add-descriptions-so-environment-options-.patch

These can be found in meta-mender layer. Note that these patches typically apply cleanly to a more recent U-boot version and you might need to adopt the patches if you are using an older version.

The patch containing all the boot logic required for Mender can be found in 0002-Generic-boot-code-for-Mender.patch, if you are curious and want to read in on the source code.

In addition to the cross platform patches you’ll also need patches specific to your board. We have documented what is needed in the board specific patch in our documentation, Manual U-boot integration

We have many reference integrations in meta-mender-community where you can find board specific patches, there might be one for your specific board or at least it can be a source of inspiration.

Compiling

You should now be ready to compile U-Boot! This process is covered under “Building the Software” inside the README file in the U-Boot source tree, so I will not go through it here. How to integrate the resulting boot loader image is board specific, and needs to follow the procedure for the board you are building for.

U-boot environment utils

In addition to the U-Boot boot loader image, some user space Linux tools are also needed, to be able to modify the U-boot environment from user-space. These tools are part of the U-boot source code and can be found at tools/env/ and to compile the you should use the same U-boot source with the Mender patches applied and run,

make env

The resulting binary, tools/env/fw_printenv , should be installed in the rootfs image twice, once as /sbin/fw_printenv and once as /sbin/fw_setenv . The tool uses the binary name as an indication of which operation to perform.

The Mender client will invoke these tools.

/etc/fw_env.config

In order for the U-boot environment tools to find the environment, we need to provide a configuration file called /etc/fw_env.config . This file should contain information of where your U-boot environment is located and its size, this must match the configuration that you have built-in in U-boot, an example is:

/dev/mmcblk0    0x400000    0x4000
/dev/mmcblk0    0x800000    0x4000

This matches the values that are set in config_mender_defines.h that was described earlier,

#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 0x400000
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 0x800000
#define MENDER_BOOTENV_SIZE 0x4000

Mender Artifact

Mender would not be much good unless we could also provide updates to the image we just made. To do this we need the mender-artifact tool.

Instructions on how to compile the mender-artifact tool can be found here,

You can also download a prebuilt mender-artifact (x86_64) Linux binary here.

The primary input to the mender-artifact tool is the filesystem image that comprises the rootfs. In addition we need to provide some metadata about the image. Here is an example invocation of the tool:

mender-artifact write rootfs-image \
    -n <artifact_name> \
    -t <compatible_devices> \
    -u <rootfs image> \
    -o <name-of-update>.mender
  • <artifact_name> - The Mender Artifact name and this should match the content of /etc/mender/artifact_info.

  • <compatible_devices> - List of compatible devices (comma separated) and this will be later compared to the content of /var/lib/mender/device_type

  • <rootfs image> - Path to the root filesystem image, e.g an ext4 image.

  • <name-of-update> - Path to the output Mender Artifact. It must have a .mender suffix to be able to upload it to the Mender Server.

4 Likes

In the context of GRUB:
As far as I understand, Mender client controls boot loading (by setting variables, such as “upgrade_available”) using fw_setenv which access /boot/ folder. It cannot be in two copies in the two A/B rootfs partitions, without mounting also the inactive partition (that would “block” writing to the inactive partition using “dd” command). So, it is in a separate partition.
How the boot loader read the /boot folder contents? Is the separate partition mounted by fstab? How GRUB identify it? Does it use the “boot” flag indication and GRUB chooses the actual partition to boot from based on the fw_setenv variables?

Your understanding is accurate.

The GRUB environment resides on a separate partition that is mounted at boot at /boot/efi using an fstab entry.

GRUB identifies the boot partition using simple indexing where first partition is assumed to be the boot partition containing the relevant content.

You can find the source on how this is all done in the following repository (both the grub.cfg and the userspace tools to modify the environment files):

Hi,

we have 2 different NXP imx7 based projects, both are using mender. And we have the same issue with both of them:
When booting the devices initially, u-boot writes the environment to mmc. u-boot and uboot-env are outside a partition, directly in memory.
But it looks like the offset of the first partition ( in our case /uboot which is not used) is overwritten by the uboot-env.
This causes mount problems during linux boot as the partition is corrupted.

My workaround to patch this behaviour: adding parameter --offset 32768 to line meta-mender/mender-part-images.bbclass at dunfell · mendersoftware/meta-mender · GitHub

With this patch the partition layout changes from (output of fdisk -l)

Disk /dev/mmcblk0: 7.29 GiB, 7822376960 bytes, 15278080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x33c68b61

Device         Boot   Start     End Sectors  Size Id Type
/dev/mmcblk0p1 *      32768   65535   32768   16M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        65536 1064959  999424  488M 83 Linux
/dev/mmcblk0p3      1064960 2064383  999424  488M 83 Linux
/dev/mmcblk0p4      2064384 4161535 2097152    1G 83 Linux

to

Disk /dev/mmcblk0: 7.29 GiB, 7822376960 bytes, 15278080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc5ed08c5

Device         Boot   Start     End Sectors  Size Id Type
/dev/mmcblk0p1 *      65536   98303   32768   16M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        98304 1097727  999424  488M 83 Linux
/dev/mmcblk0p3      1097728 2097151  999424  488M 83 Linux
/dev/mmcblk0p4      2097152 4194303 2097152    1G 83 Linux

As you can see, the starting point of the first partition is moved back.

Do you have any idea whats going wrong here? How to avoid conflicts between uboot-env and first partition?

I’m not sure but the offsets can indeed be tricky. @kacf or @oleorhagen may be able to provide more guidance.

But, if you are not using the FAT32 partition then you can set MENDER_BOOT_PART_SIZE_MB to 0 and it will be removed. Maybe that will help here.

Nice find. This could have been a fun one, considering that U-Boot does not write to both environment every time.

And I’m sorry, I’m not too familiar with how we write the disk images.

But as you’ve noticed your boot partition ends up on the 10MiB boundary, and hence collides with the redundant U-Boot env.

What the proper way to fix this, I’m not sure of unfortunately, will defer this to @kacf

I did this. But this broke my boot completely as then the uboot env overwrites the boot partition. So, same error, only with other partition.

It should not be necessary to adjust the table offsets manually, though I admit I don’t know the exact inner workings of wic. Take a look at this code. This is where the environment is added to the raw image (technically the environment is null bytes at this point, but it’s the size of the file which matters). This file comes from the U-Boot recipe, and is supposed to make wic aware of the size, and make sure that the first partition is not overlapping with it. Maybe there is some discrepancy here, and the size of this file doesn’t match BOOTENV_SIZE in the U-Boot recipe, for example. I encourage you to debug this area a bit.

We have different partition layout, How we can integrate to mender.

short-description: Create SD card image for HiFive Unleashed development board with U-Boot SPL and OpenSBI (FW_DYNAMIC)

part --source rawcopy --sourceparams=“file=u-boot-spl.bin” --ondisk mmcblk0 --fixed-size 1M --align 17 --part-type 5b193300-fc78-40cd-8002-e86c45580b47
part --source rawcopy --sourceparams=“file=u-boot.itb” --ondisk mmcblk0 --fixed-size 4M --align 1 --part-type 2e54b353-1271-4842-806f-e436d6af6985
part /boot --source bootimg-partition --sourceparams=“loader=u-boot” --ondisk mmcblk0 --fstype=vfat --label boot --active --size=100M --align 4096
part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --label root --align 4096 --size 1G

bootloader --ptable gpt --configfile=“freedom-u540-extlinux.conf”