This post is primarily for education purposes to provide you with detailed insights in how Mender is installed on a system and can be useful when porting Mender to a new build system.
The official integrations of Mender are meta-mender (Yocto/OE-core) and Debian family.
Buildroot is community supported.
This will be a quite technical blog post, and a certain knowledge of embedded builds is assumed.
As Mender is an evolving project, this post might get out date from time to time but we will do our best to keep it up to date.
Partition layout
In order to create the proper image for the memory card using Mender, we require a particular partition layout, and the build system should make sure this is followed when creating the image.
It is possible to use other layouts, but the four main pieces, the boot partition, the two rootfs partitions, and the persistent data partition, all need to be there. Some aspects of the configuration may also change, such as references to the individual partitions.
Boot partition is optional if your specific board does not require it
The exact expected partition layout is:
- Partition 1: Boot partition, mounted on
/uboot(optional) - Partition 2: Rootfs A partition, mounted on
/ - Partition 3: Rootfs B partition, inactive, mounted on
/ - Partition 4: Data partition (persistent data), mounted on
/data
NOTE This post makes assumptions where to mount certain partitions and this is mainly for consistency. You should be able to adopt this to your needs.
The Mender program
Compiling Mender
Compiling or cross-compiling Mender is covered in our official documentation,
Configuring Mender
Mender comes with several configuration files that are important for it to function. This should appear in the rootfs filesystem in the locations specified below.
/etc/mender/mender.conf
An example of this file is pasted below,
{
"InventoryPollIntervalSeconds": 5,
"RetryPollIntervalSeconds": 30,
"RootfsPartA": "/dev/mmcblk0p2",
"RootfsPartB": "/dev/mmcblk0p3",
"ServerCertificate": "/etc/mender/server.crt",
"ServerURL": "https://hosted.mender.io",
"TenantToken": "",
"UpdatePollIntervalSeconds": 5
The configuration options are documented in our official documentation,
It is also an good idea to read trough the recommendations for choosing polling intervals,
/etc/mender/server.crt
This is only needed if you are using self-signed certificates or using a CA that is not included in the ca-certificates bundle.
We provide a demo certificate for usage with the Mender demo environment. Do not use this for production, as the private key is freely available in our source code repositories.
If you are setting up an Mender on-premise server, a server.crt file will be generated if you follow the Certification and Keys section.
/usr/share/mender/identity/mender-device-identity
This script is called by the Mender client to generate a unique device identity.
This script shall exit with non 0 status code on errors. In this case the agent
will discard any output the script may have produced.
This script shall output identity data in key=value format, one entry per line. Example
$ ./mender-device-identity
mac=de:ad:ca:fe:00:01
cpuid=1112233
Multiple values are ok, but the important part is that these values do not change over time as that would result in a new device identity.
An example script that outputs the MAC address of the first valid network interface can be found in the the Mender client source.
Common stable sources for device identity:
- MAC address of a specific network interface (though some devices have randomized MAC which makes it unsuitable as an source for device identity)
- Custom serial number which is provisioned during production
- CID register of eMMC
/usr/share/mender/inventory/*
The inventory scripts are used to collect data to send to the Mender server, the data will be sent periodically and how often depends on the UpdatePollIntervalSeconds configuration option. You can have multiple scripts in the /usr/share/mender/inventory/ directory.
The Mender server only stores this data without further processing. This data can later be used to query devices using the server API based on inventory information.
The Mender server GUI will display the inventory data, e.g
This script shall output inventory data in key=value format, one entry per line. Example:
$ ./mender-inventory-hostinfo
cpu_model=ARMv7 Processor rev 1 (v7l)
mem_total_kB=102847
Example inventory scripts can be found in the Mender client source
Installing Mender
Mender should be installed into /usr/bin on the target device or some other location that is typically part of PATH.
There are some additional files that need to be configured or added to get a fully functional system with Mender.
/etc/fstab
It is important that the root filesystem is mounted with a similar entry to below,
/dev/root / auto defaults 1 1
Above is needed because we will be switching between two different parts that act as the root filesystem and then the path to the device can not be hard-coded. /dev/root is created based on the root= Linux kernel argument.
In addition to common entries such as the root filesystem, the /etc/fstab file needs to contain the following entries:
/dev/mmcblk0p1 /uboot auto defaults,sync,auto 0 0
/dev/mmcblk0p5 /data auto defaults,auto 0 0
The two partitions need to match the partitions in your layout that contain the U-Boot environment file and the ext3/ext4 data partition, respectively. See the official partition layout for more information.
Note that if you are not using a dedicated boot partition, the /uboot entry may not be necessary.
/etc/mender/artifact_info
In /etc/mender/artifact_info there should be a single line:
artifact_name=<artifact_name>
<artifact_name> is name of the image or update that. This is what the device will report that it is running, and different images must have different names
Example:
$ cat /etc/mender/artifact_info
artifact_name=release-1
/var/lib/mender/mender -> /data/mender
In this post we have assumed that the we mount the fourth partition which is the persistent data partition to /data.
The default directory where Mender will store state is /var/lib/mender, but to be able to store state across updates you typically create a symlink, /var/lib/mender -> /data/mender, making sure that anything that is written to /var/lib/mender is persistent across updates.
/var/lib/mender/device_type
NOTE! This assumes that the symlink /var/lib/mender/mender -> /data/mender is in place so this path could also be represented as /data/mender/device_type.
In /var/lib/mender/device_type there should be a single line:
device_type=<device_type>
<device_type> should be a string that uniquely identifies a group of devices.
systemd
There is an example systemd service for Mender which can be found our Yocto layer which you typically install under /lib/systemd/system and to enable it you need to run:
systemctl enable meneder && systemctl start mender
SysVinit
We do not provide an official SysVinit script for Mender but there is one in the Buildroot source code.
GRUB
More information coming soon…
GRUB does not require any patches to the actual source code and most of the integration is in the grub.cfg file. You can take a look at this repository for more information,
U-Boot
Now we’re getting into the advanced stuff! U-Boot is one of the trickiest parts to integrate in the build. We need to patch U-Boot to support Mender’s automatic partition selection and rollback features, as well as provide some auxiliary files.
NOTE! We have an U-boot fork for Raspberry Pi boards where all the necessary patches have been applied and is a good reference as well. This U-boot fork is also used by our mender-convert tool
U-Boot configuration
U-Boot needs to store its state, which is typically referred to as U-boot “environment”, to persistent storage during upgrade procedures. To accomplish this you need to add,
CONFIG_ENV_IS_IN_MMC=y
to your board defconfig file. Note if you are using device with RAW NAND then the equivalent configuration is,
CONFIG_ENV_IS_IN_NAND=y
You need to enable the following features in a board specific header file,
#define CONFIG_BOOTCOUNT_LIMIT
#define CONFIG_BOOTCOUNT_ENV
Above features are utilized to perform an automatic roll-back, a simplified explanation is that it will call alt_bootcmd if bootcount exceeds bootlimit.
Any occurrences of CONFIG_ENV_OFFSET should be removed from board specific header or defconfig files, these will be defined based on values in config_mender_defines.h which is described below.
Any occurrences *_ENV_INTERFACE, *_ENV_DEVICE, *_ENV_PART and *_ENV_DEVICE_AND_PART should be removed from board specific header or defconfig files. These will be defined based on config_mender_defines.h which is described below.
config_mender_defines.h
config_mender_defines.h is a special file used by Mender during compilation to provide configuration values to U-Boot supplied by the build system. This file is auto-generated during a Yocto build, but we need to generate it ourselves for our third party build system.
Below I have put an example of what this looks like when it’s filled out (this example is from Raspberry Pi 3 integration):
#ifndef HEADER_CONFIG_MENDER_DEFINES_H
#define HEADER_CONFIG_MENDER_DEFINES_H
/* Shell variables */
#define MENDER_BOOT_PART_NUMBER 1
#define MENDER_BOOT_PART_NUMBER_HEX 1
#define MENDER_ROOTFS_PART_A_NUMBER 2
#define MENDER_ROOTFS_PART_A_NUMBER_HEX 2
#define MENDER_ROOTFS_PART_B_NUMBER 3
#define MENDER_ROOTFS_PART_B_NUMBER_HEX 3
#define MENDER_UBOOT_STORAGE_INTERFACE "mmc"
#define MENDER_UBOOT_STORAGE_DEVICE 0
/* BB variables. */
#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 0x400000
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 0x800000
/* For sanity checks. */
#define MENDER_BOOTENV_SIZE 0x4000
#define MENDER_UBOOT_PRE_SETUP_COMMANDS ""
#define MENDER_UBOOT_POST_SETUP_COMMANDS ""
The six first entries are simply partition numbers, and the two entries containing UBOOT describe how to refer to the storage device using U-Boot’s syntax.
In this case we refer to the MultiMediaCard interface, or MMC, and we select the first (0’th) such slot. The MENDER_STORAGE_DEVICE_BASE is the string which the Linux kernel uses to refer to the MMC storage, excluding the partition number.
The MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET variables define where to store the U-boot environment on the storage medium. Take note of these values as they represent an address of on the storage medium, meaning that the first partition should be after these addresses, otherwise you can end up with conflicts. In the example presented the environment is aligned to 4MB, meaning that the first partition should start at a 12MB offset.
The MENDER_BOOTENV_SIZE is the size of the U-Boot environment and should match CONFIG_ENV_SIZE. This might feel like an extra unnecessary step and it is in this context, but our Yocto integration uses these variables to perform sanity checks and since we will re-use the same patches that are in Yocto we need to add this.
The PRE_SETUP_COMMANDS/POST_SETUP_COMMANDS are also residue from Yocto, but they need to be defined or we will get an build error.
Patches
There are a couple of board independent patches that need to be applied to U-boot, and they are:
0002-Generic-boot-code-for-Mender.patch0003-Integration-of-Mender-boot-code-into-U-Boot.patch0006-env-Kconfig-Add-descriptions-so-environment-options-.patch
These can be found in meta-mender layer. Note that these patches typically apply cleanly to a more recent U-boot version and you might need to adopt the patches if you are using an older version.
The patch containing all the boot logic required for Mender can be found in 0002-Generic-boot-code-for-Mender.patch, if you are curious and want to read in on the source code.
In addition to the cross platform patches you’ll also need patches specific to your board. We have documented what is needed in the board specific patch in our documentation, Manual U-boot integration
We have many reference integrations in meta-mender-community where you can find board specific patches, there might be one for your specific board or at least it can be a source of inspiration.
Compiling
You should now be ready to compile U-Boot! This process is covered under “Building the Software” inside the README file in the U-Boot source tree, so I will not go through it here. How to integrate the resulting boot loader image is board specific, and needs to follow the procedure for the board you are building for.
U-boot environment utils
In addition to the U-Boot boot loader image, some user space Linux tools are also needed, to be able to modify the U-boot environment from user-space. These tools are part of the U-boot source code and can be found at tools/env/ and to compile the you should use the same U-boot source with the Mender patches applied and run,
make env
The resulting binary, tools/env/fw_printenv , should be installed in the rootfs image twice, once as /sbin/fw_printenv and once as /sbin/fw_setenv . The tool uses the binary name as an indication of which operation to perform.
The Mender client will invoke these tools.
/etc/fw_env.config
In order for the U-boot environment tools to find the environment, we need to provide a configuration file called /etc/fw_env.config . This file should contain information of where your U-boot environment is located and its size, this must match the configuration that you have built-in in U-boot, an example is:
/dev/mmcblk0 0x400000 0x4000
/dev/mmcblk0 0x800000 0x4000
This matches the values that are set in config_mender_defines.h that was described earlier,
#define MENDER_STORAGE_DEVICE_BASE "/dev/mmcblk0p"
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 0x400000
#define MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 0x800000
#define MENDER_BOOTENV_SIZE 0x4000
Mender Artifact
Mender would not be much good unless we could also provide updates to the image we just made. To do this we need the mender-artifact tool.
Instructions on how to compile the mender-artifact tool can be found here,
You can also download a prebuilt mender-artifact (x86_64) Linux binary here.
The primary input to the mender-artifact tool is the filesystem image that comprises the rootfs. In addition we need to provide some metadata about the image. Here is an example invocation of the tool:
mender-artifact write rootfs-image \
-n <artifact_name> \
-t <compatible_devices> \
-u <rootfs image> \
-o <name-of-update>.mender
-
<artifact_name>- The Mender Artifact name and this should match the content of/etc/mender/artifact_info. -
<compatible_devices>- List of compatible devices (comma separated) and this will be later compared to the content of/var/lib/mender/device_type -
<rootfs image>- Path to the root filesystem image, e.g anext4image. -
<name-of-update>- Path to the output Mender Artifact. It must have a.mendersuffix to be able to upload it to the Mender Server.