Introduction
Containers, and especially docker as the management tool are seeing an ever increasing interest and demand also in embedded Linux. Lets look at how you can add those to your Yocto Project® (YP) based build!
This tutorial will guide you through how to add the necessary layers and configuration to an existing initial setup, such as described in the setup guide on Mender Hub, so docker, and eventually docker-compose can be installed on the device.
This is a high-level tutorial and the intention is not to cover the Yocto Project in detail. For detailed information we recommend that you read the Yocto Project Mega-Manual
The Yocto Project is an open source collaboration project that helps developers create custom Linux-based operating systems regardless of the hardware architecture.
The project provides a flexible set of tools and a space where embedded developers worldwide can share technologies, software stacks, configurations, and best practices that can be used to create tailored Linux images for embedded and IOT devices, or anywhere a customized Linux OS is needed.
Version notes
The tutorial has been verified on Debian 11, as of 2023-08-02
This tutorial uses kirkstone as the primary target, which is the current LTS release by the Yocto Project. You can find more infomation on releases here. Supported releases for following the tutorial are:
| Yocto Project | Tutorial applies | Maintenance |
|---|---|---|
| scarthgap (5.0) |  |
development |
| nanbield (4.3) | |
stable |
| mickledore (4.2) | |
 EOL |
| langdale (4.1) | |
EOL |
| kirkstone (4.0) | |
LTS |
| honister (3.4) | |
EOL |
| hardknott (3.3) | |
EOL |
| gatesgarth (3.2) | |
EOL |
| dunfell (3.1) | |
LTS |
| zeus (3.0) | |
EOL |
| warrior (2.7) | |
EOL |
| thud (2.6) | |
EOL |
| sumo (2.5) | |
EOL |
| rocko (2.4) | |
EOL |
Please note: a failure in the “tutorial applies” column indicates that the instructions do not work without modification. Depending on the combination of release and host Linux distribution, installing other python versions or dependencies might provide a functional state.
Prerequisites
To follow this tutorial, you will need:
- A prepared build setup We will be using an exisiting Yocto Project build setup and the Raspberry Pi 4 as an example target. As the example for the instructions, the setup as described on the Mender Hub is used.
- An initialized shell The commands assume that the shell has been initialized for the build setup, which is usually done by invoking
source poky/oe-init-build-env. Given the linked setup, we start in theyoctodirectory
The meta-virtualization layer
The meta-virtualization layer is the canonical place for all virtualization and containerization-related pieces of metadata. This mostly means recipes, but also commonly required kernel configuration fragments.
Prominent examples for provided recipes are
Getting the layer
Clone the repository into the sources directory. We are checking out the kirkstone branch here, adapt in case you are using a different release.
cd sources
git clone git://git.yoctoproject.org/meta-virtualization -b kirkstone
cd ..
Adding the layer
Change into the build directory:
cd build
meta-virtualization has a an additional layer dependency compared to our base setup,
so lets add that first:
bitbake-layers add-layer ../sources/meta-openembedded/meta-filesystems
Then add the virtualization layer, and check the layer listing:
bitbake-layer add-layer ../sources/meta-virtualization
bitbake-layer show-layers
docker
Building
docker is by itself a recipe, as there are various providers for it. You can test its build via
bitbake virtual/docker
or by selecting a specific provider, such as docker-ce by
bitbake docker-ce
Note: this just checks if docker builds for your setup. As with every package, it will not be installed without adding it to the image.
Enabling the virtualization features
You might have noticed this warning:
WARNING: You have included the meta-virtualization layer, but 'virtualization' has not been enabled in your DISTRO_FEATURES. Some bbappend files may not take effect. See the meta-virtualization README for details on enabling virtualization support.
This means, you need to add a distribution-wide setting in order to fully enable the things that meta-virtualization provides. An example would be the overlayfs support, which docker requires. Without this kernel option, docker could be installed on the device but would not work.
For our example setup, you can just add the DISTRO_FEATURES extension to local.conf. In a real life situation, this should go into your custom distribution configuration file.
echo 'DISTRO_FEATURES:append = " virtualization"' >> conf/local.conf
Adding docker to the image and testing it
To add docker to your resulting image, you can append it to the IMAGE_INSTALL variable. For testing, we put that modification into local.conf:
echo 'IMAGE_INSTALL:append = " docker-ce"' >> conf/local.conf
Then build the desired images such as core-image-base, and test docker on your machine:
bitbake core-image-base
<put on board, boot and log in>
docker run -it --rm hello-world
Congratulations, you have added docker to your build!
For proper maintenance, it is highly recommended to move the DISTRO_FEATURES extension into your custom distribution configuration now, and add the IMAGE_INSTALL modification to your project image recipe!
Advanced: docker-compose
The mickledore release
Unfortunately, docker-compose is only available from the mickledore release and onwards, see the OpenEmbedded layerindex for reference. This means, if you want to install it you need to forward all of your build setup to the mickledore release branch. Usually this is straightforwards by checking out the mickledore branch of all layers in the sources directory. After that, the next image build will be based on the new release. Please note that this will probably require a noticeable amount of time!
Adding docker-compose
Once the build setup and especially meta-virtualization are moved to mickledore, docker-compose can be added to the IMAGE_INSTALL variable. Again, for testing, putting this into local.conf can be done like this:
echo 'IMAGE_INSTALL:append = " docker-compose"' >> conf/local.conf
This will add docker-compose to any image.
Adding iptables NAT to the kernel
By default, meta-virtualization brings a kernel configuration fragment which enables NAT as a module. This can cause problems with software defined network configurations in docker-compose. To get around that, you can add a configuration fragment which moves this functionality to built-in.
This requires that you have a custom layer prepared which can hold recipes. Please see steps 1 and 2 of this article if you still need to create one.
In your layer, create a directory structure to hold kernel modifications. On the Raspberry Pi, the linux-raspberrypi kernel recipe is used, adapt accordingly if you use a different board.
mkdir -p recipes-kernel/linux-raspberrypi
cd recipes-kernel/linux-raspberrypi
Add a .bbappend file to incorporate the additional configuration fragment, and create the fragment:
cat >> linux-raspberrypi_6.1.bbappend <<EOF
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://netfilter.cfg"
EOF
mkdir linux-raspberrypi && cd linux-raspberrypi
cat >> netfilter.cfg <<EOF
CONFIG_NETFILTER_NETLINK=y
CONFIG_NF_NAT=y
CONFIG_NF_TABLES=y
CONFIG_NFT_NAT=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_NAT=y
CONFIG_NETFILTER_XT_TARGET_NETMAP=y
CONFIG_NETFILTER_XT_TARGET_REDIRECT=y
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_REDIRECT=y
EOF
Note: this matches the Linux kernel version 6.1 as correlated to the mickledore release. Adapt as necessary to your specific board and release.
After rebuilding the image, which will also trigger a kernel rebuild, you are able to fully use docker-compose on your device.
Conclusion
In this tutorial we covered the general requirements and strategy to install docker and eventually docker-compose on your Yocto Project based Linux image. We used the Raspberry Pi 4 as a reference platform, with the core concepts being applicable to any target which has a Yocto Project based board support package.
Addendum - Troubleshooting
docker: x509: certificate has expired
As the Raspberry Pi used in this example does not have a buffered RTC not is synchronizing the clock with NTP by default, you might get the error docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": x509: certificate has expired or is not yet valid: current time 2018-03-09T12:35:33Z is before 2023-05-05T00:00:00Z., or a similar timestamp mentioned.
Fix
Manually set the system time on your Raspberry Pi using the date command. An example to set August 3rd 2023, 13:15PM is
date -s 2308031315
For more details, please consult the manpage
If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!
