# Adding docker and docker-compose to a Yocto build

**URL:** <https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078>\
**Category:** Yocto Project\
**Created:** [August 3, 2023, 1:05pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078 "2023-08-03T13:05:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [August 3, 2023, 1:05pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/1 "2023-08-03T13:05:18Z")

</div>

## Introduction

Containers, and especially `docker` as the management tool are seeing an ever increasing interest and demand also in embedded Linux. Lets look at how you can add those to your [_Yocto Project®_](https://www.yoctoproject.org/) (YP) based build!

This tutorial will guide you through how to add the necessary layers and configuration to an existing initial setup, such as described in the [setup guide on Mender Hub](https://hub.mender.io/t/preparing-the-yocto-project-environment-from-scratch/801), so `docker`, and eventually `docker-compose` can be installed on the device.

This is a high-level tutorial and the intention is not to cover the Yocto Project in detail. For detailed information we recommend that you read the [Yocto Project Mega-Manual](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html)

The Yocto Project is an open source collaboration project that helps developers create custom Linux-based operating systems regardless of the hardware architecture.

The project provides a flexible set of tools and a space where embedded developers worldwide can share technologies, software stacks, configurations, and best practices that can be used to create tailored Linux images for embedded and IOT devices, or anywhere a customized Linux OS is needed.

 ![LF_17_02_Yocto-Badge-Update_Participant_Final](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/758c60e0e8309c4dc2519392b78668e7396c15bd.png)

## Version notes

The tutorial has been verified on Debian 11, as of 2023-08-02

This tutorial uses **kirkstone** as the primary target, which is the current LTS release by the Yocto Project. You can find more infomation on releases [here](https://wiki.yoctoproject.org/wiki/Releases). Supported releases for following the tutorial are:

| Yocto Project | Tutorial applies | Maintenance |
| --- | --- | --- |
| scarthgap (5.0) | :test_works: | :test_works: development |
| nanbield (4.3) | :test_works: | :test_works: stable |
| mickledore (4.2) | :test_works: | :test_fails: EOL |
| langdale (4.1) | :test_works: | :test_fails: EOL |
| kirkstone (4.0) | :test_works: | :test_works: LTS |
| honister (3.4) | :test_works: | :test_fails: EOL |
| hardknott (3.3) | :test_works: | :test_fails: EOL |
| gatesgarth (3.2) | :test_fails: | :test_fails: EOL |
| dunfell (3.1) | :test_works: | :test_works: LTS |
| zeus (3.0) | :test_fails: | :test_fails: EOL |
| warrior (2.7) | :test_fails: | :test_fails: EOL |
| thud (2.6) | :test_fails: | :test_fails: EOL |
| sumo (2.5) | :test_fails: | :test_fails: EOL |
| rocko (2.4) | :test_fails: | :test_fails: EOL |

**Please note:** a failure in the “tutorial applies” column indicates that the instructions do not work without modification. Depending on the combination of release and host Linux distribution, installing other python versions or dependencies might provide a functional state.

## Prerequisites

To follow this tutorial, you will need:

- **A prepared build setup** We will be using an exisiting Yocto Project build setup and the Raspberry Pi 4 as an example target. As the example for the instructions, the [setup as described on the Mender Hub](https://hub.mender.io/t/preparing-the-yocto-project-environment-from-scratch/801) is used.
- **An initialized shell** The commands assume that the shell has been initialized for the build setup, which is usually done by invoking `source poky/oe-init-build-env`. Given the linked setup, we start in the `yocto` directory

## The `meta-virtualization` layer

The [`meta-virtualization`](https://git.yoctoproject.org/meta-virtualization/) layer is the canonical place for all virtualization and containerization-related pieces of metadata. This mostly means recipes, but also commonly required kernel configuration fragments.

Prominent examples for provided recipes are

- [`docker`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/docker?h=kirkstone)
- [`xen`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-extended/xen?h=kirkstone)
- [`podman`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/podman?h=kirkstone)
- [`kubernetes`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/kubernetes?h=kirkstone)

### Getting the layer

Clone the repository into the sources directory. We are checking out the `kirkstone` branch here, adapt in case you are using a different release.

```auto
cd sources
git clone git://git.yoctoproject.org/meta-virtualization -b kirkstone
cd ..

```

### Adding the layer

Change into the build directory:

```auto
cd build

```

`meta-virtualization` has a an additional layer dependency compared to our base setup,  
so lets add that first:

```auto
bitbake-layers add-layer ../sources/meta-openembedded/meta-filesystems

```

Then add the virtualization layer, and check the layer listing:

```auto
bitbake-layer add-layer ../sources/meta-virtualization
bitbake-layer show-layers

```

## `docker`

### Building

`docker` is by itself a recipe, as there are various providers for it. You can test its build via

```auto
bitbake virtual/docker

```

or by selecting a specific provider, such as `docker-ce` by

```auto
bitbake docker-ce

```

**Note** : this just checks if `docker` builds for your setup. As with every package, it will not be installed without adding it to the image.

### Enabling the virtualization features

You might have noticed this warning:

```auto
WARNING: You have included the meta-virtualization layer, but 'virtualization' has not been enabled in your DISTRO_FEATURES. Some bbappend files may not take effect. See the meta-virtualization README for details on enabling virtualization support.

```

This means, you need to add a distribution-wide setting in order to fully enable the things that `meta-virtualization` provides. An example would be the `overlayfs` support, which `docker` requires. Without this kernel option, `docker` could be installed on the device but would not work.

For our example setup, you can just add the `DISTRO_FEATURES` extension to `local.conf`. In a real life situation, this should go into your custom distribution configuration file.

```auto
echo 'DISTRO_FEATURES:append = " virtualization"' >> conf/local.conf

```

### Adding `docker` to the image and testing it

To add `docker` to your resulting image, you can append it to the `IMAGE_INSTALL` variable. For testing, we put that modification into `local.conf`:

```auto
echo 'IMAGE_INSTALL:append = " docker-ce"' >> conf/local.conf

```

Then build the desired images such as `core-image-base`, and test `docker` on your machine:

```auto
bitbake core-image-base
<put on board, boot and log in>
docker run -it --rm hello-world

```

Congratulations, you have added `docker` to your build!

For proper maintenance, it is highly recommended to move the `DISTRO_FEATURES` extension into your custom distribution configuration now, and add the `IMAGE_INSTALL` modification to your project image recipe!

## Advanced: `docker-compose`

### The `mickledore` release

Unfortunately, `docker-compose` is only available from the `mickledore` release and onwards, see the [OpenEmbedded layerindex](https://layers.openembedded.org/layerindex/recipe/327567/) for reference. This means, if you want to install it you need to forward all of your build setup to the `mickledore` release branch. Usually this is straightforwards by checking out the `mickledore` branch of all layers in the `sources` directory. After that, the next image build will be based on the new release. Please note that this will probably require a noticeable amount of time!

### Adding `docker-compose`

Once the build setup and especially `meta-virtualization` are moved to `mickledore`, `docker-compose` can be added to the `IMAGE_INSTALL` variable. Again, for testing, putting this into `local.conf` can be done like this:

```auto
echo 'IMAGE_INSTALL:append = " docker-compose"' >> conf/local.conf

```

This will add `docker-compose` to any image.

### Adding `iptables` NAT to the kernel

By default, `meta-virtualization` brings a kernel configuration fragment which enables NAT as a [module](https://git.yoctoproject.org/meta-virtualization/tree/recipes-kernel/linux/linux-yocto/docker.cfg?h=mickledore). This can cause problems with software defined network configurations in `docker-compose`. To get around that, you can add a configuration fragment which moves this functionality to built-in.

**This requires that you have a custom layer prepared which can hold recipes. Please see steps 1 and 2 of [this article](https://hub.mender.io/t/how-to-create-your-first-recipe-and-enable-auto-start-using-systemd/1195) if you still need to create one.**

In your layer, create a directory structure to hold kernel modifications. On the Raspberry Pi, the `linux-raspberrypi` kernel recipe is used, adapt accordingly if you use a different board.

```auto
mkdir -p recipes-kernel/linux-raspberrypi
cd recipes-kernel/linux-raspberrypi

```

Add a `.bbappend` file to incorporate the additional configuration fragment, and create the fragment:

```auto
cat >> linux-raspberrypi_6.1.bbappend <<EOF
FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
SRC_URI += "file://netfilter.cfg"
EOF

mkdir linux-raspberrypi && cd linux-raspberrypi

cat >> netfilter.cfg <<EOF
CONFIG_NETFILTER_NETLINK=y
CONFIG_NF_NAT=y
CONFIG_NF_TABLES=y
CONFIG_NFT_NAT=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_NAT=y
CONFIG_NETFILTER_XT_TARGET_NETMAP=y
CONFIG_NETFILTER_XT_TARGET_REDIRECT=y
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_NAT=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_IP_NF_TARGET_REDIRECT=y
EOF

```

**Note: this matches the Linux kernel version 6.1 as correlated to the `mickledore` release. Adapt as necessary to your specific board and release.**

After rebuilding the image, which will also trigger a kernel rebuild, you are able to fully use `docker-compose` on your device.

## Conclusion

In this tutorial we covered the general requirements and strategy to install `docker` and eventually `docker-compose` on your Yocto Project based Linux image. We used the Raspberry Pi 4 as a reference platform, with the core concepts being applicable to any target which has a Yocto Project based board support package.

## Addendum - Troubleshooting

### `docker: x509: certificate has expired`

As the Raspberry Pi used in this example does not have a buffered RTC not is synchronizing the clock with NTP by default, you might get the error `docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": x509: certificate has expired or is not yet valid: current time 2018-03-09T12:35:33Z is before 2023-05-05T00:00:00Z.`, or a similar timestamp mentioned.

**Fix**

Manually set the system time on your Raspberry Pi using the `date` command. An example to set August 3rd 2023, 13:15PM is

```auto
date -s 2308031315

```

For more details, please consult the [manpage](https://man7.org/linux/man-pages/man1/date.1.html)

* * *

_If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!_

---

<div class="post-metadata">

**Author:** ![dHofmeister](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dhofmeister/32/2023_2.png) [@dHofmeister](https://hub.mender.io/u/dHofmeister)\
**Post date:** [September 27, 2023, 6:31pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/2 "2023-09-27T18:31:10Z")

</div>

Dear [TheYoctoJester](https://hub.mender.io/u/TheYoctoJester),

Thank you for this write up. This is amazing. Having remote docker compose support on yocto builds is great.

Forgive me if I am making novice mistakes here.

I followed your tutorial, but I tried to modify your example in two ways:

1. Create a meta-layer where your modifications are put in their correct places
2. Modify the MACHINE such that I can run this example in qemux86-64

To this extend I uploaded my work as an example to github. It is unfinished, but compiles.

> **[GitHub - Robotics-Deployment/meta-robotics-deployment: Yocto project meta layer...](https://github.com/Robotics-Deployment/meta-robotics-deployment/tree/main)**
>
> main

And I modified the conf/local.conf as such:

```auto
MACHINE ??= "qemux86-64"
DISTRO ?= "robotics-deployment"

EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
BB_NUMBER_THREADS = "${@oe.utils.cpu_count() - 1}"
PARALLEL_MAKE = "-j ${@oe.utils.cpu_count() - 1}"

USER_CLASSES ?= "buildstats"

PATCHRESOLVE = "noop"

BB_DISKMON_DIRS ??= "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    HALT,${TMPDIR},100M,1K \
    HALT,${DL_DIR},100M,1K \
    HALT,${SSTATE_DIR},100M,1K \
    HALT,/tmp,10M,1K"

PACKAGECONFIG:append:pn-qemu-system-native = " sdl"

CONF_VERSION = "2"

```

For brevity, I removed all comments that come default in local.conf

Along with the conf/bblayers.conf:

```auto
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../sources/meta-openembedded/meta-oe \
  ${TOPDIR}/../sources/meta-openembedded/meta-filesystems \
  ${TOPDIR}/../sources/meta-openembedded/meta-python \
  ${TOPDIR}/../sources/meta-openembedded/meta-networking \
  ${TOPDIR}/../sources/meta-virtualization \
  ${TOPDIR}/../sources/meta-robotics-deployment \
  "

```

After all this, I did manage to compile it all and that is all well.

However, after

> runqemu qemux86-64

and a successful boot, I run into the following:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/9/945f8b976fe18772ef8c37ebf760a8db1b5168a0.png)

I have a feeling I did something wrong with the netfilter.cfg , can it be that your configuration does not apply to qemux86-64 ? I have not tried this yet for a raspberry pi.

I have never seen the http: invalid Host header error

What I tried is modifying the following in local.conf:

```auto
PREFERRED_PROVIDER_virtual/docker = "docker-moby"

```

The installed docker version went from 23.0.2 to 24.0.0, thinking that there would be an issue in docker, but sadly the issue remains.

**Edit:** I decided to build it with **MACHINE ??= “raspberrypi4”** instead and flashed my raspberry pi and I am getting the identical error. I would love to get your input on this?

Some debug output that could maybe help:

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/0/05771e7298076a11e57956f49e3ccd53765e92fb.png)

Greetings and thank you very much for your post.

Deniz

---

<div class="post-metadata">

**Author:** ![dHofmeister](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dhofmeister/32/2023_2.png) [@dHofmeister](https://hub.mender.io/u/dHofmeister)\
**Post date:** [September 29, 2023, 12:53pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/3 "2023-09-29T12:53:28Z")

</div>

In addition to this, it seems that podman does work out of the box, also while removing the netfilter.cfg configuration from the build:

```auto
root@qemux86-64:~# podman run hello-world
Resolved "hello-world" as an alias (/var/cache/containers/short-name-aliases.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob 719385e32844 done  
Copying config 9c7a54a9a4 done  
Writing manifest to image destination
Storing signatures

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@qemux86-64:~# 

```

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [October 2, 2023, 3:16pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/4 "2023-10-02T15:16:40Z")

</div>

Hi @dHofmeister,

Thanks a lot for getting in touch. I have verified the behaviour, and talked to the `meta-virtualization` maintainers about it. It is actually a currently known bug. Fixes should already be on the `master` branch, and are planned to appear on `mickledore` shortly. I will keep you in the loop!

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![sieger](https://avatars.discourse-cdn.com/v4/letter/s/7ea924/32.png) [@sieger](https://hub.mender.io/u/sieger)\
**Post date:** [November 28, 2023, 10:53am UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/5 "2023-11-28T10:53:09Z")

</div>

@TheYoctoJester What would be the best way to get docker-compose working in a `kirkstone` yocto environment? I have tried upgrading my entire build process to the `mickledore` branch and `master-next` for the `meta-mender` layer, but I’m not really comfortable running development branches in a production build. Is there another way to get docker-compose into `kirkstone` without upgrading?

For reference, I’m running python3-docker-compose right now but for usage with [app-gen to generate a docker-compose artifact](https://docs.mender.io/3.6/artifact-creation/create-an-artifact/docker-compose#create-a-deployment) need to use docker-compose v2.

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [November 28, 2023, 11:04am UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/6 "2023-11-28T11:04:38Z")

</div>

Hi @sieger,

No path to `kirkstone` that I could really recommend. I tried to build and run as-is, but the `go` versions were giving me a hard time. In the current situation, my bet would rather be to move forward as much as possible, `nanbield` or even `master` until the next release `scarthgap` directly branches out into LTS. This might mean a bit more integration, maintenance and testing right now but will give the best long term strategy.

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [April 17, 2024, 4:01pm UTC](https://hub.mender.io/t/adding-docker-and-docker-compose-to-a-yocto-build/6078/7 "2024-04-17T16:01:35Z")

</div>

4 posts were split to a new topic: [Install docker-compose on Yocto, bbappend not being picked up](https://hub.mender.io/t/install-docker-compose-on-yocto-bbappend-not-being-picked-up/6739)