Using kas to reproduce your Yocto builds

Tutorials Yocto Project
1

While a Yocto Project/OpenEmbedded based strategy to create Linux based distributions does technically provide an excellent base for high reproducibility of the built artifacts, the actual setup of the build environment itself can be complicated, even jeopardizing the goal of reproducing the Linux distribution.

There is a variety of approaches to solve this problem.

  • git submodules
  • repo
  • bitbake-layers
  • homegrown setup scripts
  • kas

All of those have different strengths and weaknesses, which makes them suitable for different use cases. In the case of meta-mender-community, the repo tool has been used for a long time and is being increasingly replaced now by kas. This allows to provide build configurations for a variety of boards/integrations which offer a high degree of reproducibility through defined revisions.

Installation

kas should be installed through the pip tool. Depending on your host setup, you can install it for the surrent user:

pip install kas

or globally

sudo pip install kas

Concept and getting started

kas’ mental model is centered around a single configuration file which defines your desired build. This means it needs to include all information pieces that are required to re-create the build. This means primarily:

  • machine
  • distro
  • image
  • all involved layers and their
    • repository URLS
    • revisions
      This information is captured in a YAML file, which is source revision control friendly and understandable to humans at the same time.

A very simple kas-style configuration file looks like this:

header:
  version: 13

distro: poky
machine: qemuarm64

repos:
  poky:
    url: http://git.yoctoproject.org/poky
    refspec: master
    layers:
      meta:
	  meta-poky:

local_conf_header:
  base: |
    CONF_VERSION = "2"
    INIT_MANAGER = "systemd"

target:
  - core-image-minimal

With that file being saved as poky-qemuarm64.yml, the build is set up and started by calling

kas build poky-qemuarm64.yml

This will build the core-image-minimal based on the poky distribution for the qemuarm64 machine, as defined in the configuration file.

Based on this skeleton, you can add an arbitrary number of layers under the repos node, as well as blocks to be injected into local.conf under the local_conf_headers node.

Decomposing kas configurations through includes

Just having a monolithic YAML file works well as long as you only have self-contained build setups that can and should move freely, for example when a layer revision gets updated. Once you start working with related or interconnected builds, the resulting copy-paste approach leads to duplication and therefore numerous places in which to maintain the same modifications. The solution here is to move the common pieces into a shared include. Taking the above example, we can expand to build for qemux86 and qemuarm64.

File poky.yml:

header:
  version: 13

distro: poky

repos:
  poky:
    url: http://git.yoctoproject.org/poky
    refspec: master
    layers:
      meta:
	  meta-poky:

local_conf_header:
  base: |
    CONF_VERSION = "2"
    INIT_MANAGER = "systemd"

target:
  - core-image-minimal

File qemux86.yml:

header:
  version: 13
  includes:
  - poky.yml

machine: qemux86

File qemuarm64.yml:

header:
  version: 13
  includes:
  - poky.yml

machine: qemuarm64

Now you can build the two distinct machines by calling kas build qemux86.yml and kas build qemuarm64.yml, respectively. A change in the poky.yml file will directly be applied to both machine specific builds.

local override files

There are configuration items that you will want to apply to a number of builds, but which are not fit for inclusion in a publicly accessible file, or that are build machine specific. A few examples:

  • DL_DIR and SSTATE_DIR: a build configuration file should be environment agnostic, but those are usually shared across all builds on a specific host
  • MENDER_SERVER_URL and MENDER_TENANT_TOKEN: those values are specific to your fleet of devices, and should not be shared

How to inject such into a kas build? You can use an override file. To have the build use your Mender account, you could have the following file:

File: my-mender.yml

header:
  version: 13

local_conf_header:
  mender: |
    MENDER_SERVER_URL = "https://hosted.mender.io"
    MENDER_TENANT_TOKEN = "..."

To use it in your kas invocation, chain the configuration files, separated by colons:

kas build qemux86.yml:my-mender.yml

This will build the qemux86.yml-defined setup, and add the additional Mender configuration to local.conf.

in meta-mender-community

Starting with the kirkstone release branch, the meta-mender-community layer includes a set of kas configuration files. They are located in the kas directory, and are meant to provide known good build setups for the known boards.

Example kas/raspberrypi4.yml:

mkdir my-build
cd my-build
kas build ../kas/raspberrypi4.yml

will create a subdirectory called my-build to contain the build, set it up and run. After it finishes, the canonical build directory structure is located in my-build/build, including the resulting Artifacts.

Interactive mode: kas shell

During development you might want to “work” on the build in a more interactive way instead of strictly reproducing a defined configuration. kas has a way to set up the structure including specified layers, build directory and configuration and then drop you into a shell. Reusing the Raspberry Pi 4 example in meta-mender-community, building core-image-minimal

mkdir my-build
cd my-build
kas shell ../kas/raspberrypi4.yml
# you can carry out tasks just as usual here
bitbake core-image-minimal

Note: by invoking bitbake directly the target setting of the configuration files is obviously not taken into account anymore.

Common pitfalls

Duplicate configuration fragment names

In the local_conf_header section, you have one or more fragments, each identified by a name, base and mender in the above examples. Those names need to be unique, otherwise only the last parsed fragment of a specific name will be actually applied.

Native versus containerized builds

kas also provides the kas-container invocation, which moves the actual build process into a container. This is useful in some cases, and troublesome in others. If you are looking into containerizing your build, this might be an interesting starting point.

transient local.conf/bblayers.conf

kas treats local.conf and bblayers.conf as completely transient. Any modifications that are added during an interactive kas shell session, as well as preexisting files will be completely replaced. Make sure to save your work either in the YAML files or metadata so you don’t lose it upon the next kas invocation.

Further resources