# Adding k3s to a Yocto build **URL:** https://hub.mender.io/t/adding-k3s-to-a-yocto-build/6611 **Category:** Yocto Project **Created:** [March 8, 2024, 12:28pm UTC](https://hub.mender.io/t/adding-k3s-to-a-yocto-build/6611 "2024-03-08T12:28:23Z") **Posts on this page:** 1 **Page:** 1
### Author: ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester) #### Post date: [March 8, 2024, 12:28pm UTC](https://hub.mender.io/t/adding-k3s-to-a-yocto-build/6611/1 "2024-03-08T12:28:23Z")
## Introduction Containers are seeing an ever increasing interest, and based upon their technology, [kubernetes](https://kubernetes.io) has emerged as a de-facto standard for their orchestration. Let’s look at how you can add this to your [_Yocto Project®_](https://www.yoctoproject.org/) (YP) based build! This tutorial will guide you through how to add the necessary layers and configuration to an existing initial setup, such as described in the [setup guide on Mender Hub](https://hub.mender.io/t/preparing-the-yocto-project-environment-from-scratch/801), to add [k3s](https://k3s.io). K3s is a Kubernetes distribution suitable for resource constrained devices, such as IoT or edge computing. This is a high-level tutorial and the intention is not to cover the Yocto Project in detail. For detailed information we recommend that you read the [Yocto Project Mega-Manual](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html) The Yocto Project is an open source collaboration project that helps developers create custom Linux-based operating systems regardless of the hardware architecture. The project provides a flexible set of tools and a space where embedded developers worldwide can share technologies, software stacks, configurations, and best practices that can be used to create tailored Linux images for embedded and IoT devices, or anywhere a customized Linux OS is needed. ![LF_17_02_Yocto-Badge-Update_Participant_Final](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/758c60e0e8309c4dc2519392b78668e7396c15bd.png) ## Version notes The tutorial has been verified on Debian 12, as of 2024-03-08 This tutorial uses **kirkstone** as the primary target, which is the current LTS release by the Yocto Project. You can find more infomation on releases [here](https://wiki.yoctoproject.org/wiki/Releases). Supported releases for following the tutorial are: | Yocto Project | Tutorial applies | Maintenance | | --- | --- | --- | | scarthgap (5.0) | :test_works: | :test_works: development | | nanbield (4.3) | :test_works: | :test_works: stable | | mickledore (4.2) | :test_works: | :test_fails: EOL | | langdale (4.1) | :test_works: | :test_fails: EOL | | kirkstone (4.0) | :test_works: | :test_works: LTS | | honister (3.4) | :test_works: | :test_fails: EOL | | hardknott (3.3) | :test_works: | :test_fails: EOL | | gatesgarth (3.2) | : test\_works: | :test_fails: EOL | | dunfell (3.1) | :test_works: | :test_works: LTS | | zeus (3.0) | :test_fails: | :test_fails: EOL | | warrior (2.7) | :test_fails: | :test_fails: EOL | | thud (2.6) | :test_fails: | :test_fails: EOL | | sumo (2.5) | :test_fails: | :test_fails: EOL | | rocko (2.4) | :test_fails: | :test_fails: EOL | **Please note:** a failure in the “tutorial applies” column indicates that the instructions do not work without modification. Depending on the combination of release and host Linux distribution, installing other python versions or dependencies might provide a functional state. ## Prerequisites To follow this tutorial, you will need: - **A prepared build setup** We will be using an exisiting Yocto Project build setup and QEMU for AArch64 as an example target. As the example for the instructions, the [setup as described on the Mender Hub](https://hub.mender.io/t/preparing-the-yocto-project-environment-from-scratch/801) is used. - **An initialized shell** The commands assume that the shell has been initialized for the build setup, which is usually done by invoking `source poky/oe-init-build-env`. Given the linked setup, we start in the `yocto` directory ## The `meta-virtualization` layer The [`meta-virtualization`](https://git.yoctoproject.org/meta-virtualization/) layer is the canonical place for all virtualization and containerization-related pieces of metadata. This mostly means recipes, but also commonly required kernel configuration fragments. Prominent examples for provided recipes are - [`docker`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/docker?h=kirkstone) - [`xen`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-extended/xen?h=kirkstone) - [`podman`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/podman?h=kirkstone) - [`kubernetes`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/kubernetes?h=kirkstone) - [`k3s`](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/k3s_git.bb?h=kirkstone) ### Getting the layer Clone the repository into the sources directory. We are checking out the `kirkstone` branch here, adapt in case you are using a different release. ```auto cd sources git clone git://git.yoctoproject.org/meta-virtualization -b kirkstone cd .. ``` ### Adding the layer Change into the build directory: ```auto cd build ``` `meta-virtualization` has additional layer dependencies compared to our base setup, so lets add that first: ```auto bitbake-layers add-layer ../sources/meta-openembedded/meta-filesystems bitbake-layers add-layer ../sources/meta-openembedded/meta-networking bitbake-layers add-layer ../sources/meta-openembedded/meta-python ``` Then add the virtualization layer, and check the layer listing: ```auto bitbake-layer add-layer ../sources/meta-virtualization bitbake-layer show-layers ``` ## `k3s` ### Building You can test building `k3s` via ```auto bitbake k3s ``` **Note** : this just checks if `k3s` builds for your setup. As with every package, it will not be installed without adding it to the image. ### Enabling the virtualization features You might have noticed this warning: ```auto WARNING: You have included the meta-virtualization layer, but 'virtualization' has not been enabled in your `DISTRO_FEATURES. Some bbappend files may not take effect. See the meta-virtualization README for details on enabling virtualization support. ``` This means, you need to add a distribution-wide setting in order to fully enable the things that `meta-virtualization` provides. In the `k3s` context, we need two additional `DISTRO_FEATURE` flags to be set. - some kernel configuration tweaks. Those are triggered by the `DISTRO_FEATURE` setting `k3s`. - `seccomp`. This is enabled by the `DISTRO_FEATURE` setting `k3s`. For our example setup, you can just add the `DISTRO_FEATURES` extension to `local.conf`. In a real life situation, this should go into your custom distribution configuration file. ```auto echo 'DISTRO_FEATURES:append = " k3s"' >> conf/local.conf echo 'DISTRO_FEATURES:append = " seccomp"' >> conf/local.conf echo 'DISTRO_FEATURES:append = " virtualization"' >> conf/local.conf ``` ### Image adaptions The `k3s` recipes in `meta-virtualization` rely on `systemd` as init manager for automated start up. Just like `DISTRO_FEATURES`, this should be set in your distribution configuration file, for the sake of this tutorial we will use `local.conf`. ```auto echo 'INIT_MANAGER = "systemd' >> conf/local.conf ``` In order to have enough free storage for running containers, allocate 2GB of additional image space. This should be done in a custom image recipe, but `local.conf` is also possible: ```auto echo 'IMAGE_ROOTFS_EXTRA_SPACE= "2097152' >> conf/local.conf ``` ### Adding `k3s` to the image and testing it To add a working standalone `k3s` instantiation to your resulting image, you need to add a few items it to the `IMAGE_INSTALL` variable. For testing, we put that modification into `local.conf`: ```auto echo 'IMAGE_INSTALL:append = " packagegroup-k3s-host"' >> conf/local.conf echo 'IMAGE_INSTALL:append = " packagegroup-k3s-node"' >> conf/local.conf echo 'IMAGE_INSTALL:append = " ca-certificates"' >> conf/local.conf echo 'IMAGE_INSTALL:append = " kernel-modules"' >> conf/local.conf ``` Then build the desired image such as `core-image-base`. ```auto bitbake core-image-base ``` Unlike the standard images which can be tested with `runqemu`, running the `k3s` cluster requires a substantially larger amount of ram than the default, which is just 256MB. Start `runqemu` with 2048MB of RAM: ```auto runqemu nographic slirp qemuparams="-m 2048" ``` Log into the started instance, and after waiting a few moments to let the start up process complete, you can test the `k3s` cluster by running: ```auto root@qemuarm64:~# kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6-k3s1", GitCommit:"c29ac40427297294b851f1edecce76b17eaf0e96", GitTreeState:"clean", BuildDate:"2022-01-20T00:22:17Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/arm64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6-k3s1", GitCommit:"c29ac40427297294b851f1edecce76b17eaf0e96", GitTreeState:"clean", BuildDate:"2022-01-20T00:22:17Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/arm64"} root@qemuarm64:~# kubectl get nodes NAME STATUS ROLES AGE VERSION qemuarm64 Ready control-plane,master 2m13s v1.22.6-k3s1 ``` Congratulations, you have added `k3s` to your build! For proper maintenance, it is highly recommended to move the `DISTRO_FEATURES` extension into your custom distribution configuration now, and add the `IMAGE_INSTALL` modifications to your project image recipe! ## Conclusion In this tutorial we covered the general requirements and strategy to install `k3s` on your Yocto Project based Linux image. We used QEMU as a reference platform, with the core concepts being applicable to any target which has a Yocto Project based board support package. ## Additional resources `meta-virtualization` has a README file specifically for `k3s`: [Making sure you're not a bot!](https://git.yoctoproject.org/meta-virtualization/tree/recipes-containers/k3s/README.md?h=kirkstone) * * * _If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!_