Help Automating OS Image Creation with Mkosi for Mender-Compatible Deployment?

General Discussions
1

Hello!

I am currently using Mender as a deployment platform for a small fleet of robots running on x86 computers (HX401). These robots currently run Ubuntu 20.04 with ROS2 Foxy.

To create the OS image they are using, I followed the recommended workflow outlined in the Mender documentation. This involves creating a “golden image” by installing the OS, booting the device, logging in, and making runtime modifications (e.g., installing packages, changing configurations) to generate the Mender Artifact.

I have noticed that this process is not ideal. This issue is also highlighted in this Mender Hub post, where @krystof states:

The recommended workflow for golden images suggests manually booting the device, logging in, and making all modifications. This is time-consuming, error-prone, and unsuitable for CI/CD.

I completely agree with this!

This comment prompted me to explore whether I could use mkosi to create an image compatible with my platform while still supporting Mender.

I quickly discovered that this is a relatively niche area with limited information available online. While there are some snippets and mentions of using mkosi on forums, there are no concrete, up-to-date examples demonstrating compatibility with the latest versions of these tools. Many configurations I found are outdated and no longer work.

For completeness, here is a list of all the sources I referenced in my research (these might help others in the future):

I attempted the following mkosi.config to see if I could get it to build an image:

[Distribution]
Distribution=ubuntu
Release=focal
Repositories=universe

[Build]
WithNetwork=True

[Content]
Packages=
    linux-image-generic-hwe-20.04 
    systemd
    udev
    grub-efi-amd64
    sudo
    
Bootable=yes
Bootloader=grub
BiosBootloader=grub

[Output]
Format=disk

However, I encountered numerous errors during the process.

This experience has highlighted that I am out of my depth in figuring out how to get started with mkosi as an alternative to manual golden image creation. Even experimenting with it has been challenging.

If someone could assist me (and likely others who come across this in the future), I would appreciate answers to the following:

  1. What configuration is needed to create an OS image (just Ubuntu 20.04 would already be enough!) that I can use with dd to flash onto the SSD of my target platform? Is any additional conversion required to make the output of mkosi usable with dd?
  2. What considerations are necessary to ensure this image is also compatible with Mender?
2

Hi @Trab40,

Yeah, golden image based workflows are inherently problematic - but at the same time quite popular in situations like the one you described, where creating a fully automated, reproducible image creation workflow is not readily available, and constructing one would take a lot of time and effort.

Recently I have also played with mkosi a bit, and it seems to be a) quite opinionated towards all things systemd/containers, and b) also changes the supported feature set substantially over versions. That together with the fact that you would still need quite some postprocessing, like partitioning for A/B, eventual bootloader integration, your own application and so on, and so on, I can’t call it a good approach given the information I have.

Concerning your situation at hand, I guess that you are tied to Ubuntu 20.04+ROS Foxy to some extent? So how is your timeline looking for the next iteration, specifically as Ubuntu 20.04 will go out of support in 6 months time?

My suggestion would be to take this as an opportunity for migrating towards an actually reproducible solution for distribution creation. The most obvious is Yocto, which seems to have pretty neat ROS2 support via meta-ros. Alternatives might be ISAR or ELBE, both solutions to build a distribution from Debian packages. I’d expect that those are - with some effort - able to build Ubuntu.

Happy to discuss this further, but in a nutshell, my advice is kinda like “think more strategic for the whole product software maintenance”.

And CRA is in effect now, so you’re up for problems with Ubuntu+golden images anyways.

Greetz,
Josef

3

Hi Josef!

Thanks for taking the time to answer!

Regarding your response:

Thanks for the tip! I had seen it mentioned a couple of times on this forum so thought that it was an alternative that was working for others. The lack of examples and documentation should have been telling i guess.

So our application is currently written to work on ROS2 Foxy, which targets Ubuntu 20.04. I do not think that it will be very difficult to update to the newest version, we just haven’t had the time as a startup. We are actually already in dangerous waters since Foxy is already EOL. This is also why I am figuring out what the best way is to update the systems that is also somewhat future proof, hence my investigation into mkosi.

So my ideal setup (which I think is also that of many Mender users) would be one where I have a Embedded Linux solution such as Yocto with the ability to do docker container updates frequently with OS updates more sparingly (as outlined in Combining Operating System and Application updates | Mender documentation) .
However I have no experience with Yocto and was thinking that mkosi would be a nice stepping stone, where I am creating an image automatically while still having access to all the tools of ubuntu server, and getting experience with the deployment of docker containers and os images. I guess I am just afraid that I will be biting off too much than I can chew if I dive into the yocto rabbit hole. :sweat_smile:
Another important aspect of this is I do not know if I can update a Mender image created using the golden image approach with one that uses the Yocto approach, which is something I will need to do for the existing systems if I decide to make the switch!

Interesting! I am assuming that CRA refers to the Cyber Resilience Act? In what way is Ubuntu+golden images not compliant with this? Is there a way to make it compliant?

Thanks!

4

Hi @Trab40,

nice thread indeed, because I think it highlights a few challenges which are very common. Some quotes slightly shortened, but hopefully keeping the meaning intact.

Understood - like said, it’s a very common situation unfortunately. Starting out with a golden image feels easier and possibly yields faster results in the beginning, but you end up with a maintenance nightmare. Where to head depends on your specific requirements of course, but basically it means addressing the technical debt from the faster ramp-up.

Such a setup is, let’s say, comparatively easy to build. In the end it consists, as you already figured out, of two building blocks.

  1. the base OS image, which provides the infrastructure to run and update a container-based workload. I have written a tutorial on this, and an example incarnation for the Raspberry Pi at meta-mender-community/kas/demos/meta-mender-demo-app-updates at scarthgap · mendersoftware/meta-mender-community · GitHub
  2. the containerized application payload, which is then deployed on top of the base image. Mender provides the so-called application update module, and based on that I am just right now working on an example pipeline to build and deploy those artifacts. You can find the current working state (I’d say beta quality, functional but not pretty) at GitHub - TheYoctoJester/mender-app-update-pipeline: Example repository for creating and deploying a docker-compose based Mender Artifact

I’m pretty sure it can be done, also without too much hassle, if the original Ubuntu setup already is A/B enabled. For something x86/UEFI based, even better. It will definitely require a bit of tinkering and testing, but my gut feeling is that it should not be a blocker. Possibly even the easiest part of this. Maybe @kacf has some thoughts?

The first and foremost step would be finding a way to create an SBOM of what you are actually shipping. To my knowledge, that’s extremely hard to nearly impossible for golden images, because basically every time (exception: configuration) you modify the image, you actually change the software payload. And doing so at different points in time will even give you slightly different states of software, because the versions from the upstream repositories are in flux. In a nutshell: it’s just not reproducible or traceable.

So TL;DR: my advice would be to go for a combined Yocto base OS plus container updates on top of it. But again, I’m somewhat biased there.

How all of this plays together with ROS, I absolutely cannot comment on.

Greetz,
Josef

5

You definitely can. Many people have done this already. There may be some smaller obstacles that need overcoming, depending on the exact setup, but the way that Mender works does not differ substantially between Yocto and mender-convert.

The main challenge that comes to mind if using x86 is making sure the right grub.cfg is loaded, and that it in turn loads the correct kernel. You need to take a look at the boot partition of your existing conversions to see what they do, and which files they load.

6

@Trab40 This probably won’t help in your case, but in case it’s helpful for anyone else…

We are using Debos to build our Mender-ized OS images. We haven’t set up a CI/CD pipeline integrated with hosted Mender yet, but even in it’s current manual deployment state it addresses a lot of the concerns that you and @TheYoctoJester are discussing.

7

Thank you, everyone, for taking the time to respond!

As @TheYoctoJester pointed out:

This does indeed sound like an elegant and robust solution to the problem. However, I’m hesitant to dive into building my own image at this point. The main reason for this hesitation is that my expertise lies primarily in robotics software engineering rather than embedded Linux, though I find the latter very interesting.

To elaborate further, my concerns are as follows:

  • Unclear system requirements: I’m not entirely sure what my system needs to function correctly. I know that i need ubuntu server 20.04, but not the exact parts which I do and do not use. This makes it challenging to determine what should or shouldn’t be included in the Yocto image.
  • Evolving needs during development: As we are still developing the robot, it’s difficult to predict future requirements. Additionally, unexpected needs often arise during client integrations, where I rely on tools that happen to be pre-installed in standard Linux server environments.
  • Specialized skillset and time constraints: From my preliminary research, creating a Yocto image appears to be a specialized task, often a full-time job in some organizations. With limited time, I worry that creating a Yocto-based solution could lead to complications that outweigh the benefits at this stage.

These challenges reflect the prototyping phase we’re currently in, which makes me think the Yocto approach may be too much. This is why I considered mkosi—it seems to offer a practical stepping stone before committing to a full-fledged Yocto solution.

The suggestion from @bssi to explore Debos also seems promising and aligns with our current needs. I’ll definitely take a closer look at it. I am curious why it won’t help in my case! A quick google of “OS image builder” leads me to Image Builder, which may also be relevant.

Considering the input from this discussion, I’ve also been exploring the idea of streamlining the creation of a slimmed-down golden image. Currently, we use bash install scripts on a fresh Ubuntu Server 20.04 installation. However, one issue with this approach is that changes made to the OS on the device are not always easily reversible.

To address this, I’m thinking of using virtual machines to finalize the scripts and simultaneously strip down the OS as much as possible. By taking strategic snapshots of the VMs, it becomes easier to “undo” certain changes. This could potentially speed up the process significantly and make it more reliable.

All these stepping stone solutions clearly don’t take the ramifications of Cyber Resilience Act into account.