Help Automating OS Image Creation with Mkosi for Mender-Compatible Deployment?

[TheYoctoJester]

Hi @Trab40,

nice thread indeed, because I think it highlights a few challenges which are very common. Some quotes slightly shortened, but hopefully keeping the meaning intact.

@Trab40

@TheYoctoJester

Concerning your situation at hand, I guess that you are tied to Ubuntu 20.04+ROS Foxy to some extent? So how is your timeline looking for the next iteration, specifically as Ubuntu 20.04 will go out of support in 6 months time?

So our application is currently written to work on ROS2 Foxy, which targets Ubuntu 20.04. I do not think that it will be very difficult to update to the newest version, we just haven’t had the time as a startup. We are actually already in dangerous waters since Foxy is already EOL. This is also why I am figuring out what the best way is to update the systems that is also somewhat future proof, hence my investigation into mkosi.

Understood - like said, it’s a very common situation unfortunately. Starting out with a golden image feels easier and possibly yields faster results in the beginning, but you end up with a maintenance nightmare. Where to head depends on your specific requirements of course, but basically it means addressing the technical debt from the faster ramp-up.

@Trab40

@TheYoctoJester

My suggestion would be to take this as an opportunity for migrating towards an actually reproducible solution for distribution creation. The most obvious is Yocto, which seems to have pretty neat ROS2 support via meta-ros.

So my ideal setup (which I think is also that of many Mender users) would be one where I have a Embedded Linux solution such as Yocto with the ability to do docker container updates frequently with OS updates more sparingly (as outlined in Combining Operating System and Application updates | Mender documentation) .
However I have no experience with Yocto and was thinking that mkosi would be a nice stepping stone, … I guess I am just afraid that I will be biting off too much than I can chew if I dive into the yocto rabbit hole.

Such a setup is, let’s say, comparatively easy to build. In the end it consists, as you already figured out, of two building blocks.

1. the base OS image, which provides the infrastructure to run and update a container-based workload. I have written a tutorial on this, and an example incarnation for the Raspberry Pi at meta-mender-community/kas/demos/meta-mender-demo-app-updates at scarthgap · mendersoftware/meta-mender-community · GitHub
2. the containerized application payload, which is then deployed on top of the base image. Mender provides the so-called application update module, and based on that I am just right now working on an example pipeline to build and deploy those artifacts. You can find the current working state (I’d say beta quality, functional but not pretty) at GitHub - TheYoctoJester/mender-app-update-pipeline: Example repository for creating and deploying a docker-compose based Mender Artifact

@Trab40

Another important aspect of this is I do not know if I can update a Mender image created using the golden image approach with one that uses the Yocto approach, which is something I will need to do for the existing systems if I decide to make the switch!

I’m pretty sure it can be done, also without too much hassle, if the original Ubuntu setup already is A/B enabled. For something x86/UEFI based, even better. It will definitely require a bit of tinkering and testing, but my gut feeling is that it should not be a blocker. Possibly even the easiest part of this. Maybe @kacf has some thoughts?

@Trab40

@TheYoctoJester

And CRA is in effect now, so you’re up for problems with Ubuntu+golden images anyways.

Interesting! I am assuming that CRA refers to the Cyber Resilience Act? In what way is Ubuntu+golden images not compliant with this? Is there a way to make it compliant?

The first and foremost step would be finding a way to create an SBOM of what you are actually shipping. To my knowledge, that’s extremely hard to nearly impossible for golden images, because basically every time (exception: configuration) you modify the image, you actually change the software payload. And doing so at different points in time will even give you slightly different states of software, because the versions from the upstream repositories are in flux. In a nutshell: it’s just not reproducible or traceable.

So TL;DR: my advice would be to go for a combined Yocto base OS plus container updates on top of it. But again, I’m somewhat biased there.

How all of this plays together with ROS, I absolutely cannot comment on.

Greetz,
Josef