I’m looking for any best practices for automatically building images as part of a CI/CD pipeline.
Here’s how I envision things working:
- I push a new commit to a repo containing the manifest files for my application (what packages are needed, versions of my containers etc.)
- a CI/CD pipeline is triggered (for example, a GitHub Action)
- this CI/CD pipeline builds a new image in a systematic, reproducible way based on the manifests in the repo (I would like this process to be similar to Docker builds containers from Dockerfiles)
- the CI/CD pipeline runs some tests on the resulting image
- if all tests pass, the pipeline creates a new image and pushes it to my Mender server
Have any of you implemented a pipeline like this? How?
I could see some solutions ways to do it using tools like Ansible, Puppet, Chef, Saltstack etc… but I don’t quite have the complete picture figured out. I would love to learn from your experience.
For context, here’s some information about my project:
- ubuntu server-based
- runs a bunch of containers
- also runs some services on bare metal
Assuming you’re using Yocto, kas and kas-docker go a long way to ensuring reproducible build environments, especially for CI.
I essentially implemented the first three steps of your plan using Gitlab CI and kas, but I’d assume it would be similar with Github actions. Because Yocto builds need a lot of processing power, I use the Gitlab docker-machine runner to spin up a really beefy cloud VM to do the build, which means I’m only paying for the beefy cloud VM while the build is taking place and builds are relatively quick (~1 hour).
edit: Reread your post and realised you’re building on top of Ubuntu server, so maybe the Yocto advice is not so useful, sorry. For a different project I automated building up an Ubuntu image using kickstarts. Maybe you could use those to install all the packages in your base image and then build and copy your containers over?
Thanks Sam. Kickstarts go some of the way to covering what I am looking for.
How did you run those jobs? I assume you made the images in virtual machine - what did you use to do so?
The recommended workflow for golden images suggests manually booting the device, logging in, and making all modifications. This is time-consuming, and error-prone, and unsuitable for CI/CD, which is what I am trying to accomplish.
HI @krystof indeed the Golden Master approach is a bit awkward and, as you say, time consuming. I have started playing with mkosi which is a wrapper around the Debian build utilities and supports a decent number of distros. So far it seems pretty useful
mkosi looks like exactly what I need! Thanks a bunch!