Best practices for automatically building images in a CI/CD pipeline?

General Discussions
1

Hi,

I’m looking for any best practices for automatically building images as part of a CI/CD pipeline.

Here’s how I envision things working:

  • I push a new commit to a repo containing the manifest files for my application (what packages are needed, versions of my containers etc.)
  • a CI/CD pipeline is triggered (for example, a GitHub Action)
  • this CI/CD pipeline builds a new image in a systematic, reproducible way based on the manifests in the repo (I would like this process to be similar to Docker builds containers from Dockerfiles)
  • the CI/CD pipeline runs some tests on the resulting image
  • if all tests pass, the pipeline creates a new image and pushes it to my Mender server

Have any of you implemented a pipeline like this? How?

I could see some solutions ways to do it using tools like Ansible, Puppet, Chef, Saltstack etc… but I don’t quite have the complete picture figured out. I would love to learn from your experience.

For context, here’s some information about my project:

  • ubuntu server-based
  • runs a bunch of containers
  • also runs some services on bare metal

Thanks!

2

Assuming you’re using Yocto, kas and kas-docker go a long way to ensuring reproducible build environments, especially for CI.

I essentially implemented the first three steps of your plan using Gitlab CI and kas, but I’d assume it would be similar with Github actions. Because Yocto builds need a lot of processing power, I use the Gitlab docker-machine runner to spin up a really beefy cloud VM to do the build, which means I’m only paying for the beefy cloud VM while the build is taking place and builds are relatively quick (~1 hour).

edit: Reread your post and realised you’re building on top of Ubuntu server, so maybe the Yocto advice is not so useful, sorry. For a different project I automated building up an Ubuntu image using kickstarts. Maybe you could use those to install all the packages in your base image and then build and copy your containers over?

3

Is the golden image route a good starting point?

https://docs.mender.io/2.3/artifacts/debian-family/building-a-mender-debian-image

4

Thanks Sam. Kickstarts go some of the way to covering what I am looking for.

How did you run those jobs? I assume you made the images in virtual machine - what did you use to do so?

5

The recommended workflow for golden images suggests manually booting the device, logging in, and making all modifications. This is time-consuming, and error-prone, and unsuitable for CI/CD, which is what I am trying to accomplish.

6

HI @krystof indeed the Golden Master approach is a bit awkward and, as you say, time consuming. I have started playing with mkosi which is a wrapper around the Debian build utilities and supports a decent number of distros. So far it seems pretty useful

Drew

7

@drewmoseley mkosi looks like exactly what I need! Thanks a bunch!