Approach to integrate existing Ubuntu servers into Mender

Board Integrations Debian family
1

The servers / Situation

I’ve got appliances in the field and new appliances being built running Ubuntu Server on a single disk with the following partitions:

  • UEFI/GRUB boot partition (1GB)
  • Rootfs Partition A containing kernel and all data (~400GB)
  • Rootfs Partition B empty ext4 partition (~30GB)

The B partition is only prepared to give us an upgrade path in the future as the current deployment is managed using a configuration management tool. We’re having scaling issues with that tool, there’s no fallback and that’s why we are considering using Mender.

What I want

  • Switch to Mender for the Firmware Update/UI/Signing/Azure IoT integration features and more
  • Upgrade my server firmware with fallback (A/B partition)
  • Outline a process to switch to Mender for servers described above that are already out in the field
  • Create a data partition and migrate existing data if possible (I could also upload the data before the migration to some bucket but if I don’t have to I’ll save time)
  • Not to brick any servers

Approach without Mender

  1. Create a new image
  2. Use dd/partclone to flash the image onto partition B
  3. Use “update-grub” maybe with custom configuration to make sure B is the new A and A the new fallback
  4. Boot into rootfs B partition
  5. Shrink rootfs A partition
  6. Create a data partition from the empty space
  7. Mount the data partition and make sure my applications work
  8. Create a new image for the rootfs partitions to include the mounts and other potential updates
  9. Flash the new image onto rootfs partition A again using dd/partclone
  10. Update grub again
  11. Boot into partition A, and done

Obviously, this will be much more error-prone and lacks many features we’d like to have down the road. But it will work without calling back any servers, which is most important.

Question / Approach with Mender (?)

Below is my idea. I’m pretty sure this won’t work as it doesn’t seem possible, if I read the documentation correctly, to install the client and migrate partition B + Boot only at first.
So I’m looking for a process I can follow to make this happen.
Is this possible or do I have to look for other options?

  1. Install mender software on my base image and convert the base image into a mender-compatible image using mender-convert
  2. Install mender software on my server’s A rootfs partition
  3. Update rootfs B partition and UEFI/GRUB partition only using mender software on A partition (Is this somehow possible?)
  4. Boot into rootfs B partition
  5. Shrink rootfs A partition
  6. Create a data partition from the empty space
  7. Mount the data partition and make sure my applications work
  8. Create a new mender image for the rootfs partitions to include the mounts
  9. Flash the new mender image onto rootfs partition A
  10. Boot into partition A, and done

Resources

I found only this one approach that would translate to my situation: How to migrate live devices to Mender? - #10 by erikhh
But it has been 4 years, and it seems error-prone, and supporting the process I describe above seems possible (looking at this as a beginner)

2

Hi @DELFAR45,

Thanks for reaching out! Yeah, migrating a brownfield fleet comes with a lot of challenges, so there is no one size fits all approach. Reading through your idea, I think it is actually not that far off.

To gauge some more details. The servers are all x86-64, I guess? And on a homogenous hardware platform, or is some variety to be expected here?

You can install the Mender Client straight into the existing system, and if you ship along a matching configuration then it should also be able to update to partition B. On UEFI/x86-64, the bootloader integration does not require patching, just getting the scripts properly installed. So a slightly customized step which brings that, together with update-grub should be able to do the required magic.

The rest of the steps are theoretically all straightforward, but in practise, as you know - anything can happen, especially when it comes to repartitioning. So my understanding is that yes, it can be done, and yes, it will required proper and careful preparation and testing.

If you’d like to proceed and need more in-depth evaluation, please get in touch with contact@mender.io so we can figure it out.

Greetz,
Josef

3

Hi @TheYoctoJester

Thanks for your answer!
The servers are all x86-64 for sure. There are 2 different server types, the only difference between those are 250GB disk vs. 500GB disk, a different GPU, and 16GB vs. 32GB of memory.

That’s great news, we’re on UEFI/x86-64 exclusively.
What exactly do you mean by a matching configuration?

Let’s say I manage to create a data partition, migrate all my data, and mount it from there and everything is working before my mender migration.
Do you mean I can then use such a server to create artifacts using mender-convert and the outcome would then be a matching configuration?
Do the start and end sectors of the partitions have to be an exact match for this to work?

I’m very glad to hear that I can install the client directly and expect it to be able to flash my B partition.
Could you elaborate on what scripts you mean by “just getting the scripts properly installed”?

Thanks again!

4

Hi @DELFAR45,

Sorry, this one fell a bit through the cracks. By “getting the scripts properly installed” I mean getting the pieces that take care of the A/B-switching into the UEFI setup.
Matching configuration means, are those boxes really similar, like same CPU, same RAM size, same boot process etc, and just differing in disk size, or are they all more like a bit alike, but in the end actually a heterogenous selection?
All in all you would then use mender-convert for creating artifacts then, correct.

Greetz,
Josef

5

Hi @TheYoctoJester

Thanks for the clarifications!
The hardware differences between the 2 supported servers are limited to the following:

  • 250GB disk / 500GB disk
  • a different GPU
  • 16GB / 32GB of memory

The UEFI/Grub setup will be crucial but can be tested well in a controlled environment before touching any servers in the field.

I would create 2 separate images for these 2 sets of hardware.
My main concern for the remote conversion is that the start and end sectors of the partitions have to be an exact match with the source images. Is that the case or could I use the artifacts coming from mender-convert to flash the image e.g. for rootfs-B onto any partition that is big enough to hold the data?

I’m asking because remotely adding these partitions in might not be as exact as the flashing of a new base disk image would be (for newly built ones).

Thanks