Mender convert image from SATA storage golden image to NVMe storage target device: corrupt partition?

I have made a golden image on an x86 Ubuntu device that is running on a SATA 128 GB SSD. Let’s call this device A.
I have already tested a mender-convert’ed image, with the golden image of device A as input, on another device, device B, the same type on which the golden image was made. That worked without any issues, and I was able to see the device on hosted mender, which was great.
This was done using the following config

#
# Boot live USB and copy the data from internal storage (hdd, eMMC) e.g. for eMMC:
#
#     dd if=/dev/mmcblk0 of=/media/usb/centos.img bs=1M count=10000 status=progress
#
# Before running mender-convert we need to fix GPT partition table (as we didn't copy
# whole hdd). Run following command to fix GPT partition table on copied image:
#
#    sgdisk -e /media/usb/centos.img
#
# Converted with the following command:
#
#    MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert \
#   	 --disk-image input/centos.img \
#   	 --config configs/centos_x86-64_hdd_config \
#   	 --overlay rootfs_overlay_demo/
#
# and image must be then copied back after conversion to internal storage (when booted again from live USB) e.g. for hdd:
#
#    zcat /media/usb/centos-x86_64-mender.img.gz | sudo dd of=/dev/sda bs=1M status=progress
#
# after copy run again (to fix GPT partition table):
#    sgdisk -e /dev/sda
#
# for mender MB stands for MiB not MB


MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"

#50 GB to MiB is 52428800
MENDER_STORAGE_TOTAL_SIZE_MB=47684

# 10 GB to KiB is 10485760
# IMAGE_ROOTFS_EXTRA_SPACE=10485760
IMAGE_OVERHEAD_FACTOR="2.0"

# Install Mender Connect addon
MENDER_ADDON_CONNECT_INSTALL="y"

# Install Mender Configure addon
MENDER_ADDON_CONFIGURE_INSTALL="y"


# Nothing to copy
MENDER_COPY_BOOT_GAP="n"

#
# Resize the data partition to fill the remaining space, using parted, with systemd
#
function grow_data_partition() {
	log_info "Adding systemd init script to run parted and resize the data partition on boot"
	log_info "to fill all the available space on the storage media"
	run_and_log_cmd "sudo mkdir -p work/x86-64/etc/systemd/system/"
	run_and_log_cmd "sudo mkdir -p work/rootfs/etc/systemd/system/data.mount.wants/"
	cat <<- EOF > work/x86-64/etc/systemd/system/mender-grow-data.service
  [Unit]
  Description=Mender service to grow data partition size
  DefaultDependencies=no
  Before=data.mount
  Before=systemd-growfs@data.service

  [Service]
  Type=oneshot
  User=root
  Group=root
  ExecStart=/bin/growpart /dev/sda 4

  [Install]
  WantedBy=data.mount
EOF

	# Install
	run_and_log_cmd "cp work/x86-64/etc/systemd/system/mender-grow-data.service \
                  	work/rootfs/etc/systemd/system/"
	run_and_log_cmd "ln -sf work/rootfs/etc/systemd/system/mender-grow-data.service \
                      	work/rootfs/etc/systemd/system/data.mount.wants/"
}

function parse_secrets_env_set_hostname() {
	log_info "Adding systemd init script to parse secrets.env and set hostname"
	run_and_log_cmd "sudo mkdir -p work/x86-64/usr/local/bin"
	cat <<- 'EOF' > work/x86-64/usr/local/bin/parse_secrets_env_set_hostname.sh
#!/bin/bash
if [ -f /data/secrets.env ]; then
  HOSTNAME=$(grep -E '^HOSTNAME=' /data/secrets.env | cut -d'=' -f2)
  if [ -n "$HOSTNAME" ]; then
	hostnamectl set-hostname "$HOSTNAME"
	echo "Hostname set to $HOSTNAME"
  else
	echo "HOSTNAME variable not found or empty in secrets.env"
  fi
else
  echo "secrets.env not found in /data directory"
fi
EOF
	# Install
	run_and_log_cmd "cp work/x86-64/usr/local/bin/parse_secrets_env_set_hostname.sh \
                  	work/rootfs/usr/local/bin/"
	run_and_log_cmd "chmod +x work/rootfs/usr/local/bin/parse_secrets_env_set_hostname.sh"

	log_info "Adding systemd init script to run parse_secrets_env_set_hostname.sh on boot"
	run_and_log_cmd "sudo mkdir -p work/x86-64/etc/systemd/system/"
	run_and_log_cmd "sudo mkdir -p work/rootfs/etc/systemd/system/data.mount.wants/"
	cat <<- EOF > work/x86-64/etc/systemd/system/parse_secrets_env_set_hostname.service
  [Unit]
  Description=Mender service to grow data partition size
  DefaultDependencies=no
  Before=data.mount
  Before=systemd-growfs@data.service

  [Service]
  Type=oneshot
  User=root
  Group=root
  ExecStart=/usr/local/bin/parse_secrets_env_set_hostname.sh

  [Install]
  WantedBy=data.mount
EOF

	# Install
	run_and_log_cmd "cp work/x86-64/etc/systemd/system/parse_secrets_env_set_hostname.service \
                  	work/rootfs/etc/systemd/system/"
	run_and_log_cmd "ln -sf work/rootfs/etc/systemd/system/parse_secrets_env_set_hostname.service \
                      	work/rootfs/etc/systemd/system/data.mount.wants/"
}

PLATFORM_MODIFY_HOOKS+=(grow_data_partition)
PLATFORM_MODIFY_HOOKS+=(parse_secrets_env_set_hostname)

I now want to transplant this golden image to a different device, Device C, through mender-convert. Device C is also x86 but instead of 128 GB SATA storage has 256 GB NVMe storage. I thought that if I changed the config for mender-convert from:

MENDER_STORAGE_DEVICE_BASE=/dev/sda

to

MENDER_STORAGE_DEVICE_BASE=/dev/nvme0n1
# Partition number of boot partition
MENDER_BOOT_PART_NUMBER="p1"

# Partition number of root filesystem A
MENDER_ROOTFS_PART_A_NUMBER="p2"

# Partition number of root filesystem B
MENDER_ROOTFS_PART_B_NUMBER="p3"

# Partition number of persistent data partition
MENDER_DATA_PART_NUMBER="p4"

I though that this would be enough. I then ran this config through mender-convert with the golden image taken from device A and generated a new mender converted image. Using dd I then flashed this image on the NVMe storage of device C.

However, when I start up device C, I get the following errors on the screen attached to device C

the environment is corrupt,trying to boot from /dev/nvme0n1p2 in 10 seconds, but this is not garantueed to be a valid partition ...
returning from rootfs boot script, this could mean that the currently selected rootfs is corrupt. but there is no upgrade in progresss, so dropping to prompt in 10 seconds ...
entering rescue mode
grub rescue>

Is there a step that I am missing that is corrupting the partitions ? Or is it not possible to apply an image from a SATA SSD to an NVMe SSD?

Thanks in advance!

Just sharing our experience here: We build debian and ubuntu images that boot from USB stick, sata and nvme storage. We handled this by relying on partition uuids instead of device paths. We couldn’t get it to work reliably with device paths.

The relevant config fragment is

MENDER_PARTITION_SCHEME="gpt"
MENDER_ENABLE_PARTUUID="y"
MENDER_BOOT_PART="/dev/disk/by-partuuid/cb1c089e-3174-493e-b52b-a4f233a163a0"
MENDER_ROOTFS_PART_A="/dev/disk/by-partuuid/cb1c089e-3174-493e-b52b-a4f233a163a1"
MENDER_ROOTFS_PART_B="/dev/disk/by-partuuid/cb1c089e-3174-493e-b52b-a4f233a163a2"
MENDER_DATA_PART="/dev/disk/by-partuuid/cb1c089e-3174-493e-b52b-a4f233a163a3"

We had some initial trouble with the mender grubenv getting confused during boot. But I guess this has been fixed upstream a while ago.

1 Like