Mender-convert failed when converting Ubuntu 20.6 server disk image

General Discussions
1

I’m trying to convert a fresh Ubuntu 20 server installed with the following autoinstaller

#cloud-config
autoinstall:
  identity:
     hostname: ubuntu-server
     password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
     username: ubuntu
  kernel:
    package: linux-generic
  locale: en_US.UTF-8
  storage:
    config:
    - ptable: gpt
      path: /dev/vda
      wipe: superblock-recursive
      preserve: false
      name: ''
      grub_device: true
      type: disk
      id: disk-vda
    - device: disk-vda
      size: 1048576
      flag: bios_grub
      number: 1
      preserve: false
      grub_device: false
      type: partition
      id: partition-0
    - device: disk-vda
      size: 17176723456
      wipe: superblock
      flag: ''
      number: 2
      preserve: false
      grub_device: false
      type: partition
      id: partition-1
    - fstype: ext4
      volume: partition-1
      preserve: false
      type: format
      id: format-0
    - path: /
      device: format-0
      type: mount
      id: mount-0
  updates: security
  version: 1

Using KVM I can install the OS into an disk image. Without further modification of the OS I use ran it though mender-convert with following configurations

# Set the device file corresponding to the root filesystem partitions.
MENDER_STORAGE_DEVICE="/dev/sda"
MENDER_STORAGE_DEVICE_BASE="/dev/sda" ### TEST

# They can also bet defined manually
#MENDER_BOOT_PART="/dev/sda1"
#MENDER_ROOTFS_PART_A="/dev/sda2"
#MENDER_ROOTFS_PART_B="/dev/sda3"
#MENDER_DATA_PART="/dev/sda4"
#MENDER_BOOT_PART_MOUNT_LOCATION="/boot/efi"

# The size of the storage medium of the device. 119.2GB (set to 118GB)
#MENDER_STORAGE_TOTAL_SIZE_MB="120832"
MENDER_STORAGE_TOTAL_SIZE_MB="65536"

# Calculate rootfs size
#rootfs_part_sectors=$(((${MENDER_STORAGE_TOTAL_SIZE_MB(126701535232)} - ${MENDER_DATA_PART_SIZE_MB(75161927680)} - ${boot_part_sectors(536870912)} - ${overhead_sectors(65536)}) / 2))
# The size of the boot partition. 
MENDER_BOOT_PART_SIZE_MB="512"

#The size of each of the two rootfs filesystems, in KiB. 
#If this is 0, mender-convert will use the size of the filesystem content as a basis. 
#If the value is -1, mender-convert will use the maximum size that will fit inside the created partition. 
#The size is further modified by IMAGE_ROOTFS_EXTRA_SPACE and IMAGE_OVERHEAD_FACTOR.
# NOTE! By not setting this the finished image will think it only have ex 11GB space, eventhough the partition is 24GB
# (11GB filesystem 24GB partition size)
IMAGE_ROOTFS_SIZE="-1"


# The size of the root partitions

# Mender client version
#
# This is used to fetch the correct binaries
#
# Valid values are "latest" (default), "master" or a specific version
MENDER_CLIENT_VERSION="latest"

# Install Mender Connect addon
#
MENDER_ADDON_CONNECT_INSTALL="n"

# Install Mender Configure addon
#
MENDER_ADDON_CONFIGURE_INSTALL="n"


# The size of the Mender data partition. 30720
#MENDER_DATA_PART_SIZE_MB="71680" # 70 GB --> root parts (128-70)/2 = 24GB root
MENDER_DATA_PART_SIZE_MB="16384"
# https://www.unitconverters.net/data-storage/gb-to-mb.htm
# Enable/Disable automatically growing the filesystem to fill the physical storage device.
# This feature is utilizing x−systemd.growfs and will only work on systemd 242
# and newer and nothing will be done if an older systemd version is used.
MENDER_DATA_PART_GROWFS="y" 
#MENDER_DATA_PART_GROWFS="y"
# Our devices are using 245
# However Their system is somewhat wierd, since if we want to control the size of root partition.
# We can do that by setting total size and data size.


#MENDER_PARTITIONING_OVERHEAD_KB="32768"

# This is useful when you have large disk images,
# compressing them makes it easier to transfer them between a build server
# and a local machine, and saves space.
# Values: gzip(default)/lzma/none
MENDER_COMPRESS_DISK_IMAGE="gzip" 
MENDER_ARTIFACT_COMPRESSION="gzip"

# The partition alignment expressed in bytes.
MENDER_PARTITION_ALIGNMENT="8388608"

# Explicitly set the name of the generated update Artifact.
# Required for the conversion to succeed.
# However, should be specified on the command line, and not in the configuration.

MENDER_ARTIFACT_NAME="$(date +'%Y.%m.%d')"

#MENDER_FEATURES_ENABLE_append = " <FEATURE>"
#MENDER_FEATURES_DISABLE_append = " mender-bios"
#DISTRO_FEATURES="mender-grub mender-convert "


MENDER_FEATURES_ENABLE_append="mender-grub mender-image-uefi"
MENDER_FEATURES_DISABLE_append="mender-uboot mender-image-sd mender-bios"
MENDER_FEATURES_ENABLE="mender-grub mender-image-uefi"
MENDER_FEATURES_DISABLE="mender-uboot mender-image-sd"
# Specific Linux kernel boot arguments
#
# Typically you would read the content of /proc/cmdline on a "golden image"
# add the appropriate arguments here.
#
# This will override the defaults set by by grub-mender-grubenv, if not an
# empty string
MENDER_GRUB_KERNEL_BOOT_ARGS="console=tty0,115200n8 console=ttyO0,115200n8 console=ttyAMA0,115200n8" 

#MENDER_GRUB_KERNEL_IMAGETYPE=""

DEPLOY_DIR_IMAGE="/mender-convert/deploy"
MENDER_MACHINE="robotti_mender"


# Set the device type specified by the Artifact. (robotti 150D 150LR... etc.. This way it will be more easy to send specific updates)
# If left empty it will default to the value of '/etc/hostname'.
MENDER_DEVICE_TYPE="x86_64"

# Copy the boot gap, which is the area from sector 1 until first parition,
# this is done because some images might contain the bootloader embedded here
# and we need to copy the bootloader to the output image.
#
# In most cases you would like this enabled.
# Nothing to copy
MENDER_COPY_BOOT_GAP="n"


function platform_modify() {
    #
    # Make sure /lib64 exists since the Mender binary requires it.
    # Some systems put everything under /lib (ie Yocto) and a simple
    # symlink is enough to find everything Mender needs.
    #
    if [ ! -e work/rootfs/lib64 ]; then
        run_and_log_cmd "ln -s /lib work/rootfs/lib64"
    fi
}

function  write_mender_version() {
cat <<- EOF > work/rootfs/data/mender/artifact_version.txt
ARTIFACT_VERSION_PART_A=$MENDER_ARTIFACT_NAME
ARTIFACT_VERSION_PART_B=$MENDER_ARTIFACT_NAME
EOF
}
PLATFORM_MODIFY_HOOKS+=(write_mender_version)

function  check_system_permissions_and_groups_1() {
        
       
if ! [ "stat -c '%U' work/rootfs/" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/"
    else
        log_info "owner and group matches for /"
        run_and_log_cmd "stat -c '%U' work/rootfs/"  
fi


if ! [ "stat -c '%U' work/rootfs/etc" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc" 
    else
        log_info "owner and group matches for /etc"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc" 
fi


if ! [ "stat -c '%U' work/rootfs/etc/mender" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc/mender"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender" 
    else
        log_info "owner and group matches /etc/mender"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender" 
fi


}
PLATFORM_MODIFY_HOOKS+=(check_system_permissions_and_groups_1)


function  check_system_permissions_and_groups_2() {
        
       
if ! [ "stat -c '%U' work/rootfs/" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/" 
    else
        log_info "owner and group matches for /"
        run_and_log_cmd "stat -c '%U' work/rootfs/" 
fi


if ! [ "stat -c '%U' work/rootfs/etc" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc"
    else
        log_info "owner and group matches for /etc"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc"
fi


if ! [ "stat -c '%U' work/rootfs/etc/mender" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc/mender"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender"
    else
        log_info "owner and group matches /etc/mender"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender"
fi

}
OVERLAY_MODIFY_HOOKS+=(check_system_permissions_and_groups_2)


function  check_system_permissions_and_groups_3() {
        
       
if ! [ "stat -c '%U' work/rootfs/" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/" 
    else
        log_info "owner and group matches for /"
        run_and_log_cmd "stat -c '%U' work/rootfs/" 
fi


if ! [ "stat -c '%U' work/rootfs/etc" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc"
    else
        log_info "owner and group matches for /etc"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc"
fi


if ! [ "stat -c '%U' work/rootfs/etc/mender" == "root" ]; 
    then
        run_and_log_cmd "sudo chown root:root work/rootfs/etc/mender"
        log_info "owner and group is now:"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender"
    else
        log_info "owner and group matches /etc/mender"
        run_and_log_cmd "stat -c '%U' work/rootfs/etc/mender"
fi

}

#function  check_partition_sizes() {
        
#if [ "df --output=size work/roofs/dev/sda2 | awk 'NR==2 {print}'" < "sudo fdisk -h  work/roofs/dev/sda2" ]; 
#    then
#        log_info "Current size:"
#        run_and_log_cmd "df -h --out=size /dev/sda2"
        #run_and_log_cmd "sudo e2fsck -f /dev/sda2"
        #run_and_log_cmd "sudo resize2fs /dev/sda2" 
#        log_info "New size:"
 #       run_and_log_cmd "df -h --out=size /dev/sda2"
#    else
#        log_info "size match for sda2"
#        run_and_log_cmd "df -h --out=size /dev/sda2"
#fi

#if [ "df --output=size  /dev/sda3 | awk 'NR==2 {print}'" < "sudo fdisk -h  /dev/sda3" ]; 
#    then
#        log_info "Current size:"
#        run_and_log_cmd "df -h --out=size /dev/sda3"
#        #run_and_log_cmd "sudo e2fsck -f /dev/sda3"
#        #run_and_log_cmd "sudo resize2fs /dev/sda3" 
#        log_info "New size:"
#        run_and_log_cmd "df -h --out=size /dev/sda3"
#
#    else
#        log_info "size match for sda3"
#        run_and_log_cmd "df -h --out=size /dev/sda3"
#fi
#}
#PLATFORM_MODIFY_HOOKS+=(check_partition_sizes)

Then mender-convert is executed like the following

MENDER_ARTIFACT_NAME=release-2 ./mender-convert 
   --disk-image $INPUT_DISK_IMAGE 
   --config $CONFIG_1 
   --config $CONFIG_2 
   --overlay $CONFIG_OVERLAY 
   --overlay $DEVICE_OVERLAY

However mender-convert fails with the following

Running mender-convert-extract: [...]
[...]
2023-06-02 09:29:54 [INFO] [mender-convert-extract] Validating disk image
2023-06-02 09:29:54 [INFO] [mender-convert-extract] Disk parsed successfully
2023-06-02 09:29:54 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: gpt
2023-06-02 09:29:54 [INFO] [mender-convert-extract] PART 1: SIZE: 1M TYPE: 21686148-6449-6e6f-744e-656564454649
2023-06-02 09:29:54 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2023-06-02 09:29:54 [INFO] [mender-convert-extract] PART 2: SIZE: 16G TYPE: 0fc63daf-8483-4772-8e79-3d69d8477de4
2023-06-02 09:29:54 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning up...
Running mender-convert-modify: [...]
mount: .../mender-convert/work/boot: wrong fs type, bad option, bad superblock on /dev/loop33, missing codepage or helper program, or other error.
mender-convert-modify has finished. Cleaning up...
umount: work/boot: not mounted.
2023-06-02 09:30:16 [ERROR] [mender-convert] mender-convert failed
2023-06-02 09:30:16 [DEBUG] [mender-convert-extract] When running: (modules/disk.sh:63): run_and_log_cmd():  

	dd if=[...] of=work/part-2.fs skip=4096 bs=512 count=33548288 conv=sparse status=none

Tried to run the dd command manually and there’s no error. So I’m clueless of what could’ve went wrong, any advice is appreciated.