Mender-convert fails on dd step

General Discussions
, ,
1

Hi! I’ve been trying out Mender for the first time, and I’m having some trouble converting an image I generated with mkosi (Ubuntu 20.04 with some extra stuff installed). The image boots fine when I launch it from a USB stick, but when I try to run mender-convert on it it fails and I have not been able to find out why. Has anyone experienced an issue like this before?

MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert     --disk-image input/ubuntu_cuda.img     --config configs/generic_x86-64_hdd_config     --overlay rootfs_overlay_demo/
Running mender-convert --disk-image input/ubuntu_cuda.img --config configs/generic_x86-64_hdd_config --overlay rootfs_overlay_demo/
Running mender-convert-extract: --config ./work/override_compression_config --disk-image input/ubuntu_cuda.img --config configs/generic_x86-64_hdd_config --overlay rootfs_overlay_demo/
2022-01-14 07:51:58 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2022-01-14 07:51:58 [INFO] [mender-convert-extract] Using configuration file: ./work/override_compression_config
2022-01-14 07:51:58 [INFO] [mender-convert-extract] Using configuration file: configs/generic_x86-64_hdd_config
2022-01-14 07:51:58 [INFO] [mender-convert-extract] Validating disk image
2022-01-14 07:51:58 [INFO] [mender-convert-extract] Disk parsed successfully
2022-01-14 07:51:58 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: gpt
2022-01-14 07:51:58 [INFO] [mender-convert-extract] PART 1: SIZE: 512M TYPE: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
2022-01-14 07:51:58 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2022-01-14 07:51:59 [INFO] [mender-convert-extract] PART 2: SIZE: 15G TYPE: 4f68bce3-e8cd-4db1-96e7-fbcaf984b709
2022-01-14 07:51:59 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning up...
Running mender-convert-modify: --config ./work/override_compression_config --disk-image input/ubuntu_cuda.img --config configs/generic_x86-64_hdd_config --overlay rootfs_overlay_demo/
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: ./work/override_compression_config
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: configs/generic_x86-64_hdd_config
mount: /mender-convert/work/rootfs: mount(2) system call failed: File exists.
mender-convert-modify has finished. Cleaning up...
umount: work/rootfs: not mounted.
2022-01-14 07:53:13 [ERROR] [mender-convert] mender-convert failed
2022-01-14 07:53:12 [DEBUG] [mender-convert-extract] When running: (modules/disk.sh:64): run_and_log_cmd():  

	dd if=input/ubuntu_cuda.img of=work/part-2.fs skip=1050624 bs=512 count=31459295 conv=sparse status=none
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: ./work/override_compression_config
2022-01-14 07:53:12 [INFO] [mender-convert-modify] Using configuration file: configs/generic_x86-64_hdd_config
2022-01-14 07:53:13 [ERROR] [mender-convert] mender-convert failed
2022-01-14 07:53:13 [ERROR] [mender-convert] mender-convert exit code: 32

My config looks like this, only modified some values from default:

MENDER_ADDON_CONNECT_INSTALL='y'
MENDER_ADDON_CONFIGURE_INSTALL='y'

MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"

MENDER_BOOT_PART_SIZE_MB=512
MENDER_STORAGE_TOTAL_SIZE_MB=475000
IMAGE_ROOTFS_SIZE="-1" 

# Nothing to copy
MENDER_COPY_BOOT_GAP="n"

function platform_modify() {
    #
    # Make sure /lib64 exists since the Mender binary requires it.
    # Some systems put everything under /lib (ie Yocto) and a simple
    # symlink is enough to find everything Mender needs.
    #
    if [ ! -e work/rootfs/lib64 ]; then
        run_and_log_cmd "ln -s /lib work/rootfs/lib64"
    fi
}
2

Could it be related to this? It seems my system does not unmount loop devices correctly.

3

I’ve tried this on a few different machines (all Ubuntu 20.04), and the same issue happens on all of them. I also tried downloading other images from working cases in this forum, and they fail for me too. Any idea what I can do to fix this?

4

64bit build? I ask because 64bit is starting to seem like a common thread. Also, loop devices are rather important in the conversion

5

Yes, 64 bit. Actually it seems an isolated problem to my machine. When I was trying on other devices, I had the USB drive that the image was on still plugged in, which made it fail. Once I unplugged the USB, mender-convert finished successfully. Still I have not been able to run it on my machine, but that is a minor issue now.

6

Still I have not been able to run it on my machine, but that is a minor issue now.

I’m sorry, I do not understand. What are you having trouble running on your machine, and when you say your machine are you referring to your workstation where you performed the conversion of the image?

7

Sorry, it is a bit of a confusing situation.

What I did:

  • Generate disk image using mkosi (on my workstation): OK
  • Write this image to usb stick to check if it is bootable: OK
  • Run mender-convert on my workstation (Ubuntu 20.04, LVM): fails with the error in the original post
  • Run mender-convert on a different machine (actually the machine I want to apply mender to, XUbuntu 20.04, btrfs): fails with the same error when the bootable USB stick is plugged in, works succesfuly when unplugged

So by not using my own workstation, I was able to run mender-convert successfully.

8

Ok, so my next question, does the image you created on the non-workstation system actually boot when you write the image output by mender-conversion in the deploy dir? If so, awesome, you’d just now have to figure out why your workstation fails (but the loop device issue might be part of it).

If you write the image created by the conversion process and it won’t boot, do you have any output on the screen of the device?

9

Even though I can run mender-convert on the non-workstation, the resulting image fails to boot from GRUB. But it turns out the whole thing was a an issue with the filesystem on my image being “btrfs”. Once I created an image with “ext4” format everything worked smoothly on my workstation and I could boot from the the generated image. It’s still a mystery why it mender-convert was working on the non-workstation, but at least everything is working now.

10

Good to hear. I wish you luck getting everything fully working. I’ve not been able to get Ubuntu 20+ to boot after running a conversion of the image through mender-convert. Even 32bit wouldn’t work. I had hoped your issue was somehow related but I don’t believe it does. I will say however, thank you for your thread because it reminded me of mkosi, which may end up helping now.