Failure during mender-convert: cannot overwrite directory

General Discussions
1

Running mender-convert 2.5.3 on a Debian image and the convert is failing while creating a symlink in the work rootfs. The client version on the image is 3.0.2. Here is the log at the failure point.

	sudo mkdir -p work/rootfs/data/mender
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:241): run_and_log_cmd():  

	sudo mkdir -p work/rootfs/var/lib
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:242): run_and_log_cmd():  

	sudo ln -sf /data/mender work/rootfs/var/lib
	ln: work/rootfs/var/lib/mender: cannot overwrite directory

2022-07-08 20:23:23 [e[0;31mERRORe[0m] [mender-convert] mender-convert failed
2022-07-08 20:23:23 [e[0;31mERRORe[0m] [mender-convert] mender-convert exit code: 1

The image to be converted is from a 128 GB SD card. I thought maybe it was a disk space issue on my Debian 10.8 VM, but I have over 700 GB of space available before running the convert. I’m stumped as to what else to check, as this failure is occurring while convert operates on the work rootfs. Any idea as to what is going on? The full convert log is shown below. Thx.

2022-07-08 20:01:15 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2022-07-08 20:01:15 [INFO] [mender-convert-extract] Using configuration file: ./work/override_compression_config
2022-07-08 20:01:15 [INFO] [mender-convert-extract] Using configuration file: configs/raspberrypi3_config
2022-07-08 20:01:15 [INFO] [mender-convert-extract] Validating disk image
2022-07-08 20:01:15 [INFO] [mender-convert-extract] Disk parsed successfully
2022-07-08 20:01:15 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: dos
2022-07-08 20:01:15 [INFO] [mender-convert-extract] PART 1: SIZE: 256M TYPE: 0xc
2022-07-08 20:01:15 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2022-07-08 20:01:18 [DEBUG] [mender-convert-extract] Running: (modules/disk.sh:64): run_and_log_cmd():  


	dd if=input/im3-be_mender_prep_image.img of=work/part-1.fs skip=8192 bs=512 count=524288 conv=sparse status=none
2022-07-08 20:01:18 [INFO] [mender-convert-extract] PART 2: SIZE: 118.9G TYPE: 0x83
2022-07-08 20:01:18 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
2022-07-08 20:23:15 [DEBUG] [mender-convert-extract] Running: (modules/disk.sh:64): run_and_log_cmd():  


	dd if=input/im3-be_mender_prep_image.img of=work/part-2.fs skip=532480 bs=512 count=249204736 conv=sparse status=none
2022-07-08 20:23:15 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2022-07-08 20:23:15 [INFO] [mender-convert-modify] Using configuration file: ./work/override_compression_config
2022-07-08 20:23:15 [INFO] [mender-convert-modify] Using configuration file: configs/raspberrypi3_config
2022-07-08 20:23:18 [INFO] [mender-convert-modify] Installing Mender client and related files
2022-07-08 20:23:18 [INFO] [mender-convert-modify] Installing Mender client version 3.0.2
2022-07-08 20:23:21 [DEBUG] [mender-convert-modify] Running: (modules/deb.sh:86): run_and_log_cmd_noexit():  


	wget -Nq https://downloads.mender.io/repos/debian/pool/main/m/mender-client/mender-client_3.0.2-1%2Bdebian%2Bbullseye_armhf.deb -P work/deb-packages
2022-07-08 20:23:21 [INFO] [mender-convert-modify] Successfully downloaded mender-client_3.0.2-1+debian+bullseye_armhf.deb
2022-07-08 20:23:21 [DEBUG] [mender-convert-modify] Running: (modules/deb.sh:113): run_and_log_cmd():  


	ar -xv /mender-convert/work/deb-packages/mender-client_3.0.2-1+debian+bullseye_armhf.deb
	x - debian-binary
x - control.tar.xz
x - data.tar.xz

2022-07-08 20:23:21 [DEBUG] [mender-convert-modify] Running: (modules/deb.sh:115): run_and_log_cmd():  


	sudo tar xJf data.tar.xz -C files
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (modules/deb.sh:118): run_and_log_cmd():  


	sudo rsync --archive --keep-dirlinks --verbose /tmp/tmp.trcSiR8Pq1/files/ /mender-convert/work/rootfs/
	sending incremental file list
./
etc/
etc/mender/
etc/mender/artifact_info
etc/mender/scripts/
etc/mender/scripts/version
lib/
lib/systemd/
lib/systemd/system/
lib/systemd/system/mender-client.service
usr/
usr/bin/
usr/bin/directory-artifact-gen
usr/bin/docker-artifact-gen
usr/bin/mender
usr/bin/single-file-artifact-gen
usr/share/
usr/share/dbus-1/
usr/share/dbus-1/system.d/
usr/share/dbus-1/system.d/io.mender.AuthenticationManager.conf
usr/share/dbus-1/system.d/io.mender.UpdateManager.conf
usr/share/doc/
usr/share/doc/mender-client/
usr/share/doc/mender-client/changelog.Debian.gz
usr/share/doc/mender-client/copyright
usr/share/doc/mender-client/examples/
usr/share/doc/mender-client/examples/demo.crt
usr/share/doc/mender/
usr/share/mender/
usr/share/mender/identity/
usr/share/mender/identity/mender-device-identity
usr/share/mender/inventory/
usr/share/mender/inventory/mender-inventory-bootloader-integration
usr/share/mender/inventory/mender-inventory-geo
usr/share/mender/inventory/mender-inventory-hostinfo
usr/share/mender/inventory/mender-inventory-network
usr/share/mender/inventory/mender-inventory-os
usr/share/mender/inventory/mender-inventory-provides
usr/share/mender/inventory/mender-inventory-rootfs-type
usr/share/mender/inventory/mender-inventory-update-modules
usr/share/mender/modules/
usr/share/mender/modules/v3/
usr/share/mender/modules/v3/deb
usr/share/mender/modules/v3/directory
usr/share/mender/modules/v3/docker
usr/share/mender/modules/v3/rpm
usr/share/mender/modules/v3/script
usr/share/mender/modules/v3/single-file

sent 7,817,508 bytes  received 624 bytes  15,636,264.00 bytes/sec
total size is 7,813,357  speedup is 1.00

2022-07-08 20:23:22 [INFO] [mender-convert-modify] Successfully installed mender-client_3.0.2-1+debian+bullseye_armhf.deb into /mender-convert/work/rootfs/
2022-07-08 20:23:22 [INFO] [mender-convert-modify] Creating the Mender Configure state folder in the data partition
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:132): run_and_log_cmd():  


	sudo mkdir -p work/rootfs/data/mender-configure
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:133): run_and_log_cmd():  


	sudo mkdir -p work/rootfs/var/lib
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:134): run_and_log_cmd():  


	sudo ln -sf /data/mender-configure work/rootfs/var/lib
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:240): run_and_log_cmd():  


	sudo mkdir -p work/rootfs/data/mender
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:241): run_and_log_cmd():  


	sudo mkdir -p work/rootfs/var/lib
2022-07-08 20:23:22 [DEBUG] [mender-convert-modify] Running: (./mender-convert-modify:242): run_and_log_cmd():  


	sudo ln -sf /data/mender work/rootfs/var/lib
	ln: work/rootfs/var/lib/mender: cannot overwrite directory

2022-07-08 20:23:23 [e[0;31mERRORe[0m] [mender-convert] mender-convert failed
2022-07-08 20:23:23 [e[0;31mERRORe[0m] [mender-convert] mender-convert exit code: 1
2

I did some digging in the source code and determined that I should let mender-convert install the client as part of conversion. Unfortunately, since we are on Debian stretch, I cannot use 3.0.2 client as stretch is not supported, only buster and bullseye are. During conversion the download of the client failed because of our old distribution.

So my next step is to go back to convert 2.5.2 and client 3.0.1, which is before the limitation of the two last Debian releases was implemented. Hopefully it will allow the client to be installed on the converted image and I can move forward. Ultimate goal is to get our product off of stretch, but need to get it mender converted first!

3

To close the loop on this, I was able to get a successful image conversion of a Debian stretch 9.4 golden image using mender-convert 2.5.0, and specifying mender-client 2.6.1 as the MENDER_CLIENT_VERSION in mender_convert_config. This version of client runs without issue on the converted stretch image.