Unable to create artefacts from Ubuntu 18.04 x64 image using mender-convert-next

Hello!

For the past few days i’ve been trying to use mender-convert-next on a Ubuntu 18.04 image. Initially I made contact with mender at contact@mender.io where i got the following instructions:

  1. Install: https://github.com/mendersoftware/mender-convert/tree/next

  2. Make an image. (They even gave me access to an image from their own s3 account, named: Ubuntu-Bionic-x86-64.img.gz. This image was subsequently extracted and put in an input directory.)

  3. To generate the mender artefacts, the following should be run:

     MENDER_ARTIFACT_NAME=release-1 ./mender-convert \
         --disk-image input/Ubuntu-Bionic-x86-64.img  \
        --overlay rootfs_overlay_demo \
        --config configs/qemux86-64_config
    

Quote mender support: "This should create two output Artifacts of interest:

  • An “.sdimg” file which you will use to provision a device
  • An “.mender” file that you will e.g upload to the Mender server to deploy OTA updates."

The process starts. but does not create any output. Here is my entire output from the terminal:

Terminal Output

fatal: not a git repository (or any of the parent directories): .git
Running mender-convert-extract: --disk-image input/Ubuntu-Bionic-x86-64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config
2019-11-06 15:06:03 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2019-11-06 15:06:03 [INFO] [mender-convert-extract] Using configuration file: configs/qemux86-64_config
2019-11-06 15:06:03 [INFO] [mender-convert-extract] Validating disk image
2019-11-06 15:06:03 [INFO] [mender-convert-extract] Disk parsed succesfully
2019-11-06 15:06:03 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: gpt
2019-11-06 15:06:03 [INFO] [mender-convert-extract] PART 1: SIZE: 256M TYPE: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
2019-11-06 15:06:03 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2019-11-06 15:06:03 [INFO] [mender-convert-extract] PART 2: SIZE: 1,9G TYPE: 4f68bce3-e8cd-4db1-96e7-fbcaf984b709
2019-11-06 15:06:03 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
Running mender-convert-modify: --disk-image input/Ubuntu-Bionic-x86-64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config
2019-11-06 15:06:07 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2019-11-06 15:06:07 [INFO] [mender-convert-modify] Using configuration file: configs/qemux86-64_config
2019-11-06 15:06:08 [INFO] [mender-convert-modify] Installing Mender client and related files
2019-11-06 15:06:09 [INFO] [mender-convert-modify] Found Linux kernel image: 

	work/rootfs/boot/vmlinuz-4.15.0-20-generic

2019-11-06 15:06:09 [INFO] [mender-convert-modify] Found initramfs image: 

	work/rootfs/boot/initrd.img-4.15.0-20-generic

2019-11-06 15:06:09 [INFO] [mender-convert-modify] Installing GRUB
2019-11-06 15:06:09 [INFO] [mender-convert-modify] GRUB EFI: bootx64.efi
2019-11-06 15:06:10 [INFO] [mender-convert-modify] Installing a custom /etc/fstab (see work/convert.log for more info)
2019-11-06 15:06:10 [INFO] [mender-convert-modify] Performing platform specific modifications (if any)
2019-11-06 15:06:10 [INFO] [mender-convert-modify] Applying rootfs overlay: rootfs_overlay_demo
mender-convert-modify has finished. Cleaning...
Running mender-convert-package: --disk-image input/Ubuntu-Bionic-x86-64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config
2019-11-06 15:06:10 [INFO] [mender-convert-package] Using configuration file: configs/mender_convert_config
2019-11-06 15:06:10 [INFO] [mender-convert-package] Using configuration file: configs/qemux86-64_config
2019-11-06 15:06:10 [WARN] [mender-convert-package] The allocated boot part size 40 MiB is too small.
2019-11-06 15:06:10 [WARN] [mender-convert-package] The actual boot part size is 256 MiB
2019-11-06 15:06:10 [WARN] [mender-convert-package] Will adjust MENDER_BOOT_PART_SIZE_MB automatically
2019-11-06 15:06:10 [WARN] [mender-convert-package] Considered adjusting the configuration file to avoid this message
2019-11-06 15:06:11 [INFO] [mender-convert-package] Creating a file-system image from: work/rootfs/data/
2019-11-06 15:06:11 [INFO] [mender-convert-package] Creating a file-system image from: work/rootfs/
2019-11-06 15:06:19 [INFO] [mender-convert-package] Copying root filesystem image to deploy directory
2019-11-06 15:06:20 [INFO] [mender-convert-package] Writing Mender artifact to: deploy/qemux86_64-release-1.mender
2019-11-06 15:06:20 [INFO] [mender-convert-package] This can take up to 20 minutes depending on which compression method is used
modules/run.sh: line 26: mender-artifact: command not found
mender-convert-package has finished. Cleaning..

Firstly, i noticed the “fatal: not a git repository (or any of the parent directories): .git”, which i could not make sense of. I also see that: mender-artifact: command not found. I tried git cloning the repo mender-artifact to see if i could spot any missing dependencies without any luck.

Therefore, in the lack of a solution, I was hoping someone could help me work through this issue? And in general, explain how to configure mender to make a “.mender” and “.sdimg” from a Ubuntu 18.04 x64 image.

Best regards,
TL

Edit: @mirzak: formatting.

Indeed, this is the root cause of it failing. You need to have the mender-artifact binary installed on your PC and it needs to available in PATH.

You can download a pre-built binary of mender-artifact here:

https://docs.mender.io/2.2/downloads

One alternative, to avoid installing this is to utilize the “Dockerized” version of the mender-convert tool, which will run the mender-convert tool inside a Docker container which has all the dependencies installed already. Instructions can be found here,

https://github.com/mendersoftware/mender-convert/tree/next#docker-environment-for-mender-convert

So in you case, once you have built the docker image you would run,

 MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert \
     --disk-image input/Ubuntu-Bionic-x86-64.img  \
    --overlay rootfs_overlay_demo \
    --config configs/qemux86-64_config

The only difference from you original command is, mender-convertdocker-mender-convert

And this error,

is because we try to parse the git revision of of the mender-convert repository which is output when using mender-convert --version.

I suspect that you have downloaded the mender-convertrepository as a tar archive, instead of running e.g:

git clone -b next https://github.com/mendersoftware/mender-convert.git

Though this is something that we should fix, avoiding to print this confusing message.

You were absolutely right, running it from within docker solved it. And yes, i did download the .zip file, which explains the: “fatal: not a git repository (or any of the parent directories): .git”.

I now have a “.sdimg” and “.mender”-file from my Ubuntu image. Thank you!

Hello,

I’ve been trying to use mender-convert-next on a Ubuntu 18.04 image. I have downloaded Ubuntu 18.04 image from https://ubuntu.com/download/desktop

I have converted ubuntu-18.04.3-desktop-amd64.iso into ubuntu-18.04.3-desktop-amd64.img using below command:

dd if=ubuntu-18.04.3-desktop-amd64.iso of=ubuntu-18.04.3-desktop-amd64.img

#Steps followed using mender-convert-next git repo on a Ubuntu 18.04 image

git clone -b next https://github.com/mendersoftware/mender-convert.git
cd mender-convert
mkdir -p input
#Copy the ubuntu-18.04.3-desktop-amd64.img file under input folder
./scripts/bootstrap-rootfs-overlay-demo-server.sh --output-dir ${PWD}/rootfs_overlay_demo --server-ip 10.77.164.6
sudo su
./docker-build

MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert --disk-image input/ubuntu-18.04.3-desktop-amd64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config

user@user-Veriton-Series:$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

I have tried conversion both with docker & without docker as well but getting same error as below.

## Terminal Output & ERROR ##

user@user-Veriton-Series:~/Rohit/mender/mender-convert$ MENDER_ARTIFACT_NAME=release-1 ./mender-convert --disk-image input/Ubuntu-Bionic-x86-64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config
Running mender-convert-extract: --disk-image input/Ubuntu-Bionic-x86-64.img --overlay rootfs_overlay_demo --config configs/qemux86-64_config
2019-11-26 15:12:41 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2019-11-26 15:12:41 [INFO] [mender-convert-extract] Using configuration file: configs/qemux86-64_config
2019-11-26 15:12:41 [INFO] [mender-convert-extract] Validating disk image
2019-11-26 15:12:41 [INFO] [mender-convert-extract] Disk parsed succesfully
2019-11-26 15:12:41 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: dos
2019-11-26 15:12:41 [INFO] [mender-convert-extract] PART 1: SIZE: 2G TYPE: 0x0
2019-11-26 15:12:41 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2019-11-26 15:13:31 [INFO] [mender-convert-extract] PART 2: SIZE: 2.4M TYPE: 0xef
2019-11-26 15:13:31 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
dd: invalid number: ‘-1’

Could you please assist me to resolve this above dd error.

Can you run fdisk -l ubuntu-18.04.3-desktop-amd64.img and share the output here.

user@user-Veriton-Series:~/Rohit/mender/Backup_image$ fdisk -l ubuntu-18.04.3-desktop-amd64.img
Disk ubuntu-18.04.3-desktop-amd64.img: 2 GiB, 2082816000 bytes, 4068000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5092863d

Device Boot Start End Sectors Size Id Type
ubuntu-18.04.3-desktop-amd64.img1 * 0 4067999 4068000 2G 0 Empty
ubuntu-18.04.3-desktop-amd64.img2 3989132 3994059 4928 2.4M ef EFI (FAT

Yeah, this does not make any sense.

The command you ran:

dd if=ubuntu-18.04.3-desktop-amd64.iso of=ubuntu-18.04.3-desktop-amd64.img

Will not convert the .iso image to an disk image, and this simply makes a copy/rename of the `iso’ image.

mender-convert will not work with iso images and this is probably not something that you would like to use with Mender. Images downloaded from https://ubuntu.com/download/desktop are installer images.

You can take a look at the recommended workflow for mender-convert here,

https://docs.mender.io/2.2/artifacts/debian-family#recommended-workflow

@mirzak, Thanks for quick response.

Could you please share any source (S3 account or URL) for getting Ubuntu 18.04 x86 desktop for .img image i.e. Ubuntu-Bionic-x86-64.img.gz.

Note that this is simply a test image that we use while running tests on the mender-convert tool,

https://d1b0l86ne08fsf.cloudfront.net/mender-convert/images/Ubuntu-Bionic-x86-64.img.gz

Generated using:

mkosi -d ubuntu -r bionic -t gpt_ext4 -b --checksum --password password -o image.raw

@mirzak, I now have a “.sdimg” and “.mender” files from the provided test Ubuntu image.

Note: Without sd-card boot, By using qemu with below command, I am able to login successful without any error.
sudo qemu-system-x86_64 -enable-kvm -m 512 -smp 2 -bios /usr/share/ovmf/OVMF.fd -drive format=raw,file=qemux86_64-release-1.sdimg

When I booted above provided ubuntu test image into sd-card (16gb) & connect with X86 64bit m/c and also was able to login successfully after entering hostname/password into terminal (without GUI) But when I booted same with mender converted .sdimg file into sd-card & connect with X86 64bit m/c and I am getting error on terminal as “[FAILED] Failed to start Switch Root.”.

Is there any idea about this error?

Where exactly are you getting this error? Can you share a more complete log?

@mirzak, Attached error screenshot after boot .sdimg (on 16GB sdcard) on X86 m/c for reference.