Can't create Mender images for Ubuntu Server x64

General Discussions
1

I’m trying to build an Ubuntu Server x64 (20.04) image for this board, and I keep getting this error from mender-convert:

mount: /mender-convert/work/boot: wrong fs type, bad option, bad superblock on /dev/loop5, missing codepage or helper program, or other error.

Here’s what I’ve done:

1. Installed Ubuntu Server x64 20.04 on the board

2. Created an .img file from the disk like so:

sudo dd if=/dev/sdc of=ubuntu-server-image-from-HW-2020-05-07.img bs=1M conv=fdatasync

3. Run mender-convert

After several failed attempts of my own, I found the generic_x86-64_hdd_config file in the master branch of mender-convert, and tried using that, modifying just the MENDER_STORAGE_TOTAL_SIZE_MB to match my disk, and noting that it was indeed mounted at /dev/sda on my target device (I booted the device to check).

Here is the output:

$ MENDER_ARTIFACT_NAME=artifact-v0.0.1 ./docker-mender-convert \
      --disk-image input/ubuntu-server-image-from-HW-2020-05-07.img --config ./configs/krystofs-config-for-mender-convert 
Running mender-convert --disk-image input/ubuntu-server-image-from-HW-2020-05-07.img --config ./configs/krystofs-config-for-mender-convert
Running mender-convert-extract: --disk-image input/ubuntu-server-image-from-HW-2020-05-07.img --config ./configs/krystofs-config-for-mender-convert
2020-05-09 00:11:12 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2020-05-09 00:11:12 [INFO] [mender-convert-extract] Using configuration file: ./configs/krystofs-config-for-mender-convert
2020-05-09 00:11:12 [INFO] [mender-convert-extract] Validating disk image
2020-05-09 00:11:12 [INFO] [mender-convert-extract] Disk parsed successfully
2020-05-09 00:11:12 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: gpt
2020-05-09 00:11:12 [INFO] [mender-convert-extract] PART 1: SIZE: 1M TYPE: 21686148-6449-6e6f-744e-656564454649
2020-05-09 00:11:12 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2020-05-09 00:11:12 [INFO] [mender-convert-extract] PART 2: SIZE: 119.2G TYPE: 0fc63daf-8483-4772-8e79-3d69d8477de4
2020-05-09 00:11:12 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning up...
Running mender-convert-modify: --disk-image input/ubuntu-server-image-from-HW-2020-05-07.img --config ./configs/krystofs-config-for-mender-convert
2020-05-09 00:18:25 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2020-05-09 00:18:25 [INFO] [mender-convert-modify] Using configuration file: ./configs/krystofs-config-for-mender-convert
mount: /mender-convert/work/boot: wrong fs type, bad option, bad superblock on /dev/loop5, missing codepage or helper program, or other error.
mender-convert-modify has finished. Cleaning up...
umount: work/boot: not mounted.
2020-05-09 00:18:25 [ERROR] [mender-convert] mender-convert failed
2020-05-09 00:18:25 [DEBUG] [mender-convert-extract] When running: (modules/disk.sh:61): run_and_log_cmd():  

	dd if=input/ubuntu-server-image-from-HW-2020-05-07.img of=work/part-2.fs skip=4096 bs=512 count=250062848 conv=sparse status=none
2020-05-09 00:18:25 [ERROR] [mender-convert] mender-convert exit code: 32

Any idea on what I need to do to successfully create the artifacts?

Thanks,
Krystof

2

Hi,

I have added generic_x64_hdd_config and I hit same issue and problem was that GPT table was not correct. Can you please try to run on copied image from hdd command:

sgdisk -e ubuntu-server-image-from-HW-2020-05-07.img

Please look at help here: config steps are described here.

Also note LVM partitioning is not supported so you need to manually setup partitioning on your host OS. Hope this help. Pls let me know ;). Thanks.

3

Thanks @MarekBelisko. Here’s the output of the sgdisk command:

$ sgdisk -e ubuntu-server-image-from-HW-2020-05-07.img
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

Following the steps from the config file / guide you linked, I tried running mender-convert again, but the result was the same as before.

I also tried running partprobe on the image as suggested by the sgdisk output, and running mender-convert after that. The result is always the same as I wrote my original post.

For completeness, here are the contents of my config file:

MENDER_STORAGE_TOTAL_SIZE_MB="122104"
MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"

Any ideas?

I am not sure what you mean about the partitioning on the host OS. Do you mean that before I first create the .img file, that disk should not use LVM? The disk from which I created the .img. file does not use LVM.

4

@krystof can you please post output of:
fdisk -l ubuntu-server-image-from-HW-2020-05-07.img or partx -l ubuntu-server-image-from-HW-2020-05-07.img

By partitioning on host os I mean when you install Ubuntu on your test PC how do you partitioned disk (manually or automatically by installer)? I have experience with CentOS that automatic partitioner use LVM and when try to convert such image I have same issues (that mender-convert cannot detect partitions and mount them). Output of fdisk or partx should help better to understand what can be wrong. Thanks.

5

Thanks @MarekBelisko. Here’s the output:

$ fdisk -l ubuntu-server-image-from-HW-2020-05-07.img
Disk ubuntu-server-image-from-HW-2020-05-07.img: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: ABD2060A-7176-4254-A6DE-8E8EDCA92627

Device                                      Start       End   Sectors   Size Type
ubuntu-server-image-from-HW-2020-05-07.img1  2048      4095      2048     1M BIOS boot
ubuntu-server-image-from-HW-2020-05-07.img2  4096 250066943 250062848 119.2G Linux filesystem

and

$ partx -l ubuntu-server-image-from-HW-2020-05-07.img
# 1:      2048-     4095 (     2048 sectors,      1 MB)
# 2:      4096-250066943 (250062848 sectors, 128032 MB)

What do you think?

6

AFAIK mender look for boot partition (where efi is installed). Above partition layout don’t have efi partition so could you please try to install ubuntu with separate efi (can be ~200M) and separate rootfs (e.g. 15G). Then you don’t need to copy whole HDD just e.g. 16G (more convenient when transferring image from test device to PC for conversion). Thanks.

7

Interesting. So does this mean that we assume EFI only with mender-convert? I recently did an Ubuntu install on my desktop system and it ended up without EFI support. I had to manually create the EFI partition and install grub-efi using boot-repair to get it to boot un EFI mode.

@mirzak any thoughts here?

Drew

8

Hm, the image posted by @krystof seems to be BIOS (legacy boot). This we do not support yet in mender-convert.

We run tests on the following Ubuntu image (EFI boot):

$ fdisk -l Ubuntu-Bionic-x86-64.img 
Disk Ubuntu-Bionic-x86-64.img: 2,12 GiB, 2270532608 bytes, 4434634 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3B4BAD4E-D5CB-5F40-B03D-7BBF5F306C3E

Device                     Start     End Sectors  Size Type
Ubuntu-Bionic-x86-64.img1   2048  526335  524288  256M EFI System
Ubuntu-Bionic-x86-64.img2 526336 4434600 3908265  1,9G Linux root (x86-64)

Rebuilding the “golden image” image with EFI support might help, if the hardware supports EFI. Any modern hardware should support it and the linked hardware seems “modern”

9

As an aside, if you get a chance to play with mkosi as recommended in a different discussion, as a replacement for golden image, then mkosi only produces EFI images which I can confirm having had success processing with mender-convert. (assuming your hardware supports EFI)

10

Thanks everyone for your advice. The good news is that I’ve managed to get mender-convert to work.

I created a new image using mkosi, and confirmed that it was using EFI boot. I converted the image to an .img file, and running mender-convert on it succeeded.

From there, I wasn’t quite sure how to copy that image to the device, since the official documentation seems to only talk about Yocto, while I am using Ubuntu.

I tried following the steps suggested in configs/generic_x86-64_hdd_config

I copied the data to the disk with

zcat mkosi-image-x86_64-mender.img.gz | sudo dd of=/dev/sdc bs=1M status=progress

Running fdisk on it showed some partition issues:

$ sudo fdisk -l /dev/sdc
[sudo] password for krystof:
GPT PMBR size mismatch (250068991 != 250069679) will be corrected by write.
The backup GPT table is not on the end of the device. This problem will be corrected by write.
Disk /dev/sdc: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Disk model: 2115            
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 411C0010-B3FA-498A-BD46-09B615FD9042

Device         Start       End   Sectors  Size Type
/dev/sdc1      16384    540671    524288  256M EFI System
/dev/sdc2     540672 125157375 124616704 59.4G Linux filesystem
/dev/sdc3  125157376 249774079 124616704 59.4G Linux filesystem
/dev/sdc4  249774080 250036223    262144  128M Linux filesystem

I tried to correct these with sgdisk, again as suggested by generic_x86-64_hdd_config:

$ sudo sgdisk -e /dev/sdc
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.

This did remove the warning:

$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 119.25 GiB, 128035676160 bytes, 250069680 sectors
Disk model: 2115            
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 411C0010-B3FA-498A-BD46-09B615FD9042

Device         Start       End   Sectors  Size Type
/dev/sdc1      16384    540671    524288  256M EFI System
/dev/sdc2     540672 125157375 124616704 59.4G Linux filesystem
/dev/sdc3  125157376 249774079 124616704 59.4G Linux filesystem
/dev/sdc4  249774080 250036223    262144  128M Linux filesystem

…but the device does not boot.

I am contacting AdvanTech to confirm that the board I have does indeed support EFI boot.

Do any of you see anything that I’ve missed that could be a problem?

Thanks again!

11

The manual for your device does imply that it should be set to uefi boot by default, unless some of the bios settings have been changed to legacy boot.

Any boot messages output to your screen or serial port?

12

I do not see anything wrong in your workflow @krystof, and as @dellgreen pointed out I would check out that the firmware settings are set to support EFI

This is correct, but I would also add an sync call at the end, e.g:

zcat mkosi-image-x86_64-mender.img.gz | sudo dd of=/dev/sdc bs=1M status=progress && sync
13

@mirzak that’s a good catch, I recall having created the odd unbootable sdcard and usb stick in the past by forgetting the sync command, I believe newer dd versions also have a built in sync option also which I have used in the past with success.

@krystof also if you use bmaptools with the bmap file produced by the mender-convert process to image your device, you won’t need to do zcat/dd/sync combo and your imaging will also be faster.

14

Yeah, you can also do:

# Same as running 'sync' after the call
zcat mkosi-image-x86_64-mender.img.gz | sudo dd of=/dev/sdc bs=1M status=progress conv=fdatasync

or

# This will sync on each block write
zcat mkosi-image-x86_64-mender.img.gz | sudo dd of=/dev/sdc bs=1M status=progress oflag=dsync
15

Can you boot image which is generated by mkosi (without mender-convert)? Thanks.

16

OK, so the issue isn’t in the hardware/firmware. As @MarekBelisko suggested I tried booting directly the image generated by mkosi, and that works fine (after adjusting the boot order in BIOS).

With the mender-convert created image, after adjusting the boot order in BIOS in the same way that I did for the mkosi image, I get a GRUB screen that I am unfamiliar with. It does not seem to offer an easy way to boot anything:

grub
grub4032×1656 852 KB

I did add a sync call to dd command as @mirzak suggested, and also fixed the partition issue after running dd with sudo sgdisk -e /dev/sdc.

I am not sure what to do from here.

Any suggestions?

17

@krystof can you please try this commit mender Some x86 seems to doesn’t work without this (at least from my experience). Pls use this repo of cherry pick commit to mender-convert repo and try to convert and test. Thanks.

18

@MarekBelisko I cherry-picked your commit, and things changed, though I am unsure whether for the better or worse:

2020-05-13-GRUB-1
2020-05-13-GRUB-13889×1893 1.54 MB

which after 10 seconds goes to this:

2020-05-13-GRUB-2
2020-05-13-GRUB-23910×2211 1.85 MB

which does not really let me do anything.

Any ideas?

19 
   lock:OK
   lock:OK

looks good but what is issue with error: not a regular file. needs to be debuged.

I check in grub source and only places are those:

gg 'not a regular file'
grub-core/fs/btrfs.c:2070:      return grub_error (GRUB_ERR_BAD_FILE_TYPE, N_("not a regular file"));
grub-core/fs/fshelp.c:307:    return grub_error (GRUB_ERR_BAD_FILE_TYPE, N_("not a regular file"));
grub-core/fs/jfs.c:835:      grub_error (GRUB_ERR_BAD_FILE_TYPE, N_("not a regular file"));
grub-core/fs/zfs/zfs.c:3804:      return grub_error (GRUB_ERR_BAD_FILE_TYPE, N_("not a regular file"));

@krystof which fs do you use for rootfs? ext4 or something else?

@mirzak @drewmoseley ideas where this can come from?

Thanks.

20

@MarekBelisko yes, I use ext4 as the rootfs.

The images are created by mkosi based on this config file:

[Distribution]
Distribution=ubuntu
Release=focal

[Output]
Format=raw_ext4
Bootable=yes

[Partitions]
RootSize=20G