Can't create Mender images for Ubuntu Server x64

General Discussions
#23

It appears the problem was indeed with the partition sizes. I intentionally set the data partition size too small to let mender-convert figure out the appropriate sizes for the root partitions, and the resulting image booted.

This is the config that worked:

MENDER_STORAGE_TOTAL_SIZE_MB="122104"
IMAGE_ROOTFS_SIZE="20971520"

# In KiB
IMAGE_ROOTFS_EXTRA_SPACE="0"

IMAGE_OVERHEAD_FACTOR="1.0"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="61440"

MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"

Thanks again for your help!

#24

:slight_smile: @krystof ok thanks for sharing back. Can you please update original post with those info? Also do you use patch I’ve shared with you? If so we need to find why it helps and integrate it to mender-convert to have image working out of then box. Thanks and have a nice day.

#25

Thanks for the detailed reports @krystof,

To clear some things up on the variables and sizes.

What you list here as 20GB, is the partition size,

18-04_20G.img2 526336 42471390 41945055   20G Linux root (x86-64)

There is a difference between partition size and file system image size. mender-convert does not care much about input partition sizes and will try to re-calculate this based on various variables.

The rootfs partition size is always calculated based on this formula:

rootfs_part_sectors=$(((${disk_image_total_sectors} - ${data_part_sectors} - \
  ${boot_part_sectors} - ${overhead_sectors}) / 2))

which can be translated to (roughly):

rootfs_part_mb=$(( ${MENDER_STORAGE_TOTAL_SIZE_MB} - ${MENDER_DATA_PART_SIZE_MB} - \
  ${MENDER_DATA_PART_SIZE_MB} ))

You can specific the rootfs image size using the IMAGE_ROOTFS_* variables. This is an optimization, as the rootfs image size impacts what is transferred between the Mender server and the Mender client, and this can be less then the actual partition size.

The default rootfs image size will be actual rootfs content * 1.5 (50 % free space relative to actual occupied storage).

If you want the rootfs image size to be the same as the rootfs partition size you can just set:

IMAGE_ROOTFS_SIZE = "-1"

Further, the following is not aligned,

MENDER_STORAGE_TOTAL_SIZE_MB="122104"

Eg I would set it to 122880 which is aligned to 1024 if you want to represent 120GB. This a very large disk :).

#26

@mirzak @drewmoseley seems patch which I’ve created for mender-convert does also for this case. I think we need to investigate why some BIOS work without this patch (Intel NUC) and some doesn’t.

@krystof what is your target HW, can you please share? Thanks.

#27

@MarekBelisko target hardware is this: https://www.advantech.com/products/68ccaea2-9ff5-4f85-97f2-3d11244b0a08/aimb-286/mod_ad53f4ec-0d78-4995-aeed-337921f3c837

@mirzak the disk size is strange to me also, but I got that number from fdisk:

 $ sudo fdisk -l /dev/sdc 
Disk /dev/sdc: 119.25 GiB, 128035676160 bytes, 250069680 sectors

If I’m doing the calculation right, 128035676160 bytes = 125034840 KiB = 122104.3359375 MiB??? This is very strange indeed. I am what the appropriate value for MENDER_STORAGE_TOTAL_SIZE_MB should be.

I am still getting strange / inconsistent behavior. For example, I tried installing different packages in the base image (generated by mkosi), and even when the image would consistently boot in a VM, some images created by mender-convert wouldn’t. Today I also tried the “golden-image” approach, manually creating partitions including an EFI boot partition, and then using mender-convert on that image… and the result wouldn’t boot either (it showed the “Dropping to grub prompt for unknown reason. Should never get here” message).

@mirzak could you please suggest what values I should set for all the mender sizing variables, given the disk I have and that I want about rootfs partitions of about 30 GB?

Thanks!

#28

@mirzak the disk size is strange to me also, but I got that number from fdisk:

This is better to read from the actual block device on the device, e.g

$ cat /sys/block/sda/size 
62521344

In regards to variables, something like this should work:

# 62 GB total size
MENDER_STORAGE_TOTAL_SIZE_MB = "63488"
MENDER_BOOT_PART_SIZE_MB="256"

# As you are working with very large partitions, I would leave this at 256MB here and this can be resized later on the device, e.g on first boot to occupy rest of the free space available on the disk. 
MENDER_DATA_PART_SIZE_MB="256"

IMAGE_ROOTFS_SIZE = "-1"

This should give you rootfs parts of ~30GB.

#29

@krystof I could see you have generated the .img from mkosi, could you please share me the steps or how can i convert the generated raw image files?

#30

Thanks @mirzak.

I was able to get Ubuntu to start booting, but it didn’t succeed:

boot-problem
boot-problem3889×2652 2.27 MB

Any ideas based on this? Have you seen this before?

#31

@Sash I use a command like

qemu-img convert -O raw mkosi-image.raw converted.img

…but as you’ll see on this thread I haven’t been able to get things to work, so take everything I say with a pinch of salt :wink:

1 Like
#32

Not sure why it would fail on fsck. Can you check the log as instructed? That is running systemctl status systemd-fsck-root.service ?

#33

Thanks it did help generating images then giving input to mender-convert.

While flashing images to my Hardware, am also here facing the same issue Debian could able to boot, But it failed in dracut emergency shell window.

Did you had any luck with this? If yes could you please enlighten me?
Regards,
Sashank

#34

Hi All,

I’m working with @krystof to get a bootable Ubuntu image with mender_convert. I’ve been able to get an image to boot when using an Ubuntu bionic based mkosi image but not when using focal.

The issue seems to be that the focal image generated by mkosi contains some additional kernel and initramfs images that the bionic image does not.

See this excerpt of the mender_convert log output when running with a Ubuntu focal based mkosi image as input:

2020-06-02 01:25:22 [INFO] [mender-convert-modify] Installing Mender client and related files
2020-06-02 01:25:27 [INFO] [mender-convert-modify] Found Linux kernel image: 

	work/rootfs/boot/vmlinuz
work/rootfs/boot/vmlinuz-5.4.0-26-generic
work/rootfs/boot/vmlinuz.old

basename: extra operand 'work/rootfs/boot/vmlinuz.old'
Try 'basename --help' for more information.
2020-06-02 01:25:27 [INFO] [mender-convert-modify] Found initramfs image: 

	work/rootfs/boot/initrd.img-5.4.0-26-generic
work/rootfs/boot/initrd.img
work/rootfs/boot/initrd.img.old

basename: extra operand 'work/rootfs/boot/initrd.img.old'
Try 'basename --help' for more information.

It seems that when there are multiple kernel / initramfs images on the input image mender_convert fails silently. By that I mean that even when this output is in the mender_convert log it still generates an output image and returns a 0 return code. The converted image does not boot however.

For reference here are the working mkosi and mender_convert configurations.

mkosi config:

[Distribution]
Distribution=ubuntu
Release=bionic

[Output]
Format=gpt_ext4
Bootable=yes

[Partitions]
RootSize=3G


[Packages]
Packages=nano less tmux ssh dnsutils curl ethtool iproute2 isc-dhcp-client netplan.io iputils-ping ca-certificates lshw pciutils

[Validation]
Password=test

mender_convert config:

MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"
MENDER_STORAGE_TOTAL_SIZE_MB="7168"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="256"
IMAGE_ROOTFS_SIZE="-1"

By changing Release=bionic in the mkosi config to Release=focal I can reproducibly break the converted image.

#35

Whilst not a fix for the mender bug, you could drop a postinst script file into the mkosi build to delete the .old files for now to move your project forward.

#36

@marekl, great find! This we should fix in mender-convert.

I am not sure what the purpose is of e.g work/rootfs/boot/initrd.img.old but I suspect that it would be safe to filter these out during parse.

So I think the tool should do the following:

  1. If it finds more then one image, pick the first one from the list and print a warning that says it found multiple images and that the first one was picked.
  2. It should also print a suggestion that you silence this warning by setting the following variables in your config:
MENDER_GRUB_KERNEL_IMAGETYPE = "vmlinuz"
MENDER_GRUB_INITRD_IMAGETYPE = "initrd.img"

This will disable the probe, and explicitly set the image names.

#37

the .old files are normally symlinks that point to the the previous used kernel/initramdisk before they were upgraded, so in my opinion should be ignored if the goal is to extract the current versions. In my use case i am using a mkosi postinst script to clean out all unused kernels/initramdisks/headers etc prior to putting it through the mender-convert process.

#38

I have logged this here, https://tracker.mender.io/browse/MEN-3640. If someone feels up to it feel free to send a PR :slight_smile:

#39

Have updated the ticket above

#40

Was able to recreate the same issue @marekl was seeing. With fix applied, was able to create bootable ubuntu focal mender image.

Fix and pull request can be found here:


https://tracker.mender.io/browse/MEN-3640
1 Like
#41

I think I found a different bug relating to the use of MENDER_GRUB_INITRD_IMAGETYPE in mender-convert-modify

https://tracker.mender.io/browse/MEN-3642

#42

It looks like @drewmoseley found this as well, and there is a fix for this in https://github.com/mendersoftware/mender-convert/pull/211