@krystof ok thanks for info. I’ll try to build today mkosi image convert and play on my device which had same symptoms. Maybe I spot something. Keep you posted. Thanks.
I don’t have the image for you, because I had a very interesting day today. I started out by trying to build a smaller image (5 GB rootfs partitions) to share with you, and I changed a couple of other things to try to get things to work:
- removed
--qcow2flag tomkosi - used Ubuntu 18.04 instead of 20.04
When I tried booting the mender images from that smaller image… it worked!
Just to be sure, I then tried the same process again (same flags, same Ubuntu release), with 20GB rootfs partitions… and the resulting image had the same problems I’ve been experiencing.
I am pretty sure I followed exactly the same process for both builds, but I’m still copy-pasting and editing commands from my notes. I am going to write some automation scripts to be sure it’s always consistent.
Could it be that there is an issue with my partition sizes for mender for the 20GB rootfs build?
Here is my mender config for the 20GB rootfs build which did NOT work:
MENDER_STORAGE_TOTAL_SIZE_MB="122104"
IMAGE_ROOTFS_SIZE="20971520"
IMAGE_ROOTFS_EXTRA_SPACE="0"
IMAGE_OVERHEAD_FACTOR="1.0"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="80864"
MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"
And here is the config for the 5GB rootfs build which did work:
MENDER_STORAGE_TOTAL_SIZE_MB="122104"
IMAGE_ROOTFS_SIZE="5881856"
IMAGE_ROOTFS_EXTRA_SPACE="0"
IMAGE_OVERHEAD_FACTOR="1.0"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="109088"
MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"
Actually, how the numbers should be calculated is not always clear to me. In both the 20GB and 5GB builds, mender-convert complained about some of the sizes, and I had to make adjustments based on the messages to get mender-convert to run.
For example, with the 5 GB build, I had
MENDER_STORAGE_TOTAL_SIZE_MB="122104"
IMAGE_ROOTFS_SIZE="5242880"
IMAGE_ROOTFS_EXTRA_SPACE="0"
IMAGE_OVERHEAD_FACTOR="1.0"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="109088"
I think 5 GiB = 5,120 MiB = 5242880 KiB, so the sizes look correct to me. But mender-convert said:
2020-05-15 19:49:00 [WARN] [mender-convert-package] The calculated rootfs partition size 624 MiB is too small.
2020-05-15 19:49:00 [WARN] [mender-convert-package] The actual rootfs image size is 5120 MiB
2020-05-15 19:49:00 [FATAL] [mender-convert-package] You can try adjusting the MENDER_STORAGE_TOTAL_SIZE_MB variable to increase available space, or modify one of the variables IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_EXTRA_SPACE or IMAGE_OVERHEAD_FACTOR to reduce the size of the root filesystem.
Even though the original image was exactly 5GB:
Device Start End Sectors Size Type
18_04.img1 2048 526335 524288 256M EFI System
18_04.img2 526336 11014110 10487775 5G Linux root (x86-64)
I had something similar with the 20GB sizes, though I no longer have exactly how mender-convert asked me to change the numbers. In case it’s helpful, here’s the output of fdisk -l on the 20GB which I am still not able to boot:
$ fdisk -l 18-04_20G.img
Disk 18-04_20G.img: 20.26 GiB, 21745369088 bytes, 42471424 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C84B8257-7B06-1C4E-BAC2-9FCF399621F6
Device Start End Sectors Size Type
18-04_20G.img1 2048 526335 524288 256M EFI System
18-04_20G.img2 526336 42471390 41945055 20G Linux root (x86-64)
Any ideas? Is my math off on some of the partition sizes for the 20GB configuration?
Thanks so much for your continued help!
It appears the problem was indeed with the partition sizes. I intentionally set the data partition size too small to let mender-convert figure out the appropriate sizes for the root partitions, and the resulting image booted.
This is the config that worked:
MENDER_STORAGE_TOTAL_SIZE_MB="122104"
IMAGE_ROOTFS_SIZE="20971520"
# In KiB
IMAGE_ROOTFS_EXTRA_SPACE="0"
IMAGE_OVERHEAD_FACTOR="1.0"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="61440"
MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"
Thanks again for your help!
@krystof ok thanks for sharing back. Can you please update original post with those info? Also do you use patch I’ve shared with you? If so we need to find why it helps and integrate it to mender-convert to have image working out of then box. Thanks and have a nice day.
Thanks for the detailed reports @krystof,
To clear some things up on the variables and sizes.
What you list here as 20GB, is the partition size,
18-04_20G.img2 526336 42471390 41945055 20G Linux root (x86-64)
There is a difference between partition size and file system image size. mender-convert does not care much about input partition sizes and will try to re-calculate this based on various variables.
The rootfs partition size is always calculated based on this formula:
rootfs_part_sectors=$(((${disk_image_total_sectors} - ${data_part_sectors} - \
${boot_part_sectors} - ${overhead_sectors}) / 2))
which can be translated to (roughly):
rootfs_part_mb=$(( ${MENDER_STORAGE_TOTAL_SIZE_MB} - ${MENDER_DATA_PART_SIZE_MB} - \
${MENDER_DATA_PART_SIZE_MB} ))
You can specific the rootfs image size using the IMAGE_ROOTFS_* variables. This is an optimization, as the rootfs image size impacts what is transferred between the Mender server and the Mender client, and this can be less then the actual partition size.
The default rootfs image size will be actual rootfs content * 1.5 (50 % free space relative to actual occupied storage).
If you want the rootfs image size to be the same as the rootfs partition size you can just set:
IMAGE_ROOTFS_SIZE = "-1"
Further, the following is not aligned,
MENDER_STORAGE_TOTAL_SIZE_MB="122104"
Eg I would set it to 122880 which is aligned to 1024 if you want to represent 120GB. This a very large disk :).
@mirzak @drewmoseley seems patch which I’ve created for mender-convert does also for this case. I think we need to investigate why some BIOS work without this patch (Intel NUC) and some doesn’t.
@krystof what is your target HW, can you please share? Thanks.
@MarekBelisko target hardware is this: https://www.advantech.com/products/68ccaea2-9ff5-4f85-97f2-3d11244b0a08/aimb-286/mod_ad53f4ec-0d78-4995-aeed-337921f3c837
@mirzak the disk size is strange to me also, but I got that number from fdisk:
$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 119.25 GiB, 128035676160 bytes, 250069680 sectors
If I’m doing the calculation right, 128035676160 bytes = 125034840 KiB = 122104.3359375 MiB??? This is very strange indeed. I am what the appropriate value for MENDER_STORAGE_TOTAL_SIZE_MB should be.
I am still getting strange / inconsistent behavior. For example, I tried installing different packages in the base image (generated by mkosi), and even when the image would consistently boot in a VM, some images created by mender-convert wouldn’t. Today I also tried the “golden-image” approach, manually creating partitions including an EFI boot partition, and then using mender-convert on that image… and the result wouldn’t boot either (it showed the “Dropping to grub prompt for unknown reason. Should never get here” message).
@mirzak could you please suggest what values I should set for all the mender sizing variables, given the disk I have and that I want about rootfs partitions of about 30 GB?
Thanks!
@mirzak the disk size is strange to me also, but I got that number from fdisk:
This is better to read from the actual block device on the device, e.g
$ cat /sys/block/sda/size
62521344
In regards to variables, something like this should work:
# 62 GB total size
MENDER_STORAGE_TOTAL_SIZE_MB = "63488"
MENDER_BOOT_PART_SIZE_MB="256"
# As you are working with very large partitions, I would leave this at 256MB here and this can be resized later on the device, e.g on first boot to occupy rest of the free space available on the disk.
MENDER_DATA_PART_SIZE_MB="256"
IMAGE_ROOTFS_SIZE = "-1"
This should give you rootfs parts of ~30GB.
@krystof I could see you have generated the .img from mkosi, could you please share me the steps or how can i convert the generated raw image files?
Thanks @mirzak.
I was able to get Ubuntu to start booting, but it didn’t succeed:
Any ideas based on this? Have you seen this before?
@Sash I use a command like
qemu-img convert -O raw mkosi-image.raw converted.img
…but as you’ll see on this thread I haven’t been able to get things to work, so take everything I say with a pinch of salt 
1 Like
Not sure why it would fail on fsck. Can you check the log as instructed? That is running systemctl status systemd-fsck-root.service ?
Thanks it did help generating images then giving input to mender-convert.
While flashing images to my Hardware, am also here facing the same issue Debian could able to boot, But it failed in dracut emergency shell window.
Did you had any luck with this? If yes could you please enlighten me?
Regards,
Sashank
Hi All,
I’m working with @krystof to get a bootable Ubuntu image with mender_convert. I’ve been able to get an image to boot when using an Ubuntu bionic based mkosi image but not when using focal.
The issue seems to be that the focal image generated by mkosi contains some additional kernel and initramfs images that the bionic image does not.
See this excerpt of the mender_convert log output when running with a Ubuntu focal based mkosi image as input:
2020-06-02 01:25:22 [INFO] [mender-convert-modify] Installing Mender client and related files
2020-06-02 01:25:27 [INFO] [mender-convert-modify] Found Linux kernel image:
work/rootfs/boot/vmlinuz
work/rootfs/boot/vmlinuz-5.4.0-26-generic
work/rootfs/boot/vmlinuz.old
basename: extra operand 'work/rootfs/boot/vmlinuz.old'
Try 'basename --help' for more information.
2020-06-02 01:25:27 [INFO] [mender-convert-modify] Found initramfs image:
work/rootfs/boot/initrd.img-5.4.0-26-generic
work/rootfs/boot/initrd.img
work/rootfs/boot/initrd.img.old
basename: extra operand 'work/rootfs/boot/initrd.img.old'
Try 'basename --help' for more information.
It seems that when there are multiple kernel / initramfs images on the input image mender_convert fails silently. By that I mean that even when this output is in the mender_convert log it still generates an output image and returns a 0 return code. The converted image does not boot however.
For reference here are the working mkosi and mender_convert configurations.
mkosi config:
[Distribution]
Distribution=ubuntu
Release=bionic
[Output]
Format=gpt_ext4
Bootable=yes
[Partitions]
RootSize=3G
[Packages]
Packages=nano less tmux ssh dnsutils curl ethtool iproute2 isc-dhcp-client netplan.io iputils-ping ca-certificates lshw pciutils
[Validation]
Password=test
mender_convert config:
MENDER_STORAGE_DEVICE_BASE=/dev/sda
MENDER_DEVICE_TYPE="x86_64"
MENDER_COPY_BOOT_GAP="n"
MENDER_STORAGE_TOTAL_SIZE_MB="7168"
MENDER_BOOT_PART_SIZE_MB="256"
MENDER_DATA_PART_SIZE_MB="256"
IMAGE_ROOTFS_SIZE="-1"
By changing Release=bionic in the mkosi config to Release=focal I can reproducibly break the converted image.
Whilst not a fix for the mender bug, you could drop a postinst script file into the mkosi build to delete the .old files for now to move your project forward.
@marekl, great find! This we should fix in mender-convert.
I am not sure what the purpose is of e.g work/rootfs/boot/initrd.img.old but I suspect that it would be safe to filter these out during parse.
So I think the tool should do the following:
- If it finds more then one image, pick the first one from the list and print a warning that says it found multiple images and that the first one was picked.
- It should also print a suggestion that you silence this warning by setting the following variables in your config:
MENDER_GRUB_KERNEL_IMAGETYPE = "vmlinuz"
MENDER_GRUB_INITRD_IMAGETYPE = "initrd.img"
This will disable the probe, and explicitly set the image names.
the .old files are normally symlinks that point to the the previous used kernel/initramdisk before they were upgraded, so in my opinion should be ignored if the goal is to extract the current versions. In my use case i am using a mkosi postinst script to clean out all unused kernels/initramdisks/headers etc prior to putting it through the mender-convert process.
I have logged this here, https://tracker.mender.io/browse/MEN-3640. If someone feels up to it feel free to send a PR
Have updated the ticket above
Was able to recreate the same issue @marekl was seeing. With fix applied, was able to create bootable ubuntu focal mender image.
Fix and pull request can be found here:
https://tracker.mender.io/browse/MEN-3640
1 Like