Mender-Convert UEFI GPT Output Image

General Discussions
,
1

Hi,

First I would like to say thanks for all the work on mender-convert (and obviously mender itself), it is a great product.

I am attempting to create a Debian image for generic x86 hardware with UEFI and no legacy boot support. When I run the mender-convert on a disk partitioned with GPT, it always outputs an image in MBR/msdos format. I do see a comment in the mender-convert-package script that says:

# Create partition table. TODO: GPT support

Does that mean the tool will never output a GPT/UEFI img.gz file? I’m confused because the first time I ran the tool I used an msdos/MBR image for input and I could swear it created a GPT/UEFI image. I remember because the image didn’t boot at first when I wrote it back to the same device, I had to change the PC to UEFI instead of BIOS before it would boot.

Is there something I can change to get a GPT/UEFI output image? Some of the devices I’m trying to use are only UEFI and won’t boot legacy.

Just for reference, here is the input image partitions:

Disklabel type: gpt
Disk identifier: 9E3F1724-97A6-45D2-B640-361293210FA9

Device    Start      End  Sectors  Size Type
1          2048  1050623  1048576  512M EFI System
2       1050624 12769279 11718656  5.6G Linux filesystem

And here is the output image partitions:

Disklabel type: dos
Disk identifier: 0x271b6a9f

Device  Boot    Start      End  Sectors  Size Id Type
1       *       16384  1064959  1048576  512M  c W95 FAT32 (LBA)
2             1064960 17154047 16089088  7.7G 83 Linux
3            17154048 33243135 16089088  7.7G 83 Linux
4            33243136 33505279   262144  128M 83 Linux

Also, while mender-convert is running I see in work/convert.log that it does create an EFI boot partition with EFI files in the mender-convert-modify section. I’m wondering if the Mender Artifact contains an EFI image that I could extract and use?

Thanks!
Nate

2

Hi @bakern, thank you for kind words about Mender and welcome to the community.

So the the directory structure that is created is compatible with UEFI but as you have noticed the output image is not an GPT image and this has been the case so far.

As you have noticed the TODO, this is something that we need to add.

The changes needed to create an GPT image are probably in these lines, https://github.com/mendersoftware/mender-convert/blob/master/mender-convert-package#L226-L233 and should not be much.

3

Yes, GPT should be the default, as this is what is specified in the UEFI standard. However, be aware that despite what the standard says, GPT is not always supported. In particular, the Beaglebone Black only boots if the partition table is msdos/MBR style. Maybe a good idea is to make the defaults x86 -> GPT, ARM -> MBR? With options to override either?

4

Makes sense. Actually found a task for this [MEN-2151] - Mender and CFEngine (by Northern.tech) Jira, I think it was referring the old version of mender-convert but can re-purpose this one I guess as it is still valid.

5

I verified by changing the lines

run_and_log_cmd "${PARTED} -s ${img_path} mklabel msdos"
run_and_log_cmd "${PARTED} -s ${img_path} unit s mkpart primary fat32 ${boot_part_start} ${boot_part_end}"

to

run_and_log_cmd "${PARTED} -s ${img_path} mklabel gpt"
run_and_log_cmd "${PARTED} -s ${img_path} unit s mkpart ESP fat32 ${boot_part_start} ${boot_part_end}"

It seems to work fine. Just another idea, how about just use whichever type of partition table the source image is using? For example I just moved it into an if/else block checking the source image partition scheme and it seemed to work:

MENDER_PARTITION_TYPE=$(disk_get_part_value ${disk_image} 1 SCHEME)
if [ "${MENDER_PARTITION_TYPE}" == "gpt" ]; then
...
6

Thanks for looking in to this. Looks simple enough :slight_smile: .

I like the conditional approach based on input disk type,

MENDER_PARTITION_TYPE=$(disk_get_part_value ${disk_image} 1 SCHEME)
if [ "${MENDER_PARTITION_TYPE}" == "gpt" ]; then
...

Would you be able to create a pull-request for this?

7

I created a pull request: https://github.com/mendersoftware/mender-convert/pull/185

If someone could review it and let me know if any changes need to be made, that would be great.

8

I’m using mender-convert commit which add GPT support and trying to convert ubuntu image. Converted image cannot boot because grub cannot see gpt partitions (except boot only). Rootfs and data are like:
Partition hd0,gpt2: No known filesystem detected ...

My Intel NUC using 128GB SSD. I installed ubuntu (5G rootfs only) and run live image and copy content using dd command (copied 10G just to be sure :wink: )

Then mender-convert fails due to the fact that partx cannot find any partition as there was an error:

fdisk -l input/ubuntu.img
GPT PMBR size mismatch (234441647 != 22527999) will be corrected by w(rite).

so I used gdisk input/ubuntu.img and verify + write options. After this change mender-convert pass and also print that using gpt partition table:

2020-04-20 18:06:11 [INFO] [mender-convert-extract] Validating disk image
2020-04-20 18:06:11 [INFO] [mender-convert-extract] Disk parsed successfully
2020-04-20 18:06:11 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: gpt
2020-04-20 18:06:11 [INFO] [mender-convert-extract] PART 1: SIZE: 100M TYPE: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
2020-04-20 18:06:11 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2020-04-20 18:06:11 [INFO] [mender-convert-extract] PART 2: SIZE: 10G TYPE: 0fc63daf-8483-4772-8e79-3d69d8477de4
2020-04-20 18:06:11 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs

So question is am I missing something in my step to create bootable image? Thanks a lot for any pointers.

9

This might relate to the fact that you only copied 10G. The GPT backup headers are put at LBA - 1, which is relative to the disk size, so this will always be at the last block of the disk.

I am not suggesting for you to copy 128GB, this would be quite excessive, but maybe we should add something in mender-convert that ensures that the GPT backup headers are written prior to trying to extract anything,

10

I think I fixed GPT issue by using gdisk input/ubuntu.img and verify + write. After that fdisk and other tools doesn’t report issue with GPT (actually I’m not sure if it write GPT backup headers or not). I’ll investigate if we can make it somehow working in mender-convert. Thanks.

11

Yeah if you don’t copy the whole source disk you can run sgdisk -e input/ubuntu.img first and it should fix it. Other tools such as gdisk can do it as well, if you can script it. I’m not sure if it would be best to put that in mender-convert or require it to be a manual step since it does modify the image.

If you’re writing a GPT image back to the device and the image isn’t exactly the same size as the device’s disk you should also run sgdisk -e /dev/sda after writing it.

12

@bakern ok thanks a lot for valuable info. IMO it would be nice to have this handling automatic in mender convert and also documented that after flashing final image they need to run command to have system booted.

13

Though I have never seen any issues with actually booting system which do not have the GPT backup headers.

I have seen other issues, and just recently I have been struggling with this with regards to partition resizing, e.g this commit

14

I didn’t perform resize (data partition remain same). I’ll try to play with it and summarize what I’ll find. Thanks.

15

I tried to install fresh centos and copy only 10G of 12GB ssd. When image is trasfered to PC and run sgdisk -e input/centos.img and go this:

sgdisk -e input/centos.img 
Warning! Disk size is smaller than the main header indicates! Loading
secondary header from the last sector of the disk! You should use 'v' to
verify disk integrity, and perhaps options on the experts' menu to repair
the disk.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Error 25 reading partition table for CRC check!
Warning! One or more CRCs don't match. You should repair the disk!

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Warning! Secondary partition table overlaps the last partition by
213960737 blocks!
You will need to delete this partition or resize it in another utility.

Problem: partition 2 is too big for the disk.
Aborting write operation!
Aborting write of new partition table.

I’m using Ubuntu 16.04. Could it be that sgdisk is old to fix such errors? Thanks.

16

I think everything through the last Caution: is expected. I get that too and it works fine.

It looks like sgdisk thinks your 2nd partition is too large though. Can you share the partition table of the source disk before you copied 10G of it to the PC?

I have done that same type of operation and it’s worked for me. For example on a 32GB disk with a 256MB boot and 8GB root I do dd if=/dev/sda bs=1M count=10000 conv=sync,noerror | gunzip -c > /mnt/usb/image.img just making sure it gets both partitions.

I think sgdisk should be able to handle just creating a new backup GPT table at the end of the disk, but if you find a better way to script it I would say go for it. The -e operation doesn’t seem to be as well documented, but sgdisk was meant for scripting which I think is why people like to use it.

17

@bakern thansk a lot. Issue was cause my own stupidity (or Friday :wink: ). Installed original system on whole disk and not only to e.g. 10G. When install rootfs to 10G only and covers (and use sgdisk trick) converted image booted perfectly fine. Sorry for noise. I’ll post mender-convert update concerning fixing GPT in next few days. Thanks.