Boot device not found after integration mender into yocto

Hello,

I am currently trying to integrate mender for an intel based Industrial PC (x64). Before integrating mender my image boots normally, but after integrating mender and installing via the .hddimg the firmware displays “Default Boot Device Missing or Boot Failed”. But the firmware also supports “Boot From File” where I can choose an EFI file to boot from. After I choose the file “EFI/BOOT/bootx64.efi” the image starts up normally. This is why I suspect that there is nothing wrong with the image in general, but just with the GRUP setup created by mender. The “vanilla” (poky - thud 2.6 with meta-intel) yocto image did boot up just fine (without booting from file).

Is there any way to debug this issue further?

Additional Information:
The relevant part of the local.conf
MENDER_ARTIFACT_NAME = “release-1”
INHERIT += “mender-full”
ARTIFACTIMG_FSTYPE = “ext4”
MENDER_SERVER_URL = “https://hosted.mender.io
MENDER_TENANT_TOKEN = “”

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd”"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
ARTIFACTIMG_FSTYPE = "ext4"

MENDER_STORAGE_TOTAL_SIZE_MB = "4096"
MENDER_DATA_PART_SIZE_MB = "512"

MENDER_STORAGE_DEVICE = "/dev/sda"
MENDER_GRUB_STORAGE_DEVICE = "hd0"
MENDER_DEVICE_TYPE = "intel-corei7-64"
MENDER_FEATURES_ENABLE_append = " mender-grub mender-image-uefi"
MENDER_FEATURES_DISABLE_append = " mender-uboot mender-image-sd"
# Debug only
PACKAGECONFIG_append_pn-grub-mender-grubenv = " debug-pause"

Yocto layers:
meta /home/parallels/Documents/yocto/poky/meta 5
meta-poky /home/parallels/Documents/yocto/poky/meta-poky 5
meta-yocto-bsp /home/parallels/Documents/yocto/poky/meta-yocto-bsp 5
meta-intel /home/parallels/Documents/yocto/poky/meta-intel 5
meta-mender-core /home/parallels/Documents/yocto/poky/meta-mender/meta-mender-core 6
meta-mender-demo /home/parallels/Documents/yocto/poky/meta-mender/meta-mender-demo 6

Any help would be greatly appreciated.

Best regards,
c3ntry

It probably relates to usage of “.hddimg”, which is not supposed to be used with Mender.

You should have gotten a “uefiimg” as output, which you should write to the storage medium. This should include all the parts necessary to boot and update

This is probably a good reference,

Thanks for your help! The Giada VM23 Guide looks promising - I will try that as soon as I can.

My device has an internal storage medium (M2 SSD) which is hard to access. Does mender not support an install USB Stick? Is the method outlined in the Giada VM23 the only supported way to do the initial device provisioning?

The method described in the post is currently the way to do it.

Though we just merged a “live installer” on master,

Which I belive works similar do hddimg

To clarify, this is the HDDIMG which has the initramfs based installer. The installer script has been replaced with a Mender-specific one. Basically, the Mender version of the HDDIMG includes the UEFIIMG and then just uses “dd” to write it. If MENDER_STORAGE_DEVICE is set, it will attempt to use that. Otherwise, if mender-partuuid is set, it will scan for appropriate devices.

That said, prior to the PR mentioned, the HDDIMG file would not work. Now “install” is the only boot option and it does work.

Thanks for your responses and your help. I really appreciate it!

I tried following the Giada VM23 Guide - but had to make some adjustments because the network adapter doesn’t work yet. I went through the following steps:

  • Booted Ubuntu Server 18.04 from a USB thumb drive
  • Inserted a USB thumb drive which contains the .uefiimg
  • used dd to copy it to the SSD (/dev/sda)
  • Removed all USB drives and rebooted

But unfortunately, the result was the same as before.

Then I disassembled the entire IPC to flash the SSD directly with the image - but go the same result.

Some other thing I suspect could cause Issues:
When I run “sudo fdisk -l” I get the following result for the SSD:

GPT PMBR size mismatch (8388607 != 125045423) will be corrected by w(rite).
Disk /dev/sdc: 59.6 GiB, 64023257088 bytes, 125045424 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 2A5D1CDF-FFEA-4F38-82F5-8F70826E7B8B

Is the GPT PMBR size mismatch caused by me setting the STORAGE_TOTAL_SIZE_MB to 4096 and not to ~60GB? If yes, is it really a best practice to handle 60GB+ image files?

BTW: The new Live Installer seams really nice. I will definitely give it a try after I get the image working.

When I run “sudo fdisk -l” I get the following result for the SSD:

Can you print the full output of this command?

Is the GPT PMBR size mismatch caused by me setting the STORAGE_TOTAL_SIZE_MB to 4096 and not to ~60GB? If yes, is it really a best practice to handle 60GB+ image files?

Do not think this relates to STORAGE_TOTAL_SIZE_MB, and I recall seeing this printout before but it has never caused any harm.

But going back to your original report and the error message,

“Default Boot Device Missing or Boot Failed”

and a couple follow up questions,

  • Is your boot priority configured correctly?
  • Does the boot firmware support GPT?
  • Is the boot firmware BIOS or UEFI?

I am trying to figure out if you should use this configuration or not,

https://docs.mender.io/2.0/devices/yocto-project/bootloader-support/grub#bios-based-systems

Can you print the full output of this command?

What parts are you interested in? There are about 16 ramdisks listed which hide the top of the output.

  • Is your boot priority configured correctly?

I have not found this setting yet. Maybe the manufacturer didn’t integrate it because there is only one internal drive and you usually don’t boot this device from an external drive.

  • Does the boot firmware support GPT?

I am not sure - how can I find out?

  • Is the boot firmware BIOS or UEFI?

I am quite sure it is UEFI because I can boot from the .efi file if I select it manually.

Some more information: The Setup Utility is InsydeH2O.

Here you can find the BIOS documentation: BIOS Docs

Here you can find the full documentation: Full Docs

I am currently building the BIOS image and will test if that works.

I noticed that the fdisk outputs for my Ubuntu Server USB (which boots normally) and the disk with my mender image where different,

the first partition on the Ubuntu stick was Type “EFI System” an the first mender partition was “Microsoft basic data”.

So I got it to work using the following steps:

  • I used parted to resize the partion (partition 4) -> This removes the GPT PMBR size mismatch an allowes

me to use fdisk to change the partition type

  • I used fdisk to change the partition type (partition 1) from “Microsoft basic data” to “EFI System” (the “t” command)

I am about 80% sure that this fixes the problem (One time the image did boot directly without the changes, but maybe I messed up the dd command).

So it looks like my device only boots if the .efi file is located on an EFI partition type.

My qustion now is: Is there a way to set the partition type of the mender boot partition to “EFI System”?

I would expect it to already be EFI System, here is the output from one of my images I had laying around for x86_64,

$ fdisk -l agl-demo-platform-intel-corei7-64.uefiimg 
Disk agl-demo-platform-intel-corei7-64.uefiimg: 7,10 GiB, 8564786176 bytes, 16728098 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A87D9A57-A19E-4763-A67C-ED475790506F

Device                                        Start      End Sectors  Size Type
agl-demo-platform-intel-corei7-64.uefiimg1    16384    49151   32768   16M EFI System
agl-demo-platform-intel-corei7-64.uefiimg2    49152  8257535 8208384  3,9G Linux filesystem
agl-demo-platform-intel-corei7-64.uefiimg3  8257536 16465919 8208384  3,9G Linux filesystem
agl-demo-platform-intel-corei7-64.uefiimg4 16465920 16728063  262144  128M Linux filesystem

That is strange - my output is:

Disk /home/parallels/Documents/yocto/poky/build/tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 33A578DB-581B-4462-ACFF-C08D5CEAEBA8

Device                                                                                                                  Start     End Sectors  Size Type
/home/parallels/Documents/yocto/poky/build/tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg1   16384   49151   32768   16M Microsoft basic data
/home/parallels/Documents/yocto/poky/build/tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg2   49152 3686399 3637248  1.8G Linux filesystem
/home/parallels/Documents/yocto/poky/build/tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg3 3686400 7323647 3637248  1.8G Linux filesystem
/home/parallels/Documents/yocto/poky/build/tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg4 7323648 8372223 1048576  512M Linux filesystem

Do you have any special settings regarding x86_64 besides the MACHINE variable (mine is “intel-corei7-64”)?

Pls try to use exact steps as for Giada and try this image with some ubuntu live distro then dd it to eMMC and it should boot (at least it boots on Giada). Which yocto branch do you use? Thanks.

I think I followed the Giada steps as close as possible, but I have an internal SSD instead of an eMMC so I am using “/dev/sda” instead of “/dev/mmcblk0”. Other than that there aren’t a lot of features/configurations to make in the Giada Guide. Does your Giada image have the correct EFI partition?

Can you share some information about your host machine and OS?

Yes, of course. It is a bit a wired setup:

I am building Yocto on:
Ubuntu 18.04.1 LTS - 64-bit - Virtualization KVM
VM ran by -> Parallels Desktop 14 for Mac Business Edition - Version 14.1.3 (45485)

Physical Machine:

MacBookPro15,1
Intel Core i9-9880H CPU @ 2.30GHz - 16 GB RAM
Mac OS Mojave 10.14.6 (18G87)

I did a Gaida build, and indeed something is odd here. as the partition is marked Microsoft basic data

sudo fdisk -l tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg
Disk tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3A12AC2C-7A7F-4A2F-B657-3B912C2BD78E

Device                                                                       Start     End Sectors  Size Type
tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg1   16384   49151   32768   16M Microsoft basic data
tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg2   49152 4079615 4030464  1.9G Linux filesystem
tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg3 4079616 8110079 4030464  1.9G Linux filesystem
tmp/deploy/images/intel-corei7-64/core-image-base-intel-corei7-64.uefiimg4 8110080 8372223  262144  128M Linux filesystem

Could be odd but I’m sure I have enable UEFI boot on Giada, but maybe there is something wrong (and it was reason why I cannot boot it with live usb stick from mender build)

From within the build folder, could you post the output of:

grep -i -A7 "### Contents of wks file" `find -name log.do_image_uefiimg`

You can also try to run the wks file manually with wic, in case it has some special behavior in your setup. The command should be in the same log.do_image_uefiimg file.

Here it is,

./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg:+ echo ### Contents of wks file ###
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg:### Contents of wks file ###
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-+ cat /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/mender-uefiimg.wks
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --rootfs-dir /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/bootfs.image_uefiimg --ondisk "sda" --fstype=vfat --label boot --align 8192 --fixed-size 16 --active
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --ondisk "sda" --fstype=ext4 --label primary --align 8192 --fixed-size 2015232k --exclude-path data/ boot/efi/ 
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --ondisk "sda" --fstype=ext4 --label secondary --align 8192 --fixed-size 2015232k --exclude-path data/ boot/efi/ 
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --rootfs-dir /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/rootfs/data --ondisk "sda" --fstype=ext4 --label data --align 8192 --fixed-size 128
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-bootloader --ptable gpt
./tmp/work/intel_corei7_64-poky-linux/core-image-minimal-initramfs/1.0-r0/temp/log.do_image_uefiimg-+ echo ### End of contents of wks file ###
--
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg:+ echo ### Contents of wks file ###
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg:### Contents of wks file ###
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-+ cat /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/mender-uefiimg.wks
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --rootfs-dir /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/bootfs.image_uefiimg --ondisk "sda" --fstype=vfat --label boot --align 8192 --fixed-size 16 --active
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --ondisk "sda" --fstype=ext4 --label primary --align 8192 --fixed-size 2015232k --exclude-path data/ boot/efi/ 
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --ondisk "sda" --fstype=ext4 --label secondary --align 8192 --fixed-size 2015232k --exclude-path data/ boot/efi/ 
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-part --source rootfs --rootfs-dir /home/admin/mender-giada/build/tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/rootfs/data --ondisk "sda" --fstype=ext4 --label data --align 8192 --fixed-size 128
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-bootloader --ptable gpt
./tmp/work/intel_corei7_64-poky-linux/core-image-base/1.0-r0/temp/log.do_image_uefiimg-+ echo ### End of contents of wks file ###