Variscite DART-6UL - upgrade zeus to scarthgap

I’m attempting to update a Zeus build to Yocto Scarthgap. The Zeus build was created by following the instructions in the first post in this thread.
A few things required tweaks to support Scarthgap - the u-boot patches needed re-applying, and manually tweaking the conf files.
Now it’s building fine, however the boot process is having trouble.
This is the 2nd attempt I’ve had at this - the first was targeting Kirkstone, however it ran into this exact same boot issue:

Here is the boot log:

U-Boot SPL 2018.03+g9d5b55aeed8+p0 (Oct 18 2024 - 19:06:56 +0000)

Part number: VSM-VS6UL-101
Assembly: AS2104164477
Date of production: 2021 May 04
SOM configuration: eMMC WiFi
Trying to boot from MMC1
MMC Boot Device: mmc0 (SD)


U-Boot 2018.03+g9d5b55aeed8+p0 (Oct 18 2024 - 19:06:56 +0000)

CPU:   Freescale i.MX6ULL rev1.1 900 MHz (running at 396 MHz)
CPU:   Commercial temperature grade (0C to 95C) at 37C
Reset cause: POR
Board: Variscite VAR-SOM-6UL
I2C:   ready
DRAM:  512 MiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... OK
Display: VAR-WVGA-LCD (800x480)
Video: 800x480x24
** Unrecognized filesystem type **
Error: could not access storage.
In:    serial
Out:   serial
Err:   serial
eMMC:  7.3 GiB
Codec: wm8731
Net:   got MAC0 address from fuse: f8:dc:7a:5b:a4:1e
FEC0 [PRIME], usb_ether
Error: usb_ether address not set.

Normal Boot
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
32167 bytes read in 20 ms (1.5 MiB/s)
Found EFI removable media binary efi/boot/bootarm.efi
Scanning disks on usb...
Disk usb0 not ready
Disk usb1 not ready
Disk usb2 not ready
Disk usb3 not ready
Scanning disks on mmc...
MMC Device 2 not found
MMC Device 3 not found
Found 10 disks
466944 bytes read in 39 ms (11.4 MiB/s)
## Starting EFI application at 82000000 ...


EFI stub: Booting Linux Kernel...
EFI stub: Entering in SVC mode with MMU enabled
EFI stub: ERROR: install_configuration_table() failed
error: start_image() returned 0x80000009.

Dropping to grub prompt for unknown reason. Should never get here.

One thing to note - If I boot into the kernel directly from u-boot, it boots fine & everything seems correct. AFAICT it seems that the problem only arises when uefi/grub are in the loop.
These u-boot commands let me boot into the kernel directly:

setenv bootargs "console=ttymxc0,115200n8 rootwait root=/dev/mmcblk0p2" 
load mmc 0:2 ${loadaddr} boot/zImage 
load mmc 0:2 ${fdt_addr_r} boot/imx6ull-var-som-emmc-sd-card.dtb
bootz ${loadaddr} - ${fdt_addr_r}

Also, perhaps unsurprisingly, if I overwrite the zImage with one from the Zeus build (kernel 5.4.3-imx6ul+g9447bf00ed07), it boots fine.
And the opposite is true too - if I use an SD card with the Zeus build, and overwrite the zImage with one from Scarthgap, I get the exact same install_configuration_table & start_image errors.
To me, that indicates that the bootloaders are OK, and that the issue is the kernel itself - or at least, the handoff done via u-boot/grub.

Any help or guidance on how to proceed would be greatly appreciated. I’m guessing its a kernel configuration issue, however I think the config is correct. Here are the efi options in the kernel config, if it’s relevant:

root@imx6ul-var-dart-ts:~# zcat /proc/config.gz   | grep -i efi
CONFIG_EFI_STUB=y
CONFIG_EFI=y
CONFIG_EFI_PARTITION=y
# EFI (Extensible Firmware Interface) Support
CONFIG_EFI_ESRT=y
CONFIG_EFI_PARAMS_FROM_FDT=y
CONFIG_EFI_RUNTIME_WRAPPERS=y
CONFIG_EFI_GENERIC_STUB=y
CONFIG_EFI_ARMSTUB_DTB_LOADER=y
# CONFIG_EFI_BOOTLOADER_CONTROL is not set
# CONFIG_EFI_CAPSULE_LOADER is not set
# CONFIG_EFI_TEST is not set
# CONFIG_EFI_DISABLE_PCI_DMA is not set
# CONFIG_EFI_DISABLE_RUNTIME is not set
# CONFIG_EFI_COCO_SECRET is not set
# end of EFI (Extensible Firmware Interface) Support
# Software defined radio USB devices
# CONFIG_FB_EFI is not set
# CONFIG_RTC_DRV_EFI is not set
CONFIG_EFIVAR_FS=m

Hi @tloten,

Thinking a bit about your description, I think your analysis is right - unless we are both overlooking something weird. As a first shot, can you diff the kernel configurations? It should be possible to extract both from the build directory.

Greetz,
Josef

Sure, both configs are in this gist (will need to scroll down to find the 2nd file):

Hi @tloten,

Maybe a long shot, but can you check if this problem also affects you? Upgrading Mender client 2.6 to 4.0 - #18 by TheYoctoJester

Greetz,
Josef

Thanks - good idea, but didn’t seem to have an effect unfortunately. I stripped out a few areas like display/audio/bluetooth, and got the zImage to 6.9MB (down from 9.1MB). But still getting the same error :frowning:
For reference, the Zeus kernel was 8.5MB.

It’ll be a few weeks before I can get back onto this, but my next steps will be to try a few intermediary Yocto versions & their kernels to at least narrow it down. But it’s fairly tedious to do each time! I’ll also try my luck with Variscite support, but not sure how far I’ll get given it’s outside their typical boot process.

Hi @tloten,

Sorry to hear not having luck yet. I’ll also try to give the configs some more inspection, will let you know if I find something.:crossed_fingers:

Greetz,
Josef

One thing I recently noticed about the Zeus boot process is that it prints an EFI stub error, but still proceeds to boot successfully (and has been reliable for 4+ years on hundreds of devices).

Zeus boot log (doing u-boot → grub → uefi stub → kernel, as per the Zeus board integration thread here ):

U-Boot SPL 2018.03-imx_v2018.03_4.14.78_1.0.0_ga_var02+g717f29898a (Jun 04 2021 - 01:59:18 +0000)

Part number: VSM-VS6UL-211
Assembly: AS205183446
Date of production: 2022 Jun 26
SOM configuration: eMMC
Trying to boot from MMC1
MMC Boot Device: mmc1 (eMMC)


U-Boot 2018.03-imx_v2018.03_4.14.78_1.0.0_ga_var02+g717f29898a (Jun 04 2021 - 01:59:18 +0000)

CPU:   Freescale i.MX6ULL rev1.1 900 MHz (running at 396 MHz)
CPU:   Commercial temperature grade (0C to 95C) at 44C
Reset cause: POR
Board: Variscite VAR-SOM-6UL
I2C:   ready
DRAM:  512 MiB
MMC:   FSL_SDHC: 0, FSL_SDHC: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

Display: VAR-WVGA-LCD (800x480)
Video: 800x480x24
** Unrecognized filesystem type **
Error: could not access storage.
In:    serial
Out:   serial
Err:   serial
eMMC:  7.3 GiB
Net:   got MAC0 address from fuse: f8:dc:7a:85:65:2c
FEC0 [PRIME], usb_ether
Error: usb_ether address not set.

Normal Boot
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc1(part 0) is current device
Scanning mmc 1:1...
31198 bytes read in 18 ms (1.7 MiB/s)
Found EFI removable media binary efi/boot/bootarm.efi
Scanning disks on usb...
Disk usb0 not ready
Disk usb1 not ready
Disk usb2 not ready
Disk usb3 not ready
Scanning disks on mmc...
MMC Device 2 not found
MMC Device 3 not found
Found 10 disks
385024 bytes read in 26 ms (14.1 MiB/s)
## Starting EFI application at 82000000 ...
Welcome to GRUB!

lock: OK
lock: OK


EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: ERROR: Failed to install memreserve config table!
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 5.4.3-imx6ul+g9447bf00ed07 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 SMP PREEMPT Wed Jun 2 08:00:42 UTC 2021
[    0.000000] CPU: ARMv7 Processor [410fc075] revision 5 (ARMv7), cr=10c5387d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: T VAR-SOM-6UL with i.MX6ULL, eMMC & SD card support
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.70 by Das U-Boot
[    0.000000] efi:  SMBIOS=0x9df3e000
[    0.000000] cma: Reserved 32 MiB at 0x9a000000
[    0.000000] percpu: Embedded 15 pages/cpu s32332 r8192 d20916 u61440
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 129919
[    0.000000] Kernel command line: BOOT_IMAGE=(hd1,2)/boot/zImage root=/dev/mmcblk1p2 console=ttymxc0,115200n8 rootwait
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
... continues to boot successfully

As you can see, just prior to the kernel booting it prints EFI stub: ERROR: Failed to install memreserve config table!

Not sure whether it’s a red-herring - kernel/uefi/efistub is way outside my realm of knowledge - but if you compare the Zeus kernel efi-stub code with Scarthgap’s efi code (which is substantially different) in these two files:
efi-stub.c and arm32-stub.c
There are two things I noticed:
In Zeus, the memreserve error message actually results from a failed efi call to install_configuration_table, but it doesn’t care - the function’s return type is void and it continues booting.

In Scarthgap, the failure results from a call to install_configuration_table (ableit with very different args & intent), and the function returns the error (rather than void or EFI_SUCCESS), and it’s a lot earlier in the efi_stub_common method.

Where to go from here - not sure yet - the EFI stub seems to be a lot more complex compared to the Zeus one, with a large number of changes. Perhaps crudely attempting to port in the Zeus’ libstub folder to Scarthgap (and hoping there aren’t wider tendrils that need fixing) would be worthwhile? No idea… kernel changes are not something I’ve attempted before :laughing: