Mender Raspberry PI CM4 Uboot issue

Hi all,

We’re having a strange problem trying to boot an sdimg for our cm4 raspberry pi board.
After creating and flashing the sdimg file created by Yocto/mender using meta-mender the board is unable to boot, unless we log into uboot console and simply run “saveenv” then everything works as expected.

I’m at a bit of loss how to diagnose this one any further, so any help would be appreciated.

Here’s the none working uboot

U-Boot 2022.01 (Jan 10 2022 - 18:46:34 +0000)
DRAM:  3.9 GiB
RPI Compute Module 4 (0xc03141)
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb@7e980000: USB DWC2
Bus xhci_pci: probe failed, error -110
scanning bus usb@7e980000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
"Synchronous Abort" handler, esr 0x96000004
elr: 00000000000da5f0 lr : 0000000000093cec (reloc)
elr: 000000003b3b35f0 lr : 000000003b36ccec
x0 : 036d756e76656403 x1 : 000000003b3c8683
x2 : 000000000000002c x3 : 0000000000000000
x4 : 036d756e76656403 x5 : 0000000000000074
x6 : 000000003aff8810 x7 : 0000000000000000
x8 : fffffffffffffff0 x9 : 0000000000000008
x10: 000000003af5fc60 x11: 000000003af64b00
x12: 0000000000000000 x13: 0000000000000200
x14: 000000003af474f0 x15: 0000000000000021
x16: 000000003b376fdc x17: 156852f900000000
x18: 000000003af54d90 x19: 000000003b3c8683
x20: 000000003aff87b0 x21: 0000000000000000
x22: 000000003aff86a0 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000000 x27: 000000003aff8720
x28: 000000003aff8850 x29: 000000003af45dd0
Code: 3900007f d65f03c0 aa0003e4 d2800003 (38636885)
Resetting CPU ...
resetting ...

and then working

switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
299 bytes read in 6 ms (47.9 KiB/s)
## Executing script at 02400000
switch to partitions #0, OK
mmc0(part 0) is current device
22934016 bytes read in 985 ms (22.2 MiB/s)
Moving Image from 0x80000 to 0x200000, end=18e0000
## Flattened Device Tree blob at 2eff2700
   Booting using the fdt blob at 0x2eff2700
   Using Device Tree in place at 000000002eff2700, end 000000002f002f63

Starting kernel ...

At first I thought this might be something to do with the environment variables, but they all look good. Here’s a printenv before running saveenv

U-Boot> printenv
altbootcmd=run mender_altbootcmd; run bootcmd
arch=arm
baudrate=115200
board=rpi
board_name=Compute Module 4
board_rev=0x14
board_rev_scheme=1
board_revision=0xC03141
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_efi_bootmgr=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr;fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_pci_enum=pci enum
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc0 mmc1 mmc2 usb0 pxe dhcp
bootcmd=run distro_bootcmd
bootcmd_dhcp=devtype=dhcp; run boot_net_usb_start; run boot_pci_enum; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_mmc2=devnum=2; run mmc_boot
bootcmd_pxe=run boot_net_usb_start; run boot_pci_enum; dhcp; if pxe get; then pxe boot; fi
bootcmd_usb0=devnum=0; run usb_boot
bootcount=1
bootdelay=2
bootlimit=1
cpu=armv8
dhcpuboot=usb start; dhcp u-boot.uimg; bootm
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
ethaddr=d8:3a:dd:eb:5c:ab
fdt_addr=2eff2700
fdt_addr_r=0x02600000
fdt_high=ffffffffffffffff
fdtcontroladdr=3af47510
fdtfile=broadcom/bcm2711-rpi-cm4.dtb
initrd_high=ffffffffffffffff
kernel_addr_r=0x00080000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
loadaddr=0x1000000
mender_altbootcmd=if test ${mender_boot_part} = 2; then setenv mender_boot_part 3; setenv mender_boot_part_hex 3; else setenv mender_boot_part 2; setenv mender_boot_part_hex 2; fi; setenv upgrade_available 0; saveenv; run mender_setup
mender_boot_kernel_type=booti
mender_boot_part=2
mender_boot_part_hex=2
mender_check_saveenv_canary=1
mender_dtb_name=overlay_map.dtb
mender_kernel_name=Image
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; if test "${mender_systemd_machine_id}" != ""; then setenv bootargs "systemd.machine_id=${mender_systemd_machine_id} ${bootargs}"; fi; setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk0p2; else setenv mender_boot_part_name /dev/mmcblk0p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_post_setup_commands; fi
mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi
mender_uboot_boot=mmc 0:1
mender_uboot_dev=0
mender_uboot_if=mmc
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
preboot=pci enum; usb start;
pxefile_addr_r=0x02500000
ramdisk_addr_r=0x02700000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;run boot_efi_bootmgr;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x02400000
serial#=10000000f03eda59
soc=bcm283x
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
upgrade_available=0
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
usbethaddr=d8:3a:dd:eb:5c:ab
vendor=raspberrypi

Environment size: 5582/16379 bytes

As stated above I’m really a bit of a loss what “saveenv” is possibly doing that’s causing this to be fixed.

Just as an aside we use this same yocto image to build our standard raspberry pi4’s and everything “just works”, so this feels compute module specific?!?

Hi @Alan01252,

That’s indeed a surprising effect. :frowning: My hunch would be that something on mmc0 behaves slightly different on the CM (are you using an eMMC version?) as it does on an SD card. Unfortunately I don’t have a way to look into the details at the moment, but maybe @leon-anavi (from the u-boot side) or @lueschem (from the Raspi side) have some wisdom to share?

Greets,
Josef

Thanks @TheYoctoJester

We’re using the eMMC version here, whereas our pi4 generic boards are using an SD card.

It’s very confusing :slight_smile:

@Alan01252 which Yocto release are you using?

Layer meta-raspberrypi for release Scarthgap relies on layer meta-lts-mixins branch scarthgap/u-boot which provides U-Boot v2024.04. I see you in the logs are using an older U-Boot version. Although U-Boot v2024.04 is explicitly needed for Raspberry Pi 5 (not for CM4), it is recommended to give it a try.

Hi @leon-anavi

We’re using kirkstone and a CM4.

Could you please try building Scartgap with U-Boot v2024.04 to see if the issue affects it too?

That might take me a while… :slight_smile:

I can probably create a minimal image but the current image is all part of our build pipeline, and upgrading to scarthgap even if it works won’t be a small endeavor for us :frowning:

Here’s the full console output from a fresh build


U-Boot 2022.01 (Jan 10 2022 - 18:46:34 +0000)

DRAM:  3.9 GiB
RPI Compute Module 4 (0xc03141)
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb@7e980000: USB DWC2
Bus xhci_pci: probe failed, error -110
scanning bus usb@7e980000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
"Synchronous Abort" handler, esr 0x96000004
elr: 00000000000da5f0 lr : 0000000000093cec (reloc)
elr: 000000003b3b35f0 lr : 000000003b36ccec
x0 : 036d756e76656403 x1 : 000000003b3c8683
x2 : 000000000000002c x3 : 0000000000000000
x4 : 036d756e76656403 x5 : 0000000000000074
x6 : 000000003aff8810 x7 : 0000000000000000
x8 : fffffffffffffff0 x9 : 0000000000000008
x10: 000000003af5fc60 x11: 000000003af64b00
x12: 0000000000000000 x13: 0000000000000200
x14: 000000003af474f0 x15: 0000000000000021
x16: 000000003b376fdc x17: 0000000001000000
x18: 000000003af54d90 x19: 000000003b3c8683
x20: 000000003aff87b0 x21: 0000000000000000
x22: 000000003aff86a0 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000000 x27: 000000003aff8720
x28: 000000003aff8850 x29: 000000003af45dd0

Code: 3900007f d65f03c0 aa0003e4 d2800003 (38636885)
Resetting CPU ...

resetting ...


U-Boot 2022.01 (Jan 10 2022 - 18:46:34 +0000)

DRAM:  3.9 GiB
RPI Compute Module 4 (0xc03141)
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb@7e980000: USB DWC2
Bus xhci_pci: probe failed, error -110
scanning bus usb@7e980000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
U-Boot> saveenv
Saving Environment to MMC... Writing to redundant MMC(0)... OK
U-Boot> boot
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
"Synchronous Abort" handler, esr 0x96000004
elr: 00000000000da5f0 lr : 0000000000093cec (reloc)
elr: 000000003b3b35f0 lr : 000000003b36ccec
x0 : e0d51e4020100000 x1 : 000000003b3c8683
x2 : 00000000ffffffbb x3 : 0000000000000000
x4 : e0d51e4020100000 x5 : 0000000000000003
x6 : 000000003aff8a10 x7 : 0000000000000000
x8 : fffffffffffffff0 x9 : 0000000000000008
x10: 000000003aff7e30 x11: 000000003af64b00
x12: 0000000000000000 x13: 0000000000000200
x14: 000000003af474f0 x15: 0000000000000021
x16: 000000003b376fdc x17: 0000000001000000
x18: 000000003af54d90 x19: 000000003b3c8683
x20: 0000000000000061 x21: 0000000000000000
x22: 000000003aff88a0 x23: 0000000000000000
x24: 0000000000000000 x25: 0000000000000000
x26: 0000000000000000 x27: 000000003aff8920
x28: 000000003aff8a50 x29: 000000003af45bf0

Code: 3900007f d65f03c0 aa0003e4 d2800003 (38636885)
Resetting CPU ...

resetting ...


U-Boot 2022.01 (Jan 10 2022 - 18:46:34 +0000)

DRAM:  3.9 GiB
RPI Compute Module 4 (0xc03141)
MMC:   mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from MMC... OK
In:    serial
Out:   serial
Err:   serial
Net:   eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus usb@7e980000: USB DWC2
Bus xhci_pci: probe failed, error -110
scanning bus usb@7e980000 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found U-Boot script /boot.scr
299 bytes read in 6 ms (47.9 KiB/s)
## Executing script at 02400000
Saving Environment to MMC... Writing to MMC(0)... OK
switch to partitions #0, OK
mmc0(part 0) is current device
22934016 bytes read in 985 ms (22.2 MiB/s)
Moving Image from 0x80000 to 0x200000, end=18e0000
## Flattened Device Tree blob at 2eff2700
   Booting using the fdt blob at 0x2eff2700
   Using Device Tree in place at 000000002eff2700, end 000000002f002f83

Starting kernel ...