Raspberry Pi 4 stuck on boot

General Discussions
, ,
1

Hi.

We are building Raspberry Pi images using RPi-Distro/pi-gen. The images are based on Debian 12 arm64 and then they are converted using mender-convert version 4.3.0.

Recently, a newer Kernel version was released by Debian (version 6.12.62) and since then images converted by mender-convert are stuck on boot with the following output:

MESS:00:00:17.285959:0: brfs: File read: 1059 bytes
MESS:00:00:17.291733:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:17.294300:0: Read command line from file 'cmdline.txt':
MESS:00:00:17.300177:0: 'console=serial0,115200 console=tty1 root=${mender_kernel_root} rootfstype=ext4 fsck.repair=yes rootwait splash'
MESS:00:00:17.437867:0: brfs: File read: 111 bytes
MESS:00:00:17.496318:0: brfs: File read: /mfs/sd/kernel8.img
MESS:00:00:17.498870:0: Loaded 'kernel8.img' to 0x200000 size 0x9b0c8
MESS:00:00:17.505768:0: Kernel relocated to 0x80000
MESS:00:00:17.509627:0: Device tree loaded to 0x2e659e00 (size 0xe1f5)
MESS:00:00:17.518940:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:17.525182:0: uart: Baud rate change done...
MESS:00:00:17.527199:0: uart: Baud rate change done...
MESS:00:00:17.532584:0: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined
MESS:00:00:17.540201:0: Watchdog stopped
MESS:00:00:17.543698:0: arm_loader: Starting ARM with 948MB

Note, that we do not define specific versions to use when building the image with pi-gen and no changes were made in the process which builds the Raspberry Pi images.

To confirm that the images built by pi-gen are valid, we burned them using rpi-imager and tried to boot - the Raspberry Pi boot successfully with those images (with the newer Kernel version).

Any idea what can be the root cause for stuck boot encountered on images converted using mender-convert?

Thanks.

2

To me this sounds similar to the issue I encountered in 2021. With u-boot in the boot chain it gets difficult to have a synchronous update of the device tree binary (dtb) and the kernel. At some point I got stuck on an old device tree binary and the new kernel did not boot anymore.

3

@lueschem, thanks.

We are facing this issue even after burning the *.img.gz file generated by mender-convert on the SD card (i.e. not via the mender-update flow), which means that this is a fresh installation that has everything sync.

4

Hi @yakirm-cr,

My gut feeling is that it’s not necessarily the kernel, but some other part of the plumbing which Raspberry Pi OS changed. They are sometimes quite trigger happy there unfortunately. As I’m not familiar with the pi-gen workflow, is my guess correct so far:

  • the script is obtained via their Github repo
  • you have a known state which works
  • you have a known state which is broken

If that is accurate, then the most straightforward approach would be a git bisect. That will get you to the offending change quite quickly, assuming that there’s not thousands of commits to pi-gen in the mean time. Once that is known we can hopefully propose a solution.

Greetz,
Josef

5

@TheYoctoJester,

You were right - some other part of the plumbing which Raspberry Pi OS changed with the new Kernel version.

I rebuilt the U-Boot (using mendersoftware/mender-convert-integration-scripts) with additional debug logging and based on the additional debug messages I found that the new 6.12.62 device tree sets compatible = "arm,pl011-axi" for the UART. U-Boot 2024.04 only matches "arm,pl011", so the PL011 driver never binds, which causes the Raspberry Pi to hang on boot.

In order to resolve I applied the following patch on the drivers/serial/serial_pl01x.c on the branch mender-rpi-2024.04 of mendersoftware/uboot-mender:

 #if CONFIG_IS_ENABLED(OF_REAL)
 static const struct udevice_id pl01x_serial_id[] ={
 	{.compatible = "arm,pl011", .data = TYPE_PL011},
+	{.compatible = "arm,pl011-axi", .data = TYPE_PL011},
 	{.compatible = "arm,pl010", .data = TYPE_PL010},
 	{}
 };

With this change the issue is resolved. :slight_smile:

6

Great @yakirm-cr,

Thanks for sharing!

7

We ran into the same underlying arm,pl011-axi compatibility issue on a Revolution Pi Connect SE / Raspberry Pi CM4S and completed a deeper root-cause investigation here:

We confirmed that U-Boot 2024.04 does not match arm,pl011-axi when it is the only PL011 compatible string. Patching only UART0 in the failing DTB to add the legacy fallbacks makes the image boot again.

I have opened a PR against Mender’s mender-rpi-2024.04 branch to address this directly in the Mender-maintained U-Boot fork:

The long-term fix can also be considered upstream in U-Boot, but merging the small compatibility fix into Mender’s current 2024.04-based branch would immediately harden the existing Raspberry Pi integration for Mender users.

If others here are affected by the same issue, it would be helpful to reference the PR and add your test results/use case there as well.

8

Hi @bartvanbos,

Thanks for sharing and submitting!

Greetz,
Josef