Boot hangs on "Starting kernel" for running system

Hello,

I have five devices setup with Yocto and Mender with the same image, but two devices just stopped booting in the past weeks. I overwrite the first one already but now had a deeper look into the second device: It stopped being able to boot hanging at “starting kernel” and then a blank screen with just a “_” follows.

The device is a Raspberry Pi Zero 2W with a SD Card. The SD Card is still working and readable.
This happend out of nowhere for me, with no update during that time. The system ran with that version for about two weeks now. Sometimes just getting restarted.

Some notes to the image:
Based on scarthgap (including mender-full).

  • MACHINE = “raspberrypi0-2w-64”
  • RPI_USE_U_BOOT = “1”
  • ENABLE_UART = “1”
  • IMAGE_INSTALL:append = " kernel-image kernel-devicetree"
  • IMAGE_FSTYPES:remove = " rpi-sdimg"
  • DISABLE_SPLASH = “1”
  • BOOT_DELAY = “0”
  • BOOT_DELAY_MS = “0”
  • DISABLE_RPI_BOOT_LOGO = “1”
  • ENABLE_SPI_BUS = “1”
  • KERNEL_DEVICETREE = “broadcom/bcm2710-rpi-zero-2.dtb overlays/spi1-1cs.dtbo”

This image worked and booted for some weeks now.

Do you have any idea how to analyze this further and resolve the issue?
Do you have suggestion how to “harden”/stabilize the image so that this cannot happen in a productive environment?

Regards
Florian

Hi @florianb,

That’s definitely strange and unexpected. But even if the microSD cards look like usable and readable, my bet is on them being corrupted. Especially if no read-only filesystem has been used for the root partitions.
Some ideas for digging deeper:

  • comparing the hash of the kernel image file of a good device card to a bad device card
  • swapping cards between the devices

Concerning hardening: going to read-only is definitely the first step. Then come industrial grade micro SD cards, then comes proper storage, like eMMC.

Greetz,
Josef

1 Like