Standalone deployment on Raspberry Pi hangs during boot

I am trying to use mender-convert together with Standalone deployment, but the first boot after running “mender install” hangs with message:

Waiting for root device /dev/mmcblk0p3...

Here are the steps that I took:

  1. Flash a 16 GB SD card with the official Raspberry Pi image 2021-10-30-raspios-bullseye-armhf-lite.img (Linux kernel 5.10.63).

  2. Boot with this SD card and install a couple of packages (e.g. samba, unixodbc).

  3. Create an image from this SD card on Ubuntu 18.04 by running:

dd if=/dev/sdb of=vtx.img bs=1M conv=fdatasync status=progress
  1. Run mender-convert to create a mender image from vtx.img:
sigmund@kassa1:~/mender-convert$ ./scripts/bootstrap-rootfs-overlay-demo-server.sh \
    --output-dir ${PWD}/rootfs_overlay_demo \
    --server-ip 192.168.80.141

sigmund@kassa1:~/mender-convert$ cat configs/raspberrypi3_config
RASPBERRYPI_CONFIG="raspberrypi3"
RASPBERRYPI_KERNEL_IMAGE="kernel7.img"
MENDER_KERNEL_IMAGETYPE="zImage"
MENDER_DEVICE_TYPE="raspberrypi3"
MENDER_DATA_PART_SIZE_MB=2000
MENDER_STORAGE_TOTAL_SIZE_MB=12000

source configs/raspberrypi_config

sigmund@kassa1:~/mender-convert$ MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert --disk-image input/vtx.img --config configs/raspberrypi3_config --overlay rootfs_overlay_demo/
  1. Flash the new image:
sudo dd of=/dev/sdb if=deploy/vtx-raspberrypi3-mender.img bs=1M conv=fdatasync status=progress
  1. Boot and disable mender-client as described in " Standalone deployment":
sudo systemctl stop mender-client
sudo systemctl disable mender-client
sudo systemctl mask mender-client
  1. Repeat steps 1-5 using another SD card and with a newer Raspberry Pi image 2022-01-28-raspios-bullseye-armhf-lite.img (Linux kernel 5.10.92).

  2. Copy the vtx-raspberrypi3-mender.mender file from step 7 onto the first SD card.

  3. Run “mender install”:

pi@raspberrypi:/data $ sudo mender install vtx-raspberrypi3-mender.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] 'UpdateControlMapExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 2*UpdatePollIntervalSeconds
INFO[0000] 'UpdateControlMapBootExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 600 seconds
INFO[0000] Mender running on partition: /dev/mmcblk0p2
INFO[0000] Start updating from local image file: [vtx-raspberrypi3-mender.mender]
Installing Artifact of size 519545344...
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Opening device "/dev/mmcblk0p3" for writing
INFO[0000] Native sector size of block device /dev/mmcblk0p3 is 512 bytes. Mender will write in chunks of 1048576 bytes
.............................................................. - 100 %
INFO[0186] All bytes were successfully written to the new partition
INFO[0186] The optimized block-device writer wrote a total of 2013 frames, where 0 frames did need to be rewritten (i.e., skipped)
INFO[0186] Wrote 2110064640/2110064640 bytes to the inactive partition
INFO[0186] Enabling partition with new image installed to be a boot candidate: 3
Use 'commit' to update, or 'rollback' to roll back the update.
At least one payload requested a reboot of the device it updated.
  1. After running “sudo reboot”, the Raspberry Pi hangs during boot with the message:
Waiting for root device /dev/mmcblk0p3...
  1. After booting again, uname -a still gives me Linux raspberrypi 5.10.63-v7+. Unfortunately sudo grep -R mmcblk0p3 /var/log/ comes up empty, but I have attached a picture of the boot sequence log output.

I have tried a couple of times with tweaks to MENDER_STORAGE_TOTAL_SIZE_MB, but I always get the same hang. One thing I find strange is this:

pi@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.1G  1.6G  465M  77% /
devtmpfs        333M     0  333M   0% /dev
tmpfs           462M     0  462M   0% /dev/shm
tmpfs           185M  1.7M  183M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
/dev/mmcblk0p4  1.9G  502M  1.3G  28% /data
/dev/mmcblk0p1  253M   43M  210M  17% /uboot
tmpfs            93M     0   93M   0% /run/user/1000
pi@raspberrypi:~ $ cat /proc/partitions
major minor  #blocks  name

   1        0       4096 ram0
   1        1       4096 ram1
   1        2       4096 ram2
   1        3       4096 ram3
   1        4       4096 ram4
   1        5       4096 ram5
   1        6       4096 ram6
   1        7       4096 ram7
   1        8       4096 ram8
   1        9       4096 ram9
   1       10       4096 ram10
   1       11       4096 ram11
   1       12       4096 ram12
   1       13       4096 ram13
   1       14       4096 ram14
   1       15       4096 ram15
 179        0   15558144 mmcblk0
 179        1     262144 mmcblk0p1
 179        2    4976640 mmcblk0p2
 179        3    4976640 mmcblk0p3
 179        4    2048000 mmcblk0p4
pi@raspberrypi:~ $ cat /proc/cmdline
coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=HDMI-A-1:1680x1050M@60 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 fsck.repair=yes rootwait

I would expect the size of /dev/root to be that same as mmcblk0p2. The small size of dev/root is why I needed to copy the .mender file to /data.

Any ideas about what I am doing wrong?

Best regards,
Sigmund

1 Like

Resizing the partitions like this fixes the unexpected /dev/root/ and mmcblk0p2 size mismatch (thanks to Johan at IOT Solutions!):

sudo resize2fs /dev/mmcblk0p2
sudo resize2fs /dev/mmcblk0p3
sudo umount -l /data
sudo parted /dev/mmcblk0 --script resizepart 4 100%
sudo parted /dev/mmcblk0 'unit s print' free
sudo reboot
2 Likes

Hi @sigmund,

thanks a lot for sharing the solution and helping others!

Greetz,
Josef

Hi @ TheYoctoJester

Note that the resizes don’t necessarily fix the original problem. It will only fixe the partition sizes and makes all disk space available.

1 Like

@TheYoctoJester and @johan, unfortunately the problem with the hang during boot is unaffected by the resizing. Sorry about the confusion.

Oh @sigmund,

Thats too bad. One thing that just came to my mind, can you try gathering a little more information about the state of the mmc card? Specifically, the output of blkid?

Greetz,
Josef

Here you go, @TheYoctoJester:

pi@raspberrypi:~ $ blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="C839-E506" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="6634552a-01"
/dev/mmcblk0p2: UUID="e40db878-c64f-4b1b-8fe1-838735a31e8e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6634552a-02"
/dev/mmcblk0p3: UUID="e40db878-c64f-4b1b-8fe1-838735a31e8e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6634552a-03"
/dev/mmcblk0p4: UUID="5c0d1554-fb90-4ace-9260-14e9ddae83b8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6634552a-04"

Thanks @sigmund!

Can you mount /dev/mmcblk0p3 manually to some directory? So the partition is technically functional?

Greetz,
Josef

@TheYoctoJester:

pi@raspberrypi:~ $ mkdir x
pi@raspberrypi:~ $ sudo mount /dev/mmcblk0p3 x
pi@raspberrypi:~ $ ll x
total 92
drwxrwxr-x 20 root root  4096 Mar 18 14:30 ./
drwxr-xr-x  3 pi   pi    4096 Mar 24 16:02 ../
lrwxrwxrwx  1 root root     7 Jan 28 01:01 bin -> usr/bin/
drwxr-xr-x  2 root root  4096 Mar 24 14:46 boot/
drwxr-xr-x  2 root root  4096 Mar 24 14:46 data/
drwxr-xr-x  4 root root  4096 Jan 28 01:02 dev/
drwxrwxr-x 81 root root  4096 Mar 24 09:36 etc/
-rw-rw-r--  1 root root     0 Mar 18 14:28 .gitkeep
drwxr-xr-x  3 root root  4096 Jan 28 01:04 home/
lrwxrwxrwx  1 root root     7 Feb  2 13:49 lib -> usr/lib/
drwx------  2 root root 16384 Jan 28 01:22 lost+found/
drwxr-xr-x  2 root root  4096 Jan 28 01:02 media/
drwxr-xr-x  2 root root  4096 Jan 28 01:02 mnt/
drwxr-xr-x  2 root root  4096 Jan 28 01:02 opt/
drwxr-xr-x  2 root root  4096 Dec 21 15:49 proc/
drwx------  2 root root  4096 Jan 28 01:02 root/
drwxr-xr-x  5 root root  4096 Jan 28 01:06 run/
lrwxrwxrwx  1 root root     8 Jan 28 01:01 sbin -> usr/sbin/
drwxr-xr-x  2 root root  4096 Jan 28 01:02 srv/
drwxr-xr-x  2 root root  4096 Dec 21 15:49 sys/
drwxrwxrwt  7 root root  4096 Mar 20 15:21 tmp/
drwxr-xr-x  2 root root  4096 Mar 24 14:46 uboot/
drwxr-xr-x 11 root root  4096 Feb  2 13:49 usr/
drwxr-xr-x 11 root root  4096 Jan 28 01:22 var/

@TheYoctoJester, the boot also hangs when I only perform step 5 from Integration checklist:

sudo fw_setenv mender_boot_part 3
sudo fw_setenv mender_boot_part_hex 3
sudo reboot

Which is quite odd. I am trying with a different SD card now.

The hang on boot after running “mender install” happens the same way with another SD card (these are industrial grade cards which have been very robust in the past).

I am really just trying to create an artifact as described in the following note here (I made the relevant sentence bold):

Blockquote
Do not call apt upgrade on a device with Mender system updates enabled. Doing so may brick your device! Always update single applications only. The reason is that kernel and boot loader Debian packages do not expect the Mender partition and filesystem layout, and therefore will try to perform actions that conflict with Mender. If you need to run apt upgrade, do it on a pristine system without Mender installed, and then convert it to a Mender image afterwards. This restriction may be lifted in the future.

Here is the output of fw_printenv after running “mender install” and before the reboot that hangs:

pi@raspberrypi:~ $ sudo fw_printenv
altbootcmd=run mender_altbootcmd; run bootcmd
arch=arm
baudrate=115200
board=rpi
board_name=3 Model B
board_rev=0x8
board_rev_scheme=1
board_revision=0xA02082
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootarm.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc0 mmc1 usb0 pxe dhcp
bootargs=coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=HDMI-A-1:1680x1050M@60 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  console=ttyS0,115200 console=tty1 root=${mender_kernel_root} rootfstype=ext4 fsck.repair=yes rootwait
bootcmd=run distro_bootcmd
bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00010:UNDI:003000;setenv bootp_arch 0xa;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fi
bootcmd_usb0=devnum=0; run usb_boot
bootdelay=2
bootfstype=fat
bootlimit=1
cpu=armv7
devplist=1
dhcpuboot=usb start; dhcp u-boot.uimg; bootm
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
efi_dtb_prefixes=/ /dtb/ /dtb/current/
ethaddr=b8:27:eb:4c:89:2b
fdt_addr=2eff8900
fdt_addr_r=0x02600000
fdt_high=ffffffff
fdtaddr=2eff8900
fdtcontroladdr=3b3cd9a0
fdtfile=bcm2837-rpi-3-b.dtb
fileaddr=2400000
filesize=238
initrd_high=ffffffff
kernel_addr_r=0x00080000
load_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}
loadaddr=0x00200000
mender_altbootcmd=if test ${mender_boot_part} = 2; then setenv mender_boot_part 3; setenv mender_boot_part_hex 3; else setenv mender_boot_part 2; setenv mender_boot_part_hex 2; fi; setenv upgrade_available 0; saveenv; run mender_setup
mender_boot_kernel_type=bootm
mender_check_saveenv_canary=1
mender_dtb_name=bcm2710-rpi-cm3.dtb
mender_kernel_name=uImage
mender_saveenv_canary=1
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; if test "${mender_systemd_machine_id}" != ""; then setenv bootargs systemd.machine_id=${mender_systemd_machine_id} ${bootargs}; fi; setenv mender_kernel_root /dev/mmcblk0p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk0p2; else setenv mender_boot_part_name /dev/mmcblk0p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_post_setup_commands; fi
mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi
mender_uboot_boot=mmc 0:1
mender_uboot_dev=0
mender_uboot_if=mmc
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
pxefile_addr_r=0x02500000
ramdisk_addr_r=0x02700000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; if test -z "${fdtfile}" -a -n "${soc}"; then setenv efi_fdtfile ${soc}-${board}${boardver}.dtb; fi; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootarm.efi; then echo Found EFI removable media binary efi/boot/bootarm.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
scriptaddr=0x02400000
serial#=00000000e34c892b
soc=bcm283x
stderr=serial,vidconsole
stdin=serial,usbkbd
stdout=serial,vidconsole
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
usbethaddr=b8:27:eb:4c:89:2b
vendor=raspberrypi
mender_boot_part=3
mender_boot_part_hex=3
upgrade_available=1
bootcount=0

Is there a flag I can set to make Mender print out more debugging information while booting?