Update fails on reboot - Timeout waiting for nonexistent device

General Discussions
, ,
1

Environment
Mender: 2.2 -> 2.4 update.
Yocto: Poke - Zeus
Kernel: 4.19.34
Systemd: ~243
Hardware: Bespoke, based on beaglebone design.

Behavior
Deploying an artifact updating the mender client from 2.2 to 2.4, and implementing the growfs-data mechanism.
The artifact is successfully written to the filesystem and subsequently enters the reboot state, however, during reboot, a timeout waiting for device /dev/disk/by-uuid/EA4E-6CC6 results in a /boot dependency failure, and the system is dropped to the emergency shell. Due to the system landing in emergency mode, nothing beyond the single-user init is achieved, so mender isnt running and is thus unable to fail back, leaving the device indefinitely stuck, baring a power cycled.

The above /boot device does not exist.

Boot log
.
.
.
XT4-fs (mmcblk1p4): resizing filesystem from 716800 to 731136 blocks
EXT4-fs (mmcblk1p4): resized filesystem to 731136
[ OK ] Started Grow File System on /data.
[ ] A start job is running for /dev/dis…by-uuid/EA4E-6CC6 (29s / 1min 30s)
** [ TIME ] Timed out waiting for device /dev/disk/by-uuid/EA4E-6CC6.**
[DEPEND] Dependency failed for /boot.
[DEPEND] Dependency failed for Local File Systems.

/dev/disk/by-uuid/
The actual listing for /dev/disk/by-uuid/ (This is identical to the listing taken before deploying the OTA update). Note the absence of the expected device, in its place is 8BB6-604F.
~ # ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root 15 Nov 5 04:09 8BB6-604F -> …/…/mmcblk1p1
lrwxrwxrwx 1 root root 15 Nov 5 04:09 c83d7053-5c41-48cd-839e-f069afa6c3a1 -> …/…/mmcblk1p2
lrwxrwxrwx 1 root root 15 Nov 5 04:09 e5d1f708-a43b-40b1-9f86-c5328d277cce -> …/…/mmcblk1p3
lrwxrwxrwx 1 root root 15 Nov 5 04:09 f3286823-d81a-4231-bfa4-ce145921d999 -> …/…/mmcblk1p4

New fstab
From the emergency shell in the newly updated image, this is the fstab (Note, theres an entry for a vfat /boot, which isnt in our previous images):
Give root password for maintenance
(or press Control-D to continue):
~ # cat /etc/fstab
# stock fstab - you probably want to override this with a machine specific one

	/dev/root            /                    auto       defaults              1  1
	proc                 /proc                proc       defaults              0  0
	devpts               /dev/pts             devpts     mode=0620,gid=5       0  0
	tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
	tmpfs                /var/volatile        tmpfs      defaults              0  0

	# uncomment this if your device has a SD/MMC/Transflash slot
	#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0


	# Add tmpfs to host temporary sensor logs.
	tmpfs   /var/spool/redaptive    tmpfs   size=5m 0 0

	# Where the U-Boot environment resides; for devices with SD card support ONLY!
	/dev/mmcblk1p1       /uboot               auto       defaults,sync         0  2
	/dev/mmcblk1p4       /data                auto       defaults              0  2
	UUID=EA4E-6CC6  	 /boot   			  vfat    	 defaults        	   0  0

Previous fstab
fstab from native OS, before OTA update:
0c1c57f50f6a:~# cat /etc/fstab
# stock fstab - you probably want to override this with a machine specific one

	/dev/root            /                    auto       defaults              1  1
	proc                 /proc                proc       defaults              0  0
	devpts               /dev/pts             devpts     mode=0620,gid=5       0  0
	tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
	tmpfs                /var/volatile        tmpfs      defaults              0  0

	# uncomment this if your device has a SD/MMC/Transflash slot
	#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0


	# Add tmpfs to host temporary sensor logs.
	tmpfs   /var/spool/redaptive    tmpfs   size=5m 0 0

	# Where the U-Boot environment resides; for devices with SD card support ONLY!
	/dev/mmcblk1p1       /uboot               auto       defaults,sync         0  0
	/dev/mmcblk1p4       /data                auto       defaults              0  0

.
.
Any idea what may be going on here?

Thanks!
SLR-

2

Yeah, this is edge case that is hard to handle by Mender alone I think and is something that have come across as well. I have been thinking if one should disable the “emergency/rescue” targets completely in systemd, to avoid entering these and to give the watchdog a chance to trigger if it is enabled.

systemd.special
systemd.special

UUID=EA4E-6CC6 /boot vfat defaults

This does not look like something that is provided by meta-mender, but no idea where it would come from.

Maybe you can confirm that it is not written by meta-mender by investigating this part