Environment
Mender: 2.2 -> 2.4 update.
Yocto: Poke - Zeus
Kernel: 4.19.34
Systemd: ~243
Hardware: Bespoke, based on beaglebone design.
Behavior
Deploying an artifact updating the mender client from 2.2 to 2.4, and implementing the growfs-data mechanism.
The artifact is successfully written to the filesystem and subsequently enters the reboot state, however, during reboot, a timeout waiting for device /dev/disk/by-uuid/EA4E-6CC6 results in a /boot dependency failure, and the system is dropped to the emergency shell. Due to the system landing in emergency mode, nothing beyond the single-user init is achieved, so mender isnt running and is thus unable to fail back, leaving the device indefinitely stuck, baring a power cycled.
The above /boot device does not exist.
Boot log
.
.
.
XT4-fs (mmcblk1p4): resizing filesystem from 716800 to 731136 blocks
EXT4-fs (mmcblk1p4): resized filesystem to 731136
[ OK ] Started Grow File System on /data.
[ ] A start job is running for /dev/dis…by-uuid/EA4E-6CC6 (29s / 1min 30s)
** [ TIME ] Timed out waiting for device /dev/disk/by-uuid/EA4E-6CC6.**
[DEPEND] Dependency failed for /boot.
[DEPEND] Dependency failed for Local File Systems.
/dev/disk/by-uuid/
The actual listing for /dev/disk/by-uuid/ (This is identical to the listing taken before deploying the OTA update). Note the absence of the expected device, in its place is 8BB6-604F.
~ # ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root 15 Nov 5 04:09 8BB6-604F -> …/…/mmcblk1p1
lrwxrwxrwx 1 root root 15 Nov 5 04:09 c83d7053-5c41-48cd-839e-f069afa6c3a1 -> …/…/mmcblk1p2
lrwxrwxrwx 1 root root 15 Nov 5 04:09 e5d1f708-a43b-40b1-9f86-c5328d277cce -> …/…/mmcblk1p3
lrwxrwxrwx 1 root root 15 Nov 5 04:09 f3286823-d81a-4231-bfa4-ce145921d999 -> …/…/mmcblk1p4
New fstab
From the emergency shell in the newly updated image, this is the fstab (Note, theres an entry for a vfat /boot, which isnt in our previous images):
Give root password for maintenance
(or press Control-D to continue):
~ # cat /etc/fstab
# stock fstab - you probably want to override this with a machine specific one
/dev/root / auto defaults 1 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
tmpfs /var/volatile tmpfs defaults 0 0
# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1 /media/card auto defaults,sync,noauto 0 0
# Add tmpfs to host temporary sensor logs.
tmpfs /var/spool/redaptive tmpfs size=5m 0 0
# Where the U-Boot environment resides; for devices with SD card support ONLY!
/dev/mmcblk1p1 /uboot auto defaults,sync 0 2
/dev/mmcblk1p4 /data auto defaults 0 2
UUID=EA4E-6CC6 /boot vfat defaults 0 0
Previous fstab
fstab from native OS, before OTA update:
0c1c57f50f6a:~# cat /etc/fstab
# stock fstab - you probably want to override this with a machine specific one
/dev/root / auto defaults 1 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
tmpfs /var/volatile tmpfs defaults 0 0
# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1 /media/card auto defaults,sync,noauto 0 0
# Add tmpfs to host temporary sensor logs.
tmpfs /var/spool/redaptive tmpfs size=5m 0 0
# Where the U-Boot environment resides; for devices with SD card support ONLY!
/dev/mmcblk1p1 /uboot auto defaults,sync 0 0
/dev/mmcblk1p4 /data auto defaults 0 0
.
.
Any idea what may be going on here?
Thanks!
SLR-