# Update fails on reboot - Timeout waiting for nonexistent device

**URL:** <https://hub.mender.io/t/update-fails-on-reboot-timeout-waiting-for-nonexistent-device/2789>\
**Category:** General Discussions\
**Tags:** yocto, zeus, growfs-data\
**Created:** [November 6, 2020, 5:31am UTC](https://hub.mender.io/t/update-fails-on-reboot-timeout-waiting-for-nonexistent-device/2789 "2020-11-06T05:31:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![SuicidalLabRat](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/suicidallabrat/32/311_2.png) [@SuicidalLabRat](https://hub.mender.io/u/SuicidalLabRat)\
**Post date:** [November 6, 2020, 5:31am UTC](https://hub.mender.io/t/update-fails-on-reboot-timeout-waiting-for-nonexistent-device/2789/1 "2020-11-06T05:31:22Z")

</div>

**Environment**  
Mender: 2.2 -\> 2.4 update.  
Yocto: Poke - Zeus  
Kernel: 4.19.34  
Systemd: ~243  
Hardware: Bespoke, based on beaglebone design.

**Behavior**  
Deploying an artifact updating the mender client from 2.2 to 2.4, and implementing the growfs-data mechanism.  
The artifact is successfully written to the filesystem and subsequently enters the reboot state, however, during reboot, a timeout waiting for device /dev/disk/by-uuid/EA4E-6CC6 results in a /boot dependency failure, and the system is dropped to the emergency shell. Due to the system landing in emergency mode, nothing beyond the single-user init is achieved, so mender isnt running and is thus unable to fail back, leaving the device indefinitely stuck, baring a power cycled.

The above /boot device does not exist.

**Boot log**  
.  
.  
.  
XT4-fs (mmcblk1p4): resizing filesystem from 716800 to 731136 blocks  
EXT4-fs (mmcblk1p4): resized filesystem to 731136  
[OK] Started Grow File System on /data.  
**[] A start job is running for /dev/dis…by-uuid/EA4E-6CC6 (29s / 1min 30s)**  
\*\* [TIME] Timed out waiting for device /dev/disk/by-uuid/EA4E-6CC6.\*\*  
[DEPEND] Dependency failed for /boot.  
[DEPEND] Dependency failed for Local File Systems.

**/dev/disk/by-uuid/**  
The actual listing for /dev/disk/by-uuid/ (This is identical to the listing taken before deploying the OTA update). Note the absence of the expected device, in its place is 8BB6-604F.  
~ # ls -l /dev/disk/by-uuid/  
lrwxrwxrwx 1 root root 15 Nov 5 04:09 8BB6-604F -\> …/…/mmcblk1p1  
lrwxrwxrwx 1 root root 15 Nov 5 04:09 c83d7053-5c41-48cd-839e-f069afa6c3a1 -\> …/…/mmcblk1p2  
lrwxrwxrwx 1 root root 15 Nov 5 04:09 e5d1f708-a43b-40b1-9f86-c5328d277cce -\> …/…/mmcblk1p3  
lrwxrwxrwx 1 root root 15 Nov 5 04:09 f3286823-d81a-4231-bfa4-ce145921d999 -\> …/…/mmcblk1p4

**New fstab**  
From the emergency shell in the newly updated image, this is the fstab (Note, theres an entry for a vfat /boot, which isnt in our previous images):  
Give root password for maintenance  
(or press Control-D to continue):  
~ # cat /etc/fstab  
# stock fstab - you probably want to override this with a machine specific one

```
	/dev/root / auto defaults 1 1
	proc /proc proc defaults 0 0
	devpts /dev/pts devpts mode=0620,gid=5 0 0
	tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
	tmpfs /var/volatile tmpfs defaults 0 0

	# uncomment this if your device has a SD/MMC/Transflash slot
	#/dev/mmcblk0p1 /media/card auto defaults,sync,noauto 0 0

	# Add tmpfs to host temporary sensor logs.
	tmpfs /var/spool/redaptive tmpfs size=5m 0 0

	# Where the U-Boot environment resides; for devices with SD card support ONLY!
	/dev/mmcblk1p1 /uboot auto defaults,sync 0 2
	/dev/mmcblk1p4 /data auto defaults 0 2
	UUID=EA4E-6CC6 /boot vfat defaults 0 0

```

**Previous fstab**  
fstab from native OS, before OTA update:  
0c1c57f50f6a:~# cat /etc/fstab  
# stock fstab - you probably want to override this with a machine specific one

```
	/dev/root / auto defaults 1 1
	proc /proc proc defaults 0 0
	devpts /dev/pts devpts mode=0620,gid=5 0 0
	tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
	tmpfs /var/volatile tmpfs defaults 0 0

	# uncomment this if your device has a SD/MMC/Transflash slot
	#/dev/mmcblk0p1 /media/card auto defaults,sync,noauto 0 0

	# Add tmpfs to host temporary sensor logs.
	tmpfs /var/spool/redaptive tmpfs size=5m 0 0

	# Where the U-Boot environment resides; for devices with SD card support ONLY!
	/dev/mmcblk1p1 /uboot auto defaults,sync 0 0
	/dev/mmcblk1p4 /data auto defaults 0 0

```

.  
.  
Any idea what may be going on here?

Thanks!  
SLR-

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [November 9, 2020, 9:16am UTC](https://hub.mender.io/t/update-fails-on-reboot-timeout-waiting-for-nonexistent-device/2789/2 "2020-11-09T09:16:10Z")

</div>

> [@SuicidalLabRat](#):
>
> Due to the system landing in emergency mode, nothing beyond the single-user init is achieved, so mender isnt running and is thus unable to fail back, leaving the device indefinitely stuck, baring a power cycled.

Yeah, this is edge case that is hard to handle by Mender alone I think and is something that have come across as well. I have been thinking if one should disable the “emergency/rescue” targets completely in systemd, to avoid entering these and to give the watchdog a chance to trigger if it is enabled.

[systemd.special](https://www.freedesktop.org/software/systemd/man/systemd.special.html#emergency.target)  
[systemd.special](https://www.freedesktop.org/software/systemd/man/systemd.special.html#rescue.target)

> UUID=EA4E-6CC6 /boot vfat defaults

This does not look like something that is provided by meta-mender, but no idea where it would come from.

Maybe you can confirm that it is not written by meta-mender by investigating this part

> <https://github.com/mendersoftware/meta-mender/blob/027f2d720da0d4ecbb4f24e5d98614c590e59023/meta-mender-core/classes/mender-setup-image.inc#L30-L66>
