Update Module output (stderr): No space left on device

General Discussions
1

I’m building an image using Yocto and have integrated mender. I had it working with no problems on a verdin imx8mm SoM. Recently I updated to a verdin imx8mp SoM, and have been getting the error “Update Module output (stderr): No space left on device”. Below is the output of the log set to trace level. And lsblk and df -h commands. There is plenty of room on my eMMC and paritions, but I must not have a setting correct in Mender.

When it says “no space left on device”… what device? What is it trying to copy and to what partition?

Tail of mender trace log:
39%record_id=312492 severity=trace time=“2024-Oct-12 14:43:23.500547” name=“Global” msg=“Wrote 5118115840 bytes to Update Module”
39%record_id=312493 severity=trace time=“2024-Oct-12 14:43:23.500753” name=“Global” msg=“Wrote 5118132224 bytes to Update Module”
39%record_id=312494 severity=trace time=“2024-Oct-12 14:43:23.500880” name=“Global” msg=“Wrote 5118148608 bytes to Update Module”
39%record_id=312495 severity=trace time=“2024-Oct-12 14:43:23.501157” name=“Global” msg=“Wrote 5118164992 bytes to Update Module”
39%record_id=312496 severity=info time=“2024-Oct-12 14:43:23.501390” name=“Global” msg=“Update Module output (stderr): No space left on device: Error while writing data: No space left on device”
record_id=312497 severity=info time=“2024-Oct-12 14:43:25.005115” name=“Global” msg=“Sending SIGTERM to PID 4745”
record_id=312498 severity=info time=“2024-Oct-12 14:43:25.006928” name=“Global” msg=“PID 4745 exited with status 15”
record_id=312499 severity=info time=“2024-Oct-12 14:43:25.014630” name=“Global” msg=“Running State Script: /etc/mender/scripts/Download_Error_99”
record_id=312500 severity=info time=“2024-Oct-12 14:43:25.064986” name=“Global” msg=“Collected output (stderr) while running script: 7.0.0-devel-20241011151943+build.0: Download_Error_99 was called!”
record_id=312501 severity=error time=“2024-Oct-12 14:43:25.095563” name=“Global” msg=“Broken pipe: AsyncWrite failed”
Installation failed. System not modified.

Disk Information:
root@verdin-imx8mp-15128137:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 58.6G 0 disk
-sda1 8:1 1 58.6G 0 part /run/media/sda1 mmcblk2 179:0 0 29.1G 0 disk |-mmcblk2p1 179:1 0 32M 0 part /uboot |-mmcblk2p2 179:2 0 4.8G 0 part / |-mmcblk2p3 179:3 0 4.8G 0 part -mmcblk2p4 179:4 0 19.6G 0 part /data
mmcblk2boot0 179:8 0 4M 1 disk
mmcblk2boot1 179:16 0 4M 1 disk
root@verdin-imx8mp-15128137:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 4.4G 1.7G 2.6G 40% /
devtmpfs 3.7G 4.0K 3.7G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 1.6G 5.5M 1.6G 1% /run
tmpfs 3.9G 554M 3.4G 14% /tmp
tmpfs 3.9G 36K 3.9G 1% /var/volatile
/dev/mmcblk2p1 32M 4.0K 32M 1% /uboot
/dev/mmcblk2p4 20G 66M 19G 1% /data
t
mpfs 796M 4.0K 796M 1% /run/user/0
/dev/sda1 59G 3.3G 56G 6% /run/media/sda1

And my MENDER local.conf configs per meta-mender-community:
MENDER_STORAGE_DEVICE:verdin-imx8mp = “/dev/mmcblk2”
MENDER_STORAGE_TOTAL_SIZE_MB:verdin-imx8mp = “29844”
MENDER_DATA_PART_SIZE_MB = “5120”
MENDER_BOOT_PART_SIZE_MB = “32”
OFFSET_SPL_PAYLOAD = “”
MENDER_IMAGE_BOOTLOADER_BOOTSECTOR_OFFSET:verdin-imx8mp = “0”
MENDER_UBOOT_STORAGE_INTERFACE:verdin-imx8mp = “mmc”
MENDER_UBOOT_STORAGE_DEVICE:verdin-imx8mp = “2”

2

Hi @gk1097,

From a first glance that definitely sounds strange. Just to clarify: this is a standard root filesystem update process? Are there any special scripts involved?

As the log does not give away the details of the Update Module and its states being invoked, can you maybe look a bit further up, like what happens right before the sequence of “Wrote xxx bytes”?

Greetz,
Josef

3

@TheYoctoJester Thanks for the reply. It’s just a standard root filesystem update, yes. I do have an ArtifactInstall_Leave script that just replaces the user’s home directory in the new rootfs with a symlink to the persistent /data partition. But that script hasn’t run at the point where this errors so I don’t think that affects this error.

I will duplicate this issue again and check further up in the log. I’ve actually been switching back & forth between my new imx8mp environment and my old imx8mm environment as I debug other unrelated issues with the hardware switch (like gpio configs, etc). So my workspace is a bit of a mess. But my latest imx8mp build is not throwing this error. One notable change (among many seemingly unrelated changes I’ve made) is that I set MENDER_STORAGE_TOTAL_SIZE_MB:verdin-imx8mp = “14942”. That’s the value that I had in my imx8mm workspace because that chip only has a 16GB eMMC. The imx8mp has a 32GB eMMC and the command “blockdev --getsize64 /dev/mmcblk0 | xargs -i% expr % / 1048576” gives me 29844 on that chip. But that’s when I was getting the error - when I had the 29844 value. Kinda seems like a smaller MENDER_STORAGE_TOTAL_SIZE_MB works… but this issue was somewhat intermittent and I haven’t done a ton of mender-updates in the last few days, so I’m not yet confident this issue is totally resolved.

Does it make sense to you that MENDER_STORAGE_TOTAL_SIZE_MB might resolve the problem? I’m not 100% sure on the details of how that variable affects mender partitioning and size checks.

4

Hi @gk1097,

Thanks again for reporting the issue! Your question together with Raspberry Pi 4 Model B + Raspbian - #13 by amcdonald-ipa let us identify the probable root cause. We’re now working on the fix and releasing it. I will report back with more details no later than Thursday, Oct 17th.

Greetz,
Josef

5

The workaround of using a smaller MENDER_STORAGE_TOTAL_SIZE_MB value has been working. But I’m just curious, @TheYoctoJester have you been able to release a version with a long-term fix?