Device running out of space during directory update

Hi,

I try to use directory update to update a large folder (383M). Here is the output of the artifact

    Files:
      name:     update.tar
      size:     354058240
      modified: 2022-02-03 02:30:15 -0600 CST

The deployment failed because of no space left in the device

time="2022-02-03T02:34:32-06:00" level=error msg="Artifact install failed: Payload: can not install Payload: update.tar: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar: write /var/lib/mender/modules/v3/payloads/0000/tree/files/update.tar: no space left on device"

However, it looks like the device has enough space

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  7.9G  5.9G  58% /
devtmpfs        433M     0  433M   0% /dev
tmpfs           438M     0  438M   0% /dev/shm
tmpfs           438M  792K  437M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           438M     0  438M   0% /sys/fs/cgroup
/dev/mmcblk0p4  108M  296K   99M   1% /data
/dev/mmcblk0p1   43M   18M   25M  42% /uboot
tmpfs            88M     0   88M   0% /run/user/1000

What do I miss? Looking forward to any help I can get here. Thanks!

Hello @lincolnxlw, sorry for the obvious question: did you try to run “df” while the Mender client is processing the update? I see no obvious reason why this should fail. Did you also try to create a 400MB file on the device in the same path the Mender client tries to unpack the artifact?

Thank you @tranchitella for your reply. I monitored the whole update process, and there was always enough space in the device.

The path I try to install to is /home/<user>/temp/, so it is not the same path as /var/lib/mender/modules/v3/payloads/0000/tree/files/. I tried to install a smaller folder to the same path and it worked. So I am pretty sure somehow the size of the update is the issue.

From the documentation,

The tar format supports streaming, which Mender takes advantage of. As a Mender Artifact is downloaded from the Mender server or external storage, the Mender client streams the root file system within it directly to the inactive partition, without needing any temporary storage for unpacking it before it is written.

Does this also apply to directory update? Will the inactive partition have something to do with the failure?

Hello @lincolnxlw it should not affect.

The second partition is used when you perform complete root filesystem updates with the A/B strategy. Have you ever tried a full image update before? (To understand if the problem is the file size or the device size + the directory)

Regards,
Luis

Hi @lramirez, thanks for the reply!

There isn’t any problem running full image update. Both A B partition are 15G (6G of the 15G are available for each partition), so there should be enough space to store the tar from the artifact.

After I trimmed down the folder from 383M to 90M, the mender-client is able to download the artifact but failed in trying to compress the old folder (also 90M)

Feb 16 02:33:15 raspberrypi3 mender[1009]: time="2022-02-16T02:33:15-06:00" level=info msg="Running Mender client version: 2.6.1"
Feb 16 02:33:16 raspberrypi3 mender[1009]: time="2022-02-16T02:33:16-06:00" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
Feb 16 02:33:16 raspberrypi3 mender[1009]: time="2022-02-16T02:33:16-06:00" level=info msg="No public key was provided for authenticating the artifact"
Feb 16 02:33:33 raspberrypi3 mender[1009]: time="2022-02-16T02:33:33-06:00" level=info msg="State transition: update-store [Download_Enter] -> update-after-store [Download_Leave]"
Feb 16 02:33:33 raspberrypi3 mender[1009]: time="2022-02-16T02:33:33-06:00" level=info msg="State transition: update-after-store [Download_Leave] -> update-install [ArtifactInstall]"
Feb 16 02:33:36 raspberrypi3 mender[1009]: time="2022-02-16T02:33:36-06:00" level=info msg="Update module output: tar: /var/lib/mender/modules/v3/payloads/0000/tree/tmp/prev_files.tar: Wrote only 2048 of 10240 bytes"
Feb 16 02:33:36 raspberrypi3 mender[1009]: time="2022-02-16T02:33:36-06:00" level=info msg="Update module output: tar: Error is not recoverable: exiting now"

Let me know if there is any other information you need.

Thanks

Lincoln