Failed to restore state data: mdb_txn_commit: no space left on device

I am experimenting with the Directory Update Module. I only have 4GB emmc space on the Beaglebones, tight for docker so I want to push the nodejs diectory including the node_modules, captured from a development reference device.

My artifact *.mender file is 37MB in size, the unpacked nodejs tree is 200MB. I have 387MB available on the partition. I deployed the artifact and it never finished the downloading stage. I rebooted my target device and aborted the deployment. Now the device doesn’t check into Mender.io anymore, and I see complains about disk space in the journalctl output.

My first question is if there could be a stale artifact on the device now that causes this behavior? If so, how can I remove it to make it connect again? I decommissioned it and wait for it to come in.

Secondly I am trying to learn about the update. I feel like I should have enough space to deploy this directory update. If understand the “directory” shell script correctly it first tars up my old 200MB nodejs directory, probably to a similar sized 37MB tar ball, then deletes the target directory, then unpacks the new artifact. So I assume there aren’t multiple unpacked directory trees on the device at any time and it should fit, however I get this space error.

I hope someone can help me out here. I feel that I am super close, yet this is another roadblock I don’t understand. Once my device connects again I want to add a pre-script stopping my nodejs service, and then a post-script restarting it (or maybe rebooting). Is this what people do for nodejs apps if there is no room for docker?

Thanks for all support here, I really appreciate this community and hope I can give back some day :slight_smile:
Here the output:

sudo journalctl -u mender-client
-- Logs begin at Sat 2022-07-02 19:34:26 PDT, end at Sat 2022-07-02 19:38:37 PDT. --
Jul 02 19:34:57 JF-912-00031-200804F systemd[1]: Started Mender OTA update service.
Jul 02 19:35:03 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:03-07:00" level=info msg="Loaded configuration file: /var/lib/mender/mender.conf"
Jul 02 19:35:03 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:03-07:00" level=info msg="Loaded configuration file: /etc/mender/mender.conf"
Jul 02 19:35:03 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:03-07:00" level=info msg="'UpdateControlMapExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 2*UpdatePollIntervalSeconds"
Jul 02 19:35:03 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:03-07:00" level=info msg="'UpdateControlMapBootExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 600 seconds"
Jul 02 19:35:14 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:14-07:00" level=info msg="Mender running on partition: /dev/mmcblk1p2"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=info msg="State transition: init [none] -> init [none]"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="Failed to restore state data: mdb_txn_commit: no space left on device"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="Failed to restore state data: mdb_txn_commit: no space left on device"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=info msg="State transition: init [none] -> update-error [ArtifactFailure]"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="Could not write state data to persistent storage: mdb_txn_commit: no space left on device"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="Could not write state data to persistent storage: mdb_txn_commit: no space left on device"
Jul 02 19:35:22 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="fatal error: mdb_txn_commit: no space left on device"
Jul 02 19:35:23 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:22-07:00" level=error msg="fatal error: mdb_txn_commit: no space left on device"
Jul 02 19:35:23 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:23-07:00" level=error msg="fatal error: mdb_txn_commit: no space left on device"
Jul 02 19:35:23 JF-912-00031-200804F mender[1312]: time="2022-07-02T19:35:23-07:00" level=error msg="fatal error: mdb_txn_commit: no space left on device"
Jul 02 19:35:23 JF-912-00031-200804F systemd[1]: mender-client.service: Main process exited, code=exited, status=1/FAILURE
Jul 02 19:35:23 JF-912-00031-200804F systemd[1]: mender-client.service: Failed with result 'exit-code'.


df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            211M     0  211M   0% /dev
tmpfs            48M  2.6M   46M   6% /run
/dev/mmcblk1p2  1.8G  1.3G  387M  77% /
tmpfs           239M     0  239M   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           239M     0  239M   0% /sys/fs/cgroup
/dev/mmcblk1p4   12M   12M     0 100% /data
/dev/mmcblk1p1   16M  6.9M  9.1M  44% /boot/efi
/dev/mmcblk0p1   30G  7.4G   21G  27% /media
tmpfs            48M     0   48M   0% /run/user/1001

Oh I realize that my /data partition is full, I wasn’t aware that Mender directory update module copies the update.tar to: /data/mender/modules/v3/payloads/0000/tree/files

Can I configure it to be copied to my default partition /dev/mmcblk1p2?