My updates are stalling on a handful Raspberry Pi 3b+ devices because mender is running out of memory. The system’s memory utilization normally hovers around 40%, but during the update, I see it spike to 100% and then the client crashed with an “out of memory” error. Most devices update successfully…

Hi, Are you using gzip or lzma compressed Artifacts? I suspect that this might have some impact on memory utilization while the Artifacts are decompressed. gzip would probably use less memory, while lzma would be heavier versus much better compression then gzip. But saying that, I would not have …

Thanks for taking a look, Mirza! I’m using mender-convert, which I think defaults to gzip compression IIRC? I just watched an update occur, and mender was definitely consuming 60% of memory according to top (see screenshot). Here’s the output for cat /proc/$(pidof mender)/status | grep Vm: VmPea…

Thanks for sharing the information. Do you have the possibility to test this with the Mender client (2.2.0)? I know that there have been memory leaks in older versions caused by the golang standard library. As a side note: since mender.service only restarts on abort, mender does not recover from …

I have ssh access to some of the devices, so I can try updating mender client manually. Should I just follow the install from source steps at https://github.com/mendersoftware/mender ? Will that cause any issues if the image was created with an older mender-convert?

Oh nevermind, just found the debian packages at https://docs.mender.io/2.3/downloads Second question still stands of whether there would be any conflict with an older integration?

[image] neil: Second question still stands of whether there would be any conflict with an older integration? Hard to say, if you just use the binary from the deb package that would minimize any potential conflicts. But I do not know if I can recommend to this on a device that is in the field…

I’m unable to recreate the issue on a device locally. It’s only happening on 5 of the 20 devices this latest update has been pushed to so far, so I need to find a way to fix it remotely. I tried killing all user processes to free up more RAM before performing the update and noticed that mender’s me…

Can we try to eliminate that you are using lzma compressed Artifacts, as both have been default in mender-convert and it might depend on which version you are using. To check your this, can you please unpack the Artifact you are trying to deploy E.g $ tar xvf Ubuntu-Bionic-x86-64-qemux86_64-mend…

[image] neil: After lowering my memory usage even further to 20%, the system spiked to 75% and then threw the same out of memory error. I’m not sure if that’s pointing to a different issue or just a red herring. The situation does seem to somewhat environment specific, as you are not able to …

[image] mirzak: Can we try to eliminate that you are using lzma compressed Artifacts I checked this on my own, since you are using Mender Client 1.7 you are using gzip as it does not support lzma

Mender client running out of memory

General Discussions

TheYoctoJester Split this topic May 13, 2024, 8:58am 13

A post was merged into an existing topic: Docker Update Module without registry

Topic		Replies	Views
New mender server, new mender artifact, old mender client General Discussions yocto , kirkstone	3	390	September 26, 2023
High Mender client memory usage? General Discussions	1	721	June 11, 2019
System update bandwidth usage General Discussions	7	531	October 29, 2021
Malloc() error during inventory update General Discussions raspberry-pi-3 , mender-client , inventory , bug	10	63	March 19, 2026
Mender-update install artifact available storage considerations General Discussions	2	82	July 11, 2025

Mender client running out of memory

Related topics