# Mender client running out of memory **URL:** \ **Category:** General Discussions\ **Created:** [March 30, 2020, 10:00pm UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715 "2020-03-30T22:00:45Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [March 30, 2020, 10:00pm UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/1 "2020-03-30T22:00:45Z")
My updates are stalling on a handful Raspberry Pi 3b+ devices because mender is running out of memory. The system’s memory utilization normally hovers around 40%, but during the update, I see it spike to 100% and then the client crashed with an “out of memory” error. Most devices update successfully, and only spike to around 80% utilization during the update. However the devices that are failing fail repeatedly on multiple tries, even after a reboot. I wonder if the high memory usage is indicative of another issue? Any tips for things I can do to bring down mender’s memory usage and successfully complete the update? ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [March 31, 2020, 6:42am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/2 "2020-03-31T06:42:32Z")
Hi, Are you using `gzip` or `lzma` compressed Artifacts? I suspect that this might have some impact on memory utilization while the Artifacts are decompressed. `gzip` would probably use less memory, while `lzma` would be heavier versus much better compression then `gzip`. But saying that, I would not have expected to see the Mender client utilize 60 % of your available memory regardless of which compression method is used. Could you share `cat /proc/$(pidof mender)/status | grep Vm` while an update is in progress? ---
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [March 31, 2020, 7:52am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/3 "2020-03-31T07:52:12Z")
Thanks for taking a look, Mirza! I’m using mender-convert, which I think defaults to gzip compression IIRC? I just watched an update occur, and mender was definitely consuming 60% of memory according to `top` (see screenshot). Here’s the output for `cat /proc/$(pidof mender)/status | grep Vm`: ``` VmPeak: 926892 kB VmSize: 926892 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 567872 kB VmRSS: 567872 kB VmData: 681644 kB VmStk: 132 kB VmExe: 5228 kB VmLib: 1440 kB VmPTE: 600 kB VmPMD: 0 kB VmSwap: 0 kB ``` And here’s the log when it crashes: [https://gist.github.com/neilgupta/683deb58bfd0ce97006d686ac6f37dd2](https://gist.github.com/neilgupta/683deb58bfd0ce97006d686ac6f37dd2) As a side note: since mender.service only restarts on abort, mender does not recover from this crash so the server gets stuck thinking the update is still downloading and has no way to recover. The device is still running mender client 1.7, while the server is 2.3. ![Screen Shot 2020-03-31 at 2.40.38 AM](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/fc80e5f48f2c9c1983a9b51a9a4f627bedb7fe20.png) ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [March 31, 2020, 7:56am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/4 "2020-03-31T07:56:58Z")
Thanks for sharing the information. Do you have the possibility to test this with the Mender client (2.2.0)? I know that there have been memory leaks in older versions caused by the golang standard library. > As a side note: since mender.service only restarts on abort, mender does not recover from this crash so the server gets stuck thinking the update is still downloading and has no way to recover. This looks like something that maybe should be fixed. @kacf what do you think? ---
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [March 31, 2020, 8:03am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/5 "2020-03-31T08:03:12Z")
I have ssh access to some of the devices, so I can try updating mender client manually. Should I just follow the install from source steps at [https://github.com/mendersoftware/mender](https://github.com/mendersoftware/mender)? Will that cause any issues if the image was created with an older mender-convert? ---
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [March 31, 2020, 8:06am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/6 "2020-03-31T08:06:22Z")
Oh nevermind, just found the debian packages at [https://docs.mender.io/2.3/downloads](https://docs.mender.io/2.3/downloads) Second question still stands of whether there would be any conflict with an older integration? ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [March 31, 2020, 8:08am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/7 "2020-03-31T08:08:19Z")
> [@neil](#): > > Second question still stands of whether there would be any conflict with an older integration? Hard to say, if you just use the binary from the deb package that would minimize any potential conflicts. But I do not know if I can recommend to this on a device that is in the field, and best if you can test this in a controlled environment. ---
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [April 1, 2020, 6:45pm UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/8 "2020-04-01T18:45:10Z")
I’m unable to recreate the issue on a device locally. It’s only happening on 5 of the 20 devices this latest update has been pushed to so far, so I need to find a way to fix it remotely. I tried killing all user processes to free up more RAM before performing the update and noticed that mender’s memory usage appears to be fixed at around 60% somehow? Out of memory made sense when my starting utilization was 40%, but after a reboot, my system’s memory utilization spiked from 30% to 85%. After lowering my memory usage even further to 20%, the system spiked to 75% and then threw the same out of memory error. I’m not sure if that’s pointing to a different issue or just a red herring. ![Screen Shot 2020-04-01 at 1.36.18 PM](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/b66eb56a2753e7da1dff4c9709b18f08d34f2fd8.png) Devices that succeeded saw the same memory spike, but did not crash. Any ideas on how I can get the update to succeed on a remote device? ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 2, 2020, 8:04am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/9 "2020-04-02T08:04:36Z")
Can we try to eliminate that you are using `lzma` compressed Artifacts, as both have been default in mender-convert and it might depend on which version you are using. To check your this, can you please unpack the Artifact you are trying to deploy E.g ```auto $ tar xvf Ubuntu-Bionic-x86-64-qemux86_64-mender.mender version manifest header.tar.gz data/0000.tar.gz ``` The file ending of `data/0000.tar` hints which compression algorithm was used. In my example it is `gzip`. ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 2, 2020, 8:18am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/10 "2020-04-02T08:18:54Z")
> [@neil](#): > > After lowering my memory usage even further to 20%, the system spiked to 75% and then threw the same out of memory error. I’m not sure if that’s pointing to a different issue or just a red herring. The situation does seem to somewhat environment specific, as you are not able to re-produce this locally and we have not seen similar reports earlier. So I do not think that is strictly only the Mender client in play here. How much actual memory do you have on the devices, so far we have been speaking in % but can you also check with `free -h`, what is the total available memory on remote and local devices. ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 2, 2020, 10:01am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/11 "2020-04-02T10:01:45Z")
> [@mirzak](#): > > Can we try to eliminate that you are using `lzma` compressed Artifacts I checked this on my own, since you are using Mender Client 1.7 you are using gzip as it does not support `lzma` ---
**Author:** ![neil](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/neil/32/204_2.png) [@neil](https://hub.mender.io/u/neil)\ **Post date:** [April 3, 2020, 5:51am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/12 "2020-04-03T05:51:03Z")
It’s a raspberry pi 3B+, so 1GB total ram. `free -h` during normal usage (no mender update): ``` total used free shared buff/cache available Mem: 976M 242M 369M 49M 364M 633M Swap: 99M 0B 99M ``` memory usage than climbs roughly 1mb/second as it downloads the update until it hits its peak: ``` total used free shared buff/cache available Mem: 976M 788M 24M 49M 163M 87M Swap: 99M 0B 99M ``` and then it immediately returns to ``` total used free shared buff/cache available Mem: 976M 237M 645M 49M 94M 643M Swap: 99M 0B 99M ``` Also confirmed that the mender artifact is gzip compressed. ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [May 13, 2024, 8:58am UTC](https://hub.mender.io/t/mender-client-running-out-of-memory/1715/13 "2024-05-13T08:58:03Z")
A post was merged into an existing topic: [Docker Update Module without registry](https://hub.mender.io/t/docker-update-module-without-registry/6816/13)