we use docker-mender-convert (v5.1.0) on an Raspbian(Debian 12, Bookworm) image for a Raspberry Pi 3B which works without any problems.
Problem
During operation the system is executing the service mender-updated and fails during processing the inventory scripts with a C++ “malloc(): unsorted double linked list corrupted“-error:
Mar 16 12:15:03 connector systemd[1]: mender-data-dir.service - Mender persistent data dir was skipped because of an unmet condition check (ConditionPathExists=!/data/mender).
Mar 16 12:15:03 connector systemd[1]: Started mender-updated.service - Mender OTA update service.
Mar 16 12:15:03 connector systemd[1]: systemquery.service: Deactivated successfully.
Mar 16 12:15:03 connector systemd[1]: Finished systemquery.service - systemquery.
Mar 16 12:15:03 connector CRON[20999]: pam_unix(cron:session): session closed for user root
Mar 16 12:15:03 connector mender-update[21259]: record_id=1 severity=info time="2026-Mar-16 12:15:03.351876" name="Global" msg="Running Mender client 5.0.3"
Mar 16 12:15:03 connector mender-update[21259]: record_id=2 severity=info time="2026-Mar-16 12:15:03.352291" name="Global" msg="The update client daemon is now ready to handle incoming deployments"
Mar 16 12:15:03 connector mender-update[21259]: record_id=3 severity=info time="2026-Mar-16 12:15:03.361873" name="Global" msg="Running State Script: /etc/mender/scripts/Sync_Enter_01_connection-check"
Mar 16 12:15:04 connector mender-update[21259]: malloc(): unsorted double linked list corrupted
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Main process exited, code=killed, status=6/ABRT
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Failed with result 'signal'.
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Scheduled restart job, restart counter is at 7.
Mar 16 12:15:04 connector systemd[1]: Stopped mender-updated.service - Mender OTA update service.
We tested the Mender client 5.0.4 as well as 5.0.3 and the client is never able to connect to the server and the system stops the service for good after 5 restart tries.
We used Mender up until now with the same custom inventory scripts and client v3.5.0, but since updating to 5.0.x we get consistently the malloc()-error. Non of our key/value-pairs are excessively large or use protected keys and worked under client v3.5.0.
What we tested
We made sure that non of our custom script failed and restarted the service. (failed)
We cleared the mender-store and restarted the service. (failed)
We moved all inventory scripts (custom and Mender) manually to a backup folder and restarted the mender-updated.service. (success)
Further investigation
We copied the the scripts manually one by one back into the folder and restarted the mender-updated.service. We did this for every file and we tried this process multiple times in a different order. The error reappeared randomly and not after a specific script.
Once the crash happened after coping the last inventory script.
Once after coping only half of the scripts.
Once crashing after coping the last script, but upon commenting out random key/value-pairs the service was starting again and connecting to the server.
Sadly we can not recognize any pattern which would help with the investigation. If anyone could give advice on this issue we would highly appreciate it.
Thanks for reaching out! It’s the first time I hear such an error description and it sure sounds weird to debug.
The one thing which might be worth checking, is: did the problem appear some time after moving to the 4.x/5.x family (the major version bump was for something really unrelated here), or was it always present since moving beyond 3.x?
On the platform, it’s a somewhat vanilla RPi3B, I guess? Debian 12, 32bit or 64bit? Any other specialties which might be relevant?
Thx @TheYoctoJester for responding so quick, the problem happened as soon as we changed from 3.5.0 to 5.0.x.
Our platform is indeed a normal 3B+, Debian 12 32 bits.
We’re really stuck, we haven’t been able to find any additional information or logs about it… We’re considering downgrading to a lower version - would there be any version you’d recommend we could fall back on?
The main problem is that Raspberry Pi OS 32bit is the ONLY one which does not do REAL 32bit, a.k.a. armv7, but in fact is armv6 in disguise - a platform which has been effectively deprecated many years ago. Even their own kernel is by now 64bit under the 32bit distro (at least on some boards by now, see https://www.raspberrypi.com/documentation/computers/linux_kernel.html#natively-build-a-kernel, the red box), and the Pi 5 family is exclusively 64bits.
This has caused confusion and problems for other distributions and boards, so this increasingly niche use case has been deprecated now.
@TheYoctoJester : is there any workaround for us to upgrade to last mender version ? Or the only workaround is to stay on previous mender version ?
Thanks
@Jerry the last version built in the “old” way is 5.0.1. So for a quick fix, stick to that version.
The obvious possible workaround is recompiling the newer clients for armv6 manually, which should be quite straightforward.
To my understanding the main problem is that both Raspberry Pi OS 32bit (armv6) and Debian 32bit (armv7) use the armhf string identifier, which makes it mutually exclusive in a single package repo. On top of that, hosting and maintaining a distinct package repo just for RPi OS 32bit, which does not make too much sense as the platforms relevance is diminishing quickly.
Really sorry I do not have a better or more straightforwards answer here at the moment.
Great! If you run into problems, please let me know! I do regular build time testing of the client on RPi 32bit through Yocto, so I’m fairly certain that there should be no hard blockers, but of course there can be complicated bits related to Raspberry Pi OS and I’m happy to help with those as much as I can.
Going back to the client version 5.0.1 works for me. I loaded the old debian repository based on the orange box in the documentation and installed version 5.0.1.
I would say this is a good enough solution for legacy 32bit systems.
