Malloc() error during inventory update

General Discussions
, , ,
1

Hi everyone,

we use docker-mender-convert (v5.1.0) on an Raspbian(Debian 12, Bookworm) image for a Raspberry Pi 3B which works without any problems.

Problem

During operation the system is executing the service mender-updated and fails during processing the inventory scripts with a C++ “malloc(): unsorted double linked list corrupted“-error:

Mar 16 12:15:03 connector systemd[1]: mender-data-dir.service - Mender persistent data dir was skipped because of an unmet condition check (ConditionPathExists=!/data/mender).
Mar 16 12:15:03 connector systemd[1]: Started mender-updated.service - Mender OTA update service.
Mar 16 12:15:03 connector systemd[1]: systemquery.service: Deactivated successfully.
Mar 16 12:15:03 connector systemd[1]: Finished systemquery.service - systemquery.
Mar 16 12:15:03 connector CRON[20999]: pam_unix(cron:session): session closed for user root
Mar 16 12:15:03 connector mender-update[21259]: record_id=1 severity=info time="2026-Mar-16 12:15:03.351876" name="Global" msg="Running Mender client 5.0.3"
Mar 16 12:15:03 connector mender-update[21259]: record_id=2 severity=info time="2026-Mar-16 12:15:03.352291" name="Global" msg="The update client daemon is now ready to handle incoming deployments"
Mar 16 12:15:03 connector mender-update[21259]: record_id=3 severity=info time="2026-Mar-16 12:15:03.361873" name="Global" msg="Running State Script: /etc/mender/scripts/Sync_Enter_01_connection-check"
Mar 16 12:15:04 connector mender-update[21259]: malloc(): unsorted double linked list corrupted
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Main process exited, code=killed, status=6/ABRT
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Failed with result 'signal'.
Mar 16 12:15:04 connector systemd[1]: mender-updated.service: Scheduled restart job, restart counter is at 7.
Mar 16 12:15:04 connector systemd[1]: Stopped mender-updated.service - Mender OTA update service.

We tested the Mender client 5.0.4 as well as 5.0.3 and the client is never able to connect to the server and the system stops the service for good after 5 restart tries.
We used Mender up until now with the same custom inventory scripts and client v3.5.0, but since updating to 5.0.x we get consistently the malloc()-error. Non of our key/value-pairs are excessively large or use protected keys and worked under client v3.5.0.

What we tested

  1. We made sure that non of our custom script failed and restarted the service. (failed)
  2. We cleared the mender-store and restarted the service. (failed)
  3. We moved all inventory scripts (custom and Mender) manually to a backup folder and restarted the mender-updated.service. (success)

Further investigation

We copied the the scripts manually one by one back into the folder and restarted the mender-updated.service. We did this for every file and we tried this process multiple times in a different order. The error reappeared randomly and not after a specific script.

  • Once the crash happened after coping the last inventory script.
  • Once after coping only half of the scripts.
  • Once crashing after coping the last script, but upon commenting out random key/value-pairs the service was starting again and connecting to the server.

Sadly we can not recognize any pattern which would help with the investigation. If anyone could give advice on this issue we would highly appreciate it.

2

Hi @rsahlmann,

Thanks for reaching out! It’s the first time I hear such an error description and it sure sounds weird to debug. :frowning:
The one thing which might be worth checking, is: did the problem appear some time after moving to the 4.x/5.x family (the major version bump was for something really unrelated here), or was it always present since moving beyond 3.x?

On the platform, it’s a somewhat vanilla RPi3B, I guess? Debian 12, 32bit or 64bit? Any other specialties which might be relevant?

Greetz,
Josef

3

Thx @TheYoctoJester for responding so quick, the problem happened as soon as we changed from 3.5.0 to 5.0.x.
Our platform is indeed a normal 3B+, Debian 12 32 bits.
We’re really stuck, we haven’t been able to find any additional information or logs about it… We’re considering downgrading to a lower version - would there be any version you’d recommend we could fall back on?

4

We have the exact same problem with the exact same setup :

  • from 3.5.0 to 5.0.x
  • raspberry pi3

Our raspbian version is bookworm