Mender MCU client crash on ESP32-S3 during identity callback

Board Integrations
,
1

ESP32-S3 + Zephyr

Device description

Board: ESP32-S3 DevKitC
OS: Zephyr v4.2.0
Quick description:
A development board with dual-core Xtensa LX7, used here with Zephyr OS and integrated Mender MCU client for OTA updates.

Support level

  • Board supported: ESP32-S3 DevKitC

  • Zephyr OS integration tested by the user for basic build and OTA functionality.

  • Note: Crash occurs during Mender MCU client initialization when using identity callback.

Getting started

  • Zephyr project: mender-mcu-integration

  • Build steps (successful firmware build with Mender MCU integration):

    west build -b esp32s3_devkitc path/to/mender-mcu-integration
west espressif flash -p /dev/ttyUSB0
west espressif monitor -p /dev/ttyUSB0 -b 115200

  • MCUboot is used as the bootloader.

  • TLS certificates are integrated via certs.c and .cer.inc include files.

  • Mender Server: self-hosted Open Source server deployed on Ubuntu (following official Mender tutorial).

  • Mender Client Kconfig settings for self-hosted server:

    CONFIG_MENDER_SERVER_HOST="http://<server_ip>"
CONFIG_MENDER_SERVER_TENANT_TOKEN=""   # empty for on-prem server
CONFIG_MENDER_NET_CA_CERTIFICATE_TAG_PRIMARY=1
CONFIG_MENDER_CLIENT_UPDATE_POLL_INTERVAL=30
CONFIG_MENDER_CLIENT_INVENTORY_REFRESH_INTERVAL=60

References

Known issues

Problem description:
During initialization of Mender MCU client on ESP32-S3 with Zephyr, the device crashes after network connection, TLS setup, and when attempting to call the identity callback.

Crash log snippet:

[1970-01-01T00:00:04,673000Z] <dbg> mender_app: mender_network_connect_cb: network_connect_cb
[1970-01-01T00:00:04,673000Z] <inf> mender: Checking for deployment...
[1970-01-01T00:00:04,673000Z] <dbg> mender: mender_storage_get_provides: Provides not available
[1970-01-01T00:00:04,673000Z] <dbg> mender_app: mender_get_identity_cb: get_identity_cb
ASSERTION FAIL [0] @ WEST_TOPDIR/zephyr/lib/libc/picolibc/assert.c:27
[1970-01-01T00:00:04,746000Z] <err> os:  ** FATAL EXCEPTION
[1970-01-01T00:00:04,746000Z] <err> os:  ** CPU 0 EXCCAUSE 63 (zephyr exception)
[1970-01-01T00:00:04,746000Z] <err> os:  **  PC 0x40378587 VADDR 0
[1970-01-01T00:00:04,746000Z] <err> os:  **  PS 0x60e20
[1970-01-01T00:00:04,746000Z] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:14 CALLINC:2)

Identity callback used:

mender_err_t mender_get_identity_cb(const mender_identity_t **identity_ptr) {
    if (identity_ptr) {
        *identity_ptr = &identity;
        return MENDER_OK;
    }
    return MENDER_FAIL;
}

Steps attempted to resolve:

  1. Verified mender_get_identity_cb returns MENDER_OK.

  2. Integrated self-signed TLS certificates correctly.

  3. Ensured network (Wi-Fi) is up before client initialization.

  4. Tried different static identity formats.

  5. Checked memory usage; NVS storage initialized correctly.

  6. Confirmed server is reachable and accessible from other devices.

Observations:

  • Crash occurs immediately after mender_get_identity_cb is called.

  • The device successfully connects to Wi-Fi and gets an IP address.

  • TLS setup is completed and primary certificate is loaded.

  • MCUboot successfully loads the Zephyr application; “Hello World” message prints before crash.

Request for guidance:

  • Could this crash be due to environment initialization order (NVS, TLS, time)?

  • Are there known issues with ESP32-S3 + Zephyr + Mender MCU client regarding identity callback?

  • Any recommended workaround or example for safe initialization sequence on ESP32-S3 with self-hosted Mender server?

2

Hello @alian and thank you for your report,

This is something I would like to look deeper into. Certainly and exception like that one is something we need to look into!

Please clarify some points:

  • Mender Client Kconfig settings for self-hosted server:
CONFIG_MENDER_SERVER_HOST="http://<server_ip>"
CONFIG_MENDER_SERVER_TENANT_TOKEN=""   # empty for on-prem server
CONFIG_MENDER_NET_CA_CERTIFICATE_TAG_PRIMARY=1
CONFIG_MENDER_CLIENT_UPDATE_POLL_INTERVAL=30
CONFIG_MENDER_CLIENT_INVENTORY_REFRESH_INTERVAL=60

Setting SERVER_HOST to http://<server_ip> cannot work for two reasons: 1) Mender MCU won’t connect to a non-https server and 2) It must be a domain and not an ip address. If you are running on a private network you will need to set a DNS to resolve the ip address.

Can you clarify more your on-prem setup?

References

These don’t exist, is this a AI hallucination? Which guide(s) did you use?

References
TLS / certificates guidance: Mender MCU TLS documentation

Are you referring to this one?

Identity callback used:

mender_err_t mender_get_identity_cb(const mender_identity_t **identity_ptr) {
   if (identity_ptr) {
       *identity_ptr = &identity;
       return MENDER_OK;
   }
   return MENDER_FAIL;
}

Can you please share where is identity defined? I assume you used our demo code, which defines it static like static mender_identity_t mender_identity but I would like to double-check.

Lluís

3

One more thing,

In Zephyr v4.2.0, this exact line is a comment :thinking:

Lluís

4

Hi, lluiscampos

Thank you for looking into this issue.

Here is some clarification regarding my setup:

Zephyr version
Zephyr v4.2.0 running on ESP32-S3 DevKitC with MCUboot.

Mender MCU code
I am using the official integration example from:

On-prem Mender Server
My server is self-hosted using the Docker Compose setup from:

I followed the deployment instructions in that repository to bring up the full server stack locally on my Ubuntu.

I believe this deployment is running in single-tenant mode, since in the server UI I cannot find an “Organization Token”. So during the esp32s3 build I configured:

CONFIG_MENDER_SERVER_TENANT_TOKEN=""

Please let me know if this understanding is correct, or if I may have missed something in the server configuration.

Server host configuration

CONFIG_MENDER_SERVER_HOST="http://<server_ip>"
CONFIG_MENDER_NET_CA_CERTIFICATE_TAG_PRIMARY=1

Based on your feedback, I understand that Mender MCU requires HTTPS and a domain name instead of an IP address. My current setup uses an HTTP endpoint with a private network IP, so this is likely incorrect and I will update it accordingly.

For TLS, I am currently using the certificate located at:

mender-server/compose/certs/mender.crt

This certificate has been integrated into the Mender MCU firmware.

Crash context

The device successfully connects to WiFi and obtains an IP address.
TLS initialization also completes before the crash occurs.

The crash happens immediately after mender_get_identity_cb is called during the Mender client initialization.

Identity callback

Here is how the identity is defined in my application:

static char mac_address[18] = {0};
static mender-identity_t mender_identity = { .name = "mac", .value = mac_address };
mender_err_t mender_get_identity_cb(const mender_identity **identity) {
	if (NULL != identity) {
		*identity = &mender_identity;
		return MENDER_OK;
	}
	return MENDER_FAIL;
}

Please let me know if anything looks incorrect.

If there is a recommended setup for running Mender MCU against an on-prem server (especially with ESP32-S3 + Zephyr), I would greatly appreciate your guidance.

Thank you again for your support!