Hi,
When I use the Hosted Mender server and performing OTA updates on an ESP32-S3 device. However, during deployment, the artifact download consistently stalls between 20% and 40%. The exact percentage varies each time (sometimes 10%, sometimes 30%, sometimes 40%), and the download never completes successfully.
- Device: ESP32-S3 (esp32s3_devkitc)
- OS: Zephyr
- Client: Mender MCU client
- Server: Hosted Mender (US region)
- Artifact size: ~760 KB
- Transport: HTTPS
And I have ever tried to tested (Switched to a different WiFi network, Used a mobile hotspot, Adjusted socket connect timeout, Modified network connection limits, Experimented with different mbedTLS buffer and heap sizes).
And, the device does not crash. It appears that the TLS/socket read operation blocks indefinitely during the download phase.
Below is the relevant part of the log :
[1970-01-01T00:00:00,177000Z] mender_app: Using net interface wifi, index=1
[1970-01-01T00:00:00,177000Z] mender_app: Connecting to wireless network GL-MT3000-ee0…
[1970-01-01T00:00:00,206000Z] mender_app: Waiting for network up…
[1970-01-01T00:00:05,329000Z] net_dhcpv4: Received: 192.168.8.156
[1970-01-01T00:00:05,330000Z] mender_app: Address[1]: 192.168.8.156
[1970-01-01T00:00:05,330000Z] mender_app: Subnet[1]: 255.255.255.0
[1970-01-01T00:00:05,330000Z] mender_app: Router[1]: 192.168.8.1
[1970-01-01T00:00:05,330000Z] mender_app: Lease time[1]: 43200 seconds
[1970-01-01T00:00:05,330000Z] mender_app: Initializing Mender Client with:
[1970-01-01T00:00:05,330000Z] mender_app: Device type: ‘esp32s3_devkitc’
[1970-01-01T00:00:05,330000Z] mender_app: Identity: ‘{“mac”: “e8:f6:0a:8d:af:fc”}’
[1970-01-01T00:00:05,330000Z] mender: Device type: [esp32s3_devkitc]
[1970-01-01T00:00:05,341000Z] fs_nvs: 2 Sectors of 4096 bytes
[1970-01-01T00:00:05,341000Z] fs_nvs: alloc wra: 0, fd8
[1970-01-01T00:00:05,341000Z] fs_nvs: data wra: 0, 120
[1970-01-01T00:00:05,342000Z] mender: mender_storage_init: Initialized Mender NVS at 0x7f1000 with 2 sectors (4096 bytes available)
[1970-01-01T00:00:05,342000Z] mender: mender_storage_init: Initialized deployment logs FCB at 0x7f3000 with 2 sectors (8192 bytes available)
[1970-01-01T00:00:05,342000Z] mender: mender_client_init: Added dormant certificate
[1970-01-01T00:00:05,342000Z] mender_app: Mender client initialized
[1970-01-01T00:00:05,342000Z] mender_app: Update Module ‘zephyr-image’ initialized
[1970-01-01T00:00:05,342000Z] mender_app: Mender inventory callback added
[1970-01-01T00:00:05,342000Z] mender: mender_os_scheduler_work_activate: Activating mender_client_main every 30 seconds
[1970-01-01T00:00:05,342000Z] mender: mender_os_scheduler_work_activate: Activating mender_inventory every 60 seconds
[1970-01-01T00:00:05,342000Z] mender_app: Mender client activated and running!
[1970-01-01T00:00:05,343000Z] mender: mender_os_scheduler_work_handler: Executing mender_client_main work
[1970-01-01T00:00:05,343000Z] mender: mender_client_work_function: Inside work function [state: MENDER_CLIENT_STATE_INITIALIZATION]
[1970-01-01T00:00:05,343000Z] mender: mender_tls_init_authentication_keys: Trying to read authentication keys from store
[1970-01-01T00:00:05,343000Z] mender: mender_storage_get_deployment_data: Deployment data not available
[1970-01-01T00:00:05,343000Z] mender: Initialization done
[1970-01-01T00:00:05,344000Z] mender: mender_client_update_work_function: Entering state MENDER_UPDATE_STATE_DOWNLOAD
[1970-01-01T00:00:05,344000Z] mender_app: mender_network_connect_cb: network_connect_cb
[1970-01-01T00:00:05,344000Z] mender: Checking for deployment…
[1970-01-01T00:00:05,344000Z] mender: mender_storage_get_provides: Provides not available
[1970-01-01T00:00:05,344000Z] mender_app: mender_get_identity_cb: get_identity_cb
[1970-01-01T00:00:07,681000Z] mender: ensure_authenticated_and_locked: Authenticated successfully
[1970-01-01T00:00:11,572000Z] mender: Downloading artifact with id ‘93bfaeb…’, name ‘release-343’, uri ‘https://c271964d41749feb10da762816c952ee.r2.cloudflarestorage.com/mender-artifacts-us/69845d2fa5cf26b77f4f7075/c2ba3487-4aac-491d-9937-0971d648ec8c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=161d877990f252c30c4602beb38f74a8%2F20260227%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260227T054909Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D"release-343.mender"&response-content-type=application%2Fvnd.mender-artifact&x-id=GetObject&X-Amz-Signature=2f78c5f315166a56d716b77b261af1bd8c82660dcc8357779e1fae45b6eeaceb’
[1970-01-01T00:00:14,931000Z] mender_app: mender_deployment_status_cb: deployment_status_cb: downloading
[1970-01-01T00:00:18,839000Z] mender: artifact_read_version: Artifact has valid version
[1970-01-01T00:00:19,044000Z] mender: artifact_read_manifest: 384a8f8b3363eb2477a0d4221711eb7bf97caab329edad7f822dc11f6e3d4606 data/0000/zephyr.signed.bin
[1970-01-01T00:00:19,044000Z] mender: artifact_read_manifest: 2a5cc250ced94ac8e6de5b548d3c6ef927de7b77441e9cc7b97e944fe2ca9b8b header.tar
[1970-01-01T00:00:19,044000Z] mender: artifact_read_manifest: 96bcd965947569404798bcbdb614f103db5a004eb6e364cfc162c146890ea35b version
[1970-01-01T00:00:19,044000Z] mender: is_checksum_valid: Checking integrity for artifact file ‘version’
[1970-01-01T00:00:19,863000Z] mender: is_checksum_valid: Checking integrity for artifact file ‘header.tar’
[1970-01-01T00:00:19,878000Z] mender: Start flashing artifact ‘zephyr.signed.bin’ with size 751484
[1970-01-01T00:00:36,883000Z] mender: Downloading ‘zephyr-image’ 10%… [75264/751484]
[1970-01-01T00:00:52,842000Z] mender: Downloading ‘zephyr-image’ 20%… [150528/751484]
[1970-01-01T00:01:08,714000Z] mender: Downloading ‘zephyr-image’ 30%… [225792/751484]
My prj.conf is as below:
CONFIG_MENDER_SERVER_TENANT_TOKEN=“…”
CONFIG_MENDER_MCU_CLIENT=y
CONFIG_MENDER_LOG_LEVEL_INF=y
CONFIG_MENDER_CLIENT_UPDATE_POLL_INTERVAL=30
CONFIG_MENDER_CLIENT_INVENTORY_REFRESH_INTERVAL=60
CONFIG_MENDER_RETRY_ERROR_BACKOFF=5
CONFIG_MENDER_RETRY_ERROR_MAX_BACKOFF=15
CONFIG_MENDER_SERVER_HOST_US=y
CONFIG_MENDER_STORAGE_PARTITION_STORAGE_PARTITION=y
CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_PSK_ENABLED=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_RSA_ENABLED=y
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED=y
CONFIG_MBEDTLS_ECDH_C=y
CONFIG_MBEDTLS_ECDSA_C=y
CONFIG_MBEDTLS_ECP_C=y
CONFIG_MBEDTLS_ECP_DP_SECP256R1_ENABLED=y
CONFIG_MBEDTLS_ECP_DP_SECP384R1_ENABLED=y
CONFIG_MBEDTLS_ECP_NIST_OPTIM=y
CONFIG_MBEDTLS_CIPHER_CCM_ENABLED=y
CONFIG_MBEDTLS_CIPHER_GCM_ENABLED=y
CONFIG_MBEDTLS_SHA384=y
CONFIG_MBEDTLS_GENPRIME_ENABLED=y
CONFIG_MBEDTLS_PEM_CERTIFICATE_FORMAT=y
CONFIG_MBEDTLS_SERVER_NAME_INDICATION=y
CONFIG_MBEDTLS_PK_WRITE_C=y
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=16384
CONFIG_MBEDTLS_ENABLE_HEAP=y
CONFIG_MBEDTLS_HEAP_SIZE=40960
CONFIG_MBEDTLS_ENTROPY_POLL_ZEPHYR=y
CONFIG_MBEDTLS_ENTROPY_C=y
CONFIG_MBEDTLS_USER_CONFIG_ENABLE=y
CONFIG_MBEDTLS_USER_CONFIG_FILE=“config-tls-mender.h”
CONFIG_MAIN_STACK_SIZE=2048
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=4096
CONFIG_ZVFS_OPEN_MAX=5
CONFIG_LOG=y
CONFIG_LOG_BUFFER_SIZE=2048
CONFIG_LOG_MODE_DEFERRED=y
CONFIG_LOG_MODE_OVERFLOW=y
CONFIG_LOG_SPEED=y
CONFIG_POSIX_C_LANG_SUPPORT_R=y
CONFIG_LOG_OUTPUT_FORMAT_ISO8601_TIMESTAMP=y
CONFIG_NET_IPV6=n
CONFIG_NET_IPV4=y
CONFIG_NET_DHCPV4=y
CONFIG_NET_MGMT=y
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_MGMT_EVENT_STACK_SIZE=2048
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_NET_SOCKETS_TLS_MAX_CONTEXTS=2
CONFIG_NET_MAX_CONN=16
CONFIG_DNS_RESOLVER_ADDITIONAL_BUF_CTR=5
CONFIG_DNS_RESOLVER_ADDITIONAL_QUERIES=2
CONFIG_DNS_RESOLVER_MAX_SERVERS=2
CONFIG_DNS_NUM_CONCUR_QUERIES=5
CONFIG_NET_SOCKETS_CONNECT_TIMEOUT=5000
CONFIG_NET_LOG=y
CONFIG_COMPILER_WARNINGS_AS_ERRORS=y
CONFIG_SPIN_VALIDATE=n
Questions
-
Is this behavior known when using Hosted Mender + ESP32/Zephyr?
-
Are there recommended TCP/TLS configuration settings for downloading artifacts of this size (~760 KB) on resource-constrained devices?
-
Are there specific debug logs (TCP state, TLS debug level, net_buf stats, etc.) that you would like me to capture?
Thank you for your assistance.
Best regards,