Delayed detection of deployment by Mender client

General Discussions
1

@skrawn

Expected Behavior

The mender client running on device contacts the Mender server every 30 minutes to check for a software update deployment. This can also be done manually with mender check-update . This has been working well for the past 6 months.

Observed Behavior

Now when creating a deployment, devices can go into a “pending” state for several hours before before detecting a deployment. Running mender check-update has no effect.

From looking at mender-client logs on device, it is not seeing the deployment which was started at 20:18.
Oct 09 19:58:33 ec-221 mender[427]: time="2025-10-09T19:58:33Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" Oct 09 19:58:34 ec-221 mender[427]: time="2025-10-09T19:58:34Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]" Oct 09 20:28:33 ec-221 mender[427]: time="2025-10-09T20:28:33Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" Oct 09 20:28:34 ec-221 mender[427]: time="2025-10-09T20:28:34Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]" Oct 09 20:58:33 ec-221 mender[427]: time="2025-10-09T20:58:33Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" Oct 09 20:58:34 ec-221 mender[427]: time="2025-10-09T20:58:34Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]" Oct 09 21:28:33 ec-221 mender[427]: time="2025-10-09T21:28:33Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" Oct 09 21:28:34 ec-221 mender[427]: time="2025-10-09T21:28:34Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]" Oct 09 21:58:33 ec-221 mender[427]: time="2025-10-09T21:58:33Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]" Oct 09 21:58:34 ec-221 mender[427]: time="2025-10-09T21:58:34Z" level=info msg="State transition: update-check [Sync] -> check-wait [Idle]

2

Hi @skarstein

This is an interesting report.

This has been working well for the past 6 months

With this comment I should assume that you have not updated the client and then started seeing the issue, but that with no changes in the client side it started happening now?

Also, you don’t mention which kind of server you are using but can I assume you are using hosted Mender?

If both assumptions are true, it could be related to Server changes that we have deployed recently :thinking:

Can you enable debug logs on mender-update and then reproduce the issue?

To enable the logs the easiest is to edit the mender-updated systemd service with something like:

ExecStart=/usr/bin/mender-update --log-level debug daemon

Thanks

3

Thanks for the quick reply and apologies for missing out some keys details.

  • We are using hosted.mender.io
  • Devices run Yocto built mender connect (2.1.0) and client (3.5.0)

I should clarify that deployments will eventually finish. But what used to take less than an hour to complete can now take up to 12 hours due to device being in pending state.

I will be out for 2 week. I’ll enable the debug output and report findings when I get back.

One question - when do I find notification of server side releases?

Henry