Troubleshooting addon goes offline for hours

General Discussions
,
1

Hi

We observed that on our devices, mender-connect addon goes offline for long periods of time (could be few hours)

The effect is only troubleshooting is offline, and deployment and inventory updates still works

an example of log from an event this happend

Sep 03 17:43:36 raspberrypi mender-connect[574]: time=“2025-09-03T17:43:36+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:44654->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 6/10); len(token)=647”
Sep 03 17:43:38 raspberrypi mender-connect[574]: time=“2025-09-03T17:43:38+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:43:49 raspberrypi mender-connect[574]: time=“2025-09-03T17:43:49+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:00 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:00+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:11 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:11+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:22 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:22+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:26 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:26+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:54440->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 7/10); len(token)=647”
Sep 03 17:44:33 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:33+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:44 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:44+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:44:55 raspberrypi mender-connect[574]: time=“2025-09-03T17:44:55+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:45:06 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:06+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:45:16 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:16+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:35162->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 8/10); len(token)=647”
Sep 03 17:45:17 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:17+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:45:28 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:28+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:45:39 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:39+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:45:50 raspberrypi mender-connect[574]: time=“2025-09-03T17:45:50+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:01 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:01+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:06 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:06+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:51468->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 9/10); len(token)=647”
Sep 03 17:46:12 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:12+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:23 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:23+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:34 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:34+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:45 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:45+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=error msg=“eventLoop: error reconnecting: failed to connect after max number of retries”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=trace msg=“messageLoop: got response: {event:connected-error data: id:}”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=trace msg=“messageLoop: posting event: reconnect-req; waiting for response”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=trace msg=“needsReconnect: got event: reconnect-req”
Sep 03 17:46:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:46:56+03:00” level=trace msg=“dbusEventLoop: daemon needs to reconnect”
Sep 03 17:47:06 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:06+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:47:06 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:06+03:00” level=trace msg=“(dbusEventLoop) posting Event: reconnect”
Sep 03 17:47:06 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:06+03:00” level=trace msg=“eventLoop: got event: reconnect”
Sep 03 17:47:17 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:17+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:47:28 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:28+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:47:39 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:39+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:47:50 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:50+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:47:51 raspberrypi mender-connect[574]: time=“2025-09-03T17:47:51+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:57666->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 1/10); len(token)=647”
Sep 03 17:48:01 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:01+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:48:12 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:12+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:48:23 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:23+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:48:34 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:34+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:48:41 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:41+03:00” level=error msg=“connection manager failed to connect to http://127.0.0.1:32959/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:50636->127.0.0.1:32959: i/o timeout; reconnecting in 5s (try 2/10); len(token)=647”
Sep 03 17:48:45 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:45+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”
Sep 03 17:48:56 raspberrypi mender-connect[574]: time=“2025-09-03T17:48:56+03:00” level=trace msg=“dbusEventLoop: WaitForJwtTokenStateChange , err timeout waiting for signal JwtTokenStateChange”

we are using 2025-05-13-raspios-bookworm-arm64-lite.img on rpizero2w
with mender 4.0.7 and mender-connect 2.2.1 runtime: go1.21.1
on the hosted mender servers (US)

What can be done?

Edit by @TheYoctoJester: formatting of log

2

Hi @roee.douek,

Thanks for reaching out!

The easiest thing to try is bumping mender-connect to 2.3.1, as if I remember correctly there were some improvements concerning handling the connection.
Besides that, are there maybe any recognizable patterns? Like

  • always around a similar time or day,
  • or a specific action taking place?

Greetz,
Josef