# Mender Troubleshoot Add-on reliability

**URL:** <https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760>\
**Category:** General Discussions\
**Created:** [April 25, 2024, 1:21pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760 "2024-04-25T13:21:49Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [April 25, 2024, 1:21pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/1 "2024-04-25T13:21:49Z")

</div>

Hi,

we sometimes can’t access our devices via the troubleshoot remote terminal. 95% of the time it’s working fine, but if troubleshoot stops working, it does not work until restarting `mender-connect` or rebooting the device.

We have a bunch of devices set up with a custom yocto-based OS. We use EU hosted mender and have licensed the troubleshoot add-on. The devices continue to update the inventory and successfully check for and install pending deployments. Restarting the device or restarting the `mender-connect` systemd service over a separate SSH session immediatly fixes the issue. This problem comes up sporadically and we don’t have a way to reproduce it.

Are there any known issues with `mender-connect` and the troubleshooting add-on, for example when working with less reliable internet connections, wifi, …? Could problems on the server-side cause this? Does anyone have pointers what we could try? Currently my only idea is to periodically restart `mender-connect`, which would not be an ideal solution.

Best regards,  
Nils

---

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [April 30, 2024, 6:24am UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/2 "2024-04-30T06:24:26Z")

</div>

Does anyone have an idea? What additional information can I provide to aid in resolving this?

Best regards  
Nils

---

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [May 23, 2024, 4:40pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/3 "2024-05-23T16:40:47Z")

</div>

I since contacted support via e-mail and got a quick reply. By mistake I thought we already updated to current mender-connect 2.2.1, but we were still on 2.1.0, which apparently had known issues.

Side note: The support rep also suggested to migrate to mender client 4, if we did not rollout to production yet.

I’ll close this for now and re-open if we find more problems.

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [May 23, 2024, 7:21pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/4 "2024-05-23T19:21:12Z")

</div>

Hi @NiBr, thanks for the recap!

---

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [May 29, 2024, 6:42am UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/5 "2024-05-29T06:42:27Z")

</div>

Hi again,

sadly, updating mender and mender-connect did not fix the issue. We still face the same problem. The device is connected (and for our development devices also reachable via SSH), inventory updates work but troubleshoot does not. The devices sometimes are simply losing connection to troubleshoot, so neither the web-gui nor `mender-cli terminal` works.  
Restarting the `mender-connect` service or rebooting the device immediatly solves the problem.

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/d/d815b589d3de3dec7b88d7b44bf382fae6ab2150.png)

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [June 12, 2024, 1:54pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/6 "2024-06-12T13:54:51Z")

</div>

Hello Nils,

Thanks for following up. Let me answer in here as well:

There is a config based workaround that it is to set the following config in your `mender-connect.conf` :

```auto
"Sessions": {    
    "StopExpired": true,    
    "ExpireAfterIdle": 600  
}

```

This basically ensures that in the worst case scenario, the mender-connect auto heal will happen after 10 minutes (600 seconds).

Can you give it a try?

BR,  
Luis

---

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [June 12, 2024, 4:12pm UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/7 "2024-06-12T16:12:34Z")

</div>

Hi Luis,

thanks for the reply. From my side we can continue here in the forum instead of the private mail chat, so everyone can profit from our findings. For your info: We are using current versions of `mender` (3.5.2) and `mender-connect` (2.2.1) now.

I’m glad to test `ExpireAfterIdle`. Sometimes we have to use Mender troubleshoot to help customers in certain situations, e.g. when they call via phone. Does it hurt to set the `ExpireAfterIdle` time to let’s say 300 or 60 seconds instead of 600?  
Waiting 10 minutes during a call just because the connection got lost could get annoying quickly.

---

<div class="post-metadata">

**Author:** ![martin.zima](https://avatars.discourse-cdn.com/v4/letter/m/7ba0ec/32.png) [@martin.zima](https://hub.mender.io/u/martin.zima)\
**Post date:** [June 13, 2024, 10:49am UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/8 "2024-06-13T10:49:33Z")

</div>

FWIW, this happens to us quite often (had observed this for a few months now) to the point mender-connect is not something we could rely upon. Today this happened already a few times and I noticed that it we lost the connectivity to all our devices at around the same time (± 1 minute) and then when it started working, all devices came up online simultaneously again. At the same time, I was intermittently getting errors like “accepted devices couldn’t be loaded. timeout of 10000ms exceeded” in the Mender console, which got me thinking if it actually could be a server-related issue.

We are on the trial plan of the hosted Mender (EU) and using mender 3.4.0 and mender-connect 2.1.0 (will try upgrading today). Here is an excerpt of mender-connect logs that I’m seeing on of these devices:

```auto
Jun 13 07:30:32 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:30:32Z" level=error msg="eventLoop: error reconnecting: failed to connect after max number of retries"
Jun 13 07:30:57 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:30:57Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 1/10); len(token)=654"
Jun 13 07:31:22 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:31:22Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 2/10); len(token)=654"
Jun 13 07:31:42 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:31:42Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 3/10); len(token)=654"
Jun 13 07:31:58 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:31:58Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 07:34:44 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:34:44Z" level=error msg="messageLoop: error on readMessage: websocket: close 1000 (normal): read tcp 10.11.8.21:60980->20.61.30.255:443: read: connection timed out; disconnecting, waiting for reconnect."
Jun 13 07:35:14 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:35:14Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 1/10); len(token)=654"
Jun 13 07:35:39 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:35:39Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 2/10); len(token)=654"
Jun 13 07:36:04 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:36:04Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 3/10); len(token)=654"
Jun 13 07:36:29 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:36:29Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 4/10); len(token)=654"
Jun 13 07:36:54 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:36:54Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 5/10); len(token)=654"
Jun 13 07:37:19 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:37:19Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 6/10); len(token)=654"
Jun 13 07:37:44 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:37:44Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 7/10); len(token)=654"
Jun 13 07:38:09 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:38:09Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 8/10); len(token)=654"
Jun 13 07:38:34 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:38:34Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 9/10); len(token)=654"
Jun 13 07:38:45 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:38:45Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 07:58:44 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:58:44Z" level=error msg="messageLoop: error on readMessage: websocket: bad close code 1006; disconnecting, waiting for reconnect."
Jun 13 07:59:25 iot-gate-imx8 mender-connect[523]: time="2024-06-13T07:59:25Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 08:34:46 iot-gate-imx8 mender-connect[523]: time="2024-06-13T08:34:46Z" level=error msg="messageLoop: error on readMessage: websocket: close 1000 (normal): read tcp 10.11.8.21:60988->20.61.30.255:443: read: connection reset by peer; disconnecting, waiting for reconnect."
Jun 13 08:35:01 iot-gate-imx8 mender-connect[523]: time="2024-06-13T08:35:01Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 09:39:16 iot-gate-imx8 mender-connect[523]: time="2024-06-13T09:39:16Z" level=error msg="messageLoop: error on readMessage: websocket: close 1000 (normal): read tcp 10.11.8.21:60990->20.61.30.255:443: read: connection reset by peer; disconnecting, waiting for reconnect."
Jun 13 09:39:32 iot-gate-imx8 mender-connect[523]: time="2024-06-13T09:39:32Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 10:26:31 iot-gate-imx8 mender-connect[523]: time="2024-06-13T10:26:31Z" level=error msg="messageLoop: error on readMessage: websocket: close 1011 (internal server error): read tcp 172.26.65.255:8080->172.26.65.60:33946: i/o timeout; disconnecting, waiting for reconnect."
Jun 13 10:27:21 iot-gate-imx8 mender-connect[523]: time="2024-06-13T10:27:21Z" level=info msg="eventLoop: Connection established with http://127.0.0.1:39823"
Jun 13 10:29:32 iot-gate-imx8 mender-connect[523]: time="2024-06-13T10:29:32Z" level=error msg="messageLoop: error on readMessage: websocket: close 1011 (internal server error): read tcp 172.26.67.23:8080->172.26.65.60:49428: i/o timeout; disconnecting, waiting for reconnect."
Jun 13 10:30:29 iot-gate-imx8 mender-connect[523]: time="2024-06-13T10:30:29Z" level=error msg="connection manager failed to connect to http://127.0.0.1:39823/api/devices/v1/deviceconnect/connect: read tcp 127.0.0.1:35640->127.0.0.1:39823: i/o timeout; reconnecting in 5s (try 1/10); len(token)=654"

```

EDIT: @NiBr Ah, I guess that’s explains some of the connection problems. Still, this problem has been affecting us for a much longer time. I will try upgrading mender-connect and see if that helps.

---

<div class="post-metadata">

**Author:** ![NiBr](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@NiBr](https://hub.mender.io/u/NiBr)\
**Post date:** [June 13, 2024, 11:23am UTC](https://hub.mender.io/t/mender-troubleshoot-add-on-reliability/6760/9 "2024-06-13T11:23:24Z")

</div>

Hi @martin.zima,

EU hosted mender seems to have experienced an outage today: [https://mender.statuspage.io/](https://mender.statuspage.io/).  
Maybe the problems you’ve experienced are caused by that. The problem I was discussing does not affect all devices simultaneously.