# Problem with mender connect remote terminal
**URL:** https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141
**Category:** General Discussions
**Created:** [January 27, 2021, 5:32pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141 "2021-01-27T17:32:47Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@jakew009](https://hub.mender.io/u/jakew009)
#### Post date: [January 27, 2021, 5:32pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/1 "2021-01-27T17:32:47Z")
When the device first came online, clock was wrong and it couldn’t talk to Mender. All fine.
After a while it connected, but then it got this error and couldn’t recover.
```auto
Jan 27 17:18:31 ifu-273 mender-connect[525]: time="2021-01-27T17:18:31Z" level=error msg="messageLoop: error on readMessage: read tcp 192.168.1.170:43264->3.219.180.200:443: read: connection reset by peer; disconnecting, waiting for reconnect."
```
It then got stuck here for 10 minutes until I got fedup and restarted the service at which point it worked immediately.
```auto
Sep 20 10:44:30 ifu-273 mender-connect[525]: time="2020-09-20T10:44:30Z" level=error msg="connection manager failed to connect to https://hosted.mender.io/api/devices/v1/deviceconnect/connect: x509: certificate has expired or is not yet valid: current time 2020-09-20T10:44:30Z is before 2021-01-21T08:53:09Z; reconnecting in 5s (try 5/0); len(token)=884"
Jan 27 16:50:59 ifu-273 mender-connect[525]: time="2021-01-27T16:50:59Z" level=warning msg="The server certificate cannot be loaded: no file provided"
Jan 27 17:18:31 ifu-273 mender-connect[525]: time="2021-01-27T17:18:31Z" level=error msg="messageLoop: error on readMessage: read tcp 192.168.1.170:43264->3.219.180.200:443: read: connection reset by peer; disconnecting, waiting for reconnect."
Jan 27 17:18:41 ifu-273 mender-connect[525]: time="2021-01-27T17:18:41Z" level=warning msg="The server certificate cannot be loaded: no file provided"
Jan 27 17:27:53 ifu-273 systemd[1]: Stopping Mender Connect service...
Jan 27 17:27:53 ifu-273 systemd[1]: mender-connect.service: Succeeded.
Jan 27 17:27:53 ifu-273 systemd[1]: Stopped Mender Connect service.
Jan 27 17:27:53 ifu-273 systemd[1]: Started Mender Connect service.
Jan 27 17:27:53 ifu-273 mender-connect[899]: time="2021-01-27T17:27:53Z" level=info msg="Loaded configuration file: /etc/mender/mender-connect.conf"
Jan 27 17:27:53 ifu-273 mender-connect[899]: time="2021-01-27T17:27:53Z" level=warning msg="ShellCommand is empty, defaulting to /bin/sh"
Jan 27 17:27:54 ifu-273 mender-connect[899]: time="2021-01-27T17:27:54Z" level=warning msg="The server certificate cannot be loaded: no file provided"
```
Any ideas?
We have tried rttys and shellhub in the past to do the same thing (remote ssh) and they have both been plagued by this sort of thing (not reliable when you most need them).
---
### Author:  [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [January 29, 2021, 8:06pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/2 "2021-01-29T20:06:50Z")
@lluiscampos any thoughts on this one?
---
### Author:  [@oleorhagen](https://hub.mender.io/u/oleorhagen)
#### Post date: [February 1, 2021, 2:02pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/3 "2021-02-01T14:02:11Z")
I will defer this to @merlin
---
### Author:  [@lluiscampos](https://hub.mender.io/u/lluiscampos)
#### Post date: [February 4, 2021, 4:32pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/4 "2021-02-04T16:32:47Z")
@jakew009 what is really the issue?
The printed warnings, while confusing, could be completely fine depending on your setup. Which image are you using (yocto based, mender-convert)? Which server are you connecting to?
---
### Author:  [@jakew009](https://hub.mender.io/u/jakew009)
#### Post date: [February 4, 2021, 5:13pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/5 "2021-02-04T17:13:04Z")
Hello
We are using Yocto (Dunfell) and hosted mender.
The issue is that the device was “offline” (remote terminal not available) in Mender. Restarting Mender brings it back online.
This line seems to get echoed out when it goes offline:
```auto
Jan 27 17:18:31 ifu-273 mender-connect[525]: time="2021-01-27T17:18:31Z" level=error msg="messageLoop: error on readMessage: read tcp 192.168.1.170:43264->3.219.180.200:443: read: connection reset by peer; disconnecting, waiting for reconnect."
```
---
### Author:  [@lluiscampos](https://hub.mender.io/u/lluiscampos)
#### Post date: [February 4, 2021, 5:34pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/6 "2021-02-04T17:34:08Z")
@jakew009 Right, I see. I got confused with your original message.
I thought it was a one-time error and that then you were worried about the “certificate cannot be loaded” warning and jumped in the conversation.
The error itself seems to be lost connection between mender-connect and the backend. @merlin should be the right one to ping then 🙂
---
### Author:  [@jakew009](https://hub.mender.io/u/jakew009)
#### Post date: [February 4, 2021, 6:09pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/7 "2021-02-04T18:09:38Z")
@lluiscampos yep that’s correct. The connection between mender-connect and the mender server is lost, and it does not recover automatically. But if you restart the mender-connect service, it reconnects instantly.
This is the major problem we’ve had with every other solution we’ve tried before that implement similar functionalities. The only thing we’ve had almost 100% success with is a VPN daemon, as these seem to be have been designed from the get-go to be persistent and naturally they have built in watchdogs / ack packets.
Maybe it is possible to add some sort of watchdog to mender-connect, so it can figure out when it has lost it’s connection and reset itself?
---
### Author:  [@peter](https://hub.mender.io/u/peter)
#### Post date: [February 5, 2021, 8:58am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/8 "2021-02-05T08:58:29Z")
Hello @jakew009
thank you for using Mender.
mender-connect is designed to handle exactly the scenario you are describing, i.e.: it waits until in can reconnect, and resumes normal operations.
could I ask for you enable debug output? by changing the following line in the `mender-connect.service` file:
```auto
ExecStart=/usr/bin/mender-connect daemon
```
into
```auto
ExecStart=/usr/bin/mender-connect --debug daemon
```
and send the log?
thank you.
Also, the mender-client process is running at all times, right? and you restart only mender-connect? Could you share the log of mender-client as well?
best regards,
peter
---
### Author:  [@anishmonachan](https://hub.mender.io/u/anishmonachan)
#### Post date: [May 27, 2021, 6:57pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/9 "2021-05-27T18:57:03Z")
I too have the same issue. I had to restart mender-client to see the launch remote terminal button. @peter , is this issue solved?
For some devices, I tried to reboot the device without restarting the mender-client. But the button doesn’t appear on the mender server.
Once you restarted the mender-client manually, you’ll see the launch button. After that, I had no issues launching the terminal even after a reboot.
Restarting mender-client manually at runtime is the only solution. weird.
---
### Author:  [@jakew009](https://hub.mender.io/u/jakew009)
#### Post date: [May 28, 2021, 10:38am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/10 "2021-05-28T10:38:41Z")
We absolutely still have the same problem as well on random devices. Restarting mender-connect always fixes it immediately. Luckily we have a vpn and mender-connect is just a backup in case we break the vpn config somehow.
I haven’t had chance to get logs and because it happens randomly, it’s hard to reproduce / chose which devices to log.
---
### Author:  [@peter](https://hub.mender.io/u/peter)
#### Post date: [July 21, 2021, 11:12am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/11 "2021-07-21T11:12:24Z")
Hello,
the issue has been fixed with the new release. please give it a go.
best regards
peter
---
### Author:  [@HerrMuellerluedensch](https://hub.mender.io/u/HerrMuellerluedensch)
#### Post date: [December 4, 2021, 8:09pm UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/12 "2021-12-04T20:09:28Z")
@peter can you elaborate where and from which version on this was fixed exactly? `mender v2.5.x` ? `mender-connect v1.1` and up?
We’ve had the same issue and I’d like to make sure it’s solved in our next release.
---
### Author:  [@kacf](https://hub.mender.io/u/kacf)
#### Post date: [December 6, 2021, 5:15am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/13 "2021-12-06T05:15:52Z")
It was fixed in Mender client 2.5.2, 2.6.1 and later versions.
---
### Author:  [@rlaybourn](https://hub.mender.io/u/rlaybourn)
#### Post date: [February 21, 2022, 9:46am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/14 "2022-02-21T09:46:38Z")
I appear to be having a very similar problem and I am on version 2.6.1 . It is a yocto build on Zeus. Some devices loose their connection to mender with the error messages “The server certificate cannot be loaded: no file provided” and “messageLoop: error on readMessage: websocket: close 1011 (inter
nal server error): read tcp 172.23.102.249:8080-\>172.23.121.10:35918: i/o timeout; disconnecting, waiting for reconnect.”. They will not reconnect until the mender client has been restarted
---
### Author:  [@dbasner](https://hub.mender.io/u/dbasner)
#### Post date: [November 30, 2023, 9:36am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/15 "2023-11-30T09:36:40Z")
Has anyone found a solution yet? @rlaybourn have you been able to resolve this issue?
---
### Author:  [@saminurm](https://hub.mender.io/u/saminurm)
#### Post date: [September 23, 2025, 5:44am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/16 "2025-09-23T05:44:14Z")
Happens still with mender-connect 2.3.0 / Mender client 3.5.3
```auto
mender[685]: time=“2025-09-23T05:38:21Z” level=error msg=“error forwarding from backend to client: websocket: close 1011 (internal server error): read tcp 172.23.153.177:8080->172.23.149.7:45526: i/o timeout” thread=proxy
mender-connect[686]: time=“2025-09-23T05:38:21Z” level=error msg=“messageLoop: error on readMessage: websocket: close 1011 (internal server error): read tcp 172.23.153.177:8080->172.23.149.7:45526: i/o timeout; disconnecting, waiting for reconnect.”
```
---
### Author:  [@saminurm](https://hub.mender.io/u/saminurm)
#### Post date: [September 25, 2025, 9:23am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/17 "2025-09-25T09:23:57Z")
Tried to update the Mender to 5.0.2, but the mender-connect (2.3.1) still has the same problem
---
### Author:  [@saminurm](https://hub.mender.io/u/saminurm)
#### Post date: [September 29, 2025, 5:55am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/18 "2025-09-29T05:55:36Z")
There was quite high Rx packet loss in our device (about 20%). So it wasn’t mender-connect problem after all, but the error message (“internal server error”) is a bit confusing.
---
### Author:  [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)
#### Post date: [September 29, 2025, 7:29am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/19 "2025-09-29T07:29:32Z")
Hi @saminurm,
Thanks for the follow-up. So you could identify a technical problem in the network connectivity?
Greetz,
Josef
---
### Author:  [@saminurm](https://hub.mender.io/u/saminurm)
#### Post date: [September 29, 2025, 8:34am UTC](https://hub.mender.io/t/problem-with-mender-connect-remote-terminal/3141/20 "2025-09-29T08:34:49Z")
Hi @TheYoctoJester ,
Yes, technically it was an invalid configuration for a supply regulator in our device’s kernel devicetree. The mender-connect and Mender server were behaving ok, I was just confused by the error message and thought that it was some Mender server error.
Best regards,
Sami