# Remote terminal not working - on-prem server

**URL:** <https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758>\
**Category:** General Discussions\
**Tags:** yocto, mender-connect\
**Created:** [April 25, 2024, 8:26am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758 "2024-04-25T08:26:31Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 8:26am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/1 "2024-04-25T08:26:31Z")

</div>

Hi,

We have moved to our production on-prem server and now our remote terminal is not working. I have verified that mender-connect works by using mender-cli and following [Remote Terminal | Mender documentation](https://docs.mender.io/add-ons/remote-terminal#using-your-own-terminal). When using the mender UI clicking on the “Connect Terminal” I get “Connection to the remote terminal is forbidden.” I see it for a second then it disappear. I see nothing on the target that there is ongoing session that is failing. Any suggestion how to debug this would be appreciated. I have debugged the target and I cannot find any issues on the target side don’t know much about the server side.

mender-client: 3.5.2  
mender-server: 3.6.0

Looks like the actual error that we see is from the gui and is located at

> <https://github.com/mendersoftware/gui/blob/master/src/js/components/devices/troubleshoot/terminal-wrapper.js#L132>

Transfering files works which I assume is also using a similar mechanism as the terminal so the issue is very specific for opening a remote terminal using the GUI.

Thanks

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 9:48am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/2 "2024-04-25T09:48:25Z")

</div>

From the debug tools in chrome I see the following error

Websocket connection to 'wss:///api/management/v1/deviceconnect/devices//connect" failed

when running mender-cli I see the following

GET /api/management/v1/deviceconnect/devices/ HTTP/1.1

So looks like they are using the same API but for some reason when using the UI it is forbidden while when using the mender-cli it is allowed

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 10:48am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/3 "2024-04-25T10:48:50Z")

</div>

On the server side I am seeing the following

```auto
time="2024-04-25T10:41:39Z" level=info byteswritten=153 clientip=10.42.0.191 file=middleware_gin.go func=accesslog.Middleware.func1 line=131 method=GET path=/api/management/v1/deviceconnect/devices/<device-id> qs= request_id=<req-id-1> responsetime=680us status=200 ts="2024-04-25T10:41:39Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

time="2024-04-25T10:41:41Z" level=error msg="unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin" file=management.go func=http.ManagementController.Connect line=210 request_id=<req-id-2> user_id=<user-id>

time="2024-04-25T10:41:41Z" level=error byteswritten=126 clientip=10.42.0.191 error=Forbidden file=middleware_gin.go func=accesslog.Middleware.func1 line=153 method=GET path=/api/management/v1/deviceconnect/devices/<device-id>/connect qs= request_id=<req-id-2> responsetime=1588us status=403 ts="2024-04-25T10:41:41Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

```

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 10:59am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/4 "2024-04-25T10:59:04Z")

</div>

The initial error on the server side can be tracked to

> <https://github.com/mendersoftware/deviceconnect/blob/master/api/http/management.go#L209>

and

> <https://github.com/mendersoftware/deviceconnect/blob/master/api/http/management.go#L255>

I assume the issue is from the connect call. Not sure what the Upgrade call is any ideas what the purpose is and why it is is failing when using the UI and not the mender-cli?

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 11:10am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/5 "2024-04-25T11:10:03Z")

</div>

The output on the server side is from running

`kubectl logs mender-deviceconnect-<id> -f`

Is there some other service that I should check the logs for to see why we are getting this issue when using the UI and not when running the mender-cli?

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 11:51am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/6 "2024-04-25T11:51:47Z")

</div>

This is not my really my area of expertise but from looking at the code the call that is failing is trying to convert an http get request to a websocket call or what? The gui side is failing

> <https://github.com/mendersoftware/gui/blob/master/src/js/utils/sockethook.js#L136>

Is there a missmatch between the server side and the client side?

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 25, 2024, 8:21pm UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/7 "2024-04-25T20:21:02Z")

</div>

The error message indicates that the WebSocket handshake request was rejected because the origin (the domain from which the request originated) is not allowed according to the criteria defined in the `CheckOrigin` function. So some miss-configuration of the server maybe?

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 26, 2024, 8:14am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/8 "2024-04-26T08:14:57Z")

</div>

From what I can tell we have this function

> <https://github.com/mendersoftware/deviceconnect/blob/master/api/http/management.go#L79>

where CheckOrigin is set to allowAllOrigins and the only function that I could find is defined her

> <https://github.com/mendersoftware/deviceconnect/blob/master/api/http/cors.go#L21>

If this is correct then I don’t understand why we are getting

time=“2024-04-25T10:41:41Z” level=error msg=“unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin” file=management.go func=http.ManagementController.Connect line=210 request\_id= user\_id=

Since the allowAllOrigin returns true. But at the same time there is a configuration options available

> <https://github.com/mendersoftware/deviceconnect/blob/master/config/config.go#L88>

which indicates that the origin can be configured. An it looks like when server is initialized we have the following

> <https://github.com/mendersoftware/deviceconnect/blob/master/server/server.go#L44>

suggesting that the origin is set as part of the that init process.

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 26, 2024, 8:34am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/9 "2024-04-26T08:34:40Z")

</div>

I found this old post so it is not applicable directly but I assume there is something similar today

> [@Issue with cross server API access](https://hub.mender.io/t/issue-with-cross-server-api-access/1300):
>
> Hello, I am running mender 2.1 on a local Ubuntu 18 machine. I am able to login using the mender gui, web based user interface. And also able to call the APIs using curl command line utility, as described in the documentation. I am facing issue while accessing mender APIs from the web page hosted on different server. I am getting http 400 error for preflight http OPTIONS request and then CORS error. As per the documentation mender supports OPTIONS and cross server API access. So I have tested…

---

<div class="post-metadata">

**Author:** ![robgio](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/robgio/32/1760_2.png) [@robgio](https://hub.mender.io/u/robgio)\
**Post date:** [April 26, 2024, 9:18am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/10 "2024-04-26T09:18:13Z")

</div>

Hello @extm ,  
can you share more information about your setup? Where and how the Mender server is installed? Are you using a Load balancer that allows WSS connections?

Thanks

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 26, 2024, 9:27am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/11 "2024-04-26T09:27:35Z")

</div>

Hi,

Thanks for your reply. We have been using [Production installation with Kubernetes | Mender documentation](https://docs.mender.io/3.7/server-installation/production-installation-with-kubernetes) to set it up. I just found this [gui/httpd.conf at master · mendersoftware/gui · GitHub](https://github.com/mendersoftware/gui/blob/master/httpd.conf#L49) specifying a list of Content-Security-Policy which seems like an interesting candidate. Let me check regarding the load balancer and allowing WSS connections. I did not personally set it up and is mostly working on the target so you will have to excuse me if I am using the wrong terminology.

Thanks

---

<div class="post-metadata">

**Author:** ![robgio](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/robgio/32/1760_2.png) [@robgio](https://hub.mender.io/u/robgio)\
**Post date:** [April 26, 2024, 9:34am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/12 "2024-04-26T09:34:06Z")

</div>

So you set it up on a single VM, right?  
Do you have any error regarding the CSP in the browser console?

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 26, 2024, 9:39am UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/13 "2024-04-26T09:39:45Z")

</div>

Yes I am pretty sure we run this on an single VM. Don’t see that the installation instructions mentions anything about setting up multiple VMs. Regarding the error the only thing that I can see is that we get

“Connection to the remote terminal is forbidden.”

in the UI and in the browser console I can see the following error

`sockethook.js:161 WebSocket connection to 'wss://coffeemender.publicvm.com/api/management/v1/deviceconnect/devices/<device-id>/connect'`

but maybe I should look somewhere else for a specific CSP error. The actual error that indicates that it is an origin issue is from the backend when I run

`kubectl logs mender-deviceconnect-<id> -f`

and notice

> [@extm](#):
>
> `time="2024-04-25T10:41:41Z" level=error msg="unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin" file=management.go func=http.ManagementController.Connect line=210 request_id=<req-id-2> user_id=<user-id>`

---

<div class="post-metadata">

**Author:** ![extm](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/extm/32/2171_2.png) [@extm](https://hub.mender.io/u/extm)\
**Post date:** [April 26, 2024, 12:22pm UTC](https://hub.mender.io/t/remote-terminal-not-working-on-prem-server/6758/14 "2024-04-26T12:22:04Z")

</div>

Ok so it looks like the issue was a url miss-configuration in the mender-0.3.6.yml file for kubernetes the url was pointing to an azure vm url instead of our external domain. Not sure why this only caused the terminal to fail and nothing else but it seems to be working after adjusting that url. I hope this can be of assistance to anyone else having a similar issue.