Remote terminal not working - on-prem server


We have moved to our production on-prem server and now our remote terminal is not working. I have verified that mender-connect works by using mender-cli and following Remote Terminal | Mender documentation. When using the mender UI clicking on the “Connect Terminal” I get “Connection to the remote terminal is forbidden.” I see it for a second then it disappear. I see nothing on the target that there is ongoing session that is failing. Any suggestion how to debug this would be appreciated. I have debugged the target and I cannot find any issues on the target side don’t know much about the server side.

mender-client: 3.5.2
mender-server: 3.6.0

Looks like the actual error that we see is from the gui and is located at

Transfering files works which I assume is also using a similar mechanism as the terminal so the issue is very specific for opening a remote terminal using the GUI.


From the debug tools in chrome I see the following error

Websocket connection to 'wss:///api/management/v1/deviceconnect/devices//connect" failed

when running mender-cli I see the following

GET /api/management/v1/deviceconnect/devices/ HTTP/1.1

So looks like they are using the same API but for some reason when using the UI it is forbidden while when using the mender-cli it is allowed

On the server side I am seeing the following

time="2024-04-25T10:41:39Z" level=info byteswritten=153 clientip= file=middleware_gin.go func=accesslog.Middleware.func1 line=131 method=GET path=/api/management/v1/deviceconnect/devices/<device-id> qs= request_id=<req-id-1> responsetime=680us status=200 ts="2024-04-25T10:41:39Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36"

time="2024-04-25T10:41:41Z" level=error msg="unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin" file=management.go func=http.ManagementController.Connect line=210 request_id=<req-id-2> user_id=<user-id>

time="2024-04-25T10:41:41Z" level=error byteswritten=126 clientip= error=Forbidden file=middleware_gin.go func=accesslog.Middleware.func1 line=153 method=GET path=/api/management/v1/deviceconnect/devices/<device-id>/connect qs= request_id=<req-id-2> responsetime=1588us status=403 ts="2024-04-25T10:41:41Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537.36"

The initial error on the server side can be tracked to


I assume the issue is from the connect call. Not sure what the Upgrade call is any ideas what the purpose is and why it is is failing when using the UI and not the mender-cli?

The output on the server side is from running

kubectl logs mender-deviceconnect-<id> -f

Is there some other service that I should check the logs for to see why we are getting this issue when using the UI and not when running the mender-cli?

This is not my really my area of expertise but from looking at the code the call that is failing is trying to convert an http get request to a websocket call or what? The gui side is failing

Is there a missmatch between the server side and the client side?

The error message indicates that the WebSocket handshake request was rejected because the origin (the domain from which the request originated) is not allowed according to the criteria defined in the CheckOrigin function. So some miss-configuration of the server maybe?

From what I can tell we have this function

where CheckOrigin is set to allowAllOrigins and the only function that I could find is defined her

If this is correct then I don’t understand why we are getting

time=“2024-04-25T10:41:41Z” level=error msg=“unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin” file=management.go func=http.ManagementController.Connect line=210 request_id= user_id=

Since the allowAllOrigin returns true. But at the same time there is a configuration options available

which indicates that the origin can be configured. An it looks like when server is initialized we have the following

suggesting that the origin is set as part of the that init process.

I found this old post so it is not applicable directly but I assume there is something similar today

Hello @extm ,
can you share more information about your setup? Where and how the Mender server is installed? Are you using a Load balancer that allows WSS connections?



Thanks for your reply. We have been using Production installation with Kubernetes | Mender documentation to set it up. I just found this gui/httpd.conf at master · mendersoftware/gui · GitHub specifying a list of Content-Security-Policy which seems like an interesting candidate. Let me check regarding the load balancer and allowing WSS connections. I did not personally set it up and is mostly working on the target so you will have to excuse me if I am using the wrong terminology.


So you set it up on a single VM, right?
Do you have any error regarding the CSP in the browser console?

Yes I am pretty sure we run this on an single VM. Don’t see that the installation instructions mentions anything about setting up multiple VMs. Regarding the error the only thing that I can see is that we get

“Connection to the remote terminal is forbidden.”

in the UI and in the browser console I can see the following error

sockethook.js:161 WebSocket connection to 'wss://<device-id>/connect'

but maybe I should look somewhere else for a specific CSP error. The actual error that indicates that it is an origin issue is from the backend when I run

kubectl logs mender-deviceconnect-<id> -f

and notice

Ok so it looks like the issue was a url miss-configuration in the mender-0.3.6.yml file for kubernetes the url was pointing to an azure vm url instead of our external domain. Not sure why this only caused the terminal to fail and nothing else but it seems to be working after adjusting that url. I hope this can be of assistance to anyone else having a similar issue.