Remote terminal not working - on-prem server

extm · April 25, 2024, 8:26am

Hi,

We have moved to our production on-prem server and now our remote terminal is not working. I have verified that mender-connect works by using mender-cli and following Remote Terminal | Mender documentation. When using the mender UI clicking on the “Connect Terminal” I get “Connection to the remote terminal is forbidden.” I see it for a second then it disappear. I see nothing on the target that there is ongoing session that is failing. Any suggestion how to debug this would be appreciated. I have debugged the target and I cannot find any issues on the target side don’t know much about the server side.

mender-client: 3.5.2
mender-server: 3.6.0

Looks like the actual error that we see is from the gui and is located at

github.com

mendersoftware/gui/blob/master/src/js/components/devices/troubleshoot/terminal-wrapper.js#L132


      
            }
            onNotify('Health check failed: connection with the device lost.');
          }, [onNotify, socketInitialized]);
          
          const onSocketClose = useCallback(
            event => {
              // abnormal socket close might happen without socket being initialized, in case of forbidden permissions
              // this should be checked before socketInitialized condition
              if (event.code === 1006) {
                // 1006: abnormal closure
                onNotify('Connection to the remote terminal is forbidden.');
              }
          
              if (!socketInitialized) {
                return;
              }
              if (event.wasClean) {
                onNotify(`Connection with the device closed.`);
              } else {
                onNotify('Connection with the device died.');
              }

Transfering files works which I assume is also using a similar mechanism as the terminal so the issue is very specific for opening a remote terminal using the GUI.

Thanks

extm · April 25, 2024, 9:48am

From the debug tools in chrome I see the following error

Websocket connection to 'wss:///api/management/v1/deviceconnect/devices//connect" failed

when running mender-cli I see the following

GET /api/management/v1/deviceconnect/devices/ HTTP/1.1

So looks like they are using the same API but for some reason when using the UI it is forbidden while when using the mender-cli it is allowed

extm · April 25, 2024, 10:48am

On the server side I am seeing the following

time="2024-04-25T10:41:39Z" level=info byteswritten=153 clientip=10.42.0.191 file=middleware_gin.go func=accesslog.Middleware.func1 line=131 method=GET path=/api/management/v1/deviceconnect/devices/<device-id> qs= request_id=<req-id-1> responsetime=680us status=200 ts="2024-04-25T10:41:39Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

time="2024-04-25T10:41:41Z" level=error msg="unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin" file=management.go func=http.ManagementController.Connect line=210 request_id=<req-id-2> user_id=<user-id>

time="2024-04-25T10:41:41Z" level=error byteswritten=126 clientip=10.42.0.191 error=Forbidden file=middleware_gin.go func=accesslog.Middleware.func1 line=153 method=GET path=/api/management/v1/deviceconnect/devices/<device-id>/connect qs= request_id=<req-id-2> responsetime=1588us status=403 ts="2024-04-25T10:41:41Z" type=HTTP/1.1 user_id=<user-id> useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

extm · April 25, 2024, 10:59am

The initial error on the server side can be tracked to

github.com

mendersoftware/deviceconnect/blob/master/api/http/management.go#L209


      
          			"error": "failed to establish internal device session",
          		})
          		return
          	}
          	//nolint:errcheck
          	defer sub.Unsubscribe()
          
          	// upgrade get request to websocket protocol
          	conn, err := wsUpgrader.Upgrade(c.Writer, c.Request, nil)
          	if err != nil {
          		err = errors.Wrap(err, "unable to upgrade the request to websocket protocol")
          		l.Error(err)
          		// upgrader.Upgrade has already responded
          		return
          	}
          	conn.SetReadLimit(int64(app.MessageSizeLimit))
          
          	//nolint:errcheck
          	h.ConnectServeWS(ctx, conn, session, deviceChan)
          }

and

github.com

mendersoftware/deviceconnect/blob/master/api/http/management.go#L255


      
          		if err != nil {
          			sleepMilliseconds = uint(n)
          		}
          	}
          
          	l.Infof("Playing back the session session_id=%s", sessionID)
          
          	// upgrade get request to websocket protocol
          	conn, err := wsUpgrader.Upgrade(c.Writer, c.Request, nil)
          	if err != nil {
          		err = errors.Wrap(err, "unable to upgrade the request to websocket protocol")
          		l.Error(err)
          		return
          	}
          	conn.SetReadLimit(int64(app.MessageSizeLimit))
          
          	deviceChan := make(chan *natsio.Msg, channelSize)
          	errChan := make(chan error, 1)
          
          	//nolint:errcheck
          	go h.websocketWriter(ctx,

I assume the issue is from the connect call. Not sure what the Upgrade call is any ideas what the purpose is and why it is is failing when using the UI and not the mender-cli?

extm · April 25, 2024, 11:10am

The output on the server side is from running

kubectl logs mender-deviceconnect-<id> -f

Is there some other service that I should check the logs for to see why we are getting this issue when using the UI and not when running the mender-cli?

extm · April 25, 2024, 11:51am

This is not my really my area of expertise but from looking at the code the call that is failing is trying to convert an http get request to a websocket call or what? The gui side is failing

github.com

mendersoftware/gui/blob/master/src/js/utils/sockethook.js#L136


      
            e => {
              console.log('closing');
              onClose(e);
              close();
            },
            [close, onClose]
          );
          
          const connect = useCallback(
            deviceId => {
              const uri = `wss://${window.location.host}${apiUrl.v1}/deviceconnect/devices/${deviceId}/connect`;
              setSessionId();
              cookies.set('JWT', token, { path: '/', secure: true, sameSite: 'strict', maxAge: 5 });
              try {
                socketRef.current = new WebSocket(uri);
              } catch (error) {
                console.log(error);
              }
            },
            [token]
          );

Is there a missmatch between the server side and the client side?

extm · April 25, 2024, 8:21pm

The error message indicates that the WebSocket handshake request was rejected because the origin (the domain from which the request originated) is not allowed according to the criteria defined in the CheckOrigin function. So some miss-configuration of the server maybe?

extm · April 26, 2024, 8:14am

From what I can tell we have this function

github.com

mendersoftware/deviceconnect/blob/master/api/http/management.go#L79


      
          	//possible delay. We round down to this if the real delay is larger
          	keyStrokeMaxDelayRecording = int64(65535 * 1000000)
          )
          
          const channelSize = 25 // TODO make configurable
          
          const (
          	PropertyUserID = "user_id"
          )
          
          var wsUpgrader = websocket.Upgrader{
          	Subprotocols: []string{"protomsg/msgpack"},
          	CheckOrigin:  allowAllOrigins,
          	Error: func(
          		w http.ResponseWriter, r *http.Request, s int, e error,
          	) {
          		w.WriteHeader(s)
          		enc := json.NewEncoder(w)
          		_ = enc.Encode(rest.Error{
          			Err:       e.Error(),
          			RequestID: requestid.FromContext(r.Context())},

where CheckOrigin is set to allowAllOrigins and the only function that I could find is defined her

github.com

mendersoftware/deviceconnect/blob/master/api/http/cors.go#L21


      
          //    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          //    See the License for the specific language governing permissions and
          //    limitations under the License.
          
          package http
          
          import (
          	"net/http"
          )
          
          func allowAllOrigins(r *http.Request) bool { return true }
          
          func SetAcceptedOrigins(origins []string) {
          	switch len(origins) {
          	case 0:
          		wsUpgrader.CheckOrigin = allowAllOrigins
          	case 1:
          		origin := origins[0]
          		wsUpgrader.CheckOrigin = func(r *http.Request) bool {
          			if actual, ok := r.Header[HdrKeyOrigin]; ok {
          				return len(actual) > 0 && origin == actual[0]

If this is correct then I don’t understand why we are getting

time=“2024-04-25T10:41:41Z” level=error msg=“unable to upgrade the request to websocket protocol: websocket: request origin not allowed by Upgrader.CheckOrigin” file=management.go func=http.ManagementController.Connect line=210 request_id= user_id=

Since the allowAllOrigin returns true. But at the same time there is a configuration options available

github.com

mendersoftware/deviceconnect/blob/master/config/config.go#L88


      
          	// SettingEnableAuditLogs enables/disables audit logging.
          	SettingEnableAuditLogs = "enable_audit"
          	// SettingEnableAuditLogsDefault is disabled by default.
          	SettingEnableAuditLogsDefault = false
          
          	// SettingLogExpireSec is the config key for how long logs will be
          	// retained in the database.
          	SettingRecordingExpireSec     = "recording_expire_seconds"
          	SettingRecordingExpireDefault = 30 * 24 * 60 * 60
          
          	// SettingWSAllowedOrigin configures the allowed origins to use the websocket APIs.
          	// An empty list will disable cors checks
          	SettingWSAllowedOrigins        = "ws.allowed_origins"
          	SettingWSAllowedOriginsDefault = ""
          
          	// SettingGracefulShutdownTimeout is the config key for the
          	// graceful shutdown timeout.
          	SettingGracefulShutdownTimeout        = "graceful_shutdown_timeout"
          	SettingGracefulShutdownTimeoutDefault = "60s"
          )

which indicates that the origin can be configured. An it looks like when server is initialized we have the following

github.com

mendersoftware/deviceconnect/blob/master/server/server.go#L44


      
          	"github.com/mendersoftware/deviceconnect/store"
          )
          
          // InitAndRun initializes the server and runs it
          func InitAndRun(conf config.Reader, dataStore store.DataStore) error {
          	ctx := context.Background()
          
          	log.Setup(conf.GetBool(dconfig.SettingDebugLog))
          	l := log.FromContext(ctx)
          
          	allowedOrigin := conf.GetStringSlice(dconfig.SettingWSAllowedOrigins)
          	if allowedOrigin != nil {
          		api.SetAcceptedOrigins(allowedOrigin)
          	}
          
          	natsClient, err := nats.NewClientWithDefaults(
          		config.Config.GetString(dconfig.SettingNatsURI),
          	)
          	if err != nil {
          		return err
          	}

suggesting that the origin is set as part of the that init process.

extm · April 26, 2024, 8:34am

I found this old post so it is not applicable directly but I assume there is something similar today

robgio · April 26, 2024, 9:18am

Hello @extm ,
can you share more information about your setup? Where and how the Mender server is installed? Are you using a Load balancer that allows WSS connections?

Thanks

extm · April 26, 2024, 9:27am

Hi,

Thanks for your reply. We have been using Production installation with Kubernetes | Mender documentation to set it up. I just found this gui/httpd.conf at master · mendersoftware/gui · GitHub specifying a list of Content-Security-Policy which seems like an interesting candidate. Let me check regarding the load balancer and allowing WSS connections. I did not personally set it up and is mostly working on the target so you will have to excuse me if I am using the wrong terminology.

Thanks

robgio · April 26, 2024, 9:34am

So you set it up on a single VM, right?
Do you have any error regarding the CSP in the browser console?

extm · April 26, 2024, 9:39am

Yes I am pretty sure we run this on an single VM. Don’t see that the installation instructions mentions anything about setting up multiple VMs. Regarding the error the only thing that I can see is that we get

“Connection to the remote terminal is forbidden.”

in the UI and in the browser console I can see the following error

sockethook.js:161 WebSocket connection to 'wss://coffeemender.publicvm.com/api/management/v1/deviceconnect/devices/<device-id>/connect'

but maybe I should look somewhere else for a specific CSP error. The actual error that indicates that it is an origin issue is from the backend when I run

kubectl logs mender-deviceconnect-<id> -f

and notice

extm · April 26, 2024, 12:22pm

Ok so it looks like the issue was a url miss-configuration in the mender-0.3.6.yml file for kubernetes the url was pointing to an azure vm url instead of our external domain. Not sure why this only caused the terminal to fail and nothing else but it seems to be working after adjusting that url. I hope this can be of assistance to anyone else having a similar issue.

Topic		Replies	Views
Remote terminal connection failed General Discussions yocto , raspberry-pi-3 , mender-connect	18	1656	June 30, 2021
Unable to connect terminal : WebSocket connection failed General Discussions mender-connect	5	607	February 15, 2024
Remote terminal not working General Discussions mender-client , mender-connect , dbus	4	524	January 19, 2023
Mender Remote Terminal Unavailable? General Discussions	5	489	March 11, 2022
Mender Troubleshoot Add-on reliability General Discussions	8	191	June 13, 2024

Related topics