# Hosted Mender Connect Remote Terminal Forbidden

**URL:** <https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121>\
**Category:** General Discussions\
**Created:** [July 14, 2022, 6:31pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121 "2022-07-14T18:31:43Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 14, 2022, 6:31pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/1 "2022-07-14T18:31:43Z")

</div>

I am trying to stand up some test machines running Debian 11 and installing the mender-client and mender-connect deb packages using the docs form the APT repo.

I have successful installed the mender-client, mender-connect, and mender-configure deb packages and I am able to register my test machines in the mender ui using a basic demo config.

I have also paid for the configure and troubleshooting add-ons for my account.

When attempting to Remote terminal from the Mender Hosted UI I get a forbidden.

Watching the debug logs on the mender client show no errors, however watching a trace output from the mender-connect logs show the following

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/646b2f69e8b047bfa1290845865bd89acffccdab.png)

I did find some references in previous posts to running a dbus-send command to check if the JWT was being provided by the localhost endpoint. I can confirm that is it and the server in the logs of the mender-connect matches the server returned by the dbus-send command.

Otherwise I can find no other errors in either the mender-connect or mender-client logs and am wondering what is going on.

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [July 15, 2022, 12:57am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/2 "2022-07-15T00:57:00Z")

</div>

Hello @vhbgraham,

Is the mender-connect service running?

```auto
$ systemctl status mender-connect

```

The mender client is run by systemd, meaning, it uses root permissions, while if you run the mender-connect as a regular user probably that is the reason it is not working as it is not able to get access to the credentials the mender-client has. If you don’t want to use it as a regular systemd service, at least you need to run it with `sudo` or the root user.

Best regards,  
Luis

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 15, 2022, 1:59am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/3 "2022-07-15T01:59:03Z")

</div>

Both are run as systemd services as that is how the .deb sets them up

mender-client.service

```auto
root@ip-172-31-1-10:/var/lib/systemd# cat /lib/systemd/system/mender-client.service
[Unit]
Description=Mender OTA update service
Wants=network-online.target
After=systemd-resolved.service network-online.target mender.service mender-client-data-dir.service data.mount
Conflicts=mender.service

[Service]
Type=idle
User=root
Group=root
ExecStart=/usr/bin/mender daemon
Restart=on-abort

[Install]
WantedBy=multi-user.target

```

mender-connect.service

```auto
root@ip-172-31-1-10:/var/lib/systemd# cat /lib/systemd/system/mender-connect.service
[Unit]
Description=Mender Connect service
Wants=network-online.target
After=systemd-resolved.service network-online.target mender-client.service
Requires=mender-client.service

[Service]
Type=idle
User=root
Group=root
ExecStart=/usr/bin/mender-connect daemon
Restart=on-abort

[Install]
WantedBy=multi-user.target

```

```auto
root@ip-172-31-1-10:/var/lib/systemd# systemctl status mender-client
● mender-client.service - Mender OTA update service
     Loaded: loaded (/lib/systemd/system/mender-client.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-07-15 01:47:50 UTC; 9min ago
   Main PID: 575 (mender)
      Tasks: 11 (limit: 1112)
     Memory: 40.7M
        CPU: 2.031s
     CGroup: /system.slice/mender-client.service
             └─575 /usr/bin/mender daemon

```

```auto
root@ip-172-31-1-10:/var/lib/systemd# systemctl status mender-connect
● mender-connect.service - Mender Connect service
     Loaded: loaded (/lib/systemd/system/mender-connect.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-07-15 01:47:50 UTC; 10min ago
   Main PID: 576 (mender-connect)
      Tasks: 10 (limit: 1112)
     Memory: 21.1M
        CPU: 128ms
     CGroup: /system.slice/mender-connect.service
             └─576 /usr/bin/mender-connect daemon

```

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 15, 2022, 2:03am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/4 "2022-07-15T02:03:03Z")

</div>

Here is a fresh restart of mender-connect with --trace turned on

```auto
● mender-connect.service - Mender Connect service
     Loaded: loaded (/lib/systemd/system/mender-connect.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-07-15 02:00:36 UTC; 15s ago
   Main PID: 1456 (mender-connect)
      Tasks: 11 (limit: 1112)
     Memory: 5.4M
        CPU: 22ms
     CGroup: /system.slice/mender-connect.service
             └─1456 /usr/bin/mender-connect --trace daemon

Jul 15 02:00:36 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:36Z" level=trace msg="messageLoop: starting"
Jul 15 02:00:36 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:36Z" level=trace msg="messageLoop: posting event: reconnect-req; waiting for response"
Jul 15 02:00:36 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:36Z" level=trace msg="needsReconnect: got event: reconnect-req"
Jul 15 02:00:36 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:36Z" level=trace msg="dbusEventLoop: daemon needs to reconnect"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateCha>
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="(dbusEventLoop) posting Event: reconnect"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="eventLoop: got event: reconnect"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=info msg="eventLoop: Connection established with http://localhost:46299"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="messageLoop: got response: {event:connected data: id:}"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="messageLoop: calling readMessage"
Jul 15 02:00:57 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:57Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:00:46 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:46Z" level=trace msg="messageLoop: calling readMessage"
Jul 15 02:00:57 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:00:57Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:01:08 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:08Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:01:19 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:19Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:01:30 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:30Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:01:40 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:40Z" level=debug msg="ping message"
Jul 15 02:01:40 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:40Z" level=debug msg="PongHandler called"
Jul 15 02:01:41 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:41Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:01:52 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:01:52Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:02:03 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:02:03Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:02:14 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:02:14Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:02:25 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:02:25Z" level=trace msg="dbusEventLoop: WaitForJwtTokenStateChange [], err timeout waiting for signal JwtTokenStateChange"
Jul 15 02:02:34 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:02:34Z" level=debug msg="ping message"
Jul 15 02:02:34 ip-172-31-1-10 mender-connect[1456]: time="2022-07-15T02:02:34Z" level=debug msg="PongHandler called"

```

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 15, 2022, 2:29am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/5 "2022-07-15T02:29:39Z")

</div>

What is interesting and that file uploads and download work fine and that uses the mender-connect daemon I believe

Log after a successful upload

```auto
Jul 15 02:25:22 ip-172-31-1-10 mender-connect[1550]: time="2022-07-15T02:25:22Z" level=info msg="session: accepting new session with ID: 8a60b59c-c29c-4980-a4db-7526723fa817"
Jul 15 02:25:22 ip-172-31-1-10 mender-connect[1550]: time="2022-07-15T02:25:22Z" level=info msg="session: closed 8a60b59c-c29c-4980-a4db-7526723fa817"

```

Log after successful download

```auto
Jul 15 02:28:13 ip-172-31-1-10 mender-connect[1550]: time="2022-07-15T02:28:13Z" level=info msg="session: accepting new session with ID: 624524e6-8a63-423f-b859-1132d3e26310"
Jul 15 02:28:13 ip-172-31-1-10 mender-connect[1550]: time="2022-07-15T02:28:13Z" level=info msg="session: closed 624524e6-8a63-423f-b859-1132d3e26310"

```

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 15, 2022, 3:39pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/6 "2022-07-15T15:39:16Z")

</div>

Let me also note the the installation enviroment is as follows

1. QEMU Virtual machine, X86-64 running Debian 11 (bullseye) net-install/minimal installation with not extra packages selected, and mender is deployed following

`https://docs.mender.io/client-installation/install-with-debian-package` – using the script to install the client, and then apt-get install mender-connect mender-configure.

mender setup is using hosted and mender-connect.conf is a follows

```auto
{
  "ShellCommand": "/bin/bash",
  "User": "root"
}

```

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 16, 2022, 12:02am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/7 "2022-07-16T00:02:43Z")

</div>

Update from my end. I tried using mender-cli which did work. I am able to upload, download, port-forward, and terminal into my device for the CLI running on a local machine to me.

So this seems to be an issues with the [hosted.mender.io](http://hosted.mender.io) web interface – Still an issue as that is part of the troubleshoot offering, but at least some progress.

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/1/15875860d21c20aba0482a5a4d5e3addb2519230.png)

---

<div class="post-metadata">

**Author:** ![peter](https://avatars.discourse-cdn.com/v4/letter/p/278dde/32.png) [@peter](https://hub.mender.io/u/peter)\
**Post date:** [July 18, 2022, 4:55am UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/8 "2022-07-18T04:55:55Z")

</div>

Hello @vhbgraham

Thank you for using Mender. Could we try two things?

1. reload without cache (command/control+r in some browsers), and/or use incognito/private window/tab?

2. could you open developer tools and show me both the console and the network tabs when you try to open the terminal?

best regards,  
peter

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 18, 2022, 2:25pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/9 "2022-07-18T14:25:26Z")

</div>

Load the page with both Cache Disabled in Dev tools, and in Incognito Modes results in the same error – below is a screen shot of the dev tools network during the Incognito Run – tried connecting 3 times

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/6cd92a78daed2783d3d22a8ba940bcc7f209feda.png)

Here is output from a “normal” browser session (its the same result as the above attempt)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/b/b928ff23bf470bbe066d673394d6efb69b033707.png)

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 18, 2022, 2:29pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/10 "2022-07-18T14:29:41Z")

</div>

Here is the output from one of the connect in the network log (I am excluding cookies since there is a secret in there)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/0/097f0dcfb682cab0dfdfa3cf0edddc48d02760d3.png)

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/c/c612699e53aeca8ad508d39f3ce7b22693e574a7.png)

Here are the cookies that aren’t the JWT  
grav-site-8c069df=6e4d692979579274335dbd080f3a1a9c; \_ga=GA1.2.857832636.1658154050; \_gid=GA1.2.1618402795.1658154050; \_gat=1

Seems I am getting a 400 due to an error in the websocket handshake

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/6ebbfd5d8923f52a30eded3d91068ce08e24be46.png)

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [July 21, 2022, 4:19pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/11 "2022-07-21T16:19:00Z")

</div>

Bumping this as I am still having this issue on the webui side

---

<div class="post-metadata">

**Author:** ![pv3](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pv3/32/1614_2.png) [@pv3](https://hub.mender.io/u/pv3)\
**Post date:** [September 29, 2022, 10:04pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/12 "2022-09-29T22:04:21Z")

</div>

Im also bumping this as it seems to be a [hosted.mender.io](http://hosted.mender.io)

---

<div class="post-metadata">

**Author:** ![vhbgraham](https://avatars.discourse-cdn.com/v4/letter/v/ecb155/32.png) [@vhbgraham](https://hub.mender.io/u/vhbgraham)\
**Post date:** [September 29, 2022, 10:24pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/13 "2022-09-29T22:24:17Z")

</div>

I agree. It actually worked for a bit and then stopped working the CLI seems to work generally most of the time. It it so sad the support if so terrible even when paying for hosted mender.

---

<div class="post-metadata">

**Author:** ![pv3](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/pv3/32/1614_2.png) [@pv3](https://hub.mender.io/u/pv3)\
**Post date:** [September 30, 2022, 4:41pm UTC](https://hub.mender.io/t/hosted-mender-connect-remote-terminal-forbidden/5121/14 "2022-09-30T16:41:26Z")

</div>

After a series of reboots (luckily I have a reboot function on the apps) I was able to connect for a few minutes and get a VPN setup…which is what we were doing before using this add-on. I get that it’s not a SSH session and mender has their work cut out for them, but would really appreciate support/transparency.