Mender connect trying to authenticate against a local service?

General Discussions
1

Greetings,

I am evaluating Mender for my company, and I came across an error and I did not successfully got over it after trying for over a day. Important notice: We are not running an enterprise installation, we only use the open source version of the software

I have made a prod installation in my K8s cluster successfully, and I was able to add new devices as expected. The only (super important) missing part is the troubleshooting part. This seems to not work at all:

image
image855×221 9.24 KB

From what I have read in the docs, this simply requires a working config of mender-connect. When I run sudo mender-connect daemon I get the following logs:

WARN[2024-03-25T12:21:39+01:00] SkipVerify field is deprecated, ignoring.
INFO[2024-03-25T12:21:39+01:00] Loaded configuration file: /etc/mender/mender-connect.conf
WARN[2024-03-25T12:21:39+01:00] ShellArguments is empty, defaulting to [--login]
ERRO[2024-03-25T12:21:49+01:00] connection manager failed to connect to http://127.0.0.1:36323/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 1/10); len(token)=1024
ERRO[2024-03-25T12:21:54+01:00] connection manager failed to connect to http://127.0.0.1:36323/api/devices/v1/deviceconnect/connect: websocket: bad handshake; reconnecting in 5s (try 2/10); len(token)=1024
...

mender.conf:

{
    "HttpsClient": {},
    "Security": {},
    "Connectivity": {},
    "DeviceTypeFile": "/var/lib/mender/device_type",
    "DBus": {
        "Enabled": true
    },
    "UpdatePollIntervalSeconds": 1800,
    "InventoryPollIntervalSeconds": 28800,
    "RetryPollIntervalSeconds": 300,
    "ServerURL": "https://mender.<my-domain>.com"
}

mender-connect.conf

{
  "User": "root",
  "ShellCommand": "/bin/bash"
}

Output of the command sudo dbus-send --system --dest=io.mender.AuthenticationManager --print-reply /io/mender/AuthenticationManager io.mender.Authentication1.GetJwtToken:

   string "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJlMDU3MTY2ZC0yYjZhLTQyMTMtYWIzZi0wMTczNmY4MjdiZGMiLCJzdWIiOiIyOTdjZmI2ZC1kYTVmLTRjNDMtYmEwNy1hMjdmY2IwZWViYjgiLCJpc3MiOiJNZW5kZXIiLCJleHAiOjE3MTE5NzAxNDgsImlhdCI6MTcxMTM2NTM0OCwibmJmIjotNjIxMzU1OTY4MDAsIm1lbmRlci5kZXZpY2UiOnRydWUsIm1lbmRlci5wbGFuIjoiZW50ZXJwcmlzZSIsIm1lbmRlci50cmlhbCI6ZmFsc2UsIm1lbmRlci5hZGRvbnMiOlt7Im5hbWUiOiJjb25maWd1cmUiLCJlbmFibGVkIjp0cnVlfSx7Im5hbWUiOiJ0cm91Ymxlc2hvb3QiLCJlbmFibGVkIjp0cnVlfSx7Im5hbWUiOiJtb25pdG9yIiwiZW5hYmxlZCI6dHJ1ZX1dfQ.PYprJGO9EVlKnYJyK-zwJc43D6k8kCBf5L4Rg5Lr0gLXEmzmBAdvWChfx0SBAzh9kqzdbn9aMVlOBnds86zj2LX95hKrMAYn7XLy2_EaIwVPNhsbOnG176BE-h5Yp8N8X5RQVwpWIJr5HM0qwW_ZakaIsZWsobIOoz6Xzp31Hu0cxEAmcNT3YWHk3FpeyHxumnoxWftGf4xKQs-jpGTavY0qLsoDqfaKRcdvIpiqVe4hUCpyFN4cQdTcGuKgrVYjNkuOwCsMT2qEp5FkanWHVSI1pBDFL9IoB2xUaL8xgU_rwbnz3-s3D0nKk1Qm7Jd-uwQIvqiyHCmofw0c6oLeXkypknYSbAeD7mhUTlhSRsnyuztxyGSIVN5L5MsKON8-cZRAtdhDWzT10WUwC1bmsCSr5rAgIUMaI_GG5PtE1w1V_DFLJ4Y7F8TCoR4NoOac_Zau1ii0Z_gNFMI3MHpkIM2qrfM0Ld4CcjEvwcqhYKU9rI-7YFLQaEKbIkrRFoQg"
   string "http://127.0.0.1:36323"

What am I missing? Is it normal that this tries to run against the host 127.0.0.1 instead of the host of my mender server?

Thanks a lot in advance!

2

Just in case somebody comes across the same problem, you simply need to enable ws in your ingress controller, otherwise this will not work. I think the Mender team should update the documentation on this regard. The following ingress did the trick for me:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mender-ingress
  namespace: mender
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    acme.cert-manager.io/http01-edit-in-place: "true"
    kubernetes.io/ingress.class: nginx
    nginx.org/client-max-body-size: 128m
    nginx.org/websocket-services: mender-api-gateway
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - mender.mydomain.com
    secretName: mender-ingress-tls
  rules:
  - host: "mender.mydomain.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mender-api-gateway
            port:
              number: 80
1 Like
3

Hello @vorkutt ,
thanks for your kind report! We are going to update the documentation for the Nginx ingress controller.