Login issues with mender 3.2 on kubernetes

General Discussions
1

Hi,

I’m facing right now is login in through the UI, by inspecting the network log, I can see that the login request was successful, but the subsequent request fails with an 401.
/api/management/v1/useradm/auth/login return with status 200
but
/api/management/v1/useradm/users/me return 401

From the useradm pod log Im getting:
“time=“2022-02-15T23:16:57Z” level=info msg=“200 346646μs POST /api/management/v1/useradm/auth/login HTTP/1.1 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36” byteswritten=816 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=POST path=/api/management/v1/useradm/auth/login qs= request_id=15d5fdef-139b-4f38-8d29-baaf9733cd98 responsetime=0.346646272 status=200 ts=“2022-02-15 23:16:57.536197765 +0000 UTC” type=http”

“time=“2022-02-15T23:16:58Z” level=warning msg=“Failed to parse extracted JWT: identity: incorrect token format” file=middleware.go func=“identity.(*IdentityMiddleware).MiddlewareFunc.func1” line=51 request_id=4a00ffcf-81b5-46f6-a36f-fccfe4ece8a7”

“time=“2022-02-15T23:16:58Z” level=error msg=“invalid jwt” file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=4a00ffcf-81b5-46f6-a36f-fccfe4ece8a7”

“time=“2022-02-15T23:16:58Z” level=info msg=“401 317μs GET /api/internal/v1/useradm/auth/verify HTTP/1.1 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36” byteswritten=75 file=middleware.go func=“accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1” line=71 method=GET path=/api/internal/v1/useradm/auth/verify qs= request_id=4a00ffcf-81b5-46f6-a36f-fccfe4ece8a7 responsetime=0.000317221 status=401 ts=“2022-02-15 23:16:58.307000967 +0000 UTC” type=http”

Would really appreciate any support on the issue.

2

Hello @kingman

thank you for using Mender!
I am sorry for the late response.
could you check, that you have correct private key for useradm service installed, and available in the useradm pod in /etc/useradm-enterprise/rsa?

best regards,
peter

3

the folder /etc/useradm-enterprise/ does not exist on my useradm pod

4

@kingman can you please check if you get an error when starting the useradm service, from the logs? (I think it’s /etc/useradm-enterprise/ is only if you’re using Mender Enterprise)

For your installation, if it doesn’t find the keys it should tell you where it’s looking for them.

This might be a bug in our installation docs / setup. Let’s solve your issue first and we can figure out what is wrong.

5

right. ok could you share the output of this command:

kubectl exec mender-useradm-pod-name -- sh -c set | grep PRIV_KEY

it should output a path. also could I see the useradm deployment manifest?

peter

6

not key was found by grep

7 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: mender
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-02-15T22:28:23Z"
  generation: 1
  labels:
    app.kubernetes.io/component: useradm
    app.kubernetes.io/instance: mender
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: useradm
    app.kubernetes.io/part-of: mender
    helm.sh/chart: mender
  name: useradm
  namespace: default
  resourceVersion: "55854"
  uid: 3a45be5f-948c-4b21-b5be-1e7ed0a3d78f
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: useradm
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: useradm
    spec:
      containers:
      - args:
        - server
        - --automigrate
        env:
        - name: USERADM_MIDDLEWARE
          value: prod
        - name: USERADM_JWT_ISSUER
          value: Mender Users
        - name: USERADM_JWT_EXP_TIMEOUT
          value: "604800"
        - name: USERADM_MIDDLEWARE
          value: prod
        - name: USERADM_TOTP_ISSUER
          value: Mender
        envFrom:
        - prefix: USERADM_
          secretRef:
            name: mongodb-common
        image: docker.io/mendersoftware/useradm:mender-3.2.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/internal/v1/useradm/alive
            port: 8080
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        name: useradm
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/internal/v1/useradm/health
            port: 8080
            scheme: HTTP
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 150m
            memory: 128M
          requests:
            cpu: 150m
            memory: 64M
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/useradm/rsa/
          name: rsa
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: rsa
        secret:
          defaultMode: 420
          secretName: rsa-useradm
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-02-15T22:28:24Z"
    lastUpdateTime: "2022-02-15T22:28:24Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-02-15T22:28:23Z"
    lastUpdateTime: "2022-02-15T22:28:29Z"
    message: ReplicaSet "useradm-59c9449649" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
8

thanks for all the information. and sorry for yet another question: do you have something inside /etc/useradm/rsa/ in the pod:

kubectl exec mender-useradm-pod-name -- ls -al /etc/useradm/rsa

if you have something there, could you verify that it is a valid RSA key?

pg

9

yes I have a private RSA key, which was generate during the installation process, according to this step:

lrwxrwxrwx    1 root     root            18 Feb 15 22:28 private.pem -> ..data/private.pem
10

to double check that this file exists, and the key is valid:

kubectl exec mender-useradm-pod-name -- cat /etc/useradm/rsa/private.pem | openssl rsa -modulus -noout -in - | openssl md5

just to be sure, include this in the deployment:

        - name: USERADM_SERVER_PRIV_KEY_PATH
          value: /etc/useradm/rsa/private.pem

and restart the pods?

pg

11

I added the new ENV in the manifest and restarted the pod, but still the login fails with the same behavior.

12

ok then.
lets approach from the other side.
could you login via curl:

jwt=`curl -X POST -H "Content-Type: application/json;" -u 'user:pass' -d '{"token2fa" : "'${token}'"}' https://SERVER_URL/api/management/v1/useradm/auth/login`

(set token if you are using two-factor auth, otherwise do not worry about it and set user and pass only)

and then verify if the JWT looks something like that:

#echo "$jwt"  | cut -f2 -d. | base64 --decode | jq .
base64: invalid input
{
  "jti": "xxxx",
  "sub": "xxxx",
  "exp": xxxx,
  "iat": xxxx,
  "mender.tenant": "xxxx",
  "mender.user": true,
  "iss": "Mender Users",
  "scp": "mender.*",
  "mender.plan": "enterprise",
  "mender.trial": true,
  "mender.addons": [
    {
      "name": "configure",
      "enabled": true
    },
    {
      "name": "troubleshoot",
      "enabled": true
    },
    {
      "name": "monitor",
      "enabled": true
    }
  ],
  "nbf": xxxx
}

best regards,
peter

13

The following is what I got

base64: invalid input
{
  "jti": "3450e615-a61c-4d73-a8e3-31775c6e7e8c",
  "sub": "140336f2-e1ec-494a-97c2-8a1afd0a9b63",
  "exp": 1646338866,
  "iat": 1645734066,
  "mender.user": true,
  "iss": "Mender Users",
  "scp": "mender.*",
  "nbf": 1645734066
}
14

ok, last thing before I give up :slight_smile: could we check:

curl -v -X POST -H "Authorization: Bearer $jwt" -H "Content-Type: application/json;" -d '{"page":1,"per_page":1,"filters":[{"scope":"identity","attribute":"status","type":"$eq","value":"accepted"}],"attributes":[{"scope":"identity","attribute":"status"},{"scope":"inventory","attribute":"artifact_name"},{"scope":"inventory","attribute":"device_type"},{"scope":"inventory","attribute":"rootfs-image.version"},{"scope":"monitor","attribute":"alerts"},{"scope":"system","attribute":"created_ts"},{"scope":"system","attribute":"updated_ts"},{"scope":"tags","attribute":"name"}]}' https://SERVER_URL/api/management/v2/inventory/filters/search

what does that say?

pg

15

here is the response:

Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying xx.xxx.xx.xxx:80...
* Connected to xx.xxx.xx.xxx (xx.xxx.xx.xxx) port 80 (#0)
> POST /api/management/v2/inventory/filters/search HTTP/1.1
> Host: xx.xxx.xx.xxx
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIwNmJiZjkzNC1kMTMyLTRjNWUtOTlmMS0zZWNlZmUwYzZlNzEiLCJzdWIiOiIxNDAzMzZmMi1lMWVjLTQ5NGEtOTdjMi04YTFhZmQwYTliNjMiLCJleHAiOjE2NDYzOTE0MzgsImlhdCI6MTY0NTc4NjYzOCwibWVuZGVyLnVzZXIiOnRydWUsImlzcyI6Ik1lbmRlciBVc2VycyIsInNjcCI6Im1lbmRlci4qIiwibmJmIjoxNjQ1Nzg2NjM4fQ.LTmZBgMhFNYKvAxtfHJSYOZAJYeTYvZ6HgKM-uw1K8oxpPpnyePTx0di5an8rjQiR23pwJZCmwHYSp9fpfw5jNipnnudHjlQ06CP9s4t08sOB9wL2H4VNRYBSPnVA7P5UfmugqqhXpqar4OsF9gQ6ysyHU6dOV2rLSa74LacHyEMiXXdUrH5eYuyXxYNqWS8CiaNQI1n87zsalydNlCz9RgSY6wKIuQzVEZyhkv9PNMMIHY4CAwczWnEMBk4w7rm5F1NN9Rdz9vx9RjRyuvUh7nwD0_s0Odn6k_fJMhZQjtduFdOvYUsGRMeT4-JjH-909Qwv9JizB3um6Jna6yTeYdrqAtmYZWUvdnRk5hwOsPlVHlaFLJIsKs4BLJXZaZw8dg5rHuQB0OlHzoLGTHvvlRSXUwCFLJbGOaLrEJGargqIApS0nYuOaTMC6yqcPmCNnqh_oFwzINOLsdZC-_ZebFUspbjPwhT7Ij54tLgIOMvMHoFpd46euk-augSPq5h
> Content-Type: application/json;
> Content-Length: 486
>
* upload completely sent off: 486 out of 486 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Date: Fri, 25 Feb 2022 10:59:19 GMT
< Referrer-Policy: no-referrer
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Men-Requestid: a5bdf823-1f95-41d9-9831-0b94fbfc0286
< X-Total-Count: 0
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host xx.xxx.xx.xxx left intact
16

ok so it works with inventory, does it work with /me:

curl -v -X GET -H "Authorization: Bearer $jwt" -H "Content-Type: application/json;"  https://SERVER_URL/api/management/v1/useradm/users/me

pg

17

/me seem to work with curl as well, but not from the web-ui

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying xx.xxx.xx.xxx:80...
* Connected to xx.xxx.xx.xxx (xx.xxx.xx.xxx) port 80 (#0)
> GET /api/management/v1/useradm/users/me HTTP/1.1
> Host: xx.xxx.xx.xxx
> User-Agent: curl/7.74.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI3MzNhMDhmMS1jOGY0LTQ4NTktODE3My1iZDM3NTljZTRkZTUiLCJzdWIiOiIxNDAzMzZmMi1lMWVjLTQ5NGEtOTdjMi04YTFhZmQwYTliNjMiLCJleHAiOjE2NDYzOTM1OTgsImlhdCI6MTY0NTc4ODc5OCwibWVuZGVyLnVzZXIiOnRydWUsImlzcyI6Ik1lbmRlciBVc2VycyIsInNjcCI6Im1lbmRlci4qIiwibmJmIjoxNjQ1Nzg4Nzk4fQ.Au83gMLOGXgi5W1PU6eyV23a_CAy84lCOTBHkEUeqERgjGwC2Y91DaSkpKN3vVoEAX4ibKMlen8uaMgCUnHYyg9gWOpp8xLaXiuS4zA115e4hnAsPcD41qKtpg5dHKHMqFR0c77_dPdFtIEWwlhkozdgzCjDA8RrlOvpiROfT0rKAiJjaFopl02oUK0lgvognQK_cblSdEbbzSE31jelaRvy2JhPAxT_sFv693W36R7BDKzT7Nn-t5-kNqgsdJ-udJJgraoKoaEIHypySgzl9IpG-OTNYXfvr19i5s4UQiICFN46AypozdTpaUBMM3cF41U57pX3LAmDUkRVU2lEbGEucJQNHxQ_NI9w8YyKiUk7BAy2RbxEdf20xQFLnW1F9ZzFY9NqRG9mwh525mFMkJuWMGuYZHNjbN1HzXtNix1fIcRVdhC3AMjdTF-YeVLdvVgFA4TaDZwCBc-kGo6DHBDhiinYtEEhXuBKLrBOr0NUOJg2abj-4iMhaRznF3Af
> Content-Type: application/json;
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 192
< Content-Type: application/json; charset=utf-8
< Date: Fri, 25 Feb 2022 11:34:46 GMT
< Referrer-Policy: no-referrer
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Men-Requestid: 46f9e60b-ff9c-4539-a587-8b1d9ef9bc5a
< X-Xss-Protection: 1; mode=block
<
* Connection #0 to host xx.xxx.xx.xxx left intact
{"id":"140336f2-e1ec-494a-97c2-8a1afd0a9b63","email":"mender@example.com","created_ts":"2022-02-15T20:34:05.797Z","updated_ts":"2022-02-15T20:34:05.797Z","login_ts":"2022-02-25T11:33:18.175Z"}
18

ok, we have to open an incognito window in a browser, refresh without cache: ctrl/cmd+r and re-login.

pg

19

I have try with incognito window and cleared cache, but I’m getting the same behavior:
/me 401

20

wow. how can that be?
can we verify if the browser is sending the same POST to /login and GET to /me? can we check the network tab and copy-paste the JWT from the browser, and to the same trick with base64 decoding? also, does the copied-pasted JWT from the browser cookies work with inventory curl above?

peter