Deployment of AKS based mender-server Version 3.6.2 not completely successful

General Discussions
21

The password from the Connection string in mongodb-common matches exactly my value from cofig param mongodb.auth.rootPassword

During the helm upgrade flow, the mongodb-common-prerelease is still having a broken connection string

kubectl get secret mongodb-common-prerelease -o jsonpath='{.data.MONGO}' | base64 -d
mongodb://root:<rootPW>@mender-mongodb-headless

here are the logs of the workflows-server-migration containers:

$ kubectl logs mender-db-data-migration-h498l -c workflows-server-migration
2023/09/08 10:41:09 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
$ kubectl logs mender-db-data-migration-kx4b5 -c workflows-server-migration
2023/09/08 10:40:50 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
$ kubectl logs mender-db-data-migration-7wql4 -c workflows-server-migration
2023/09/08 10:40:40 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
22

Maybe a special character in the mongodb password? If so, could you please retry a fresh install with a complex password composed by only numbers and letters?

23

not in the secret

secret_contents
secret_contents784×136 3.54 KB

24

so the temporary resource is the mongodb-common-prerelease right? This one is not deleted and has not the same content as mongodb-common.

P.S.

I did run:

kubectl delete mongodb-common-prerelease
helm upgrade mender mender/mender -f mender-3.6.2.yml

This is how mongodb-common-prerelease got created:

\o$ kubectl get secrets mongodb-common-prerelease
NAME                        TYPE     DATA   AGE
mongodb-common-prerelease   Opaque   2      24s
\o$ kubectl get secret mongodb-common-prerelease -o jsonpath='{.data.MONGO}' | base64 -d
mongodb://root:<rootPW>@mender-mongodb-headless
25

I will re-install. I dug into the chart and have seen quite some logic around the mongo uri generation.

I had a request to scale down pods in the cluster and did in fact change the mongodb architecture at one helm revision.

26

Should I keep these properties in my config for fresh installation regarding the NATS issue?

workflows:
  automigrate: false

create_artifact_worker:
  automigrate: false
27

Yes, you can keep the automigrate: false flags.
The mongodb-common-prerelease could be originating from the old setup, then. When doing a fresh install you also have to take care of old volumes, that usually are kept even after a helm delete.

Tldr: for a real fresh install you have to delete all the secrets, pvc, pv, jobs, statefulset resources that could be kept even after an helm delete

28

The cluster is cleaned and ready.

Now I get an yaml error when I set
workflows.automigrate=false

\o$ helm template mender mender/mender -f mender-3.6.2.yml --set workflows.automigrate=true > /dev/null
\o$ helm template mender mender/mender -f mender-3.6.2.yml --set workflows.automigrate=false > /dev/null
Error: YAML parse error on mender/templates/create-artifact-worker-deploy.yaml: error converting YAML to JSON: yaml: line 55: mapping values are not allowed in this context

Use --debug flag to render out invalid YAML
29

Checking… Can you please try without the automigrate flags meanwhile?

30

I did run 2 template of the specific file:

helm template mender ./mender -f mender-3.6.2-2.yml -s templates/create-artifact-worker-deploy.yaml --debug --version 5.2.3 --set workflows.automigrate=true > good_create-artifact-worker-deploy.yaml
helm template mender ./mender -f mender-3.6.2-2.yml -s templates/create-artifact-worker-deploy.yaml --debug --version 5.2.3 --set workflows.automigrate=false > cursed_create-artifact-worker-deploy.yaml

diff cursed_create-artifact-worker-deploy.yaml good_create-artifact-worker-deploy.yaml 
57c57,58
<             memory: 128Mcommand: ["workflows", "--config", "/etc/workflows/config.yaml", "worker"]
---
>             memory: 128M
>         command: ["workflows", "--config", "/etc/workflows/config.yaml", "worker", "--automigrate"]

There’s no linebreak after memory: 128M :face_with_spiral_eyes:

31

The clean setup even without automigrate flags was successful. I can’t tell what the difference was to the previous installation. Maybe some special character as I’m not working 100% with linux :frowning:

Thanks for the quick and excellent support!

kubectl get pods
NAME                                             READY   STATUS    RESTARTS        AGE
mender-api-gateway-7b75c59bdd-bpsf8              1/1     Running   0               8m44s
mender-create-artifact-worker-6b9f44d949-wd4wz   1/1     Running   3 (8m3s ago)    8m44s
mender-deployments-6f6764995c-pk28g              1/1     Running   0               8m44s
mender-device-auth-989c8bd96-jdzvk               1/1     Running   0               8m44s
mender-deviceconfig-67b9779b8d-qhhl8             1/1     Running   0               8m44s
mender-deviceconnect-5b67d847f-58qtx             1/1     Running   0               8m44s
mender-gui-5d999776c4-sr22k                      1/1     Running   0               8m44s
mender-inventory-8899485fb-4c85p                 1/1     Running   0               8m44s
mender-iot-manager-587ff944d-sshmr               1/1     Running   0               8m44s
mender-mongodb-0                                 1/1     Running   0               9m42s
mender-mongodb-1                                 1/1     Running   0               9m
mender-mongodb-arbiter-0                         1/1     Running   0               9m41s
mender-nats-0                                    3/3     Running   0               8m44s
mender-nats-1                                    3/3     Running   0               8m44s
mender-nats-2                                    3/3     Running   0               8m44s
mender-nats-box-7bbc67486c-hzrcm                 1/1     Running   0               8m44s
mender-redis-master-0                            1/1     Running   0               9m39s
mender-redis-replicas-0                          1/1     Running   0               9m38s
mender-redis-replicas-1                          1/1     Running   0               8m52s
mender-redis-replicas-2                          1/1     Running   0               8m27s
mender-useradm-7d9d56b6fd-nmn5z                  1/1     Running   0               8m44s
mender-workflows-server-5cc4b8b965-j78lz         1/1     Running   2 (8m21s ago)   8m44s
mender-workflows-worker-859b66449-zgzbt          1/1     Running   3 (8m7s ago)    8m44s
32

Thanks for the debug! FYI, here’s a PR for the fix you reported: fix: automigrate command newline fix by oldgiova · Pull Request #196 · mendersoftware/mender-helm · GitHub

Have a great weekend.