Deployment of AKS based mender-server Version 3.6.2 not completely successful

General Discussions
21

The password from the Connection string in mongodb-common matches exactly my value from cofig param mongodb.auth.rootPassword

During the helm upgrade flow, the mongodb-common-prerelease is still having a broken connection string

kubectl get secret mongodb-common-prerelease -o jsonpath='{.data.MONGO}' | base64 -d
mongodb://root:<rootPW>@mender-mongodb-headless

here are the logs of the workflows-server-migration containers:

$ kubectl logs mender-db-data-migration-h498l -c workflows-server-migration
2023/09/08 10:41:09 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
$ kubectl logs mender-db-data-migration-kx4b5 -c workflows-server-migration
2023/09/08 10:40:50 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
$ kubectl logs mender-db-data-migration-7wql4 -c workflows-server-migration
2023/09/08 10:40:40 failed to connect to db: Error reaching mongo server: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.
22

Maybe a special character in the mongodb password? If so, could you please retry a fresh install with a complex password composed by only numbers and letters?

23

not in the secret

secret_contents
secret_contents784×136 3.54 KB

24

so the temporary resource is the mongodb-common-prerelease right? This one is not deleted and has not the same content as mongodb-common.

P.S.

I did run:

kubectl delete mongodb-common-prerelease
helm upgrade mender mender/mender -f mender-3.6.2.yml

This is how mongodb-common-prerelease got created:

\o$ kubectl get secrets mongodb-common-prerelease
NAME                        TYPE     DATA   AGE
mongodb-common-prerelease   Opaque   2      24s
\o$ kubectl get secret mongodb-common-prerelease -o jsonpath='{.data.MONGO}' | base64 -d
mongodb://root:<rootPW>@mender-mongodb-headless
25

I will re-install. I dug into the chart and have seen quite some logic around the mongo uri generation.

I had a request to scale down pods in the cluster and did in fact change the mongodb architecture at one helm revision.

26

Should I keep these properties in my config for fresh installation regarding the NATS issue?

workflows:
  automigrate: false

create_artifact_worker:
  automigrate: false
27

Yes, you can keep the automigrate: false flags.
The mongodb-common-prerelease could be originating from the old setup, then. When doing a fresh install you also have to take care of old volumes, that usually are kept even after a helm delete.

Tldr: for a real fresh install you have to delete all the secrets, pvc, pv, jobs, statefulset resources that could be kept even after an helm delete

1 Like
28

The cluster is cleaned and ready.

Now I get an yaml error when I set
workflows.automigrate=false

\o$ helm template mender mender/mender -f mender-3.6.2.yml --set workflows.automigrate=true > /dev/null
\o$ helm template mender mender/mender -f mender-3.6.2.yml --set workflows.automigrate=false > /dev/null
Error: YAML parse error on mender/templates/create-artifact-worker-deploy.yaml: error converting YAML to JSON: yaml: line 55: mapping values are not allowed in this context

Use --debug flag to render out invalid YAML
29

Checking… Can you please try without the automigrate flags meanwhile?

30

I did run 2 template of the specific file:

helm template mender ./mender -f mender-3.6.2-2.yml -s templates/create-artifact-worker-deploy.yaml --debug --version 5.2.3 --set workflows.automigrate=true > good_create-artifact-worker-deploy.yaml
helm template mender ./mender -f mender-3.6.2-2.yml -s templates/create-artifact-worker-deploy.yaml --debug --version 5.2.3 --set workflows.automigrate=false > cursed_create-artifact-worker-deploy.yaml

diff cursed_create-artifact-worker-deploy.yaml good_create-artifact-worker-deploy.yaml 
57c57,58
<             memory: 128Mcommand: ["workflows", "--config", "/etc/workflows/config.yaml", "worker"]
---
>             memory: 128M
>         command: ["workflows", "--config", "/etc/workflows/config.yaml", "worker", "--automigrate"]

There’s no linebreak after memory: 128M :face_with_spiral_eyes:

31

The clean setup even without automigrate flags was successful. I can’t tell what the difference was to the previous installation. Maybe some special character as I’m not working 100% with linux :frowning:

Thanks for the quick and excellent support!

kubectl get pods
NAME                                             READY   STATUS    RESTARTS        AGE
mender-api-gateway-7b75c59bdd-bpsf8              1/1     Running   0               8m44s
mender-create-artifact-worker-6b9f44d949-wd4wz   1/1     Running   3 (8m3s ago)    8m44s
mender-deployments-6f6764995c-pk28g              1/1     Running   0               8m44s
mender-device-auth-989c8bd96-jdzvk               1/1     Running   0               8m44s
mender-deviceconfig-67b9779b8d-qhhl8             1/1     Running   0               8m44s
mender-deviceconnect-5b67d847f-58qtx             1/1     Running   0               8m44s
mender-gui-5d999776c4-sr22k                      1/1     Running   0               8m44s
mender-inventory-8899485fb-4c85p                 1/1     Running   0               8m44s
mender-iot-manager-587ff944d-sshmr               1/1     Running   0               8m44s
mender-mongodb-0                                 1/1     Running   0               9m42s
mender-mongodb-1                                 1/1     Running   0               9m
mender-mongodb-arbiter-0                         1/1     Running   0               9m41s
mender-nats-0                                    3/3     Running   0               8m44s
mender-nats-1                                    3/3     Running   0               8m44s
mender-nats-2                                    3/3     Running   0               8m44s
mender-nats-box-7bbc67486c-hzrcm                 1/1     Running   0               8m44s
mender-redis-master-0                            1/1     Running   0               9m39s
mender-redis-replicas-0                          1/1     Running   0               9m38s
mender-redis-replicas-1                          1/1     Running   0               8m52s
mender-redis-replicas-2                          1/1     Running   0               8m27s
mender-useradm-7d9d56b6fd-nmn5z                  1/1     Running   0               8m44s
mender-workflows-server-5cc4b8b965-j78lz         1/1     Running   2 (8m21s ago)   8m44s
mender-workflows-worker-859b66449-zgzbt          1/1     Running   3 (8m7s ago)    8m44s
1 Like
32

Thanks for the debug! FYI, here’s a PR for the fix you reported: fix: automigrate command newline fix by oldgiova · Pull Request #196 · mendersoftware/mender-helm · GitHub

Have a great weekend.