Kubernetes setup fail. Secret Mongodb-common no found

General Discussions
1

I have been trying to setup the openrsource mender server according to the instructions for version 3.5. As mongodb v11.2.0 is now not available I have used version 12.1.31 as is on the readme on the mender-helm github. Setup all appears to go fine but when i attempt to install mender 3.5.0 the helm command just hangs. Looking with kubectl I can see a pod “index-reporting” in status “CreateContainerConfigError”. Inside its description the error reported is

  Normal   Scheduled  76s                default-scheduler  Successfully assigned default/index-reporting-2s9g6 to pool-8ygyp6lie-q6uq2
  Warning  Failed     22s (x6 over 75s)  kubelet            Error: secret "mongodb-common" not found
  Normal   Pulled     22s (x6 over 75s)  kubelet            Container image "docker.io/mendersoftware/deployments:mender-3.5.0" already present on machine
  Warning  Failed     22s (x6 over 75s)  kubelet            Error: secret "mongodb-common" not found
  Normal   Pulled     7s (x7 over 75s)   kubelet            Container image "docker.io/mendersoftware/deviceauth:mender-3.5.0" already present on machine

the description is refering to the image “docker.io/mendersoftware/deviceauth:mender-3.5.0”.
I have tried a few versions of mongodb which no effect. I am using credentials for an s3 bucket on aws so have not installed minio but I can no see how that could effect it.
the mender-3.5.0.yml file , produced as per the instructions is shown bellow

global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:<rootpass>@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  s3:
    AWS_URI: "https://<bucket-name>.s3.eu-west-1.amazonaws.com"
    AWS_BUCKET: "<bucket-name>"
    AWS_REGION: "eu-west-1"
    AWS_ACCESS_KEY_ID: "<keyid>"
    AWS_SECRET_ACCESS_KEY: "<access key>"
    AWS_FORCE_PATH_STYLE: "false"
  url: "https://mender.example.com"

api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
              <KEY>
useradm:
  certs:
    key: | 
            <KEY>

Any ideas or suggestions how to proceed would be much appreciated

1 Like
2

As an update to this I found the job “index-reporting” runs before the template that creates the “mongodb-common” secret. I do not know enough about kubernetes and helm to know why this is however I am working around this by creating the secret myself first and modifying the helm package to remove the template that creates that secret. This is not really a fix though

3

The hack mentioned in the previous post got a server up and running only just. I suspect there may be some issue with the database as I now have 3 pods that are constantly crashing . “create-artifact-worker”,“workflows-server” and “workflows-worker”. The logs from each of these 3 services is identical

time="2023-03-09T13:29:01Z" level=info msg="migrating workflows" file=migrations.go func=mongo.Migrate line=38
time="2023-03-09T13:29:01Z" level=info msg="migration to version 1.0.0 skipped" db=workflows file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-09T13:29:01Z" level=info msg="DB migrated to version 1.0.0" db=workflows file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
2023/03/09 13:29:06 context deadline exceeded

The interface is active and I can log in but without these services running the page javascript is reporting 502 errors and an experimental upload of an artifact just returns a http 413 (too large).

4

Hello @rlaybourn

thank you for the report, and for using Mender. we will take a close look at it.
in the meantime could you try to check it you have nats up and running? you should see something like:

# kubectl get pods | grep nats
nats-0                                     3/3     Running            12 (4m19s ago)   14d
nats-box-845d4d68f5-w4g97                  1/1     Running            4 (4m19s ago)    14d

if not you can use the following to install and deploy it:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update
helm install nats nats/nats --version 0.15.1 --set "nats.image=nats:2.7.4-alpine" --set "nats.jetstream.enabled=true"

in case you already have it, you can remove it with helm uninstall nats and repeating the install command. after nats is up and running just delete the pods that are misbehaving (kubectl delete pod pod_name_from_kubectl_get_pods)

best regards,
peter

5

Thanks for the reponse I found a solution to the problem in this other ticket Mender on K8S problems. I needed to add the correct storage type in the nats.yml for the cluster i am using . In this case I am testing setting up the server using minikube. (in this case I needed storageClassName: “standard”). I now have a server mostly running but I am currently unable to upload any artifacts to this test . The error reported back from the interface when i try is and error 413

6

Ok the 413 size error was from me misconfiguring my ingress

7

I am facing the same problem. In order to get past the index-reporting-abdef CreateContainerConfigError, I rendered the secret-mongodb-common.yaml locally:

helm template secret-mongodb-common.yaml ./mender-3.5.1.tgz -f mender-values.yaml > secret-mongodb.yaml

and then applied it:

kubectl apply -f secret-mongodb.yaml

UPDATE: Not unexpectedly, a bunch of the pods are now in CrashLoopBackOff state, but that’s for another day.

But, given that I am trying to do the installation fully automated with Terraform, this is rather annoying.

I’ll also note that the 3.5.1 helm chart has not been published yet and the 3.5 docs still refer to the obsolete 11.2.0 mongodb helm chart.