# Kubernetes setup fail. Secret Mongodb-common no found

**URL:** <https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660>\
**Category:** General Discussions\
**Created:** [March 8, 2023, 4:56pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660 "2023-03-08T16:56:43Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![rlaybourn](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rlaybourn](https://hub.mender.io/u/rlaybourn)\
**Post date:** [March 8, 2023, 4:56pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/1 "2023-03-08T16:56:43Z")

</div>

I have been trying to setup the openrsource mender server according to the instructions for version 3.5. As mongodb v11.2.0 is now not available I have used version 12.1.31 as is on the readme on the mender-helm github. Setup all appears to go fine but when i attempt to install mender 3.5.0 the helm command just hangs. Looking with kubectl I can see a pod “index-reporting” in status “CreateContainerConfigError”. Inside its description the error reported is

```auto
  Normal Scheduled 76s default-scheduler Successfully assigned default/index-reporting-2s9g6 to pool-8ygyp6lie-q6uq2
  Warning Failed 22s (x6 over 75s) kubelet Error: secret "mongodb-common" not found
  Normal Pulled 22s (x6 over 75s) kubelet Container image "docker.io/mendersoftware/deployments:mender-3.5.0" already present on machine
  Warning Failed 22s (x6 over 75s) kubelet Error: secret "mongodb-common" not found
  Normal Pulled 7s (x7 over 75s) kubelet Container image "docker.io/mendersoftware/deviceauth:mender-3.5.0" already present on machine

```

the description is refering to the image “[docker.io/mendersoftware/deviceauth:mender-3.5.0](http://docker.io/mendersoftware/deviceauth:mender-3.5.0)”.  
I have tried a few versions of mongodb which no effect. I am using credentials for an s3 bucket on aws so have not installed minio but I can no see how that could effect it.  
the mender-3.5.0.yml file , produced as per the instructions is shown bellow

```auto
global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:<rootpass>@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  s3:
    AWS_URI: "https://<bucket-name>.s3.eu-west-1.amazonaws.com"
    AWS_BUCKET: "<bucket-name>"
    AWS_REGION: "eu-west-1"
    AWS_ACCESS_KEY_ID: "<keyid>"
    AWS_SECRET_ACCESS_KEY: "<access key>"
    AWS_FORCE_PATH_STYLE: "false"
  url: "https://mender.example.com"

api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
              <KEY>
useradm:
  certs:
    key: | 
            <KEY>

```

Any ideas or suggestions how to proceed would be much appreciated

---

<div class="post-metadata">

**Author:** ![rlaybourn](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rlaybourn](https://hub.mender.io/u/rlaybourn)\
**Post date:** [March 8, 2023, 7:34pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/2 "2023-03-08T19:34:03Z")

</div>

As an update to this I found the job “index-reporting” runs before the template that creates the “mongodb-common” secret. I do not know enough about kubernetes and helm to know why this is however I am working around this by creating the secret myself first and modifying the helm package to remove the template that creates that secret. This is not really a fix though

---

<div class="post-metadata">

**Author:** ![rlaybourn](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rlaybourn](https://hub.mender.io/u/rlaybourn)\
**Post date:** [March 9, 2023, 1:33pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/3 "2023-03-09T13:33:24Z")

</div>

The hack mentioned in the previous post got a server up and running only just. I suspect there may be some issue with the database as I now have 3 pods that are constantly crashing . “create-artifact-worker”,“workflows-server” and “workflows-worker”. The logs from each of these 3 services is identical

```auto
time="2023-03-09T13:29:01Z" level=info msg="migrating workflows" file=migrations.go func=mongo.Migrate line=38
time="2023-03-09T13:29:01Z" level=info msg="migration to version 1.0.0 skipped" db=workflows file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-09T13:29:01Z" level=info msg="DB migrated to version 1.0.0" db=workflows file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
2023/03/09 13:29:06 context deadline exceeded

```

The interface is active and I can log in but without these services running the page javascript is reporting 502 errors and an experimental upload of an artifact just returns a http 413 (too large).

---

<div class="post-metadata">

**Author:** ![peter](https://avatars.discourse-cdn.com/v4/letter/p/278dde/32.png) [@peter](https://hub.mender.io/u/peter)\
**Post date:** [March 10, 2023, 9:14am UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/4 "2023-03-10T09:14:35Z")

</div>

Hello @rlaybourn

thank you for the report, and for using Mender. we will take a close look at it.  
in the meantime could you try to check it you have nats up and running? you should see something like:

```auto
# kubectl get pods | grep nats
nats-0 3/3 Running 12 (4m19s ago) 14d
nats-box-845d4d68f5-w4g97 1/1 Running 4 (4m19s ago) 14d

```

if not you can use the following to install and deploy it:

```auto
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update
helm install nats nats/nats --version 0.15.1 --set "nats.image=nats:2.7.4-alpine" --set "nats.jetstream.enabled=true"

```

in case you already have it, you can remove it with `helm uninstall nats` and repeating the install command. after nats is up and running just delete the pods that are misbehaving (`kubectl delete pod pod_name_from_kubectl_get_pods`)

best regards,  
peter

---

<div class="post-metadata">

**Author:** ![rlaybourn](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rlaybourn](https://hub.mender.io/u/rlaybourn)\
**Post date:** [March 10, 2023, 9:41am UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/5 "2023-03-10T09:41:40Z")

</div>

Thanks for the reponse I found a solution to the problem in this other ticket [Mender on K8S problems](https://hub.mender.io/t/mender-on-k8s-problems/4746). I needed to add the correct storage type in the nats.yml for the cluster i am using . In this case I am testing setting up the server using minikube. (in this case I needed storageClassName: “standard”). I now have a server mostly running but I am currently unable to upload any artifacts to this test . The error reported back from the interface when i try is and error 413

---

<div class="post-metadata">

**Author:** ![rlaybourn](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rlaybourn](https://hub.mender.io/u/rlaybourn)\
**Post date:** [March 10, 2023, 11:02am UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/6 "2023-03-10T11:02:32Z")

</div>

Ok the 413 size error was from me misconfiguring my ingress

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 1, 2023, 1:17am UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/7 "2023-04-01T01:17:02Z")

</div>

I am facing the same problem. In order to get past the `index-reporting-abdef` `CreateContainerConfigError`, I rendered the `secret-mongodb-common.yaml` locally:

```auto
helm template secret-mongodb-common.yaml ./mender-3.5.1.tgz -f mender-values.yaml > secret-mongodb.yaml

```

and then applied it:

```auto
kubectl apply -f secret-mongodb.yaml

```

> UPDATE: Not unexpectedly, a bunch of the pods are now in `CrashLoopBackOff` state, but that’s for another day.

But, given that I am trying to do the installation fully automated with Terraform, this is rather annoying.

I’ll also note that the 3.5.1 helm chart has not been published yet and the 3.5 docs still refer to the obsolete 11.2.0 mongodb helm chart.

---

<div class="post-metadata">

**Author:** ![mister\_kanister](https://avatars.discourse-cdn.com/v4/letter/m/8edcca/32.png) [@mister\_kanister](https://hub.mender.io/u/mister_kanister)\
**Post date:** [April 3, 2023, 9:18am UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/8 "2023-04-03T09:18:24Z")

</div>

I adjusted the versions of the charts of various services and solved that problem by using **mender 3.4.** However I’m facing now [another problem](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724) where `mender-deployments` in in crash loopback. To be honest the documentation is a mess. I tried several mender versions starting from 3.1.

Here are the commands I used that. It’s not a full-auto install script.

```auto
curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --data-dir /mnt
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Install Helm
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

# Install Cert-Manager
export CERT_MANAGER_CHART_VERSION="v1.10.0"
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager \
  --version $CERT_MANAGER_CHART_VERSION \
  --set installCRDs=true \
  --set namespace=kube-system

# Provide ClusterIssuer
export LETSENCRYPT_SERVER_URL="https://acme-v02.api.letsencrypt.org/directory"
export LETSENCRYPT_EMAIL="<email>"

cat >issuer-letsencrypt.yml <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: ${LETSENCRYPT_SERVER_URL}
    email: ${LETSENCRYPT_EMAIL}
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress: {}
EOF

# Wait until cert-manager is ready
kubectl apply -f issuer-letsencrypt.yml

# Install MongoDB
# Install password generator
sudo apt install pwgen

export MONGODB_ROOT_PASSWORD=$(pwgen 32 1)
export MONGODB_REPLICA_SET_KEY=$(pwgen 32 1)
export MONGODB_CHART_VERSION="12.1.31"
export MONGODB_TAG="5.0.10-debian-11-r7"

cat >mongodb.yml <<EOF
architecture: "replicaset"
replicaCount: 1
arbiter:
  enabled: true
auth:
  rootPassword: ${MONGODB_ROOT_PASSWORD}
  replicaSetKey: ${MONGODB_REPLICA_SET_KEY}
readinessProbe:
  timeoutSeconds: 20
image:
  tag: "${MONGODB_TAG}"
persistence:
  size: "4Gi"
EOF

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm upgrade --install mongodb bitnami/mongodb --version $MONGODB_CHART_VERSION -f mongodb.yml 

# Get connection string
echo mongodb://root:${MONGODB_ROOT_PASSWORD}@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017

# Install NATS, message broker
export NATS_IMAGE="nats:2.7.4-alpine"
export NATS_CHART_VERSION="0.15.1"

cat >nats.yml <<EOF
cluster:
  enabled: true
  replicas: 2
nats:
  image: "${NATS_IMAGE}"
  jetstream:
    enabled: true
    memStorage:
      enabled: true
      size: "1Gi"

    fileStorage:
      enabled: true
      size: "2Gi"
      storageDirectory: /data/
EOF

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update
helm install nats nats/nats --version $NATS_CHART_VERSION -f nats.yml

# Install MinIO, storage interface provider
export MINIO_TAG="RELEASE.2021-06-17T00-10-46Z"
export MINIO_CHART_VERSION="4.1.7"
export MINIO_ACCESS_KEY=$(pwgen 32 1)
export MINIO_SECRET_KEY=$(pwgen 32 1)

cat >minio-operator.yml <<EOF
tenants: {}
EOF

helm repo add minio https://operator.min.io/
helm repo update
helm install minio-operator minio/minio-operator --version $MINIO_CHART_VERSION -f minio-operator.yml

cat >minio.yml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: minio-creds-secret
type: Opaque
data:
  accesskey: $(echo -n $MINIO_ACCESS_KEY | base64)
  secretkey: $(echo -n $MINIO_SECRET_KEY | base64)
---
apiVersion: minio.min.io/v2
kind: Tenant
metadata:
  name: minio
  labels:
    app: minio
spec:
  image: minio/minio:${MINIO_TAG}
  credsSecret:
    name: minio-creds-secret
  pools:
    - servers: 2
      volumesPerServer: 2
      volumeClaimTemplate:
        metadata:
          name: data
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: "local-path"
  mountPath: /export
  requestAutoCert: false
EOF

kubectl apply -f minio.yml

# Create ingress for minio where artifacts will be uploaded
export MINIO_DOMAIN_NAME="<domain>"

cat >minio-ingress.yml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minio-ingress
  annotations:
    cert-manager.io/issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - ${MINIO_DOMAIN_NAME}
    secretName: minio-ingress-tls
  rules:
  - host: "${MINIO_DOMAIN_NAME}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: minio
            port:
              number: 80
EOF

kubectl apply -f minio-ingress.yml

# Mender deployment
# Need two keys. For device authentication and user generation
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out device_auth.key
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 | openssl rsa -out useradm.key

helm repo add mender https://charts.mender.io
helm repo update

export MENDER_SERVER_DOMAIN="<domain>"
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
export MENDER_VERSION="3.4.0"

cat >mender-${MENDER_VERSION}.yml <<EOF
global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:${MONGODB_ROOT_PASSWORD}@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  url: "${MENDER_SERVER_URL}"

api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
$(cat device_auth.key | sed -e 's/^/ /g')

useradm:
  certs:
    key: |-
$(cat useradm.key | sed -e 's/^/ /g')
EOF

helm upgrade --install mender mender/mender --version $MENDER_VERSION -f mender-$MENDER_VERSION.yml

cat >mender-ingress.yml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mender-ingress
  annotations:
    cert-manager.io/issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - ${MENDER_SERVER_DOMAIN}
    secretName: mender-ingress-tls
  rules:
  - host: "${MENDER_SERVER_DOMAIN}"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mender-api-gateway
            port:
              number: 80
EOF

kubectl apply -f mender-ingress.yml

# Create a new user
export USER_EMAIL="<mail>"
export USER_PASSWORD="<password>"
USERADM_POD=$(kubectl get pod -l 'app.kubernetes.io/name=useradm' -o name | head -1)
kubectl exec $USERADM_POD -- useradm create-user --username "${USER_EMAIL}" --password "${USER_PASSWORD}"

```

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 5, 2023, 10:06pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/9 "2023-04-05T22:06:09Z")

</div>

I was able to get the helm chart to at least install (still `CrashLoopBackOff` issues) with the following:

```diff
diff --git a/mender/values.yaml b/mender/values.yaml
index bcb805f..12cc2ab 100644
--- a/mender/values.yaml
+++ b/mender/values.yaml
@@ -39,7 +39,7 @@ global:
 
 dataMigration:
   reindexReporting:
- enable: true
+ enable: false
     annotations: {}
     backoffLimit: 5
     affinity: {}

```

And then:

```auto
make package
helm install mender ./mender-3.5.1.tgz --namespace mender -f mender-values.yaml

```

---

<div class="post-metadata">

**Author:** ![robgio](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/robgio/32/1760_2.png) [@robgio](https://hub.mender.io/u/robgio)\
**Post date:** [April 6, 2023, 1:27pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/10 "2023-04-06T13:27:06Z")

</div>

Hi @moto-timo ,  
thanks for your report. We updated today the Mender Helm Chart v3.5.0 with the fix for the secret creation.

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 6, 2023, 3:36pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/11 "2023-04-06T15:36:27Z")

</div>

@mister_kanister thank you for sharing your complete steps.

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 6, 2023, 11:38pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/12 "2023-04-06T23:38:17Z")

</div>

The `minio-operator` has disappeared:

```shell
$ helm install \
  --namespace minio-operator \
  --create-namespace \
  minio-operator minio/operator
Error: INSTALLATION FAILED: chart "operator" matching not found in minio index. (try 'helm repo update'): no chart name found

```

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 19, 2023, 6:05pm UTC](https://hub.mender.io/t/kubernetes-setup-fail-secret-mongodb-common-no-found/5660/13 "2023-04-19T18:05:15Z")

</div>

If you want to install the `4.1.7` version of the `minio-operator` helm chart as shown in the [documentation](https://docs.mender.io/3.5/server-installation/production-installation-with-kubernetes/minio), you can use:

```auto
$ helm repo add minio-github https://raw.githubusercontent.com/minio/operator/v4.1.3/
$ helm search repo minio-github
NAME CHART VERSION APP VERSION DESCRIPTION                    
minio-github/minio-operator 4.1.7 v4.1.3 A Helm chart for MinIO Operator

```