Open Source Mender Server Tutorial

I am working on getting the open source mender tutorial up and running but I keep running into errors. Mender Server | Mender documentation

I have started a MinIO pod, which seems to be up and running just fine:

sudo kubectl get all
NAME        READY   STATUS    RESTARTS   AGE
pod/minio   1/1     Running   0          6s

I have set up an access key, created a bucket (“test-bucket”), and also set environment variable MINIO_DOMAIN=minio.example.net

Next, I followed the instructions in the link above to set up the mender helm chart. Please see my helm chart below, with tokens and secrets commented out

global:
  enterprise: false
  image:
    tag: mender-3.7.8
  mongodb:
    URL: ""
  nats:
    URL: ""
  s3:
    AWS_URI: "https://mender.example.com"
    AWS_BUCKET: test-bucket
    AWS_ACCESS_KEY_ID: <removed for security reasons>
    AWS_SECRET_ACCESS_KEY: <removed for security reasons>
  url: "https://mender.example.com"

# This enables bitnami/mongodb sub-chart
mongodb:
  enabled: true
  auth:
    enabled: true
    rootPassword: <removed for security reasons>
    replicaSetKey: <removed for security reasons>

# This enabled nats sub-chart
nats:
  enabled: true

api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
      -----BEGIN PRIVATE KEY-----
      <removed for security reasons>
      -----END PRIVATE KEY-----

useradm:
  certs:
    key: |-
      -----BEGIN PRIVATE KEY-----
      <removed for security reasons>
      -----END PRIVATE KEY-----
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  path: /
  ingressClassName: nginx
  hosts:
    - mender.example.com
  tls:
  # this secret must exists or it can be created from a working cert-manager instance
   - secretName: mender-ingress-tls
     hosts:
       - mender.example.com

When I do the command below via the tutorial, I get the errors below related to nats, and I never see the nats pods listed when I do “sudo kubectl get all”

helm upgrade --install mender mender/mender  --version 5.11.2 -f mender-3.7.8.yml 

Warnings about nats:

Release "mender" does not exist. Installing it now.
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])
coalesce.go:286: warning: cannot overwrite table with non table for mender.nats.nats.image (map[pullPolicy:IfNotPresent repository:nats tag:2.9.20-alpine])

Error

sudo kubectl get all
NAME                                     READY   STATUS    RESTARTS   AGE
pod/mender-deployments-migration-5cjxq   0/1     Error     0          33s
pod/mender-mongodb-0                     1/1     Running   0          17s
pod/mender-mongodb-1                     0/1     Running   0          5s
pod/mender-redis-master-0                1/1     Running   0          34s
pod/mender-redis-replicas-0              1/1     Running   0          34s
pod/mender-redis-replicas-1              0/1     Running   0          3s
pod/minio                                1/1     Running   0          45s
sudo kubectl logs pod/mender-deployments-migration-5cjxq 
time="2025-01-22T19:43:53Z" level=warning msg="'presign.secret' not configured. Generating a random secret." caller="config.Setup@config.go:246"
failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: mender-mongodb-0.mender-mongodb-headless.default.svc.cluster.local:27017, Type: RSSecondary, Average RTT: 672594 }, { Addr: mender-mongodb-1.mender-mongodb-headless.default.svc.cluster.local:27017, Type: RSSecondary, Average RTT: 1089670 }, { Addr: mender-mongodb-arbiter-0.mender-mongodb-arbiter-headless.default.svc.cluster.local:27017, Type: Unknown, Last error: dial tcp: lookup mender-mongodb-arbiter-0.mender-mongodb-arbiter-headless.default.svc.cluster.local on 10.43.0.10:53: no such host }, ] }

Any and all help would be really appreciated, thank you!

Hello @mld
it looks like both MongoDB and Redis are not completely running; could you please check the logs for the MongoDB statefulset? In particular the pod which is not ready.

Thanks

NAME                                                 READY   STATUS             RESTARTS         AGE
pod/mender-api-gateway-75f8dff5b5-p9q9w              1/1     Running            0                28m
pod/mender-create-artifact-worker-7cf6d444d7-rmj5x   1/1     Running            0                28m
pod/mender-deployments-577bfdf64d-wwpcx              0/1     CrashLoopBackOff   10 (2m31s ago)   28m
pod/mender-deployments-86c987878-j7gpk               0/1     Running            0                3s
pod/mender-device-auth-78bcdd687-hpzr4               1/1     Running            0                28m
pod/mender-deviceconfig-66967cd7c7-gjvxd             1/1     Running            0                28m
pod/mender-deviceconnect-559f4496f4-7m9bb            1/1     Running            0                28m
pod/mender-gui-7944884849-pqmpl                      1/1     Running            0                28m
pod/mender-inventory-6f7df89889-7kwjq                1/1     Running            0                28m
pod/mender-iot-manager-7cc67d98bc-nswgc              1/1     Running            0                28m
pod/mender-redis-master-0                            1/1     Running            0                30m
pod/mender-redis-replicas-0                          1/1     Running            0                30m
pod/mender-redis-replicas-1                          1/1     Running            0                29m
pod/mender-redis-replicas-2                          1/1     Running            0                29m
pod/mender-useradm-7775ccd87f-jqdkm                  1/1     Running            0                28m
pod/mender-workflows-server-6494d5c5f9-zhhdj         1/1     Running            0                28m
pod/mender-workflows-worker-765b5bc789-tcdpf         1/1     Running            0                28m
pod/mongodb-0                                        1/1     Running            0                34m
pod/mongodb-arbiter-0                                1/1     Running            0                34m
pod/nats-0                                           3/3     Running            0                33m
pod/nats-1                                           3/3     Running            0                33m
pod/nats-box-5fb7f4979f-z5d2s                        1/1     Running            0                33m
pod/seaweedfs-filer-0                                1/1     Running            0                78m
pod/seaweedfs-master-0                               1/1     Running            0                78m
pod/seaweedfs-s3-57ffbb7694-78hjw                    1/1     Running            0                78m
pod/seaweedfs-volume-0                               1/1     Running            0                78m

NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                                 AGE
service/external-nginx             ExternalName   <none>          <host_ip_address_running_nginx_container>   <none>                                                  86m
service/kubernetes                 ClusterIP      10.43.0.1       <none>          443/TCP                                                 123m
service/mender-api-gateway         ClusterIP      10.43.207.141   <none>          80/TCP                                                  28m
service/mender-deployments         ClusterIP      10.43.149.87    <none>          8080/TCP                                                28m
service/mender-device-auth         ClusterIP      10.43.113.82    <none>          8080/TCP                                                28m
service/mender-deviceconfig        ClusterIP      10.43.94.65     <none>          8080/TCP                                                28m
service/mender-deviceconnect       ClusterIP      10.43.248.232   <none>          8080/TCP                                                28m
service/mender-gui                 ClusterIP      10.43.215.185   <none>          80/TCP,8080/TCP                                         28m
service/mender-inventory           ClusterIP      10.43.234.50    <none>          8080/TCP                                                28m
service/mender-iot-manager         ClusterIP      10.43.29.206    <none>          8080/TCP                                                28m
service/mender-redis-headless      ClusterIP      None            <none>          6379/TCP                                                30m
service/mender-redis-master        ClusterIP      10.43.70.37     <none>          6379/TCP                                                30m
service/mender-redis-replicas      ClusterIP      10.43.39.5      <none>          6379/TCP                                                30m
service/mender-useradm             ClusterIP      10.43.150.1     <none>          8080/TCP                                                28m
service/mender-workflows-server    ClusterIP      10.43.144.29    <none>          8080/TCP                                                28m
service/mongodb-arbiter-headless   ClusterIP      None            <none>          27017/TCP                                               34m
service/mongodb-headless           ClusterIP      None            <none>          27017/TCP                                               34m
service/nats                       ClusterIP      None            <none>          4222/TCP,6222/TCP,8222/TCP,7777/TCP,7422/TCP,7522/TCP   33m
service/seaweedfs-filer            ClusterIP      None            <none>          8888/TCP,18888/TCP,8333/TCP,9327/TCP                    78m
service/seaweedfs-filer-client     ClusterIP      None            <none>          8888/TCP,18888/TCP,9327/TCP                             78m
service/seaweedfs-master           ClusterIP      None            <none>          9333/TCP,19333/TCP,9327/TCP                             78m
service/seaweedfs-s3               ClusterIP      10.43.110.67    <none>          8333/TCP,9327/TCP                                       78m
service/seaweedfs-volume           ClusterIP      None            <none>          8080/TCP,18080/TCP,9327/TCP                             78m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mender-api-gateway              1/1     1            1           28m
deployment.apps/mender-create-artifact-worker   1/1     1            1           28m
deployment.apps/mender-deployments              0/1     1            0           28m
deployment.apps/mender-device-auth              1/1     1            1           28m
deployment.apps/mender-deviceconfig             1/1     1            1           28m
deployment.apps/mender-deviceconnect            1/1     1            1           28m
deployment.apps/mender-gui                      1/1     1            1           28m
deployment.apps/mender-inventory                1/1     1            1           28m
deployment.apps/mender-iot-manager              1/1     1            1           28m
deployment.apps/mender-useradm                  1/1     1            1           28m
deployment.apps/mender-workflows-server         1/1     1            1           28m
deployment.apps/mender-workflows-worker         1/1     1            1           28m
deployment.apps/nats-box                        1/1     1            1           33m
deployment.apps/seaweedfs-s3                    1/1     1            1           78m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/mender-api-gateway-75f8dff5b5              1         1         1       28m
replicaset.apps/mender-create-artifact-worker-7cf6d444d7   1         1         1       28m
replicaset.apps/mender-deployments-577bfdf64d              1         1         0       28m
replicaset.apps/mender-deployments-86c987878               1         1         0       2m4s
replicaset.apps/mender-device-auth-78bcdd687               1         1         1       28m
replicaset.apps/mender-deviceconfig-66967cd7c7             1         1         1       28m
replicaset.apps/mender-deviceconnect-559f4496f4            1         1         1       28m
replicaset.apps/mender-gui-7944884849                      1         1         1       28m
replicaset.apps/mender-inventory-6f7df89889                1         1         1       28m
replicaset.apps/mender-iot-manager-7cc67d98bc              1         1         1       28m
replicaset.apps/mender-useradm-7775ccd87f                  1         1         1       28m
replicaset.apps/mender-workflows-server-6494d5c5f9         1         1         1       28m
replicaset.apps/mender-workflows-worker-765b5bc789         1         1         1       28m
replicaset.apps/nats-box-5fb7f4979f                        1         1         1       33m
replicaset.apps/seaweedfs-s3-57ffbb7694                    1         1         1       78m

NAME                                     READY   AGE
statefulset.apps/mender-redis-master     1/1     30m
statefulset.apps/mender-redis-replicas   3/3     30m
statefulset.apps/mongodb                 1/1     34m
statefulset.apps/mongodb-arbiter         1/1     34m
statefulset.apps/nats                    2/2     33m
statefulset.apps/seaweedfs-filer         1/1     78m
statefulset.apps/seaweedfs-master        1/1     78m
statefulset.apps/seaweedfs-volume        1/1     78m

NAME                                              SCHEDULE     TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/mender-deployments-storage-daemon   15 * * * *   <none>     False     0        <none>          28m

I switched over to using SeaweedFS as I see MinIO is no longer compatible.

I now get this error in pod/mender-deployments-577bfdf64d-wwpcx

kubectl logs pod/mender-deployments-577bfdf64d-wwpcx   
time="2025-01-23T17:02:47Z" level=warning msg="'presign.secret' not configured. Generating a random secret." caller="config.Setup@config.go:246"
time="2025-01-23T17:02:47Z" level=info msg="Deployments Service starting up" caller="main.cmdServer@main.go:158"
time="2025-01-23T17:02:47Z" level=info msg="automigrate is ON, will apply migrations" caller="mongo.Migrate@migrations.go:49"
time="2025-01-23T17:02:47Z" level=info msg="migrating deployment_service" caller="mongo.MigrateSingle@migrations.go:71"
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.1 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.2 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.3 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.4 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.5 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.6 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.7 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.9 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.10 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.11 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.13 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.14 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="migration to version 1.2.15 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-23T17:02:47Z" level=info msg="DB migrated to version 1.2.15" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:139" db=deployment_service
main: failed to setup storage client: s3: failed to check bucket preconditions: operation error S3: HeadBucket, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "https://mender.example.com/test-bucket": dial tcp: lookup mender.example.com on 10.43.0.10:53: no such host
kubectl get ingress
NAME                     CLASS     HOSTS                ADDRESS         PORTS     AGE
external-nginx-ingress   traefik   mender.example.com   <host_ip_address_running_nginx_container>   80, 443   92m

Why does that error make it seem like mongoDB and redis are not running? Maybe my most recent attempt shown in this have it running correctly now?

I decided to use external mongoDB and nats, as I kept running into issues with having mender helm bundle them. The error I’m getting now makes it seem like it cannot connect to the SeaweedFS test bucket. I’m still confused by the presign.secret error though too, as I keep getting that on every attempt I do