Self Signed Certs for mender-deployments k3s helm

General Discussions
, ,
1

I see that there are docs for exposing self-signed certs to all of the mender server docker containers for mender 2.3 Certificates and keys | Mender documentation

I am using the latest mender 3.7 but with k3s. I have gotten the pods to start up and can reach mender at https://mender.example.com through the self-signed cert, but the deployments pod keeps crashing. I think this has to do with not exposing the self-signed cert to all of the mender pods.

How can I easily expose a self-signed cert via mender helm? I do not see volumeMounts on the helm charts for mender, so I am not sure how they would be able to get access.

kubectl get all
NAME                                                   READY   STATUS             RESTARTS        AGE
pod/mender-api-gateway-75f8dff5b5-68j8g                1/1     Running            0               145m
pod/mender-auditlogs-migration-5xn86                   0/1     ImagePullBackOff   0               3h45m
pod/mender-create-artifact-worker-5c64f7498d-jsbn2     1/1     Running            0               145m
pod/mender-deployments-648b65cb77-xrcvk                0/1     CrashLoopBackOff   7 (3m22s ago)   14m
pod/mender-deployments-7b76655698-zm22c                0/1     CrashLoopBackOff   7 (3m28s ago)   14m
pod/mender-deployments-storage-daemon-28969635-f25dj   0/1     Completed          0               50m
pod/mender-device-auth-688cc46fbb-kkc8t                1/1     Running            0               145m
pod/mender-deviceconfig-6c474b6866-dcs8w               1/1     Running            0               145m
pod/mender-deviceconnect-579d5579cc-pq7ss              1/1     Running            0               145m
pod/mender-gui-7d77ff88bb-6lhbl                        1/1     Running            0               145m
pod/mender-inventory-68d5f45474-6tllk                  1/1     Running            0               145m
pod/mender-iot-manager-7c8bc89894-7s5m6                1/1     Running            0               145m
pod/mender-mongodb-0                                   1/1     Running            0               3h45m
pod/mender-mongodb-1                                   1/1     Running            0               3h45m
pod/mender-mongodb-arbiter-0                           1/1     Running            0               3h46m
pod/mender-nats-0                                      3/3     Running            0               3h43m
pod/mender-nats-1                                      3/3     Running            0               3h43m
pod/mender-nats-2                                      3/3     Running            0               3h43m
pod/mender-nats-box-7d447f45b-dwhcx                    1/1     Running            0               3h43m
pod/mender-redis-master-0                              1/1     Running            0               3h46m
pod/mender-redis-replicas-0                            1/1     Running            0               3h46m
pod/mender-redis-replicas-1                            1/1     Running            0               3h45m
pod/mender-redis-replicas-2                            1/1     Running            0               3h45m
pod/mender-useradm-7b9bd49f66-9k7wf                    1/1     Running            0               145m
pod/mender-workflows-server-6f754485f8-g9tjf           1/1     Running            0               145m
pod/mender-workflows-worker-5655b65dd4-g87s9           1/1     Running            0               145m
pod/seaweedfs-filer-0                                  1/1     Running            0               4h36m
pod/seaweedfs-master-0                                 1/1     Running            0               4h36m
pod/seaweedfs-s3-57ffbb7694-4fr6x                      1/1     Running            0               4h36m
pod/seaweedfs-volume-0                                 1/1     Running            0               4h36m

NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                 AGE
service/kubernetes                        ClusterIP   10.43.0.1       <none>        443/TCP                                                 4h58m
service/mender-api-gateway                ClusterIP   10.43.248.74    <none>        80/TCP                                                  3h43m
service/mender-deployments                ClusterIP   10.43.247.59    <none>        8080/TCP                                                3h43m
service/mender-device-auth                ClusterIP   10.43.196.162   <none>        8080/TCP                                                3h43m
service/mender-deviceconfig               ClusterIP   10.43.176.109   <none>        8080/TCP                                                3h43m
service/mender-deviceconnect              ClusterIP   10.43.105.30    <none>        8080/TCP                                                3h43m
service/mender-gui                        ClusterIP   10.43.202.65    <none>        80/TCP,8080/TCP                                         3h43m
service/mender-inventory                  ClusterIP   10.43.92.73     <none>        8080/TCP                                                3h43m
service/mender-iot-manager                ClusterIP   10.43.155.160   <none>        8080/TCP                                                3h43m
service/mender-mongodb-arbiter-headless   ClusterIP   None            <none>        27017/TCP                                               3h46m
service/mender-mongodb-headless           ClusterIP   None            <none>        27017/TCP                                               3h46m
service/mender-nats                       ClusterIP   None            <none>        4222/TCP,6222/TCP,8222/TCP,7777/TCP,7422/TCP,7522/TCP   3h43m
service/mender-redis-headless             ClusterIP   None            <none>        6379/TCP                                                3h46m
service/mender-redis-master               ClusterIP   10.43.67.198    <none>        6379/TCP                                                3h46m
service/mender-redis-replicas             ClusterIP   10.43.239.154   <none>        6379/TCP                                                3h46m
service/mender-useradm                    ClusterIP   10.43.21.1      <none>        8080/TCP                                                3h43m
service/mender-workflows-server           ClusterIP   10.43.120.115   <none>        8080/TCP                                                3h43m
service/seaweedfs-filer                   ClusterIP   None            <none>        8888/TCP,18888/TCP,8333/TCP,9327/TCP                    4h36m
service/seaweedfs-filer-client            ClusterIP   None            <none>        8888/TCP,18888/TCP,9327/TCP                             4h36m
service/seaweedfs-master                  ClusterIP   None            <none>        9333/TCP,19333/TCP,9327/TCP                             4h36m
service/seaweedfs-s3                      ClusterIP   10.43.132.45    <none>        8333/TCP,9327/TCP                                       4h36m
service/seaweedfs-volume                  ClusterIP   None            <none>        8080/TCP,18080/TCP,9327/TCP                             4h36m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mender-api-gateway              1/1     1            1           3h43m
deployment.apps/mender-create-artifact-worker   1/1     1            1           3h43m
deployment.apps/mender-deployments              0/1     1            0           3h43m
deployment.apps/mender-device-auth              1/1     1            1           3h43m
deployment.apps/mender-deviceconfig             1/1     1            1           3h43m
deployment.apps/mender-deviceconnect            1/1     1            1           3h43m
deployment.apps/mender-gui                      1/1     1            1           3h43m
deployment.apps/mender-inventory                1/1     1            1           3h43m
deployment.apps/mender-iot-manager              1/1     1            1           3h43m
deployment.apps/mender-nats-box                 1/1     1            1           3h43m
deployment.apps/mender-useradm                  1/1     1            1           3h43m
deployment.apps/mender-workflows-server         1/1     1            1           3h43m
deployment.apps/mender-workflows-worker         1/1     1            1           3h43m
deployment.apps/seaweedfs-s3                    1/1     1            1           4h36m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/mender-api-gateway-5f45645774              0         0         0       170m
replicaset.apps/mender-api-gateway-75f8dff5b5              1         1         1       3h43m
replicaset.apps/mender-create-artifact-worker-5c64f7498d   1         1         1       3h43m
replicaset.apps/mender-create-artifact-worker-7d678664cb   0         0         0       170m
replicaset.apps/mender-deployments-648b65cb77              1         1         0       14m
replicaset.apps/mender-deployments-74cf4949d9              0         0         0       3h43m
replicaset.apps/mender-deployments-7b76655698              1         1         0       170m
replicaset.apps/mender-device-auth-688cc46fbb              1         1         1       3h43m
replicaset.apps/mender-device-auth-775b4bcfbb              0         0         0       170m
replicaset.apps/mender-deviceconfig-689f8d9b66             0         0         0       170m
replicaset.apps/mender-deviceconfig-6c474b6866             1         1         1       3h43m
replicaset.apps/mender-deviceconnect-579d5579cc            1         1         1       3h43m
replicaset.apps/mender-deviceconnect-586789c549            0         0         0       170m
replicaset.apps/mender-gui-7d77ff88bb                      1         1         1       3h43m
replicaset.apps/mender-gui-7fb54c8b86                      0         0         0       170m
replicaset.apps/mender-inventory-68d5f45474                1         1         1       3h43m
replicaset.apps/mender-inventory-c6f45c7f4                 0         0         0       170m
replicaset.apps/mender-iot-manager-748b96555d              0         0         0       170m
replicaset.apps/mender-iot-manager-7c8bc89894              1         1         1       3h43m
replicaset.apps/mender-nats-box-7d447f45b                  1         1         1       3h43m
replicaset.apps/mender-useradm-6f9d4657bd                  0         0         0       170m
replicaset.apps/mender-useradm-7b9bd49f66                  1         1         1       3h43m
replicaset.apps/mender-workflows-server-6f754485f8         1         1         1       3h43m
replicaset.apps/mender-workflows-server-cf9df944b          0         0         0       170m
replicaset.apps/mender-workflows-worker-5655b65dd4         1         1         1       3h43m
replicaset.apps/mender-workflows-worker-79d87cf5c4         0         0         0       170m
replicaset.apps/seaweedfs-s3-57ffbb7694                    1         1         1       4h36m

NAME                                      READY   AGE
statefulset.apps/mender-mongodb           2/2     3h46m
statefulset.apps/mender-mongodb-arbiter   1/1     3h46m
statefulset.apps/mender-nats              3/3     3h43m
statefulset.apps/mender-redis-master      1/1     3h46m
statefulset.apps/mender-redis-replicas    3/3     3h46m
statefulset.apps/seaweedfs-filer          1/1     4h36m
statefulset.apps/seaweedfs-master         1/1     4h36m
statefulset.apps/seaweedfs-volume         1/1     4h36m

NAME                                              SCHEDULE     TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/mender-deployments-storage-daemon   15 * * * *   <none>     False     0        50m             3h43m

NAME                                                   STATUS     COMPLETIONS   DURATION   AGE
job.batch/mender-auditlogs-migration                   Running    0/1           3h49m      3h49m
job.batch/mender-deployments-storage-daemon-28969635   Complete   1/1           6s         50m

kubectl logs pod/mender-deployments-7b76655698-zm22c  
time="2025-01-29T20:01:50Z" level=warning msg="'presign.secret' not configured. Generating a random secret." caller="config.Setup@config.go:246"
time="2025-01-29T20:01:50Z" level=info msg="Deployments Service starting up" caller="main.cmdServer@main.go:158"
time="2025-01-29T20:01:50Z" level=info msg="automigrate is ON, will apply migrations" caller="mongo.Migrate@migrations.go:49"
time="2025-01-29T20:01:50Z" level=info msg="migrating deployment_service" caller="mongo.MigrateSingle@migrations.go:71"
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.1 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.2 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.3 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.4 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.5 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.6 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.7 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.9 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.10 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.11 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.13 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.14 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="migration to version 1.2.15 skipped" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:124" db=deployment_service
time="2025-01-29T20:01:50Z" level=info msg="DB migrated to version 1.2.15" caller="migrate.(*SimpleMigrator).Apply@migrator_simple.go:139" db=deployment_service
main: failed to setup storage client: s3: failed to check bucket preconditions: operation error S3: HeadBucket, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head "https://mender.example.com/test-bucket": tls: failed to verify certificate: x509: certificate signed by unknown authority
2

Nevermind. There is a typo in the docs, which caused this issue. AWS_URI should be STORAGE_ENDPOINT, not MENDER_SERVER_URL . Mender Server | Mender documentation

export MENDER_SERVER_DOMAIN="mender.example.com"
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"

cat <<-EOF >> mender-values.yml
global:
  s3:
    AWS_URI: "${STORAGE_ENDPOINT}"
    AWS_BUCKET: "${STORAGE_BUCKET}"
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
  url: "${MENDER_SERVER_URL}"

api_gateway:
  storage_proxy:
    enabled: true
    url: "${STORAGE_ENDPOINT}"
    customRule: "PathRegexp(\`^/${STORAGE_BUCKET}\`)"
EOF
3

Hello @mld ,
please be aware that the documentation you’re using is referring to the next 4.0 version which has not being released yet. Feel free to test it, though and please share your feedback ;). For making it work you have to use this helm install command:
helm upgrade --install mender mender/mender --set default.image.tag=v4.0.0-rc.8 -f mender-values.yml --devel

If you want to test the stable 3.7 release, though, you should go through this documentation, but this requires a public DNS and a public signed Certificate.

4

@robgio thanks for letting me know! So for the 4.0 version with the helm install you shared, will using self-signed certs work if I follow those instructions? Mender Server | Mender documentation

5

It should, yes. But then when you have to accept a new Mender Device, the device has to trust the cert, so up to you to still use the self signed cert or moving to a trusted one.
For demo purposes, I think you can just create your own CA and sign your own server cert and use it in the Ingress; then on the device side you have just to trust that CA.

6

Thanks, I’ll plan on testing that out. So for open source mender v4.0, should the AWS_URI = http://seaweedfs-s3:8333 instead of the MENDER_SERVER_URL?

7

Yes, correct

1 Like