Kubernetes deployments endpoint is not working, pod crashing

General Discussions
,
1

Good Day, Friends!

I installed self-hosted mender on Azure VM using K3s like in the documentaion. I needed to make some adjustments because it’s not working like described in the documentaion (outdated/wrong). Every pod don’t report any errors so far. I can login and use Mender UI except for the releases tab.

Deployments pod is constantly crashing because the livenessProbe failes
deployments-7b765947b9-cb9sh 0/1 CrashLoopBackOff 251 (27s ago) 21h

Pod logs report an error I can’t comprehend

~$ kubectl logs deployments-7b765947b9-cb9sh
time="2023-03-31T10:40:28Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
time="2023-03-31T10:40:28Z" level=info msg="Deployments Service starting up" file=main.go func=main.cmdServer line=138
time="2023-03-31T10:40:28Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2023-03-31T10:40:28Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.3 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.4 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.5 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.6 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.7 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.9 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.10 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.11 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="DB migrated to version 1.2.11" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestCanceled: request context canceled
caused by: context deadline exceeded

Here are my variables I used for the K3S installation. I used .yaml snippets from the documentation with minor tweaks.

export CERT_MANAGER_CHART_VERSION="v1.10.0"
export LETSENCRYPT_SERVER_URL="https://acme-v02.api.letsencrypt.org/directory"
export LETSENCRYPT_EMAIL="<email>"
export MONGODB_CHART_VERSION="12.1.31"
export MONGODB_TAG="5.0.10-debian-11-r7"
export NATS_IMAGE="nats:2.7.4-alpine"
export NATS_CHART_VERSION="0.15.1"
export MINIO_TAG="RELEASE.2021-06-17T00-10-46Z"
export MINIO_CHART_VERSION="4.1.7"
export MINIO_DOMAIN_NAME="<public-accessible-domain>" # I can upload files here
export MENDER_SERVER_DOMAIN="<public-accessible-domain>" # UI accessible
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
export MENDER_VERSION="3.4.0" # Chart and app version

Thanks in advance!

2

One question I have is whether Mender is compatible with 5.0.x MongoDB?

3

While installing I’ve tried several versions including MongoDB 5 and 6. Ending up with the same problem.

4

I was trying with 4.4.15-debian-10-r8. I am not sure if Mender requires 4.4.x or not.

5

Hi @mister_kanister :wave:

I have to admit, that log does not seem very helpful. I would guess that the deployments service times out when trying to connect to the object storage backend. Could you verify that the storage settings are correct? That is, that the minio instance is reachable and credentials are correct.

6

Thanks for your answer!
I’ve increased the time and attempts threshold in the livenessProbe of deployments service pod (even removed completly). MinIO is accessible through a public URL. I can login and upload/download files via MinIO UI.

I’m looking through the helm chart documentation of mender 3.5. It comes to my attention that you can’t select MinIO as your default storage provider. In values.yaml under global.storage there is only the option for Azure or AWS. I don’t use both. I want to use the local storage of my VM. That means if leave global.storage blank, MinIO became my default provider. Furthermore under global.s3.<URI, KEY and SECRET> are mentioned in the documentation as equivalent to MinIO credentials. Is my assumption right so far?

global:
  enterprise: false
  hosted: false
  auditlogs: false
  # Deleted this, because I assume that MinIO will became my default storage provider
  # storage: "aws"
  image:
    registry: docker.io
  mongodb:
    URL: mongodb://mongodb
  nats:
    URL: "nats://nats:4222"
  url: "https://<domain>.com"
  # Are these actually MinIO credential now?
  s3:
    AWS_URI: 
    AWS_ACCESS_KEY_ID:
    AWS_SECRET_ACCESS_KEY:
7

Yes, your assumption is correct. Minio is serving the AWS s3 API, so they are equivalent - you need to configure the Minio access key using the “AWS” configuration values.

8

That was it!

I must admit the naming convention is little bit confusing. I wrongly assumed I need to exclude AWS variables so MinIO will be used. I was able to push an artifact from CI/CD to my mender server.

Adjusted the values like this:

# AWS variables are actually used for your MinIO instance
cat >mender-${MENDER_VERSION}.yml <<EOF
global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:${MONGODB_ROOT_PASSWORD}@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  url: "${MENDER_SERVER_URL}"
  s3:
    AWS_URI: "https://${MINIO_DOMAIN_NAME}"
    AWS_BUCKET: "mender-artifact-storage"
    AWS_ACCESS_KEY_ID: "${MINIO_ACCESS_KEY}"
    AWS_SECRET_ACCESS_KEY: "${MINIO_SECRET_KEY}"
api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
$(cat device_auth.key | sed -e 's/^/      /g')

useradm:
  certs:
    key: |-
$(cat useradm.key | sed -e 's/^/      /g')
EOF

Thanks!

1 Like