# Kubernetes deployments endpoint is not working, pod crashing

**URL:** <https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724>\
**Category:** General Discussions\
**Tags:** deployments, kubernetes\
**Created:** [March 31, 2023, 10:56am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724 "2023-03-31T10:56:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mister\_kanister](https://avatars.discourse-cdn.com/v4/letter/m/8edcca/32.png) [@mister\_kanister](https://hub.mender.io/u/mister_kanister)\
**Post date:** [March 31, 2023, 10:56am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/1 "2023-03-31T10:56:34Z")

</div>

Good Day, Friends!

I installed self-hosted mender on Azure VM using K3s like in the documentaion. I needed to make some adjustments because it’s not working like described in the documentaion (outdated/wrong). Every pod don’t report any errors so far. I can login and use Mender UI except for the releases tab.

Deployments pod is constantly crashing because the livenessProbe failes  
`deployments-7b765947b9-cb9sh 0/1 CrashLoopBackOff 251 (27s ago) 21h`

Pod logs report an error I can’t comprehend

```auto
~$ kubectl logs deployments-7b765947b9-cb9sh
time="2023-03-31T10:40:28Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
time="2023-03-31T10:40:28Z" level=info msg="Deployments Service starting up" file=main.go func=main.cmdServer line=138
time="2023-03-31T10:40:28Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2023-03-31T10:40:28Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.3 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.4 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.5 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.6 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.7 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.9 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.10 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="migration to version 1.2.11 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2023-03-31T10:40:28Z" level=info msg="DB migrated to version 1.2.11" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestCanceled: request context canceled
caused by: context deadline exceeded

```

Here are my variables I used for the K3S installation. I used .yaml snippets from the documentation with minor tweaks.

```auto
export CERT_MANAGER_CHART_VERSION="v1.10.0"
export LETSENCRYPT_SERVER_URL="https://acme-v02.api.letsencrypt.org/directory"
export LETSENCRYPT_EMAIL="<email>"
export MONGODB_CHART_VERSION="12.1.31"
export MONGODB_TAG="5.0.10-debian-11-r7"
export NATS_IMAGE="nats:2.7.4-alpine"
export NATS_CHART_VERSION="0.15.1"
export MINIO_TAG="RELEASE.2021-06-17T00-10-46Z"
export MINIO_CHART_VERSION="4.1.7"
export MINIO_DOMAIN_NAME="<public-accessible-domain>" # I can upload files here
export MENDER_SERVER_DOMAIN="<public-accessible-domain>" # UI accessible
export MENDER_SERVER_URL="https://${MENDER_SERVER_DOMAIN}"
export MENDER_VERSION="3.4.0" # Chart and app version

```

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 3, 2023, 6:50pm UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/2 "2023-04-03T18:50:20Z")

</div>

One question I have is whether Mender is compatible with `5.0.x` MongoDB?

---

<div class="post-metadata">

**Author:** ![mister\_kanister](https://avatars.discourse-cdn.com/v4/letter/m/8edcca/32.png) [@mister\_kanister](https://hub.mender.io/u/mister_kanister)\
**Post date:** [April 4, 2023, 9:43am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/3 "2023-04-04T09:43:52Z")

</div>

While installing I’ve tried several versions including MongoDB 5 and 6. Ending up with the same problem.

---

<div class="post-metadata">

**Author:** ![moto-timo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/moto-timo/32/1736_2.png) [@moto-timo](https://hub.mender.io/u/moto-timo)\
**Post date:** [April 4, 2023, 6:36pm UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/4 "2023-04-04T18:36:13Z")

</div>

I was trying with `4.4.15-debian-10-r8`. I am not sure if Mender requires `4.4.x` or not.

---

<div class="post-metadata">

**Author:** ![alfrunes](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/alfrunes/32/703_2.png) [@alfrunes](https://hub.mender.io/u/alfrunes)\
**Post date:** [April 5, 2023, 7:55am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/5 "2023-04-05T07:55:47Z")

</div>

Hi @mister_kanister 👋

I have to admit, that log does not seem very helpful. I would guess that the deployments service times out when trying to connect to the object storage backend. Could you verify that the storage settings are correct? That is, that the minio instance is reachable and credentials are correct.

---

<div class="post-metadata">

**Author:** ![mister\_kanister](https://avatars.discourse-cdn.com/v4/letter/m/8edcca/32.png) [@mister\_kanister](https://hub.mender.io/u/mister_kanister)\
**Post date:** [April 5, 2023, 8:22am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/6 "2023-04-05T08:22:21Z")

</div>

Thanks for your answer!  
I’ve increased the time and attempts threshold in the livenessProbe of deployments service pod (even removed completly). MinIO is accessible through a public URL. I can login and upload/download files via MinIO UI.

I’m looking through the helm chart documentation of mender 3.5. It comes to my attention that you can’t select MinIO as your default storage provider. In `values.yaml` under `global.storage` there is only the option for Azure or AWS. I don’t use both. I want to use the local storage of my VM. That means if leave `global.storage` blank, MinIO became my default provider. Furthermore under `global.s3.<URI, KEY and SECRET>` are mentioned in the documentation as equivalent to MinIO credentials. Is my assumption right so far?

```auto
global:
  enterprise: false
  hosted: false
  auditlogs: false
  # Deleted this, because I assume that MinIO will became my default storage provider
  # storage: "aws"
  image:
    registry: docker.io
  mongodb:
    URL: mongodb://mongodb
  nats:
    URL: "nats://nats:4222"
  url: "https://<domain>.com"
  # Are these actually MinIO credential now?
  s3:
    AWS_URI: 
    AWS_ACCESS_KEY_ID:
    AWS_SECRET_ACCESS_KEY:

```

---

<div class="post-metadata">

**Author:** ![alfrunes](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/alfrunes/32/703_2.png) [@alfrunes](https://hub.mender.io/u/alfrunes)\
**Post date:** [April 5, 2023, 8:36am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/7 "2023-04-05T08:36:38Z")

</div>

Yes, your assumption is correct. Minio is serving the AWS s3 API, so they are equivalent - you need to configure the Minio access key using the “AWS” configuration values.

---

<div class="post-metadata">

**Author:** ![mister\_kanister](https://avatars.discourse-cdn.com/v4/letter/m/8edcca/32.png) [@mister\_kanister](https://hub.mender.io/u/mister_kanister)\
**Post date:** [April 5, 2023, 9:32am UTC](https://hub.mender.io/t/kubernetes-deployments-endpoint-is-not-working-pod-crashing/5724/8 "2023-04-05T09:32:19Z")

</div>

That was it!

I must admit the naming convention is little bit confusing. I wrongly assumed I need to exclude AWS variables so MinIO will be used. I was able to push an artifact from CI/CD to my mender server.

Adjusted the values like this:

```auto
# AWS variables are actually used for your MinIO instance
cat >mender-${MENDER_VERSION}.yml <<EOF
global:
  enterprise: false
  mongodb:
    URL: "mongodb://root:${MONGODB_ROOT_PASSWORD}@mongodb-0.mongodb-headless.default.svc.cluster.local:27017,mongodb-1.mongodb-headless.default.svc.cluster.local:27017"
  nats:
    URL: "nats://nats:4222"
  url: "${MENDER_SERVER_URL}"
  s3:
    AWS_URI: "https://${MINIO_DOMAIN_NAME}"
    AWS_BUCKET: "mender-artifact-storage"
    AWS_ACCESS_KEY_ID: "${MINIO_ACCESS_KEY}"
    AWS_SECRET_ACCESS_KEY: "${MINIO_SECRET_KEY}"
api_gateway:
  env:
    SSL: false

device_auth:
  certs:
    key: |-
$(cat device_auth.key | sed -e 's/^/ /g')

useradm:
  certs:
    key: |-
$(cat useradm.key | sed -e 's/^/ /g')
EOF

```

Thanks!