# After upgrade to Kubernetes 1.29, Mender no longer works **URL:** \ **Category:** General Discussions\ **Tags:** kubernetes\ **Created:** [September 23, 2024, 10:41pm UTC](https://hub.mender.io/t/after-upgrade-to-kubernetes-1-29-mender-no-longer-works/7193 "2024-09-23T22:41:19Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![archibaldodlc](https://avatars.discourse-cdn.com/v4/letter/a/f17d59/32.png) [@archibaldodlc](https://hub.mender.io/u/archibaldodlc)\ **Post date:** [September 23, 2024, 10:41pm UTC](https://hub.mender.io/t/after-upgrade-to-kubernetes-1-29-mender-no-longer-works/7193/1 "2024-09-23T22:41:19Z")
Hi, everyone! I have an issue and want to know if you can help me. Recently, I upgraded my EKS on AWS to 1.29 and after that, I cannot get Mender to work. The workflows-server, the workflows-worker and create-artifact-worker continuosly restart with CrashLoopBackOff ```auto kubectl get pods NAME READY STATUS RESTARTS AGE api-gateway-865ffc4768-mqzkg 1/1 Running 0 51m create-artifact-worker-69646cf645-464dq 0/1 CrashLoopBackOff 14 (4m48s ago) 51m deployments-5d9b6d6bd5-lm7sl 0/1 Running 0 11m device-auth-77bdb4868f-h2fv2 0/1 Running 0 11m deviceconfig-586d87974-jqxxb 1/1 Running 0 11m deviceconnect-8565cf5d45-khxhf 1/1 Running 0 11m gui-5d76f4dc89-8gpt7 1/1 Running 0 41m inventory-dcc7dc886-wgzq8 1/1 Running 0 11m iot-manager-675488bd48-8zhxk 1/1 Running 0 11m nats-0 3/3 Running 0 41m nats-box-56d4878784-4q2j7 1/1 Running 0 51m sonarqube-sonarqube-0 1/1 Running 0 111m useradm-99478d77b-p8wb6 1/1 Running 0 11m workflows-server-58495d7489-rlmvq 0/1 CrashLoopBackOff 8 (3m46s ago) 19m workflows-worker-6bdb9d8cbc-k56n7 0/1 CrashLoopBackOff 7 (56s ago) 11m ``` This are the logs: Workflows-server logs ```auto time="2024-09-23T22:18:10Z" level=info msg="migrating workflows" file=migrations.go func=mongo.Migrate line=38 2024/09/23 22:18:10 nats: no responders available for request ``` Workflows-server Describe ```auto Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 31m default-scheduler Successfully assigned default/workflows-server-58495d7489-rlmvq to ip-10-127-1-254.ec2.internal Normal Pulled 29m (x5 over 31m) kubelet Container image "docker.io/mendersoftware/workflows:mender-3.4.0" already present on machine Normal Created 29m (x5 over 31m) kubelet Created container workflows Normal Started 29m (x5 over 31m) kubelet Started container workflows Warning BackOff 69s (x151 over 31m) kubelet Back-off restarting failed container workflows in pod workflows-server-58495d7489-rlmvq_default(86d06ee9-850f-4ee5-814e-b00a534c7352) ``` Workflows-worker logs ```auto time="2024-09-23T22:21:04Z" level=info msg="migrating workflows" file=migrations.go func=mongo.Migrate line=38 2024/09/23 22:21:04 nats: no responders available for request ``` Workflows-worker describe ```auto Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 22m default-scheduler Successfully assigned default/workflows-worker-6bdb9d8cbc-k56n7 to ip-10-127-2-101.ec2.internal Normal Pulling 22m kubelet Pulling image "docker.io/mendersoftware/workflows-worker:mender-3.4.0" Normal Pulled 22m kubelet Successfully pulled image "docker.io/mendersoftware/workflows-worker:mender-3.4.0" in 839ms (839ms including waiting) Normal Created 20m (x5 over 22m) kubelet Created container workflows Normal Started 20m (x5 over 22m) kubelet Started container workflows Normal Pulled 20m (x4 over 22m) kubelet Container image "docker.io/mendersoftware/workflows-worker:mender-3.4.0" already present on machine Warning BackOff 2m (x94 over 22m) kubelet Back-off restarting failed container workflows in pod workflows-worker-6bdb9d8cbc-k56n7_default(649c33c3-e550-4989-9397-6d3acc07a2f0) ``` Creater-Artifact worker logs ```auto time="2024-09-23T22:22:08Z" level=info msg="migrating workflows" file=migrations.go func=mongo.Migrate line=38 2024/09/23 22:22:08 nats: no responders available for request ``` The nats are working fine. The Deployments pod is running good, but I got the following 503 error, I supposed because the worksflows server is not working: ```auto time="2024-09-23T22:32:12Z" level=error msg="Workflows service unhealthy: Get \"http://mender-workflows-server:8080/api/v1/health\": dial tcp 172.20.50.161:8080: connect: connection refused" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=1bd8f4a1-8357-4e09-be55-f505633ca3f7 time="2024-09-23T22:32:12Z" level=info msg="503 39289μs GET /api/internal/v1/deployments/health HTTP/1.1 - kube-probe/1.29+" byteswritten=208 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/health qs= request_id=1bd8f4a1-8357-4e09-be55-f505633ca3f7 responsetime=0.0 39289687 status=503 ts="2024-09-23 22:32:12.495906076 +0000 UTC" type=http time="2024-09-23T22:32:16Z" level=error msg="Workflows service unhealthy: Get \"http://mender-workflows-server:8080/api/v1/health\": dial tcp 172.20.50.161:8080: connect: connection refused" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=c7030c45-2578-4441-81fe-76b833dd8439 time="2024-09-23T22:32:16Z" level=info msg="503 23544μs GET /api/internal/v1/deployments/health HTTP/1.1 - kube-probe/1.29+" byteswritten=208 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/health qs= request_id=c7030c45-2578-4441-81fe-76b833dd8439 responsetime=0.0 23544842 status=503 ts="2024-09-23 22:32:16.673091063 +0000 UTC" type=http time="2024-09-23T22:32:17Z" level=info msg="204 53μs GET /api/internal/v1/deployments/alive HTTP/1.1 - kube-probe/1.29+" byteswritten=0 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/alive qs= request_id=5181b61c-ed85-4de3-b851-cc4e9dc39752 responsetime=5.3792e-05 status=204 ts="2024-09-23 22:32:17.495558402 +0000 UTC" type=http time="2024-09-23T22:32:22Z" level=info msg="204 53μs GET /api/internal/v1/deployments/alive HTTP/1.1 - kube-probe/1.29+" byteswritten=0 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/alive qs= request_id=85316c95-e6a1-470e-80f1-646b98449b14 responsetime=5.3454e-05 status=204 ts="2024-09-23 22:32:22.495332492 +0000 UTC" type=http time="2024-09-23T22:32:27Z" level=info msg="204 66μs GET /api/internal/v1/deployments/alive HTTP/1.1 - kube-probe/1.29+" byteswritten=0 file=middleware.go func="accesslog.(*AccessLogMiddleware).MiddlewareFunc.func1" line=71 method=GET path=/api/internal/v1/deployments/alive qs= request_id=28578432-71d3-4a04-b075-617073ffe78d responsetime=6.6035e-05 status=204 ts="2024-09-23 22:32:27.495704439 +0000 UTC" type=http time="2024-09-23T22:32:27Z" level=error msg="Workflows service unhealthy: Get \"http://mender-workflows-server:8080/api/v1/health\": dial tcp 172.20.50.161:8080: connect: connection refused" file=response_helpers.go func=rest_utils.restErrWithLogMsg line=110 request_id=14aa22b5-4a4a-449f-b4b4-dbebb888a614 ``` I tried to redeploy the Mender with Helm, but didn’t work. The stack I’m using is: Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.8-eks-a737599 Mender: 3.4.0 (Via Helm) What could be happening? Thanks in advance. Regards, Víctor. ---
**Author:** ![archibaldodlc](https://avatars.discourse-cdn.com/v4/letter/a/f17d59/32.png) [@archibaldodlc](https://hub.mender.io/u/archibaldodlc)\ **Post date:** [September 26, 2024, 7:47pm UTC](https://hub.mender.io/t/after-upgrade-to-kubernetes-1-29-mender-no-longer-works/7193/2 "2024-09-26T19:47:58Z")
I resolved. If helps anyone, I had disabled the jetstreams on Nats helm values.yml. I don’t know why this was working in previous Kubernetes version. But enable it in Kubernetes 1.29 seems to work. Nats helm values.yml (nats version 2.7.4) ```auto jetstream: **enabled: true** domain: memStorage: enabled: true size: 1Gi fileStorage: enabled: true storageDirectory: /data size: 10Gi accessModes: - ReadWriteOnce annotations: ``` Hope it helps. Regards, Víctor ---
**Author:** ![robgio](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/robgio/32/1760_2.png) [@robgio](https://hub.mender.io/u/robgio)\ **Post date:** [September 27, 2024, 8:27am UTC](https://hub.mender.io/t/after-upgrade-to-kubernetes-1-29-mender-no-longer-works/7193/3 "2024-09-27T08:27:33Z")
Thanks @archibaldodlc. Just a note: jetstream should be enable in Mender. Can you please try to use NATS version to 2.9.20 instead? There are known issues at startup with the 2.7.4. The [default settings of the helm chart v5.7.1](https://github.com/mendersoftware/mender-helm/blob/master/mender/CHANGELOG.md#version-571) include the right NATS version.