Clean mender open-source install keeps failing

General Discussions
1

Trying to install Mender Server on a clean Ubuntu VM running k3s. I followed the tutorial to the letter (changed the domain names etc of course), but the helm deployment of the server keeps failing (it times out after 5 minutes). How do I troubleshoot this?
Below is the output of the helm upgrade command with --debug flag. Also included kubectl get pods output.

Release "mender" does not exist. Installing it now.
Error: failed pre-install: 1 error occurred:
	* timed out waiting for the condition


root@mender-server:/home/thijs# helm upgrade --install mender mender/mender --wait -f mender-values.yml --debug
history.go:56: 2026-07-02 20:57:00.473799696 +0000 UTC m=+0.075558821 [debug] getting history for release mender
upgrade.go:164: 2026-07-02 20:57:00.591067798 +0000 UTC m=+0.192826921 [debug] preparing upgrade for mender
upgrade.go:172: 2026-07-02 20:57:01.442325284 +0000 UTC m=+1.044084438 [debug] performing update for mender
upgrade.go:375: 2026-07-02 20:57:02.314117047 +0000 UTC m=+1.915876208 [debug] creating upgraded release for mender
client.go:506: 2026-07-02 20:57:02.564907485 +0000 UTC m=+2.166666607 [debug] Starting delete for "mongodb-common-prerelease" Secret
client.go:510: 2026-07-02 20:57:02.573730958 +0000 UTC m=+2.175490102 [debug] Ignoring delete failure for "mongodb-common-prerelease" /v1, Kind=Secret: secrets "mongodb-common-prerelease" not found
wait.go:114: 2026-07-02 20:57:02.573797189 +0000 UTC m=+2.175556310 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:144: 2026-07-02 20:57:02.845484747 +0000 UTC m=+2.447243874 [debug] creating 1 resource(s)
client.go:506: 2026-07-02 20:57:02.88292093 +0000 UTC m=+2.484680088 [debug] Starting delete for "mender-deployments-migration" Job
wait.go:114: 2026-07-02 20:57:02.902860833 +0000 UTC m=+2.504619989 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:144: 2026-07-02 20:57:03.203593431 +0000 UTC m=+2.805352601 [debug] creating 1 resource(s)
client.go:753: 2026-07-02 20:57:03.246117172 +0000 UTC m=+2.847876293 [debug] Watching for changes to Job mender-deployments-migration with timeout of 5m0s
client.go:781: 2026-07-02 20:57:03.272347793 +0000 UTC m=+2.874106954 [debug] Add/Modify event for mender-deployments-migration: ADDED
client.go:820: 2026-07-02 20:57:03.272397574 +0000 UTC m=+2.874156697 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:03.681651545 +0000 UTC m=+3.283410726 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:03.681873445 +0000 UTC m=+3.283632569 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:04.686010758 +0000 UTC m=+4.287769892 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:04.68605245 +0000 UTC m=+4.287811572 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:15.863583696 +0000 UTC m=+15.465342873 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:15.863653112 +0000 UTC m=+15.465412235 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:16.868474909 +0000 UTC m=+16.470234063 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:16.868518934 +0000 UTC m=+16.470278058 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:16.883346566 +0000 UTC m=+16.485105725 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:16.883401323 +0000 UTC m=+16.485160444 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 1, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:24.014492305 +0000 UTC m=+23.616251486 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:24.014633673 +0000 UTC m=+23.616392837 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:25.018737629 +0000 UTC m=+24.620496806 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:25.018902335 +0000 UTC m=+24.620661498 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:36.95684604 +0000 UTC m=+36.558605163 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:36.956927212 +0000 UTC m=+36.558686365 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:37.962274366 +0000 UTC m=+37.564033524 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:37.962344635 +0000 UTC m=+37.564103771 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 1, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:37.975406823 +0000 UTC m=+37.577165976 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:37.975458046 +0000 UTC m=+37.577217173 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 2, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:54.024820205 +0000 UTC m=+53.626579378 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:54.024873795 +0000 UTC m=+53.626632919 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:781: 2026-07-02 20:57:56.025777145 +0000 UTC m=+55.627536306 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:57:56.025845667 +0000 UTC m=+55.627604791 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:07.135174644 +0000 UTC m=+66.736933810 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:07.135223934 +0000 UTC m=+66.736983056 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:08.153445292 +0000 UTC m=+67.755204452 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:08.153625885 +0000 UTC m=+67.755385043 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 2, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:08.206065573 +0000 UTC m=+67.807824735 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:08.206123345 +0000 UTC m=+67.807882469 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 3, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:45.059191239 +0000 UTC m=+104.660950403 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:45.059280344 +0000 UTC m=+104.661039466 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 3, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:46.046644643 +0000 UTC m=+105.648403839 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:46.046882546 +0000 UTC m=+105.648641708 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 3, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:57.30372025 +0000 UTC m=+116.905479433 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:57.303780173 +0000 UTC m=+116.905539299 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 3, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:58.324257541 +0000 UTC m=+117.926016709 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:58.324400967 +0000 UTC m=+117.926160129 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 3, jobs succeeded: 0
client.go:781: 2026-07-02 20:58:58.374981019 +0000 UTC m=+117.976740186 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 20:58:58.37502319 +0000 UTC m=+117.976782312 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 4, jobs succeeded: 0
client.go:781: 2026-07-02 21:00:15.069732054 +0000 UTC m=+194.671491230 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 21:00:15.069799171 +0000 UTC m=+194.671558295 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 4, jobs succeeded: 0
client.go:781: 2026-07-02 21:00:16.084505982 +0000 UTC m=+195.686265131 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 21:00:16.084581275 +0000 UTC m=+195.686340395 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 4, jobs succeeded: 0
client.go:781: 2026-07-02 21:00:27.629222459 +0000 UTC m=+207.230981624 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 21:00:27.629263744 +0000 UTC m=+207.231022867 [debug] mender-deployments-migration: Jobs active: 1, jobs failed: 4, jobs succeeded: 0
client.go:781: 2026-07-02 21:00:28.658998428 +0000 UTC m=+208.260757551 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 21:00:28.659088336 +0000 UTC m=+208.260847458 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 4, jobs succeeded: 0
client.go:781: 2026-07-02 21:00:28.676013935 +0000 UTC m=+208.277773082 [debug] Add/Modify event for mender-deployments-migration: MODIFIED
client.go:820: 2026-07-02 21:00:28.676062234 +0000 UTC m=+208.277821356 [debug] mender-deployments-migration: Jobs active: 0, jobs failed: 5, jobs succeeded: 0
client.go:506: 2026-07-02 21:02:03.271121213 +0000 UTC m=+302.872880349 [debug] Starting delete for "mongodb-common-prerelease" Secret
wait.go:114: 2026-07-02 21:02:03.2760893 +0000 UTC m=+302.877848426 [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
upgrade.go:487: 2026-07-02 21:02:03.280110827 +0000 UTC m=+302.881869950 [debug] warning: Upgrade "mender" failed: pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition


Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition


helm.go:92: 2026-07-02 21:02:03.342637398 +0000 UTC m=+302.944396538 [debug] pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition


UPGRADE FAILED
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:244
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.10.2/command.go:1015
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.10.2/command.go:1148
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.10.2/command.go:1071
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:91
runtime.main
	runtime/proc.go:290
runtime.goexit
	runtime/asm_amd64.s:1771

kubectl get pods:



NAME                                 READY   STATUS                       RESTARTS      AGE

mender-deployments-migration-2tmwv   0/1     Error                        0             5m13s

mender-deployments-migration-497b2   0/1     Error                        0             6m4s

mender-deployments-migration-4xdbg   0/1     Error                        0             6m55s

mender-deployments-migration-64xmc   0/1     CreateContainerConfigError   0             53s

mender-deployments-migration-9mxjd   0/1     Error                        0             3m43s

mender-deployments-migration-b4flr   0/1     Error                        0             6m34s

mender-mongodb-59f9cd9c5f-gzksc      0/1     CrashLoopBackOff             7 (47s ago)   12m
2

Hi @thm,

Thanks for reaching out! The most likely reason is that the MongoDB version we reference in the Helm chart requires the AVX cpu extensions to be available. Can you check if those are present in your hypervisor configuration?

Greetz,
Josef

3

Hi Josef,

Thanks for your help! I progressed a lot over the weekend. Had to resolve several issues, of which the AVX extension was one of them.

Now the only issue I appear to be having is connecting to S3 storage.. I aim to use CloudFlare R2, but can’t seem to get that to work… One of the deployments pods stays in retrying mode.

Created an AWS S3 bucket and that seems to work..

But I would prefer to use R2…

4

Hi @thm it should work fine with R2 as well. Can you share the deployments service log that are failing to access R2?