# Cannot upgrade mender server from 2.4.0 to 2.5.0 - mongo container is "restarting"
**URL:** https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506
**Category:** General Discussions
**Tags:** mender-server
**Created:** [September 28, 2020, 8:56am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506 "2020-09-28T08:56:47Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Ks89](https://hub.mender.io/u/Ks89)
#### Post date: [September 28, 2020, 8:56am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/1 "2020-09-28T08:56:48Z")
After the upgrade from mender server open source 2.4.0 to 2.5.0, mongo container is restarting:
```
menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp, 80/tcp
menderproduction_mender-create-artifact-worker_1 /usr/bin/workflows --confi ... Up 8080/tcp
menderproduction_mender-deployments_1 /entrypoint.sh --config /e ... Restarting
menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf ... Up 8080/tcp
menderproduction_mender-gui_1 /entrypoint.sh nginx Up (healthy) 80/tcp
menderproduction_mender-inventory_1 /usr/bin/inventory --confi ... Up 8080/tcp
menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Restarting
menderproduction_mender-useradm_1 /usr/bin/useradm --config ... Up 8080/tcp
menderproduction_mender-workflows-server_1 /usr/bin/workflows --confi ... Up 8080/tcp
menderproduction_mender-workflows-worker_1 /usr/bin/workflows --confi ... Up
menderproduction_minio_1 /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp
menderproduction_storage-proxy_1 /usr/local/openresty/bin/o ... Up 0.0.0.0:9000->9000/tcp
```
Obviously, also other containers restart because they are not able to connect with mongo.
**docker logs idOfContainer** returns:
```
{"t":{"$date":"2020-09-28T08:29:27.068+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2020-09-28T08:29:27.073+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-09-28T08:29:27.074+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"9928341b1f12"}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I", "c":"STORAGE", "id":22297, "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3217M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"W", "c":"STORAGE", "id":22347, "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F", "c":"STORAGE", "id":28595, "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1101}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
```
What can I do?
Thanks.
---
### Author:  [@Ks89](https://hub.mender.io/u/Ks89)
#### Post date: [September 28, 2020, 9:08am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/2 "2020-09-28T09:08:56Z")
I found also this [https://tracker.mender.io/projects/MEN/issues/MEN-3975?filter=allopenissues](https://tracker.mender.io/projects/MEN/issues/MEN-3975?filter=allopenissues)
---
### Author:  [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [September 28, 2020, 3:55pm UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/3 "2020-09-28T15:55:51Z")
Just to clarify, did you run all the steps from here: [https://docs.mender.io/2.5/server-installation/upgrading](https://docs.mender.io/2.5/server-installation/upgrading)?
@tranchitella @merlin any thoughts here?
Drew
---
### Author:  [@Ks89](https://hub.mender.io/u/Ks89)
#### Post date: [September 28, 2020, 7:12pm UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/4 "2020-09-28T19:12:18Z")
yes.
I made a backup of mongo, cleaned deviceauth, updated and restarted as explained in the tutorial.
At the moment I didn’t restore mongo database, I only made backups and upgraded, but if required I can restore it, but I suppose that the problem is not related to the data itself in the db.
---
### Author:  [@drewmoseley](https://hub.mender.io/u/drewmoseley)
#### Post date: [September 29, 2020, 1:59pm UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/5 "2020-09-29T13:59:34Z")
@tranchitella @merlin any ideas here?
Drew
---
### Author:  [@tranchitella](https://hub.mender.io/u/tranchitella)
#### Post date: [October 1, 2020, 7:34am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/6 "2020-10-01T07:34:34Z")
@Ks89 ciao Stefano, Mender 2.5 uses mongodb version 4.4, while Mender 2.4 uses mongodb version 3.6; it is not possible to upgrade from mongodb 3.6 to 4.4 directly, you have to manually upgrade the database version by version, 3.6 -\> 4.0 -\> 4.2 -\> 4.4.
The official mongodb documentation applies:
> **[Upgrade a Standalone to 4.0 — MongoDB Manual](https://docs.mongodb.com/manual/release-notes/4.0-upgrade-standalone/)**
> **[Upgrade a Standalone to 4.2 — MongoDB Manual](https://docs.mongodb.com/manual/release-notes/4.2-upgrade-standalone/)**
> **[Upgrade a Standalone to 4.4 — MongoDB Manual](https://docs.mongodb.com/manual/release-notes/4.4-upgrade-standalone/)**
You can manually change the image tag of the mongodb container to 4.0, 4.2 and finally 4.4 as you proceed with the migration of the data, using the `db.adminCommand( { setFeatureCompatibilityVersion: "VERSION" } )` command, where VERSION is `4.0`, `4.2` and finally `4.4`.
I hope it helps.
---
### Author:  [@Ks89](https://hub.mender.io/u/Ks89)
#### Post date: [October 2, 2020, 2:12pm UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/7 "2020-10-02T14:12:13Z")
Hi!
> [@tranchitella](#):
>
> Mender 2.5 uses mongodb version 4.4, while Mender 2.4 uses mongodb version 3.6; it is not possible to upgrade from mongodb 3.6 to 4.4 directly, you have to manually upgrade the database version by version, 3.6 → 4.0 → 4.2 → 4.4.
Ok thank you for info
If I modify docker-compose file with mongo:3.6, I able to run it again, however using the older version.
I tried to run the command `db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } )` but it throws an error on 3.6, so I changed docker-compose to 4.0 to get the new version (as explained in the guide you posted), however the container is restarting exactly like 4.4.
I cannot run this command on 4.0 “db.adminCommand( { setFeatureCompatibilityVersion: “VERSION” } )” because it throw this error:
Error response from daemon: Container 1a4b848de777f0e0939eae3fcaedfe5ad440b813227fcb0f98265ae955d4c3c3 is restarting, wait until the container is running
Am I missing something? Do you have suggestion?
PS:
I suggest to update the official documentation (upgrade section) with this procedure, because it’s missing right now.
---
### Author:  [@tranchitella](https://hub.mender.io/u/tranchitella)
#### Post date: [October 2, 2020, 4:16pm UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/8 "2020-10-02T16:16:45Z")
Hello Stefano,
you have to run the next stable release (mongodb 4.0, if you are running mongodb 3.6) and upgrade using the db.adminCommand. When done, you can repeat the procedure upgrading to 4.2, and then finally to 4.4.
We are updating our docs to provide guidance in upgrading mongodb, you can track the progress here:
> **[\[MEN-3975\] Upgrade from 2.4.0 to 2.5.0 fails due to mongodb - Northern.tech AS](https://tracker.mender.io/browse/MEN-3975)**
---
### Author:  [@Ks89](https://hub.mender.io/u/Ks89)
#### Post date: [October 26, 2020, 10:46am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/9 "2020-10-26T10:46:14Z")
Oh I forgot to mark it as solution. Done 🙂
thanks.
---
### Author:  [@dimant](https://hub.mender.io/u/dimant)
#### Post date: [November 11, 2020, 7:53am UTC](https://hub.mender.io/t/cannot-upgrade-mender-server-from-2-4-0-to-2-5-0-mongo-container-is-restarting/2506/10 "2020-11-11T07:53:25Z")
Happy to see that the solution has worked for you.
I’m facing your same issue, same log output and mender-mongo keep restarting.
My environment is an azure VM that come out with a storage of 30 GB.
This was the problem, the lack of storage. I’ve solved the issue by:
- stop docker, erase docker disk and image (bacukp it before), erase the content of /var/lib/docker
- adding a storage disk
- partition new disk 50 GB ext4
- mount new partition it to /var/lib/docker (must be \>= 50GB)
- start docker, it will regenerate the files on the new parttion (or you can restore the previously backuped)
now it start with no problem