Cannot upgrade mender server from 2.4.0 to 2.5.0 - mongo container is "restarting"

After the upgrade from mender server open source 2.4.0 to 2.5.0, mongo container is restarting:

menderproduction_mender-api-gateway_1              /entrypoint.sh                   Up             0.0.0.0:443->443/tcp, 80/tcp
menderproduction_mender-create-artifact-worker_1   /usr/bin/workflows --confi ...   Up             8080/tcp
menderproduction_mender-deployments_1              /entrypoint.sh --config /e ...   Restarting
menderproduction_mender-device-auth_1              /usr/bin/deviceauth --conf ...   Up             8080/tcp
menderproduction_mender-gui_1                      /entrypoint.sh nginx             Up (healthy)   80/tcp
menderproduction_mender-inventory_1                /usr/bin/inventory --confi ...   Up             8080/tcp
menderproduction_mender-mongo_1                    docker-entrypoint.sh mongod      Restarting
menderproduction_mender-useradm_1                  /usr/bin/useradm --config  ...   Up             8080/tcp
menderproduction_mender-workflows-server_1         /usr/bin/workflows --confi ...   Up             8080/tcp
menderproduction_mender-workflows-worker_1         /usr/bin/workflows --confi ...   Up
menderproduction_minio_1                           /usr/bin/docker-entrypoint ...   Up (healthy)   9000/tcp
menderproduction_storage-proxy_1                   /usr/local/openresty/bin/o ...   Up             0.0.0.0:9000->9000/tcp

Obviously, also other containers restart because they are not able to connect with mongo.

docker logs idOfContainer returns:

{"t":{"$date":"2020-09-28T08:29:27.068+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2020-09-28T08:29:27.073+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2020-09-28T08:29:27.074+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"9928341b1f12"}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","openSSLVersion":"OpenSSL 1.1.1  11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2020-09-28T08:29:27.075+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-09-28T08:29:27.079+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=3217M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1101}}
{"t":{"$date":"2020-09-28T08:29:27.783+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

What can I do?

Thanks.

I found also this https://tracker.mender.io/projects/MEN/issues/MEN-3975?filter=allopenissues

Just to clarify, did you run all the steps from here: https://docs.mender.io/2.5/server-installation/upgrading?

@tranchitella @merlin any thoughts here?

Drew

yes.
I made a backup of mongo, cleaned deviceauth, updated and restarted as explained in the tutorial.

At the moment I didn’t restore mongo database, I only made backups and upgraded, but if required I can restore it, but I suppose that the problem is not related to the data itself in the db.

@tranchitella @merlin any ideas here?
Drew

@Ks89 ciao Stefano, Mender 2.5 uses mongodb version 4.4, while Mender 2.4 uses mongodb version 3.6; it is not possible to upgrade from mongodb 3.6 to 4.4 directly, you have to manually upgrade the database version by version, 3.6 -> 4.0 -> 4.2 -> 4.4.

The official mongodb documentation applies:



You can manually change the image tag of the mongodb container to 4.0, 4.2 and finally 4.4 as you proceed with the migration of the data, using the db.adminCommand( { setFeatureCompatibilityVersion: "VERSION" } ) command, where VERSION is 4.0, 4.2 and finally 4.4.

I hope it helps.

Hi!

Ok thank you for info

If I modify docker-compose file with mongo:3.6, I able to run it again, however using the older version.
I tried to run the command db.adminCommand( { setFeatureCompatibilityVersion: "4.0" } ) but it throws an error on 3.6, so I changed docker-compose to 4.0 to get the new version (as explained in the guide you posted), however the container is restarting exactly like 4.4.
I cannot run this command on 4.0 “db.adminCommand( { setFeatureCompatibilityVersion: “VERSION” } )” because it throw this error:

Error response from daemon: Container 1a4b848de777f0e0939eae3fcaedfe5ad440b813227fcb0f98265ae955d4c3c3 is restarting, wait until the container is running

Am I missing something? Do you have suggestion?

PS:
I suggest to update the official documentation (upgrade section) with this procedure, because it’s missing right now.

Hello Stefano,

you have to run the next stable release (mongodb 4.0, if you are running mongodb 3.6) and upgrade using the db.adminCommand. When done, you can repeat the procedure upgrading to 4.2, and then finally to 4.4.

We are updating our docs to provide guidance in upgrading mongodb, you can track the progress here:

Oh I forgot to mark it as solution. Done :slight_smile:
thanks.

Happy to see that the solution has worked for you.
I’m facing your same issue, same log output and mender-mongo keep restarting.
My environment is an azure VM that come out with a storage of 30 GB.

This was the problem, the lack of storage. I’ve solved the issue by:

  • stop docker, erase docker disk and image (bacukp it before), erase the content of /var/lib/docker
  • adding a storage disk
  • partition new disk 50 GB ext4
  • mount new partition it to /var/lib/docker (must be >= 50GB)
  • start docker, it will regenerate the files on the new parttion (or you can restore the previously backuped)

now it start with no problem

1 Like