Self-hosted 3.0.0 fails to connect to mongo, demo runs fine

Kris · July 23, 2021, 9:25pm

We’re spinning up a feature bump for a customer’s setup and completely stuck trying to get any recent version of Mender running.

We have a test 2.2.1 server that still runs, but minio complains that the version of Mender is too old.

If I clone and follow the demo instructions for 3.0.0, the demo and all docker containers run without issue.

But following the instructions for self-hosted, verbatim, ends up with a handful of containers complaining they cannot reach mongodb, with mongo itself showing no errors. When in this state, I’m not able to log in with the created user. However, it appears the user create command from commandline is failing. See below for some logs which I assume are relevant. If there is any other detail I can provide, let me know.

OS info:

Linux mender 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 18.04

Create user command return

kris@mender:~/mender-server/production$ ./run exec mender-useradm /usr/bin/useradm create-user --username=myusername@host.com --password=mysecretpassword
kris@mender:~/mender-server/production$ echo $?
137

docker ps

kris@mender:~/mender-server/production$ docker ps -a
CONTAINER ID   IMAGE                                                COMMAND                  CREATED          STATUS                           PORTS                                           NAMES
12d7f5b9ce3a   mendersoftware/deployments:mender-3.0.0              "/entrypoint.sh --co…"   13 minutes ago   Up 2 seconds                     8080/tcp                                        menderproduction_mender-deployments_1
9544e0fca9f5   traefik:v2.4                                         "/entrypoint.sh --ac…"   13 minutes ago   Up 13 minutes                    80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   menderproduction_mender-api-gateway_1
8972d6f36a2b   mendersoftware/deviceauth:mender-3.0.0               "/usr/bin/deviceauth…"   13 minutes ago   Up Less than a second            8080/tcp                                        menderproduction_mender-device-auth_1
f63d7218db97   mendersoftware/deviceconnect:mender-3.0.0            "/usr/bin/deviceconn…"   13 minutes ago   Up 1 second                      8080/tcp                                        menderproduction_mender-deviceconnect_1
58c52b1a3fa9   mendersoftware/deviceconfig:mender-3.0.0             "/usr/bin/deviceconf…"   13 minutes ago   Up 19 seconds                    8080/tcp                                        menderproduction_mender-deviceconfig_1
9d4856e11cbb   mendersoftware/create-artifact-worker:mender-3.0.0   "/usr/bin/workflows …"   13 minutes ago   Up 6 seconds                     8080/tcp                                        menderproduction_mender-create-artifact-worker_1
7e659881fe9c   mendersoftware/inventory:mender-3.0.0                "/usr/bin/inventory …"   13 minutes ago   Up 19 seconds                    8080/tcp                                        menderproduction_mender-inventory_1
c76b16cbc5a2   mendersoftware/useradm:mender-3.0.0                  "/usr/bin/useradm --…"   13 minutes ago   Up 5 seconds                     8080/tcp                                        menderproduction_mender-useradm_1
a6afa115cb40   mendersoftware/workflows:mender-3.0.0                "/usr/bin/workflows …"   14 minutes ago   Up 2 seconds                     8080/tcp                                        menderproduction_mender-workflows-server_1
6bfc5b8df0f7   mendersoftware/workflows-worker:mender-3.0.0         "/usr/bin/workflows …"   14 minutes ago   Up 6 seconds                                                                     menderproduction_mender-workflows-worker_1
05f8916ae244   nats:2.1.9-alpine3.12                                "docker-entrypoint.s…"   14 minutes ago   Up 13 minutes                    4222/tcp, 6222/tcp, 8222/tcp                    menderproduction_mender-nats_1
2002ace8325a   mendersoftware/gui:mender-3.0.0                      "/entrypoint.sh nginx"   14 minutes ago   Up 14 minutes (healthy)          80/tcp, 8080/tcp                                menderproduction_mender-gui_1
e27b5199a6da   minio/minio:RELEASE.2019-04-23T23-50-36Z             "/usr/bin/docker-ent…"   14 minutes ago   Up 13 minutes (healthy)          9000/tcp                                        menderproduction_minio_1
bb1e2f8d642f   mongo:4.4                                            "docker-entrypoint.s…"   14 minutes ago   Restarting (14) 33 seconds ago                                                   menderproduction_mender-mongo_1

mongodb log

kris@mender:~/mender-server/production$ ./run logs mender-mongo
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:19:05.151+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55BDA513572E","b":"55BDA4164000","o":"FD172E",
"s":"_start","s+":"2E"}}}                                                                                                                                                                                                                    
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.704+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"
}                                                                                                                                                                                                                                            
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.706+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}                                 
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.706+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastO
penClient, and tcpFastOpenQueueSize."}                                                                                                                                                                                                       
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.707+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64$
bit","host":"bb1e2f8d642f"}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.707+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.7","gitVersion":"abb6b9c2bf675e9e2aea$
cba05f0f8359d99e203","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.707+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.707+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.709+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEng$
ne":"wiredTiger"}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.710+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. S
ee http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:05.710+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=1460M,session_max=33000,evicti
on=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wai
t=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.120+00:00"},"s":"F",  "c":"STORAGE",  "id":4671205, "ctx":"initandlisten","msg":"This version of MongoDB is too recent to start up on the existing data files. Try Mong
oDB 4.2 or earlier."}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.122+00:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":4671205,"file":"src/mongo/db/storage/wiredtiger/wired
tiger_kv_engine.cpp","line":919}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.123+00:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.123+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.237+00:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"initandlisten","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55F58BAED69A","b":"55F588D44000","o"
:"2DA969A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"55F58BAEF129","b":"55F588D44000","o":"2DAB129","s":"_ZN5mongo15printStackTraceEv","s+"
:"29"},{"a":"55F58BAEC4B6","b":"55F588D44000","o":"2DA84B6","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7F0959F793C0","b":"7F0959F64000","o":"153C0","s":"funlockfile","s+":"60"},{"a":"7F0959DB818B","b"
:"7F0959D72000","o":"4618B","s":"gsignal","s+":"CB"},{"a":"7F0959D97859","b":"7F0959D72000","o":"25859","s":"abort","s+":"12B"},{"a":"55F589C7D788","b":"55F588D44000","o":"F39788","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12
B"},{"a":"55F5899513D7","b":"55F588D44000","o":"C0D3D7","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_.cold.1865","s+":"1C7"},{"a":"55F589D8C642","b":"55F588D44000","o":"1048
642","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb","s+":"1362"},{"a":"55F589D6A3E0","b":"55F588D44000","o":"10263E0","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFa
ctory6createERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","s+":"120"},{"a":"55F58A66ECE7","b":"55F588D44000","o":"192ACE7","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE","s+":"4F7"
},{"a":"55F589D162F7","b":"55F588D44000","o":"FD22F7","s":"_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1584","s+":"3F7"},{"a":"55F589D19F8B","b":"55F588D44000","o":"FD5F8B","s":"_ZN5mongo12_GLOBAL__N_111mongoDbMai
nEiPPcS2_","s+":"104B"},{"a":"55F589C8CE99","b":"55F588D44000","o":"F48E99","s":"main","s+":"9"},{"a":"7F0959D990B3","b":"7F0959D72000","o":"270B3","s":"__libc_start_main","s+":"F3"},{"a":"55F589D1572E","b":"55F588D44000","o":"FD172E","s
":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"4.4.7","gitVersion":"abb6b9c2bf675e9e2aeaecba05f0f8359d99e203","compiledModules":[],"uname":{"sysname":"Linux","release":"4.15.0-151-generic","version":"#157-Ubuntu SMP Fri Jul 9 23
:07:57 UTC 2021","machine":"x86_64"},"somap":[{"b":"55F588D44000","elfType":3,"buildId":"4F8B26A04CD4B51B127724FF7CAE2E74E9FFAF32"},{"b":"7F0959F64000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"E54761F7B554D0F
CC1562959665D93DFFBEBDAF0"},{"b":"7F0959D72000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"099B9225BCB0D019D9D60884BE583EB31BB5F44E"}]}}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F58BAED69A","b":"55F588D44000","o":"2DA969A"
,"s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F58BAEF129","b":"55F588D44000","o":"2DAB129"
,"s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F58BAEC4B6","b":"55F588D44000","o":"2DA84B6"
,"s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F0959F793C0","b":"7F0959F64000","o":"153C0","
s":"funlockfile","s+":"60"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F0959DB818B","b":"7F0959D72000","o":"4618B","
s":"gsignal","s+":"CB"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F0959D97859","b":"7F0959D72000","o":"25859","
s":"abort","s+":"12B"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589C7D788","b":"55F588D44000","o":"F39788",
"s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"12B"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F5899513D7","b":"55F588D44000","o":"C0D3D7",
"s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_.cold.1865","s+":"1C7"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589D8C642","b":"55F588D44000","o":"1048642"
,"s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb","s+":"1362"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589D6A3E0","b":"55F588D44000","o":"10263E0"
,"s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","s+":"120"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F58A66ECE7","b":"55F588D44000","o":"192ACE7"
,"s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE","s+":"4F7"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589D162F7","b":"55F588D44000","o":"FD22F7",
"s":"_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1584","s+":"3F7"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589D19F8B","b":"55F588D44000","o":"FD5F8B",
"s":"_ZN5mongo12_GLOBAL__N_111mongoDbMainEiPPcS2_","s+":"104B"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589C8CE99","b":"55F588D44000","o":"F48E99",
"s":"main","s+":"9"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"7F0959D990B3","b":"7F0959D72000","o":"270B3","
s":"__libc_start_main","s+":"F3"}}}
mender-mongo_1                   | {"t":{"$date":"2021-07-23T21:20:07.240+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"initandlisten","msg":"  Frame: {frame}","attr":{"frame":{"a":"55F589D1572E","b":"55F588D44000","o":"FD172E",
"s":"_start","s+":"2E"}}}

useradm log

mender-useradm_1                 | time="2021-07-23T21:23:18Z" level=info msg="User Administration Service, version unknown starting up"
mender-useradm_1                 | failed to connect to db: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-useradm:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-useradm on 127.0.0.11:53: server misbehaving }, ] }
mender-useradm_1                 | time="2021-07-23T21:23:28Z" level=info msg="User Administration Service, version unknown starting up"
mender-useradm_1                 | failed to connect to db: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-useradm:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-useradm on 127.0.0.11:53: server misbehaving }, ] }
mender-useradm_1                 | time="2021-07-23T21:23:39Z" level=info msg="User Administration Service, version unknown starting up"
mender-useradm_1                 | failed to connect to db: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-useradm:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-useradm on 127.0.0.11:53: server misbehaving }, ] }
mender-useradm_1                 | time="2021-07-23T21:23:50Z" level=info msg="User Administration Service, version unknown starting up"
mender-useradm_1                 | failed to connect to db: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-useradm:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-useradm on 127.0.0.11:53: server misbehaving }, ] }
mender-useradm_1                 | time="2021-07-23T21:24:01Z" level=info msg="User Administration Service, version unknown starting up"
mender-useradm_1                 | failed to connect to db: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-useradm:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-useradm on 127.0.0.11:53: server misbehaving }, ] }
mender-useradm_1                 | time="2021-07-23T21:24:15Z" level=info msg="User Administration Service, version unknown starting up"

deployments log

mender-deployments_1             | time="2021-07-23T21:24:30Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-23T21:24:30Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1             | failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-deployments:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-deployments: Try again }, ] }
mender-deployments_1             | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1             | time="2021-07-23T21:24:41Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-23T21:24:41Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1             | failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-deployments:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-deployments: Try again }, ] }
mender-deployments_1             | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1             | time="2021-07-23T21:24:52Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-23T21:24:52Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1             | failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-deployments:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-deployments: Try again }, ] }
mender-deployments_1             | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1             | time="2021-07-23T21:25:04Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-23T21:25:04Z" level=info msg="Deployments Service, version unknown starting up"

If there is any info that is missing or additional details needed, let me know, I’d be happy to provide.

Kris · July 23, 2021, 10:36pm

I’ve also tried downgrading to 2.7, following the 2.7 installation instructions verbatim, from a fresh clone of the repo, and the same failures seem to occur.

tranchitella · July 24, 2021, 3:35am

@Kris from the error I see the different microservices cannot resolve the mongo aliases (e.g. mongo-useradm, mongo-deployments). These aliases are defined in docker-compose.yml:

github.com

mendersoftware/integration/blob/master/docker-compose.yml#L172

    
      
                  extends:
                      file: common.yml
                      service: mender-base
                  networks:
                      mender:
                          aliases:
                              - mongo-tenantadm
                              - mongo-deployments
                              - mongo-device-auth
                              - mongo-inventory
                              - mongo-useradm
                              - mongo-workflows
          
          
networks:
              mender:

Can you please check those alias are there in your configuration, and you can successfully resolve them from the containers running in your docker composition?

Kris · July 26, 2021, 4:44pm

Thanks for the feedback.

docker-compose.yml is unmodified and indeed has all of those aliases defined.

I’m not sure if the best way to go about testing resolution from within containers, so if the following is not sufficient, let me know what a better way is:

kris@mender:~/mender-server/production$ docker exec -it menderproduction_mender-gui_1 sh
/var/www/mender-gui/dist # ping mongo-useradm
ping: bad address 'mongo-useradm'
/var/www/mender-gui/dist # ping mongo-workflows
ping: bad address 'mongo-workflows'
/var/www/mender-gui/dist # ping mender-gui
PING mender-gui (172.21.0.3): 56 data bytes
64 bytes from 172.21.0.3: seq=0 ttl=64 time=0.062 ms
64 bytes from 172.21.0.3: seq=1 ttl=64 time=0.113 ms
^C
--- mender-gui ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.062/0.087/0.113 ms
/var/www/mender-gui/dist # ping mender-useradm
PING mender-useradm (172.21.0.7): 56 data bytes
64 bytes from 172.21.0.7: seq=0 ttl=64 time=0.148 ms
64 bytes from 172.21.0.7: seq=1 ttl=64 time=0.102 ms
^C
--- mender-useradm ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.102/0.125/0.148 ms
/var/www/mender-gui/dist # ping mongo-deployments
ping: bad address 'mongo-deployments'

It would appear none of the mongo aliases are resolvable. Any clues as to where to look at next?

tranchitella · July 28, 2021, 7:32am

@Kris this is definitely strange, as you pointed out none of the aliases is working. I suppose something is wrong in the networking setup of your Docker daemon. To fix it, from Mender’s PoV, you can override the configuration settings for each of the services.

For example, for deployments:

github.com

mendersoftware/deployments/blob/master/config.yaml#L30

    
      
          # Overwrite with environment variables:
          # - DEPLOYMENTS_HTTPS_CERTIFICATE
          # - DEPLOYMENTS_HTTPS_KEY
          
          
# https:
          #     certificate: /path/to/certificate
          #     key: /path/to/private_key
          
          
# Mongodb connection string
          # Defaults to: "mongodb://mongo-deployments:27017"
          # Overwrite with environment variable: DEPLOYMENTS_MONGO_URL
          
          
mongo-url: "mongodb://mongo-deployments:27017"
          
          
# Enable SSL for mongo connections
          # Defaults to: false
          # Overwrite with environment variable: DEPLOYMENTS_MONGO_SSL
          
          
# mongo_ssl: false
          
          
# SkipVerify controls whether a mongo client verifies the

As you can see, you can set the DEPLOYMENTS_MONGO_URL env for deployments to use mender-mongo instead of mongo-deployments. The same configuration must be done for the other services.

Kris · July 28, 2021, 5:43pm

Setting that env var before starting services seems to have no effect:

kris@mender:~/mender-server/production$ export
declare -x DEPLOYMENTS_MONGO_URL="mongodb://mender-mongo:27017"
...

kris@mender:~/mender-server/production$ ./run up -d

kris@mender:~/mender-server/production$ ./run logs mender-deployments
Attaching to menderproduction_mender-deployments_1
mender-deployments_1             | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1             | time="2021-07-28T17:18:23Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-28T17:18:23Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1             | failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mongo-deployments:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mongo-deployments: Try again }, ] }
...

Any thoughts on the root cause of this? Like I said, this is standard Ubuntu. I suppose I could try spinning it up again from scratch, but this VM is otherwise a base install of 18.04 that was only used for an older 2.2 Mender server test that I’m unable to continue to use due to minio not liking Mender being so old.

tranchitella · July 28, 2021, 7:30pm

@kris the env variable must be set in the docker-compose.yaml file, not in the shell.
As an example:

github.com

mendersoftware/integration/blob/master/docker-compose.yml#L75

    
      
                  - mender-device-auth
                  - mender-gui
                  - mender-useradm
                  - mender-inventory
          
          
#
          # mender-device-auth
          #
          mender-device-auth:
              image: mendersoftware/deviceauth:mender-master
              environment:
                  DEVICEAUTH_ORCHESTRATOR_ADDR: http://mender-workflows-server:8080/
              extends:
                  file: common.yml
                  service: mender-base
              networks:
                  - mender
              depends_on:
                  - mender-mongo
                  - mender-workflows-server

You need to set the deployments environment in the corresponding section:

github.com

mendersoftware/integration/blob/master/docker-compose.yml#L7

    
      
          version: '2.1'
          services:
          
          
    #
              # mender-deployments
              #
              mender-deployments:
                  image: mendersoftware/deployments:mender-master
                  extends:
                      file: common.yml
                      service: mender-base
                  networks:
                      - mender
                  depends_on:
                      - mender-mongo
          
          
    #

Kris · July 28, 2021, 8:38pm

Ah, my mistake. This does show trying to reach the correct name, but same error:

kris@mender:~/mender-server/production$ ./run logs mender-deployments
Attaching to menderproduction_mender-deployments_1
mender-deployments_1             | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1             | time="2021-07-28T19:57:23Z" level=info msg="'presign.secret' not configured. Generating a random secret." file=main.go func=main.doMain.func1 line=99
mender-deployments_1             | time="2021-07-28T19:57:23Z" level=info msg="Deployments Service, version unknown starting up"
mender-deployments_1             | failed to connect to db: Error reaching mongo server: server selection error: context deadline exceeded, current topology: { Type: Unknown, Servers: [{ Addr: mender-mongo:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp: lookup mender-mongo: Try again }, ] }
...

I can get in to mender-deployments container for a few seconds, enough to ping:

kris@mender:~/mender-server/production$ docker exec -it menderproduction_mender-deployments_1 sh
/ # ping mender-mongo
ping: bad address 'mender-mongo'
/ # ping mender-gui
PING mender-gui (172.26.0.4): 56 data bytes
64 bytes from 172.26.0.4: seq=0 ttl=64 time=0.173 ms
64 bytes from 172.26.0.4: seq=1 ttl=64 time=0.083 ms

I am unable to enter mender-mongo in any meaningful way, it remains in the “Restarting” state for so long:

Error response from daemon: Container 1ea664a92ecc3e02d078f15726dceb41746fd5f9fb2d2b0f09662f92657b8cf7 is restarting, wait until the container is running

Looking at the logs for mongo, I noticed this line this time around, not sure if this has significance or what do to about it:

mender-mongo_1                   | {"t":{"$date":"2021-07-28T20:04:43.007+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=1460M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
mender-mongo_1                   | {"t":{"$date":"2021-07-28T20:04:43.603+00:00"},"s":"F",  "c":"STORAGE",  "id":4671205, "ctx":"initandlisten","msg":"This version of MongoDB is too recent to start up on the existing data files. Try MongoDB 4.2 or earlier."}
mender-mongo_1                   | {"t":{"$date":"2021-07-28T20:04:43.603+00:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":4671205,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":919}}
mender-mongo_1                   | {"t":{"$date":"2021-07-28T20:04:43.603+00:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
mender-mongo_1                   | {"t":{"$date":"2021-07-28T20:04:43.603+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

So, I blasted away all of /var/lib/docker/volumes/ re-installed all of the docker components, re-ran the pulls and setup, and, everything is happy now. It looks like mongo was trying to use the old database despite being a different container, tags, etc. So that was a bit of a pain.

kris@mender:~$ docker ps
CONTAINER ID   IMAGE                                                COMMAND                  CREATED         STATUS                   PORTS                                           NAMES
a18fc8d330de   mendersoftware/deployments:mender-3.0.0              "/entrypoint.sh --co…"   3 minutes ago   Up 3 minutes             8080/tcp                                        menderproduction_mender-deployments_1
ee8edd5aec5f   traefik:v2.4                                         "/entrypoint.sh --ac…"   4 minutes ago   Up 4 minutes             80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   menderproduction_mender-api-gateway_1
e618e45be50e   mendersoftware/deviceauth:mender-3.0.0               "/usr/bin/deviceauth…"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-device-auth_1
6380c9c7fa6c   mendersoftware/create-artifact-worker:mender-3.0.0   "/usr/bin/workflows …"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-create-artifact-worker_1
147d7df37751   mendersoftware/useradm:mender-3.0.0                  "/usr/bin/useradm --…"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-useradm_1
5f515c28c6d2   mendersoftware/workflows-worker:mender-3.0.0         "/usr/bin/workflows …"   4 minutes ago   Up 4 minutes                                                             menderproduction_mender-workflows-worker_1
2cef524b48eb   mendersoftware/inventory:mender-3.0.0                "/usr/bin/inventory …"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-inventory_1
11d235c3f0b2   mendersoftware/deviceconnect:mender-3.0.0            "/usr/bin/deviceconn…"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-deviceconnect_1
cc39c18c230e   mendersoftware/workflows:mender-3.0.0                "/usr/bin/workflows …"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-workflows-server_1
88026b4fc785   mendersoftware/deviceconfig:mender-3.0.0             "/usr/bin/deviceconf…"   4 minutes ago   Up 4 minutes             8080/tcp                                        menderproduction_mender-deviceconfig_1
e700c663e6da   mongo:4.4                                            "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes             27017/tcp                                       menderproduction_mender-mongo_1
6d06c563cfb3   mendersoftware/gui:mender-3.0.0                      "/entrypoint.sh nginx"   7 minutes ago   Up 4 minutes (healthy)   80/tcp, 8080/tcp                                menderproduction_mender-gui_1
125eba9430f7   nats:2.1.9-alpine3.12                                "docker-entrypoint.s…"   7 minutes ago   Up 4 minutes             4222/tcp, 6222/tcp, 8222/tcp                    menderproduction_mender-nats_1
0fd4f62450d0   minio/minio:RELEASE.2019-04-23T23-50-36Z             "/usr/bin/docker-ent…"   7 minutes ago   Up 4 minutes (healthy)   9000/tcp                                        menderproduction_minio_1

Thanks for your suggestions! Hope this helps someone else coming from an older install.

Topic		Replies	Views
Demo Server 2.3.0 General Discussions	8	613	April 16, 2020
Unable to start demo General Discussions	15	1178	April 27, 2020
[Self-hosted Mender 1.7] useradm failed to open mgo session for remote db General Discussions	1	332	February 20, 2019
There is a problem while logging in General Discussions	9	271	August 29, 2023
Demo server: docker exec error: 137 General Discussions	2	542	March 20, 2020

Self-hosted 3.0.0 fails to connect to mongo, demo runs fine

Related Topics