Unable to start on-premise demo server per "Getting Started" instructions

Kris · January 8, 2020, 9:31pm

I originally posted this on tracker.mender.io (https://tracker.mender.io/browse/MEN-3022) and ultimately they asked me to come here.
Basically, after following the official guide under a number of different configurations, I’m unable to run the Mender server in a VirtualBox VM. The output below is all that I ever get:

./demo up
Pulling mender-gui           ... done 
Pulling mender-mongo         ... done 
Pulling mender-useradm       ... done 
Pulling mender-inventory     ... done 
Pulling mender-deployments   ... done 
Pulling mender-elasticsearch ... done 
Pulling mender-redis         ... done 
Pulling mender-conductor     ... done 
Pulling mender-device-auth   ... done 
Pulling mender-api-gateway   ... done 
Starting the Mender demo environment... 
Creating network "integration221_mender" with the default driver 
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)... 
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio 
c67f3896b22c: Pull complete 
28537c0b7408: Pull complete 
57b9c24c4fa7: Pull complete 
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0 
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z 
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)... 
1.13.6.2-0-alpine: Pulling from openresty/openresty 
ff3a5c916c92: Already exists 
ede0a2a1012b: Already exists 
0e0a11843023: Already exists 
246b2c6f4992: Already exists 
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2 
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine 
Creating integration221_minio_1                ... done 
Creating integration221_mender-elasticsearch_1 ... done 
Creating integration221_mender-redis_1         ... done 
Creating integration221_mender-mongo_1         ... done 
Creating integration221_mender-gui_1           ... done 
Creating integration221_storage-proxy_1        ... done 
Creating integration221_mender-useradm_1       ... done 
Creating integration221_mender-inventory_1     ... done 
Creating integration221_mender-deployments_1   ... done 
Creating integration221_mender-conductor_1     ... done 
Creating integration221_mender-device-auth_1   ... done 
Creating integration221_mender-api-gateway_1   ... done 
It does not seem the useradm service is up and running yet. Retrying... 
It does not seem the useradm service is up and running yet. Retrying... 
It does not seem the useradm service is up and running yet. Retrying... 
It does not seem the useradm service is up and running yet. Retrying... 
It does not seem the useradm service is up and running yet. Retrying... 
It does not seem the useradm service is up and running yet. Retrying... 
Retried 5 times without success. Giving up.

This is running on Windows 7 Host, tested against the following guests:

Ubuntu 18.04 amd64 
Debian Stretch amd64
Debian Buster amd64

Using the following Virtual Box releases:

VirtualBox Version 5.2.18 r124319 (Qt5.6.2)
VirtualBox Version 6.1.0 r135406 (Qt5.6.2)

And the following network configurations:

Bridged
NAT

The only way the demo server was able to run, was on a native install of Debian Buster amd64 which worked the first time with no hassle. However, I need this to be running in a VM in order to move forward. The original issue I opened also referenced this: https://mender.io/blog/getting-started-with-mender-2-1-beta-on-windows which I followed to the letter and again had no success; every attempt ends with the same error as noted above.

Does anyone have any idea why this wouldn’t be working as-is? Is there some VM or specific VirtualBox tweak/setting that needs to be used?

A coworker did initial testing about 8-12 months back and was able to get it up and running without issue. However, that was long enough ago that all of those setup details were lost and it was just assumed that it will always work moving forward. This was also done with Virtual Box as well.

peter · January 13, 2020, 8:16am

Hello and thanks for trying Mender.

Since this seems to be hanging on docker pull, I would like you to add:

set -x

on the second line of ./demo script and post the results.

peter

0lmi · January 13, 2020, 11:13pm

@Kris can you please share the output of docker ps -a and docker logs integration221_mender-useradm_1 commands?

Kris · January 14, 2020, 8:11pm

@peter
Output of ‘./demo up’ with “set -x” added:

$ ./demo up
+ ./verify-docker-versions
+ '[' 1 -eq 0 ']'
+ '[' up = -h ']'
+ '[' up = --help ']'
++ tr -d .-
+ DOCKER_COMPOSE_PROJECT_NAME=integration221
+ CLIENT=
+ CLIENT_ARGS='-f docker-compose.client.yml -f docker-compose.client.demo.yml'
+ hash curl
+ hash jq
+ '[' -n up ']'
+ '[' up = --no-client ']'
+ '[' up = --client ']'
+ '[' up = -p ']'
+ '[' up = --project-name ']'
+ '[' up = --kvm ']'
+ '[' up = down ']'
+ '[' up = rm ']'
+ '[' up = stop ']'
+ break
+ DEMO_ARTIFACT_NAME=mender-demo-artifact.mender
+ curl -sz mender-demo-artifact.mender -o mender-demo-artifact.mender https://dgsbl4vditpls.cloudfront.net/mender-demo-artifact.mender
+ retval=0
+ '[' 0 -ne 0 ']'
+ [[ linux-gnu == \d\a\r\w\i\n* ]]
++ stat -c %s mender-demo-artifact.mender
+ ARTIFACT_SIZE_BYTES=2002432
++ awk '{print $7;exit}'
++ ip route get 1
+ export GATEWAY_IP=10.0.2.15
+ GATEWAY_IP=10.0.2.15
+ grep -q mendersoftware/deployments
+ docker images
+ [[ 1 -eq 1 ]]
+ compose_args=
++ docker-compose pull -h
+ docker_compose_output='Pulls images for services defined in a Compose file, but does not start the containers.

Usage: pull [options] [SERVICE...]

Options:
    --ignore-pull-failures  Pull what it can and ignores images with pull failures.
    --parallel              Deprecated, pull multiple images in parallel (enabled by default).
    --no-parallel           Disable parallel pulling.
    -q, --quiet             Pull without printing progress information
    --include-deps          Also pull services declared as dependencies'
+ grep -q -- --no-parallel
+ echo 'Pulls images for services defined in a Compose file, but does not start the containers.

Usage: pull [options] [SERVICE...]

Options:
    --ignore-pull-failures  Pull what it can and ignores images with pull failures.
    --parallel              Deprecated, pull multiple images in parallel (enabled by default).
    --no-parallel           Disable parallel pulling.
    -q, --quiet             Pull without printing progress information
    --include-deps          Also pull services declared as dependencies'
+ [[ 0 -eq 1 ]]
+ docker-compose pull
Pulling mender-gui           ... done
Pulling mender-mongo         ... done
Pulling mender-useradm       ... done
Pulling mender-inventory     ... done
Pulling mender-deployments   ... done
Pulling mender-elasticsearch ... done
Pulling mender-redis         ... done
Pulling mender-conductor     ... done
Pulling mender-device-auth   ... done
Pulling mender-api-gateway   ... done
++ git describe --tags --abbrev=0
+ export INTEGRATION_VERSION=2.2.1
+ INTEGRATION_VERSION=2.2.1
++ extra/release_tool.py -g mender-artifact
+ export MENDER_ARTIFACT_VERSION=3.2.1
+ MENDER_ARTIFACT_VERSION=3.2.1
++ extra/release_tool.py -g mender
+ export MENDER_VERSION=2.1.2
+ MENDER_VERSION=2.1.2
+ export MENDER_DEB_PACKAGE_VERSION=2.1.2
+ MENDER_DEB_PACKAGE_VERSION=2.1.2
++ grep '\bup\b'
++ echo up
+ RUN_UP=up
+ [[ up == '' ]]
+ MENDER_SERVER_URI=https://localhost
+ trap exitfunc SIGINT
+ trap exitfunc SIGTERM
+ echo 'Starting the Mender demo environment...'
Starting the Mender demo environment...
+ docker-compose -f docker-compose.yml -f docker-compose.storage.minio.yml -f docker-compose.demo.yml -p integration221 up -d
Creating network "integration221_mender" with the default driver
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)...
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio
c67f3896b22c: Pull complete
28537c0b7408: Pull complete
57b9c24c4fa7: Pull complete
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)...
1.13.6.2-0-alpine: Pulling from openresty/openresty
ff3a5c916c92: Already exists
ede0a2a1012b: Already exists
0e0a11843023: Already exists
246b2c6f4992: Already exists
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine
Creating integration221_mender-mongo_1         ... done
Creating integration221_mender-gui_1           ... done
Creating integration221_minio_1                ... done
Creating integration221_mender-redis_1         ... done
Creating integration221_mender-elasticsearch_1 ... done
Creating integration221_storage-proxy_1        ... done
Creating integration221_mender-conductor_1     ... done
Creating integration221_mender-useradm_1       ... done
Creating integration221_mender-deployments_1   ... done
Creating integration221_mender-inventory_1     ... done
Creating integration221_mender-device-auth_1   ... done
Creating integration221_mender-api-gateway_1   ... done
+ retval=0
+ [[ 0 -ne 0 ]]
+ USER=mender-demo@example.com
++ head -c 12
++ base64 /dev/urandom
+ PASSWORD=Upx6eJLWlFxZ
+ RETRY_LIMIT=5
+ RETRIES=0
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 0 -ge 5 ]]
+ RETRIES=1
+ sleep 5
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 1 -ge 5 ]]
+ RETRIES=2
+ sleep 5
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 2 -ge 5 ]]
+ RETRIES=3
+ sleep 5
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 3 -ge 5 ]]
+ RETRIES=4
+ sleep 5
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 4 -ge 5 ]]
+ RETRIES=5
+ sleep 5
+ :
+ curl --silent -k -X POST -u mender-demo@example.com:Upx6eJLWlFxZ --fail --connect-timeout 5 https://localhost/api/management/v1/useradm/auth/login
+ retval=35
+ case $retval in
+ echo 'It does not seem the useradm service is up and running yet. Retrying...'
It does not seem the useradm service is up and running yet. Retrying...
+ [[ 5 -ge 5 ]]
+ echo 'Retried 5 times without success. Giving up.'
Retried 5 times without success. Giving up.
+ exit 1

Kris · January 14, 2020, 8:16pm

@0lmi

Output of docker ps -a:

CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS                            PORTS                              NAMES
88abd7b4acc8        mendersoftware/api-gateway:2.0.0           "/entrypoint.sh"         59 minutes ago      Restarting (132) 49 seconds ago                                      integration221_mender-api-gateway_1
069d0a192439        mendersoftware/deviceauth:2.1.0            "/usr/bin/deviceauth…"   59 minutes ago      Up 59 minutes                     8080/tcp                           integration221_mender-device-auth_1
170e65059d61        mendersoftware/useradm:1.9.1               "/usr/bin/useradm --…"   59 minutes ago      Up 59 minutes                     8080/tcp                           integration221_mender-useradm_1
1b521cae8de8        mendersoftware/deployments:1.8.1           "/entrypoint.sh --co…"   59 minutes ago      Restarting (4) 43 seconds ago                                        integration221_mender-deployments_1
4df4f8b1dc8e        mendersoftware/inventory:1.6.0             "/usr/bin/inventory …"   59 minutes ago      Up 59 minutes                     8080/tcp                           integration221_mender-inventory_1
a7b8a72743ba        mendersoftware/mender-conductor:1.5.0      "/srv/start_conducto…"   59 minutes ago      Up 59 minutes                     0.0.0.0:8080->8080/tcp, 8090/tcp   integration221_mender-conductor_1
e4a2c164aab4        openresty/openresty:1.13.6.2-0-alpine      "/usr/local/openrest…"   59 minutes ago      Exited (132) 59 minutes ago                                          integration221_storage-proxy_1
1bc6df092a43        minio/minio:RELEASE.2018-09-25T21-34-43Z   "/usr/bin/docker-ent…"   59 minutes ago      Up 59 minutes (healthy)           9000/tcp                           integration221_minio_1
a297a49a13fa        redis:5-alpine3.8                          "/redis/entrypoint.sh"   59 minutes ago      Up 59 minutes                     6379/tcp                           integration221_mender-redis_1
7b97559b3a3c        elasticsearch:5-alpine                     "/docker-entrypoint.…"   59 minutes ago      Up 59 minutes                     9200/tcp, 9300/tcp                 integration221_mender-elasticsearch_1
f8e0e7df52f0        mendersoftware/gui:2.2.1                   "/entrypoint.sh nginx"   59 minutes ago      Up 59 minutes                     80/tcp                             integration221_mender-gui_1
452a584d355c        mongo:3.4                                  "docker-entrypoint.s…"   59 minutes ago      Up 59 minutes                     27017/tcp                          integration221_mender-mongo_1
82ba7751afac        hello-world                                "/hello"                 About an hour ago   Exited (0) About an hour ago                                         ecstatic_engelbart

Output of docker logs integration221_mender-useradm_1

time="2020-01-14T19:13:18Z" level=info msg="User Administration Service, version unknown starting up" file=main.go func=main.runServer line=154
time="2020-01-14T19:13:18Z" level=info msg="running migrations in single tenant mode" file=datastore_mongo.go func="mongo.(*DataStoreMongo).Migrate" line=358
time="2020-01-14T19:13:18Z" level=info msg="automigrate is ON, will apply migrations" file=datastore_mongo.go func="mongo.(*DataStoreMongo).Migrate" line=362
time="2020-01-14T19:13:18Z" level=info msg="migrating useradm" file=datastore_mongo.go func="mongo.(*DataStoreMongo).Migrate" line=368
time="2020-01-14T19:13:18Z" level=info msg="applying migration from version 0.0.0 to 0.1.0" db=useradm file=migrator_dummy.go func="migrate.(*DummyMigrator).Apply" line=61
time="2020-01-14T19:13:18Z" level=info msg="setting up prod middleware" file=middleware.go func=main.SetupMiddleware line=136
time="2020-01-14T19:13:18Z" level=info msg="listening on :8080" file=server.go func=main.RunServer line=95

0lmi · January 14, 2020, 10:52pm

@Kris Can you also please post the logs for the services which has status “Exited” or “Restarting” (docker logs <service_name>). In this case they are integration221_mender-api-gateway_1, integration221_mender-deployments_1 and integration221_storage-proxy_1.

Kris · January 20, 2020, 7:48pm

@0lmi

Thanks for the help. Below is the requested output.

Output of docker logs integration221_mender-deployments_1:

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:06Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:06Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:06Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:06Z" level=info msg="applying migration from version 0.0.0 to 1.2.1" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=101
time="2020-01-20T19:36:06Z" level=info msg="applying migration from version 1.2.1 to 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=101
time="2020-01-20T19:36:06Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:15Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:15Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:15Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:15Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:15Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:15Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:21Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:21Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:21Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:21Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:21Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:21Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:29Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:29Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:29Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:29Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:29Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:29Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:35Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:35Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:35Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:35Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:35Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:35Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:42Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:42Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:42Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:42Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:42Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:42Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:50Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:50Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:50Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:50Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:50Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:50Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:00Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:00Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:00Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:00Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:00Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:00Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:16Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:16Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:16Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:16Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:16Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:16Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:43Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:43Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:43Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:43Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:43Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:43Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:38:36Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:38:36Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:38:36Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:38:36Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:38:36Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:38:36Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:39:38Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:39:38Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:39:38Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:39:38Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:39:38Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:39:38Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:40:40Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:40:40Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:40:40Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:40:40Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:40:40Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:40:40Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:41:42Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:41:42Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:41:42Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:41:42Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:41:42Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:41:42Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:42:44Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:42:44Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:42:44Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:42:44Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:42:44Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:42:44Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host

Output of docker logs integration221_mender-api-gateway_1

setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui

Output of docker logs integration221_storage-proxy_1 is completely empty.

0lmi · January 21, 2020, 6:09pm

Looks like a mistake in configuration. Hard to say where exactly but it’s definitely something messed up with certificates.

If you go through the documentation and copy/paste commands from there then it just works. You probably missed something or made a mistake/typo somewhere.

Kris · January 21, 2020, 6:48pm

@0lmi, I appreciate the feedback, but, nope, that’s not a situation that has occurred. If it is a configuration issue, it is through no fault of mine in setting up Mender (could still be something else VM related still, though).

I’ve run the commands, copy/pasted, directly from the Mender documentation as well as docker documentation (as linked by the Mender manual). At this point, probably close to 2 dozen times. The ONLY situation where it has worked, is when I did it, the same way as I always have, on a native Debian Buster installation. First installation as complex as this, I follow the manual to the letter, and resolve errors as I can as they come up. And, between each test, I revert the VM to a prior snapshot, before ANYTHING related to mender/docker was installed. So, the instructions need to be copy/pasted each time I do it as nothing is even in history.

I would gladly provide you with ‘history’ output so you can check this; or if you prefer, you are welcome to copy/paste commands here and I will input them exactly in to my terminal to test again.

EDIT: Added a clarification of testing process.

0lmi · January 21, 2020, 8:02pm

In deployments service logs you see
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host

It means s3.docker.mender.io domain name is used for Minio (works through storage proxy) and it’s linked to localhost. Both are incorrect, first it has to be your’s FQDN and it should link to real IP if you want to deliver updates to your devices. In particular configuration is described in the following sections:

Minio - https://docs.mender.io/2.2/administration/production-installation#minio
Deployments service - https://docs.mender.io/2.2/administration/production-installation#deployments-service
Storage proxy - https://docs.mender.io/2.2/administration/production-installation#storage-proxy
API gateway - https://docs.mender.io/2.2/administration/production-installation#api-gateway

All of those are related to the issues you have (Minio is related to storage proxy and deployments services).

Kris · January 21, 2020, 8:26pm

@0lmi Can you please clarify what you mean?

The documentation states: “Mender currently requires two entries in your /etc/hosts file to work with the Docker networking (typically 127.0.0.1 s3.docker.mender.io and 127.0.0.1 docker.mender.io).” https://docs.mender.io/2.2/getting-started/on-premise-installation/create-a-test-environment

This is confusing because:

I never manually have added those entries to /etc/hosts, in my VMs or on the working native install, and they never appear (even though the docs say they will be auto added by ‘./demo up’).
The documentation seems like it contradicts what you just said as the demo server WANTS those to link to localhost. If I’m misunderstanding something there, I would appreciate a correction.

0lmi · January 21, 2020, 8:42pm

Looks like you are following by the wrong document

Very first line in Install a Mender demo server guide is:
Following this tutorial will create a demo installation of the Mender, appropriate for testing and experimenting. When you are ready to install for production, please follow the Production installation documentation.

There is the link to Production installation guide which you need to follow by - https://docs.mender.io/2.2/administration/production-installation

Kris · January 21, 2020, 8:58pm

@0lmi Yes, testing and experimenting is precisely what I’m trying to do. Its in the title of this post even. I’m working on bootstraping a brand new device, having never used Mender before, and am attempting to explore and understand how this whole process works so I can roll this out without issue.

Additionally, looking at the production guide, the steps are the same up until the point of running ‘./demo up’ vs creating a custom production setup. So, in theory, following either guide should allow the demo server to run.

Is there some issue with bringing up a demo server that I’m unaware of? Note from my original output that the demo server cannot even connect to itself. I cannot log in to the demo server. I’m not even to the point of having a real client to connect to it.

EDIT: I looked at the prod. guide and added a comment about it.

0lmi · January 22, 2020, 12:50pm

@Kris I checked Mender demo script by running a VM on aws, installing everything by the script and everything works without any issues. Looks like this is something with your local virtual environment, linux, etc…

Steps to reproduce:

aws ec2 run-instances \
  --image-id <ubuntu-18.04-instance-id-in-your-region> \
  --count 1 \
  --instance-type t3.medium \
  --key-name <key-name> \
 --user-data https://raw.githubusercontent.com/0lmi/mender-it/master/server/scripts/mender-demo-server-setup.sh

then ssh to the server and check sudo docker ps -a

P.S.: image-id will be different for different regions (in my case it’s ami-0ac05733838eabc06 in eu-central-1 region), so, make sure you specify the correct one

Kris · January 22, 2020, 5:09pm

@0lmi Great! Do you have any comments on what that incorrect configuration might be? As noted in my initial post here, a coworker was able to run through the demo steps some months ago. I pinged them and found the VM was run via ESXi while I’ve got a VirtualBox setup here as that is likely what our customer will be using.

I also linked to instructions using an Ubuntu guest on a Win10 host via VirtualBox that worked but did not really highlight much of the configuration. As I said, I am using Win 7 host and I’m not sure if that impacts anything. I’ve also tried the different networking modes as listed. I’ve also tested in VMWare Player with the same failures as above. Additionally, the guest OS installations are fresh, with Docker and Mender being the first thing installed.

Thanks for any further insight you may be able to provide!

0lmi · January 22, 2020, 5:39pm

@Kris I would recommend you to create a new VM (with at least 2 CPUs and 4Gb of RAM), install Ubuntu 18.04 from scratch and use mentioned in my previous post script for configuring and preparing the VM (with just commenting last line in it).

p.s.: you can also try to do the same VM configuration like in the ticket which you mentioned in first post, there are posted screenshots of working win10+virtualbox setup

Kris · January 22, 2020, 8:18pm

@0lmi

I reverted my 18.04 snapshot back to its post-installation state, updated the system to the latest, and then download and ran your script; same issue. I made two modifications, adding ‘set -x’ near the top, and changing the absolute path with ‘cd’ to relative because of username differences. See the output below.

As I said, I’m not able to use Win10 host right now. I would be surprised if thats what fixes things in this situation.

karl@karl:~$ ./mender-demo-server-setup.sh 
+ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease      
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 163 kB in 1s (159 kB/s)                                               
Reading package lists... Done
+ sudo apt-get install -y git apt-transport-https ca-certificates curl gnupg-agent software-properties-common jq
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20180409).
ca-certificates set to manually installed.
software-properties-common is already the newest version (0.96.24.32.12).
software-properties-common set to manually installed.
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5
  libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0
  libe-book-0.1-1 libedataserverui-1.2-2 libeot0 libepubgen-0.1-1
  libetonyek-0.1-1 libevent-2.1-6 libexiv2-14 libfreerdp-client2-2
  libfreerdp2-2 libgc1c2 libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6
  libgpod-common libgpod4 liblangtag-common liblangtag1 liblirc-client0
  libllvm8 liblua5.3-0 libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1
  libqqwing2v5 libraw16 librevenge-0.0-0 libsgutils2-2 libssh-4
  libsuitesparseconfig5 libvncclient1 libwinpr2-2 libxapian30 libxmlsec1
  libxmlsec1-nss lp-solve media-player-info python3-mako python3-markupsafe
  syslinux syslinux-common syslinux-legacy usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  git-man libcurl4 liberror-perl libjq1 libonig4
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  apt-transport-https curl git git-man gnupg-agent jq libcurl4 liberror-perl
  libjq1 libonig4
0 upgraded, 10 newly installed, 0 to remove and 3 not upgraded.
Need to get 5,393 kB of archives.
After this operation, 36.1 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 apt-transport-https all 1.6.12 [1,692 B]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.8 [214 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 curl amd64 7.58.0-2ubuntu3.8 [159 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 liberror-perl all 0.17025-1 [22.8 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git-man all 1:2.17.1-1ubuntu0.5 [803 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.5 [3,912 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 gnupg-agent all 2.2.4-1ubuntu1.2 [4,880 B]
Get:8 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB]
Fetched 5,393 kB in 3s (1,991 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 149809 files and directories currently installed.)
Preparing to unpack .../0-apt-transport-https_1.6.12_all.deb ...
Unpacking apt-transport-https (1.6.12) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../1-libcurl4_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package curl.
Preparing to unpack .../2-curl_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../3-liberror-perl_0.17025-1_all.deb ...
Unpacking liberror-perl (0.17025-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../4-git-man_1%3a2.17.1-1ubuntu0.5_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package git.
Preparing to unpack .../5-git_1%3a2.17.1-1ubuntu0.5_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package gnupg-agent.
Preparing to unpack .../6-gnupg-agent_2.2.4-1ubuntu1.2_all.deb ...
Unpacking gnupg-agent (2.2.4-1ubuntu1.2) ...
Selecting previously unselected package libonig4:amd64.
Preparing to unpack .../7-libonig4_6.7.0-1_amd64.deb ...
Unpacking libonig4:amd64 (6.7.0-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../8-libjq1_1.5+dfsg-2_amd64.deb ...
Unpacking libjq1:amd64 (1.5+dfsg-2) ...
Selecting previously unselected package jq.
Preparing to unpack .../9-jq_1.5+dfsg-2_amd64.deb ...
Unpacking jq (1.5+dfsg-2) ...
Setting up git-man (1:2.17.1-1ubuntu0.5) ...
Setting up apt-transport-https (1.6.12) ...
Setting up liberror-perl (0.17025-1) ...
Setting up libonig4:amd64 (6.7.0-1) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Setting up libjq1:amd64 (1.5+dfsg-2) ...
Setting up gnupg-agent (2.2.4-1ubuntu1.2) ...
Setting up git (1:2.17.1-1ubuntu0.5) ...
Setting up jq (1.5+dfsg-2) ...
Setting up curl (7.58.0-2ubuntu3.8) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ sudo apt-key add -
+ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg
OK
++ lsb_release -cs
+ sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu    bionic    stable'
Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                    
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]   
Get:4 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [9,852 B]
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease            
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 238 kB in 1s (200 kB/s)                                               
Reading package lists... Done
+ sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                    
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]   
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]  
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 252 kB in 1s (215 kB/s)                               
Reading package lists... Done
+ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5
  libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0
  libe-book-0.1-1 libedataserverui-1.2-2 libeot0 libepubgen-0.1-1
  libetonyek-0.1-1 libevent-2.1-6 libexiv2-14 libfreerdp-client2-2
  libfreerdp2-2 libgc1c2 libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6
  libgpod-common libgpod4 liblangtag-common liblangtag1 liblirc-client0
  libllvm8 liblua5.3-0 libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1
  libqqwing2v5 libraw16 librevenge-0.0-0 libsgutils2-2 libssh-4
  libsuitesparseconfig5 libvncclient1 libwinpr2-2 libxapian30 libxmlsec1
  libxmlsec1-nss lp-solve media-player-info python3-mako python3-markupsafe
  syslinux syslinux-common syslinux-legacy usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  aufs-tools cgroupfs-mount pigz
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli pigz
0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.
Need to get 85.5 MB of archives.
After this operation, 384 MB of additional disk space will be used.
Get:1 https://download.docker.com/linux/ubuntu bionic/stable amd64 containerd.io amd64 1.2.10-3 [20.0 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 aufs-tools amd64 1:4.9+20170918-1ubuntu1 [104 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B]
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce-cli amd64 5:19.03.5~3-0~ubuntu-bionic [42.5 MB]
Get:6 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce amd64 5:19.03.5~3-0~ubuntu-bionic [22.8 MB]
Fetched 85.5 MB in 8s (10.3 MB/s)                                             
Selecting previously unselected package pigz.
(Reading database ... 150754 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package aufs-tools.
Preparing to unpack .../1-aufs-tools_1%3a4.9+20170918-1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:4.9+20170918-1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ...
Unpacking cgroupfs-mount (1.4) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../3-containerd.io_1.2.10-3_amd64.deb ...
Unpacking containerd.io (1.2.10-3) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../4-docker-ce-cli_5%3a19.03.5~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.5~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../5-docker-ce_5%3a19.03.5~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ...
Setting up containerd.io (1.2.10-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up cgroupfs-mount (1.4) ...
Setting up docker-ce-cli (5:19.03.5~3-0~ubuntu-bionic) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
++ uname -s
++ uname -m
+ sudo curl -L https://github.com/docker/compose/releases/download/1.25.1/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0   1757      0 --:--:-- --:--:-- --:--:--  1752
100 16.2M  100 16.2M    0     0  2456k      0  0:00:06  0:00:06 --:--:-- 2889k
+ sudo chmod +x /usr/local/bin/docker-compose
+ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+ cd
+ git clone -b 2.2.1 https://github.com/mendersoftware/integration.git integration-2.2.1
Cloning into 'integration-2.2.1'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 8899 (delta 5), reused 5 (delta 1), pack-reused 8873
Receiving objects: 100% (8899/8899), 2.24 MiB | 6.06 MiB/s, done.
Resolving deltas: 100% (6069/6069), done.
Note: checking out 'a9216ae3e0c27166a10e58880407af8a3732e23b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

karl@karl:~$ + sudo bash -c 'cd integration-2.2.1 && ./demo up'
Pulling mender-gui           ... done
Pulling mender-mongo         ... done
Pulling mender-useradm       ... done
Pulling mender-inventory     ... done
Pulling mender-deployments   ... done
Pulling mender-elasticsearch ... done
Pulling mender-redis         ... done
Pulling mender-conductor     ... done
Pulling mender-device-auth   ... done
Pulling mender-api-gateway   ... done
Starting the Mender demo environment...
Creating network "integration221_mender" with the default driver
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)...
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio
c67f3896b22c: Pull complete
28537c0b7408: Pull complete
57b9c24c4fa7: Pull complete
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)...
1.13.6.2-0-alpine: Pulling from openresty/openresty
ff3a5c916c92: Already exists
ede0a2a1012b: Already exists
0e0a11843023: Already exists
246b2c6f4992: Already exists
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine
Creating integration221_minio_1                ... done
Creating integration221_mender-elasticsearch_1 ... done
Creating integration221_mender-mongo_1         ... done
Creating integration221_mender-gui_1           ... done
Creating integration221_mender-redis_1         ... done
Creating integration221_mender-inventory_1     ... done
Creating integration221_mender-deployments_1   ... done
Creating integration221_mender-useradm_1       ... done
Creating integration221_storage-proxy_1        ... done
Creating integration221_mender-conductor_1     ... done
Creating integration221_mender-device-auth_1   ... done
Creating integration221_mender-api-gateway_1   ... done
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
Retried 5 times without success. Giving up.

0lmi · January 22, 2020, 8:55pm

Can you show output of the following commands:

docker ps -a
cat /proc/cpuinfo
top -n1

Kris · February 14, 2020, 11:23pm

@0lmi Sorry for the delay; was moved off this project for a bit, have time to look in to it again.

Same host settings, same guest settings, did a fresh install again of Ubuntu 18.04, used your script (basically the same output, but, I’ve added it below again) as well as the commands you requested. Same issues still.

Installation:

karl@mender-test:~$ ./mender-demo-server-setup.sh 
+ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 252 kB in 1s (296 kB/s)                                
Reading package lists... Done
+ sudo apt-get install -y git apt-transport-https ca-certificates curl gnupg-agent software-properties-common jq
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20180409).
ca-certificates set to manually installed.
software-properties-common is already the newest version (0.96.24.32.12).
software-properties-common set to manually installed.
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5 libclucene-core1v5
  libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0 libe-book-0.1-1
  libedataserverui-1.2-2 libeot0 libepubgen-0.1-1 libetonyek-0.1-1
  libevent-2.1-6 libexiv2-14 libfreerdp-client2-2 libfreerdp2-2 libgc1c2
  libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6 libgpod-common libgpod4
  liblangtag-common liblangtag1 liblirc-client0 libllvm8 liblua5.3-0
  libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1 libqqwing2v5 libraw16
  librevenge-0.0-0 libsgutils2-2 libssh-4 libsuitesparseconfig5 libvncclient1
  libwinpr2-2 libxapian30 libxmlsec1 libxmlsec1-nss lp-solve media-player-info
  python3-mako python3-markupsafe syslinux syslinux-common syslinux-legacy
  usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  git-man libcurl4 liberror-perl libjq1 libonig4
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  apt-transport-https curl git git-man gnupg-agent jq libcurl4 liberror-perl
  libjq1 libonig4
0 upgraded, 10 newly installed, 0 to remove and 3 not upgraded.
Need to get 4,996 kB/5,393 kB of archives.
After this operation, 36.1 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git-man all 1:2.17.1-1ubuntu0.5 [803 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.5 [3,912 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 gnupg-agent all 2.2.4-1ubuntu1.2 [4,880 B]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB]
Fetched 4,832 kB in 7s (685 kB/s)                                              
Selecting previously unselected package apt-transport-https.
(Reading database ... 154812 files and directories currently installed.)
Preparing to unpack .../0-apt-transport-https_1.6.12_all.deb ...
Unpacking apt-transport-https (1.6.12) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../1-libcurl4_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package curl.
Preparing to unpack .../2-curl_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../3-liberror-perl_0.17025-1_all.deb ...
Unpacking liberror-perl (0.17025-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../4-git-man_1%3a2.17.1-1ubuntu0.5_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package git.
Preparing to unpack .../5-git_1%3a2.17.1-1ubuntu0.5_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package gnupg-agent.
Preparing to unpack .../6-gnupg-agent_2.2.4-1ubuntu1.2_all.deb ...
Unpacking gnupg-agent (2.2.4-1ubuntu1.2) ...
Selecting previously unselected package libonig4:amd64.
Preparing to unpack .../7-libonig4_6.7.0-1_amd64.deb ...
Unpacking libonig4:amd64 (6.7.0-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../8-libjq1_1.5+dfsg-2_amd64.deb ...
Unpacking libjq1:amd64 (1.5+dfsg-2) ...
Selecting previously unselected package jq.
Preparing to unpack .../9-jq_1.5+dfsg-2_amd64.deb ...
Unpacking jq (1.5+dfsg-2) ...
Setting up git-man (1:2.17.1-1ubuntu0.5) ...
Setting up apt-transport-https (1.6.12) ...
Setting up liberror-perl (0.17025-1) ...
Setting up libonig4:amd64 (6.7.0-1) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Setting up libjq1:amd64 (1.5+dfsg-2) ...
Setting up gnupg-agent (2.2.4-1ubuntu1.2) ...
Setting up git (1:2.17.1-1ubuntu0.5) ...
Setting up jq (1.5+dfsg-2) ...
Setting up curl (7.58.0-2ubuntu3.8) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ sudo apt-key add -
+ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg
OK
++ lsb_release -cs
+ sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu    bionic    stable'
Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                     
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [10.3 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 327 kB in 1s (391 kB/s)                               
Reading package lists... Done
+ sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                     
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]    
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 252 kB in 1s (299 kB/s)                                
Reading package lists... Done
+ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5 libclucene-core1v5
  libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0 libe-book-0.1-1
  libedataserverui-1.2-2 libeot0 libepubgen-0.1-1 libetonyek-0.1-1
  libevent-2.1-6 libexiv2-14 libfreerdp-client2-2 libfreerdp2-2 libgc1c2
  libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6 libgpod-common libgpod4
  liblangtag-common liblangtag1 liblirc-client0 libllvm8 liblua5.3-0
  libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1 libqqwing2v5 libraw16
  librevenge-0.0-0 libsgutils2-2 libssh-4 libsuitesparseconfig5 libvncclient1
  libwinpr2-2 libxapian30 libxmlsec1 libxmlsec1-nss lp-solve media-player-info
  python3-mako python3-markupsafe syslinux syslinux-common syslinux-legacy
  usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  aufs-tools cgroupfs-mount pigz
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli pigz
0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.
Need to get 85.6 MB of archives.
After this operation, 384 MB of additional disk space will be used.
Get:1 https://download.docker.com/linux/ubuntu bionic/stable amd64 containerd.io amd64 1.2.10-3 [20.0 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 aufs-tools amd64 1:4.9+20170918-1ubuntu1 [104 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B]
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce-cli amd64 5:19.03.6~3-0~ubuntu-bionic [42.5 MB]
Get:6 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce amd64 5:19.03.6~3-0~ubuntu-bionic [22.9 MB]
Fetched 85.6 MB in 9s (9,248 kB/s)                                             
Selecting previously unselected package pigz.
(Reading database ... 155757 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package aufs-tools.
Preparing to unpack .../1-aufs-tools_1%3a4.9+20170918-1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:4.9+20170918-1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ...
Unpacking cgroupfs-mount (1.4) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../3-containerd.io_1.2.10-3_amd64.deb ...
Unpacking containerd.io (1.2.10-3) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../4-docker-ce-cli_5%3a19.03.6~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.6~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../5-docker-ce_5%3a19.03.6~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:19.03.6~3-0~ubuntu-bionic) ...
Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ...
Setting up containerd.io (1.2.10-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up cgroupfs-mount (1.4) ...
Setting up docker-ce-cli (5:19.03.6~3-0~ubuntu-bionic) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce (5:19.03.6~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
++ uname -s
++ uname -m
+ sudo curl -L https://github.com/docker/compose/releases/download/1.25.1/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617  100   617    0     0   6232      0 --:--:-- --:--:-- --:--:--  6232
100 16.2M  100 16.2M    0     0  2694k      0  0:00:06  0:00:06 --:--:-- 3096k
+ sudo chmod +x /usr/local/bin/docker-compose
+ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+ cd
+ git clone -b 2.2.1 https://github.com/mendersoftware/integration.git integration-2.2.1
Cloning into 'integration-2.2.1'...
remote: Enumerating objects: 98, done.
remote: Counting objects: 100% (98/98), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 9128 (delta 47), reused 58 (delta 29), pack-reused 9030
Receiving objects: 100% (9128/9128), 2.32 MiB | 10.27 MiB/s, done.
Resolving deltas: 100% (6227/6227), done.
Note: checking out 'a9216ae3e0c27166a10e58880407af8a3732e23b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

+ sudo bash -c 'cd integration-2.2.1 && ./demo up'
Pulling mender-gui           ... done
Pulling mender-mongo         ... done
Pulling mender-useradm       ... done
Pulling mender-inventory     ... done
Pulling mender-deployments   ... done
Pulling mender-elasticsearch ... done
Pulling mender-redis         ... done
Pulling mender-conductor     ... done
Pulling mender-device-auth   ... done
Pulling mender-api-gateway   ... done
Starting the Mender demo environment...
Creating network "integration221_mender" with the default driver
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)...
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio
c67f3896b22c: Pull complete
28537c0b7408: Pull complete
57b9c24c4fa7: Pull complete
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)...
1.13.6.2-0-alpine: Pulling from openresty/openresty
ff3a5c916c92: Already exists
ede0a2a1012b: Already exists
0e0a11843023: Already exists
246b2c6f4992: Already exists
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine
Creating integration221_mender-mongo_1         ... done
Creating integration221_mender-elasticsearch_1 ... done
Creating integration221_mender-redis_1         ... done
Creating integration221_mender-gui_1           ... done
Creating integration221_minio_1                ... done
Creating integration221_storage-proxy_1        ... done
Creating integration221_mender-inventory_1     ... done
Creating integration221_mender-deployments_1   ... done
Creating integration221_mender-useradm_1       ... done
Creating integration221_mender-conductor_1     ... done
Creating integration221_mender-device-auth_1   ... done
Creating integration221_mender-api-gateway_1   ... done
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
Retried 5 times without success. Giving up.

docker ps -a:

CONTAINER ID        IMAGE                                      COMMAND                  CREATED              STATUS                            PORTS                              NAMES
607dd8424278        mendersoftware/api-gateway:2.0.0           "/entrypoint.sh"         About a minute ago   Restarting (132) 5 seconds ago                                       integration221_mender-api-gateway_1
ed0f92211b3e        mendersoftware/deviceauth:2.1.0            "/usr/bin/deviceauth…"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-device-auth_1
8bcd5c137344        mendersoftware/mender-conductor:1.5.0      "/srv/start_conducto…"   About a minute ago   Up About a minute                 0.0.0.0:8080->8080/tcp, 8090/tcp   integration221_mender-conductor_1
0bd7ec33a8a2        mendersoftware/useradm:1.9.1               "/usr/bin/useradm --…"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-useradm_1
d34a19217af1        mendersoftware/deployments:1.8.1           "/entrypoint.sh --co…"   About a minute ago   Restarting (4) 8 seconds ago                                         integration221_mender-deployments_1
22cd6c381bb6        mendersoftware/inventory:1.6.0             "/usr/bin/inventory …"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-inventory_1
48850941b73a        openresty/openresty:1.13.6.2-0-alpine      "/usr/local/openrest…"   About a minute ago   Exited (132) About a minute ago                                      integration221_storage-proxy_1
56417bbe8583        mendersoftware/gui:2.2.1                   "/entrypoint.sh nginx"   About a minute ago   Up About a minute                 80/tcp                             integration221_mender-gui_1
90bae0d4a2a1        minio/minio:RELEASE.2018-09-25T21-34-43Z   "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)       9000/tcp                           integration221_minio_1
cbd513e40d89        elasticsearch:5-alpine                     "/docker-entrypoint.…"   About a minute ago   Up About a minute                 9200/tcp, 9300/tcp                 integration221_mender-elasticsearch_1
04ff8a530fe9        redis:5-alpine3.8                          "/redis/entrypoint.sh"   About a minute ago   Up About a minute                 6379/tcp                           integration221_mender-redis_1
0fa7d7c4fcd8        mongo:3.4                                  "docker-entrypoint.s…"   About a minute ago   Up About a minute                 27017/tcp                          integration221_mender-mongo_1

cat /proc/cpuinfo:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 4
model name	: AMD Phenom(tm) II X4 965 Processor
stepping	: 3
microcode	: 0x1000086
cpu MHz		: 3422.738
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch vmmcall
bugs		: tlb_mmatch fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips	: 6845.47
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 4
model name	: AMD Phenom(tm) II X4 965 Processor
stepping	: 3
microcode	: 0x1000086
cpu MHz		: 3422.738
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch vmmcall
bugs		: tlb_mmatch fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips	: 6845.47
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management:

top -n 1 (screenshot because formatting)

0lmi · February 15, 2020, 6:08am

Considering that it can’t start services which listens on public IP (deployments doesn’t do it but depends from storage-proxy) I would check network configuration. Can you try to disable bridged network interface and do it from scratch once again?

Topic		Replies	Views
Unable to start demo General Discussions	15	1241	April 27, 2020
Unable to install Mender Server on Amazon EC2 General Discussions	6	283	October 6, 2023
Self-hosted 3.0.0 fails to connect to mongo, demo runs fine General Discussions mender-server	7	2577	July 28, 2021
Integration 2.3 demo server storage proxy error General Discussions	13	1574	May 7, 2020
Mender minio: trouble connecting with other containers General Discussions minio	9	1673	September 10, 2019

Unable to start on-premise demo server per "Getting Started" instructions

Related topics