Unable to start on-premise demo server per "Getting Started" instructions

@Kris Can you also please post the logs for the services which has status “Exited” or “Restarting” (docker logs <service_name>). In this case they are integration221_mender-api-gateway_1, integration221_mender-deployments_1 and integration221_storage-proxy_1.

@0lmi

Thanks for the help. Below is the requested output.

Output of docker logs integration221_mender-deployments_1:

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:06Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:06Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:06Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:06Z" level=info msg="applying migration from version 0.0.0 to 1.2.1" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=101
time="2020-01-20T19:36:06Z" level=info msg="applying migration from version 1.2.1 to 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=101
time="2020-01-20T19:36:06Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:15Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:15Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:15Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:15Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:15Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:15Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:21Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:21Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:21Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:21Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:21Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:21Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:29Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:29Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:29Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:29Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:29Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:29Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:35Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:35Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:35Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:35Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:35Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:35Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:42Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:42Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:42Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:42Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:42Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:42Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:36:50Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:36:50Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:36:50Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:36:50Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:50Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:36:50Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:00Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:00Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:00Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:00Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:00Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:00Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:16Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:16Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:16Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:16Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:16Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:16Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:37:43Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:37:43Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:37:43Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:37:43Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:43Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:37:43Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:38:36Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:38:36Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:38:36Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:38:36Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:38:36Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:38:36Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:39:38Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:39:38Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:39:38Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:39:38Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:39:38Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:39:38Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:40:40Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:40:40Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:40:40Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:40:40Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:40:40Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:40:40Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:41:42Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:41:42Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:41:42Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:41:42Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:41:42Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:41:42Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2020-01-20T19:42:44Z" level=info msg="Deployments Service, version unknown starting up"
time="2020-01-20T19:42:44Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2020-01-20T19:42:44Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2020-01-20T19:42:44Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:42:44Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2020-01-20T19:42:44Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host

Output of docker logs integration221_mender-api-gateway_1

setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui

Output of docker logs integration221_storage-proxy_1 is completely empty.

Looks like a mistake in configuration. Hard to say where exactly but it’s definitely something messed up with certificates.

If you go through the documentation and copy/paste commands from there then it just works. You probably missed something or made a mistake/typo somewhere.

@0lmi, I appreciate the feedback, but, nope, that’s not a situation that has occurred. If it is a configuration issue, it is through no fault of mine in setting up Mender (could still be something else VM related still, though).

I’ve run the commands, copy/pasted, directly from the Mender documentation as well as docker documentation (as linked by the Mender manual). At this point, probably close to 2 dozen times. The ONLY situation where it has worked, is when I did it, the same way as I always have, on a native Debian Buster installation. First installation as complex as this, I follow the manual to the letter, and resolve errors as I can as they come up. And, between each test, I revert the VM to a prior snapshot, before ANYTHING related to mender/docker was installed. So, the instructions need to be copy/pasted each time I do it as nothing is even in history.

I would gladly provide you with ‘history’ output so you can check this; or if you prefer, you are welcome to copy/paste commands here and I will input them exactly in to my terminal to test again.

EDIT: Added a clarification of testing process.

In deployments service logs you see
caused by: Put https://s3.docker.mender.io:9000/mender-artifact-storage: dial tcp: lookup s3.docker.mender.io on 127.0.0.11:53: no such host

It means s3.docker.mender.io domain name is used for Minio (works through storage proxy) and it’s linked to localhost. Both are incorrect, first it has to be your’s FQDN and it should link to real IP if you want to deliver updates to your devices. In particular configuration is described in the following sections:

All of those are related to the issues you have (Minio is related to storage proxy and deployments services).

@0lmi Can you please clarify what you mean?

The documentation states: “Mender currently requires two entries in your /etc/hosts file to work with the Docker networking (typically 127.0.0.1 s3.docker.mender.io and 127.0.0.1 docker.mender.io).” https://docs.mender.io/2.2/getting-started/on-premise-installation/create-a-test-environment

This is confusing because:

  1. I never manually have added those entries to /etc/hosts, in my VMs or on the working native install, and they never appear (even though the docs say they will be auto added by ‘./demo up’).
  2. The documentation seems like it contradicts what you just said as the demo server WANTS those to link to localhost. If I’m misunderstanding something there, I would appreciate a correction.

Looks like you are following by the wrong document :slight_smile:

Very first line in Install a Mender demo server guide is:
Following this tutorial will create a demo installation of the Mender, appropriate for testing and experimenting. When you are ready to install for production, please follow the Production installation documentation.

There is the link to Production installation guide which you need to follow by - https://docs.mender.io/2.2/administration/production-installation

@0lmi Yes, testing and experimenting is precisely what I’m trying to do. Its in the title of this post even. I’m working on bootstraping a brand new device, having never used Mender before, and am attempting to explore and understand how this whole process works so I can roll this out without issue.

Additionally, looking at the production guide, the steps are the same up until the point of running ‘./demo up’ vs creating a custom production setup. So, in theory, following either guide should allow the demo server to run.

Is there some issue with bringing up a demo server that I’m unaware of? Note from my original output that the demo server cannot even connect to itself. I cannot log in to the demo server. I’m not even to the point of having a real client to connect to it.

EDIT: I looked at the prod. guide and added a comment about it.

@Kris I checked Mender demo script by running a VM on aws, installing everything by the script and everything works without any issues. Looks like this is something with your local virtual environment, linux, etc…

Steps to reproduce:

aws ec2 run-instances \
  --image-id <ubuntu-18.04-instance-id-in-your-region> \
  --count 1 \
  --instance-type t3.medium \
  --key-name <key-name> \
 --user-data https://raw.githubusercontent.com/0lmi/mender-it/master/server/scripts/mender-demo-server-setup.sh

then ssh to the server and check sudo docker ps -a

P.S.: image-id will be different for different regions (in my case it’s ami-0ac05733838eabc06 in eu-central-1 region), so, make sure you specify the correct one

@0lmi Great! Do you have any comments on what that incorrect configuration might be? As noted in my initial post here, a coworker was able to run through the demo steps some months ago. I pinged them and found the VM was run via ESXi while I’ve got a VirtualBox setup here as that is likely what our customer will be using.

I also linked to instructions using an Ubuntu guest on a Win10 host via VirtualBox that worked but did not really highlight much of the configuration. As I said, I am using Win 7 host and I’m not sure if that impacts anything. I’ve also tried the different networking modes as listed. I’ve also tested in VMWare Player with the same failures as above. Additionally, the guest OS installations are fresh, with Docker and Mender being the first thing installed.

Thanks for any further insight you may be able to provide!

@Kris I would recommend you to create a new VM (with at least 2 CPUs and 4Gb of RAM), install Ubuntu 18.04 from scratch and use mentioned in my previous post script for configuring and preparing the VM (with just commenting last line in it).

p.s.: you can also try to do the same VM configuration like in the ticket which you mentioned in first post, there are posted screenshots of working win10+virtualbox setup

@0lmi

I reverted my 18.04 snapshot back to its post-installation state, updated the system to the latest, and then download and ran your script; same issue. I made two modifications, adding ‘set -x’ near the top, and changing the absolute path with ‘cd’ to relative because of username differences. See the output below.

As I said, I’m not able to use Win10 host right now. I would be surprised if thats what fixes things in this situation.

karl@karl:~$ ./mender-demo-server-setup.sh 
+ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease      
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 163 kB in 1s (159 kB/s)                                               
Reading package lists... Done
+ sudo apt-get install -y git apt-transport-https ca-certificates curl gnupg-agent software-properties-common jq
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20180409).
ca-certificates set to manually installed.
software-properties-common is already the newest version (0.96.24.32.12).
software-properties-common set to manually installed.
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5
  libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0
  libe-book-0.1-1 libedataserverui-1.2-2 libeot0 libepubgen-0.1-1
  libetonyek-0.1-1 libevent-2.1-6 libexiv2-14 libfreerdp-client2-2
  libfreerdp2-2 libgc1c2 libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6
  libgpod-common libgpod4 liblangtag-common liblangtag1 liblirc-client0
  libllvm8 liblua5.3-0 libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1
  libqqwing2v5 libraw16 librevenge-0.0-0 libsgutils2-2 libssh-4
  libsuitesparseconfig5 libvncclient1 libwinpr2-2 libxapian30 libxmlsec1
  libxmlsec1-nss lp-solve media-player-info python3-mako python3-markupsafe
  syslinux syslinux-common syslinux-legacy usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  git-man libcurl4 liberror-perl libjq1 libonig4
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  apt-transport-https curl git git-man gnupg-agent jq libcurl4 liberror-perl
  libjq1 libonig4
0 upgraded, 10 newly installed, 0 to remove and 3 not upgraded.
Need to get 5,393 kB of archives.
After this operation, 36.1 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 apt-transport-https all 1.6.12 [1,692 B]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.8 [214 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 curl amd64 7.58.0-2ubuntu3.8 [159 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 liberror-perl all 0.17025-1 [22.8 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git-man all 1:2.17.1-1ubuntu0.5 [803 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.5 [3,912 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 gnupg-agent all 2.2.4-1ubuntu1.2 [4,880 B]
Get:8 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB]
Get:9 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB]
Fetched 5,393 kB in 3s (1,991 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 149809 files and directories currently installed.)
Preparing to unpack .../0-apt-transport-https_1.6.12_all.deb ...
Unpacking apt-transport-https (1.6.12) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../1-libcurl4_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package curl.
Preparing to unpack .../2-curl_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../3-liberror-perl_0.17025-1_all.deb ...
Unpacking liberror-perl (0.17025-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../4-git-man_1%3a2.17.1-1ubuntu0.5_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package git.
Preparing to unpack .../5-git_1%3a2.17.1-1ubuntu0.5_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package gnupg-agent.
Preparing to unpack .../6-gnupg-agent_2.2.4-1ubuntu1.2_all.deb ...
Unpacking gnupg-agent (2.2.4-1ubuntu1.2) ...
Selecting previously unselected package libonig4:amd64.
Preparing to unpack .../7-libonig4_6.7.0-1_amd64.deb ...
Unpacking libonig4:amd64 (6.7.0-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../8-libjq1_1.5+dfsg-2_amd64.deb ...
Unpacking libjq1:amd64 (1.5+dfsg-2) ...
Selecting previously unselected package jq.
Preparing to unpack .../9-jq_1.5+dfsg-2_amd64.deb ...
Unpacking jq (1.5+dfsg-2) ...
Setting up git-man (1:2.17.1-1ubuntu0.5) ...
Setting up apt-transport-https (1.6.12) ...
Setting up liberror-perl (0.17025-1) ...
Setting up libonig4:amd64 (6.7.0-1) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Setting up libjq1:amd64 (1.5+dfsg-2) ...
Setting up gnupg-agent (2.2.4-1ubuntu1.2) ...
Setting up git (1:2.17.1-1ubuntu0.5) ...
Setting up jq (1.5+dfsg-2) ...
Setting up curl (7.58.0-2ubuntu3.8) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ sudo apt-key add -
+ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg
OK
++ lsb_release -cs
+ sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu    bionic    stable'
Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                    
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]   
Get:4 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [9,852 B]
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease            
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 238 kB in 1s (200 kB/s)                                               
Reading package lists... Done
+ sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                    
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]   
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]  
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Fetched 252 kB in 1s (215 kB/s)                               
Reading package lists... Done
+ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5
  libclucene-core1v5 libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0
  libe-book-0.1-1 libedataserverui-1.2-2 libeot0 libepubgen-0.1-1
  libetonyek-0.1-1 libevent-2.1-6 libexiv2-14 libfreerdp-client2-2
  libfreerdp2-2 libgc1c2 libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6
  libgpod-common libgpod4 liblangtag-common liblangtag1 liblirc-client0
  libllvm8 liblua5.3-0 libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1
  libqqwing2v5 libraw16 librevenge-0.0-0 libsgutils2-2 libssh-4
  libsuitesparseconfig5 libvncclient1 libwinpr2-2 libxapian30 libxmlsec1
  libxmlsec1-nss lp-solve media-player-info python3-mako python3-markupsafe
  syslinux syslinux-common syslinux-legacy usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  aufs-tools cgroupfs-mount pigz
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli pigz
0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.
Need to get 85.5 MB of archives.
After this operation, 384 MB of additional disk space will be used.
Get:1 https://download.docker.com/linux/ubuntu bionic/stable amd64 containerd.io amd64 1.2.10-3 [20.0 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 aufs-tools amd64 1:4.9+20170918-1ubuntu1 [104 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B]
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce-cli amd64 5:19.03.5~3-0~ubuntu-bionic [42.5 MB]
Get:6 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce amd64 5:19.03.5~3-0~ubuntu-bionic [22.8 MB]
Fetched 85.5 MB in 8s (10.3 MB/s)                                             
Selecting previously unselected package pigz.
(Reading database ... 150754 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package aufs-tools.
Preparing to unpack .../1-aufs-tools_1%3a4.9+20170918-1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:4.9+20170918-1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ...
Unpacking cgroupfs-mount (1.4) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../3-containerd.io_1.2.10-3_amd64.deb ...
Unpacking containerd.io (1.2.10-3) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../4-docker-ce-cli_5%3a19.03.5~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.5~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../5-docker-ce_5%3a19.03.5~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ...
Setting up containerd.io (1.2.10-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up cgroupfs-mount (1.4) ...
Setting up docker-ce-cli (5:19.03.5~3-0~ubuntu-bionic) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce (5:19.03.5~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.33) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
++ uname -s
++ uname -m
+ sudo curl -L https://github.com/docker/compose/releases/download/1.25.1/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0   1757      0 --:--:-- --:--:-- --:--:--  1752
100 16.2M  100 16.2M    0     0  2456k      0  0:00:06  0:00:06 --:--:-- 2889k
+ sudo chmod +x /usr/local/bin/docker-compose
+ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+ cd
+ git clone -b 2.2.1 https://github.com/mendersoftware/integration.git integration-2.2.1
Cloning into 'integration-2.2.1'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 8899 (delta 5), reused 5 (delta 1), pack-reused 8873
Receiving objects: 100% (8899/8899), 2.24 MiB | 6.06 MiB/s, done.
Resolving deltas: 100% (6069/6069), done.
Note: checking out 'a9216ae3e0c27166a10e58880407af8a3732e23b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

karl@karl:~$ + sudo bash -c 'cd integration-2.2.1 && ./demo up'
Pulling mender-gui           ... done
Pulling mender-mongo         ... done
Pulling mender-useradm       ... done
Pulling mender-inventory     ... done
Pulling mender-deployments   ... done
Pulling mender-elasticsearch ... done
Pulling mender-redis         ... done
Pulling mender-conductor     ... done
Pulling mender-device-auth   ... done
Pulling mender-api-gateway   ... done
Starting the Mender demo environment...
Creating network "integration221_mender" with the default driver
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)...
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio
c67f3896b22c: Pull complete
28537c0b7408: Pull complete
57b9c24c4fa7: Pull complete
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)...
1.13.6.2-0-alpine: Pulling from openresty/openresty
ff3a5c916c92: Already exists
ede0a2a1012b: Already exists
0e0a11843023: Already exists
246b2c6f4992: Already exists
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine
Creating integration221_minio_1                ... done
Creating integration221_mender-elasticsearch_1 ... done
Creating integration221_mender-mongo_1         ... done
Creating integration221_mender-gui_1           ... done
Creating integration221_mender-redis_1         ... done
Creating integration221_mender-inventory_1     ... done
Creating integration221_mender-deployments_1   ... done
Creating integration221_mender-useradm_1       ... done
Creating integration221_storage-proxy_1        ... done
Creating integration221_mender-conductor_1     ... done
Creating integration221_mender-device-auth_1   ... done
Creating integration221_mender-api-gateway_1   ... done
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
Retried 5 times without success. Giving up.

Can you show output of the following commands:

  • docker ps -a
  • cat /proc/cpuinfo
  • top -n1

@0lmi Sorry for the delay; was moved off this project for a bit, have time to look in to it again.

Same host settings, same guest settings, did a fresh install again of Ubuntu 18.04, used your script (basically the same output, but, I’ve added it below again) as well as the commands you requested. Same issues still.

Installation:

karl@mender-test:~$ ./mender-demo-server-setup.sh 
+ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 252 kB in 1s (296 kB/s)                                
Reading package lists... Done
+ sudo apt-get install -y git apt-transport-https ca-certificates curl gnupg-agent software-properties-common jq
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20180409).
ca-certificates set to manually installed.
software-properties-common is already the newest version (0.96.24.32.12).
software-properties-common set to manually installed.
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5 libclucene-core1v5
  libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0 libe-book-0.1-1
  libedataserverui-1.2-2 libeot0 libepubgen-0.1-1 libetonyek-0.1-1
  libevent-2.1-6 libexiv2-14 libfreerdp-client2-2 libfreerdp2-2 libgc1c2
  libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6 libgpod-common libgpod4
  liblangtag-common liblangtag1 liblirc-client0 libllvm8 liblua5.3-0
  libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1 libqqwing2v5 libraw16
  librevenge-0.0-0 libsgutils2-2 libssh-4 libsuitesparseconfig5 libvncclient1
  libwinpr2-2 libxapian30 libxmlsec1 libxmlsec1-nss lp-solve media-player-info
  python3-mako python3-markupsafe syslinux syslinux-common syslinux-legacy
  usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  git-man libcurl4 liberror-perl libjq1 libonig4
Suggested packages:
  git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk
  gitweb git-cvs git-mediawiki git-svn
The following NEW packages will be installed:
  apt-transport-https curl git git-man gnupg-agent jq libcurl4 liberror-perl
  libjq1 libonig4
0 upgraded, 10 newly installed, 0 to remove and 3 not upgraded.
Need to get 4,996 kB/5,393 kB of archives.
After this operation, 36.1 MB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git-man all 1:2.17.1-1ubuntu0.5 [803 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.5 [3,912 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 gnupg-agent all 2.2.4-1ubuntu1.2 [4,880 B]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB]
Fetched 4,832 kB in 7s (685 kB/s)                                              
Selecting previously unselected package apt-transport-https.
(Reading database ... 154812 files and directories currently installed.)
Preparing to unpack .../0-apt-transport-https_1.6.12_all.deb ...
Unpacking apt-transport-https (1.6.12) ...
Selecting previously unselected package libcurl4:amd64.
Preparing to unpack .../1-libcurl4_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package curl.
Preparing to unpack .../2-curl_7.58.0-2ubuntu3.8_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.8) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../3-liberror-perl_0.17025-1_all.deb ...
Unpacking liberror-perl (0.17025-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../4-git-man_1%3a2.17.1-1ubuntu0.5_all.deb ...
Unpacking git-man (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package git.
Preparing to unpack .../5-git_1%3a2.17.1-1ubuntu0.5_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.5) ...
Selecting previously unselected package gnupg-agent.
Preparing to unpack .../6-gnupg-agent_2.2.4-1ubuntu1.2_all.deb ...
Unpacking gnupg-agent (2.2.4-1ubuntu1.2) ...
Selecting previously unselected package libonig4:amd64.
Preparing to unpack .../7-libonig4_6.7.0-1_amd64.deb ...
Unpacking libonig4:amd64 (6.7.0-1) ...
Selecting previously unselected package libjq1:amd64.
Preparing to unpack .../8-libjq1_1.5+dfsg-2_amd64.deb ...
Unpacking libjq1:amd64 (1.5+dfsg-2) ...
Selecting previously unselected package jq.
Preparing to unpack .../9-jq_1.5+dfsg-2_amd64.deb ...
Unpacking jq (1.5+dfsg-2) ...
Setting up git-man (1:2.17.1-1ubuntu0.5) ...
Setting up apt-transport-https (1.6.12) ...
Setting up liberror-perl (0.17025-1) ...
Setting up libonig4:amd64 (6.7.0-1) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.8) ...
Setting up libjq1:amd64 (1.5+dfsg-2) ...
Setting up gnupg-agent (2.2.4-1ubuntu1.2) ...
Setting up git (1:2.17.1-1ubuntu0.5) ...
Setting up jq (1.5+dfsg-2) ...
Setting up curl (7.58.0-2ubuntu3.8) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
+ sudo apt-key add -
+ sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg
OK
++ lsb_release -cs
+ sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/ubuntu    bionic    stable'
Get:1 https://download.docker.com/linux/ubuntu bionic InRelease [64.4 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                     
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages [10.3 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 327 kB in 1s (391 kB/s)                               
Reading package lists... Done
+ sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease                     
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]    
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]   
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 
Fetched 252 kB in 1s (299 kB/s)                                
Reading package lists... Done
+ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  fonts-liberation2 fonts-opensymbol gir1.2-geocodeglib-1.0
  gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gir1.2-gudev-1.0
  gir1.2-udisks-2.0 grilo-plugins-0.3-base gstreamer1.0-gtk3
  libboost-date-time1.65.1 libboost-filesystem1.65.1 libboost-iostreams1.65.1
  libboost-locale1.65.1 libcdr-0.1-1 libclucene-contribs1v5 libclucene-core1v5
  libcmis-0.5-5v5 libcolamd2 libdazzle-1.0-0 libe-book-0.1-1
  libedataserverui-1.2-2 libeot0 libepubgen-0.1-1 libetonyek-0.1-1
  libevent-2.1-6 libexiv2-14 libfreerdp-client2-2 libfreerdp2-2 libgc1c2
  libgee-0.8-2 libgexiv2-2 libgom-1.0-0 libgpgmepp6 libgpod-common libgpod4
  liblangtag-common liblangtag1 liblirc-client0 libllvm8 liblua5.3-0
  libmediaart-2.0-0 libmspub-0.1-1 libodfgen-0.1-1 libqqwing2v5 libraw16
  librevenge-0.0-0 libsgutils2-2 libssh-4 libsuitesparseconfig5 libvncclient1
  libwinpr2-2 libxapian30 libxmlsec1 libxmlsec1-nss lp-solve media-player-info
  python3-mako python3-markupsafe syslinux syslinux-common syslinux-legacy
  usb-creator-common
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  aufs-tools cgroupfs-mount pigz
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli pigz
0 upgraded, 6 newly installed, 0 to remove and 3 not upgraded.
Need to get 85.6 MB of archives.
After this operation, 384 MB of additional disk space will be used.
Get:1 https://download.docker.com/linux/ubuntu bionic/stable amd64 containerd.io amd64 1.2.10-3 [20.0 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 aufs-tools amd64 1:4.9+20170918-1ubuntu1 [104 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic/universe amd64 cgroupfs-mount all 1.4 [6,320 B]
Get:5 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce-cli amd64 5:19.03.6~3-0~ubuntu-bionic [42.5 MB]
Get:6 https://download.docker.com/linux/ubuntu bionic/stable amd64 docker-ce amd64 5:19.03.6~3-0~ubuntu-bionic [22.9 MB]
Fetched 85.6 MB in 9s (9,248 kB/s)                                             
Selecting previously unselected package pigz.
(Reading database ... 155757 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package aufs-tools.
Preparing to unpack .../1-aufs-tools_1%3a4.9+20170918-1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:4.9+20170918-1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../2-cgroupfs-mount_1.4_all.deb ...
Unpacking cgroupfs-mount (1.4) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../3-containerd.io_1.2.10-3_amd64.deb ...
Unpacking containerd.io (1.2.10-3) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../4-docker-ce-cli_5%3a19.03.6~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:19.03.6~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../5-docker-ce_5%3a19.03.6~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:19.03.6~3-0~ubuntu-bionic) ...
Setting up aufs-tools (1:4.9+20170918-1ubuntu1) ...
Setting up containerd.io (1.2.10-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up cgroupfs-mount (1.4) ...
Setting up docker-ce-cli (5:19.03.6~3-0~ubuntu-bionic) ...
Setting up pigz (2.4-1) ...
Setting up docker-ce (5:19.03.6~3-0~ubuntu-bionic) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Processing triggers for systemd (237-3ubuntu10.38) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for ureadahead (0.100.0-21) ...
++ uname -s
++ uname -m
+ sudo curl -L https://github.com/docker/compose/releases/download/1.25.1/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617  100   617    0     0   6232      0 --:--:-- --:--:-- --:--:--  6232
100 16.2M  100 16.2M    0     0  2694k      0  0:00:06  0:00:06 --:--:-- 3096k
+ sudo chmod +x /usr/local/bin/docker-compose
+ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
+ cd
+ git clone -b 2.2.1 https://github.com/mendersoftware/integration.git integration-2.2.1
Cloning into 'integration-2.2.1'...
remote: Enumerating objects: 98, done.
remote: Counting objects: 100% (98/98), done.
remote: Compressing objects: 100% (68/68), done.
remote: Total 9128 (delta 47), reused 58 (delta 29), pack-reused 9030
Receiving objects: 100% (9128/9128), 2.32 MiB | 10.27 MiB/s, done.
Resolving deltas: 100% (6227/6227), done.
Note: checking out 'a9216ae3e0c27166a10e58880407af8a3732e23b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

+ sudo bash -c 'cd integration-2.2.1 && ./demo up'
Pulling mender-gui           ... done
Pulling mender-mongo         ... done
Pulling mender-useradm       ... done
Pulling mender-inventory     ... done
Pulling mender-deployments   ... done
Pulling mender-elasticsearch ... done
Pulling mender-redis         ... done
Pulling mender-conductor     ... done
Pulling mender-device-auth   ... done
Pulling mender-api-gateway   ... done
Starting the Mender demo environment...
Creating network "integration221_mender" with the default driver
Pulling minio (minio/minio:RELEASE.2018-09-25T21-34-43Z)...
RELEASE.2018-09-25T21-34-43Z: Pulling from minio/minio
c67f3896b22c: Pull complete
28537c0b7408: Pull complete
57b9c24c4fa7: Pull complete
Digest: sha256:635d93e85eaf126ac855ed1287b0eedaf0c041e63b010eeb1fc82daada9bdcb0
Status: Downloaded newer image for minio/minio:RELEASE.2018-09-25T21-34-43Z
Pulling storage-proxy (openresty/openresty:1.13.6.2-0-alpine)...
1.13.6.2-0-alpine: Pulling from openresty/openresty
ff3a5c916c92: Already exists
ede0a2a1012b: Already exists
0e0a11843023: Already exists
246b2c6f4992: Already exists
Digest: sha256:23ff32a1e7d5a10824ab44b24a0daf86c2df1426defe8b162d8376079a548bf2
Status: Downloaded newer image for openresty/openresty:1.13.6.2-0-alpine
Creating integration221_mender-mongo_1         ... done
Creating integration221_mender-elasticsearch_1 ... done
Creating integration221_mender-redis_1         ... done
Creating integration221_mender-gui_1           ... done
Creating integration221_minio_1                ... done
Creating integration221_storage-proxy_1        ... done
Creating integration221_mender-inventory_1     ... done
Creating integration221_mender-deployments_1   ... done
Creating integration221_mender-useradm_1       ... done
Creating integration221_mender-conductor_1     ... done
Creating integration221_mender-device-auth_1   ... done
Creating integration221_mender-api-gateway_1   ... done
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
It does not seem the useradm service is up and running yet. Retrying...
Retried 5 times without success. Giving up.

docker ps -a:

CONTAINER ID        IMAGE                                      COMMAND                  CREATED              STATUS                            PORTS                              NAMES
607dd8424278        mendersoftware/api-gateway:2.0.0           "/entrypoint.sh"         About a minute ago   Restarting (132) 5 seconds ago                                       integration221_mender-api-gateway_1
ed0f92211b3e        mendersoftware/deviceauth:2.1.0            "/usr/bin/deviceauth…"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-device-auth_1
8bcd5c137344        mendersoftware/mender-conductor:1.5.0      "/srv/start_conducto…"   About a minute ago   Up About a minute                 0.0.0.0:8080->8080/tcp, 8090/tcp   integration221_mender-conductor_1
0bd7ec33a8a2        mendersoftware/useradm:1.9.1               "/usr/bin/useradm --…"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-useradm_1
d34a19217af1        mendersoftware/deployments:1.8.1           "/entrypoint.sh --co…"   About a minute ago   Restarting (4) 8 seconds ago                                         integration221_mender-deployments_1
22cd6c381bb6        mendersoftware/inventory:1.6.0             "/usr/bin/inventory …"   About a minute ago   Up About a minute                 8080/tcp                           integration221_mender-inventory_1
48850941b73a        openresty/openresty:1.13.6.2-0-alpine      "/usr/local/openrest…"   About a minute ago   Exited (132) About a minute ago                                      integration221_storage-proxy_1
56417bbe8583        mendersoftware/gui:2.2.1                   "/entrypoint.sh nginx"   About a minute ago   Up About a minute                 80/tcp                             integration221_mender-gui_1
90bae0d4a2a1        minio/minio:RELEASE.2018-09-25T21-34-43Z   "/usr/bin/docker-ent…"   About a minute ago   Up About a minute (healthy)       9000/tcp                           integration221_minio_1
cbd513e40d89        elasticsearch:5-alpine                     "/docker-entrypoint.…"   About a minute ago   Up About a minute                 9200/tcp, 9300/tcp                 integration221_mender-elasticsearch_1
04ff8a530fe9        redis:5-alpine3.8                          "/redis/entrypoint.sh"   About a minute ago   Up About a minute                 6379/tcp                           integration221_mender-redis_1
0fa7d7c4fcd8        mongo:3.4                                  "docker-entrypoint.s…"   About a minute ago   Up About a minute                 27017/tcp                          integration221_mender-mongo_1

cat /proc/cpuinfo:

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 4
model name	: AMD Phenom(tm) II X4 965 Processor
stepping	: 3
microcode	: 0x1000086
cpu MHz		: 3422.738
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 2
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch vmmcall
bugs		: tlb_mmatch fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips	: 6845.47
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 4
model name	: AMD Phenom(tm) II X4 965 Processor
stepping	: 3
microcode	: 0x1000086
cpu MHz		: 3422.738
cache size	: 512 KB
physical id	: 0
siblings	: 2
core id		: 1
cpu cores	: 2
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni cx16 x2apic popcnt hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch vmmcall
bugs		: tlb_mmatch fxsave_leak sysret_ss_attrs null_seg amd_e400 spectre_v1 spectre_v2
bogomips	: 6845.47
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management:

top -n 1 (screenshot because formatting)

Considering that it can’t start services which listens on public IP (deployments doesn’t do it but depends from storage-proxy) I would check network configuration. Can you try to disable bridged network interface and do it from scratch once again?

If I recall, there were 3 domains that I had to add to /etc/hosts to make it work in my virtual box VM. I think I looked through prod.yml for all listed domains that needed to be resolved to localhost. I believe a storage proxy domain was one of the domains added.

I’m away from office but I have a live running production ready instance of mender running on Ubuntu guest in virtual box, running on a Linux host using bridged networking. I don’t think I had to do anything special in virtual box to make it work.

1 Like

@0lmi The VM is currently in NAT mode for networking and has been since I started this thread. Is there another network configuration that I should be using?

@dellgreen That would be great if you could pass along the hosts modifications if that has an impact!

@Kris I installed latest Ubuntu 18.04.4 Desktop in VirtualBox v.6.0.16 (on MacOS 10.15.3) with only NAT network interface connected, applied all available updates, run the script mentioned above and it works.

The weirdness in api-gateway logs is that the following message is repetitive.

setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui

Here in api-gateway entrypoint script this echo which prints the message and there are no any loops, so, it should not repeat.

Can you try stop everything by ./demo down and then in one terminal tab run the following:

docker run -it -p 8888:80 openresty/openresty:1.13.6.2-0-alpine

and in another terminal tab execute:

curl -X GET http://localhost:8888

And post the output from both terminals in addition to dmesg output.

I have suspicious that this is happening because sse4_2 CPU instruction is missing. The issue is described in official openresty github project - here. Symptoms are the same - container exits with 132 code. So, let’s check by executing above commands and if this is the case then the case is just in the CPU, and it has nothing to do with VirtualBox, network configuration, docker, Mender, etc…

@0lmi I am absolutely speechless. I think that is absolutely the problem and thank you so much for your help.

The docker command does return 132:

karl@mender-test:~/integration-2.2.1$ sudo docker run -it -p 8888:80 openresty/openresty:1.13.6.2-0-alpine
karl@mender-test:~/integration-2.2.1$ echo $?
132

And dmesg snip shows the same opcode error:

[  688.851186] traps: openresty[3560] trap invalid opcode ip:7f986ae17448 sp:7ffe09c479b0 error:0 in libluajit-5.1.so.2.1.0[7f986ae0f000+277000]
[  689.326609] docker0: port 1(veth7b49596) entered disabled state
[  689.328498] vethd100f45: renamed from eth0
[  689.397711] docker0: port 1(veth7b49596) entered disabled state
[  689.402546] device veth7b49596 left promiscuous mode
[  689.402551] docker0: port 1(veth7b49596) entered disabled state

Seriously though, great catch, never would have expected that.