Production installation - Couldn't load deployments. Error: Bad Gateway Retrying in 7 seconds

Hi,
Following the instructions from this page Production installation, I successfully installed Mender Server for production, except the deployments part “Couldn’t load deployments. Error: Bad Gateway Retrying in 7 seconds” (this is shown on the GUI page).

./run ps

             Name                                Command                  State               Ports         

menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp
menderproduction_mender-conductor_1 /srv/start_conductor.sh Up 8080/tcp, 8090/tcp
menderproduction_mender-deployments_1 /entrypoint.sh --config /e … Restarting
menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf … Up 8080/tcp
menderproduction_mender-elasticsearch_1 /docker-entrypoint.sh elas … Up 9200/tcp, 9300/tcp
menderproduction_mender-gui_1 /entrypoint.sh Up 80/tcp
menderproduction_mender-inventory_1 /usr/bin/inventory --confi … Up 8080/tcp
menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp
menderproduction_mender-redis_1 /redis/entrypoint.sh Up 6379/tcp
menderproduction_mender-useradm_1 /usr/bin/useradm --config … Up 8080/tcp
menderproduction_minio_1 /usr/bin/docker-entrypoint … Up (healthy) 9000/tcp
menderproduction_storage-proxy_1 /usr/local/openresty/bin/o … Up 0.0.0.0:9000->9000/tcp

Attached is prod.yml file (prod.yml.yaml).

All URIs are directed to the IP address of the computer, where Production Mender Server is installed:

ping mender.bg.centrak.com
64 bytes from s3.bg.centrak.com (192.168.88.50): icmp_seq=1 ttl=64 time=0.039 ms

ping s3.bg.centrak.com
64 bytes from s3.bg.centrak.com (192.168.88.50): icmp_seq=1 ttl=64 time=0.037 ms

Where is the problem? What is wrong?

This looks to be the culprit:

menderproduction_mender-deployments_1 /entrypoint.sh --config /e … Restarting

My guess is that you have an older version of the docker container running or something. Is this a new installation? Can you “docker stop” and “docker rm” all existing containers and rerun?

Yes, after stopping and removing all docker containers, the services started up.
The installation is new, but before few days, I had made a Mender demo server installation, that should have been stopped and not working, after stopping and restarting of the computer.

Now I have the same message again: “Bad Gateway Retrying in 7 Seconds”

image

I checked the processes again with “./run ps” and “menderproduction_mender-deployments_1 /entrypoint.sh --config /e …” is again with status “Restarting”.

After stopping ("./run stop"), the following processes are active (“docker ps -a”):

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

  • 2bc49fc2668e mendersoftware/api-gateway:2.0.0 “/entrypoint.sh” 18 minutes ago Exited (0) 2 minutes ago menderproduction_mender-api-gateway_1
  • a8da6d5b14f2 mendersoftware/deviceauth:2.0.0 “/usr/bin/deviceauth…” 18 minutes ago Exited (2) 2 minutes ago menderproduction_mender-device-auth_1
  • 497df1aa71da mendersoftware/inventory:1.6.0 “/usr/bin/inventory …” 18 minutes ago Exited (2) 2 minutes ago menderproduction_mender-inventory_1
  • 21aeb0dcd8d4 mendersoftware/deployments:1.7.0 “/entrypoint.sh --co…” 18 minutes ago Exited (4) 2 minutes ago menderproduction_mender-deployments_1
  • 8bfb2d96e873 mendersoftware/useradm:1.7.0 “/usr/bin/useradm --…” 18 minutes ago Exited (2) 2 minutes ago menderproduction_mender-useradm_1
  • a862f07ec838 mendersoftware/mender-conductor:1.3.1 “/srv/start_conducto…” 18 minutes ago Exited (137) About a minute ago menderproduction_mender-conductor_1
  • 23ceec2272aa openresty/openresty:1.13.6.2-0-alpine “/usr/local/openrest…” 18 minutes ago Exited (0) 2 minutes ago menderproduction_storage-proxy_1
  • 5ee3ae5987cd redis:5-alpine3.8 “/redis/entrypoint.sh” 18 minutes ago Exited (137) About a minute ago menderproduction_mender-redis_1
  • c9d5b0fb88b8 elasticsearch:5-alpine “/docker-entrypoint.…” 18 minutes ago Exited (143) About a minute ago menderproduction_mender-elasticsearch_1
  • ee4dd6aeb0ff mongo:3.4 “docker-entrypoint.s…” 18 minutes ago Exited (0) About a minute ago menderproduction_mender-mongo_1
  • f144c05f2bf0 mendersoftware/gui:2.0.1 “/entrypoint.sh” 18 minutes ago Exited (137) About a minute ago menderproduction_mender-gui_1
  • 833e2152641a minio/minio:RELEASE.2018-09-25T21-34-43Z “/usr/bin/docker-ent…” 18 minutes ago Exited (0) About a minute ago menderproduction_minio_1

Again stopping and removing:
docker stop $(docker ps -a -q)

docker rm $(docker ps -a -q)

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

All processes are removed.

./run up -d

  • Creating menderproduction_mender-elasticsearch_1 … done
  • Creating menderproduction_mender-gui_1 … done
  • Creating menderproduction_minio_1 … done
  • Creating menderproduction_mender-redis_1 … done
  • Creating menderproduction_mender-mongo_1 … done
  • Creating menderproduction_mender-deployments_1 … done
  • Creating menderproduction_mender-inventory_1 … done
  • Creating menderproduction_mender-useradm_1 … done
  • Creating menderproduction_mender-conductor_1 … done
  • Creating menderproduction_storage-proxy_1 … done
  • Creating menderproduction_mender-device-auth_1 … done
  • Creating menderproduction_mender-api-gateway_1 … done

./run ps

             Name                                Command                  State               Ports         

  • menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp
  • menderproduction_mender-conductor_1 /srv/start_conductor.sh Up 8080/tcp, 8090/tcp
  • menderproduction_mender-deployments_1 /entrypoint.sh --config /e … Restarting
  • menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf … Up 8080/tcp
  • menderproduction_mender-elasticsearch_1 /docker-entrypoint.sh elas … Up 9200/tcp, 9300/tcp
  • menderproduction_mender-gui_1 /entrypoint.sh Up 80/tcp
  • menderproduction_mender-inventory_1 /usr/bin/inventory --confi … Up 8080/tcp
  • menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp
  • menderproduction_mender-redis_1 /redis/entrypoint.sh Up 6379/tcp
  • menderproduction_mender-useradm_1 /usr/bin/useradm --config … Up 8080/tcp
  • menderproduction_minio_1 /usr/bin/docker-entrypoint … Up (healthy) 9000/tcp
  • menderproduction_storage-proxy_1 /usr/local/openresty/bin/o … Up 0.0.0.0:9000->9000/tcp

@mnachev
it’s reasonable to check logs from deployments service
docker logs menderproduction_mender-deployments_1

Hello,
My problem seems like the same. Did you managed to solve this issue?

Are these requirements satisfied?

Minimum of 10 GB free disk space and 4 GB RAM available for Mender and its services.

Can you share what you are seeing, e.g output of docker ps

I got the same restart issues after trying the OS installation as outlined here:
https://docs.mender.io/2.2/administration/production-installation

root@ec2-lala:/usr/local/mender-server/production# ./run logs --tail 10 mender-deployments
Attaching to menderproduction_mender-deployments_1
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-lala.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-lala.eu-west-1.compute.amazonaws.com
mender-deployments_1 | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“Deployments Service, version unknown starting up”
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=mongo.Migrate line=48
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migrating deployment_service” file=migrations.go func=mongo.MigrateSingle line=70
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“migration to version 1.2.1 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
mender-deployments_1 | time=“2019-12-02T09:26:38Z” level=info msg=“DB migrated to version 1.2.1” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://ec2-lala.eu-west-1.compute.amazonaws.com:9000/mender-artifact-storage: x509: certificate is not valid for any names, but wanted to match ec2-lala.eu-west-1.compute.amazonaws.com

It seems i have ended in certificate hell my favorite pastime