Mender server production installation failed with "Couldn't load deployments. Error: Bad Gateway Retrying in 26 seconds"

Hi All,

I am trying to deploy the mender server production installation by following the given docs which provided by mender Production installation | Mender documentation .

below is the error which i faced:

below is my mender.conf file which running on board:
root@qemux86-64:~# cat /etc/mender/mender.conf
{
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“ServerURL”: “https://exaleap-mender/ui/”,
“TenantToken”: “dummy”,
“UpdateControlMapBootExpirationTimeSeconds”: 45,
“UpdateControlMapExpirationTimeSeconds”: 90,
“UpdatePollIntervalSeconds”: 5

Any help will be highly appreciated.

Hi @rock63
do you have any specific use-case to try the 2.4 Mender version?

If not could you please restart the process with the latest (and supported) version, which is 3.4?

Also, with this version I’d recommend to use the Helm Setup for Kubernetes.

Thanks
Roberto

Hi @robgio

I dont have any fix use case for any version just trying to setup mender server so I took the version 3.4 but here also facing issue just look at once.

Some of the services not starting and giving error:

mender/mender-server/production$ ./run up -d
Creating network “menderproduction_mender” with the default driver
Creating menderproduction_minio_1 …
Creating menderproduction_mender-nats_1 …
Creating menderproduction_mender-mongo_1 …
Creating menderproduction_mender-gui_1 …
Creating menderproduction_minio_1
Creating menderproduction_mender-nats_1
Creating menderproduction_mender-mongo_1
Creating menderproduction_mender-mongo_1 … done
Creating menderproduction_mender-iot-manager_1 …
Creating menderproduction_mender-workflows-worker_1 …
Creating menderproduction_mender-inventory_1 …
Creating menderproduction_mender-deviceconfig_1 …
Creating menderproduction_mender-workflows-server_1 …
Creating menderproduction_mender-create-artifact-worker_1 …
Creating menderproduction_mender-useradm_1 …
Creating menderproduction_mender-deviceconnect_1 …
Creating menderproduction_mender-iot-manager_1
Creating menderproduction_mender-workflows-worker_1
Creating menderproduction_mender-inventory_1
Creating menderproduction_mender-workflows-server_1
Creating menderproduction_mender-deviceconfig_1
Creating menderproduction_mender-useradm_1
Creating menderproduction_mender-create-artifact-worker_1
Creating menderproduction_mender-workflows-server_1 … done
Creating menderproduction_mender-device-auth_1 …
Creating menderproduction_mender-device-auth_1 … done
Creating menderproduction_mender-api-gateway_1 …
Creating menderproduction_mender-api-gateway_1 … done
Creating menderproduction_mender-deployments_1 …
Creating menderproduction_mender-deployments_1

ERROR: for menderproduction_mender-deployments_1 UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)

ERROR: for mender-deployments UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

mender/mender-server/production$ ./run ps
Name Command State Ports

menderproduction_mender-api-gateway_1 /entrypoint.sh --accesslog … Up 0.0.0.0:443->443/tcp,:::443->443/tcp, 80/tcp
menderproduction_mender-create-artifact-worker_1 /usr/bin/workflows --confi … Up
menderproduction_mender-deployments_1 /entrypoint.sh --config /e … Up 8080/tcp
menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf … Up 8080/tcp
menderproduction_mender-deviceconfig_1 /usr/bin/deviceconfig --co … Exit 0
menderproduction_mender-deviceconnect_1 /usr/bin/deviceconnect --c … Up 8080/tcp
menderproduction_mender-gui_1 /entrypoint.sh nginx Up 80/tcp, 8080/tcp
menderproduction_mender-inventory_1 /usr/bin/inventory --confi … Up 8080/tcp
menderproduction_mender-iot-manager_1 /usr/bin/iot-manager serve … Up 8080/tcp
menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Restarting
menderproduction_mender-nats_1 docker-entrypoint.sh -js Up 4222/tcp, 6222/tcp, 8222/tcp
menderproduction_mender-useradm_1 /usr/bin/useradm --config … Restarting
menderproduction_mender-workflows-server_1 /usr/bin/workflows --confi … Up 8080/tcp
menderproduction_mender-workflows-worker_1 /usr/bin/workflows --confi … Restarting
menderproduction_minio_1 /usr/bin/docker-entrypoint … Up 9000/tcp

Hi Roberto,
After doing $./run up -d all the services are up without any issue but This problem is coming at the of creating user name and password, I don t known why?.

log:
mender/mender-server/production$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a784ea416ef4 mendersoftware/deployments:mender-3.4.0 “/entrypoint.sh --co…” 41 seconds ago Up 15 seconds 8080/tcp menderproduction_mender-deployments_1
efdaa23dc773 traefik:v2.8 “/entrypoint.sh --ac…” About a minute ago Up 24 seconds 80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp menderproduction_mender-api-gateway_1
f581cce00f8d mendersoftware/deviceauth:mender-3.4.0 “/usr/bin/deviceauth…” About a minute ago Up 3 seconds 8080/tcp menderproduction_mender-device-auth_1
7db75f147bef mendersoftware/deviceconnect:mender-3.4.0 “/usr/bin/deviceconn…” About a minute ago Up 32 seconds 8080/tcp menderproduction_mender-deviceconnect_1
17dff12591d0 mendersoftware/workflows:mender-3.4.0 “/usr/bin/workflows …” About a minute ago Up 27 seconds 8080/tcp menderproduction_mender-workflows-server_1
4678073f2ac3 mendersoftware/workflows-worker:mender-3.4.0 “/usr/bin/workflows …” About a minute ago Up 22 seconds 8080/tcp menderproduction_mender-workflows-worker_1
5617079a8f1a mendersoftware/create-artifact-worker:mender-3.4.0 “/usr/bin/workflows …” About a minute ago Up 22 seconds menderproduction_mender-create-artifact-worker_1
793f94864289 mendersoftware/useradm:mender-3.4.0 “/usr/bin/useradm --…” About a minute ago Up 3 seconds 8080/tcp menderproduction_mender-useradm_1
de5c8c75775d mendersoftware/deviceconfig:mender-3.4.0 “/usr/bin/deviceconf…” About a minute ago Up About a minute 8080/tcp menderproduction_mender-deviceconfig_1
e9b6742fafc4 mendersoftware/inventory:mender-3.4.0 “/usr/bin/inventory …” About a minute ago Up About a minute 8080/tcp menderproduction_mender-inventory_1
2c7d768315ea mendersoftware/iot-manager:mender-3.4.0 “/usr/bin/iot-manage…” About a minute ago Up 48 seconds 8080/tcp menderproduction_mender-iot-manager_1
0aa25dd9daa1 minio/minio:RELEASE.2019-04-23T23-50-36Z “/usr/bin/docker-ent…” 2 minutes ago Up About a minute (healthy) 9000/tcp menderproduction_minio_1
e977cc26eb15 mendersoftware/gui:mender-3.4.0 “/entrypoint.sh nginx” 2 minutes ago Up About a minute (healthy) 80/tcp, 8080/tcp menderproduction_mender-gui_1
f9ff5007dec1 mongo:4.4 “docker-entrypoint.s…” 2 minutes ago Up About a minute 27017/tcp menderproduction_mender-mongo_1
e73b8f188010 nats:2.6-alpine “docker-entrypoint.s…” 2 minutes ago Up About a minute 4222/tcp, 6222/tcp, 8222/tcp menderproduction_mender-nats_1

mender/mender-server/production$ ./run exec mender-useradm /usr/bin/useradm create-user --username=abhi@host.com --password=Walia@12345
ERROR: Container
793f9486428963c5cf63e03de5e0c7f93946e82c1cc5fcdc4d4033338407cd82 is restarting, wait until the container is running

mender/mender-server/production$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a784ea416ef4 mendersoftware/deployments:mender-3.4.0 “/entrypoint.sh --co…” 4 minutes ago Up 3 minutes 8080/tcp menderproduction_mender-deployments_1
efdaa23dc773 traefik:v2.8 “/entrypoint.sh --ac…” 4 minutes ago Up 3 minutes 80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp menderproduction_mender-api-gateway_1
f581cce00f8d mendersoftware/deviceauth:mender-3.4.0 “/usr/bin/deviceauth…” 5 minutes ago Restarting (4) 33 seconds ago menderproduction_mender-device-auth_1
7db75f147bef mendersoftware/deviceconnect:mender-3.4.0 “/usr/bin/deviceconn…” 5 minutes ago Up 4 minutes 8080/tcp menderproduction_mender-deviceconnect_1
17dff12591d0 mendersoftware/workflows:mender-3.4.0 “/usr/bin/workflows …” 5 minutes ago Up 4 minutes 8080/tcp menderproduction_mender-workflows-server_1
4678073f2ac3 mendersoftware/workflows-worker:mender-3.4.0 “/usr/bin/workflows …” 5 minutes ago Up 3 minutes 8080/tcp menderproduction_mender-workflows-worker_1
5617079a8f1a mendersoftware/create-artifact-worker:mender-3.4.0 “/usr/bin/workflows …” 5 minutes ago Up 3 minutes menderproduction_mender-create-artifact-worker_1
793f94864289 mendersoftware/useradm:mender-3.4.0 “/usr/bin/useradm --…” 5 minutes ago Restarting (4) 33 seconds ago menderproduction_mender-useradm_1
de5c8c75775d mendersoftware/deviceconfig:mender-3.4.0 “/usr/bin/deviceconf…” 5 minutes ago Up 5 minutes 8080/tcp menderproduction_mender-deviceconfig_1
e9b6742fafc4 mendersoftware/inventory:mender-3.4.0 “/usr/bin/inventory …” 5 minutes ago Up 5 minutes 8080/tcp menderproduction_mender-inventory_1
2c7d768315ea mendersoftware/iot-manager:mender-3.4.0 “/usr/bin/iot-manage…” 5 minutes ago Up 4 minutes 8080/tcp menderproduction_mender-iot-manager_1
0aa25dd9daa1 minio/minio:RELEASE.2019-04-23T23-50-36Z “/usr/bin/docker-ent…” 5 minutes ago Up 5 minutes (healthy) 9000/tcp menderproduction_minio_1
e977cc26eb15 mendersoftware/gui:mender-3.4.0 “/entrypoint.sh nginx” 5 minutes ago Up 5 minutes (healthy) 80/tcp, 8080/tcp menderproduction_mender-gui_1
f9ff5007dec1 mongo:4.4 “docker-entrypoint.s…” 5 minutes ago Up 5 minutes 27017/tcp menderproduction_mender-mongo_1
e73b8f188010 nats:2.6-alpine “docker-entrypoint.s…” 5 minutes ago Up 5 minutes 4222/tcp, 6222/tcp, 8222/tcp menderproduction_mender-nats_1

why some of the services are showing restarting after executing the above command to create username and password.

I fixed the issue by doing following changes in the integration repo sources.

mender/mender-server$ sudo chmod +777 keys-generated/keys/useradm/private.key
mender/mender-server$ sudo chmod +777 keys-generated/keys/deviceauth/private.key

then again run the up command:
mender/mender-server/production$ ./run up -d

create the username and password:
mender/mender-server/production$ ./run exec mender-useradm /usr/bin/useradm create-user --username=myuser@host.com --password=pass@12345

Now just login with your given url and enter username and password ,it is working :blush:

Regards,
rock63

Hi @robgio ,

I login mender server successfully but mender client is not able to connect with the mender server ,there is no token present in open source mender server so how to connect.can you please help me.

my server:

Hi @rock63 ,

I fixed the issue by doing following changes in the integration repo sources.

Right, there’s a known issue ongoing. We’re working on that but you can use your workaround.

there is no token present in open source mender server

That’s right, it the Organization Token is present on the multitenant setup, which is an Enterprise only feature.
To connect a Mender device you can follow the tutorial on the UI, or the Documentation. For example if you want to install Mender on an existing Debian device you can follow this documentation: Downloads | Mender documentation

Hi @robgio ,

The mender installation has been done by following this doc Downloads | Mender documentation but when coming to mender client should connect with mender server which is not happening, Hosted mender server is connecting if we configure the mender client as follows:

/etc/mender/mender.conf
“HttpsClient”: {},
“Security”: {},
“Connectivity”: {},
“DeviceTypeFile”: “/var/lib/mender/device_type”,
“DBus”: {
“Enabled”: true
},
“UpdateControlMapExpirationTimeSeconds”: 90,
“UpdateControlMapBootExpirationTimeSeconds”: 45,
“UpdatePollIntervalSeconds”: 5,
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“TenantToken”: “eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJtZW5kZXIudGVuYW50IjoiNjQxNDNhYzZiNDY1ODI5ZmE3Y2IzMjlkIiwiaXNzIjoiTWVuZGVyIiwic3ViIjoiNjQxNDNhYzZiNDY1ODI5ZmE3Y2IzMjlkIn0.MaUqVy6tu6N6pB_Ws1fUutRgOXDxH4whfvENOI9Lz4aj-76oCtwcVyawq5lBHO_M-5D7N28hpEa5NS4SUfwQAKYtzUuoioUQF2q6qLsJvNbp7MaRzzeXg4FrJVelMTGFrak8eAuD3lOmJKmsShbG17XL90Zr-lh_gyNKdKmK9Qj42D3DV1jOQaM6TNcHbqoIO4AeXoI7bOz2HRQw5QpELlJfvFwarkO1jNMGvJvtlr50_UnnGSFBlbHYNCM5mmDUZ09Fj_fcIZ-L9AwbbeSn9855PSHlv0QErQC6JVg0uQIYwrAdif4XahzThOk5hrl1EyCNpRrzBIIO_Oe9VbXnYnt8CCPqIvgCJAN0OBR9P0z3guefUZhkAJTmo_9qPPhZ-oPtBkpSiQrG8JPKVd5lqZTOQ5ubY7g4jGFLyCaAx0b0gtfpEFpA-dmfqNMUrr53ecyro57bCGIemwO_OL_SpNdiGbmIXlQDVsuDOwdIhms2guegSUpVo33IpJNEG8Mb”,
“Servers”: [
{
“ServerURL”: “https://hosted.mender.io
}
]

But when I set self hosted server configurations then it is not showing any pending device on my server, here is my configurations .

/etc/mender/mender.conf
“HttpsClient”: {},
“Security”: {},
“Connectivity”: {},
“DeviceTypeFile”: “/var/lib/mender/device_type”,
“DBus”: {
“Enabled”: true
},
“UpdateControlMapExpirationTimeSeconds”: 90,
“UpdateControlMapBootExpirationTimeSeconds”: 45,
“UpdatePollIntervalSeconds”: 5,
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“ServerCertificate”: “/mender/mender-server/production/keys-generated/cert/cert.crt”,
“Servers”: [
{
“ServerURL”: “https://myserver-mender
}
]
Can you please check what I missed here …

Hi @rock63,
the conf seems good, I have to check. Anyway, the docker compose setup is going to be discontinued, I suggest to try the Helm Chart on Kubernetes instead.

Please don’t share your Tenant Token or anything sensible in this forum :wink:

Roberto