Mender-server: fresh production installation continuously restarting

joelguittet · February 25, 2021, 8:16am

Hello,

I’m not very familiar with mender yet, but it seems a pretty nice solution and currently evaluating the potential with a yocto machine and client working at the moment with the hosted version of mender-server.

Currently trying to have a local production installation. I have followed the tutorial Production installation | Mender documentation step by step and seems every thing is fine during the installation.

Setup:

local machine behind my router, forwarding of ports TCP 443 and TCP 9000 to the local machine
DNS available thanks to synology (xxx.synology.me) where xxx is my personal configuration
local machine running ubuntu 16.04 LTS, docker version 20.10.3 and docker-compose version 1.28.4 freshly installed.

At the verification step, it fails. I notice several containers at continuously restarting running ./run ps command:

Name                                    Command                  State                  Ports
-------------------------------------------------------------------------------------------------------------------------------
menderproduction_mender-api-gateway_1              /entrypoint.sh                   Restarting
menderproduction_mender-create-artifact-worker_1   /usr/bin/workflows --confi ...   Up             8080/tcp
menderproduction_mender-deployments_1              /entrypoint.sh --config /e ...   Restarting
menderproduction_mender-device-auth_1              /usr/bin/deviceauth --conf ...   Up             8080/tcp
menderproduction_mender-deviceconnect_1            /usr/bin/deviceconnect --c ...   Up             8080/tcp
menderproduction_mender-gui_1                      /entrypoint.sh nginx             Up (healthy)   80/tcp, 8080/tcp
menderproduction_mender-inventory_1                /usr/bin/inventory --confi ...   Up             8080/tcp
menderproduction_mender-mongo_1                    docker-entrypoint.sh mongod      Up             27017/tcp
menderproduction_mender-nats_1                     docker-entrypoint.sh nats- ...   Up             4222/tcp, 6222/tcp, 8222/tcp
menderproduction_mender-useradm_1                  /usr/bin/useradm --config  ...   Up             8080/tcp
menderproduction_mender-workflows-server_1         /usr/bin/workflows --confi ...   Up             8080/tcp
menderproduction_mender-workflows-worker_1         /usr/bin/workflows --confi ...   Up
menderproduction_minio_1                           /usr/bin/docker-entrypoint ...   Up (healthy)   9000/tcp
menderproduction_storage-proxy_1                   /usr/local/openresty/bin/o ...   Restarting

I have check the logs and two containers are interesting:

docker logs -f menderproduction_mender-deployments_1

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
time="2021-02-25T07:32:52Z" level=info msg="Deployments Service, version unknown starting up"
time="2021-02-25T07:32:52Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48
time="2021-02-25T07:32:52Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70
time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.3 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.4 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.5 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125
time="2021-02-25T07:32:52Z" level=info msg="DB migrated to version 1.2.5" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140
RequestError: send request failed
caused by: Put "https://xxx.synology.me:9000/mender-artifact-storage": dial tcp 77.128.54.18:9000: connect: connection refused

(periodically printed)

and:

docker logs -f menderproduction_mender-api-gateway_1

setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
Illegal instruction (core dumped)
-- reload
setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui

(periodically printed)

I’m a little bit lost because I have not found similar logs in mender hub so I wonder what’s wrong here…

The single point where I have notice a difference with the tutorial is the ubuntu version (16.04 instead of recommended 18.04 in the tutorial), but as soon as the docker version is enough I suppose it does not matter.

Any help or ideas are appreciated

Thanks
Joel

drewmoseley · February 25, 2021, 2:22pm

Hi @joelguittet

On your server, did you have an earlier version of the Mender server components installed? I have seen issues where older containers were left unintentionally and caused issues such as this.

@tranchitella can you provide any further help here?

Drew

tranchitella · February 25, 2021, 3:19pm

@joelguittet deployments is restarting because storage-proxy is not available, and this is normal. I see from the logs that both storage-proxy and api-gateway are restarting. Both containers are based on openresty (nginx).

May I ask you to manually run the openresty/openresty:1.13.6.2-0-alpine container, so we can check if the stock one starts fine on your machine?

docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine

You can now curl http://localhost:8080 to get back a stock index.html

joelguittet · February 25, 2021, 8:56pm

Hello @drewmoseley and @tranchitella

And thanks for the quick reply to this topic !

@drewmoseley this is fresh install. There was no mender installed before and it’s event the first time docker is installed on this machine. There is no other containers/images, just the mender ones.

@tranchitella I have run:

No error returned, but then:

curl: (7) Failed to connect to localhost port 8080: Connection refused

The three containers are still continuously restarting. What does it means ?

Thanks for the help,

Regards,
Joel

EDIT: one thing I have to mention that I have not previously said: when I have run mender-useradm the first time, I have forgotten to add some quotes on the password. It produced an error because the password as special characters. So I have executed the command again with quotes on the password. Not sure it’s related to the issue, but wanted to give the detail.

tranchitella · February 26, 2021, 6:04am

When you run this command:

Does it terminate immediately, or keep running? Can you paste the full output?
What I’m trying to discover here is whether you have issues running nginx/openresty locally, independently from Mender. Both storage-proxy and api-gateway use this specific version of openresty.

joelguittet · February 26, 2021, 9:06am

I have tried to search by myself in parallel. First to answer your question: it runs a few seconds and stops, nothing on stdout.

I have run:

docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine
echo $?

This return error code 132. Googling gave me some results:

Then checking the exact instruction set of my CPU (lscpu => Atom D525, Flags: sse3).
Confirmed googling the characteristics of the CPU, I have not SSE 4.2 which seems to be required to run the openresty container.

Then going to Docker Hub they indicate nosse42 version is available.

Tried to change openresty to the latest version with no SSE 4.2 which is 1.19.3.1-0-alpine-nosse42 (files docker-compose.storage.minio.yml and extra/failover-testing/docker-compose.failover-server.yml) => api-gateway container still continuously restarting with code 132 (the other are fine).

Tried to change openresty to the first version with no SSE 4.2 which is 1.15.8.1-0-alpine-nosse42 (files docker-compose.storage.minio.yml and extra/failover-testing/docker-compose.failover-server.yml) => api-gateway container still continuously restarting with code 132 (the other are fine).

So we need a container that will be “1.13.6.2-0-alpine-nosse42” but it’s not provided.

I decided to build this container from sources

git clone https://github.com/openresty/docker-openresty.git
git checkout 1.13.6.2-0
docker build -t openresty/openresty:1.13.6.2-0-alpine-nosse42 -f alpine/Dockerfile --build-arg "RESTY_LUAJIT_OPTIONS=--with-luajit-xcflags='-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT -mno-sse4.2'" .

This took about 40minutes on my machine, it’s very long, but I did it

Then restarted again mender-server with “./run up” and unfortunatly the api-gateway still restarts periodically

Log of the run script (there are plenty of logs from the other containers that seems to be happy):

mender-api-gateway_1             | setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui
menderproduction_mender-api-gateway_1 exited with code 132

This error is also seen at Unable to start on-premise demo server per "Getting Started" instructions - #25 by Kris, but there is no conclusion to the post.

Does the api-gateway container requires SSE 4.2 too ?

Joel

joelguittet · February 26, 2021, 10:39pm

Just notice mender-api-gateway is also based on openresty/openresty:1.13.6.2-0-alpine

Cloning and building the image but based on openresty/openresty:1.13.6.2-0-alpine-nosse42 previously build:

git clone https://github.com/mendersoftware/mender-api-gateway-docker.git
cd mender-api-gateway-docker
# Edit Dockerfile to change base image
sudo docker build -t mendersoftware/api-gateway:mender-2.6.0-nosse42 .

Now back in mender-server local clone, edition of docker-compose.yml file to use this image instead of mendersoftware/api-gateway:mender-2.6.0

And now executing “./run up” again.

Server working on my machine.

So basically the support of SSE 4.2 for both containers, openresty and api-gateway, required depending of the processor on the host.

Do you think this is something that can be introduced in the official mender ?

Suggestion to avoid building lot’s of images locally:

First update openresty to a newer version
Provide an alternative api-gateway image based on selected nosse42 version of openresty
run script option “nosse42” that pull and start the correct openresty and api-gateway images when this option is provided only
Documentation of the option

I can help with some Pull Request but will maybe not be able to test various configurations.

Joel

joelguittet · March 2, 2021, 5:54pm

@drewmoseley @tranchitella what do you think about the question above ? Do you imagine that nosse42 option is something that can be introduced in Mender ?

Regards
Joel

drewmoseley · March 2, 2021, 6:17pm

Heh. That’s definitely beyond my knowledge. @tranchitella will be able to help though.

tranchitella · March 4, 2021, 4:57am

@joelguittet FYI, next Mender release (2.7, upcoming in a few weeks) will switch to Traefik as API gateway, fully replacing openresty.

joelguittet · March 4, 2021, 8:06am

@tranchitella great information, thanks ! In this case will wait 2.7

Topic		Replies	Views
“menderproduction_mender-deployments_1” keeps restarting on self-hosted mender server General Discussions	11	483	November 3, 2021
container deviceauth and useradm and mender-deployments continuously restarting General Discussions	0	194	June 6, 2023
Failed container in Mender prod deployment General Discussions	0	368	April 16, 2020
Mender production Server not able to ping General Discussions	8	582	January 27, 2020
Mender server 3.2: container deviceauth and useradm continuously restarting General Discussions mender-server	5	943	March 30, 2023

Mender-server: fresh production installation continuously restarting

Related Topics