# Mender-server: fresh production installation continuously restarting **URL:** \ **Category:** General Discussions\ **Tags:** mender-server\ **Created:** [February 25, 2021, 8:16am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270 "2021-02-25T08:16:13Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [February 25, 2021, 8:16am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/1 "2021-02-25T08:16:13Z")
Hello, I’m not very familiar with mender yet, but it seems a pretty nice solution and currently evaluating the potential with a yocto machine and client working at the moment with the hosted version of mender-server. Currently trying to have a local production installation. I have followed the tutorial [Production installation | Mender documentation](https://docs.mender.io/2.6/server-installation/production-installation) step by step and seems every thing is fine during the installation. Setup: - local machine behind my router, forwarding of ports TCP 443 and TCP 9000 to the local machine - DNS available thanks to synology ([xxx.synology.me](http://xxx.synology.me)) where xxx is my personal configuration - local machine running ubuntu 16.04 LTS, docker version 20.10.3 and docker-compose version 1.28.4 freshly installed. At the verification step, it fails. I notice several containers at continuously restarting running ./run ps command: ``` Name Command State Ports ------------------------------------------------------------------------------------------------------------------------------- menderproduction_mender-api-gateway_1 /entrypoint.sh Restarting menderproduction_mender-create-artifact-worker_1 /usr/bin/workflows --confi ... Up 8080/tcp menderproduction_mender-deployments_1 /entrypoint.sh --config /e ... Restarting menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf ... Up 8080/tcp menderproduction_mender-deviceconnect_1 /usr/bin/deviceconnect --c ... Up 8080/tcp menderproduction_mender-gui_1 /entrypoint.sh nginx Up (healthy) 80/tcp, 8080/tcp menderproduction_mender-inventory_1 /usr/bin/inventory --confi ... Up 8080/tcp menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp menderproduction_mender-nats_1 docker-entrypoint.sh nats- ... Up 4222/tcp, 6222/tcp, 8222/tcp menderproduction_mender-useradm_1 /usr/bin/useradm --config ... Up 8080/tcp menderproduction_mender-workflows-server_1 /usr/bin/workflows --confi ... Up 8080/tcp menderproduction_mender-workflows-worker_1 /usr/bin/workflows --confi ... Up menderproduction_minio_1 /usr/bin/docker-entrypoint ... Up (healthy) 9000/tcp menderproduction_storage-proxy_1 /usr/local/openresty/bin/o ... Restarting ``` I have check the logs and two containers are interesting: ``` docker logs -f menderproduction_mender-deployments_1 WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping time="2021-02-25T07:32:52Z" level=info msg="Deployments Service, version unknown starting up" time="2021-02-25T07:32:52Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48 time="2021-02-25T07:32:52Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70 time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125 time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125 time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.3 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125 time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.4 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125 time="2021-02-25T07:32:52Z" level=info msg="migration to version 1.2.5 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=125 time="2021-02-25T07:32:52Z" level=info msg="DB migrated to version 1.2.5" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140 RequestError: send request failed caused by: Put "https://xxx.synology.me:9000/mender-artifact-storage": dial tcp 77.128.54.18:9000: connect: connection refused ``` (periodically printed) and: ``` docker logs -f menderproduction_mender-api-gateway_1 setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui Illegal instruction (core dumped) -- reload setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui ``` (periodically printed) I’m a little bit lost because I have not found similar logs in mender hub so I wonder what’s wrong here… The single point where I have notice a difference with the tutorial is the ubuntu version (16.04 instead of recommended 18.04 in the tutorial), but as soon as the docker version is enough I suppose it does not matter. Any help or ideas are appreciated 🙂 Thanks Joel ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [February 25, 2021, 2:22pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/2 "2021-02-25T14:22:54Z")
Hi @joelguittet On your server, did you have an earlier version of the Mender server components installed? I have seen issues where older containers were left unintentionally and caused issues such as this. @tranchitella can you provide any further help here? Drew ---
**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\ **Post date:** [February 25, 2021, 3:19pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/3 "2021-02-25T15:19:31Z")
@joelguittet deployments is restarting because storage-proxy is not available, and this is normal. I see from the logs that both storage-proxy and api-gateway are restarting. Both containers are based on openresty (nginx). May I ask you to manually run the openresty/openresty:1.13.6.2-0-alpine container, so we can check if the stock one starts fine on your machine? `docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine` You can now curl `http://localhost:8080` to get back a stock index.html ---
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [February 25, 2021, 8:56pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/4 "2021-02-25T20:56:42Z")
Hello @drewmoseley and @tranchitella And thanks for the quick reply to this topic ! @drewmoseley this is fresh install. There was no mender installed before and it’s event the first time docker is installed on this machine. There is no other containers/images, just the mender ones. @tranchitella I have run: > [@tranchitella](#): > > docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine No error returned, but then: > [@tranchitella](#): > > curl [http://localhost:8080](http://localhost:8080) > curl: (7) Failed to connect to localhost port 8080: Connection refused The three containers are still continuously restarting. What does it means ? Thanks for the help, Regards, Joel EDIT: one thing I have to mention that I have not previously said: when I have run mender-useradm the first time, I have forgotten to add some quotes on the password. It produced an error because the password as special characters. So I have executed the command again with quotes on the password. Not sure it’s related to the issue, but wanted to give the detail. ---
**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\ **Post date:** [February 26, 2021, 6:04am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/5 "2021-02-26T06:04:37Z")
When you run this command: > [@tranchitella](#): > > docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine Does it terminate immediately, or keep running? Can you paste the full output? What I’m trying to discover here is whether you have issues running nginx/openresty locally, independently from Mender. Both storage-proxy and api-gateway use this specific version of openresty. ---
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [February 26, 2021, 9:06am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/6 "2021-02-26T09:06:46Z")
I have tried to search by myself in parallel. First to answer your question: it runs a few seconds and stops, nothing on stdout. I have run: ```auto docker run -p 8080:80 openresty/openresty:1.13.6.2-0-alpine echo $? ``` This return error code 132. Googling gave me some results: - [process exit with code 132 · Issue #39 · openresty/docker-openresty · GitHub](https://github.com/openresty/docker-openresty/issues/39) - [Exit code 132 · Issue #138 · Kong/docker-kong · GitHub](https://github.com/Kong/docker-kong/issues/138) - [Unable to start on-premise demo server per "Getting Started" instructions - #19 by Kris](https://hub.mender.io/t/unable-to-start-on-premise-demo-server-per-getting-started-instructions/1414/19) 🙂 Then checking the exact instruction set of my CPU (lscpu =\> Atom D525, Flags: sse3). Confirmed googling the characteristics of the CPU, I have not SSE 4.2 which seems to be required to run the openresty container. Then going to [Docker Hub](https://hub.docker.com/r/openresty/openresty) they indicate nosse42 version is available. Tried to change openresty to the latest version with no SSE 4.2 which is 1.19.3.1-0-alpine-nosse42 (files docker-compose.storage.minio.yml and extra/failover-testing/docker-compose.failover-server.yml) =\> api-gateway container still continuously restarting with code 132 (the other are fine). Tried to change openresty to the first version with no SSE 4.2 which is 1.15.8.1-0-alpine-nosse42 (files docker-compose.storage.minio.yml and extra/failover-testing/docker-compose.failover-server.yml) =\> api-gateway container still continuously restarting with code 132 (the other are fine). So we need a container that will be “1.13.6.2-0-alpine-nosse42” but it’s not provided. I decided to build this container from sources ```auto git clone https://github.com/openresty/docker-openresty.git git checkout 1.13.6.2-0 docker build -t openresty/openresty:1.13.6.2-0-alpine-nosse42 -f alpine/Dockerfile --build-arg "RESTY_LUAJIT_OPTIONS=--with-luajit-xcflags='-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT -mno-sse4.2'" . ``` This took about 40minutes on my machine, it’s very long, but I did it 😃 Then restarted again mender-server with “./run up” and unfortunatly the api-gateway still restarts periodically ☹ Log of the run script (there are plenty of logs from the other containers that seems to be happy): ```auto mender-api-gateway_1 | setting up automatic reload on host IP address changes for DNS names: mender-useradm mender-inventory mender-deployments mender-device-auth mender-device-adm mender-gui menderproduction_mender-api-gateway_1 exited with code 132 ``` This error is also seen at [Unable to start on-premise demo server per "Getting Started" instructions - #25 by Kris](https://hub.mender.io/t/unable-to-start-on-premise-demo-server-per-getting-started-instructions/1414/25), but there is no conclusion to the post. Does the api-gateway container requires SSE 4.2 too ? Joel ---
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [February 26, 2021, 10:39pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/7 "2021-02-26T22:39:00Z")
Just notice mender-api-gateway is also based on openresty/openresty:1.13.6.2-0-alpine 🙂 Cloning and building the image but based on openresty/openresty:1.13.6.2-0-alpine-nosse42 previously build: ```auto git clone https://github.com/mendersoftware/mender-api-gateway-docker.git cd mender-api-gateway-docker # Edit Dockerfile to change base image sudo docker build -t mendersoftware/api-gateway:mender-2.6.0-nosse42 . ``` Now back in mender-server local clone, edition of docker-compose.yml file to use this image instead of mendersoftware/api-gateway:mender-2.6.0 And now executing “./run up” again. 🎉 🎉 🎉 🎉 😃 Server working on my machine. So basically the support of SSE 4.2 for both containers, openresty and api-gateway, required depending of the processor on the host. Do you think this is something that can be introduced in the official mender ? Suggestion to avoid building lot’s of images locally: - First update openresty to a newer version - Provide an alternative api-gateway image based on selected nosse42 version of openresty - run script option “nosse42” that pull and start the correct openresty and api-gateway images when this option is provided only - Documentation of the option I can help with some Pull Request but will maybe not be able to test various configurations. Joel ---
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [March 2, 2021, 5:54pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/8 "2021-03-02T17:54:47Z")
@drewmoseley @tranchitella what do you think about the question above ? Do you imagine that nosse42 option is something that can be introduced in Mender ? Regards Joel ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [March 2, 2021, 6:17pm UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/9 "2021-03-02T18:17:42Z")
Heh. That’s definitely beyond my knowledge. @tranchitella will be able to help though. ---
**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\ **Post date:** [March 4, 2021, 4:57am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/10 "2021-03-04T04:57:54Z")
@joelguittet FYI, next Mender release (2.7, upcoming in a few weeks) will switch to Traefik as API gateway, fully replacing openresty. ---
**Author:** ![joelguittet](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/joelguittet/32/1064_2.png) [@joelguittet](https://hub.mender.io/u/joelguittet)\ **Post date:** [March 4, 2021, 8:06am UTC](https://hub.mender.io/t/mender-server-fresh-production-installation-continuously-restarting/3270/11 "2021-03-04T08:06:46Z")
@tranchitella great information, thanks ! In this case will wait 2.7 🙂