Failed container in Mender prod deloyment

General Discussions
1

mender_container_fail
mender_container_fail1400×435 173 KB

Hi,
I have followed this link https://docs.mender.io/2.3/administration/production-installation to install mender production in ec2 instance. Everything worked but when i do docker ps, i could see a container (mendersoftware/deployments:1.9.0) is restarting continuously. I was able to access the mender login screen although.

I have attached the docker ps output image and logs

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

time="2020-04-14T07:30:06Z" level=info msg="Deployments Service, version unknown starting up"

time="2020-04-14T07:30:06Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48

time="2020-04-14T07:30:06Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70

time="2020-04-14T07:30:06Z" level=info msg="got info: '{1.2.1 2020-04-14 07:29:59.782 +0000 UTC}'" file=db.go func=migrate.GetMigrationInfo line=63

time="2020-04-14T07:30:06Z" level=info msg="got info: '{1.2.2 2020-04-14 07:30:00.207 +0000 UTC}'" file=db.go func=migrate.GetMigrationInfo line=63

time="2020-04-14T07:30:06Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" lin

e=125

time="2020-04-14T07:30:06Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" lin

e=125

time="2020-04-14T07:30:06Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140

RequestError: send request failed

caused by: Put https://:9000/mender-artifact-storage: dial tcp :9000: connect: connection refused

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

time="2020-04-14T07:30:09Z" level=info msg="Deployments Service, version unknown starting up"

time="2020-04-14T07:30:09Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48

time="2020-04-14T07:30:09Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70

time="2020-04-14T07:30:09Z" level=info msg="got info: '{1.2.1 2020-04-14 07:29:59.782 +0000 UTC}'" file=db.go func=migrate.GetMigrationInfo line=63

time="2020-04-14T07:30:09Z" level=info msg="got info: '{1.2.2 2020-04-14 07:30:00.207 +0000 UTC}'" file=db.go func=migrate.GetMigrationInfo line=63

time="2020-04-14T07:30:09Z" level=info msg="migration to version 1.2.1 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" lin

e=125

time="2020-04-14T07:30:09Z" level=info msg="migration to version 1.2.2 skipped" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" lin

e=125

time="2020-04-14T07:30:09Z" level=info msg="DB migrated to version 1.2.2" db=deployment_service file=migrator_simple.go func="migrate.(*SimpleMigrator).Apply" line=140

RequestError: send request failed

caused by: Put https://:9000/mender-artifact-storage: dial tcp :9000: connect: connection refused

WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping

time="2020-04-14T07:30:14Z" level=info msg="Deployments Service, version unknown starting up"

time="2020-04-14T07:30:14Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=mongo.Migrate line=48

time="2020-04-14T07:30:14Z" level=info msg="migrating deployment_service" file=migrations.go func=mongo.MigrateSingle line=70

time="2020-04-14T07:30:14Z" level=info msg="got info: '{1.2.1 2020-04-14 07:29:59.782 +0000 UTC}'" file=db.go func=migrate.GetMigrationInfo line=63''''

‘’‘’

2

Hi @vikram,

I have followed this link to install mender production in ec2 instance

There is no link available in above. Could you please share it again.

3

Hi sorry that i have missed the link. Here it is Production installation

I have successfully executed all the steps mentioned in the tutorial for open source mender deployment in EC2 instance

4

Hello Vikram!

thanks for using Mender.

the above log line seems to be pointing to bad minio (or other s3 storage) configuration, do you have minio (or tother s3 storage) configured and DEPLOYMENTS_AWS_URI is pointing to it?

peter

5

Hi Peter,
As mentioned in the tutorial, i have configured these 2 parameters with my ec2 instance’s public dns: ec2-xxxx-xxxx-xxxx.compute-1.amazonaws.com
As i’m not using any s3 storage, used the same value for both of them.

API_GATEWAY_DOMAIN_NAME=“ec2-xxxx-xxxx-xxxx.compute-1.amazonaws.com” # NB! replace with your server’s public domain name
STORAGE_PROXY_DOMAIN_NAME=“$API_GATEWAY_DOMAIN_NAME” # change if you are hosting the Storage Proxy on a different domain name (not the default)

In case if i have to use s3 and configure the same, kindly let me know the procedure as i’m new to Mender

6

Hi,

https://docs.mender.io/2.3/administration/production-installation#deployments-service
I would suggest to start with minio:
https://docs.mender.io/2.3/administration/production-installation#minio

peter

7

Hello Peter,
Thanks for your support.
I was able to fix the issue!
The issue is with the sed command not replacing some of the keys like DEPLOYMENTS_AWS_URI. I have manually replaced some keys and verified config/prod.yaml file. Now all containers are up!!!

8

Hi Vikram!

great to hear it! I hope you will enjoy Mender.
let me know if there are any things you would like to discuss.

peter

9

I have few questions here

  1. I need to setup mender in my organization where VMs don’t have access to public internet to pull latest images for Mender. In that case how can i manage this scenario?
  2. Also i have been using ec2 instance’s DNS in the prod.yaml file while deploying using AWS. If i need to run this locally in my organization what should be this DNS filed to be updated in the yaml file? localhost??
10

I believe you solved that issue and there was another post about it.

peter