Server upgrade from 1.7 to 2.0, deployments container doesn't start

General Discussions
1

Today, I decided was a good time to upgrade my production server from 1.7 to 2.0.0
Everything went smooth, except for the deployments container not starting.

The logs show:

mender-deployments_1 | time=“2019-05-25T09:57:32Z” level=info msg=“DB migrated to version 1.2.1” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://[domain]:9000/mender-artifact-storage: dial tcp 172.24.0.7:9000: connect: connection refused
mender-deployments_1 | time=“2019-05-25T09:58:33Z” level=info msg=“Deployments Service, version unknown starting up”
mender-deployments_1 | time=“2019-05-25T09:58:33Z” level=info msg=“automigrate is ON, will apply migrations” file=migrations.go func=migrations.Migrate line=48
mender-deployments_1 | time=“2019-05-25T09:58:33Z” level=info msg=“migrating deployment_service” file=migrations.go func=migrations.MigrateSingle line=70
mender-deployments_1 | time=“2019-05-25T09:58:33Z” level=info msg=“migration to version 1.2.1 skipped” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=125
mender-deployments_1 | time=“2019-05-25T09:58:33Z” level=info msg=“DB migrated to version 1.2.1” db=deployment_service file=migrator_simple.go func=“migrate.(*SimpleMigrator).Apply” line=140
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://[domain]:9000/mender-artifact-storage: dial tcp 172.24.0.7:9000: connect: connection refused

The ip address 172.24.0.7 points to the storage-proxy.

What am I missing here?

2

Do you use Minio + storage proxy or S3? If you use Minio then can you please share logs from storage proxy and Minio containers?

3

The storage proxy container doesn’t hae any logs available. minio has:

2019-05-27T00:07:04.499934018Z 
2019-05-27T00:07:04.650988345Z  You are running an older version of Minio released 7 months ago 
2019-05-27T00:07:04.651012972Z  Update: docker pull minio/minio:RELEASE.2019-05-23T00-29-34Z 
2019-05-27T00:07:04.651017870Z 
2019-05-27T00:07:04.651021666Z 
2019-05-27T00:07:05.084630594Z Endpoint:  http://172.24.0.6:9000  http://127.0.0.1:9000
2019-05-27T00:07:05.084752289Z AccessKey: mender-deployments 
2019-05-27T00:07:05.084794493Z SecretKey: chaereegh0fohPoo6agh0phaiNg5hoox 
2019-05-27T00:07:05.084873066Z 
2019-05-27T00:07:05.084883511Z Browser Access:
2019-05-27T00:07:05.084961962Z    http://172.24.0.6:9000  http://127.0.0.1:9000
2019-05-27T00:07:05.085022124Z 
2019-05-27T00:07:05.085031453Z Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
2019-05-27T00:07:05.085078979Z    $ mc config host add myminio http://172.24.0.6:9000 mender-deployments chaereegh0fohPoo6agh0phaiNg5hoox
2019-05-27T00:07:05.085156152Z 
2019-05-27T00:07:05.085164865Z Object API (Amazon S3 compatible):
2019-05-27T00:07:05.085216262Z    Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
2019-05-27T00:07:05.085276469Z    Java:       https://docs.minio.io/docs/java-client-quickstart-guide
2019-05-27T00:07:05.085325010Z    Python:     https://docs.minio.io/docs/python-client-quickstart-guide
2019-05-27T00:07:05.085340128Z    JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
2019-05-27T00:07:05.085393892Z    .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide
4

Storage proxy should write access logs to stdout, so, if you don’t see any logs there it means it’s something wrong with network connection and dial tcp 172.24.0.7:9000: connect: connection refused log message says the same. Do you have firewalls configured?

5

I dont have any firewalls configured inside the mender docker network. Of course there’s an external firewall in play, which runs outside of my host.

Any ideas on how to track this further?

I’ll try (will be few days, likely) to access from a different container inside the same network.

6

Further I would check from host system if storage proxy replays:

$ telnet 127.0.0.1 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

And it sounds weird to me that you don’t have any log entries in storage proxy logs because when nginx starts it should write at least those messages about DOWNLOAD_SPEED and MAX_CONNECTIONS.

7

I find the issue. Of course, after finding it was a stupid thing.

I missed adding the - ./storage-proxy/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf

line in the prod.yml file.

I had to do some tweaking with elasticsearch as well, as it suddenly took way more ram, but that’s all figured out as well…

@Alex, thanks for pointing me at the storage proxy and its lack of logging the initial message!