After exporting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, I ran “./run up -d”. But “./run ps” showed that mender-deployments service kept restarting.
$ ./run ps
Name Command State Ports
--------------------------------------------------------------------------------------------------------------
menderproduction_mender-api-gateway_1 /entrypoint.sh Up 0.0.0.0:443->443/tcp
menderproduction_mender-conductor_1 /srv/start_conductor.sh Up 8080/tcp
menderproduction_mender-deployments_1 /entrypoint.sh --config /e ... Restarting
menderproduction_mender-device-auth_1 /usr/bin/deviceauth --conf ... Up 8080/tcp
menderproduction_mender-elasticsearch_1 /docker-entrypoint.sh elas ... Up 9200/tcp, 9300/tcp
menderproduction_mender-gui_1 /entrypoint.sh Up 80/tcp
menderproduction_mender-inventory_1 /usr/bin/inventory --confi ... Up 8080/tcp
menderproduction_mender-mongo_1 docker-entrypoint.sh mongod Up 27017/tcp
menderproduction_mender-redis_1 /redis/entrypoint.sh Up 6379/tcp
menderproduction_mender-useradm_1 /usr/bin/useradm --config ... Up 8080/tcp
menderproduction_storage-proxy_1 /usr/local/openresty/bin/o ... Up 0.0.0.0:9000->9000/tcp
Some logs from mender-deployments:
$ ./run logs --tail 10 mender-deployments
Attaching to menderproduction_mender-deployments_1
mender-deployments_1 | caused by: Put https://<MY_DOMAIN>:9000/<MY_BUCKET_NAME>: dial tcp 172.18.0.6:9000: connect: connection refused
mender-deployments_1 | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=103
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=migrations.Migrate line=48
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="migrating deployment_service" file=migrations.go func=migrations.MigrateSingle line=70
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="migration to version 1.2.1 skipped" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=125
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="DB migrated to version 1.2.1" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=140
mender-deployments_1 | time="2019-02-18T14:29:40Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=123
mender-deployments_1 | RequestError: send request failed
mender-deployments_1 | caused by: Put https://<MY_DOMAIN>:9000/<MY_BUCKET_NAME>: dial tcp 172.18.0.6:9000: connect: connection refused
Could you help where to go next?
By the way, the AWS access key I’m using for the development has admin rights.
You don’t need the storage proxy if you use a ‘real’ S3 bucket.
And you’d need to set an actual S3 bucket name here, it’s a placeholder not a variable.
DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>
If you’re using the provided demo script to start and stop all this be aware that it by default does not include the docker-compose.storage.s3.yml compose file. I tested my setup by changing the mender-deployments: service definition in the docker-compose.demo.yml file.
And you’d need to set an actual S3 bucket name here, it’s a placeholder not a variable.
I’m sorry for making any confusion. I’m using real bucket name to DEPLOYMENTS_AWS_BUCKET. I just wanted to hide my real S3 bucket name.
Let me just place my configuration (diff from the result of following administration/production-installation section in the doc) hoping to help someone else having similar problem.
diff --git a/docker-compose.storage.s3.yml b/docker-compose.storage.s3.yml
index feedd77..7bdebd8 100644
--- a/docker-compose.storage.s3.yml
+++ b/docker-compose.storage.s3.yml
@@ -17,7 +17,7 @@ services:
DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
- DEPLOYMENTS_AWS_REGION: us-west-1
- DEPLOYMENTS_AWS_URI: https://s3-us-west-1.amazonaws.com
- DEPLOYMENTS_AWS_BUCKET: mender-artifacts-int-testing-us
+ DEPLOYMENTS_AWS_REGION: ap-northeast-1
+ DEPLOYMENTS_AWS_URI: https://s3.ap-northeast-1.amazonaws.com
+ DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>
diff --git a/production/prod.yml b/production/prod.yml
index 6e6698d..4e8398a 100644
--- a/production/prod.yml
+++ b/production/prod.yml
@@ -64,62 +64,20 @@ services:
environment:
ALLOWED_HOSTS: mender.zw-develop.com
- storage-proxy:
- ports:
- # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
- - "9000:9000"
- networks:
- mender:
- aliases:
- # change the alias to DNS name that storage will be
- # available on, for instance if devices will access storage
- # using https://s3.acme.org:9000, then set this to
- # s3.acme.org
- - <MY_BUCKET_NAME>
- environment:
-
- # use nginx syntax for rate limiting, see
- # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
- # Examples:
- # 1m - 1MB/s
- # 512k - 512kB/s
- DOWNLOAD_SPEED: 1m
- MAX_CONNECTIONS: 100
- volumes:
- - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro
- - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro
-
mender-deployments:
command: server --automigrate
- volumes:
- - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro
environment:
- STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
- # access key, the same value as MINIO_ACCESS_KEY
- DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
- # secret, the same valie as MINIO_SECRET_KEY
- DEPLOYMENTS_AWS_AUTH_SECRET: eeN6quooph9waewe
-
- # deployments service uses signed URLs, hence it needs to access
- # storage-proxy using exactly the same name as devices will; if
- # devices will access storage using https://s3.acme.org:9000, then
- # set this to https://s3.acme.org:9000
- DEPLOYMENTS_AWS_URI: https://<MY_BUCKET_NAME>:9000
+ DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
+ DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
+ DEPLOYMENTS_AWS_URI: https://s3.ap-northeast-1.amazonaws.com
+ DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
+ DEPLOYMENTS_AWS_REGION: ap-northeast-1
+ DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>
logging:
options:
max-file: "10"
max-size: "50m"
- minio:
- environment:
- # access key
- MINIO_ACCESS_KEY: mender-deployments
- # secret
- MINIO_SECRET_KEY: eeN6quooph9waewe
- volumes:
- # mounts a docker volume named `mender-artifacts` as /export directory
- - mender-artifacts:/export:rw
-
mender-conductor:
volumes:
- ./conductor/server/config:/app/config
diff --git a/production/run b/production/run
index 34c0703..8a46860 100755
--- a/production/run
+++ b/production/run
@@ -6,6 +6,5 @@ set -e
exec docker-compose \
-p menderproduction \
-f ../docker-compose.yml \
- -f ../docker-compose.storage.minio.yml \
-f ./prod.yml \
"$@"
Thank you for posting your example here. We replicated it against a standalone Minio Server instance running on our own network vs. using Amazon S3. Everything seemed to work great: we can upload artifacts through the UI or API, trigger deployments of those artifacts, and watch the device download the image as expected from our standalone Minio Server instance URL.
But we have one problem we can’t seem to solve: since we switched over to this config, our deployments get to 99% and then never complete. The device gets flashed on the new image, reboots, and fails to POST back the success status of its update. Eventually, the deployment changes to Failed on the server and the client rolls back to the previous image.
The logs show no sign of malfunction with our Minio Server instance or with the S3 configuration we’ve created in Mender. In fact the logs show no relation at all.
So I am reaching out to you in hopes you might have updates on how your config ended up working. Have you successfully deployed devices off the images on S3? Did you run into any issues like this?
I experienced a deployment failure, but it was because I foolishly saved WiFi settings on the rootfs, so with new image, the device simply failed to connect to the network…
I’m relatively new to Mender, so I don’t know how much I can help. But you may want to share client logs when the device “fails to POST back the success status of its update”.
Thanks,
Koki
P.S.
Just forgot to mention that I have not changed my configuration from the one I shared here other than some name changes.
Thanks @koki! I’m glad to hear everything is working ok for you. We opened up a separate thread and will work on the POST failure situation like you suggest.