[Self-hosted Mender 1.7] Configuration to Use Amazon S3

Hello.

I followed the documentation
https://docs.mender.io/1.7/administration/production-installation
to set up Mender server, which went fine.

Now I want to use Amazon S3 instead of Minio. I have read
https://docs.mender.io/1.7/administration/storage
but I couldn’t succeed in getting things work.

Could someone kindly share the following pieces of information, or point me where to look for?

  • Which API needs to be allowed for the S3 bucket?
    ** Sharing a sample policy would be greatly appreciated
  • Is there any sample configuration that uses S3?

Thank you.

Hi @koki

There’s an example of the deployment service configured for S3 in the integration repository here.

Cheers,
Erik

Hello @erikhh

Thank you for your reply.

I edited docker-compose.storage.s3.yml, production/prod.yml and production/run (which are created by following https://docs.mender.io/1.7/administration/production-installation) script in the following manner:

diff --git a/docker-compose.storage.s3.yml b/docker-compose.storage.s3.yml
index feedd77..7bdebd8 100644
--- a/docker-compose.storage.s3.yml
+++ b/docker-compose.storage.s3.yml
@@ -17,7 +17,7 @@ services:
             DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
             DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
             DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
-            DEPLOYMENTS_AWS_REGION: us-west-1
-            DEPLOYMENTS_AWS_URI: https://s3-us-west-1.amazonaws.com
-            DEPLOYMENTS_AWS_BUCKET: mender-artifacts-int-testing-us
+            DEPLOYMENTS_AWS_REGION: ap-northeast-1
+            DEPLOYMENTS_AWS_URI: https://s3.ap-northeast-1.amazonaws.com
+            DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>

diff --git a/production/prod.yml b/production/prod.yml
index 6e6698d..311af0f 100644
--- a/production/prod.yml
+++ b/production/prod.yml
@@ -65,6 +65,7 @@ services:
             ALLOWED_HOSTS: mender.zw-develop.com

     storage-proxy:
+        image: openresty/openresty:1.13.6.2-0-alpine
         ports:
             # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
             - "9000:9000"
@@ -110,16 +111,6 @@ services:
                 max-file: "10"
                 max-size: "50m"

-    minio:
-        environment:
-            # access key
-            MINIO_ACCESS_KEY: mender-deployments
-            # secret
-            MINIO_SECRET_KEY: eeN6quooph9waewe
-        volumes:
-            # mounts a docker volume named `mender-artifacts` as /export directory
-            - mender-artifacts:/export:rw
-
     mender-conductor:
         volumes:
             - ./conductor/server/config:/app/config
diff --git a/production/run b/production/run
index 34c0703..a9432b2 100755
--- a/production/run
+++ b/production/run
@@ -6,6 +6,6 @@ set -e
 exec docker-compose \
      -p menderproduction \
      -f ../docker-compose.yml \
-     -f ../docker-compose.storage.minio.yml \
+     -f ../docker-compose.storage.s3.yml \
      -f ./prod.yml \
      "$@"

After exporting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, I ran “./run up -d”. But “./run ps” showed that mender-deployments service kept restarting.

$ ./run ps
                 Name                                Command                 State              Ports
--------------------------------------------------------------------------------------------------------------
menderproduction_mender-api-gateway_1     /entrypoint.sh                   Up           0.0.0.0:443->443/tcp
menderproduction_mender-conductor_1       /srv/start_conductor.sh          Up           8080/tcp
menderproduction_mender-deployments_1     /entrypoint.sh --config /e ...   Restarting
menderproduction_mender-device-auth_1     /usr/bin/deviceauth --conf ...   Up           8080/tcp
menderproduction_mender-elasticsearch_1   /docker-entrypoint.sh elas ...   Up           9200/tcp, 9300/tcp
menderproduction_mender-gui_1             /entrypoint.sh                   Up           80/tcp
menderproduction_mender-inventory_1       /usr/bin/inventory --confi ...   Up           8080/tcp
menderproduction_mender-mongo_1           docker-entrypoint.sh mongod      Up           27017/tcp
menderproduction_mender-redis_1           /redis/entrypoint.sh             Up           6379/tcp
menderproduction_mender-useradm_1         /usr/bin/useradm --config  ...   Up           8080/tcp
menderproduction_storage-proxy_1          /usr/local/openresty/bin/o ...   Up           0.0.0.0:9000->9000/tcp

Some logs from mender-deployments:

$ ./run logs --tail 10 mender-deployments
Attaching to menderproduction_mender-deployments_1
mender-deployments_1    | caused by: Put https://<MY_DOMAIN>:9000/<MY_BUCKET_NAME>: dial tcp 172.18.0.6:9000: connect: connection refused
mender-deployments_1    | WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=103
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="automigrate is ON, will apply migrations" file=migrations.go func=migrations.Migrate line=48
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="migrating deployment_service" file=migrations.go func=migrations.MigrateSingle line=70
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="migration to version 1.2.1 skipped" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=125
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="DB migrated to version 1.2.1" db="deployment_service" file="migrator_simple.go" func="migrate.(*SimpleMigrator).Apply" line=140
mender-deployments_1    | time="2019-02-18T14:29:40Z" level=info msg="Deployments Service, version unknown starting up" file=main.go func=main.cmdServer line=123
mender-deployments_1    | RequestError: send request failed
mender-deployments_1    | caused by: Put https://<MY_DOMAIN>:9000/<MY_BUCKET_NAME>: dial tcp 172.18.0.6:9000: connect: connection refused

Could you help where to go next?
By the way, the AWS access key I’m using for the development has admin rights.

Thank you,
Koki

You don’t need the storage proxy if you use a ‘real’ S3 bucket.

And you’d need to set an actual S3 bucket name here, it’s a placeholder not a variable.

DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>

If you’re using the provided demo script to start and stop all this be aware that it by default does not include the docker-compose.storage.s3.yml compose file. I tested my setup by changing the mender-deployments: service definition in the docker-compose.demo.yml file.

1 Like

Hello @erikhh

Thank you for your help again. Now it worked!

And you’d need to set an actual S3 bucket name here, it’s a placeholder not a variable.

I’m sorry for making any confusion. I’m using real bucket name to DEPLOYMENTS_AWS_BUCKET. I just wanted to hide my real S3 bucket name.

Let me just place my configuration (diff from the result of following administration/production-installation section in the doc) hoping to help someone else having similar problem.

diff --git a/docker-compose.storage.s3.yml b/docker-compose.storage.s3.yml
index feedd77..7bdebd8 100644
--- a/docker-compose.storage.s3.yml
+++ b/docker-compose.storage.s3.yml
@@ -17,7 +17,7 @@ services:
             DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
             DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
             DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
-            DEPLOYMENTS_AWS_REGION: us-west-1
-            DEPLOYMENTS_AWS_URI: https://s3-us-west-1.amazonaws.com
-            DEPLOYMENTS_AWS_BUCKET: mender-artifacts-int-testing-us
+            DEPLOYMENTS_AWS_REGION: ap-northeast-1
+            DEPLOYMENTS_AWS_URI: https://s3.ap-northeast-1.amazonaws.com
+            DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>
 
diff --git a/production/prod.yml b/production/prod.yml
index 6e6698d..4e8398a 100644
--- a/production/prod.yml
+++ b/production/prod.yml
@@ -64,62 +64,20 @@ services:
         environment:
             ALLOWED_HOSTS: mender.zw-develop.com
 
-    storage-proxy:
-        ports:
-            # outside port mapping for artifact storage (note that storage-proxy listens on port 9000)
-            - "9000:9000"
-        networks:
-            mender:
-                aliases:
-                    # change the alias to DNS name that storage will be
-                    # available on, for instance if devices will access storage
-                    # using https://s3.acme.org:9000, then set this to
-                    # s3.acme.org
-                    - <MY_BUCKET_NAME>
-        environment:
-
-            # use nginx syntax for rate limiting, see
-            # https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
-            # Examples:
-            #   1m - 1MB/s
-            #   512k - 512kB/s
-            DOWNLOAD_SPEED: 1m
-            MAX_CONNECTIONS: 100
-        volumes:
-            - ./production/keys-generated/certs/storage-proxy/cert.crt:/var/www/storage-proxy/cert/cert.crt:ro
-            - ./production/keys-generated/certs/storage-proxy/private.key:/var/www/storage-proxy/cert/private.key:ro
-
     mender-deployments:
         command: server --automigrate
-        volumes:
-            - ./production/keys-generated/certs/storage-proxy/cert.crt:/etc/ssl/certs/storage-proxy.crt:ro
         environment:
-            STORAGE_BACKEND_CERT: /etc/ssl/certs/storage-proxy.crt
-            # access key, the same value as MINIO_ACCESS_KEY
-            DEPLOYMENTS_AWS_AUTH_KEY: mender-deployments
-            # secret, the same valie as MINIO_SECRET_KEY
-            DEPLOYMENTS_AWS_AUTH_SECRET: eeN6quooph9waewe
-
-            # deployments service uses signed URLs, hence it needs to access
-            # storage-proxy using exactly the same name as devices will; if
-            # devices will access storage using https://s3.acme.org:9000, then
-            # set this to https://s3.acme.org:9000
-            DEPLOYMENTS_AWS_URI: https://<MY_BUCKET_NAME>:9000
+            DEPLOYMENTS_AWS_AUTH_KEY: ${AWS_ACCESS_KEY_ID}
+            DEPLOYMENTS_AWS_AUTH_SECRET: ${AWS_SECRET_ACCESS_KEY}
+            DEPLOYMENTS_AWS_URI: https://s3.ap-northeast-1.amazonaws.com
+            DEPLOYMENTS_AWS_TAG_ARTIFACT: "true"
+            DEPLOYMENTS_AWS_REGION: ap-northeast-1
+            DEPLOYMENTS_AWS_BUCKET: <MY_BUCKET_NAME>
         logging:
             options:
                 max-file: "10"
                 max-size: "50m"
 
-    minio:
-        environment:
-            # access key
-            MINIO_ACCESS_KEY: mender-deployments
-            # secret
-            MINIO_SECRET_KEY: eeN6quooph9waewe
-        volumes:
-            # mounts a docker volume named `mender-artifacts` as /export directory
-            - mender-artifacts:/export:rw
-
     mender-conductor:
         volumes:
             - ./conductor/server/config:/app/config
diff --git a/production/run b/production/run
index 34c0703..8a46860 100755
--- a/production/run
+++ b/production/run
@@ -6,6 +6,5 @@ set -e
 exec docker-compose \
      -p menderproduction \
      -f ../docker-compose.yml \
-     -f ../docker-compose.storage.minio.yml \
      -f ./prod.yml \
      "$@"
1 Like

Hello @koki,

Thank you for posting your example here. We replicated it against a standalone Minio Server instance running on our own network vs. using Amazon S3. Everything seemed to work great: we can upload artifacts through the UI or API, trigger deployments of those artifacts, and watch the device download the image as expected from our standalone Minio Server instance URL.

But we have one problem we can’t seem to solve: since we switched over to this config, our deployments get to 99% and then never complete. The device gets flashed on the new image, reboots, and fails to POST back the success status of its update. Eventually, the deployment changes to Failed on the server and the client rolls back to the previous image.

The logs show no sign of malfunction with our Minio Server instance or with the S3 configuration we’ve created in Mender. In fact the logs show no relation at all.

So I am reaching out to you in hopes you might have updates on how your config ended up working. Have you successfully deployed devices off the images on S3? Did you run into any issues like this?

Thanks,
Daniel

Hello @boomtown-dd,

I haven’t encountered such issue.

I experienced a deployment failure, but it was because I foolishly saved WiFi settings on the rootfs, so with new image, the device simply failed to connect to the network…

I’m relatively new to Mender, so I don’t know how much I can help. But you may want to share client logs when the device “fails to POST back the success status of its update”.

Thanks,
Koki

P.S.
Just forgot to mention that I have not changed my configuration from the one I shared here other than some name changes.

Thanks @koki! I’m glad to hear everything is working ok for you. We opened up a separate thread and will work on the POST failure situation like you suggest.