I’ve launched the integration production setup on an ec2 instance and everything seems to be working, however I haven’t provisioned any persistence – i.e., an artifact store, any mongodb instances, etc. – and I don’t see any fields int he prod.yml that allow you to directly specify where these persistence services live. I was able to upload an artifact and then run a deployment successfully. Can someone clarify how or how much I can control where the persistence for each service is actually living? I’m guessing setting up some kind of scheduled backup would be a good idea if the mongodb is just provisioned on the same node as the mender server.
Hi @mluz as far as I know it’s all handled by the docker volumes created as part of the production setup.
I’m hardly an expert on the server side of things so I may be misunderstanding your question. If you have other requirements, please let us know. @merlin li do you have any thoughts on this one?
Drew
Hello @mluz
thanks for running Mender.
Backup is always a good idea: https://docs.mender.io/2.5/server-installation/backup-and-restore
about the persistence please take a look at the mongodb docker compose definition you will see that it is a normal docker container, and also how it uses aliases. you can choose how to modify the settings to suit your needs. In addition you can set a mongo url for each service see deployments for example.
peter
Thanks – the mongodb makes sense, I think i’m just confused how the Storage Proxy is working out of the box and how configurable it is – the alias has “s3” in the name but I’m not providing any AWS credentials to access any buckets I know of
This makes it seem like I can leave out the minio service entirely and just provide those three env variables to the deployments service and it will work fine – is this correct?
Yes if you want to use S3 rather than Minio you can do that.
I’m still not grasping the distinction between the storage proxy and minio wrt to it being s3 compatible as far as how many URLs/endpoints are involved. According to the docs above, if I want to replace minio with S3 I provide “https://s3.amazonaws.com” as the DEPLOYMENTS_AWS_URI, which I understood to be the exact endpoint point devices hit when the connect and when they download artifacts, which the other docs say should end in :9000. I provided DEPLOYMENTS_AWS_BUCKET as well as credentials and everything started OK, however when I attempted to upload an artifact it hangs at 1% and I get 502s. In the logs it says:
time=“2020-10-30T17:18:56Z” level=error msg=“Removing file: AccessDenied: Access Denied\n\tstatus code: 403, request id: BFC71BAFEC513C68, host id: UxZkucO2JdQGbFMBaVoU9cJa3g6iZ0Ajni9UerVp22XTetrZ2a1Rqk9ROaJrHCFfpTIicFDKEhQ=: reading artifact error: Payload: can not install Payload: root-part.ext4: AccessDenied: Access Denied\n\tstatus code: 403, request id: 85B1ED392E7BE212, host id: YpPQaZfhHMMuYMEpPWoypEvemPwjcII0w+tieVgSoMza8LBUgWstTrwVGDGAmqDszNg1fTt66bA=: Cannot parse artifact file” file=api_deployments.go func=“http.(*DeploymentsApiHandlers).newImageWithContext” line=356 request_id=321a4e22-a656-4398-96d5-9d47b36c56c8 user_id=c35add45-653c-580e-95ab-7c24cbed6e1a
Again I’m trying grasp how exactly many endpoints are involved in the replacement of minio with s3, and how to update the integration configurations so I can upload artifacts to s3 and then have devices connect to the mender server as well as download said artifacts from s3.
Minio is an S3-compatible service so that the Mender code simply uses S3 APIs. In the default docker-compose configuration, Minio is running as a microservice on the same node as the other Mender server services. It needs to run on a separate port from the API gateway, thus port 9000. If you switch to raw S3, then you need to just use the correct URI for whatever S3 provider service you are using. Most of the configs I’ve seen have used a more specific URI such as https://s3-us-west-1.amazonaws.com so perhaps something like that will help.
@tranchitella @merlin can either of you help further here?
Drew
I updated my prod.yaml so that the DEPLOYMENTS_AWS_AUTH_KEY and DEPLOYMENTS_AWS_AUTH_SECRET are my AWS credentials, as well as the MINIO_ACCESS_KEY and MINIO_SECRET_KEY. I also set the DEPLOYMENTS_AWS_BUCKET for the Deployments service, and the bucket was successfully created on startup.
However, when I attempt to upload a 1.3GB artifact, the process fails at 1GB:
Error: disk path full
1: cmd/logger/logger.go:294:logger.LogIf()
2: cmd/fs-v1-helpers.go:326:cmd.fsCreateFile()
3: cmd/fs-v1-multipart.go:303:cmd.(*FSObjects).PutObjectPart()
4: cmd/object-handlers.go:1619:cmd.(ObjectLayer).PutObjectPart-fm()
5: cmd/object-handlers.go:1623:cmd.objectAPIHandlers.PutObjectPartHandler()
6: cmd/api-router.go:54:cmd.(objectAPIHandlers).PutObjectPartHandler-fm()
7: pkg/handlers/http-tracer.go:144:handlers.TraceReqHandlerFunc.func1()
8: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()
I realized this seems to be maxing out the mender-artifacts docker volume I created when provisioning the setup. So a.) it appears Minio was able to create my S3 bucket but is still not uploading to S3, instead using the docker volume, and b.) the docker volume is somehow being limited to 1GB (I’ve checked my top and the system has well over 1GB of RAM when the error is thrown).
How do I get Minio to actually upload to S3 and thereby not use the docker volume and thus not run into this disk path full error? Or is it a buffer such that it stores all of the artifact in the volume until the multipart is complete, and then uploads the entire object at once? If so, how can I increase the docker volume size so that I can buffer artifacts larger than 1GB?
Update: I’m currently able to upload artifacts to s3 via the GUI. I had to provide AWS credentials to both the deployments service as well as the minio instance. I also had to provide the bucket to the deployments service, along with a deployments URI of “https://s3.amazonaws.com”. The artifact uploads to AWS as expected.
I’m still however confused how this actually works. I can’t tell if or how Minio necessary. In the config file notes of the deployments service, it says if no AWS credentials are provided, it uses the IAM Role of the instance, which is what I definitely prefer. However when I leave these blank, and comment out the Minio entry in the prod.yaml file, I get 403 errors and the uploads do not work.
Again, I am trying to understand how I can either a.) bypass Minio entirely, or b.) configure the deployments/Minio service pair to use the ec2 instance’s IAM Role to authenticate instead of passing keys as environment variables.
I’m also still unclear what this note in the prod.yaml template about the DEPLOYMENTS_AWS_URI means:
# deployments service uses signed URLs, hence it needs to access
# storage-proxy using exactly the same name as devices will; if
# devices will access storage using https://s3.acme.org:9000, then
# set this to https://s3.acme.org:9000
Like I said I currently have it set to “https://s3.amazonaws.com” and the artifacts upload to S3 as long as I provide my AWS credentials to both deployments and minio. Assuming I can get this to work with the IAM Role, i.e., I can omit the credentials env vars from deployments and minio, what URI should the device clients be connecting to?
Thank you for your help!
As far as I know, you don’t need Minio at all when using S3.
Thanks Drew – after some massaging I got it working. However when I omit the AWS credentials from the deployments service’s environment variables, I get 403s when attempting to upload an artifact.
In the config file for the deployments service it says:
# In case when none of the credential retrieving methods are set, service will default to retrieving authentication
# credentials locally from AWS IAM which is prefered method then running the service in EC2
I’ve confirmed the EC2 instance has the appropriate IAM role and permissions – any idea why this doesn’t seem to be working as the notes say it should?
Hi @mluz sorry I am not familiar with the AWS setup here. @tranchitella @merlin do you guys know?
@mluz To use AWS S3 as storage layer with Mender, you have to set the following env variables for the deployments service:
- AWS_BUCKET
- AWS_URI
To authenticate with S3, as you are running the Mender server on an EC2 instance, the best solution is to assign an IAM role to the EC2 instance which grants S3 full access to the bucket you want to use.
Otherwise, you can explicitly set the auth key and secrets setting the following env variables:
- AWS_AUTH_KEY
- AWS_AUTH_SECRET
Let me know if this solves your issue!
It ended up being an issue with the IAM Policy – you need to grant access to both the bucket (bucket) and the contents (bucket/*) as separate resources
Thanks for all the help!