alright, so i’ve hit a problem moving Mender into a Kubernetes setup (more specifically, an OpenShift setup).
In Kubernetes, all pods (containers) can be communicated to by other pods using their name. This is basically the same as how it work with Docker Compose.
However, Kubernetes use a concept of a route to route traffic into the network from the outside world. This is as opposed to just doing a port forward from the docker container.
So, the problem i’m facing is this: In OpenShift, pods cannot have names which contain periods. This means that I cannot make the name of the minio pod the same as the route. However, mender-deployments does not have any concept of this and so when an artifact is requested, mender-deployments uses the name of the minio service as the domain name.
The result is that the minio URL passed back to the client is wrong and the client is unable to access the resource.
So, basically, I have a feature request: add an optional ENV to mender-deployments which would allow for this distinction between the service name and the route name.
While this feature would only be valuable in the Kuberenets world, it would definitely allow for a little more flexibility in the application.
mender-deployments service should not use minio’s internal name for communication, communication with minio happens through storage-proxy. Here in storage-proxy config you specify your minio’s public domain name, then here in mender-deployments config you specify that public name (including port 9000). Which means mender-deployments service communicates with minio by using specified public domain name instead of internal minio’s pod name. Please take a look on the architectural overview for better visualisation.
And in mender-deploymentsconfigurationDEPLOYMENTS_AWS_URI: https://storage.your-cool-domain-name.io:9000 plus important to not forget about credentials (probably variables naming is not intuitive for custom S3 compatible storage configuration, DEPLOYMENTS_AWS_URI means “public FQDN of S3 compatible storage you use” and DEPLOYMENTS_AWS_AUTH_KEY/DEPLOYMENTS_AWS_AUTH_SECRET - credentials for the storage)
Thanks for getting back to me! Looks like this is in fact the component I was missing.
I was under the impression that a Kuberenetes service would take the place of the Storage Proxy but now I see it’s still important.
so, I just want to point out where the architectural overview falls apart in terms of Kuberenets.
The diagram shows what would effectively be an internal connection between mender-deployments and storage-proxy. However, when using a route in Kubernetes–generating the URL for clients to connect–the mender-deployments component must communicate with storage-proxy by going outside of the internal network and communicating in.
This is because, in a purely Docker configuration–such as that defined with the Docker Compose files in the Integration repo–the alias of the storage-proxy container becomes this URL.
HOWEVER, with Kubernetes (or at least in the OpenShift implementation of Kubernetes) you cannot create names containing periods. Thus, you cannot call storage-proxy “storage.your-cool-domain-name.io”.
As far as I can tell, this means the URL (e.g. storage.your-cool-domain-name.io) will need to be setup in the DNS before using the service (or in the HOSTS file on deployments?) in order for deployments to access the storage-proxy.
So, again, it would actually still be very useful to have a second ENV to allow for an internal vs external connections for deployments. storage-proxy doesn’t actually solve this problem…
It’s reasonable to consider docker-compose based and kubernetes based setups as absolutely different things for different purposes. docker-compose based setup satisfies all dependencies in one bundle and fits the best development and testing purposes, kubernates based setup is something which allows you to build HA, scalable and productive production solution.
If you build production HA and scalable solution by using kubernetes then for my opinion reasonable to pay attention on data store layer, which in our case this is MongoDB, Redis, Elasticsearch and Minio. Each of those services has configuration options to satisfy mentioned needs and requires separate attention from configuration and support perspectives.
Running those services on k8s together with backend services in most cases affects at least one of the requirements (HA, performance, etc…) or requires significant configuration efforts, so, I wouldn’t consider having them there without serious arguments.
To make sure my assumption regarding configuring minio in k8s is correct I made k8s configuration for storage-proxy and minio services. So, the setup (k8s started by kops on AWS) is the following:
mender-deployments service’s DEPLOYMENTS_AWS_URI variable is https://storage.my-cool-domain-name.io:9000.
valid DNS entry is created and let’s encrypt certificate for the domain is generated
AWS ELBs are used for mender-api-gateway and storage-proxy instead of ingress (k8s) / route (openshift)
After this is applied everything works as expected without any issues. Personally I wouldn’t use such setup for critical production systems, but it’s completely working and can be used if it satisfies yours requirements.
P.S.: let me know if you want me to share those files
Again, what I am worried about is that the deployments component will be accessing the storage-proxy via the storage-proxy's route, instead of the internal connection. Please check to see if this is the case. I just want to make sure I’m not missing something.
Also, I do want to say thanks. It does mean a lot to me that you’re putting in so much effort. I hope I don’t seem unreasonable in my concerns.
Again, what I am worried about is that the deployments component will be accessing the storage-proxy via the storage-proxy 's route, instead of the internal connection.
Minio is separate product and not a part of Mender backend. Mender requires for S3-compatible storage and Minio is only one of the options you can use. It’s placed in docker-compose to have everything is one bundle for easy demo/testing/development purposes.
I also have serious doubts that Minio supports that kind of communication which you are referring to, when you make API calls to Minio’s DN_PRIVATE and it generates links by using DN_PUBLIC, this should be Minio’s option in first turn. mender-deployments service doesn’t generate links for clients, Minio does this.
Communication here is the following:
Mender client makes API call to mender-deployments service to check if an update is available
if an update is available for the device mender-deployments service requests unique link to access required artifact by making API call to S3-compatible storage
storage responds with the link
mender-deployments returns the link to the client
Mender client downloads the artifact directly from storage by using the link
As far as I can tell, this means the URL (e.g. storage.your-cool-domain-name.io) will need to be setup in the DNS before using the service (or in the HOSTS file on deployments ?) in order for deployments to access the storage-proxy.
You have to have public domain name and valid ssl certificate for your storage because your devices (Mender clients) are supposed to be distributed and should be able to download artifacts through Internet in secure way. So, this is not about communication between microservices in your cluster this is about ability to deliver updates in general.
I also have serious doubts that Minio supports that kind of communication which you are referring to, when you make API calls to Minio’s DN_PRIVATE and it generates links by using DN_PUBLIC, this should be Minio’s option in first turn. mender-deployments service doesn’t generate links for clients, Minio does this.
We are talking about the same kind of communication. The sequence of events you described is exactly how I understood this setup.
However, what i’m trying to point out is that with Docker Compose mender-deployments communicates internally to storage-proxy (let’s just ignore minio or whatever backend). In an OpenShift environment, storage-proxy cannot be referenced internally as storage.your-cool-domain.io. Thus there are two configurations allowing mender-deployments to communicate with storage-proxy:
mender-deployments is given storage-proxy’s internal name and communicates internally BUT the URL returned by storage-proxy will use this name and thus the link cannot be used outside of the cluster.
mender-deployments is given storage-proxy’s external route and communicates by connecting in through this route (a route being the external URL for the service). This will work but I would prefer to minimize communication between service through outside routes.
mender-deployments service doesn’t generate links for clients, Minio does this.
Yes, I know. But mender-deployments is what defines the URI through which everything connects to storage-proxy. Neither storage-proxy, nor the actual storage backend, have any way of knowing how they are communicated to. It is mender-deployment which provides this. Since mender-deployments also returns the DN_PUBLIC to api-gateway, it would be the ideal location for URL modification.
I’m not sure if i’m expressing this clearly enough. Basically, this is an edge case which is causing some pains (especially for development where i’m running a local OpenShift cluster).
But it is understandable if it’s not a priority.
Thanks for the communication. Despite the disagreement, I do actually feel like this helped clarify some things and show a path for moving forward.
Yes, the plan is to use this setup in production. It’s a small network of devices and there are organizational policy restrictions which prevent the use of an external data storage service (i.e. S3).
@sudobhat It sounds like you use Kubernetes (not OpenShift) and face with some unrelated to this topic issues. I recommend you to start new topic with complete description of what you are doing and the issues you are experiencing.
P.S.: please note that shared files are not production grade, it’s just POC for this topic