# \[Self-hosted Mender with Kubernetes\] (OpenShift) Communication between mender-deployments and Minio **URL:** \ **Category:** General Discussions\ **Created:** [February 9, 2019, 12:34am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208 "2019-02-09T00:34:22Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 9, 2019, 12:34am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/1 "2019-02-09T00:34:22Z")
alright, so i’ve hit a problem moving Mender into a Kubernetes setup (more specifically, an OpenShift setup). In Kubernetes, all pods (containers) can be communicated to by other pods using their name. This is basically the same as how it work with Docker Compose. However, Kubernetes use a concept of a `route` to route traffic into the network from the outside world. This is as opposed to just doing a port forward from the docker container. So, the problem i’m facing is this: In OpenShift, pods cannot have names which contain periods. This means that I cannot make the name of the `minio` pod the same as the `route`. However, `mender-deployments` does not have any concept of this and so when an artifact is requested, `mender-deployments` uses the name of the `minio` service as the domain name. The result is that the minio URL passed back to the client is wrong and the client is unable to access the resource. So, basically, I have a feature request: add an optional ENV to `mender-deployments` which would allow for this distinction between the service name and the route name. While this feature would only be valuable in the Kuberenets world, it would definitely allow for a little more flexibility in the application. ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 10, 2019, 12:38pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/2 "2019-02-10T12:38:33Z")
`mender-deployments` service should not use minio’s internal name for communication, communication with `minio` happens through `storage-proxy`. [Here](https://github.com/mendersoftware/integration/blob/master/template/prod.yml#L76) in `storage-proxy` config you specify your minio’s public domain name, then [here](https://github.com/mendersoftware/integration/blob/master/template/prod.yml#L105) in `mender-deployments` config you specify that public name (including port 9000). Which means `mender-deployments` service communicates with `minio` by using specified public domain name instead of internal minio’s pod name. Please take a look on the [architectural overview](https://github.com/mendersoftware/integration#how-to-use-in-production) for better visualisation. So, complete picture should be like: 1. [mender.your-cool-domain-name.io:443](http://mender.your-cool-domain-name.io:443) (`route_1`) -\> mender-api-gateway 2. [storage.your-cool-domain-name.io:9000](http://storage.your-cool-domain-name.io:9000) (`route_2`) -\> storage-proxy 3. And in `mender-deployments` [configuration](https://github.com/mendersoftware/integration/blob/master/template/prod.yml#L105) `DEPLOYMENTS_AWS_URI: https://storage.your-cool-domain-name.io:9000` plus important to not forget about [credentials](https://github.com/mendersoftware/integration/blob/master/template/prod.yml#L96-L99) (probably variables naming is not intuitive for custom S3 compatible storage configuration, `DEPLOYMENTS_AWS_URI` means “public FQDN of S3 compatible storage you use” and `DEPLOYMENTS_AWS_AUTH_KEY`/`DEPLOYMENTS_AWS_AUTH_SECRET` - credentials for the storage) ---
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 11, 2019, 6:55pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/3 "2019-02-11T18:55:54Z")
Thanks for getting back to me! Looks like this is in fact the component I was missing. I was under the impression that a Kuberenetes service would take the place of the Storage Proxy but now I see it’s still important. Greatly appreciated! ---
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 11, 2019, 7:58pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/4 "2019-02-11T19:58:03Z")
so, I just want to point out where the `architectural overview` falls apart in terms of Kuberenets. The diagram shows what would effectively be an internal connection between `mender-deployments` and `storage-proxy`. However, when using a route in Kubernetes–generating the URL for clients to connect–the `mender-deployments` component must communicate with `storage-proxy` by going outside of the internal network and communicating in. This is because, in a purely Docker configuration–such as that defined with the Docker Compose files in the Integration repo–the alias of the `storage-proxy` container becomes this URL. HOWEVER, with Kubernetes (or at least in the OpenShift implementation of Kubernetes) you cannot create names containing periods. Thus, you cannot call `storage-proxy` “[storage.your-cool-domain-name.io](http://storage.your-cool-domain-name.io)”. As far as I can tell, this means the URL (e.g. [storage.your-cool-domain-name.io](http://storage.your-cool-domain-name.io)) will need to be setup in the DNS before using the service (or in the HOSTS file on `deployments`?) in order for `deployments` to access the `storage-proxy`. So, again, it would actually still be very useful to have a second ENV to allow for an internal vs external connections for `deployments`. `storage-proxy` doesn’t actually solve this problem… ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 12, 2019, 12:50am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/5 "2019-02-12T00:50:20Z")
It’s reasonable to consider docker-compose based and kubernetes based setups as absolutely different things for different purposes. docker-compose based setup satisfies all dependencies in one bundle and fits the best development and testing purposes, kubernates based setup is something which allows you to build HA, scalable and productive production solution. If you build production HA and scalable solution by using kubernetes then for my opinion reasonable to pay attention on data store layer, which in our case this is MongoDB, Redis, Elasticsearch and Minio. Each of those services has configuration options to satisfy mentioned needs and requires separate attention from configuration and support perspectives. Running those services on k8s together with backend services in most cases affects at least one of the requirements (HA, performance, etc…) or requires significant configuration efforts, so, I wouldn’t consider having them there without serious arguments. To make sure my assumption regarding configuring minio in k8s is correct I made k8s configuration for storage-proxy and minio services. So, the setup (k8s started by kops on AWS) is the following: - `minio-deployment.yaml` - kind: Deployment, name: minio - `minio-service.yaml` - kind: Service, type: ClusterIP, name: minio - `storage-proxy-configmap.yaml` - kind: ConfigMap, name: storage-proxy-config, contains [nginx.conf](https://github.com/mendersoftware/integration/blob/master/storage-proxy/nginx.conf) - `storage-proxy-deployment.yaml` - kind: Deployment, name: storage-proxy - `storage-proxy-secret.yaml` - kind: Secret, name: storage-proxy-certificates - `storage-proxy-service.yaml` - kind: Service, type: LoadBalancer, name: storage-proxy - mender-deployments service’s DEPLOYMENTS\_AWS\_URI variable is `https://storage.my-cool-domain-name.io:9000`. - valid DNS entry is created and let’s encrypt certificate for the domain is generated - AWS ELBs are used for mender-api-gateway and storage-proxy instead of ingress (k8s) / route (openshift) After this is applied everything works as expected without any issues. Personally I wouldn’t use such setup for critical production systems, but it’s completely working and can be used if it satisfies yours requirements. P.S.: let me know if you want me to share those files ---
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 12, 2019, 1:37am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/6 "2019-02-12T01:37:59Z")
Hi, Yes, please share the files if you don’t mind. Again, what I am worried about is that the `deployments` component will be accessing the `storage-proxy` via the `storage-proxy`'s route, instead of the internal connection. Please check to see if this is the case. I just want to make sure I’m not missing something. Also, I do want to say thanks. It does mean a lot to me that you’re putting in so much effort. I hope I don’t seem unreasonable in my concerns. ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 12, 2019, 12:34pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/7 "2019-02-12T12:34:20Z")
@sabjorn > Again, what I am worried about is that the `deployments` component will be accessing the `storage-proxy` via the `storage-proxy` 's route, instead of the internal connection. Minio is separate product and not a part of Mender backend. Mender requires for S3-compatible storage and Minio is only one of the options you can use. It’s placed in docker-compose to have everything is one bundle for easy demo/testing/development purposes. I also have serious doubts that Minio supports that kind of communication which you are referring to, when you make API calls to Minio’s DN\_PRIVATE and it generates links by using DN\_PUBLIC, this should be Minio’s option in first turn. mender-deployments service doesn’t generate links for clients, Minio does this. Communication here is the following: 1. Mender client makes API call to mender-deployments service to check if an update is available 2. if an update is available for the device mender-deployments service requests unique link to access required artifact by making API call to S3-compatible storage 3. storage responds with the link 4. mender-deployments returns the link to the client 5. Mender client downloads the artifact directly from storage by using the link > As far as I can tell, this means the URL (e.g. [storage.your-cool-domain-name.io](http://storage.your-cool-domain-name.io/)) will need to be setup in the DNS before using the service (or in the HOSTS file on `deployments` ?) in order for `deployments` to access the `storage-proxy`. You have to have public domain name and valid ssl certificate for your storage because your devices (Mender clients) are supposed to be distributed and should be able to download artifacts through Internet in secure way. So, this is not about communication between microservices in your cluster this is about ability to deliver updates in general. ---
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 12, 2019, 5:27pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/8 "2019-02-12T17:27:04Z")
> I also have serious doubts that Minio supports that kind of communication which you are referring to, when you make API calls to Minio’s DN\_PRIVATE and it generates links by using DN\_PUBLIC, this should be Minio’s option in first turn. mender-deployments service doesn’t generate links for clients, Minio does this. We are talking about the same kind of communication. The sequence of events you described is exactly how I understood this setup. However, what i’m trying to point out is that with Docker Compose `mender-deployments` communicates internally to `storage-proxy` (let’s just ignore `minio` or whatever backend). In an OpenShift environment, `storage-proxy` cannot be referenced internally as [storage.your-cool-domain.io](http://storage.your-cool-domain.io). Thus there are two configurations allowing `mender-deployments` to communicate with `storage-proxy`: 1. `mender-deployments` is given `storage-proxy`’s internal name and communicates internally BUT the URL returned by `storage-proxy` will use this name and thus the link cannot be used outside of the cluster. 2. `mender-deployments` is given `storage-proxy`’s external route and communicates by connecting in through this route (a route being the external URL for the service). This **will** work but I would prefer to minimize communication between service through outside routes. > mender-deployments service doesn’t generate links for clients, Minio does this. Yes, I know. But `mender-deployments` is what defines the URI through which everything connects to `storage-proxy`. Neither `storage-proxy`, nor the actual storage backend, have any way of knowing how they are communicated to. It is `mender-deployment` which provides this. Since `mender-deployments` also returns the DN\_PUBLIC to `api-gateway`, it would be the ideal location for URL modification. I’m not sure if i’m expressing this clearly enough. Basically, this is an edge case which is causing some pains (especially for development where i’m running a local OpenShift cluster). But it is understandable if it’s not a priority. Thanks for the communication. Despite the disagreement, I do actually feel like this helped clarify some things and show a path for moving forward. ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 12, 2019, 7:52pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/9 "2019-02-12T19:52:46Z")
The pain is understandable in this case. Do you plan to go in production with the setup you are making? Config files for minio and storage-proxy: [storage-proxy-service.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/213504f6599c282f00779bc6259f3e6687534d97.yaml) (220 Bytes) [storage-proxy-secret.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/2def1a94a2709d235a5a312510c4a8122d9cb1e0.yaml) (163 Bytes) [storage-proxy-deployment.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/e923b129ad635d3495472103a5af571b347d43ea.yaml) (1008 Bytes) [storage-proxy-configmap.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/b2b384443e8a6e2551dd321e087ddfe24167ed3e.yaml) (4.3 KB) [minio-deployment.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/c846cd9b33cea8e71bda3610121db6cbbccc5660.yaml) (879 Bytes) [minio-service.yaml](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/41d6bdc1325f35bdaa76d5ef7e37f822f5e73ed8.yaml) (172 Bytes) ---
**Author:** ![sabjorn](https://avatars.discourse-cdn.com/v4/letter/s/91b2a8/32.png) [@sabjorn](https://hub.mender.io/u/sabjorn)\ **Post date:** [February 13, 2019, 5:11pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/10 "2019-02-13T17:11:45Z")
Thanks for the config files. Yes, the plan is to use this setup in production. It’s a small network of devices and there are organizational policy restrictions which prevent the use of an external data storage service (i.e. S3). ---
**Author:** ![sudobhat](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@sudobhat](https://hub.mender.io/u/sudobhat)\ **Post date:** [February 17, 2020, 9:57am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/11 "2020-02-17T09:57:10Z")
Hello, is it possible to share other .yaml files too? thanks ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 17, 2020, 10:54am UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/12 "2020-02-17T10:54:12Z")
@sudobhat I believe someone from community was working on helm charts for Kubernetes. ---
**Author:** ![sudobhat](https://avatars.discourse-cdn.com/v4/letter/s/ee7513/32.png) [@sudobhat](https://hub.mender.io/u/sudobhat)\ **Post date:** [February 17, 2020, 12:18pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/13 "2020-02-17T12:18:41Z")
hi @0lmi, it is not available yet. I am getting an error that : `failed to connect to db: failed to open mgo session: no reachable servers` for 4 containers. I am really new to kubernetes, and I don’t know how to solve it yet. A suggestion would be really helpful! PS: the files you shared before were really helpful. thanks! ---
**Author:** ![0lmi](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/0lmi/32/133_2.png) [@0lmi](https://hub.mender.io/u/0lmi)\ **Post date:** [February 17, 2020, 12:42pm UTC](https://hub.mender.io/t/self-hosted-mender-with-kubernetes-openshift-communication-between-mender-deployments-and-minio/208/14 "2020-02-17T12:42:30Z")
@sudobhat It sounds like you use Kubernetes (not OpenShift) and face with some unrelated to this topic issues. I recommend you to start new topic with complete description of what you are doing and the issues you are experiencing. P.S.: please note that shared files are not production grade, it’s just POC for this topic