Mender-api-gateway liveness fails (seems like?)

General Discussions
1

Hi all,

Thanks all for creating such a great project. I’m setting up a mender server 3.0.0. in a kubernetes cluster using the helm charts. However the problem is setting an Ingress for external HTTP(s) load balancing doesn’t work. For the mender-api-gateway service below:

Screen Shot 2021-09-14 at 6.21.23 PM
Screen Shot 2021-09-14 at 6.21.23 PM1382×762 79.8 KB

I setup a LoadBalancer mender-api-gateway-lb as below:

Screen Shot 2021-09-14 at 6.21.57 PM
Screen Shot 2021-09-14 at 6.21.57 PM1382×800 82.6 KB

Everything works fine if I port forward the LB service and I can easily access the mender UI and login with no problem. However, when I setup an ingress over this load balancer, it does not work. Accessing the website gives an 502 (bad gateway) error.

Screen Shot 2021-09-14 at 6.25.17 PM
Screen Shot 2021-09-14 at 6.25.17 PM2366×1130 323 KB

Looking at the ingress description, it clearly says that backend is unhealthy (first one is the default backend which is healthy). Which means probably the liveness or readiness probes of the mender-api-gateway doesn’t work. I can confirm this because if I setup a dummy hello world backend for the ingress, it works perfectly fine (HEALTHY). But the mender server doesn’t.

Any idea what is wrong here? And what maybe the remedy?

Thank you.

2

Looking around, one reason for this could be that since the gateway redirects URL with 302 code, it will unacceptable by the kubernetes ingress controller which requires port response 200 only. That’s what I’ve go so far. If mender api gateway would implement the readiness and aliveness with httpGet to a known path (/healthz) instead of tcpPort, it could probably work.

3

per Configuring Ingress features  |  Kubernetes Engine Documentation GKE does not support TCP healthchecks so ingress cannot be setup for mender-api-gateway. I think this is a bug on mender that needs to be fixed.

4

@gigilibala you need to customize the healthcheck in your ingress:

I’d suggest to set requestPath: /ui/ to get a 200 OK.

5

Thanks. Yeah, I tried a few paths that return 200 including this, unfortunately the backend still shows unhealthy :frowning:

6

We were able to fix it by using the containrPort instead of 80 for BackendConfig. That gave it away. Thanks :slight_smile:

7

@gigilibala Thanks, would you mind sharing an excerpt of your configuration here so people facing the same issue in GCP with Ingresses can find the answer?

8

@tranchitella
Sure, here it goes. I think this would be good to add to the example Ingress. That example alone without the following does not work.

You need to setup a BackendConfig to change the healthcheck config of ingress.

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: mender-api-gateway-backend
spec:
  healthCheck:
    type: HTTP
    # mender-api-gateway redirects the root path to /ui/, but the Ingress
    # healthcheck by default uses / (root) and expects 200 code. Changing the
    # request path here fixes that issue.
    requestPath: /ui/

Then add the following annotation to the k8s service that ingress is pointing to:

cloud.google.com/backend-config: '{"ports": {"80":"mender-api-gateway-backend"}}'

Then ingress should be able to do proper healthcheck.

9

Thanks for sharing, @gigilibala