Backend Service Timeouts on GCP

General Discussions
1

Hi!

I have found something missing in the doc. I first suspected an error on server/client side but it actually had something to do with my cluster, which is:

On GCP, when deploying mender using helm, the ssh connection always timeouts after ~ 30 seconds. The reason for that is a missing backendConfig in kubernetes. What needs to be done is applying the following yaml to k8s

apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: ws-timeout-backend-config
spec:
  timeoutSec: 90

followed by an additional annotation to the mender-api-gateway service, which is

cloud.google.com/backend-config: '{"default": "ws-timeout-backend-config"}'

Hope this helps some of you out there. Of course, you can set the timeout to more than 90 seconds, more information can be found here in the load-balancing doc

3 Likes
2

Hi @dbasner ,
thanks for your note and the solution, this is really helpful!

3

Hi,

I ran into a similar issue on Azure with AKS and Application Gateway as Ingress.

Here’s the fix for the Applicaiton Gateway Backend Setting using an annitation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:  
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/request-timeout: "90"
...
2 Likes
4

Thanks @dbasner and @oldev for the feedback. Is this something that could go into the helm charts? Or am I mistaken here?

Greetz,
Josef

5

Looks like it does not make much sense to include these in the helm chart.

AFAIK, the Ingress.yaml is not part of the helm chart. And exposing the cluster is the responsibility of the operator.

see section Exposing the service:
https://docs.mender.io/3.6/server-installation/production-installation-with-kubernetes/mender-server#exposing-the-service

But there could be a hint in the " Exposing the service" documentation, that the mender-connect ping is hardcoded to 60 seconds.
https://docs.mender.io/add-ons/remote-terminal/troubleshoot#remote-terminal-sometimes-not-working

6

I agree that exposing the cluster is the responsibility of the operator. However, the ingress is actually included in the Helm Chart: https://github.com/mendersoftware/mender-helm/blob/master/mender/values.yaml#L79

By default it’s disabled, but you can enabled it and provide custom annotations as required.

1 Like