Mender-proxy/deployments unable to upload files to minio

General Discussions
, ,
10

Hello,

I have the same issue after a fresh install of mender version 2.3 when I try to upload an artficat that exceed 300MB.

The time out log found in my docker logs:

mender-api-gateway_1             | 2020/03/31 20:31:50 [error] 32#32: *1115 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.110.194.15, server: mender.hosted.io, request: "POST /api/management/v1/deployments/artifacts HTTP/1.1", upstream: "http://172.18.0.14:8080/api/management/v1/deployments/artifacts", host: "mender.hosted.io", referrer: "https://mender.hosted.io/ui/"

So this time out issue is generated by the mender-api-gateway service on the location api/management/v1/deployments/artifacts

I have a quick fix:

  • Connect into the mender-api-gateway container:
./run exec mender-api-gateway sh
  • Edit /usr/local/openresty/nginx/conf/common.nginx.conf and update location /api/management/v1/deployments/artifacts section to add proxy_read_timeout 300; 
location = /api/management/v1/deployments/artifacts {
        auth_request /userauth;
        auth_request_set $requestid $upstream_http_x_men_requestid;

        client_max_body_size 10G;
        proxy_read_timeout 300;
  • reload openresty:
openresty -s reload

This fix is not persistent, if you restart your server it will be lost… The solution is to use a docker volume to mount the customized nginx configuration from the host to the container.

Mender team, is it possible to fix definitely this issue by increasing the default proxy_read_timeout (60s) to 300s in the common nginx configuration?

Thank you

2 Likes