Mender-proxy/deployments unable to upload files to minio

Hello everyone,
beginning this week i’ve made a second mender-server to test the new 2.3 and migrate my 1.8 to it.

everything seems to be working fine, but when i want to upload an artifact, the upload hangs forever on 100%.
yet i can upload very small txt files ↓


my first instinct was that my certificates where probably not right, after redoing that, that seems not to be the case.

CERT_API_CN=menderdev.mywebsite.com CERT_STORAGE_CN=menderdev-minio.mywebsite.com …/keygen

looking inside the docker mender-proxy logs i see a pattern when uploading the txt files
(PUT GET DELETE PUT)
but not when uploading my .mender file
(only PUT GET)

log docker mender-proxy↓
https://pastebin.com/NeskiH5d

Is there a way to see if the certificates are faulty and blocking the correct working of my uploads? because i still assume the problem is still something in those lines.

i have made a pastebin of the compose file also if needed (— confidental)
https://pastebin.com/XTi0MdwB

Thanks in advance, Gene

Mender server versions prior to 2.1.0 had a bug where upload would sometimes fail depending on exact block sizes used by the upload tool (the browser). Most people were able to work around it by using a different browser.

Hello @kacf
After redoing my mender-server 2.3 several times, iam still unable to upload any .menders, even in different browsers.

Are there any other clues i could follow to debug this?
Greets

so i tried uploading my .mender using the api with mender-cli

this give me somewhat an idea what is going wrong

 912.07 MiB / 912.07 MiB [=====] 100.00% 44m50s
Processing uploaded file. This may take around one minute.

FAILURE: artifact upload failed with status 504, reason: <html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>openresty/1.13.6.2</center>
</body>
</html>

looking back into the docker logs been generated, the openresty shows an PUT and DELETE, but not really errors

172.26.0.1 - - [16/Mar/2020:16:59:46 +0000] “GET / HTTP/1.1” 403 206 “-” “check_http/v2.2.1.git (nagios-plugins 2.2.1)”
172.26.0.1 - - [16/Mar/2020:17:25:28 +0000] “PUT /mender-artifact-storage/83731ca9-a4e9-4e43-a0a0-810792b168f2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=mender-deplo
yments%2F20200316%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200316T172424Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=8bc51c502af00c04faf79ba3a99bb62b41fcc988ca11c669fbdff90af6a384d2 HTTP/1.1”
200 0 “-” “Go-http-client/1.1”
172.26.0.1 - - [16/Mar/2020:17:25:28 +0000] “DELETE /mender-artifact-storage/83731ca9-a4e9-4e43-a0a0-810792b168f2 HTTP/1.1” 204 0 “-” “aws-sdk-go/1.12.27 (go1.11.4; linux; amd64)”

@genedupre can you please provide us the logs of your storage-proxy container?

Hello @tranchitella
ive generated some logs and put them here
https://filebin.net/bh7nk0f9gwyvbzkd

Thanks for looking into it

Hi genedupre,

I am having a case similar (mender production 2.3.0) to what you comment in your query,
In addition to the same thing happening with the artifacts, i see this:

  • two containers are like healthy (mender conductor, mender minio).
  • I can upload any type of file, as long as it doesn’t exceed 1Mb.

Has this case been detected for you?

Thank you!

Hello @blackdevice
yes both these cases are also present in my problems
when i have time i will look into it more
Greets Gene

I’m having the same thing with a mender production 2.3.0 instance, fresh install.

If I upload a very small artifact on the web interface it says it was successful, and I can see the file in minio, but it doesn’t show up on the Releases page. If I upload a large artifact it gets to 100% and just sits there.

If I upload a large artifact using mender-cli I get the same 504 timeout.

Has anyone solved this yet, or have any advice on what to check? After looking at logs I don’t see anything obvious, other than using self-signed certificates, but I thought that was okay.

Thanks,
Nate

Hello,

I have the same issue after a fresh install of mender version 2.3 when I try to upload an artficat that exceed 300MB.

The time out log found in my docker logs:

mender-api-gateway_1             | 2020/03/31 20:31:50 [error] 32#32: *1115 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.110.194.15, server: mender.hosted.io, request: "POST /api/management/v1/deployments/artifacts HTTP/1.1", upstream: "http://172.18.0.14:8080/api/management/v1/deployments/artifacts", host: "mender.hosted.io", referrer: "https://mender.hosted.io/ui/"

So this time out issue is generated by the mender-api-gateway service on the location api/management/v1/deployments/artifacts

I have a quick fix:

  • Connect into the mender-api-gateway container:
./run exec mender-api-gateway sh
  • Edit /usr/local/openresty/nginx/conf/common.nginx.conf and update location /api/management/v1/deployments/artifacts section to add proxy_read_timeout 300;
location = /api/management/v1/deployments/artifacts {
        auth_request /userauth;
        auth_request_set $requestid $upstream_http_x_men_requestid;

        client_max_body_size 10G;
        proxy_read_timeout 300;
  • reload openresty:
openresty -s reload

This fix is not persistent, if you restart your server it will be lost… The solution is to use a docker volume to mount the customized nginx configuration from the host to the container.

Mender team, is it possible to fix definitely this issue by increasing the default proxy_read_timeout (60s) to 300s in the common nginx configuration?

Thank you

1 Like

@opops

You are right, to upload large files you need to increase the proxy_read_timeout in mender-api-gateway. This is going to be fixed in our next patch release, in the meantime you can make the configuration permanent as follows:

diff a/docker-compose.yml b/docker-compose.yml
index a2302f7..d3b2b66 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -42,6 +42,8 @@ services:
            service: mender-base
        networks:
            - mender
+        volumes:
+            - ./increase_timeouts.conf:/usr/local/openresty/nginx/conf/optional/endpoints/increased_timeouts.conf
        # critical - otherwise nginx may not detect
        # these servers and exits with 'upstream server not found'
        depends_on:
diff a/increase_timeouts.conf b/increase_timeouts.conf
new file mode 100644
index 0000000..22e1882
--- /dev/null
+++ b/increase_timeouts.conf
@@ -0,0 +1,2 @@
+client_max_body_size 10G;
+proxy_read_timeout 600;

This is not the optimal solution, but fixes the issue till the next release.

1 Like

Thank you @opopops and @tranchitella, that fixed the upload for me.

I’m still not able to upload a small individual file as an artifact on the web UI. It goes through the steps successfully and says it was uploaded, and it shows up in minio, but it doesn’t show up in the UI as an artifact. I’m guessing that is either an issue with the artifact generator/worker or a misunderstanding though, and I’ll create a new thread for that if necessary.

@bakern Can you please try to see if this fixes your issue?

@tranchitella Thanks for your response. Assuming I applied that image correctly, it doesn’t seem to have fixed it. I basically just changed the mender-workflows-server image to the one you gave in docker-compose.yml, and then stopped mender and started again. During startup it showed that it pulled the new image, and docker ps shows that the new image is running.

If I can help by sending any other info or testing anything else, please let me know.

@bakern can you please send me the logs from all your containers (or, if it makes your life easier, at least workflows-server and create-artifact-worker) when uploading a single file artifact from the UI?