# Mender-proxy/deployments unable to upload files to minio

**URL:** <https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637>\
**Category:** General Discussions\
**Tags:** minio, proxy, deployment\
**Created:** [March 12, 2020, 1:12pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637 "2020-03-12T13:12:21Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![genedupre](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/genedupre/32/289_2.png) [@genedupre](https://hub.mender.io/u/genedupre)\
**Post date:** [March 12, 2020, 1:12pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/1 "2020-03-12T13:12:21Z")

</div>

Hello everyone,  
beginning this week i’ve made a second mender-server to test the new 2.3 and migrate my 1.8 to it.

everything seems to be working fine, but when i want to upload an artifact, the upload hangs forever on 100%.  
yet i can upload very small txt files ↓

 ![mender-2.3-updloadhanging2](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/a78a8badf63ebd400ea62d35a6525db322cca614.png)  
my first instinct was that my certificates where probably not right, after redoing that, that seems not to be the case.

> CERT\_API\_CN=[menderdev.mywebsite.com](http://menderdev.mywebsite.com) CERT\_STORAGE\_CN=[menderdev-minio.mywebsite.com](http://menderdev-minio.mywebsite.com) …/keygen

 ![2020-03-12_13-37](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/1X/446eedaab27a7bda3459ddd6808459fd75bd9ea5.png)

looking inside the docker mender-proxy logs i see a pattern when uploading the txt files  
(PUT GET DELETE PUT)  
but not when uploading my .mender file  
(only PUT GET)

log docker mender-proxy↓  
[https://pastebin.com/NeskiH5d](https://pastebin.com/NeskiH5d)

Is there a way to see if the certificates are faulty and blocking the correct working of my uploads? because i still assume the problem is still something in those lines.

i have made a pastebin of the compose file also if needed (— confidental)  
[https://pastebin.com/XTi0MdwB](https://pastebin.com/XTi0MdwB)

Thanks in advance, Gene

---

<div class="post-metadata">

**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\
**Post date:** [March 13, 2020, 7:11am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/2 "2020-03-13T07:11:04Z")

</div>

Mender server versions prior to 2.1.0 had a bug where upload would sometimes fail depending on exact block sizes used by the upload tool (the browser). Most people were able to work around it by using a different browser.

---

<div class="post-metadata">

**Author:** ![genedupre](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/genedupre/32/289_2.png) [@genedupre](https://hub.mender.io/u/genedupre)\
**Post date:** [March 16, 2020, 2:56pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/3 "2020-03-16T14:56:54Z")

</div>

Hello @kacf  
After redoing my mender-server 2.3 several times, iam still unable to upload any .menders, even in different browsers.

Are there any other clues i could follow to debug this?  
Greets

---

<div class="post-metadata">

**Author:** ![genedupre](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/genedupre/32/289_2.png) [@genedupre](https://hub.mender.io/u/genedupre)\
**Post date:** [March 16, 2020, 6:17pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/4 "2020-03-16T18:17:22Z")

</div>

so i tried uploading my .mender using the api with mender-cli

this give me somewhat an idea what is going wrong

```
 912.07 MiB / 912.07 MiB [=====] 100.00% 44m50s
Processing uploaded file. This may take around one minute.

FAILURE: artifact upload failed with status 504, reason: <html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>openresty/1.13.6.2</center>
</body>
</html>

```

looking back into the docker logs been generated, the openresty shows an PUT and DELETE, but not really errors

> 172.26.0.1 - - [16/Mar/2020:16:59:46 +0000] “GET / HTTP/1.1” 403 206 “-” “check\_http/v2.2.1.git (nagios-plugins 2.2.1)”  
> 172.26.0.1 - - [16/Mar/2020:17:25:28 +0000] “PUT /mender-artifact-storage/83731ca9-a4e9-4e43-a0a0-810792b168f2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=mender-deplo  
> yments%2F20200316%2Fus-east-1%2Fs3%2Faws4\_request&X-Amz-Date=20200316T172424Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=8bc51c502af00c04faf79ba3a99bb62b41fcc988ca11c669fbdff90af6a384d2 HTTP/1.1”  
> 200 0 “-” “Go-http-client/1.1”  
> 172.26.0.1 - - [16/Mar/2020:17:25:28 +0000] “DELETE /mender-artifact-storage/83731ca9-a4e9-4e43-a0a0-810792b168f2 HTTP/1.1” 204 0 “-” “aws-sdk-go/1.12.27 (go1.11.4; linux; amd64)”

---

<div class="post-metadata">

**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\
**Post date:** [March 17, 2020, 8:51am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/5 "2020-03-17T08:51:46Z")

</div>

@genedupre can you please provide us the logs of your storage-proxy container?

---

<div class="post-metadata">

**Author:** ![genedupre](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/genedupre/32/289_2.png) [@genedupre](https://hub.mender.io/u/genedupre)\
**Post date:** [March 18, 2020, 12:15pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/6 "2020-03-18T12:15:37Z")

</div>

Hello @tranchitella  
ive generated some logs and put them here  
[https://filebin.net/bh7nk0f9gwyvbzkd](https://filebin.net/bh7nk0f9gwyvbzkd)

Thanks for looking into it

---

<div class="post-metadata">

**Author:** ![blackdevice](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/blackdevice/32/394_2.png) [@blackdevice](https://hub.mender.io/u/blackdevice)\
**Post date:** [March 21, 2020, 3:27pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/7 "2020-03-21T15:27:49Z")

</div>

Hi genedupre,

I am having a case similar (mender production 2.3.0) to what you comment in your query,  
In addition to the same thing happening with the artifacts, i see this:

- two containers are like healthy (mender conductor, mender minio).
- I can upload any type of file, as long as it doesn’t exceed 1Mb.

Has this case been detected for you?

Thank you!

---

<div class="post-metadata">

**Author:** ![genedupre](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/genedupre/32/289_2.png) [@genedupre](https://hub.mender.io/u/genedupre)\
**Post date:** [March 23, 2020, 10:14am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/8 "2020-03-23T10:14:01Z")

</div>

Hello @blackdevice  
yes both these cases are also present in my problems  
when i have time i will look into it more  
Greets Gene

---

<div class="post-metadata">

**Author:** ![bakern](https://avatars.discourse-cdn.com/v4/letter/b/3bc359/32.png) [@bakern](https://hub.mender.io/u/bakern)\
**Post date:** [March 31, 2020, 6:25pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/9 "2020-03-31T18:25:31Z")

</div>

I’m having the same thing with a mender production 2.3.0 instance, fresh install.

If I upload a very small artifact on the web interface it says it was successful, and I can see the file in minio, but it doesn’t show up on the Releases page. If I upload a large artifact it gets to 100% and just sits there.

If I upload a large artifact using mender-cli I get the same 504 timeout.

Has anyone solved this yet, or have any advice on what to check? After looking at logs I don’t see anything obvious, other than using self-signed certificates, but I thought that was okay.

Thanks,  
Nate

---

<div class="post-metadata">

**Author:** ![opopops](https://avatars.discourse-cdn.com/v4/letter/o/ed655f/32.png) [@opopops](https://hub.mender.io/u/opopops)\
**Post date:** [March 31, 2020, 10:22pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/10 "2020-03-31T22:22:29Z")

</div>

Hello,

I have the same issue after a fresh install of mender version 2.3 when I try to upload an artficat that exceed 300MB.

The time out log found in my docker logs:

```auto
mender-api-gateway_1 | 2020/03/31 20:31:50 [error] 32#32: *1115 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.110.194.15, server: mender.hosted.io, request: "POST /api/management/v1/deployments/artifacts HTTP/1.1", upstream: "http://172.18.0.14:8080/api/management/v1/deployments/artifacts", host: "mender.hosted.io", referrer: "https://mender.hosted.io/ui/"

```

So this time out issue is generated by the mender-api-gateway service on the location _api/management/v1/deployments/artifacts_

I have a quick fix:

- Connect into the **mender-api-gateway** container:

```shell
./run exec mender-api-gateway sh

```

- Edit _/usr/local/openresty/nginx/conf/common.nginx.conf_ and update location **/api/management/v1/deployments/artifacts** section to add **proxy\_read\_timeout 300;**

```auto
location = /api/management/v1/deployments/artifacts {
        auth_request /userauth;
        auth_request_set $requestid $upstream_http_x_men_requestid;

        client_max_body_size 10G;
        proxy_read_timeout 300;

```

- reload openresty:

```shell
openresty -s reload

```

This fix is not persistent, if you restart your server it will be lost… The solution is to use a docker volume to mount the customized nginx configuration from the host to the container.

Mender team, is it possible to fix definitely this issue by increasing the default **proxy\_read\_timeout** (60s) to 300s in the common nginx configuration?

Thank you

---

<div class="post-metadata">

**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\
**Post date:** [April 1, 2020, 3:16pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/11 "2020-04-01T15:16:28Z")

</div>

@opops

You are right, to upload large files you need to increase the proxy\_read\_timeout in mender-api-gateway. This is going to be fixed in our next patch release, in the meantime you can make the configuration permanent as follows:

```
diff a/docker-compose.yml b/docker-compose.yml
index a2302f7..d3b2b66 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -42,6 +42,8 @@ services:
            service: mender-base
        networks:
            - mender
+ volumes:
+ - ./increase_timeouts.conf:/usr/local/openresty/nginx/conf/optional/endpoints/increased_timeouts.conf
        # critical - otherwise nginx may not detect
        # these servers and exits with 'upstream server not found'
        depends_on:
diff a/increase_timeouts.conf b/increase_timeouts.conf
new file mode 100644
index 0000000..22e1882
--- /dev/null
+++ b/increase_timeouts.conf
@@ -0,0 +1,2 @@
+client_max_body_size 10G;
+proxy_read_timeout 600;

```

This is not the optimal solution, but fixes the issue till the next release.

---

<div class="post-metadata">

**Author:** ![bakern](https://avatars.discourse-cdn.com/v4/letter/b/3bc359/32.png) [@bakern](https://hub.mender.io/u/bakern)\
**Post date:** [April 1, 2020, 8:33pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/12 "2020-04-01T20:33:03Z")

</div>

Thank you @opopops and @tranchitella, that fixed the upload for me.

I’m still not able to upload a small individual file as an artifact on the web UI. It goes through the steps successfully and says it was uploaded, and it shows up in minio, but it doesn’t show up in the UI as an artifact. I’m guessing that is either an issue with the artifact generator/worker or a misunderstanding though, and I’ll create a new thread for that if necessary.

---

<div class="post-metadata">

**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\
**Post date:** [April 2, 2020, 3:42am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/13 "2020-04-02T03:42:51Z")

</div>

@bakern Can you please try to see if this fixes your issue?

> [@Mender Production update from v2.2.0 to v2.3.0](https://hub.mender.io/t/mender-production-update-from-v2-2-0-to-v2-3-0/1554/25):
>
> @blackdevice this is unfortunately a bug; may I ask you to run the latest master of the workflows service to verify if it solves your issue? image: mendersoftware/workflows:master\_218ef4159f6ff21361d58d4267b07e107f8a3e35

---

<div class="post-metadata">

**Author:** ![bakern](https://avatars.discourse-cdn.com/v4/letter/b/3bc359/32.png) [@bakern](https://hub.mender.io/u/bakern)\
**Post date:** [April 2, 2020, 6:40am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/14 "2020-04-02T06:40:14Z")

</div>

@tranchitella Thanks for your response. Assuming I applied that image correctly, it doesn’t seem to have fixed it. I basically just changed the mender-workflows-server image to the one you gave in docker-compose.yml, and then stopped mender and started again. During startup it showed that it pulled the new image, and `docker ps` shows that the new image is running.

If I can help by sending any other info or testing anything else, please let me know.

---

<div class="post-metadata">

**Author:** ![tranchitella](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/tranchitella/32/606_2.png) [@tranchitella](https://hub.mender.io/u/tranchitella)\
**Post date:** [April 2, 2020, 6:51am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/15 "2020-04-02T06:51:42Z")

</div>

@bakern can you please send me the logs from all your containers (or, if it makes your life easier, at least workflows-server and create-artifact-worker) when uploading a single file artifact from the UI?

---

<div class="post-metadata">

**Author:** ![blackdevice](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/blackdevice/32/394_2.png) [@blackdevice](https://hub.mender.io/u/blackdevice)\
**Post date:** [June 16, 2020, 3:49pm UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/16 "2020-06-16T15:49:26Z")

</div>

Hello, any news on this topic? I have tested the new version of 2.4.0b1 in a completely clean instance, and the mender-gui and minio containers are still healthy.  
I am still able to upload artifacts of up to 1mb, nothing above that is loaded in minium or in artifacts in the control web gui …

Is this version supposed to be correcting this problem? or should we wait for a higher version?

Thank you!

---

<div class="post-metadata">

**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\
**Post date:** [June 18, 2020, 8:03am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/17 "2020-06-18T08:03:59Z")

</div>

I’m told by our backend team that this is apparently fixed for the master branch, and in Hosted Mender. It is not yet cherry-picked to our 2.4.x release branch, but it will be soon, so the fix should be part of the 2.4.0 final.

---

<div class="post-metadata">

**Author:** ![blackdevice](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/blackdevice/32/394_2.png) [@blackdevice](https://hub.mender.io/u/blackdevice)\
**Post date:** [June 18, 2020, 8:29am UTC](https://hub.mender.io/t/mender-proxy-deployments-unable-to-upload-files-to-minio/1637/18 "2020-06-18T08:29:29Z")

</div>

Ok!, good news knowing you have solved it.  
Thank you! we wait for the new version 🙂