looking inside the docker mender-proxy logs i see a pattern when uploading the txt files
(PUT GET DELETE PUT)
but not when uploading my .mender file
(only PUT GET)
Is there a way to see if the certificates are faulty and blocking the correct working of my uploads? because i still assume the problem is still something in those lines.
Mender server versions prior to 2.1.0 had a bug where upload would sometimes fail depending on exact block sizes used by the upload tool (the browser). Most people were able to work around it by using a different browser.
I am having a case similar (mender production 2.3.0) to what you comment in your query,
In addition to the same thing happening with the artifacts, i see this:
two containers are like healthy (mender conductor, mender minio).
I can upload any type of file, as long as it doesn’t exceed 1Mb.
I’m having the same thing with a mender production 2.3.0 instance, fresh install.
If I upload a very small artifact on the web interface it says it was successful, and I can see the file in minio, but it doesn’t show up on the Releases page. If I upload a large artifact it gets to 100% and just sits there.
If I upload a large artifact using mender-cli I get the same 504 timeout.
Has anyone solved this yet, or have any advice on what to check? After looking at logs I don’t see anything obvious, other than using self-signed certificates, but I thought that was okay.
This fix is not persistent, if you restart your server it will be lost… The solution is to use a docker volume to mount the customized nginx configuration from the host to the container.
Mender team, is it possible to fix definitely this issue by increasing the default proxy_read_timeout (60s) to 300s in the common nginx configuration?
You are right, to upload large files you need to increase the proxy_read_timeout in mender-api-gateway. This is going to be fixed in our next patch release, in the meantime you can make the configuration permanent as follows:
diff a/docker-compose.yml b/docker-compose.yml
index a2302f7..d3b2b66 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -42,6 +42,8 @@ services:
service: mender-base
networks:
- mender
+ volumes:
+ - ./increase_timeouts.conf:/usr/local/openresty/nginx/conf/optional/endpoints/increased_timeouts.conf
# critical - otherwise nginx may not detect
# these servers and exits with 'upstream server not found'
depends_on:
diff a/increase_timeouts.conf b/increase_timeouts.conf
new file mode 100644
index 0000000..22e1882
--- /dev/null
+++ b/increase_timeouts.conf
@@ -0,0 +1,2 @@
+client_max_body_size 10G;
+proxy_read_timeout 600;
This is not the optimal solution, but fixes the issue till the next release.
I’m still not able to upload a small individual file as an artifact on the web UI. It goes through the steps successfully and says it was uploaded, and it shows up in minio, but it doesn’t show up in the UI as an artifact. I’m guessing that is either an issue with the artifact generator/worker or a misunderstanding though, and I’ll create a new thread for that if necessary.
@tranchitella Thanks for your response. Assuming I applied that image correctly, it doesn’t seem to have fixed it. I basically just changed the mender-workflows-server image to the one you gave in docker-compose.yml, and then stopped mender and started again. During startup it showed that it pulled the new image, and docker ps shows that the new image is running.
If I can help by sending any other info or testing anything else, please let me know.
@bakern can you please send me the logs from all your containers (or, if it makes your life easier, at least workflows-server and create-artifact-worker) when uploading a single file artifact from the UI?
Hello, any news on this topic? I have tested the new version of 2.4.0b1 in a completely clean instance, and the mender-gui and minio containers are still healthy.
I am still able to upload artifacts of up to 1mb, nothing above that is loaded in minium or in artifacts in the control web gui …
Is this version supposed to be correcting this problem? or should we wait for a higher version?
I’m told by our backend team that this is apparently fixed for the master branch, and in Hosted Mender. It is not yet cherry-picked to our 2.4.x release branch, but it will be soon, so the fix should be part of the 2.4.0 final.
