Upload of Mender Artifacts constantly failing due to POST EOF FAILURE, likely a server side problem

General Discussions
, ,
1

I’m trying to upload different mender artifacts via the mender-cli and browser, both are failing. With the mender-cli I get this error message:

machine:~$ mender-cli artifacts --server https://hosted.mender.io upload core-image-base-xxx-20200603064344.mender
460.88 MiB / 460.88 MiB [======================================================================================================================================================================================================] 100.00% 19s
Processing uploaded file. This may take around one minute.

FAILURE: POST /artifacts request failed: Post https://hosted.mender.io/api/management/v1/deployments/artifacts: EOF

I’m pretty sure this is a server side problem, as on May 29, 2020 the same artifact was uploaded successfully. Tried it on June 2, 2020 again and it gave me the above error again and again, for different artifacts.

Can you please help here.

2

Interesting. I was not able to reproduce however. I take it this is consistent.

In general uploading the same Artifact twice should be an error anyways though. This is what I get:

FAILURE: artifact upload failed with status 422, reason: {"error":"Artifact not unique","request_id":"274e366b-486c-4450-880e-43202b9168d5"}

But I feel like I have seen this EOF failure somewhere before. Give me a few minutes.

3

Thanks for your quick reply. I have to add, that we are on Yocto warrior and thus using tag warrior-v2020.05 of meta-mender.

4

Hi, I work on the same project with deffo. Unfortunately I see the same bug in a different build environment. But I could “hot-fix” the bug by reducing MENDER_STORAGE_TOTAL_SIZE_MB from about 30GB to 4GB. I’m very confused, that can/should have nothing to do with it?

5

Hm, we have had similar issues before, one example:

https://tracker.mender.io/browse/MEN-2428

Above is fixed though, and not sure if it is related.

There was also another thread on this topic, Mender-proxy/deployments unable to upload files to minio. This was related to on-premise though, and should be resolved in hosted Mender.

In conclusion, there has been issues with very large Artifacts, where the API gateway timeouts.

@tranchitella probably has the most up to date information here.

6

Any news here? The problem is kind of critical for us as we cannot use our desired partition layout. How can we help fix the problem? The strange thing in our case is that the actual artifact size does not seem to be the problem.

7

We’re having the same issue here with a 1.02GiB artifact. The upload fails consistently with the EOF error:
FAILURE: POST /artifacts request failed: Post https://hosted.mender.io/api/management/v1/deployments/artifacts: EOF

8

Alright, good to hear @nielsavonds. We are investigating the issue atm. However, it seems to be somewhat tricky to dig this one out.

Are you uploading through the UI, or how are you uploading your Artifacts? We seem to only be able to reproduce this with the mender-cli tool.

9

We are using mender-cli, haven’t tried the gui yet. This is our Jenkins server doing the upload.

10

I see. This should be a temporary fix.

I’m not sure if it is deployed yet or not. @merlin can you confirm this?

11

We found an issue with the proxy causing requests with chunked transfer-encoding (which mender-cli uses) to become stale for too long and time out. This issue should now be resolved in Hosted Mender.

12

Unfortunately, we see a new variant of the error. The upload works, but our pipeline (jenkins) stucks at the upload step forever. We have been seeing this behavior since December 22. (Hosted Mender (Enterprise), CLI-1.3 / CLI-1.5)

Do you have any ideas?

grafik

13

Hi @lukas_e I just did a quick test on my desktop with an Artifact of 325MiBs, and it uploaded just fine to my hosted mender account. Is this present only on your CI service, or can you reproduce locally as well?

14

Hi @oleorhagen, sorry for the late response. Unfortunately, I haven’t had much time for further investigations. In a first local test, the behavior actually did not occur :confused: . I will let you know when I have new information.
Thanks for you help

1 Like