Can't upload signed artifacts compressed with LZMA

ster · July 12, 2019, 3:51pm

I can’t upload signed artifacts via GUI.
Integration is on V2.0.0 and mender-artifact is on v.3.0.1. Artifacts are created using mender-artifact from .ext4. I don’t use --version with mender-artifact, so it should use artifact V3 I suppose.

mender-deplyments container throws these errors:

logs from mender-deplyments

time="2019-07-12T14:51:22Z" level=error msg="reading artifact error: readHeaderV3: handleHeaderReads: readHeader: readNext: Failed to copy from tarReader to the writer: unexpected EOF: Cannot parse artifact file" file=images.go func="controller.(*SoftwareImagesController).NewImage" line=239 request_id=bd965f2e-eb8a-41cb-b0b9-564699d67c8f user_id=5ec43064-1fe1-49bd-92b2-c3a7ee220460
time="2019-07-12T14:51:22Z" level=error msg="reading artifact error: readHeaderV3: handleHeaderReads: readHeader: readNext: Failed to copy from tarReader to the writer: unexpected EOF" file=view.go func="view.(*RESTView).RenderError" line=51 request_id=bd965f2e-eb8a-41cb-b0b9-564699d67c8f user_id=5ec43064-1fe1-49bd-92b2-c3a7ee220460

The problem only exists with signed artifact. This problematic artifact passes mender-artifact validate successfully. If the same artifact is generated without --key argument, it can be uploaded without any issue.

I checked docs about keys and it doesn’t seem that I should somehow provide Server with any artifact keys. What have I missed?

Also, little UX flaw: when this error happens, GUI shows nothing, the upload progress bar just getting stuck at random between 4% and 10%. It should probably notify the user that something went wrong.

mirzak · July 12, 2019, 4:07pm

Can you share the commands you use to create the Mender Artifact.

I tried to reproduce but was not successful, it uploaded just fine. Though I was using Hosted Mender but should be equivalent to v2.0.0.

ster · July 12, 2019, 4:22pm

Sure:

mender-artifact --compression "lzma" write rootfs-image
      --file "$releaseName-mender.ext4"
      --device-type "raspberrypi3" 
      --artifact-name "$releaseName-mender"
      --output-path "$releaseName-mender.mender"
      --script "$scriptsDir/ArtifactInstall_Enter_01_retain_wpa_supplicant"
      --key "$pathToSignKey"

The command to generate unsigned artifact is the same but without --key. I regenerated signed artifact once again and still have the issue.

mirzak · July 12, 2019, 4:32pm

Thanks, now I can see it as well. It seems to work if you remove --compression "lzma". Can you confirm this as well?

mirzak · July 12, 2019, 4:43pm

I created an bug report,

https://tracker.mender.io/browse/MEN-2645

ster · July 13, 2019, 11:33am

Thanks, now I can see it as well. It seems to work if you remove --compression "lzma" . Can you confirm this as well?

I confirm, everything works fine with gzip

oleorhagen · July 16, 2019, 5:44am

@ster, could you give the version of mender-artifact that you are using?

ster · July 16, 2019, 8:45am

As mentioned in the first message, I used mender-artifact V3.0.1, precompiled Linux binary downloaded from here

oleorhagen · July 16, 2019, 9:11am

Apologies! That was weak.

However, I am still not able to reproduce this. Downloaded the binary, and ran:

mender-artifact --compression "lzma" write rootfs-image
      --file test.img
      --device-type "raspberrypi3" 
      --artifact-name release-1
      --output-path test.mender
      --key ~/mendersoftware/integration/keys/deviceauth/private.key

oleorhagen · July 16, 2019, 9:14am

That is with hosted mender, and the demo environment locally.
I am not doubting that you are having issues, just to be clear, so bear over with me.
I will have a look at the backend

ster · July 16, 2019, 9:39am

Maybe the keys are the key?
I use RSA with length of 3072 bits

ster · July 16, 2019, 9:42am

also I (as well as @mirzak) am generating artifact from .ext4 and not .img, just a note.

oleorhagen · July 16, 2019, 11:09am

So is the key in the integration repo
I also, just to be sure, did convert the raw image to an ext4 image, to no avail.
@kacf, perhaps you have some ideas here?

mirzak · July 16, 2019, 12:10pm

You are also missing a state script in your command, which the original command had to reproduce

oleorhagen · July 16, 2019, 3:32pm

I’ve tried both with and without scripts, still works.
Tomorrow I will make a bash script to run the whole thing, so that we can standardize this

mirzak · July 16, 2019, 4:29pm

For additional reference, when I performed my test I downloaded this image,

https://d1b0l86ne08fsf.cloudfront.net/2.0.1/raspberrypi3/raspberrypi3_release_1_2.0.1.mender

Unpacked it, and then created a Mender Artifact with the mentioned command using the files from the extracted Artifact

oleorhagen · July 17, 2019, 8:41am

Alright, have a look at this, and if this is the proper workflow, see if it works at yours:
https://pastebin.com/rnmsYheZ

ster · July 17, 2019, 10:15am

@oleorhagen, I used your script with the only difference that I used GUI instead of mender-cli to upload the artifact and the issue is present, with the same logs from mender-deployments.

One additional thing is that I have relatively low upload speed with my ISP (1-2 Mbit/s), maybe this could play the role?

Here I uploaded test.mender (result of your script) I have issues with, can you try this file?
https://we.tl/t-78hBQlwY3b

And the key pair:

testkey.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEQyJuskzajlaW3Bd8JSYFF6AjH1CgJvSsOugywQOz21nofi2jEm5qpa/rL9+0OUX+7Vw4rIw+3tGIltL43yNylm49TJCn32oWoMjUGXFMzReifD9/ixBb+9QKDC2s8e8MMqYUSM4R0VQbQZedfj+q3pBAxcEUJEdIL6r+nynxy90GCJm+LAQU+F/UQYtKMP8dKTl+0BcSVifKTy1zFVSs16PEjj/ahQTOSZ4tJAuUZ1k2Tq+gfCifAQ+Ke6xxHKK30NGB+G3VkmjvshiI7pQyipBw+9MisWOE9kjpJYLsAYj5mAXy8BuqfiTOzrOtxN1pAvzUTc0MLxxDOYOH3yhzKpQY4wUkTyBxDjpBqTEBDraXIzHUD1+8T4DTNyH8vkctqzHXuFrm8aDjoTNfWFYoRrZY+MuV2IFn1b92Hboz8vqtKgc9S0znnRboVykiX0LfGHe0jCbeZ0J1GN4EqhiFgzZcst52rflKRJqJhfwX1ijhFqj7689m7qcN5XM5EXU= ster@sterPC

testkey

oleorhagen · July 17, 2019, 10:44am

Hmm, that is interesting indeed, cause the artifact you linked uploaded just fine for me.
I guess this calls for some knowledge from @0lmi

0lmi · July 17, 2019, 10:54am

I used your script with the only difference that I used GUI instead of mender-cli to upload the artifact

Did you try to use mender-cli? Does it work for you?

And how does you infrastructure look like?