Mender declining artifact from file system, accepting from http

General Discussions
1

Hi all,

I’ve got an artifact which gets accepted by mender client 2.5.2 only when presented via http. When installing from a local file, mender fails like so:

INFO[0000] Installer: authenticated digital signature of artifact 
ERRO[0000] Reading headers failed: installer: failed to read Artifact: readHeaderV3: handleHeaderReads: reader: reading header error: invalid checksum; expected: [d43e9e0e7f812c1b4d1eaafc9fcb2389dd3dd4d110812fab08c70a63f780d236]; actual: [ef6b4d69cfbed50c947e7b9eb6639e2922d68b918bfbd6c0ede9f986f34c5f0e] 
INFO[0000] Collected output (stderr) while running script /etc/mender/scripts/Download_Error_10_sendState

The manifest file in the artifacts root dir (for those who don’t know, mender artifacts are posix tar archives) reads

d43e9e0e7f812c1b4d1eaafc9fcb2389dd3dd4d110812fab08c70a63f780d236  header.tar.gz

, so I expected header.tar.gz to have an incorrect checksum as reported by mender. But to my surprise, the file appears valid:

$ sha256sum header.tar.gz 
d43e9e0e7f812c1b4d1eaafc9fcb2389dd3dd4d110812fab08c70a63f780d236  header.tar.gz

Is there a known problem in mender client 2.5.2? I checked the changelog but didn’t see anything obvious.

Thanks, regards,
Manuel

2

Hello mender support,

feel free to take a look and inspect the artifact in question yourself. It is 0-nellieOS-2.13.8-drdc.1.1.0-vps19dev-cifeaturebuild-500. It appears valid to me. If you want, we can do a tmate session so you can observe the installation procedure fail yourself.

3

Hi @manuel_vps,

Really sorry for the delay! Here’s a couple of pointers to hopefully get on track to resolve the topic.

  • can you check with the current client instead of 2.5.2?
  • is this reproducible also with a small dummy file? Like 1M of zeroes or such?
  • and related, does it apply to system updates only, or also to application updates like the simple file dropper?

Thanks!

4

Hi @TheYoctoJester ,

  • can you check with the current client instead of 2.5.2?

will do

  • is this reproducible also with a small dummy file? Like 1M of zeroes or such?

Do you suggest the mender artifact itself to be 1M of zeroes, or the payload contained inside the mender artifact to be 1M of zeroes? Please note we are using mender to deploy rootfs updates; not just individual files. Would need to look into how to create a mender rootfs artifact containing a pseudo rootfs .tar.gz.

  • and related, does it apply to system updates only, or also to application updates like the simple file dropper?

Never did anything else than system updates. Also, the problem was observed only in one artifact yet.

5

Hi @TheYoctoJester,

I let yocto recreate the rootfs artifact from the same commit, and this new artifact does not suffer from the same issues. That is, the mender-client accepts it even when presented on filesystem. (Meanwhile, the initial one still gets declined. It was not just a one-time occurence.)

This is not quite satisfactory for me as I’d really like to find out what caused the initial artifact to be declined, but as it seems to work for now, I’ll just leave this be and hope this issue won’t resurface. If it does I’ll have to dig deeper.

The two artifacts have subtle differences content-wise. They were created with different MENDER_ARTIFACT_NAMEs as the CI inserts the build job number there, and also some content in the rootfs gets created differently with every yocto bitbake run.

Thanks