Server matches wrong artifact during deployment

We have a test machine booted with release version 6 (which was installed manually with a rootfs-image local file with mender install version6.mender).

Our cloud server for release version 1 has these artifacts uploaded:

  • rootfs-image for 1
  • mender-binary-delta from 0 to 1
  • mender-binary-delta from 2 to 1

The differentials are produced by mender-binary-delta-1.1.0.
If I were to start a deployment to that test machine for release 1, one would expect the server would choose rootfs-image for 1. In reality server sends mender-binary-delta from 2 to 1 which of course fails (“rootfs_image_checksum” not satisfied by currently installed artifact ([hash_of_2] != [hash_of_6]).
Deleting mender-binary-delta from 2 to 1 and retrying, the server picks the other differential and fails again.
Deleting every other differential, the server finally chooses rootfs-image for 1 and succeeds.
Uploading another differential from 6 to 1 (while keeping the rootfs-image for 1) chooses correctly that differential and installs it fine.
Can something be wrong on the test machine or is it a bug on the server?

@bender a couple of questions:

  1. Are you using Hosted Mender, or is it self-hosted?
  2. How do you set up the deployment? The exact list of steps would be beneficial.
  1. Hosted Mender
  2. Choose the version 1 release, scroll to the bottom and click on “Create deployment with this release”. Select the group where the test machine is the only member > Next > Next > Create

Hey @bender,

I’d like you to provide some info for further troubleshooting:

  1. Can you call mender-artifact read <ARTIFACT> on all three artifacts and post the output here?
  2. Can you call mender show-provides on the device, and post the output here?

@kacf I replied as a DM…

Thank you, I’ve verified that there is nothing wrong with the artifacts themselves, at least not as far as I can see.

Our cloud server for release version 1 has these artifacts uploaded:

You did say Hosted Mender earlier, but I just want to double check: When you say “Our cloud server”, you mean that you host it yourself?

The reason I ask is that the Open Source backend does not support dependency resolution, this is an Enterprise feature. The Open Source backend just ignores them and picks the smallest artifact available, so it would produce the results you see.

Apologies for the confusion. We are on a Mender Professional account hosted by you. The differential feature is something we have used a lot already so its not new.

@bender: According to Features | Mender, “Automatic assignment of delta updates” is only available in Enterprise plans. In Professional and lower plans they will behave like the Open Source version, dependencies are ignored. How have you been installed them previously?

if the selection on non-Enterprise plans is by choosing the smaller size then it may be the case that by chance that was the expected diff to be chosen leaving us the false impression that it works with multiple diffs uploaded.
I guess the only way is to keep only one diff along with the full rootfs-image. Correct?

Without the “automatic assignment of delta updates” feature, you basically need to name each artifact independently and ensure that you are deploying with the correct one.

Honestly I don’t know if the behavior is defined where you have multiple artifacts, named the same, with the same compatible machines list but one is full image and one is delta.

@kacf @tranchitella is this defined? Should we maybe issue an error and not allow this situation?

Drew

What do you mean? The only field I have access to modify is the description and the release number.
The description I believe is an optional free-text field for informational purposes. The release number defeats the purpose of having a differential update as the target machine will end up with a different release installed. I think you mean something else…

So are you saying that if we have one full rootfs-image and one mender-binary-delta for the same device type for the same release, it is not guaranteed that always the delta will be chosen in case the target machine’s hash match the delta’s depends hash? I’m finding hard to believe that we have been that lucky so far in our tests…

It’s defined: It always picks the smallest Artifact if both match the device type.

@drewmoseley is right, if you keep both the diff and the rootfs-image, then the rootfs-image will never be selected. In the absence of automatic delta assignment you will have to name each delta artifact differently, since each one needs to be its own release to be selected. I believe you can use mender-artifact modify -n NAME on the delta artifact to do name it rootfs-v2-from-v1 for example. This still keeps software versioning as is, only the name of the artifact changes.

Of course another choice is to bite the bullet and go Enterprise, then all this is handled automatically and you won’t have to worry about it.

You got my meaning correctly here. Without the automatic assignment you need to use separate release numbers.