Downgrade from yocto zeus to thud and state version incompatibilty

General Discussions
1

Hi,

I ran into a problem with mender when trying to downgrade a device from a yocto zeus rootfs (mender client version 2.2.0) to a previous thud rootfs (mender client version 1.7.0).

What is observed is that the downgrade is done but after a reboot (the reboot is not initiated by mender), the device goes back to the zeus rootfs. And it gets even worse: it is now impossible to upgrade/downgrade from mender.

What happen is:

  • Once updated to mender 2.2.0, /data/mender-store will store states in “version 2”
  • In case of a downgrade, the mender 1.7.0 client is unable to read state:
mender[226]: time="2020-04-30T16:21:51+02:00" level=error msg="failed to restore state data: unsupported state data version" module=state
  • So it considers the update as failed, and want to upload status report:
mender[226]: time="2020-04-30T16:21:51+02:00" level=info msg="State transition: init [none] -> update-error [ArtifactFailure]" module=mender
mender[226]: time="2020-04-30T16:21:51+02:00" level=info msg="State transition: update-error [ArtifactFailure] -> update-status-report [none]" module=mender
  • But as it was unable to read the state, it doesn’t know the deployment id so it tries to connect to /api/devices/v1/deployments/device/deployments/unknown/status and got 500 error
mender[226]: time="2020-04-30T16:21:51+02:00" level=error msg="got unexpected HTTP status when reporting status: 500" module="client_status"
  • As the update is never commited, a reboot makes that the device boots on zeus partition
  • From there, there is always the state “update-status-report” that tries to send to /api/devices/v1/deployments/device/deployments/unknown/status, and mender is “bricked”, as it won’t ever change state.

As a workaround, we can delete /data/mender/store and it will “unbrick” mender, then downgrade and once downgrade is done, again delete /data/mender/store, restart mender and run mender -commit.

You will understand that this issue is quite serious because:

  • It prevents mender from downgrading
  • It “bricks” mender itself

I will try to add mender 2.2.0 to my yocto thud image, but I think it should be fixed or at least be clearly specified in the documentation that you can’t go back from a mender client > 2.0 to a < 2.0.

Or maybe I am missing something ?

2

It’s correct, downgrading to 1.7.x from a 2.x.x version is not supported, except when the upgrade from 1.7.x to 2.x.x fails and it has to roll back. In this specific case it is supported.

Not for that deployment, but if you make a new deployment, will it not be part of it?

3

It’s correct, downgrading to 1.7.x from a 2.x.x version is not supported, except when the upgrade from 1.7.x to 2.x.x fails and it has to roll back. In this specific case it is supported.

Well, it’s a bummer, very surprising for an update software and a very important thing to say in documentation !

Not for that deployment, but if you make a new deployment, will it not be part of it?

No, as I said, the mender daemon stays in state update-status-report and won’t change. It won’t even update inventory or check for new updates

4

You’re right, I have added a note about it here.

1 Like