I ran into a problem with mender when trying to downgrade a device from a yocto zeus rootfs (mender client version 2.2.0) to a previous thud rootfs (mender client version 1.7.0).
What is observed is that the downgrade is done but after a reboot (the reboot is not initiated by mender), the device goes back to the zeus rootfs. And it gets even worse: it is now impossible to upgrade/downgrade from mender.
What happen is:
- Once updated to mender 2.2.0,
/data/mender-storewill store states in “version 2”
- In case of a downgrade, the mender 1.7.0 client is unable to read state:
mender: time="2020-04-30T16:21:51+02:00" level=error msg="failed to restore state data: unsupported state data version" module=state
- So it considers the update as failed, and want to upload status report:
mender: time="2020-04-30T16:21:51+02:00" level=info msg="State transition: init [none] -> update-error [ArtifactFailure]" module=mender mender: time="2020-04-30T16:21:51+02:00" level=info msg="State transition: update-error [ArtifactFailure] -> update-status-report [none]" module=mender
- But as it was unable to read the state, it doesn’t know the deployment id so it tries to connect to
/api/devices/v1/deployments/device/deployments/unknown/statusand got 500 error
mender: time="2020-04-30T16:21:51+02:00" level=error msg="got unexpected HTTP status when reporting status: 500" module="client_status"
- As the update is never commited, a reboot makes that the device boots on zeus partition
- From there, there is always the state “update-status-report” that tries to send to
/api/devices/v1/deployments/device/deployments/unknown/status, and mender is “bricked”, as it won’t ever change state.
As a workaround, we can delete
/data/mender/store and it will “unbrick” mender, then downgrade and once downgrade is done, again delete
/data/mender/store, restart mender and run
You will understand that this issue is quite serious because:
- It prevents mender from downgrading
- It “bricks” mender itself
I will try to add mender 2.2.0 to my yocto thud image, but I think it should be fixed or at least be clearly specified in the documentation that you can’t go back from a mender client > 2.0 to a < 2.0.
Or maybe I am missing something ?