Directory update gets spontaneously reverted

Hey there!

I’ve encountered a bug on a system that had been updated multiple times with directory updates in the course of a year.

It appears like the device downgraded itself to a very old software version, without a deployment being issued from the hosted control panel.
Curiously, I don’t see any record for this downgrade in the finished deployments tab on hosted mender.
Sadly, by the time I noticed, the logs for mender-client had rolled over so I can’t see exactly what happened in the relevant time frame.

Has anyone encountered a similar issue before?
Any clues where I can look? i.e which logs could be insightful apart from the mender-client logs

Some data:

  • Running hosted mender, with mender client version 3.0.0
  • The device is a Raspberry Pi 4
  • I’m only pushing directory updates to the device

Thanks!

Hello @g-bulgarit ,

This sounds strange, and should not happen…

It is hard to diagnose without any logs, my guess would be that an unrelated deployment (e.g. full system update) has overwritten your directory update on the device. It depends on how you see that the device has downgraded, is it based on the file system itself or the versions reported ?

I see that the device had downgraded in the ‘current software’ tab, which also corresponds with the files on the filesystem belonging to a very old update. I’ve never pushed any full-system-updates.

Even though both of these things exist, the finished-deployments tab (where I see finished updates to my entire device fleet) does not indicate that an update was issued / had been completed.

I’m monitoring the device more closely, to hopefully catch this early if it happens again, so that I will have some logs to examine…

Hi @g-bulgarit,

Were you able to confirm whether the device actually downgraded to a older software by logging into it directly? Or the hosted Mender is showing a wrong version.