Mender OTA Updates - Client power reset interrupt during update

Hello,

I’ve been experimenting with interrupting the mender client during an OTA system update (deployment) instigated from our Mender Enterprise Server.

Essentially the experiment compared the time taken for updates with no interruption to the client versus update time taken when the client is interrupted (power reset).
The power reset interrupts were instigated when the progress indicator was at greater than 50%.
Mender version 2.5.1
System File Sizer : 400 MB

Results.
Probably as you would expect:-
The number of retries had to be increased to greater than zero to achieve a successful update when the Client was interrupted,.
The update time increased significantly when a power reset interrupted the client during an update, inferring that if a power interrupt occurs any part of the system file that has been uploaded to the client is lost and the file update has to transfer the whole file on the retry.

Which is not a good outcome in our situation where system updates would be to multiple clients over a possible limited bandwidth.

Can you confirm these results are as expected - that the mender file is not fragmented during file transfer and so any part of the file transferred is lost if the client is reset during this process?
Can you advise whether there are any options presently available, or planned for the future, which will cater for interruptions to the client device during a system update, i.e. have a file transfer history so only the remaining part of the system file needs to be sent after the interruption?

Hi @GrahamTooms

The client should indeed pick up from where it left off, as long as it is the same deployment which is being retried.

The power reset interrupts were instigated when the progress indicator was at greater than 50%.

Dirty secret: The progress indicator is simply a mock. It has no bearing on the reality of the update process.

Can you confirm these results are as expected - that the mender file is not fragmented during file transfer and so any part of the file transferred is lost if the client is reset during this process?

The client does honour the HTTP Content Range headers in the udpate, so it should be able to restart at where it left off in the face of issues.

On second look, I’m afraid it looks like I was wrong.

The client currently only picks up from where it left off in the case of dropped connections and interrupts, not in the case of spontaneous reboots.

It seems it has been close previously, and it is also on our road-map, so it will eventually show up as a feature, whenever it gains priority :slight_smile:

Sorry for the initial confusion.

Based on the historical notes in https://tracker.mender.io/browse/MEN-2293 it looks like there was a working solution for ‘image’ updates back in 2019, but it didn’t address all the different types of artifacts (ie. using update modules).

I’m quite surprised that this feature is not already in the mender functionality, but maybe a lot of other customers don’t turn off the devices. The products that we’re using can be turned off (isolated) at any time for safety reasons and having to restart the download from scratch can potentially prevent the update from occurring at all (eg. download to 99%, restart, download to 99%, restart,…)

If the feature is already on the roadmap, can we have an idea as to which upcoming version (and timing) it’s likely to be available in ? Maybe the roadmap is publicly visible and I haven’t looked hard enough for it, in which case I apologize.

Hi, yes, this has been near a few times, then it has slipped away.

Unfortunately our roadmap is not public atm, but perhaps @eystein could offer some help here?

We do offer roadmap acceleration if you are so inclined.
@StephenCourt

Hi @StephenCourt ,

Thank you for your feedback, I do not have much to add here.

Indeed it’s on the roadmap but we do not have an ETA for it currently. If you need it it within a specific timeline I would recommend that you reach out commercially (e.g. to existing representative you may have, or mender.io contact form). They can help slot it in for your project’s timelines.