Power failure testing - issue with linux sync command

I’m running mender daemon as non root user on the raspberry pi and using only directory and single-file updates. No rootfs. I’m using mender client 3.0 and opensource server 3.0.

I am doing power failure tests at critical moments, so I have commit_enter hooked up to a command that triggers a relay that kills the power to the Pi itself. (a little excessive I know). I have ext4 commit=500 in /etc/fstab, so i’m relying heavily on mender syncing the file system between states. The problem is that, upon restart, all the files in /var/lib/mender/modules/v3/payloads/0000/tree/files/ are empty. They exist, but are 0kb. This indicates to me that an fssync was performed when the files were created, but not when the files were written with the artifact details.

The backup artifact is correct and synced properly, it just the payload data required by rollback is not.

This means the rollback cannot happen upon daemon reload when the device is restarted. Performing a “sync -f /var/lib/mender” at the start of the commit enter state script fixes the issue, but I would have assumed this should be done during state transition by the mender client ?

I’m sure this problem is also manually reproduceable by placing a sleep in commit_enter and physically removing the power. Are by assumptions above about sync correct or should I be doing it my self for each state ?

Extra Note: The file I am updating is with the single-file artifact is /var/lib/mender/artifact.key . I am using artifacts to perform key rotation.

1 Like

Nice investigation, @grandfield! You are right, there is a sync call after populating the streams tree, but it appears to be missing after populating the file tree. I have put up a candidate fix here, does this improve things for you?

built and tested that change. works for me. thanks for that.

1 Like