I managed to resolve the issue before posting here, but I’m sharing my experience in case it helps someone else.
- Our system is running on Ubuntu.
- We utilize delta updates.
- We use a package called
overlayroot
to achieve a read-only root file system, which is necessary for delta updates.
However, we encountered difficulties with delta updates and received the xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
error message. Surprisingly, an unidentified process had altered the data on the root file system’s block device, even though the file system was mounted as read-only.
We eventually realized that we had overlooked the util-linux
service fstrim.service
.
fstrim is used on a mounted filesystem to discard (or “trim”) blocks that are not in use by the filesystem.
This means that those blocks would be zeroed out, resulting in the checksum mismatch.
Adding to the confusion, this service runs only on a weekly basis (see fstrim.timer
). Moreover, fstrim
failed to update the metadata in the ext4 superblock, making it more challenging to debug.
To resolve the issue, we configured fstrim.service
to operate on all file systems except the root file system.
It might be helpful if Mender’s documentation mentioned and recommended disabling services like fstrim.service
when using delta updates.