# Delta apply fail with "xdelta3: target window checksum mismatch: XD3\_INVALID\_INPUT" **URL:** \ **Category:** General Discussions\ **Created:** [April 20, 2020, 7:42pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805 "2020-04-20T19:42:32Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 20, 2020, 7:42pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/1 "2020-04-20T19:42:32Z")
Hello Everyone, I’m trying to generate and apply a delta from $MENDER\_ARTIFACT\_FROM to $MENDER\_ARTIFACT\_TO` these are the steps I took: ```auto $ export MENDER_ARTIFACT_FROM="1/image-qemux86-64-20200420165809.mender" $ export MENDER_ARTIFACT_TO="2/image-qemux86-64-20200420180715.mender" $ mender-artifact read $MENDER_ARTIFACT_FROM Mender artifact: Name: image-qemux86-64-mo-develop-20200420165809-0.0.2 Format: mender Version: 3 Signature: no signature Compatible devices: '[qemux86-64]' Provides group: Depends on one of artifact(s): [] Depends on one of group(s): [] State scripts: Updates: 0: Type: rootfs-image Provides: Nothing Depends: Nothing Metadata: Nothing Files: name: image-qemux86-64.ext4 size: 1056964608 modified: 2020-04-20 17:28:42 +0000 UTC checksum: e6ae6a20ae563692782e61561786cadfb8a163ec2a5ce7d5b42169f0e9ec0aff ``` ```auto $ mender-artifact read $MENDER_ARTIFACT_TO Mender artifact: Name: image-qemux86-64-mo-develop-20200420180715-0.0.2 Format: mender Version: 3 Signature: no signature Compatible devices: '[qemux86-64]' Provides group: Depends on one of artifact(s): [] Depends on one of group(s): [] State scripts: Updates: 0: Type: rootfs-image Provides: Nothing Depends: Nothing Metadata: Nothing Files: name: image-qemux86-64.ext4 size: 1056964608 modified: 2020-04-20 18:15:24 +0000 UTC checksum: 7ff174f1c457e02d83a59022dc851ae191c304a780582c57a8fdb90fe032892c ``` ```auto $ export COMMENT="my-delta" $ mender-binary-delta-generator -n \ $COMMENT \ -o delta.mender \ $MENDER_ARTIFACT_TO \ $MENDER_ARTIFACT_FROM ``` ```auto $ mender-artifact read delta.mender Mender artifact: Name: my-delta Format: mender Version: 3 Signature: no signature Compatible devices: '[qemux86-64]' Provides group: Depends on one of artifact(s): [] Depends on one of group(s): [] State scripts: Updates: 0: Type: mender-binary-delta Provides: Nothing Depends: Nothing Metadata: { "delta_algorithm": "xdelta3", "rootfs_file_size": 1056964608, "rootfs_image_checksum": "e6ae6a20ae563692782e61561786cadfb8a163ec2a5ce7d5b42169f0e9ec0aff" } Files: name: image-qemux86-64.ext4.delta size: 2813261 modified: 2020-04-20 19:20:26 +0000 UTC checksum: aed49fe3f46bc274a753f7b781f1a159cbf56f57260baeea8215c5440f59abc9 ``` the `delta rootfs_image_checksum` is equal to the checksum `$MENDER_ARTIFACT_FROM` but when deployed via mender-client the OTA fails with ‘xdelta3: target window checksum mismatch: XD3\_INVALID\_INPUT’ ```auto Apr 20 19:23:36 qemux86-64 mender[452]: time="2020-04-20T19:23:36Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]" module=mender Apr 20 19:23:36 qemux86-64 mender[452]: time="2020-04-20T19:23:36Z" level=info msg="no public key was provided for authenticating the artifact" module=installer Apr 20 19:23:37 qemux86-64 mender[452]: time="2020-04-20T19:23:37Z" level=info msg="Update module output: field artifact_provides not found in the type-info file." module=modules Apr 20 19:23:37 qemux86-64 mender[452]: time="2020-04-20T19:23:37Z" level=info msg="Update module output: Trying to parse transitional meta-data instead..." module=modules Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT" module=modules Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: normally this indicates that the source file is incorrect" module=modules Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: please verify the source file with sha1sum or equivalent" module=modules Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: Failed to apply the delta, err: 1" module=modules ``` some idea where I’m going wrong?, because I’m definitely making some mistakes 😉 Thank you! ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 21, 2020, 6:35am UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/2 "2020-04-21T06:35:48Z")
> [@malveo](#): > > the `delta rootfs_image_checksum` is equal to the checksum `$MENDER_ARTIFACT_FROM` but Does the checksum from `$MENDER_ARTIFACT_FROM` match what is on you device? Because the delta is applied using the rootfs partition on you device, while the Mender Artifact is used to generate the patch. E.g run the following if the currently active part is `/dev/sda2`: ```auto sha256 /dev/sda2 ``` Compare above to the content of `$MENDER_ARTIFACT_FROM` ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 22, 2020, 2:50pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/3 "2020-04-22T14:50:28Z")
Hi Mirzak, you’re correct, the checksum returned by `mender-artifact read` don’t match with the sha256 of the `/dev/hda2` partition (I’m on qemu) I’m investigation why at boot something on the filesystem change (I have “read-only-rootfs” feature enable), and I confirm that `/dev/hda3` have the correct checksum. I will keep you posted. Thank you, Mauro ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 22, 2020, 3:25pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/4 "2020-04-22T15:25:43Z")
> I’m investigation why at boot something on the filesystem change Which version of Yocto are you using? There could be one thing that is causing problems, and that is that the checksum in the `uefiimg` does not match what ends up in the Mender Artifact. This I believe only exists on older branches, that is warrior is expected to work without any further actions. But if you are on something older then warrior, you need to deploy an full image updates **once** after the `uefiimg` has been provisioned on the device (or booted in QEMU in you case). This is to align the checksum with the Mender Artifact content. You can read more about this issue at the end of [Robust delta update rootfs](https://hub.mender.io/t/robust-delta-update-rootfs/1144) ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [April 22, 2020, 3:32pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/5 "2020-04-22T15:32:08Z")
Another possibility is that the tool used to image the system is causing the change. I know that using bmap will change the checksum since it does not write the empty space whereas using dd maintains the checksum properly. Drew ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 22, 2020, 3:51pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/6 "2020-04-22T15:51:09Z")
Hi and thank you, I’m using the branch `zeus-next` I’ve already verified that the partition n.5 and n.6 from the table below of the `uefiimg`, and they have the same SHA256 hash. **The hash of both partition is the same of the `.mender` artifact.** ```auto GUID Partition Table (EFI) Offset Sector: 0 Units are in 512-byte sectors Slot Start End Length Description 000: Meta 0000000000 0000000000 0000000001 Safety Table 001: ------- 0000000000 0000016383 0000016384 Unallocated 002: Meta 0000000001 0000000001 0000000001 GPT Header 003: Meta 0000000002 0000000033 0000000032 Partition Table 004: 000 0000016384 0000049151 0000032768 boot 005: 001 0000049152 0002113535 0002064384 primary 006: 002 0002113536 0004177919 0002064384 primary 007: 003 0004177920 0008388574 0004210655 primary 008: ------- 0008388575 0008388607 0000000033 Unallocated ``` ```auto ➜ qemux86-64 git:(develop) dd if=image-qemux86-64.uefiimg of=1 skip=49152 count=2064384 ➜ qemux86-64 git:(develop) dd if=image-qemux86-64.uefiimg of=2 skip=2113536 count=2064384 ``` ```auto ➜ qemux86-64 git:(develop) sha256sum 1 2 fcbcf3c9465b11e103067f954f502b981d4fd15566240d00d18277fead8f404f 1 fcbcf3c9465b11e103067f954f502b981d4fd15566240d00d18277fead8f404f 2 ``` so, it’s not an issue with the building process of the image, but something is written at runtime before mounting in ro, i suspect someting systemd related. Best, Mauro ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 22, 2020, 3:52pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/7 "2020-04-22T15:52:29Z")
Thank you Drew, but for the moment I’m spinning up a test ‘vanilla’ image with QEMU. Best, Mauro ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 22, 2020, 4:12pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/8 "2020-04-22T16:12:18Z")
Hm, could it be that systemd is running a `fsck` and “repairing”, hence modifying the file system image. You can disable the `fsck` with adding `fsck.mode=skip` to kernel arguments. ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 22, 2020, 6:18pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/9 "2020-04-22T18:18:08Z")
Hi Mirzak, `systemd` writes files (ex. /etc/machine-id) and I think others… enabling the “read-only-rootfs” is not enought, I continue the investigation. Best, Mauro ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [April 22, 2020, 9:49pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/10 "2020-04-22T21:49:18Z")
Specifically for the machine-id file, if systemd detects that the rootfs is read-only then it just creates a temporary machine id which, if I remember correctly, is bind mounted over top so that the base rootfs is unmodified. There certainly may be other things inadvertently writing though. Please let us know what you find. Drew ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 24, 2020, 11:52am UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/11 "2020-04-24T11:52:19Z")
Gave it a go my self, and indeed I can reproduce your problems. But to me it looks like `mount` changes something, e.g ```auto root@qemux86-64:~# sha256sum /dev/hda2 /dev/hda3 54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84 /dev/hda2 # Here /dev/hda3 has the expected value (same as in Mender Artifact) 852c1a0e9e36488d4c7746b5857d309037ad8f43185ee2ab81c7bec07dd69c11 /dev/hda3 root@qemux86-64:~# mount -o ro /dev/hda3 /mnt/ [130.417154] EXT4-fs (hda3): mounted filesystem with ordered data mode. Opts: (null) root@qemux86-64:~# sha256sum /dev/hda2 /dev/hda3 54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84 /dev/hda2 # Here /dev/hda3 changed just after running the mount command 54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84 /dev/hda3 ``` So I would suspect an `ext4` feature is causing this. ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [April 24, 2020, 1:05pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/12 "2020-04-24T13:05:54Z")
@malveo, it looks like the following solves it, ```auto EXTRA_IMAGECMD_ext4 = "-i 4096 -O ^64bit" ``` A deeper investigation is pending to understand this better, but it seems that on `zeus` branch `64bit` feature is enabled by default and it seems that it re-writes meta data. Edit: For additional context read [https://tracker.mender.io/browse/MEN-3513](https://tracker.mender.io/browse/MEN-3513) (in the end this fix was applied to zeus branch) ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 24, 2020, 2:50pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/13 "2020-04-24T14:50:54Z")
Hi Mirzak, ext4 mounted in RW change ~ 1k, I’ve tested it loop mounting ext4 fs ```auto ➜ qemux86-64 git:(develop) ✗ dd if=image-qemux86-64.uefiimg of=1 skip=49152 count=2064384 2064384+0 records in 2064384+0 records out 1056964608 bytes (1.1 GB, 1008 MiB) copied, 4.43723 s, 238 MB/s ➜ qemux86-64 git:(develop) ✗ dd if=image-qemux86-64.uefiimg of=2 skip=49152 count=2064384 2064384+0 records in 2064384+0 records out 1056964608 bytes (1.1 GB, 1008 MiB) copied, 4.42777 s, 239 MB/s ➜ qemux86-64 git:(develop) ✗ sha256sum 1 2 8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce 1 8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce 2 ➜ qemux86-64 git:(develop) ✗ sudo mount -o,loop 2 /mnt/2 ➜ qemux86-64 git:(develop) ✗ umount /mnt/2 ➜ qemux86-64 git:(develop) ✗ sha256sum 1 2 8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce 1 857f24207987f5c4a3b55e5c5c3b92650b3bf5f185aeb4955e6a77b8ce8cd3f5 2 ➜ qemux86-64 git:(develop) ✗ cmp 1 2 1 2 differ: byte 1069, line 1 ``` Best, Mauro ---
**Author:** ![malveo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/malveo/32/704_2.png) [@malveo](https://hub.mender.io/u/malveo)\ **Post date:** [April 24, 2020, 2:55pm UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/14 "2020-04-24T14:55:20Z")
Hi Mirzak, I confirm that by adding ```auto EXTRA_IMAGECMD_ext4 = "-i 4096 -O ^64bit" ``` the /dev/hda2 and /dev/hda3 are equal. ```auto root@qemux86-64:~# cmp /dev/hda2 /dev/hda3 ``` and the fix works as expected. Thanks, Mauro ---
**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\ **Post date:** [June 26, 2020, 11:06am UTC](https://hub.mender.io/t/delta-apply-fail-with-xdelta3-target-window-checksum-mismatch-xd3-invalid-input/1805/15 "2020-06-26T11:06:33Z")
For additional context read [https://tracker.mender.io/browse/MEN-3513](https://tracker.mender.io/browse/MEN-3513) (in the end this fix was applied to zeus branch)