Delta apply fail with "xdelta3: target window checksum mismatch: XD3_INVALID_INPUT"

Hello Everyone,

I’m trying to generate and apply a delta from $MENDER_ARTIFACT_FROM to $MENDER_ARTIFACT_TO`
these are the steps I took:

$ export MENDER_ARTIFACT_FROM="1/image-qemux86-64-20200420165809.mender"
$ export MENDER_ARTIFACT_TO="2/image-qemux86-64-20200420180715.mender"

$ mender-artifact read $MENDER_ARTIFACT_FROM
Mender artifact:
  Name: image-qemux86-64-mo-develop-20200420165809-0.0.2
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[qemux86-64]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   rootfs-image
    Provides: Nothing
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     image-qemux86-64.ext4
      size:     1056964608
      modified: 2020-04-20 17:28:42 +0000 UTC
      checksum: e6ae6a20ae563692782e61561786cadfb8a163ec2a5ce7d5b42169f0e9ec0aff
$ mender-artifact read $MENDER_ARTIFACT_TO
Mender artifact:
  Name: image-qemux86-64-mo-develop-20200420180715-0.0.2
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[qemux86-64]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   rootfs-image
    Provides: Nothing
    Depends: Nothing
    Metadata: Nothing
    Files:
      name:     image-qemux86-64.ext4
      size:     1056964608
      modified: 2020-04-20 18:15:24 +0000 UTC
      checksum: 7ff174f1c457e02d83a59022dc851ae191c304a780582c57a8fdb90fe032892c
$ export COMMENT="my-delta"
$ mender-binary-delta-generator -n \
    $COMMENT \
    -o delta.mender \
    $MENDER_ARTIFACT_TO \
    $MENDER_ARTIFACT_FROM
$ mender-artifact read delta.mender
Mender artifact:
  Name: my-delta
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[qemux86-64]'
  Provides group:
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:

Updates:
    0:
    Type:   mender-binary-delta
    Provides: Nothing
    Depends: Nothing
    Metadata:
        {
          "delta_algorithm": "xdelta3",
          "rootfs_file_size": 1056964608,
          "rootfs_image_checksum": "e6ae6a20ae563692782e61561786cadfb8a163ec2a5ce7d5b42169f0e9ec0aff"
        }
    Files:
      name:     image-qemux86-64.ext4.delta
      size:     2813261
      modified: 2020-04-20 19:20:26 +0000 UTC
      checksum: aed49fe3f46bc274a753f7b781f1a159cbf56f57260baeea8215c5440f59abc9

the delta rootfs_image_checksum is equal to the checksum $MENDER_ARTIFACT_FROM but when deployed via mender-client the OTA fails with ‘xdelta3: target window checksum mismatch: XD3_INVALID_INPUT’

Apr 20 19:23:36 qemux86-64 mender[452]: time="2020-04-20T19:23:36Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]" module=mender
Apr 20 19:23:36 qemux86-64 mender[452]: time="2020-04-20T19:23:36Z" level=info msg="no public key was provided for authenticating the artifact" module=installer
Apr 20 19:23:37 qemux86-64 mender[452]: time="2020-04-20T19:23:37Z" level=info msg="Update module output: field artifact_provides not found in the type-info file." module=modules
Apr 20 19:23:37 qemux86-64 mender[452]: time="2020-04-20T19:23:37Z" level=info msg="Update module output: Trying to parse transitional meta-data instead..." module=modules
Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT" module=modules
Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: normally this indicates that the source file is incorrect" module=modules
Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: xdelta3: please verify the source file with sha1sum or equivalent" module=modules
Apr 20 19:23:38 qemux86-64 mender[452]: time="2020-04-20T19:23:38Z" level=info msg="Update module output: Failed to apply the delta, err: 1" module=modules

some idea where I’m going wrong?, because I’m definitely making some mistakes :wink:

Thank you!

Does the checksum from $MENDER_ARTIFACT_FROM match what is on you device? Because the delta is applied using the rootfs partition on you device, while the Mender Artifact is used to generate the patch.

E.g run the following if the currently active part is /dev/sda2:

sha256 /dev/sda2

Compare above to the content of $MENDER_ARTIFACT_FROM

Hi Mirzak,

you’re correct, the checksum returned by mender-artifact read don’t match with the sha256 of the /dev/hda2 partition (I’m on qemu) I’m investigation why at boot something on the filesystem change (I have “read-only-rootfs” feature enable), and I confirm that /dev/hda3 have the correct checksum. I will keep you posted.

Thank you,
Mauro

I’m investigation why at boot something on the filesystem change

Which version of Yocto are you using?

There could be one thing that is causing problems, and that is that the checksum in the uefiimg does not match what ends up in the Mender Artifact. This I believe only exists on older branches, that is warrior is expected to work without any further actions.

But if you are on something older then warrior, you need to deploy an full image updates once after the uefiimg has been provisioned on the device (or booted in QEMU in you case). This is to align the checksum with the Mender Artifact content.

You can read more about this issue at the end of Robust delta update rootfs

Another possibility is that the tool used to image the system is causing the change. I know that using bmap will change the checksum since it does not write the empty space whereas using dd maintains the checksum properly.

Drew

Hi and thank you,

I’m using the branch zeus-next I’ve already verified that the partition n.5 and n.6 from the table below of the uefiimg, and they have the same SHA256 hash.

The hash of both partition is the same of the .mender artifact.

GUID Partition Table (EFI)
Offset Sector: 0
Units are in 512-byte sectors

      Slot      Start        End          Length       Description
000:  Meta      0000000000   0000000000   0000000001   Safety Table
001:  -------   0000000000   0000016383   0000016384   Unallocated
002:  Meta      0000000001   0000000001   0000000001   GPT Header
003:  Meta      0000000002   0000000033   0000000032   Partition Table
004:  000       0000016384   0000049151   0000032768   boot
005:  001       0000049152   0002113535   0002064384   primary
006:  002       0002113536   0004177919   0002064384   primary
007:  003       0004177920   0008388574   0004210655   primary
008:  -------   0008388575   0008388607   0000000033   Unallocated
➜  qemux86-64 git:(develop) dd if=image-qemux86-64.uefiimg  of=1 skip=49152 count=2064384
➜  qemux86-64 git:(develop) dd if=image-qemux86-64.uefiimg  of=2 skip=2113536 count=2064384
➜  qemux86-64 git:(develop) sha256sum 1 2
fcbcf3c9465b11e103067f954f502b981d4fd15566240d00d18277fead8f404f  1
fcbcf3c9465b11e103067f954f502b981d4fd15566240d00d18277fead8f404f  2

so, it’s not an issue with the building process of the image, but something is written at runtime before mounting in ro, i suspect someting systemd related.

Best,
Mauro

Thank you Drew, but for the moment I’m spinning up a test ‘vanilla’ image with QEMU.

Best,
Mauro

Hm, could it be that systemd is running a fsck and “repairing”, hence modifying the file system image.

You can disable the fsck with adding fsck.mode=skip to kernel arguments.

Hi Mirzak,

systemd writes files (ex. /etc/machine-id) and I think others… enabling the “read-only-rootfs” is not enought, I continue the investigation.

Best,
Mauro

Specifically for the machine-id file, if systemd detects that the rootfs is read-only then it just creates a temporary machine id which, if I remember correctly, is bind mounted over top so that the base rootfs is unmodified.

There certainly may be other things inadvertently writing though. Please let us know what you find.

Drew

Gave it a go my self, and indeed I can reproduce your problems. But to me it looks like mount changes something, e.g

root@qemux86-64:~# sha256sum /dev/hda2 /dev/hda3 
54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84  /dev/hda2

# Here /dev/hda3 has the expected value (same as in Mender Artifact)
852c1a0e9e36488d4c7746b5857d309037ad8f43185ee2ab81c7bec07dd69c11  /dev/hda3
root@qemux86-64:~# mount -o ro /dev/hda3 /mnt/
[  130.417154] EXT4-fs (hda3): mounted filesystem with ordered data mode. Opts: (null)
root@qemux86-64:~# sha256sum /dev/hda2 /dev/hda3 
54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84  /dev/hda2

# Here /dev/hda3 changed just after running the mount command
54eea8451754947be8f41d2ed3470dc63c8a599ed884b5a3938a96ea0f441b84  /dev/hda3

So I would suspect an ext4 feature is causing this.

@malveo, it looks like the following solves it,

EXTRA_IMAGECMD_ext4 = "-i 4096 -O ^64bit"

A deeper investigation is pending to understand this better, but it seems that on zeus branch 64bit feature is enabled by default and it seems that it re-writes meta data.

Edit: For additional context read https://tracker.mender.io/browse/MEN-3513 (in the end this fix was applied to zeus branch)

1 Like

Hi Mirzak,

ext4 mounted in RW change ~ 1k, I’ve tested it loop mounting ext4 fs

➜  qemux86-64 git:(develop) ✗ dd if=image-qemux86-64.uefiimg  of=1 skip=49152 count=2064384
2064384+0 records in
2064384+0 records out
1056964608 bytes (1.1 GB, 1008 MiB) copied, 4.43723 s, 238 MB/s

➜  qemux86-64 git:(develop) ✗ dd if=image-qemux86-64.uefiimg  of=2 skip=49152 count=2064384
2064384+0 records in
2064384+0 records out
1056964608 bytes (1.1 GB, 1008 MiB) copied, 4.42777 s, 239 MB/s

➜  qemux86-64 git:(develop) ✗ sha256sum 1 2
8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce  1
8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce  2

➜  qemux86-64 git:(develop) ✗ sudo mount -o,loop 2 /mnt/2
➜  qemux86-64 git:(develop) ✗ umount /mnt/2

➜  qemux86-64 git:(develop) ✗ sha256sum 1 2
8dffecc1d550f85ff202f5a051b732cf97e6944e52ef3c68ea66bd7c4359bdce  1
857f24207987f5c4a3b55e5c5c3b92650b3bf5f185aeb4955e6a77b8ce8cd3f5  2

➜  qemux86-64 git:(develop) ✗ cmp 1 2
1 2 differ: byte 1069, line 1

Best,
Mauro

Hi Mirzak,

I confirm that by adding

EXTRA_IMAGECMD_ext4 = "-i 4096 -O ^64bit"

the /dev/hda2 and /dev/hda3 are equal.

root@qemux86-64:~# cmp /dev/hda2 /dev/hda3

and the fix works as expected.

Thanks,
Mauro

For additional context read https://tracker.mender.io/browse/MEN-3513 (in the end this fix was applied to zeus branch)