VAR-SOM-IMX8MM binary-delta update bad behavior

Environment and symptoms

I integrated yocto Dunfell on this board with uboot only (No grub). Full-Artifact update is running perfectly.

The logic was added to run delta artifact and everything looks OK but the delta artifact can’t be applied. Delta artifact is generated and correctly uploaded onto the server.

In case an accurate delta is provided to the server. the client find it and try to apply but fail very early with:
2021-05-12 07:05:13 +0000 UTC info: Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 2021-05-12 07:05:13 +0000 UTC info: Update module output: xdelta3: normally this indicates that the source file is incorrect 2021-05-12 07:05:13 +0000 UTC info: Update module output: xdelta3: please verify the source file with sha1sum or equivalent

Looks like some formatting issue…
I’m using mender-binary-delta-1.2.1 everywhere
The delta artifact is correctly updated to the server. Generated with command like below:
./mender-binary-delta-1.2.1/x86_64/mender-binary-delta-generator -o bg-image-debug-imx8mm-var-dart-V9.3fromV9.2.mender bg-image-debug-imx8mm-var-dart-V9.2.mender bg-image-debug-imx8mm-var-dart-V9.3.mender

In case I’m providing corresponding delta artifact + full artifact, I’ll always get the same issue. If I’m not providing the right delta artifact, then the fallback will happen and full artifact is correctly deployed.

Provisioning is working correctly. First mender update is working fine (This can’t be a real delta update due to some required init). The second update and successive using delta will always fail.

Configuration

mender works fine for full update and below was added to conf file to support delta:

IMAGE_INSTALL_append = " mender-binary-delta"
LICENSE_FLAGS_WHITELIST_append = " commercial_mender-binary-delta"
FILESEXTRAPATHS_prepend_pn-mender-binary-delta := "${HOME}/mender-binary-delta-1.2.1/:"

Here below is uboot printenv which is where some customization can happen.
root@imx8mm-var-dart:~# fw_printenv altbootcmd=run mender_altbootcmd loadm4bin=load mmc ${mmcdev}:${mender_boot_part} ${loadaddr} ${bootdir}/${m4_bin}; cp.b ${loadaddr} ${m4_addr} ${filesize} baudrate=115200 board_name=VAR-SOM-MX8M-MINI boot_fdt=try bootcmd=run ramsize_check; mmc dev ${mmcdev}; run mender_setup; if test ${upgrade_available} = 1; then setenv mmcpart $mender_boot_part; fi; saveenv; if mmc rescan; then if test ${use_m4} = yes && run loadm4bin; then run runm4bin; fi; if run loadbootscript; then run bootscript; else if run loadimage; then run mmcboot; run mender_try_to_recover; else run mender_try_to_recover; run netboot; fi; fi; else booti ${loadaddr} - ${fdt_addr}; fi; bootcmd_mfg=run mfgtool_args;if iminfo ${initrd_addr}; then if test ${tee} = yes; then bootm ${tee_addr} ${initrd_addr} ${fdt_addr}; else booti ${loadaddr} ${initrd_addr} ${fdt_addr}; fi; else echo “Run fastboot …”; fastboot 0; fi; bootcount=1 bootdelay=1 bootdir=/boot bootlimit=1 bootscript=echo Running bootscript from mmc …; source carrier_rev=sym-1.4a cma_size=cma=640M@1376M console=ttymxc3,115200 emmc_dev=2 ethaddr=f8:dc:7a:56:8a:4c ethprime=FEC fastboot_dev=mmc1 fdt_addr=0x43000000 fdt_file=undefined fdt_high=0xffffffffffffffff fdtcontroladdr=bdf202d0 findfdt=if test $fdt_file = undefined; then if test $board_name = VAR-SOM-MX8M-MINI; then if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-som-symphony-legacy.dtb; else setenv fdt_file imx8mm-var-som-symphony.dtb; fi; else if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-dart-dt8mcustomboard-legacy.dtb; else setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb; fi; fi; fi; image=Image.gz img_addr=0x42000000 initrd_addr=0x43800000 initrd_high=0xffffffffffffffff ip_dyn=yes kboot=booti loadaddr=0x40480000 loadbootscript=load mmc ${mmcdev}:${mmcpart} ${loadaddr} ${bootdir}/${script}; loadfdt=run findfdt; echo fdt_file=${fdt_file}; load mmc ${mmcdev}:${mmcpart} ${fdt_addr} ${bootdir}/${fdt_file} loadimage=load mmc ${mmcdev}:${mmcpart} ${img_addr} ${bootdir}/${image};unzip ${img_addr} ${loadaddr} m4_addr=0x7e0000 m4_bin=hello_world.bin mender_altbootcmd=run mender_swap_part; setenv upgrade_available 0; run mender_setupsaveenv; mender_boot_part=2 mender_boot_part_hex=2 mender_check_saveenv_canary=1 mender_saveenv_canary=1 mender_setup=if test “${mender_saveenv_canary}” != “1”; then setenv mender_saveenv_canary 1; saveenv; fi; mender_swap_part=if test ${mender_boot_part} = 1; then setenv mender_boot_part 2; setenv mender_boot_part_hex 2; else setenv mender_boot_part 1; setenv mender_boot_part_hex 1; fi; mfgtool_args=setenv bootargs console=${console},${baudrate} rdinit=/linuxrc clk_ignore_unused mmcargs=setenv bootargs console=${console} root=/dev/mmcblk${mmcblk}p${mmcpart} rootwait rw ${cma_size} mmcautodetect=yes mmcblk=1 mmcboot=echo Booting from mmc …; run mmcargs; run optargs; if test ${boot_fdt} = yes || test ${boot_fdt} = try; then if run loadfdt; then booti ${loadaddr} - ${fdt_addr}; else echo WARN: Cannot load the DT; fi; else echo wait for boot; fi; mmcdev=1 mmcpart=2 nandfit_part=yes netargs=setenv bootargs console=${console} root=/dev/nfs ${cma_size} ip=dhcp nfsroot=${serverip}:${nfsroot},v3,tcp netboot=echo Booting from net …; if test ${ip_dyn} = yes; then setenv get_cmd dhcp; else setenv get_cmd tftp; fi; ${get_cmd} ${img_addr} ${image}; unzip ${img_addr} ${loadaddr};run netargs; run optargs; if test ${boot_fdt} = yes || test ${boot_fdt} = try; then run findfdt; echo fdt_file=${fdt_file}; if ${get_cmd} ${fdt_addr} ${fdt_file}; then booti ${loadaddr} - ${fdt_addr}; else echo WARN: Cannot load the DT; fi; else booti; fi; optargs=setenv bootargs ${bootargs} ${kernelargs}; ramsize_check=if test $sdram_size -le 512; then setenv cma_size cma=320M; else setenv cma_size cma=640M@1376M; fi; runm4bin=if test ${m4_addr} = 0x7e0000; then echo Booting M4 from TCM; else echo Booting M4 from DRAM; dcache flush; fi; bootaux ${m4_addr}; script=boot.scr sd_dev=1 sdram_size=2048 serial#=101c1a09dab4d6b7 soc_type=imx8mm som_rev=som_rev12 use_m4=no upgrade_available=0

Let me know if anything looks inaccurate.

NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk2      179:0    0 14.6G  0 disk 
|-mmcblk2p1  179:1    0  7.2G  0 part 
|-mmcblk2p2  179:2    0  7.2G  0 part 
`-mmcblk2p3  179:3    0  128M  0 part 
mmcblk2boot0 179:32   0    4M  1 disk 
mmcblk2boot1 179:64   0    4M  1 disk 
mmcblk1      179:96   0 14.5G  0 disk 
|-mmcblk1p1  179:97   0  1.9G  0 part 
|-mmcblk1p2  179:98   0  1.9G  0 part /
`-mmcblk1p3  179:99   0  128M  0 part /data

@kacf, @lluiscampos, @oleorhagen can you guys help here? @jneanne and I discussed this somewhat in a private thread and it does seem that the binary deltas are having an issue here. My guess is it is something to do with the artifactprovides but I don’t have any data to back that up.

Drew

Here below more detail about artifacts generated:

`mender-artifact read bg-image-debug-imx8mm-var-dart-V10.1.mender
Reading Artifact…
… - 100 %
Mender artifact:
Name: fixSwapTest10.1
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[imx8mm-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: rootfs-image
Provides:
rootfs-image.checksum: d14824f0555fa09522f7522c2ea4d819e12112abe7cf99bd3050432c84f9514b
rootfs-image.version: fixSwapTest10.1
Depends: Nothing
Clears Provides: [“artifact_group”, “rootfs_image_checksum”, “rootfs-image.*”]
Metadata: Nothing
Files:
name: bg-image-debug-imx8mm-var-dart.ext4
size: 2063597568
modified: 2021-05-17 14:29:50 +0200 CEST
checksum: d14824f0555fa09522f7522c2ea4d819e12112abe7cf99bd3050432c84f9514b
jerome@jerome-BL:~/mender/BuffaloGrid/artifacts/delta$ mender-artifact read bg-image-debug-imx8mm-var-dart-V10.2fromV10.1.mender
Reading Artifact…
… - 100 %
Mender artifact:
Name: fixSwapTest10.2
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[imx8mm-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: mender-binary-delta
Provides:
rootfs-image.checksum: 0fe577adeb58aaa0d53b22f5ce04c88f01f3c7c32269b33cd292341eebaefb86
rootfs-image.version: fixSwapTest10.2
Depends:
rootfs-image.checksum: d14824f0555fa09522f7522c2ea4d819e12112abe7cf99bd3050432c84f9514b
Clears Provides: [“artifact_group”, “rootfs_image_checksum”, “rootfs-image.*”]
Metadata:
{
“delta_algorithm”: “xdelta3”,
“rootfs_file_size”: 2063597568
}
Files:
name: bg-image-debug-imx8mm-var-dart.ext4.delta
size: 190823
modified: 2021-05-17 15:19:01 +0200 CEST
checksum: b2df090e73945aaa763ad06dec9b55a6b6d4db390b03e3acc1c0afe5927767d5
jerome@jerome-BL:~/mender/BuffaloGrid/artifacts/delta$ mender-artifact read bg-image-debug-imx8mm-var-dart-V10.2.mender
Reading Artifact…
… - 100 %
Mender artifact:
Name: fixSwapTest10.2
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[imx8mm-var-dart]’
Provides group:
Depends on one of artifact(s): []
Depends on one of group(s): []
State scripts:

Updates:
0:
Type: rootfs-image
Provides:
rootfs-image.checksum: 0fe577adeb58aaa0d53b22f5ce04c88f01f3c7c32269b33cd292341eebaefb86
rootfs-image.version: fixSwapTest10.2
Depends: Nothing
Clears Provides: [“artifact_group”, “rootfs_image_checksum”, “rootfs-image.*”]
Metadata: Nothing
Files:
name: bg-image-debug-imx8mm-var-dart.ext4
size: 2063597568
modified: 2021-05-17 14:43:38 +0200 CEST
checksum: 0fe577adeb58aaa0d53b22f5ce04c88f01f3c7c32269b33cd292341eebaefb86
jerome@jerome-BL:~/mender/BuffaloGrid/artifacts/delta$ `

This look very similar with this thread: Delta apply fail with "xdelta3: target window checksum mismatch: XD3_INVALID_INPUT"

Except that the WA does not change the behavior:
EXTRA_IMAGECMD_ext4 = "-i 4096 -O ^64bit"

Have you verified the checksum of the running partition manually (adjust block device to the running root partition if needed)?

sha256sum /dev/mmcblk1p1

Also, what is the output of this:

mount

Yes I did:
root@imx8mm-var-dart:~# sha256sum /dev/mmcblk1p2 a6c21287df47c0a09409e406d970f4334b7bd7ca1c950f2b6e5304ab1d59fb08

and on the other hand doing mender-artifact myartifact.mender:
checksum: d14824f0555fa09522f7522c2ea4d819e12112abe7cf99bd3050432c84f9514b

can’t check for mount I let the board to my collegue so He can double check. He will come back to you on this.