Mender delta update questions

General Discussions
1

Hello

I’m trying to get mender delta updates working on an intel keem bay embedded device with yocto dunfell

I was able to make a delta update (A → B), but if I try installing it on an active partition with firmware A, I get this error.

INFO[0000] Update module output: parse_metadata_file: decoder_arguments_count:0 
INFO[0000] Update module output: running with decoder args: xdelta3 -9 -f  
INFO[0000] Update module output: -d -s /dev/mmcblk1p8 streams/core-image-minimal-keembay.ext4.delta /dev/mmcblk1p5  
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0000] Update module terminated abnormally: exit status 1

But if I install firmware A on the inactive partition first, then apply the delta update, the delta update works.

INFO[0000] Update module output: parse_metadata_file: decoder_arguments_count:0 
INFO[0000] Update module output: running with decoder args: xdelta3 -9 -f  
INFO[0000] Update module output: -d -s /dev/mmcblk1p5 streams/core-image-minimal-keembay.ext4.delta /dev/mmcblk1p8  <--- note that the args reversed
Use -commit to update, or -rollback to roll back the update.
At least one payload requested a reboot of the device it updated.

But strangely, the update seems to get applied to the currently active partition, not the inactive one.

I am still investigating and gathering information on the above, but I would like clarification on the expected behavior:

  1. If I have firmware A installed on (active) partition mmcblk1p8, and a different firmware on (inactive) partition mmcblk1p5, then apply a delta update with mender -install delta.mender, should the install be successful? I would expect the firmware on partition mmcblk1p5 should not matter, but it seems to need to match the base firmware for the install to be successful.
  2. If I successfully install the A->B update, is the update expected to be applied to partition mmcblk1p8 (active) or mmcblk1p5 (inactive)? I would expect it would install on mmcblk1p8 but I’m seeing it on mmcblk1p5.

Thank you

2

I think there were metadata and/or journal changes in the filesystem that were causing a checksum mismatch.

I modified /etc/fstab to ro,noload instead of defaults. This is an overlayfs change so does not affect checksum.

root@keembay:~# cat /etc/fstab 
# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    auto       ro,noload             1  1
proc                 /proc                proc       defaults              0  0
...

Then I modified the bootargs in the u-boot environment to make sure the kernel mounted the filesystem as readonly

root@keembay:~# fw_printenv | grep bootargs
bootargs=rootwait ro mender.data=/dev/mmcblk1p11 systemd.show_status=0 earlycon=uart8250,mmio32,0x20180000

(Note: fw_setenv didn’t work for me, I had to flash a vendor-specific binary file with a modified bootarg value)

After all this I was still getting a checksum difference, but the partition was a lot closer to the unmodified ext4

The only difference was two bytes

root@keembay:~# xxd -s $((768 * 1024 * 1024)) -l 1024 /dev/mmcblk1p8 > chunk2.hex
root@keembay:~# xxd -s $((768 * 1024 * 1024)) -l 1024 /dev/mmcblk1p5 > chunk1.hex
root@keembay:~# diff chunk2.hex chunk1.hex 
--- chunk2.hex
+++ chunk1.hex
@@ -1,9 +1,9 @@
 30000000: c03b 3998 0000 0004 0000 0000 0000 1000  .;9.............
 30000010: 0000 2000 0000 0001 0000 0001 0000 0000  .. .............
-30000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
+30000020: 0000 0000 0000 0000 0000 0012 0000 0000  ................
 30000030: 65f1 e8ab c92b 4b69 9dfb 1b8c 3501 ac27  e....+Ki....5..'
 30000040: 0000 0001 0000 0000 0000 0000 0000 0000  ................
-30000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
+30000050: 0400 0000 0000 0000 0000 0000 0000 0000  ................
 30000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 30000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
 30000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................

And it looks like if I switch partitions, those bytes disappear and I get a checksum match. So I’m guessing mender or u-boot must be adding some flag at boot that it switches on/off depending on if this is the active partition.

If I manually set these bytes back to zero on the active partition, the checksum matches. Then if I attempt a mender -install with the delta update, it works as expected: The delta update installs sucessfully and appears on the inactive partition.

So I think I’m very close to something, but I’m not sure why those two bytes are still being set

3

To update this, I was able to get delta updates working after disabling several ext4 features in yocto.

local.conf

EXTRA_IMAGECMD_ext4_append = " -O ^has_journal,^resize_inode,^metadata_csum"

After this, in addition to the above changes, the firmware flashed on the device was identical to the image in the .mender file, and did not change after reboots