Delta update update module checksum error

Hi!

I’m experimenting with delta updates for our Yocto-based flash image (Nvidia Jetson tegra-based system with OE4T). We have an A/B updating system with readonly root filesystem. The base implementation of Mender full root filesystem updates in standalone mode is fully integrated and tested.

As a next step we’d like to add delta updating. I followed the documentation and I’ve added the necessary tooling to the Yocto build which all seems to work fine. The update module is there and gets fired up when a delta update is started. However, something goes wrong.

I created a full root filesystem update A.mender, slightly tweaked the Yocto build and generated B.mender from this. Full update A was installed first on the device, to have the proper base on the device side. Then using the mender-binary-delta-generator I generated a delta update A2B.mender. But this fails with the following error in /var/log/messages:

Mar 14 15:50:53 test123 user.err mender[5874]: time="2022-03-14T15:50:53Z" level=error msg="Download failed: Update module terminated abnormally: exit status 1"
Mar 14 15:50:53 test123 user.err mender[5874]: time="2022-03-14T15:50:53Z" level=error msg="Update module terminated abnormally: exit status 1"

I’ve tried to run the installation manually, and get this output:

$ sudo mender install A2B.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 228352...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0000] Update module terminated abnormally: exit status 1

I’ve verified, and really have A.mender installed as the base version, and generated A2B.mender like this:

mender-binary-delta-generator -o A2B.mender  A.mender B.mender

Can someone point me at the direction to further analyze the checksum of the delta artifact? And where the mismatch actually is?

Any help is appreciated! Thanks very much :slight_smile:

(and thanks very much for making such a great product :slight_smile:)

Hi @krisvanrens,

thanks for getting in touch, and the perfect description. I think the situation that you see is caused by trying to deploy a delta update as the first update ever on a freshly deployed device. This is currently a corner case and will be improved respectively covered in the upcoming 3.4 release.

To check, you can use mender show-provides, and if you are missing the checksum there then you have hit exactly this case.

Hope this helps!

Greetz,
Josef

Hi Josef,

Thanks for the swift response!

I’m aware of this corner-case, that’s why I installed full root filesystem update A.mender as an update first. So the output of show-provides does indeed show a checksum:

$ sudo mender show-provides
artifact_name=somethingsomething
rootfs-image.checksum=be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b
rootfs-image.version=somethingsomething

I have another question on the side now. What’s the update scheme for delta updates? Is it the same as a full rootfs update? E.g. let’s say 0 is the active firmware slot 1 is the inactive firmware slot, if run a full rootfs update from slot 0 to 1 I get a checksum verification of the update artifact against the provides of slot 0, then when the installation succeeds I will be in slot 1 with the new version?

If I read the documentation on delta updating, I’m a bit puzzled:

The above approach can save considerable time and bandwidth, but it requires read-only root filesystem support to ensure that the delta calculated offline will apply properly to the active root filesystem.

Hmmm. Does that mean that the delta update is actually applied to the active firmware slot? Not the inactive one like a full root filesystem update? If so, where do Mender state scripts in delta updates come in? Our current product update validation workflow is partly based on the state scripts.

I can actually add some additional information: I actually made two delta updates now, just for testing. There’s A2B.mender for A :arrow_forward: B, and B2A.mender for A :arrow_backward: B. If I try to install the B2A variant, I actually get the expected error with a checksum failure:

$ sudo mender install B2A.mender 

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [B2A.mender] 
Installing Artifact of size 228864...
INFO[0000] No public key was provided for authenticating the artifact.
ERRO[0000] Artifact dependency "rootfs-image.checksum" not satisfied by currently installed artifact (fc1aa6b9cc9d020d461ab394334870d9bae594c605519a70ef78d5eec1d24228 != be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b). 
ERRO[0000] Artifact dependency "rootfs-image.checksum" not satisfied by currently installed artifact (fc1aa6b9cc9d020d461ab394334870d9bae594c605519a70ef78d5eec1d24228 != be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b).

But the other error is definitely different:

$ sudo mender install A2B.mender 

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 228352...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0000] Update module terminated abnormally: exit status 1

Does this point to some kind of a tooling issue perhaps?

Not sure. As you pointed out, the one is expected. How do you generate the artifacts? Is this something I could try and reproduce?

Greetz,
Josef

I’m running the delta generator from the Yocto build output directory like so:

$ ../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender

The error seemed to me that something with xdelta3 went wrong. I checked and we have xdelta3 not in the Yocto build (is mentioned nowhere in the Mender documentation). I added it just for sure but this doesn’t help unfortunately. I checked and I have xdelta3 installed locally and on the device now and get the same results.

mender-artifact is installed system-wide and is at version 3.7.0, I verified this.

I’ve tried some more options, just to rule out stuff, but I’m not getting it to work.

I also tried disabling the xdelta3 checksum when building the delta artifact:

../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender -- -- -n

Then when installing the installation takes much longer, but still fails in another spot:

$ sudo mender install A2B.mender 

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 227840...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0100] Update module output: Fatal failure: The expected and the actual checksums do not match! 
ERRO[0100] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0100] Update module terminated abnormally: exit status 1

So the update module itself errors out. Again I verified that A.mender is properly installed (on both slots 0 and 1).

Is there a possibility to enable more debugging information in the delta updating module? And does anyone have an idea how the checksum is calculated? I’m using a readonly root filesystem with various overlay mounts to be able store non-volatile configuration changes – could this be of any influence?

No luck still :confused:

If it helps any: here’s the strace output for an update attempt using the default delta update build (without the -n for xdelta3).

$ ls -l A2B.mender

-rw-rw-r-- 1 kris kris 230K Mar 18 15:18 A2B.mender
$ sudo strace mender install A2B.mender

...
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULLINFO[0000] Mender running on partition: /dev/mmcblk0p1
INFO[0000] Start updating from local image file: [A2B.mender]
Installing Artifact of size 242176...
INFO[0000] No public key was provided for authenticating the artifact
                                                               -   1 %) = 0
sched_yield()                           = 0
futex(0xb81838, FUTEX_WAKE_PRIVATE, 1)  = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=3000}, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
.                                                              -   2 %) = 0
futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81d88, FUTEX_WAKE_PRIVATE, 1)  = 1
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)
epoll_pwait(4, [], 128, 0, NULL, 12065536) = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)
..                                                             -   4 %) = 0
.............................................................. - 100 %futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL
) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
read(12, 0x400019a000, 32768)           = -1 EAGAIN (Resource temporarily unavailable)
read(14, 0x40001a2000, 32768)           = -1 EAGAIN (Resource temporarily unavailable)
waitid(P_PID, 12046, INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent
INFO[0000] Update module output: Failed to apply the delta, err: 1
{si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12046, si_uid=0, si_status=1, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
wait4(12046, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, {ru_utime={tv_sec=0, tv_usec=40000}, ru_stime={tv_sec=0, tv_usec=52000}, ...}) = 12046
futex(0x40002004c8, FUTEX_WAKE_PRIVATE, 1) = 1
openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree/stream-next", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0
fcntl(12, F_GETFL)                      = 0x20800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE)
fcntl(12, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x40000cdc88) = 0
close(12)                               = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
openat(AT_FDCWD, "/etc//localtime", O_RDONLY) = 11
read(11, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 111
read(11, "", 4096)                      = 0
close(11)                               = 0
getpid()                                = 12025
write(3, "<11>Mar 18 14:15:19 mender[12025"..., 124) = 124
write(2, "\33[31mERRO\33[0m[0000] Download fai"..., 89ERRO[0000] Download failed: Update module terminated abnormally: exit status 1
) = 89
openat(AT_FDCWD, "/etc/mender/scripts", O_RDONLY|O_CLOEXEC) = 11
epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184b38) = -1 EPERM (Operation not permitted)
getdents64(11, /* 3 entries */, 8192)   = 80
getdents64(11, /* 0 entries */, 8192)   = 0
newfstatat(AT_FDCWD, "/etc/mender/scripts/version", {st_mode=S_IFREG|0644, st_size=1, ...}, AT_SYMLINK_NOFOLLOW) = 0
close(11)                               = 0
openat(AT_FDCWD, "/etc/mender/scripts/version", O_RDONLY|O_CLOEXEC) = 11
epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184bd8) = -1 EPERM (Operation not permitted)
read(11, "3", 512)                      = 1
read(11, "", 1535)                      = 0
newfstatat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, 0) = 0
openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted)
pipe2([13, 14], O_CLOEXEC)              = 0
epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = 0
fcntl(13, F_GETFL)                      = 0 (flags O_RDONLY)
fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0
fcntl(14, F_GETFL)                      = 0x1 (flags O_WRONLY)
fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK) = 0
pipe2([15, 16], O_CLOEXEC)              = 0
epoll_ctl(4, EPOLL_CTL_ADD, 15, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = 0
fcntl(15, F_GETFL)                      = 0 (flags O_RDONLY)
fcntl(15, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538296, u64=547619384888}}) = 0
fcntl(16, F_GETFL)                      = 0x1 (flags O_WRONLY)
fcntl(16, F_SETFL, O_WRONLY|O_NONBLOCK) = 0
fcntl(14, F_GETFL)                      = 0x801 (flags O_WRONLY|O_NONBLOCK)
fcntl(14, F_SETFL, O_WRONLY)            = 0
fcntl(16, F_GETFL)                      = 0x801 (flags O_WRONLY|O_NONBLOCK)
fcntl(16, F_SETFL, O_WRONLY)            = 0
pipe2([17, 18], O_CLOEXEC)              = 0
getpid()                                = 12025
rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 12056
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
close(18)                               = 0
read(17, "", 8)                         = 0
close(17)                               = 0
close(12)                               = 0
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184c48) = 0
close(14)                               = 0
epoll_ctl(4, EPOLL_CTL_DEL, 16, 0x4000184c48) = 0
close(16)                               = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
write(6, "\0", 1)                       = 1
waitid(P_PID, 12056, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]})                 = 0
futex(0xb81938, FUTEX_WAKE_PRIVATE, 1)  = 1
wait4(12056, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={tv_sec=0, tv_usec=4000}, ru_stime={tv_sec=0, tv_usec=0}, ...}) = 12056
unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", 0) = -1 EISDIR (Is a directory)
unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", AT_REMOVEDIR) = -1 ENOTEMPTY (Directory not empty)
openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000", O_RDONLY|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184dd8) = -1 EPERM (Operation not permitted)
unlinkat(12, "tree", 0)                 = -1 EISDIR (Is a directory)
newfstatat(12, "tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(12, "tree", O_RDONLY|O_CLOEXEC)  = 13
epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 13, 0x4000184c98) = -1 EPERM (Operation not permitted)
getdents64(13, /* 10 entries */, 8192)  = 336
getdents64(13, /* 0 entries */, 8192)   = 0
unlinkat(13, "current_device_type", 0)  = 0
unlinkat(13, "header", 0)               = -1 EISDIR (Is a directory)
newfstatat(13, "header", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "header", O_RDONLY|O_CLOEXEC) = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 8 entries */, 8192)   = 256
getdents64(14, /* 0 entries */, 8192)   = 0
unlinkat(14, "artifact_group", 0)       = 0
unlinkat(14, "artifact_name", 0)        = 0
unlinkat(14, "type-info", 0)            = 0
unlinkat(14, "header-info", 0)          = 0
unlinkat(14, "meta-data", 0)            = 0
unlinkat(14, "payload_type", 0)         = 0
close(14)                               = 0
unlinkat(13, "header", AT_REMOVEDIR)    = 0
unlinkat(13, "version", 0)              = 0
unlinkat(13, "tmp", 0)                  = -1 EISDIR (Is a directory)
newfstatat(13, "tmp", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "tmp", O_RDONLY|O_CLOEXEC)   = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 2 entries */, 8192)   = 48
getdents64(14, /* 0 entries */, 8192)   = 0
close(14)                               = 0
unlinkat(13, "tmp", AT_REMOVEDIR)       = 0
unlinkat(13, "stream-next", 0)          = 0
unlinkat(13, "streams", 0)              = -1 EISDIR (Is a directory)
newfstatat(13, "streams", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "streams", O_RDONLY|O_CLOEXEC) = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 3 entries */, 8192)   = 120
getdents64(14, /* 0 entries */, 8192)   = 0
unlinkat(14, "A.ext4.delta", 0) = 0
close(14)                               = 0
unlinkat(13, "streams", AT_REMOVEDIR)   = 0
unlinkat(13, "current_artifact_name", 0) = 0
unlinkat(13, "current_artifact_group", 0) = 0
close(13)                               = 0
unlinkat(12, "tree", AT_REMOVEDIR)      = 0
close(12)                               = 0
close(10)                               = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
ERRO[0000] Update module terminated abnormally: exit status 1
+++ exited with 1 +++

Does anyone else perhaps have an idea?

Hi!

One more reply to myself. I’ve checked with the tegrademo distro (see: GitHub - OE4T/tegra-demo-distro: Reference/demonstration distro for meta-tegra) to verify if the error was originating from our customized Yocto distro.

It turns out however, that a base image build of tegrademo shows the same problem:

root@jetson-tx2-devkit:~# mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/mmcblk0p33
INFO[0000] Start updating from local image file: [A2B.mender]
Installing Artifact of size 291328...
INFO[0000] No public key was provided for authenticating the artifact
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent
INFO[0000] Update module output: Failed to apply the delta, err: 1
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1
ERRO[0000] Update module terminated abnormally: exit status 1

I’m not sure what this means, but perhaps the integration of Mender in OE4T is broken already? Not sure really.

OK, so it turns out delta updates don’t work with the base image integration for tegra-demo-distro. See this issue as well: Mender delta update issues with demo distro · Discussion #198 · OE4T/tegra-demo-distro · GitHub

I’d really like to solve this problem, we’re currently holding back on getting a professional license for Mender (for the delta updates) because we can’t get it to work. I’d gladly contribute to the project to submit a fix, if only I had a strategy to dig deeper into it.

Is it possible for someone who knows about the Yocto/Mender integration to look at this? Thanks in advance. @kacf perhaps? (forgive me for the direct referral, I’m including you because you’ve written this post).

Nothing immediately stands out from your description. But perhaps you can tell me a bit more about the overlay setup. I don’t think it should have an effect, but it depends. What is in /etc/mender/mender-binary-delta.conf

Also, perhaps you can run strace with the -f argument? Then it will follow the fork into the Update Module, which is more interesting than what the client itself does.

Hi Kristian,

Thanks very much for your response :slightly_smiling_face:

Here’s the output again, with the strace -f log attached as a separate log file.

$ strace -o A2B.log -f mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p33 
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 14524928...
INFO[0000] No public key was provided for authenticating the artifact 
.....................................................          -  86 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe 
ERRO[0000] Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe

Again, the above command was carried out on a vanilla tegra-demo-distro build.

P.S. I renamed the .log file to a .yml file because I wasn’t able to upload it otherwise…

Regarding the contents of /etc/mender/mender-binary-delta.conf:

$ cat /etc/mender/mender-binary-delta.conf 
{
  "RootfsPartA": "/dev/mmcblk0p1",
  "RootfsPartB": "/dev/mmcblk0p33"
}

Regarding the overlay setup; basically it’s /home and var/lib that’s overlayed:

$ cat /etc/fstab 
/dev/root            /                    auto       ro              1  0
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0
overlay              /var/lib             overlay    lowerdir=/var/lib,upperdir=/data/overlays/var-lib-upper,workdir=/data/overlays/var-lib-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0  0
overlay              /home                overlay    lowerdir=/home,upperdir=/data/overlays/home-upper,workdir=/data/overlays/home-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0  0
/dev/mmcblk0p34      /data                auto       defaults,data=journal 0  2

Nothing really special w.r.t. the Mender updating I would think.

A2B.yml (803.3 KB)

By the way: what I could make up out of the strace log:

It seems as if the state scripts are run? On line 3062 I see:

30446 unlinkat(13, "ArtifactInstall_Leave_80_bl-update", 0 <unfinished ...>

This deletes the post-install state script. This, and the stuff preceding it seems as though if the state script is fully run, which indicates a successful installation…?

However, then on line 4098 I see the mender-binary-delta tool is run. It seems weird to me that the state scripts would be run before the delta updating tool is run, so probably my previous assumption is false.

The first signs of something going wrong start at ~9540, but I’m not sure how to interpret this.

I can’t find anything out of the ordinary, the strace log looks pretty normal. It reads correctly, but then decides the checksum is wrong, which is still a mystery.

What is the output of the two commands:

dumpe2fs /dev/mmcblk0p1 | head -n 60
dumpe2fs /dev/mmcblk0p33 | head -n 60

We know that there are certain filesystem features that mess up the checksum.

I think it’s not run. It’s just created in preparation for running, but never actually does it, as far as I can see.

1 Like

OK, that seems plausible yes.

Here you go:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 60
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          7b256d5a-ceec-4958-b87c-baea4df77123
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747512
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 23 17:17:27 2022
Last mount time:          Wed Mar 23 19:40:40 2022
Last write time:          Wed Mar 23 19:40:40 2022
Mount count:              1
Maximum mount count:      -1
Last checked:             Wed Mar 23 17:17:28 2022
Check interval:           0 (<none>)
Lifetime writes:          281 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      3fead4d3-25ac-4954-81ff-eb64fe838aa4
Checksum type:            crc32c
Checksum:                 0xc34b830e


Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-736
  Block bitmap at 737 (+737), csum 0x00008ea6
  Inode bitmap at 753 (+753), csum 0x0000762a
  Inode table at 769-2816 (+769)
  0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes
  Free blocks: 
  Free inodes: 3711-32768
Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED]
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-33504
  Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6
  Inode bitmap at 754 (bg #0 + 754), csum 0x00000000
  Inode table at 2817-4864 (bg #0 + 2817)

And:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p33 | head -n 60
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          7b256d5a-ceec-4958-b87c-baea4df77123
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747512
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 23 17:17:27 2022
Last mount time:          Tue Mar 29 14:18:19 2022
Last write time:          Tue Mar 29 14:18:19 2022
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Mar 23 17:17:28 2022
Check interval:           0 (<none>)
Lifetime writes:          282 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      3fead4d3-25ac-4954-81ff-eb64fe838aa4
Checksum type:            crc32c
Checksum:                 0x658d3914


Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-736
  Block bitmap at 737 (+737), csum 0x00008ea6
  Inode bitmap at 753 (+753), csum 0x0000762a
  Inode table at 769-2816 (+769)
  0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes
  Free blocks: 
  Free inodes: 3711-32768
Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED]
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-33504
  Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6
  Inode bitmap at 754 (bg #0 + 754), csum 0x00000000
  Inode table at 2817-4864 (bg #0 + 2817)

Again, thanks for taking your time to help me! :slight_smile:

Thanks, we might be on to something: The metadata_csum is known to generate unstable checksums, even on read-only filesystems. Try removing it when creating the filesystem, using -O ^metadata_csum.

@kacf Great! I’m going to test this.

Unfortunately it still doesn’t work:

root@jetson-tx2-devkit:~# mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 702464...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0000] Update module terminated abnormally: exit status 1

I changed the Yocto build by adding EXTRA_IMAGECMD_append = " -O ^metadata_csum" to local.conf. This seemed to work:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 42
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          64d53d86-c500-4ab6-9838-02d134974d94
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747514
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 30 14:21:46 2022
Last mount time:          Wed Mar 30 14:34:49 2022
Last write time:          Wed Mar 30 14:34:49 2022
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Mar 30 14:21:46 2022
Check interval:           0 (<none>)
Lifetime writes:          282 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      1fe4bb93-b652-4dfd-8dbc-58c97d2d9dbd

The metadata_csum option is not there in the new build.

I followed the same updating procedure as before, first installing/committing A.mender on a clean build, resulting in correct show-provides options. Then I added a line of comment to a file that’s installed, rebuilt the rootfs resulting in B.mender. Generated delta A2B.mender from the two artifacts and tried to install that.

I added a new strace -f log (renamed to YML to make this possible).

A2B.yml (544.3 KB)

Hmm, that’s disappointing. For the record I compared the rest of the filesystem features with the ones used in our test farm, and they are exactly identical to what we use.

Could you call this:

mender-artifact read A2B.mender

Make a note of the rootfs_file_size and the Depends: rootfs-image.checksum values. Then execute:

dd if=/dev/mmcblk0p1 bs=1M count=<rootfs_file_size> iflag=count_bytes | sha256sum -

Replace <rootfs_file_size> with the value you found earlier, and make sure /dev/mmcblk0p1 is the currently running partition.

The question now is: Does the output from sha256sum match what is in the Depends value?