# Delta update update module checksum error **URL:** \ **Category:** General Discussions\ **Created:** [March 15, 2022, 9:00am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750 "2022-03-15T09:00:46Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 15, 2022, 9:00am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/1 "2022-03-15T09:00:46Z")
Hi! I’m experimenting with delta updates for our Yocto-based flash image (Nvidia Jetson tegra-based system with OE4T). We have an A/B updating system with readonly root filesystem. The base implementation of Mender full root filesystem updates in standalone mode is fully integrated and tested. As a next step we’d like to add delta updating. I followed the documentation and I’ve added the necessary tooling to the Yocto build which all seems to work fine. The update module is there and gets fired up when a delta update is started. However, _something_ goes wrong. I created a full root filesystem update `A.mender`, slightly tweaked the Yocto build and generated `B.mender` from this. Full update A was installed first on the device, to have the proper base on the device side. Then using the `mender-binary-delta-generator` I generated a delta update `A2B.mender`. But this fails with the following error in `/var/log/messages`: ```nohighlight Mar 14 15:50:53 test123 user.err mender[5874]: time="2022-03-14T15:50:53Z" level=error msg="Download failed: Update module terminated abnormally: exit status 1" Mar 14 15:50:53 test123 user.err mender[5874]: time="2022-03-14T15:50:53Z" level=error msg="Update module terminated abnormally: exit status 1" ``` I’ve tried to run the installation manually, and get this output: ```nohighlight $ sudo mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 228352... INFO[0000] No public key was provided for authenticating the artifact .............................................................. - 100 % INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 ERRO[0000] Update module terminated abnormally: exit status 1 ``` I’ve verified, and really have `A.mender` installed as the base version, and generated `A2B.mender` like this: ```nohighlight mender-binary-delta-generator -o A2B.mender A.mender B.mender ``` Can someone point me at the direction to further analyze the checksum of the delta artifact? And where the mismatch actually is? Any help is appreciated! Thanks very much 🙂 (and thanks very much for making such a great product 🙂) ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [March 15, 2022, 9:28am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/2 "2022-03-15T09:28:34Z")
Hi @krisvanrens, thanks for getting in touch, and the perfect description. I think the situation that you see is caused by trying to deploy a delta update as the first update ever on a freshly deployed device. This is currently a corner case and will be improved respectively covered in the upcoming 3.4 release. To check, you can use `mender show-provides`, and if you are missing the checksum there then you have hit exactly this case. Hope this helps! Greetz, Josef ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 15, 2022, 9:45am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/3 "2022-03-15T09:45:25Z")
Hi Josef, Thanks for the swift response! I’m aware of this corner-case, that’s why I installed full root filesystem update `A.mender` as an update first. So the output of `show-provides` does indeed show a checksum: ```plaintext $ sudo mender show-provides artifact_name=somethingsomething rootfs-image.checksum=be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b rootfs-image.version=somethingsomething ``` I have another question on the side now. What’s the update scheme for delta updates? Is it the same as a full rootfs update? E.g. let’s say `0` is the active firmware slot `1` is the inactive firmware slot, if run a full rootfs update from slot `0` to `1` I get a checksum verification of the update artifact against the _provides_ of slot `0`, then when the installation succeeds I will be in slot `1` with the new version? If I read the documentation on delta updating, I’m a bit puzzled: > The above approach can save considerable time and bandwidth, but it requires read-only root filesystem support to ensure that the delta calculated offline will apply properly **to the active root filesystem**. Hmmm. Does that mean that the delta update is actually applied to the active firmware slot? Not the inactive one like a full root filesystem update? If so, where do Mender state scripts in delta updates come in? Our current product update validation workflow is partly based on the state scripts. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 15, 2022, 11:11am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/4 "2022-03-15T11:11:04Z")
I can actually add some additional information: I actually made two delta updates now, just for testing. There’s `A2B.mender` for A ▶ B, and `B2A.mender` for A ◀ B. If I try to install the B2A variant, I actually get the expected error with a checksum failure: ```nohighlight $ sudo mender install B2A.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [B2A.mender] Installing Artifact of size 228864... INFO[0000] No public key was provided for authenticating the artifact. ERRO[0000] Artifact dependency "rootfs-image.checksum" not satisfied by currently installed artifact (fc1aa6b9cc9d020d461ab394334870d9bae594c605519a70ef78d5eec1d24228 != be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b). ERRO[0000] Artifact dependency "rootfs-image.checksum" not satisfied by currently installed artifact (fc1aa6b9cc9d020d461ab394334870d9bae594c605519a70ef78d5eec1d24228 != be261ac5e8a47f29ce96205ca328cec36fe8cb634f0ec5504f707d4e98d4341b). ``` But the other error is definitely different: ```nohighlight $ sudo mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 228352... INFO[0000] No public key was provided for authenticating the artifact .............................................................. - 100 % INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 ERRO[0000] Update module terminated abnormally: exit status 1 ``` Does this point to some kind of a tooling issue perhaps? ---
**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\ **Post date:** [March 15, 2022, 1:40pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/5 "2022-03-15T13:40:23Z")
Not sure. As you pointed out, the one is expected. How do you generate the artifacts? Is this something I could try and reproduce? Greetz, Josef ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 15, 2022, 3:43pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/6 "2022-03-15T15:43:22Z")
I’m running the delta generator from the Yocto build output directory like so: ```nohighlight $ ../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender ``` The error seemed to me that something with xdelta3 went wrong. I checked and we have xdelta3 not in the Yocto build (is mentioned nowhere in the Mender documentation). I added it just for sure but this doesn’t help unfortunately. I checked and I have xdelta3 installed locally and on the device now and get the same results. `mender-artifact` is installed system-wide and is at version 3.7.0, I verified this. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 16, 2022, 10:16am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/7 "2022-03-16T10:16:54Z")
I’ve tried some more options, just to rule out stuff, but I’m not getting it to work. I also tried disabling the `xdelta3` checksum when building the delta artifact: ```auto ../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender -- -- -n ``` Then when installing the installation takes much longer, but still fails in another spot: ```auto $ sudo mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 227840... INFO[0000] No public key was provided for authenticating the artifact .............................................................. - 100 % INFO[0100] Update module output: Fatal failure: The expected and the actual checksums do not match! ERRO[0100] Download failed: Update module terminated abnormally: exit status 1 ERRO[0100] Update module terminated abnormally: exit status 1 ``` So the update module itself errors out. Again I verified that `A.mender` is properly installed (on both slots 0 _and_ 1). Is there a possibility to enable more debugging information in the delta updating module? And does anyone have an idea how the checksum is calculated? I’m using a readonly root filesystem with various overlay mounts to be able store non-volatile configuration changes – could this be of any influence? ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 18, 2022, 3:08pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/8 "2022-03-18T15:08:48Z")
No luck still 😕 If it helps any: here’s the `strace` output for an update attempt using the default delta update build (without the `-n` for `xdelta3`). ```auto $ ls -l A2B.mender -rw-rw-r-- 1 kris kris 230K Mar 18 15:18 A2B.mender ``` ```auto $ sudo strace mender install A2B.mender ... futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULLINFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 242176... INFO[0000] No public key was provided for authenticating the artifact - 1 %) = 0 sched_yield() = 0 futex(0xb81838, FUTEX_WAKE_PRIVATE, 1) = 0 futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 nanosleep({tv_sec=0, tv_nsec=3000}, NULL) = 0 futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1 . - 2 %) = 0 futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xb81d88, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable) epoll_pwait(4, [], 128, 0, NULL, 12065536) = 0 futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out) .. - 4 %) = 0 .............................................................. - 100 %futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL ) = 0 futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1 read(12, 0x400019a000, 32768) = -1 EAGAIN (Resource temporarily unavailable) read(14, 0x40001a2000, 32768) = -1 EAGAIN (Resource temporarily unavailable) waitid(P_PID, 12046, INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12046, si_uid=0, si_status=1, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0 wait4(12046, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, {ru_utime={tv_sec=0, tv_usec=40000}, ru_stime={tv_sec=0, tv_usec=52000}, ...}) = 12046 futex(0x40002004c8, FUTEX_WAKE_PRIVATE, 1) = 1 openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree/stream-next", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 12 epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0 fcntl(12, F_GETFL) = 0x20800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE) fcntl(12, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x40000cdc88) = 0 close(12) = 0 futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1 openat(AT_FDCWD, "/etc//localtime", O_RDONLY) = 11 read(11, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 111 read(11, "", 4096) = 0 close(11) = 0 getpid() = 12025 write(3, "<11>Mar 18 14:15:19 mender[12025"..., 124) = 124 write(2, "\33[31mERRO\33[0m[0000] Download fai"..., 89ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 ) = 89 openat(AT_FDCWD, "/etc/mender/scripts", O_RDONLY|O_CLOEXEC) = 11 epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184b38) = -1 EPERM (Operation not permitted) getdents64(11, /* 3 entries */, 8192) = 80 getdents64(11, /* 0 entries */, 8192) = 0 newfstatat(AT_FDCWD, "/etc/mender/scripts/version", {st_mode=S_IFREG|0644, st_size=1, ...}, AT_SYMLINK_NOFOLLOW) = 0 close(11) = 0 openat(AT_FDCWD, "/etc/mender/scripts/version", O_RDONLY|O_CLOEXEC) = 11 epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184bd8) = -1 EPERM (Operation not permitted) read(11, "3", 512) = 1 read(11, "", 1535) = 0 newfstatat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, 0) = 0 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 12 epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted) pipe2([13, 14], O_CLOEXEC) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = 0 fcntl(13, F_GETFL) = 0 (flags O_RDONLY) fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0 fcntl(14, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 pipe2([15, 16], O_CLOEXEC) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 15, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = 0 fcntl(15, F_GETFL) = 0 (flags O_RDONLY) fcntl(15, F_SETFL, O_RDONLY|O_NONBLOCK) = 0 epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538296, u64=547619384888}}) = 0 fcntl(16, F_GETFL) = 0x1 (flags O_WRONLY) fcntl(16, F_SETFL, O_WRONLY|O_NONBLOCK) = 0 fcntl(14, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK) fcntl(14, F_SETFL, O_WRONLY) = 0 fcntl(16, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK) fcntl(16, F_SETFL, O_WRONLY) = 0 pipe2([17, 18], O_CLOEXEC) = 0 getpid() = 12025 rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0 rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0 clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 12056 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 close(18) = 0 read(17, "", 8) = 0 close(17) = 0 close(12) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184c48) = 0 close(14) = 0 epoll_ctl(4, EPOLL_CTL_DEL, 16, 0x4000184c48) = 0 close(16) = 0 futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1 write(6, "\0", 1) = 1 waitid(P_PID, 12056, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- rt_sigreturn({mask=[]}) = 0 futex(0xb81938, FUTEX_WAKE_PRIVATE, 1) = 1 wait4(12056, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={tv_sec=0, tv_usec=4000}, ru_stime={tv_sec=0, tv_usec=0}, ...}) = 12056 unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", 0) = -1 EISDIR (Is a directory) unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", AT_REMOVEDIR) = -1 ENOTEMPTY (Directory not empty) openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000", O_RDONLY|O_CLOEXEC) = 12 epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184dd8) = -1 EPERM (Operation not permitted) unlinkat(12, "tree", 0) = -1 EISDIR (Is a directory) newfstatat(12, "tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0 openat(12, "tree", O_RDONLY|O_CLOEXEC) = 13 epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 13, 0x4000184c98) = -1 EPERM (Operation not permitted) getdents64(13, /* 10 entries */, 8192) = 336 getdents64(13, /* 0 entries */, 8192) = 0 unlinkat(13, "current_device_type", 0) = 0 unlinkat(13, "header", 0) = -1 EISDIR (Is a directory) newfstatat(13, "header", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0 openat(13, "header", O_RDONLY|O_CLOEXEC) = 14 epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted) getdents64(14, /* 8 entries */, 8192) = 256 getdents64(14, /* 0 entries */, 8192) = 0 unlinkat(14, "artifact_group", 0) = 0 unlinkat(14, "artifact_name", 0) = 0 unlinkat(14, "type-info", 0) = 0 unlinkat(14, "header-info", 0) = 0 unlinkat(14, "meta-data", 0) = 0 unlinkat(14, "payload_type", 0) = 0 close(14) = 0 unlinkat(13, "header", AT_REMOVEDIR) = 0 unlinkat(13, "version", 0) = 0 unlinkat(13, "tmp", 0) = -1 EISDIR (Is a directory) newfstatat(13, "tmp", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0 openat(13, "tmp", O_RDONLY|O_CLOEXEC) = 14 epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted) getdents64(14, /* 2 entries */, 8192) = 48 getdents64(14, /* 0 entries */, 8192) = 0 close(14) = 0 unlinkat(13, "tmp", AT_REMOVEDIR) = 0 unlinkat(13, "stream-next", 0) = 0 unlinkat(13, "streams", 0) = -1 EISDIR (Is a directory) newfstatat(13, "streams", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0 openat(13, "streams", O_RDONLY|O_CLOEXEC) = 14 epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted) epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted) getdents64(14, /* 3 entries */, 8192) = 120 getdents64(14, /* 0 entries */, 8192) = 0 unlinkat(14, "A.ext4.delta", 0) = 0 close(14) = 0 unlinkat(13, "streams", AT_REMOVEDIR) = 0 unlinkat(13, "current_artifact_name", 0) = 0 unlinkat(13, "current_artifact_group", 0) = 0 close(13) = 0 unlinkat(12, "tree", AT_REMOVEDIR) = 0 close(12) = 0 close(10) = 0 futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1 ERRO[0000] Update module terminated abnormally: exit status 1 +++ exited with 1 +++ ``` Does anyone else perhaps have an idea? ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 22, 2022, 4:49pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/9 "2022-03-22T16:49:34Z")
Hi! One more reply to myself. I’ve checked with the tegrademo distro (see: [GitHub - OE4T/tegra-demo-distro: Reference/demonstration distro for meta-tegra](https://github.com/OE4T/tegra-demo-distro)) to verify if the error was originating from our customized Yocto distro. It turns out however, that a base image build of tegrademo shows the same problem: ```auto root@jetson-tx2-devkit:~# mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p33 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 291328... INFO[0000] No public key was provided for authenticating the artifact .............................................................. - 100 % INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 ERRO[0000] Update module terminated abnormally: exit status 1 ``` I’m not sure what this means, but perhaps the integration of Mender in OE4T is broken already? Not sure really. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 28, 2022, 8:00am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/10 "2022-03-28T08:00:03Z")
OK, so it turns out delta updates don’t work with the base image integration for `tegra-demo-distro`. See this issue as well: [Mender delta update issues with demo distro · Discussion #198 · OE4T/tegra-demo-distro · GitHub](https://github.com/OE4T/tegra-demo-distro/discussions/198) I’d really like to solve this problem, we’re currently holding back on getting a professional license for Mender (for the delta updates) because we can’t get it to work. I’d gladly contribute to the project to submit a fix, if only I had a strategy to dig deeper into it. Is it possible for someone who knows about the Yocto/Mender integration to look at this? Thanks in advance. @kacf perhaps? (forgive me for the direct referral, I’m including you because you’ve written [this](https://hub.mender.io/t/robust-delta-update-rootfs/1144) post). ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 29, 2022, 7:53am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/11 "2022-03-29T07:53:59Z")
Nothing immediately stands out from your description. But perhaps you can tell me a bit more about the overlay setup. I don’t think it should have an effect, but it depends. What is in `/etc/mender/mender-binary-delta.conf` Also, perhaps you can run `strace` with the `-f` argument? Then it will follow the fork into the Update Module, which is more interesting than what the client itself does. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 29, 2022, 1:18pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/12 "2022-03-29T13:18:03Z")
Hi Kristian, Thanks very much for your response 🙂 Here’s the output again, with the `strace -f` log attached as a separate log file. ```auto $ strace -o A2B.log -f mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p33 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 14524928... INFO[0000] No public key was provided for authenticating the artifact ..................................................... - 86 % INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 ERRO[0000] Download failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe ERRO[0000] Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe ``` Again, the above command was carried out on a vanilla `tegra-demo-distro` build. P.S. I renamed the `.log` file to a `.yml` file because I wasn’t able to upload it otherwise… Regarding the contents of `/etc/mender/mender-binary-delta.conf`: ```auto $ cat /etc/mender/mender-binary-delta.conf { "RootfsPartA": "/dev/mmcblk0p1", "RootfsPartB": "/dev/mmcblk0p33" } ``` Regarding the overlay setup; basically it’s `/home` and `var/lib` that’s overlayed: ```auto $ cat /etc/fstab /dev/root / auto ro 1 0 proc /proc proc defaults 0 0 devpts /dev/pts devpts mode=0620,ptmxmode=0666,gid=5 0 0 tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0 tmpfs /var/volatile tmpfs defaults 0 0 overlay /var/lib overlay lowerdir=/var/lib,upperdir=/data/overlays/var-lib-upper,workdir=/data/overlays/var-lib-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0 0 overlay /home overlay lowerdir=/home,upperdir=/data/overlays/home-upper,workdir=/data/overlays/home-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0 0 /dev/mmcblk0p34 /data auto defaults,data=journal 0 2 ``` Nothing really special w.r.t. the Mender updating I would think. [A2B.yml](https://hub.mender.io/uploads/short-url/rs0mYsxU6WUSTS1YbQqLM9cmtOb.yml) (803.3 KB) ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 29, 2022, 1:33pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/13 "2022-03-29T13:33:21Z")
By the way: what I could make up out of the `strace` log: It seems as if the state scripts are run? On line 3062 I see: ```auto 30446 unlinkat(13, "ArtifactInstall_Leave_80_bl-update", 0 ``` This deletes the post-install state script. This, and the stuff preceding it seems as though if the state script is fully run, which indicates a successful installation…? However, then on line 4098 I see the `mender-binary-delta` tool is run. It seems weird to me that the state scripts would be run before the delta updating tool is run, so probably my previous assumption is false. The first signs of something going wrong start at ~9540, but I’m not sure how to interpret this. ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 30, 2022, 7:20am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/14 "2022-03-30T07:20:51Z")
I can’t find anything out of the ordinary, the strace log looks pretty normal. It reads correctly, but then decides the checksum is wrong, which is still a mystery. What is the output of the two commands: ```bash dumpe2fs /dev/mmcblk0p1 | head -n 60 dumpe2fs /dev/mmcblk0p33 | head -n 60 ``` We know that there are certain filesystem features that mess up the checksum. ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 30, 2022, 7:22am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/15 "2022-03-30T07:22:17Z")
> [@krisvanrens](#): > > It seems as if the state scripts are run? On line 3062 I see: > > ```auto > 30446 unlinkat(13, "ArtifactInstall_Leave_80_bl-update", 0 > > ``` > > This deletes the post-install state script. This, and the stuff preceding it seems as though if the state script is fully run, which indicates a successful installation…? I think it’s not run. It’s just created in preparation for running, but never actually does it, as far as I can see. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 30, 2022, 8:26am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/16 "2022-03-30T08:26:57Z")
OK, that seems plausible yes. Here you go: ```auto root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 60 dumpe2fs 1.45.7 (28-Jan-2021) Filesystem volume name: Last mounted on: / Filesystem UUID: 7b256d5a-ceec-4958-b87c-baea4df77123 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 3014656 Block count: 3013491 Reserved block count: 150674 Free blocks: 2747512 Free inodes: 3010946 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 735 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 32768 Inode blocks per group: 2048 Flex block group size: 16 Filesystem created: Wed Mar 23 17:17:27 2022 Last mount time: Wed Mar 23 19:40:40 2022 Last write time: Wed Mar 23 19:40:40 2022 Mount count: 1 Maximum mount count: -1 Last checked: Wed Mar 23 17:17:28 2022 Check interval: 0 () Lifetime writes: 281 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Default directory hash: half_md4 Directory Hash Seed: 3fead4d3-25ac-4954-81ff-eb64fe838aa4 Checksum type: crc32c Checksum: 0xc34b830e Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED] Primary superblock at 0, Group descriptors at 1-1 Reserved GDT blocks at 2-736 Block bitmap at 737 (+737), csum 0x00008ea6 Inode bitmap at 753 (+753), csum 0x0000762a Inode table at 769-2816 (+769) 0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes Free blocks: Free inodes: 3711-32768 Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED] Backup superblock at 32768, Group descriptors at 32769-32769 Reserved GDT blocks at 32770-33504 Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6 Inode bitmap at 754 (bg #0 + 754), csum 0x00000000 Inode table at 2817-4864 (bg #0 + 2817) ``` And: ```auto root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p33 | head -n 60 dumpe2fs 1.45.7 (28-Jan-2021) Filesystem volume name: Last mounted on: / Filesystem UUID: 7b256d5a-ceec-4958-b87c-baea4df77123 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 3014656 Block count: 3013491 Reserved block count: 150674 Free blocks: 2747512 Free inodes: 3010946 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 735 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 32768 Inode blocks per group: 2048 Flex block group size: 16 Filesystem created: Wed Mar 23 17:17:27 2022 Last mount time: Tue Mar 29 14:18:19 2022 Last write time: Tue Mar 29 14:18:19 2022 Mount count: 2 Maximum mount count: -1 Last checked: Wed Mar 23 17:17:28 2022 Check interval: 0 () Lifetime writes: 282 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Default directory hash: half_md4 Directory Hash Seed: 3fead4d3-25ac-4954-81ff-eb64fe838aa4 Checksum type: crc32c Checksum: 0x658d3914 Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED] Primary superblock at 0, Group descriptors at 1-1 Reserved GDT blocks at 2-736 Block bitmap at 737 (+737), csum 0x00008ea6 Inode bitmap at 753 (+753), csum 0x0000762a Inode table at 769-2816 (+769) 0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes Free blocks: Free inodes: 3711-32768 Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED] Backup superblock at 32768, Group descriptors at 32769-32769 Reserved GDT blocks at 32770-33504 Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6 Inode bitmap at 754 (bg #0 + 754), csum 0x00000000 Inode table at 2817-4864 (bg #0 + 2817) ``` Again, thanks for taking your time to help me! 🙂 ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 30, 2022, 11:34am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/17 "2022-03-30T11:34:51Z")
Thanks, we might be on to something: The `metadata_csum` is known to generate unstable checksums, even on read-only filesystems. Try removing it when creating the filesystem, using `-O ^metadata_csum`. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 30, 2022, 11:50am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/18 "2022-03-30T11:50:02Z")
@kacf Great! I’m going to test this. ---
**Author:** ![krisvanrens](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/krisvanrens/32/1454_2.png) [@krisvanrens](https://hub.mender.io/u/krisvanrens)\ **Post date:** [March 30, 2022, 2:57pm UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/19 "2022-03-30T14:57:32Z")
Unfortunately it still doesn’t work: ```auto root@jetson-tx2-devkit:~# mender install A2B.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf INFO[0000] Mender running on partition: /dev/mmcblk0p1 INFO[0000] Start updating from local image file: [A2B.mender] Installing Artifact of size 702464... INFO[0000] No public key was provided for authenticating the artifact .............................................................. - 100 % INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent INFO[0000] Update module output: Failed to apply the delta, err: 1 ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 ERRO[0000] Update module terminated abnormally: exit status 1 ``` I changed the Yocto build by adding `EXTRA_IMAGECMD_append = " -O ^metadata_csum"` to `local.conf`. This seemed to work: ```auto root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 42 dumpe2fs 1.45.7 (28-Jan-2021) Filesystem volume name: Last mounted on: / Filesystem UUID: 64d53d86-c500-4ab6-9838-02d134974d94 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 3014656 Block count: 3013491 Reserved block count: 150674 Free blocks: 2747514 Free inodes: 3010946 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 735 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 32768 Inode blocks per group: 2048 Flex block group size: 16 Filesystem created: Wed Mar 30 14:21:46 2022 Last mount time: Wed Mar 30 14:34:49 2022 Last write time: Wed Mar 30 14:34:49 2022 Mount count: 2 Maximum mount count: -1 Last checked: Wed Mar 30 14:21:46 2022 Check interval: 0 () Lifetime writes: 282 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Default directory hash: half_md4 Directory Hash Seed: 1fe4bb93-b652-4dfd-8dbc-58c97d2d9dbd ``` The `metadata_csum` option is not there in the new build. I followed the same updating procedure as before, first installing/committing `A.mender` on a clean build, resulting in correct `show-provides` options. Then I added a line of comment to a file that’s installed, rebuilt the rootfs resulting in `B.mender`. Generated delta `A2B.mender` from the two artifacts and tried to install that. I added a new `strace -f` log (renamed to YML to make this possible). [A2B.yml](https://hub.mender.io/uploads/short-url/uqFG8u6dIs91ecrR8zLblRFKpGE.yml) (544.3 KB) ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [March 31, 2022, 7:19am UTC](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750/20 "2022-03-31T07:19:36Z")
Hmm, that’s disappointing. For the record I compared the rest of the filesystem features with the ones used in our test farm, and they are exactly identical to what we use. Could you call this: ```bash mender-artifact read A2B.mender ``` Make a note of the `rootfs_file_size` and the `Depends: rootfs-image.checksum` values. Then execute: ```bash dd if=/dev/mmcblk0p1 bs=1M count= iflag=count_bytes | sha256sum - ``` Replace `` with the value you found earlier, and make sure `/dev/mmcblk0p1` is the currently running partition. The question now is: Does the output from sha256sum match what is in the Depends value? [Next page](https://hub.mender.io/t/delta-update-update-module-checksum-error/4750.md?page=2)