Delta update update module checksum error

Not sure. As you pointed out, the one is expected. How do you generate the artifacts? Is this something I could try and reproduce?

Greetz,
Josef

I’m running the delta generator from the Yocto build output directory like so:

$ ../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender

The error seemed to me that something with xdelta3 went wrong. I checked and we have xdelta3 not in the Yocto build (is mentioned nowhere in the Mender documentation). I added it just for sure but this doesn’t help unfortunately. I checked and I have xdelta3 installed locally and on the device now and get the same results.

mender-artifact is installed system-wide and is at version 3.7.0, I verified this.

I’ve tried some more options, just to rule out stuff, but I’m not getting it to work.

I also tried disabling the xdelta3 checksum when building the delta artifact:

../../../../path/to/mender-binary-delta/mender-binary-delta-1.3.0/x86_64/mender-binary-delta-generator -o A2B.mender A.mender B.mender -- -- -n

Then when installing the installation takes much longer, but still fails in another spot:

$ sudo mender install A2B.mender 

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 227840...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0100] Update module output: Fatal failure: The expected and the actual checksums do not match! 
ERRO[0100] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0100] Update module terminated abnormally: exit status 1

So the update module itself errors out. Again I verified that A.mender is properly installed (on both slots 0 and 1).

Is there a possibility to enable more debugging information in the delta updating module? And does anyone have an idea how the checksum is calculated? I’m using a readonly root filesystem with various overlay mounts to be able store non-volatile configuration changes – could this be of any influence?

No luck still :confused:

If it helps any: here’s the strace output for an update attempt using the default delta update build (without the -n for xdelta3).

$ ls -l A2B.mender

-rw-rw-r-- 1 kris kris 230K Mar 18 15:18 A2B.mender
$ sudo strace mender install A2B.mender

...
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULLINFO[0000] Mender running on partition: /dev/mmcblk0p1
INFO[0000] Start updating from local image file: [A2B.mender]
Installing Artifact of size 242176...
INFO[0000] No public key was provided for authenticating the artifact
                                                               -   1 %) = 0
sched_yield()                           = 0
futex(0xb81838, FUTEX_WAKE_PRIVATE, 1)  = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
nanosleep({tv_sec=0, tv_nsec=3000}, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
.                                                              -   2 %) = 0
futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x4000080848, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81d88, FUTEX_WAKE_PRIVATE, 1)  = 1
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)
epoll_pwait(4, [], 128, 0, NULL, 12065536) = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xb81950, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)
..                                                             -   4 %) = 0
.............................................................. - 100 %futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL
) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0xb823e8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
read(12, 0x400019a000, 32768)           = -1 EAGAIN (Resource temporarily unavailable)
read(14, 0x40001a2000, 32768)           = -1 EAGAIN (Resource temporarily unavailable)
waitid(P_PID, 12046, INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent
INFO[0000] Update module output: Failed to apply the delta, err: 1
{si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12046, si_uid=0, si_status=1, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
wait4(12046, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, {ru_utime={tv_sec=0, tv_usec=40000}, ru_stime={tv_sec=0, tv_usec=52000}, ...}) = 12046
futex(0x40002004c8, FUTEX_WAKE_PRIVATE, 1) = 1
openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree/stream-next", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0
fcntl(12, F_GETFL)                      = 0x20800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE)
fcntl(12, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x40000cdc88) = 0
close(12)                               = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
openat(AT_FDCWD, "/etc//localtime", O_RDONLY) = 11
read(11, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 111
read(11, "", 4096)                      = 0
close(11)                               = 0
getpid()                                = 12025
write(3, "<11>Mar 18 14:15:19 mender[12025"..., 124) = 124
write(2, "\33[31mERRO\33[0m[0000] Download fai"..., 89ERRO[0000] Download failed: Update module terminated abnormally: exit status 1
) = 89
openat(AT_FDCWD, "/etc/mender/scripts", O_RDONLY|O_CLOEXEC) = 11
epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184b38) = -1 EPERM (Operation not permitted)
getdents64(11, /* 3 entries */, 8192)   = 80
getdents64(11, /* 0 entries */, 8192)   = 0
newfstatat(AT_FDCWD, "/etc/mender/scripts/version", {st_mode=S_IFREG|0644, st_size=1, ...}, AT_SYMLINK_NOFOLLOW) = 0
close(11)                               = 0
openat(AT_FDCWD, "/etc/mender/scripts/version", O_RDONLY|O_CLOEXEC) = 11
epoll_ctl(4, EPOLL_CTL_ADD, 11, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 11, 0x4000184bd8) = -1 EPERM (Operation not permitted)
read(11, "3", 512)                      = 1
read(11, "", 1535)                      = 0
newfstatat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, 0) = 0
openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184ba8) = -1 EPERM (Operation not permitted)
pipe2([13, 14], O_CLOEXEC)              = 0
epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537624, u64=547619384216}}) = 0
fcntl(13, F_GETFL)                      = 0 (flags O_RDONLY)
fcntl(13, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158537848, u64=547619384440}}) = 0
fcntl(14, F_GETFL)                      = 0x1 (flags O_WRONLY)
fcntl(14, F_SETFL, O_WRONLY|O_NONBLOCK) = 0
pipe2([15, 16], O_CLOEXEC)              = 0
epoll_ctl(4, EPOLL_CTL_ADD, 15, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = 0
fcntl(15, F_GETFL)                      = 0 (flags O_RDONLY)
fcntl(15, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538296, u64=547619384888}}) = 0
fcntl(16, F_GETFL)                      = 0x1 (flags O_WRONLY)
fcntl(16, F_SETFL, O_WRONLY|O_NONBLOCK) = 0
fcntl(14, F_GETFL)                      = 0x801 (flags O_WRONLY|O_NONBLOCK)
fcntl(14, F_SETFL, O_WRONLY)            = 0
fcntl(16, F_GETFL)                      = 0x801 (flags O_WRONLY|O_NONBLOCK)
fcntl(16, F_SETFL, O_WRONLY)            = 0
pipe2([17, 18], O_CLOEXEC)              = 0
getpid()                                = 12025
rt_sigprocmask(SIG_SETMASK, NULL, [], 8) = 0
rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 12056
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
close(18)                               = 0
read(17, "", 8)                         = 0
close(17)                               = 0
close(12)                               = 0
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184c48) = 0
close(14)                               = 0
epoll_ctl(4, EPOLL_CTL_DEL, 16, 0x4000184c48) = 0
close(16)                               = 0
futex(0x4000200148, FUTEX_WAKE_PRIVATE, 1) = 1
write(6, "\0", 1)                       = 1
waitid(P_PID, 12056, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=12056, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
rt_sigreturn({mask=[]})                 = 0
futex(0xb81938, FUTEX_WAKE_PRIVATE, 1)  = 1
wait4(12056, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, {ru_utime={tv_sec=0, tv_usec=4000}, ru_stime={tv_sec=0, tv_usec=0}, ...}) = 12056
unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", 0) = -1 EISDIR (Is a directory)
unlinkat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000/tree", AT_REMOVEDIR) = -1 ENOTEMPTY (Directory not empty)
openat(AT_FDCWD, "/var/lib/mender/modules/v3/payloads/0000", O_RDONLY|O_CLOEXEC) = 12
epoll_ctl(4, EPOLL_CTL_ADD, 12, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 12, 0x4000184dd8) = -1 EPERM (Operation not permitted)
unlinkat(12, "tree", 0)                 = -1 EISDIR (Is a directory)
newfstatat(12, "tree", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(12, "tree", O_RDONLY|O_CLOEXEC)  = 13
epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 13, 0x4000184c98) = -1 EPERM (Operation not permitted)
getdents64(13, /* 10 entries */, 8192)  = 336
getdents64(13, /* 0 entries */, 8192)   = 0
unlinkat(13, "current_device_type", 0)  = 0
unlinkat(13, "header", 0)               = -1 EISDIR (Is a directory)
newfstatat(13, "header", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "header", O_RDONLY|O_CLOEXEC) = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 8 entries */, 8192)   = 256
getdents64(14, /* 0 entries */, 8192)   = 0
unlinkat(14, "artifact_group", 0)       = 0
unlinkat(14, "artifact_name", 0)        = 0
unlinkat(14, "type-info", 0)            = 0
unlinkat(14, "header-info", 0)          = 0
unlinkat(14, "meta-data", 0)            = 0
unlinkat(14, "payload_type", 0)         = 0
close(14)                               = 0
unlinkat(13, "header", AT_REMOVEDIR)    = 0
unlinkat(13, "version", 0)              = 0
unlinkat(13, "tmp", 0)                  = -1 EISDIR (Is a directory)
newfstatat(13, "tmp", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "tmp", O_RDONLY|O_CLOEXEC)   = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 2 entries */, 8192)   = 48
getdents64(14, /* 0 entries */, 8192)   = 0
close(14)                               = 0
unlinkat(13, "tmp", AT_REMOVEDIR)       = 0
unlinkat(13, "stream-next", 0)          = 0
unlinkat(13, "streams", 0)              = -1 EISDIR (Is a directory)
newfstatat(13, "streams", {st_mode=S_IFDIR|0700, st_size=1024, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(13, "streams", O_RDONLY|O_CLOEXEC) = 14
epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2158538072, u64=547619384664}}) = -1 EPERM (Operation not permitted)
epoll_ctl(4, EPOLL_CTL_DEL, 14, 0x4000184af8) = -1 EPERM (Operation not permitted)
getdents64(14, /* 3 entries */, 8192)   = 120
getdents64(14, /* 0 entries */, 8192)   = 0
unlinkat(14, "A.ext4.delta", 0) = 0
close(14)                               = 0
unlinkat(13, "streams", AT_REMOVEDIR)   = 0
unlinkat(13, "current_artifact_name", 0) = 0
unlinkat(13, "current_artifact_group", 0) = 0
close(13)                               = 0
unlinkat(12, "tree", AT_REMOVEDIR)      = 0
close(12)                               = 0
close(10)                               = 0
futex(0x400003f648, FUTEX_WAKE_PRIVATE, 1) = 1
ERRO[0000] Update module terminated abnormally: exit status 1
+++ exited with 1 +++

Does anyone else perhaps have an idea?

Hi!

One more reply to myself. I’ve checked with the tegrademo distro (see: GitHub - OE4T/tegra-demo-distro: Reference/demonstration distro for meta-tegra) to verify if the error was originating from our customized Yocto distro.

It turns out however, that a base image build of tegrademo shows the same problem:

root@jetson-tx2-devkit:~# mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/mmcblk0p33
INFO[0000] Start updating from local image file: [A2B.mender]
Installing Artifact of size 291328...
INFO[0000] No public key was provided for authenticating the artifact
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent
INFO[0000] Update module output: Failed to apply the delta, err: 1
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1
ERRO[0000] Update module terminated abnormally: exit status 1

I’m not sure what this means, but perhaps the integration of Mender in OE4T is broken already? Not sure really.

OK, so it turns out delta updates don’t work with the base image integration for tegra-demo-distro. See this issue as well: Mender delta update issues with demo distro · Discussion #198 · OE4T/tegra-demo-distro · GitHub

I’d really like to solve this problem, we’re currently holding back on getting a professional license for Mender (for the delta updates) because we can’t get it to work. I’d gladly contribute to the project to submit a fix, if only I had a strategy to dig deeper into it.

Is it possible for someone who knows about the Yocto/Mender integration to look at this? Thanks in advance. @kacf perhaps? (forgive me for the direct referral, I’m including you because you’ve written this post).

Nothing immediately stands out from your description. But perhaps you can tell me a bit more about the overlay setup. I don’t think it should have an effect, but it depends. What is in /etc/mender/mender-binary-delta.conf

Also, perhaps you can run strace with the -f argument? Then it will follow the fork into the Update Module, which is more interesting than what the client itself does.

Hi Kristian,

Thanks very much for your response :slightly_smiling_face:

Here’s the output again, with the strace -f log attached as a separate log file.

$ strace -o A2B.log -f mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p33 
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 14524928...
INFO[0000] No public key was provided for authenticating the artifact 
.....................................................          -  86 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe 
ERRO[0000] Update module terminated abnormally: exit status 1: Payload: can not install Payload: demo-image-base-jetson-tx2-devkit.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/demo-image-base-jetson-tx2-devkit.ext4.delta: broken pipe

Again, the above command was carried out on a vanilla tegra-demo-distro build.

P.S. I renamed the .log file to a .yml file because I wasn’t able to upload it otherwise…

Regarding the contents of /etc/mender/mender-binary-delta.conf:

$ cat /etc/mender/mender-binary-delta.conf 
{
  "RootfsPartA": "/dev/mmcblk0p1",
  "RootfsPartB": "/dev/mmcblk0p33"
}

Regarding the overlay setup; basically it’s /home and var/lib that’s overlayed:

$ cat /etc/fstab 
/dev/root            /                    auto       ro              1  0
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0
overlay              /var/lib             overlay    lowerdir=/var/lib,upperdir=/data/overlays/var-lib-upper,workdir=/data/overlays/var-lib-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0  0
overlay              /home                overlay    lowerdir=/home,upperdir=/data/overlays/home-upper,workdir=/data/overlays/home-work,x-systemd.requires-mounts-for=/data,x-systemd.requires=data-overlay-setup.service 0  0
/dev/mmcblk0p34      /data                auto       defaults,data=journal 0  2

Nothing really special w.r.t. the Mender updating I would think.

A2B.yml (803.3 KB)

By the way: what I could make up out of the strace log:

It seems as if the state scripts are run? On line 3062 I see:

30446 unlinkat(13, "ArtifactInstall_Leave_80_bl-update", 0 <unfinished ...>

This deletes the post-install state script. This, and the stuff preceding it seems as though if the state script is fully run, which indicates a successful installation…?

However, then on line 4098 I see the mender-binary-delta tool is run. It seems weird to me that the state scripts would be run before the delta updating tool is run, so probably my previous assumption is false.

The first signs of something going wrong start at ~9540, but I’m not sure how to interpret this.

I can’t find anything out of the ordinary, the strace log looks pretty normal. It reads correctly, but then decides the checksum is wrong, which is still a mystery.

What is the output of the two commands:

dumpe2fs /dev/mmcblk0p1 | head -n 60
dumpe2fs /dev/mmcblk0p33 | head -n 60

We know that there are certain filesystem features that mess up the checksum.

I think it’s not run. It’s just created in preparation for running, but never actually does it, as far as I can see.

1 Like

OK, that seems plausible yes.

Here you go:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 60
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          7b256d5a-ceec-4958-b87c-baea4df77123
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747512
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 23 17:17:27 2022
Last mount time:          Wed Mar 23 19:40:40 2022
Last write time:          Wed Mar 23 19:40:40 2022
Mount count:              1
Maximum mount count:      -1
Last checked:             Wed Mar 23 17:17:28 2022
Check interval:           0 (<none>)
Lifetime writes:          281 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      3fead4d3-25ac-4954-81ff-eb64fe838aa4
Checksum type:            crc32c
Checksum:                 0xc34b830e


Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-736
  Block bitmap at 737 (+737), csum 0x00008ea6
  Inode bitmap at 753 (+753), csum 0x0000762a
  Inode table at 769-2816 (+769)
  0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes
  Free blocks: 
  Free inodes: 3711-32768
Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED]
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-33504
  Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6
  Inode bitmap at 754 (bg #0 + 754), csum 0x00000000
  Inode table at 2817-4864 (bg #0 + 2817)

And:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p33 | head -n 60
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          7b256d5a-ceec-4958-b87c-baea4df77123
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747512
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 23 17:17:27 2022
Last mount time:          Tue Mar 29 14:18:19 2022
Last write time:          Tue Mar 29 14:18:19 2022
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Mar 23 17:17:28 2022
Check interval:           0 (<none>)
Lifetime writes:          282 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      3fead4d3-25ac-4954-81ff-eb64fe838aa4
Checksum type:            crc32c
Checksum:                 0x658d3914


Group 0: (Blocks 0-32767) csum 0xc2fb [ITABLE_ZEROED]
  Primary superblock at 0, Group descriptors at 1-1
  Reserved GDT blocks at 2-736
  Block bitmap at 737 (+737), csum 0x00008ea6
  Inode bitmap at 753 (+753), csum 0x0000762a
  Inode table at 769-2816 (+769)
  0 free blocks, 29058 free inodes, 307 directories, 29058 unused inodes
  Free blocks: 
  Free inodes: 3711-32768
Group 1: (Blocks 32768-65535) csum 0x4c5e [INODE_UNINIT, ITABLE_ZEROED]
  Backup superblock at 32768, Group descriptors at 32769-32769
  Reserved GDT blocks at 32770-33504
  Block bitmap at 738 (bg #0 + 738), csum 0x00008ea6
  Inode bitmap at 754 (bg #0 + 754), csum 0x00000000
  Inode table at 2817-4864 (bg #0 + 2817)

Again, thanks for taking your time to help me! :slight_smile:

Thanks, we might be on to something: The metadata_csum is known to generate unstable checksums, even on read-only filesystems. Try removing it when creating the filesystem, using -O ^metadata_csum.

@kacf Great! I’m going to test this.

Unfortunately it still doesn’t work:

root@jetson-tx2-devkit:~# mender install A2B.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/mmcblk0p1  
INFO[0000] Start updating from local image file: [A2B.mender] 
Installing Artifact of size 702464...
INFO[0000] No public key was provided for authenticating the artifact 
.............................................................. - 100 %
INFO[0000] Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT 
INFO[0000] Update module output: xdelta3: normally this indicates that the source file is incorrect 
INFO[0000] Update module output: xdelta3: please verify the source file with sha1sum or equivalent 
INFO[0000] Update module output: Failed to apply the delta, err: 1 
ERRO[0000] Download failed: Update module terminated abnormally: exit status 1 
ERRO[0000] Update module terminated abnormally: exit status 1

I changed the Yocto build by adding EXTRA_IMAGECMD_append = " -O ^metadata_csum" to local.conf. This seemed to work:

root@jetson-tx2-devkit:~# dumpe2fs /dev/mmcblk0p1 | head -n 42
dumpe2fs 1.45.7 (28-Jan-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          64d53d86-c500-4ab6-9838-02d134974d94
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              3014656
Block count:              3013491
Reserved block count:     150674
Free blocks:              2747514
Free inodes:              3010946
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      735
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         32768
Inode blocks per group:   2048
Flex block group size:    16
Filesystem created:       Wed Mar 30 14:21:46 2022
Last mount time:          Wed Mar 30 14:34:49 2022
Last write time:          Wed Mar 30 14:34:49 2022
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Mar 30 14:21:46 2022
Check interval:           0 (<none>)
Lifetime writes:          282 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Default directory hash:   half_md4
Directory Hash Seed:      1fe4bb93-b652-4dfd-8dbc-58c97d2d9dbd

The metadata_csum option is not there in the new build.

I followed the same updating procedure as before, first installing/committing A.mender on a clean build, resulting in correct show-provides options. Then I added a line of comment to a file that’s installed, rebuilt the rootfs resulting in B.mender. Generated delta A2B.mender from the two artifacts and tried to install that.

I added a new strace -f log (renamed to YML to make this possible).

A2B.yml (544.3 KB)

Hmm, that’s disappointing. For the record I compared the rest of the filesystem features with the ones used in our test farm, and they are exactly identical to what we use.

Could you call this:

mender-artifact read A2B.mender

Make a note of the rootfs_file_size and the Depends: rootfs-image.checksum values. Then execute:

dd if=/dev/mmcblk0p1 bs=1M count=<rootfs_file_size> iflag=count_bytes | sha256sum -

Replace <rootfs_file_size> with the value you found earlier, and make sure /dev/mmcblk0p1 is the currently running partition.

The question now is: Does the output from sha256sum match what is in the Depends value?

Here you go:

$ mender-artifact read A2B.mender
Reading Artifact...
.............................................................. - 100 %
Mender artifact:
  Name: demo-image-base_3.1.15-9b9df01-dirty-kris
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[jetson-tx2-devkit jetson-tx2]'
  Provides group: 
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:
    ArtifactInstall_Leave_80_bl-update

Updates:
    0:
    Type:   mender-binary-delta
    Provides:
        rootfs-image.checksum: 693cd521c0aa1ecd46c1f81f8f91df9d44d9b5ed2ebc014dab3f8cd539cb20a4
        rootfs-image.version: demo-image-base_3.1.15-9b9df01-dirty-kris
    Depends:
        rootfs-image.checksum: 38e485758ba550532ca19fa9ec4bd61d0282e05924e5a24f304a90316199b1b0
    Clears Provides: ["artifact_group", "rootfs_image_checksum", "rootfs-image.*"]
    Metadata:
        {
          "delta_algorithm": "xdelta3",
          "rootfs_file_size": 12343259136
        }
    Files:
      name:     demo-image-base-jetson-tx2-devkit.ext4.delta
      size:     727497
      modified: 2022-03-30 16:44:49 +0200 CEST
      checksum: d283c647696e31e34403174228e1c43fdac64e8a2d094aebfbf6a281a11bdccb

On the device (with A.mender installed), it shows:

$ mender show-provides
artifact_name=demo-image-base_3.1.15-9b9df01-dirty-kris
rootfs-image.checksum=38e485758ba550532ca19fa9ec4bd61d0282e05924e5a24f304a90316199b1b0
rootfs-image.version=demo-image-base_3.1.15-9b9df01-dirty-kris

So at least the baked in root checksums correspond.

I’m having trouble executing the dd command you propose, because the bare tegra-demo-image uses busybox (which doesn’t have the iflags option). I guess running the following command is the same:

$ dd if=/dev/mmcblk0p1 bs=1 count=12343259136 | sha256sum -

But of course reading a root partition byte-for-byte takes ages.

I will try to install the ‘real’ coreutils, and get back with results later.

Allright, I did a rebuild of Yocto with coreutils included, and repeated the whole A/B/A2B installation and update process.

The mender-artifact output:

$ mender-artifact read A2B.mender
Reading Artifact...
.............................................................. - 100 %
Mender artifact:
  Name: demo-image-base_3.1.15-9b9df01-dirty-kris
  Format: mender
  Version: 3
  Signature: no signature
  Compatible devices: '[jetson-tx2-devkit jetson-tx2]'
  Provides group: 
  Depends on one of artifact(s): []
  Depends on one of group(s): []
  State scripts:
    ArtifactInstall_Leave_80_bl-update

Updates:
    0:
    Type:   mender-binary-delta
    Provides:
        rootfs-image.checksum: 88e7c0241dc48df3a887ef87c61c119e271a56cad36836d10f823332a56c818c
        rootfs-image.version: demo-image-base_3.1.15-9b9df01-dirty-kris
    Depends:
        rootfs-image.checksum: 84b5a7a89ea1b5fc947d8df6bc20a27102e9483caf18f1ea4977b00e2d1007db
    Clears Provides: ["artifact_group", "rootfs_image_checksum", "rootfs-image.*"]
    Metadata:
        {
          "delta_algorithm": "xdelta3",
          "rootfs_file_size": 12343259136
        }
    Files:
      name:     demo-image-base-jetson-tx2-devkit.ext4.delta
      size:     729948
      modified: 2022-03-31 10:19:34 +0200 CEST
      checksum: 56002eb6c0a58572003b2f17807e8aefdece378cdbbc33946604ff0c8c8b19ec

The dd output on the device:

root@jetson-tx2-devkit:~# dd if=/dev/mmcblk0p1 bs=1M count=12343259136 iflag=count_bytes | sha256sum -
11771+1 records in
11771+1 records out
12343259136 bytes (12 GB, 11 GiB) copied, 96.1862 s, 128 MB/s
9b6d570b8bbafa5ddf3335926164382b664ce8e6e0c745feed941300fc8a6ecb  -

So, apparently there is a difference indeed.

What’s interesting by the way: I ran the dd command on both read-only root partitions and get different results:

root@jetson-tx2-devkit:~# dd if=/dev/mmcblk0p1 bs=1M count=12343259136 iflag=count_bytes | sha256sum -
11771+1 records in
11771+1 records out
12343259136 bytes (12 GB, 11 GiB) copied, 96.1862 s, 128 MB/s
9b6d570b8bbafa5ddf3335926164382b664ce8e6e0c745feed941300fc8a6ecb  -
root@jetson-tx2-devkit:~# dd if=/dev/mmcblk0p33 bs=1M count=12343259136 iflag=count_bytes | sha256sum -
11771+1 records in
11771+1 records out
12343259136 bytes (12 GB, 11 GiB) copied, 97.3028 s, 127 MB/s
06df80590d916f30d29f14020f7be28c4fe13d38b673f4c4c0e10b3054b3df7f  -

Even though right before I installed/committed A.mender on both partitions in the preparation process.

I will verify now if this is also the case for a clean system flash (without installing any Mender updates).

Hmmm. OK. It seems even a freshly flashed system shows different hash values for either root partition:

root@jetson-tx2-devkit:~# dd if=/dev/mmcblk0p1 bs=1M count=12343259136 iflag=count_bytes | sha256sum -
11771+1 records in
11771+1 records out
12343259136 bytes (12 GB, 11 GiB) copied, 94.952 s, 130 MB/s
6864964e08e50e1021d24f845fc50f84a8fc06b87a1977a1f973adf765236dea  -
root@jetson-tx2-devkit:~# dd if=/dev/mmcblk0p33 bs=1M count=12343259136 iflag=count_bytes | sha256sum -
11771+1 records in
11771+1 records out
12343259136 bytes (12 GB, 11 GiB) copied, 96.8051 s, 128 MB/s
cc8017f288d6cdb17f1f61d6e0f1b368f6e223159fac8b437e0b69e24a2c1114  -

(this is run from boot slot 0, right after the flashing procedure, no Mender artifacts installed)

I also checked if the above checksums remain the same if I switch root partition (which, I guess will rule out the overlay mounts etc.). But the checksums are in fact equal regardless the active boot slot (though still different between the root partitions).

I also asked in the OE4T project if they can think of any reason why this is happening.