No Artifacts found error when trying delta-updates

General Discussions
1

I’m testing delta updates on my jetson tx2. I’ve followed the tutorial steps->

  • I’m using mender client v2.3.0
  • I have created deltas as explained
  • I did a full first rootfs update
  • I’ve mender enterprise account

After the first rootfs update, I’ve removed the full rootfs image to test delta artifact update. I’ve never managed to do it. The log says

’ No compatible artifact is found’

2

Can you provide full shell logs of your creation of the details as well as the full deployment logs and the client side logs (ie journalctl -u mender-client)?

Drew

3

Here is the log > `Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: update-status-report [none] → status-report-error [ArtifactFailure]”

Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=error msg=“Handling report error state with status: failure”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=error msg=“Error while performing update: failure ({{{https://s3.amazonaws.com/hosted-mender-artifacts/5f58a9d1fd4581aa90322f38/2f2b414d-aae3-406d-a51a-9965fbc70d94?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAQWI25QR6MOCESDWO%2F20201002%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20201002T081450Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=78df6b0e66056fe8506ed03df37c588103fe2329b5402a3a31edf156416c20dc 2020-10-02T08:14:50.586736904Z} [j140-tx2] [rootfs-image] dev-delta-update-test-6 map[rootfs_image_checksum:6d3fa52bae3458215a90d2d85793603817d6ed52be43f94d21d98f68469588fd]} 67d9711f-f2f9-49b9-b596-47c06340576d 4 false})”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: status-report-error [ArtifactFailure] → idle [Idle]”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“authorization data present and valid”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: idle [Idle] → check-wait [Idle]”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: check-wait [Idle] → inventory-update [Sync]”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: inventory-update [Sync] → check-wait [Idle]”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: check-wait [Idle] → update-check [Sync]”
Oct 02 08:16:05 j140-tx2 mender[5719]: time=“2020-10-02T08:16:05Z” level=info msg=“State transition: update-check [Sync] → check-wait [Idle]”

`

4

I made some progress: I have the following log while I tried to deploy delta

2020-10-02 09:03:47 +0000 UTC info: Running Mender client version: 2.3.0-dirty
2020-10-02 09:03:47 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2020-10-02 09:03:47 +0000 UTC info: No public key was provided for authenticating the artifact
2020-10-02 09:03:53 +0000 UTC info: Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
2020-10-02 09:03:53 +0000 UTC info: Update module output: xdelta3: normally this indicates that the source file is incorrect
2020-10-02 09:03:53 +0000 UTC info: Update module output: xdelta3: please verify the source file with sha1sum or equivalent
2020-10-02 09:03:53 +0000 UTC info: Update module output: Failed to apply the delta, err: 1
2020-10-02 09:03:53 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: image-name.ext4.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/image-name.ext4.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/image-name.ext4.delta: broken pipe
2020-10-02 09:03:53 +0000 UTC info: State transition: update-store [Download_Enter] → cleanup [Error]
2020-10-02 09:03:53 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

5

What do the following commands show on your target board?

# mount
# cat /etc/fstab
6

mount

proc on /proc type proc (rw,relatime)
none on /dev type devtmpfs (rw,relatime,size=3986104k,nr_inodes=996526,mode=755)
sysfs on /sys type sysfs (rw,relatime)
/dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/debug type cgroup (rw,nosuid,nodev,noexec,relatime,debug)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/mmcblk0p31 on /data type ext4 (rw,relatime,data=ordered,x-systemd.growfs)

cat /etc/fstab

/dev/root / auto defaults 1 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0
tmpfs /var/volatile tmpfs defaults 0 0

uncomment this if your device has a SD/MMC/Transflash slot

#/dev/mmcblk0p1 /media/card auto defaults,sync,noauto 0 0

/dev/mmcblk0p31 /data auto defaults,x-systemd.growfs 0 0

7

Your root filesystem is mounted read/write which means that it has likely been updated. Any change to the contents will invalidate the binary delta.

Drew

8

Drew,
I’m not sure I understand it well. Talking about the rootfs, which is readonly rootfs.

Also did you look at my last message with a log?
It shows one error during the update Update module output: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT. Same has been discussed here link.

I had noticed that the checksums are different in my device with the mender artifact checksum after the first mender rootfs update.

Suggested in this post is to add variable EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit”. I had added it to my local.conf. And yet I didn’t manage to perform a delta update.

9

Your mount command includes the following:

/dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered)

which indicates that the root filesystem is mounted read-write. The binary delta is calculated against the unmodified original image and if you attempt to apply it against a modified filesystem you will get checksum mismatch errors.

You need to rerun your Yocto build with the following in your local.conf to make the root filesystem readonly:

EXTRA_IMAGE_FEATURES_append = " read-only-rootfs "

Drew

10

I can tell you my local.conf already had this variable set

IMAGE_FEATURES_append = " read-only-rootfs "

I was already using readonly rootfs for these delta tests :slight_smile:

11

Drew, following the latest mount command ouput after using the IMAGE_FEATURES_append = " read-only-rootfs "
.> `proc on /proc type proc (rw,relatime)

none on /dev type devtmpfs (rw,relatime,size=3986104k,nr_inodes=996526,mode=755)
sysfs on /sys type sysfs (rw,relatime)
/dev/mmcblk0p30 on / type ext4 (ro,relatime,data=ordered)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/debug type cgroup (rw,nosuid,nodev,noexec,relatime,debug)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
tmpfs on /var/lib type tmpfs (rw,relatime)
tmpfs on /var/cache type tmpfs (rw,relatime)
tmpfs on /var/spool type tmpfs (rw,relatime)
`

I still have the same error with delta update. I’m striking the issue xdelta3: target window checksum mismatch: XD3_INVALID_INPUT

12

OK. Can you provide the following information?

From the client:

# cat /etc/mender/mender.conf
# cat /data/mender/mender.conf
# mount | grep mmc
# sha256sum /dev/mmcblk0p1
# sha256sum /devmmcblk0p30

And then from your development host:

$ mender-artifact read <full-path-to-first-mender-artifact>
$ mender-artifact read <full-path-to-second-mender-artifact>
$ mender-artifact read <full-path-to-binary-delta-artifact>

You may need a mender-artifact binary. You can download a prebuilt one from here.

Do you have anything in your system that would remount the root filesystem as read-write?

Drew

13

Do you have anything in your system that would remount the root filesystem as read-write?
No there is no recipe from my side doing this.

cat /etc/mender/mender.conf

{“InventoryPollIntervalSeconds”: 1800, “RetryPollIntervalSeconds”: 300, “RootfsPartA”: “/dev/mmcblk0p1”, “RootfsPartB”: “/dev/mmcblk0p30”, “ServerURL”: “https://hosted.mender.io”, “TenantToken”: “original-tenant-token”, “UpdatePollIntervalSeconds”: 1800}

cat /data/mender/mender.conf

{“InventoryPollIntervalSeconds”: 1800, “RetryPollIntervalSeconds”: 300, “RootfsPartA”: “/dev/mmcblk0p1”, “RootfsPartB”: “/dev/mmcblk0p30”, “ServerURL”: “https://hosted.mender.io”, “TenantToken”: “original-tenant-token”, “UpdatePollIntervalSeconds”: 1800}

mount | grep mmc

/dev/mmcblk0p30 on / type ext4 (ro,relatime,data=ordered)
/dev/mmcblk0p31 on /data type ext4 (rw,relatime,data=ordered)

sha256sum /dev/mmcblk0p1

1da143d99d8aa61091be593002cf030b6fbaef32f24bc3c1414872dc60cf6577 /dev/mmcblk0p1

sha256sum /dev/mmcblk0p30

be1a2a73e66dc00fe84694e6346bdbd28489b3598516c09a93ada7250ca1b787 /dev/mmcblk0p30

mender-artifact read v11.mender

Mender artifact:
Name: dev-delta-update-test-11
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[j140-tx2]’
Provides group:
Depends on one of artifact(s):
Depends on one of group(s):
State scripts:
ArtifactReboot_Enter_50

Updates:
0:
Type: rootfs-image
Provides:
rootfs_image_checksum: 011b7511e377a2566b20f2c6ae43407d15c4d7981f4e1c5bc73ef0a8fc0ebd1b
Depends: Nothing
Metadata: Nothing
Files:
name: image-j140-tx2.ext4
size: 3078619136
modified: 2020-10-05 11:15:10 +0200 CEST
checksum: 011b7511e377a2566b20f2c6ae43407d15c4d7981f4e1c5bc73ef0a8fc0ebd1b

mender-artifact read v12.mender

Mender artifact:
Name: dev-delta-update-test-12
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[j140-tx2]’
Provides group:
Depends on one of artifact(s):
Depends on one of group(s):
State scripts:
ArtifactReboot_Enter_50

Updates:
0:
Type: rootfs-image
Provides:
rootfs_image_checksum: a35ab70339e25c214c29f4063e09bdf9c07916c4daa88878f9fb3aa8e83bedf6
Depends: Nothing
Metadata: Nothing
Files:
name: image-j140-tx2.ext4
size: 3078619136
modified: 2020-10-05 18:04:12 +0200 CEST
checksum: a35ab70339e25c214c29f4063e09bdf9c07916c4daa88878f9fb3aa8e83bedf6

mender-artifact read v11.12.mender

Mender artifact:
Name: dev-delta-update-test-12
Format: mender
Version: 3
Signature: no signature
Compatible devices: ‘[j140-tx2]’
Provides group:
Depends on one of artifact(s):
Depends on one of group(s):
State scripts:
ArtifactReboot_Enter_50

Updates:
0:
Type: mender-binary-delta
Provides:
rootfs_image_checksum: a35ab70339e25c214c29f4063e09bdf9c07916c4daa88878f9fb3aa8e83bedf6
Depends:
rootfs_image_checksum: 011b7511e377a2566b20f2c6ae43407d15c4d7981f4e1c5bc73ef0a8fc0ebd1b
Metadata:
{
“delta_algorithm”: “xdelta3”,
“rootfs_file_size”: 3078619136
}
Files:
name: image-j140-tx2.ext4.delta
size: 222933
modified: 2020-10-06 09:54:39 +0200 CEST
checksum: c3bfb32f4eaeb8fb2fdd4f5ae364ba8378d96346765c79e274a10277bf44a345

Nb: I have set the variable ‘EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit”’ to generate image and artifacts

14

To be able to perform an delta update, you must install v11.mender and then deploy v11.12.mender.

I am not sure which versions you are running on /dev/mmcblk0p1 and /dev/mmcblk0p30 but none of them match the checksum of in the delta artifact as stated:

Depends:
rootfs_image_checksum: 011b7511e377a2566b20f2c6ae43407d15c4d7981f4e1c5bc73ef0a8fc0ebd1b

Just to confirm v11.mender is deployed and the current active partition?

EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit

Can you try doing this instead:

EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit -O ^has_journal

Above is a fix that currently exists in master branch of meta-mender I believe. See here for more info.

15

Yes, I confirm that I have performed first rootfs update with v11.mender. So the current version installed is v11.mender.

16

Thanks for confirming. Then something is changing the checksum “behind the scene”.

So please try:

EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit -O ^has_journal

I also see that all your Artifacts have a ArtifactReboot_Enter_50 script. What does it do?

17

ArtifactReboot_Enter_50 script is looks to be a mender-state-script which is enabling A/B partitions during update.

#!/bin/sh
echo "Starting nvidia mender enter reboot redundant boot steps" >&2

set -e
num_slots=`nvbootctrl get-number-slots`
if [ $num_slots != 2 ]; then
echo "Enabling A/B update mode using nv_update_engine" >&2
nv_update_engine --enable-ab 1>&2
fi

new_boot_part=`fw_printenv -n mender_boot_part`
new_mender_boot_part_mountpoint=/tmp/new_mender_boot_part
echo "Mounting new install partition ${new_boot_part}" >&2
mkdir -p ${new_mender_boot_part_mountpoint}
mount /dev/mmcblk0p${new_boot_part} ${new_mender_boot_part_mountpoint}
mount -oremount,rw /
cp ${new_mender_boot_part_mountpoint}/opt/ota_package/bl_update_payload_current /opt/ota_package/bl_update_payload_new
rm /opt/ota_package/bl_update_payload
ln -s /opt/ota_package/bl_update_payload_new /opt/ota_package/bl_update_payload
nv_update_engine --install 1>&2
# Should never get here, nv_update_engine will reboot on its own after successful update
echo "A failure occurred in nv_update_engine (rc $?).. nv_update_engine did not reboot" >&2
rm /opt/ota_package/bl_update_payload
ln -s /opt/ota_package/bl_update_payload_current /opt/ota_package/bl_update_payload
exit 1

There I see mount rw command. is that to write the latest image or is this causing checksum modification in my case?

18

I still have the same error even after using this build option EXTRA_IMAGECMD_ext4 = “-i 4096 -O ^64bit -O ^has_journal".

There is still checksum mismatch after the update.

19

Hmm, this is probably it. It actually also writes a file (changing the filesystem).

@dwalkes @madisox, are you aware of this and is there a way to workaround this?

20

No it does not. It is copies it out of inactive partition. But this I would guess to a problem with read-only rootfs anyway