# Delta Upgrades on UBI volumes

**URL:** <https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195>\
**Category:** Yocto Project\
**Created:** [September 12, 2023, 10:13am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195 "2023-09-12T10:13:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 12, 2023, 10:13am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/1 "2023-09-12T10:13:46Z")

</div>

# Board description

Owa5x: IMX 8MP

URL: [https://www.owasys.com/en/products/owa5x](https://www.owasys.com/en/products/owa5x)

Hi,

I have integrated Mender for our board which is running on NAND flash using UBI partitions. I can run full upgrades successfully but am getting errors when I attempt to perform a delta upgrade. Here is the log message of the failed upgrade:

> 2023-09-12 04:55:43 +0000 UTC info: Running Mender client version: 3.5.0  
> 2023-09-12 04:55:44 +0000 UTC info: State transition: update-fetch [Download\_Enter] → update-store [Download\_Enter]  
> 2023-09-12 04:55:45 +0000 UTC info: No public key was provided for authenticating the artifact  
> 2023-09-12 04:55:45 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: parse\_metadata\_file: decoder\_arguments\_count:0  
> 2023-09-12 04:55:46 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: running with decoder args: xdelta3 -9 -f -d -s ubi0\_1 streams/asset-health-owa5x.ubifs.delta ubi0\_0  
> 2023-09-12 04:55:49 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: file open failed: read: ubi0\_1: No such file or directory  
> 2023-09-12 04:55:49 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: Failed to apply the delta, err: 1  
> 2023-09-12 04:55:50 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: asset-health-owa5x.ubifs.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: broken pipe  
> 2023-09-12 04:55:50 +0000 UTC info: State transition: update-store [Download\_Enter] → cleanup [Error]  
> 2023-09-12 04:55:50 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

Is anyone able to explain what is going wrong? Why is it complaining about not being able to read from ubi0\_1? I can see ubi0\_0 and ubi0\_1 under /dev (redundant rootfs partitions).

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [September 13, 2023, 1:47am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/2 "2023-09-13T01:47:17Z")

</div>

Hello @smartin,

I just ran some tests on my end (with a regular SD card and EXT4) but I found interesting that the line was:

```bash
... Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": running with decoder args: xdelta3 -9 -f -d -s /dev/mmcblk0p2 streams/core-image-base.ext4.delta /dev/mmcblk0p3

```

Please notice it gets full paths to the partitions, so the Update Module should run something like:

```bash
xdelta3 -9 -f -d -s /dev/ubi0_1 streams/asset-health-owa5x.ubifs.delta /dev/ubi0_0

```

Instead of

```sh
xdelta3 -9 -f -d -s ubi0_1 streams/asset-health-owa5x.ubifs.delta ubi0_0

```

Quick question, in your `/etc/mender/mender-binary-delta.conf` does the `RootfsPartA` and `RootfsPartB` look with the full path as below or just the name?

```json
{
  "RootfsPartA": "/dev/mmcblk0p2",
  "RootfsPartB": "/dev/mmcblk0p3"
}

```

If this image is built from Yocto, please make sure `MENDER_ROOTFS_PART_A` and `MENDER_ROOTFS_PART_B` holds the full path instead of just the name of the partition.

Have a nice day!

Luis

---

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 16, 2023, 4:43am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/3 "2023-09-16T04:43:19Z")

</div>

Luis,

When I appended /dev/ to the device name, it prevented full upgrades from working. I switched /data/mender/mender.conf back to:

{  
“RootfsPartA”: “ubi0\_0”,  
“RootfsPartB”: “ubi0\_1”  
}

The full upgrades worked again after that. I left /etc/mender/mender-binary-delta.conf as this:

{  
“RootfsPartA”: “/dev/ubi0\_0”,  
“RootfsPartB”: “/dev/ubi0\_1”  
}

I then attempted to do a delta upgrade, but am now getting the following error:

2023-09-16 04:32:20 +0000 UTC info: Running Mender client version: 3.5.0  
2023-09-16 04:32:22 +0000 UTC info: State transition: update-fetch [Download\_Enter] → update-store [Download\_Enter]  
2023-09-16 04:32:22 +0000 UTC info: No public key was provided for authenticating the artifact  
2023-09-16 04:32:22 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: parse\_metadata\_file: decoder\_arguments\_count:0  
2023-09-16 04:32:23 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: running with decoder args: xdelta3 -9 -f -d -s /dev/ubi0\_0 streams/asset-health-owa5x.ubifs.delta /dev/ubi0\_1  
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: target window checksum mismatch: XD3\_INVALID\_INPUT  
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: normally this indicates that the source file is incorrect  
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: please verify the source file with sha1sum or equivalent  
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: Failed to apply the delta, err: 1  
2023-09-16 04:32:30 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: asset-health-owa5x.ubifs.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: broken pipe  
2023-09-16 04:32:30 +0000 UTC info: State transition: update-store [Download\_Enter] → cleanup [Error]  
2023-09-16 04:32:30 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]

I have been following the instructions from [Create a Delta update Artifact | Mender documentation](https://docs.mender.io/artifact-creation/create-a-delta-update-artifact) for building a delta artifact. Why would it be complaining about the source file like this?

Regards,  
Scott

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [September 17, 2023, 9:19pm UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/4 "2023-09-17T21:19:31Z")

</div>

Hello @smartin,

So we have progress now. The `xdelta3` utility can locate the origin rootfs and the destination rootfs, and now the issue is to understand on why the deltas can not be installed.

Most of the times, this happens because the rootfs is remounted during the booting/load process or because you have journaling enabled in your rootfs that makes the origin to change slightly its checksum.

Can you check the “[Delta updates section](https://docs.mender.io/troubleshoot/mender-client#delta-updates)” from the docs? I think there are several checks you can perform on your end to find the root cause.

Have a nice day!  
Luis

---

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 18, 2023, 2:28am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/5 "2023-09-18T02:28:57Z")

</div>

Thanks Luis,

I’m continuing to make some progress. I’m confused though why /data/mender/mender.conf needs different configuration than /etc/mender/mender-binary-delta.conf. That seems to be a bug to me, although I can work around it for the time being.

I’m now getting the following response when I attempt a delta upgrade.

Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:41Z” level=info msg=“State transition: update-fetch [Download\_Enter] → update-store [Download\_Enter]”  
Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:42Z” level=info msg=“No public key was provided for authenticating the artifact”  
Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:42Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": parse\_metadata\_file: decoder\_arguments\_count:0”  
Sep 18 02:15:44 owa5x mender[470]: time=“2023-09-18T02:15:44Z” level=info msg="Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": running with decoder args: xdelta3 -9 -f -d -s /dev/ubi0\_1 streams/asset-health-owa5x.ubifs.delta /dev/ubi0\_0 "  
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": xdelta3: write failed: /dev/ubi0\_0: Operation not permitted”  
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Failed to apply the delta, err: 1”  
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=error msg=“Artifact install failed: Update module terminated abnormally: exit status 1”

I’ve noted that /dev/ubi0\_0 is owned by root and has 0600 permissions. Do I need to change these permissions? I tried changing the permissions manually but got the same result.

Regards,  
Scott

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [September 18, 2023, 5:22am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/6 "2023-09-18T05:22:27Z")

</div>

Hello @smartin,

`mender.conf` is the configuration file for mender as a whole, while `mender-binary-delta.conf` is intended for the delta utility only. It allow us to have a different license for both as the delta feature is proprietary while the mender client itself is Open Source. Also, if your integration consumes the name of the partition while the feature requires the whole path, then it gives you that versatility.

Usually you should specify the whole paths in the `mender.conf` file as well, but as this is a custom integration, then we will avoid to change it in order to break anything that is already working for you.

The Update Module is ran by Mender as the root user, so it should not have any permission problem.

Just to get some context, how did you solve the checksum mismatch?

BR,  
Luis

---

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 18, 2023, 5:41am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/7 "2023-09-18T05:41:20Z")

</div>

Luis,

I had modified rootfs and then made it read-only. I have fixed this now by applying overlayfs to some directories.

The confusing thing about mender.conf and mender-binary-delta.conf is that when I change mender.conf to use /dev/ubi0\_0/1, the full upgrade no longer works. This is what the Yocto recipe will give me when I set MENDER\_ROOTFS\_PART\_A/B in local.conf. But like I said, I’m able to work around that problem so it isn’t holding me up, it’s just confusing.

Is there any other way that I can investigate this “Operation not permitted” problem? I have confirmed that it is attempting to write to the inactive ubi volume and that volume has not been mounted by any other process. I don’t see anything in the troubleshooting section on this problem.

Regards,  
Scott

---

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 18, 2023, 7:41am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/8 "2023-09-18T07:41:58Z")

</div>

Just to add to this, I can manually mount the redundant volume and write to it. I’m not understanding why Mender is reporting that it can’t write to this volume.

Regards,  
Scott

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [September 20, 2023, 4:40pm UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/9 "2023-09-20T16:40:44Z")

</div>

Hello @smartin,

My apologies for the delay as I was trying to figure out a way to get more information.  
Have you tried to perform a Standalone installation? That means you copy the artifact to the device (to the data partition for example) and then run `mender install /data/path/to/artifact.mender`. If you can run that, can you share an strace of that command? You can use our support channel support.northern.tech (or [support@mender.io](mailto:support@mender.io)) if you consider these traces includes any sensitive information.

Best regards,  
Luis

---

<div class="post-metadata">

**Author:** ![smartin](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@smartin](https://hub.mender.io/u/smartin)\
**Post date:** [September 25, 2023, 3:32am UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/10 "2023-09-25T03:32:33Z")

</div>

Luis,

Here is the strace log. I can see many of the following kinds of messages:

openat(AT\_FDCWD, “/var/lib/mender/mender.conf”, O\_RDONLY|O\_CLOEXEC) = 7  
epoll\_ctl(4, EPOLL\_CTL\_ADD, 7, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=2150759968, u64=281472832503328}}) = -1 EPERM (Operation not permitted)

Do you have any explanation for these errors?

Regards,  
Scott

(Attachment strace.log is missing)

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [September 26, 2023, 10:20pm UTC](https://hub.mender.io/t/delta-upgrades-on-ubi-volumes/6195/11 "2023-09-26T22:20:20Z")

</div>

Hello @smartin,

I can confirm I received the trace log by a private chanel and I see there are several files not being able to be open, including some ones related to certificates. Are all these files in the overlay instead of being available in the RO rootfs? Is there any chance you can move them to the RO partition for the sake of testing?

Best regards,  
Luis