I have integrated Mender for our board which is running on NAND flash using UBI partitions. I can run full upgrades successfully but am getting errors when I attempt to perform a delta upgrade. Here is the log message of the failed upgrade:
2023-09-12 04:55:43 +0000 UTC info: Running Mender client version: 3.5.0
2023-09-12 04:55:44 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2023-09-12 04:55:45 +0000 UTC info: No public key was provided for authenticating the artifact
2023-09-12 04:55:45 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: parse_metadata_file: decoder_arguments_count:0
2023-09-12 04:55:46 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: running with decoder args: xdelta3 -9 -f -d -s ubi0_1 streams/asset-health-owa5x.ubifs.delta ubi0_0
2023-09-12 04:55:49 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: file open failed: read: ubi0_1: No such file or directory
2023-09-12 04:55:49 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: Failed to apply the delta, err: 1
2023-09-12 04:55:50 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: asset-health-owa5x.ubifs.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: broken pipe
2023-09-12 04:55:50 +0000 UTC info: State transition: update-store [Download_Enter] → cleanup [Error]
2023-09-12 04:55:50 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]
Is anyone able to explain what is going wrong? Why is it complaining about not being able to read from ubi0_1? I can see ubi0_0 and ubi0_1 under /dev (redundant rootfs partitions).
If this image is built from Yocto, please make sure MENDER_ROOTFS_PART_A and MENDER_ROOTFS_PART_B holds the full path instead of just the name of the partition.
I then attempted to do a delta upgrade, but am now getting the following error:
2023-09-16 04:32:20 +0000 UTC info: Running Mender client version: 3.5.0
2023-09-16 04:32:22 +0000 UTC info: State transition: update-fetch [Download_Enter] → update-store [Download_Enter]
2023-09-16 04:32:22 +0000 UTC info: No public key was provided for authenticating the artifact
2023-09-16 04:32:22 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: parse_metadata_file: decoder_arguments_count:0
2023-09-16 04:32:23 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: running with decoder args: xdelta3 -9 -f -d -s /dev/ubi0_0 streams/asset-health-owa5x.ubifs.delta /dev/ubi0_1
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: target window checksum mismatch: XD3_INVALID_INPUT
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: normally this indicates that the source file is incorrect
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: xdelta3: please verify the source file with sha1sum or equivalent
2023-09-16 04:32:30 +0000 UTC info: Output (stderr) from command “/usr/share/mender/modules/v3/mender-binary-delta”: Failed to apply the delta, err: 1
2023-09-16 04:32:30 +0000 UTC error: Artifact install failed: Update module terminated abnormally: exit status 1: Payload: can not install Payload: asset-health-owa5x.ubifs.delta: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: write /var/lib/mender/modules/v3/payloads/0000/tree/streams/asset-health-owa5x.ubifs.delta: broken pipe
2023-09-16 04:32:30 +0000 UTC info: State transition: update-store [Download_Enter] → cleanup [Error]
2023-09-16 04:32:30 +0000 UTC info: State transition: cleanup [Error] → update-status-report [none]
So we have progress now. The xdelta3 utility can locate the origin rootfs and the destination rootfs, and now the issue is to understand on why the deltas can not be installed.
Most of the times, this happens because the rootfs is remounted during the booting/load process or because you have journaling enabled in your rootfs that makes the origin to change slightly its checksum.
Can you check the “Delta updates section” from the docs? I think there are several checks you can perform on your end to find the root cause.
I’m continuing to make some progress. I’m confused though why /data/mender/mender.conf needs different configuration than /etc/mender/mender-binary-delta.conf. That seems to be a bug to me, although I can work around it for the time being.
I’m now getting the following response when I attempt a delta upgrade.
Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:41Z” level=info msg=“State transition: update-fetch [Download_Enter] → update-store [Download_Enter]”
Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:42Z” level=info msg=“No public key was provided for authenticating the artifact”
Sep 18 02:15:42 owa5x mender[470]: time=“2023-09-18T02:15:42Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": parse_metadata_file: decoder_arguments_count:0”
Sep 18 02:15:44 owa5x mender[470]: time=“2023-09-18T02:15:44Z” level=info msg="Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": running with decoder args: xdelta3 -9 -f -d -s /dev/ubi0_1 streams/asset-health-owa5x.ubifs.delta /dev/ubi0_0 "
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": xdelta3: write failed: /dev/ubi0_0: Operation not permitted”
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=info msg=“Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Failed to apply the delta, err: 1”
Sep 18 02:15:51 owa5x mender[470]: time=“2023-09-18T02:15:51Z” level=error msg=“Artifact install failed: Update module terminated abnormally: exit status 1”
I’ve noted that /dev/ubi0_0 is owned by root and has 0600 permissions. Do I need to change these permissions? I tried changing the permissions manually but got the same result.
mender.conf is the configuration file for mender as a whole, while mender-binary-delta.conf is intended for the delta utility only. It allow us to have a different license for both as the delta feature is proprietary while the mender client itself is Open Source. Also, if your integration consumes the name of the partition while the feature requires the whole path, then it gives you that versatility.
Usually you should specify the whole paths in the mender.conf file as well, but as this is a custom integration, then we will avoid to change it in order to break anything that is already working for you.
The Update Module is ran by Mender as the root user, so it should not have any permission problem.
Just to get some context, how did you solve the checksum mismatch?
I had modified rootfs and then made it read-only. I have fixed this now by applying overlayfs to some directories.
The confusing thing about mender.conf and mender-binary-delta.conf is that when I change mender.conf to use /dev/ubi0_0/1, the full upgrade no longer works. This is what the Yocto recipe will give me when I set MENDER_ROOTFS_PART_A/B in local.conf. But like I said, I’m able to work around that problem so it isn’t holding me up, it’s just confusing.
Is there any other way that I can investigate this “Operation not permitted” problem? I have confirmed that it is attempting to write to the inactive ubi volume and that volume has not been mounted by any other process. I don’t see anything in the troubleshooting section on this problem.
Just to add to this, I can manually mount the redundant volume and write to it. I’m not understanding why Mender is reporting that it can’t write to this volume.
My apologies for the delay as I was trying to figure out a way to get more information.
Have you tried to perform a Standalone installation? That means you copy the artifact to the device (to the data partition for example) and then run mender install /data/path/to/artifact.mender. If you can run that, can you share an strace of that command? You can use our support channel support.northern.tech (or support@mender.io) if you consider these traces includes any sensitive information.
I can confirm I received the trace log by a private chanel and I see there are several files not being able to be open, including some ones related to certificates. Are all these files in the overlay instead of being available in the RO rootfs? Is there any chance you can move them to the RO partition for the sake of testing?