Mender deployment using delta reports "Reached size of the destination volume, source too big."

General Discussions
1

I am currently on the trial plan to integrate our project with hosted Mender to use delta upgrades.

  • Our project targets Pi4 (and Pi3), using Yocto kirkstone and read-only rootfs.
  • mender-client version 3.5.3
  • mender-binary-delta version 1.5.0
  • The storage is partitioned into 2 x 7.3GB rootfs partitions and a 15G data partition. We actually only use about 776MB of space on the rootfs according to df.
Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       24576   106495    81920   40M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        106496 15409151 15302656  7.3G 83 Linux
/dev/mmcblk0p3      15409152 30711807 15302656  7.3G 83 Linux
/dev/mmcblk0p4      30711808 62333951 31622144 15.1G 83 Linux

I have created a rootfs artifact versioned 7.7.93-83802 (252MB)

The rootfs artifact was flashed to the target devices. Since I’m coming from a pre-kirkstone era, I also did a mender install of the bootstrap artifact.

I made some changes to our source and made a new artifact versioned 7.7.93-84332. (252MB)

I generated a delta artifact using

./mender-binary-delta-generator -o v7.7.93.84332-deltafrom-v7.7.93.83802.mender sf-rpi-image-7.7.93.83802-raspberrypi4.mender sf-rpi-image-7.7.93.84332-raspberrypi4.mender

This created an artifact of 1017KB. I uploaded this to hosted.mender.io.

I created a deployment to install this delta to the target devices. The deployment started fine, downloading the artifact, but after a couple of minutes the deployment failed. I can post the full logs, however, the most log-worthy is:

2024-07-26 02:39:05 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Running flash tool with args: mender-flash -i - -o /dev/mmcblk0p2 -s 7834959872
2024-07-26 02:42:11 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": running with decoder args: xdelta3 -9 -f -d -c -s /dev/mmcblk0p3 streams/sf-rpi-image-7.7.93.84332-raspberrypi4.ext4.delta
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Input/output error: Reached size of the destination volume, source too big.
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": xdelta3: write failed: /dev/stdout: Broken pipe
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Failed to apply the delta, err: 1
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": FlashTool returned error: 1

I do not see how the source could be too big since mmcblk0p2 has exactly the same number of sectors as mmcblk0p3.

2

I found this post while searching for this error text. I’m experiencing a similar issue when I create a full rootfs snapshot artifact using mender-artifact.

  • mender-convert version 4.2.2
  • mender-artifact version 3.11.2
  • mender-update version 4.0.2

I’m essentially following the same steps from the documentation: Deploy an Operating System Update or Create an Artifact with system snapshot, but using a customized Raspberry Pi OS (32-bit bookworm) image that was converted using mender-convert. This image boots up the board perfectly (as far as I can tell). To test the rootfs update mechanism, I’m creating snapshots that have small but obvious modifications using:

mender-artifact write rootfs-image \
    -f ssh://"${USER}@${ADDR}" \
    -n "${DEVICE_TYPE}_${VERSION}" \
    -t "${DEVICE_TYPE}" \
    -o "${DEVICE_TYPE}_${VERSION}.mender"

This appears to create appropriate artifacts which I have uploaded to hosted Mender and the Release is created with no apparent issue. When I attempt to run any Deployment using these Releases back to the same board, the Deployment fails with a similar error (partial logs below):

2024-08-19 16:21:13.768 +0000 UTC info: Sending status update to server
2024-08-19 16:21:14.481 +0000 UTC info: Installing artifact...
2024-08-19 16:23:28.646 +0000 UTC info: Update Module output (stderr): Input/output error: Reached size of the destination volume, source too big.
2024-08-19 16:23:28.821 +0000 UTC error: Broken pipe: AsyncWrite failed
2024-08-19 16:23:28.822 +0000 UTC info: Sending SIGTERM to PID 3164
2024-08-19 16:23:28.822 +0000 UTC info: PID 3164 exited with status 1
2024-08-19 16:23:28.881 +0000 UTC info: Sending status update to server

At first I assumed that the Mender client requires some non-trivial disk space overhead to handle full rootfs artifacts. I tried adjusting my mender-convert partition scheme a few times to distribute more free space to the rootfs partitions or data partition, with no change.