Mender deployment using delta reports "Reached size of the destination volume, source too big."

General Discussions
1

I am currently on the trial plan to integrate our project with hosted Mender to use delta upgrades.

  • Our project targets Pi4 (and Pi3), using Yocto kirkstone and read-only rootfs.
  • mender-client version 3.5.3
  • mender-binary-delta version 1.5.0
  • The storage is partitioned into 2 x 7.3GB rootfs partitions and a 15G data partition. We actually only use about 776MB of space on the rootfs according to df.
Device         Boot    Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *       24576   106495    81920   40M  c W95 FAT32 (LBA)
/dev/mmcblk0p2        106496 15409151 15302656  7.3G 83 Linux
/dev/mmcblk0p3      15409152 30711807 15302656  7.3G 83 Linux
/dev/mmcblk0p4      30711808 62333951 31622144 15.1G 83 Linux

I have created a rootfs artifact versioned 7.7.93-83802 (252MB)

The rootfs artifact was flashed to the target devices. Since I’m coming from a pre-kirkstone era, I also did a mender install of the bootstrap artifact.

I made some changes to our source and made a new artifact versioned 7.7.93-84332. (252MB)

I generated a delta artifact using

./mender-binary-delta-generator -o v7.7.93.84332-deltafrom-v7.7.93.83802.mender sf-rpi-image-7.7.93.83802-raspberrypi4.mender sf-rpi-image-7.7.93.84332-raspberrypi4.mender

This created an artifact of 1017KB. I uploaded this to hosted.mender.io.

I created a deployment to install this delta to the target devices. The deployment started fine, downloading the artifact, but after a couple of minutes the deployment failed. I can post the full logs, however, the most log-worthy is:

2024-07-26 02:39:05 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Running flash tool with args: mender-flash -i - -o /dev/mmcblk0p2 -s 7834959872
2024-07-26 02:42:11 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": running with decoder args: xdelta3 -9 -f -d -c -s /dev/mmcblk0p3 streams/sf-rpi-image-7.7.93.84332-raspberrypi4.ext4.delta
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Input/output error: Reached size of the destination volume, source too big.
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": xdelta3: write failed: /dev/stdout: Broken pipe
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": Failed to apply the delta, err: 1
2024-07-26 02:42:12 +0000 UTC info: Output (stderr) from command "/usr/share/mender/modules/v3/mender-binary-delta": FlashTool returned error: 1

I do not see how the source could be too big since mmcblk0p2 has exactly the same number of sectors as mmcblk0p3.

2

I found this post while searching for this error text. I’m experiencing a similar issue when I create a full rootfs snapshot artifact using mender-artifact.

  • mender-convert version 4.2.2
  • mender-artifact version 3.11.2
  • mender-update version 4.0.2

I’m essentially following the same steps from the documentation: Deploy an Operating System Update or Create an Artifact with system snapshot, but using a customized Raspberry Pi OS (32-bit bookworm) image that was converted using mender-convert. This image boots up the board perfectly (as far as I can tell). To test the rootfs update mechanism, I’m creating snapshots that have small but obvious modifications using:

mender-artifact write rootfs-image \
    -f ssh://"${USER}@${ADDR}" \
    -n "${DEVICE_TYPE}_${VERSION}" \
    -t "${DEVICE_TYPE}" \
    -o "${DEVICE_TYPE}_${VERSION}.mender"

This appears to create appropriate artifacts which I have uploaded to hosted Mender and the Release is created with no apparent issue. When I attempt to run any Deployment using these Releases back to the same board, the Deployment fails with a similar error (partial logs below):

2024-08-19 16:21:13.768 +0000 UTC info: Sending status update to server
2024-08-19 16:21:14.481 +0000 UTC info: Installing artifact...
2024-08-19 16:23:28.646 +0000 UTC info: Update Module output (stderr): Input/output error: Reached size of the destination volume, source too big.
2024-08-19 16:23:28.821 +0000 UTC error: Broken pipe: AsyncWrite failed
2024-08-19 16:23:28.822 +0000 UTC info: Sending SIGTERM to PID 3164
2024-08-19 16:23:28.822 +0000 UTC info: PID 3164 exited with status 1
2024-08-19 16:23:28.881 +0000 UTC info: Sending status update to server

At first I assumed that the Mender client requires some non-trivial disk space overhead to handle full rootfs artifacts. I tried adjusting my mender-convert partition scheme a few times to distribute more free space to the rootfs partitions or data partition, with no change.

3

@scottbragg If you want my non-expert, possibly ill-informed suggestion, perhaps try substantially shrinking the rootfs partitions (by about half perhaps) and see if that makes any difference with deploying delta updates, even if just as an experiment.

I’ve been doing more experimentation with my use case, and the trend I’m seeing so far is that any time I configure MENDER_STORAGE_TOTAL_SIZE_MB to anything greater than 8192 (the default in mender-convert), snapshot artifacts created from and deployed to a Raspberry Pi device using that converted image no longer works with the error “Reached size of the destination volume, source too big.” Values less than or equal to 8192 seem to work as expected as long as it’s still large enough to accommodate the input image.

This is unfortunate, because my project’s current “golden” image requires just a bit more than 8192 MB of overall storage space to pass through mender-convert. I really think I’m overlooking something important, but I don’t know what.