Mender app update with `--software-filesystem=data-partition` still uses rootfs disk space

General Discussions
1

Hi there,

I recently started trying out mender to do remote updates on some industrial x86_64 PCs that have an Nvidia GPU which are meant to serve as on-edge AI compute. I managed to build a custom image based on Ubuntu Server with mender-convert that seems to be working well. I configured the partitions in mender-convert to the following sizes:

2025-10-23 07:09:24 [INFO] [mender-convert-package] Total disk size: 122100 MiB
2025-10-23 07:09:24 [INFO] [mender-convert-package]   Boot partition    2048 MiB
2025-10-23 07:09:24 [INFO] [mender-convert-package]   RootFS partitions 19016 MiB x 2
2025-10-23 07:09:24 [INFO] [mender-convert-package]   Data partition    82000 MiB

My idea being that I want to keep the root fs partitions just large enough for the os (with a couple GiBs of headroom) and make the data-partition as large as possible to store my applications and application data so that I can update the underlying os without affecting the applications and data on the device.

I then built a docker compose application artifact with app-gen with --software-filesystem=data-partition and tried loading it onto a test PC from the hosted mender server, which fails with an error No space left on device in the logs. The application is quite large, about 8GiB compressed and 18GiB uncompressed with most of the size coming from one container image, but should fit into the data-partition without issue. I retried and monitored the device storage while it was updating (using watch -n 2 df -h) and saw that the artifact file was being downloaded to the data-partition, then it seemed to be getting decompressed also on the data-partition, all as expected. But finally once that was done, suddenly the root fs partition started filling up the few GiBs it had left, hence the error.

So, what is happening here, why does the application need space on the root fs if the application partition is set to the data-partition? Is my idea to make the root fs partitions tight and make the data-partition as large as possible valid? Do i just need some more headroom on the root fs partitions for some intermediate update operation, is 5GiB not enough?

Thanks in advance!

2

After retrying a deployment on a machine with 2 larger rootfs partitions it seemed to get into the update process further than before. If I had to guess, it looks like the artifact file is being downloaded to and decompressed in the data-partition, after which the images are transferred to into dockers image storage in /var/lib/docker hence doubling the storage needs during the update. I have a 128GB disk in the system and it is almost getting saturated with this one update. Its using something like 40GB for everything (22GB on my data-partition which is 98% use and ±20GB from my rootfs partition which has 14GB of os files and packages which was somewhere >80% use).

3

Hi @JonathanC,

To me that sounds like you’re really done everything right. As far as I can tell, yes, there will be a caching area needed for unpacking, but to my understanding it does not necessarily have to be root nor data partitions.

Oh and just seeing your update, you point out the writes to /var/lib/docker. Do you have that factored out into a separate partition, or something like that?

I think the main issue is containers not being able to be stream-updated but requiring a scratch area for unpacking. Do you have a strategy where to put it? If so, we can work on implementing that.

Greetz,
Josef

4

Hi @TheYoctoJester,

Thanks for your reply.

The comment about /var/lib/docker is just a guess. I thought maybe docker would want to store the images in its regular spot but it is hard to tell because everything is deleted the moment something fails in the update. I also just noticed that the logs refer to images with paths like /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9, i.e. in /var/lib/mender, which is in the rootfs, right. I am doing nothing special regarding partitions, I just built an image with mender-convert starting from a disk image with a standard ubuntu server 22.04 installation with a couple of extra packages installed, with just 1 partition.

Here is my latest log file if it could help. I think I got past the out of storage issue now by increasing the rootfs partition size to 45GB (at the cost of my data-partition size which is now only 22GB, reaching 99% use during the update, so lucky guess maybe?). I am now failing on some other error I think, which I don’t quite understand (maybe better for another thread). But you can still see the paths it is referring to:

"timestamp":"2025-10-24T08:08:39.273948Z","level":"info","message":"Running Mender client 5.0.1"}
{"timestamp":"2025-10-24T08:08:39.274101Z","level":"info","message":"Deployment with ID 672baa6d-8c91-4900-a72c-fd054679cc35 started."}
{"timestamp":"2025-10-24T08:08:39.274494Z","level":"info","message":"Sending status update to server"}
{"timestamp":"2025-10-24T08:08:39.750629Z","level":"info","message":"Installing artifact..."}
{"timestamp":"2025-10-24T08:12:35.987163Z","level":"info","message":"Sending status update to server"}
{"timestamp":"2025-10-24T08:12:40.311087Z","level":"info","message":"Update Module output (stderr): warning: cannot find command 'xdelta3' in PATH."}
{"timestamp":"2025-10-24T08:12:48.168578Z","level":"info","message":"Update Module output (stdout): decompressing images"}
{"timestamp":"2025-10-24T08:12:48.173567Z","level":"info","message":"Update Module output (stdout): images/"}
{"timestamp":"2025-10-24T08:12:48.173646Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/"}
{"timestamp":"2025-10-24T08:12:48.173673Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/url-new.txt"}
{"timestamp":"2025-10-24T08:12:48.173693Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/sums-current.txt"}
{"timestamp":"2025-10-24T08:12:48.173721Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/sums-new.txt"}
{"timestamp":"2025-10-24T08:12:48.173740Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/image.img"}
{"timestamp":"2025-10-24T08:12:49.995585Z","level":"info","message":"Update Module output (stdout): images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9/url-current.txt"}
{"timestamp":"2025-10-24T08:12:49.995640Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/"}
{"timestamp":"2025-10-24T08:12:49.995650Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/url-new.txt"}
{"timestamp":"2025-10-24T08:12:49.995659Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/sums-current.txt"}
{"timestamp":"2025-10-24T08:12:49.995666Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/sums-new.txt"}
{"timestamp":"2025-10-24T08:12:49.995674Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/image.img"}
{"timestamp":"2025-10-24T08:12:55.754828Z","level":"info","message":"Update Module output (stdout): images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb/url-current.txt"}
{"timestamp":"2025-10-24T08:12:55.754891Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/"}
{"timestamp":"2025-10-24T08:12:55.754902Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/url-new.txt"}
{"timestamp":"2025-10-24T08:12:55.754910Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/sums-current.txt"}
{"timestamp":"2025-10-24T08:12:55.754917Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/sums-new.txt"}
{"timestamp":"2025-10-24T08:12:55.754925Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/image.img"}
{"timestamp":"2025-10-24T08:12:57.355459Z","level":"info","message":"Update Module output (stdout): images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2/url-current.txt"}
{"timestamp":"2025-10-24T08:12:57.355497Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/"}
{"timestamp":"2025-10-24T08:12:57.355529Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/url-new.txt"}
{"timestamp":"2025-10-24T08:12:57.355538Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/sums-current.txt"}
{"timestamp":"2025-10-24T08:12:57.355553Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/sums-new.txt"}
{"timestamp":"2025-10-24T08:12:57.355561Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/image.img"}
{"timestamp":"2025-10-24T08:12:57.922213Z","level":"info","message":"Update Module output (stdout): images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160/url-current.txt"}
{"timestamp":"2025-10-24T08:12:57.922241Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/"}
{"timestamp":"2025-10-24T08:12:57.922279Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/url-new.txt"}
{"timestamp":"2025-10-24T08:12:57.922291Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/sums-current.txt"}
{"timestamp":"2025-10-24T08:12:57.922305Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/sums-new.txt"}
{"timestamp":"2025-10-24T08:12:57.922318Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/image.img"}
{"timestamp":"2025-10-24T08:16:17.014775Z","level":"info","message":"Update Module output (stdout): images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6/url-current.txt"}
{"timestamp":"2025-10-24T08:16:17.015089Z","level":"info","message":"Update Module output (stdout): decompressing manifests"}
{"timestamp":"2025-10-24T08:16:17.018842Z","level":"info","message":"Update Module output (stdout): manifests/"}
{"timestamp":"2025-10-24T08:16:17.018931Z","level":"info","message":"Update Module output (stdout): manifests/compose.production.yaml"}
{"timestamp":"2025-10-24T08:16:17.019015Z","level":"info","message":"Update Module output (stdout): manifests/ray-worker.compose.production.yaml"}
{"timestamp":"2025-10-24T08:16:17.019078Z","level":"info","message":"Update Module output (stdout): manifests/compose.base.yaml"}
{"timestamp":"2025-10-24T08:16:17.019152Z","level":"info","message":"Update Module output (stdout): manifests/ray-worker.compose.staging.yaml"}
{"timestamp":"2025-10-24T08:16:17.019215Z","level":"info","message":"Update Module output (stdout): manifests/ray-worker.compose.base.yaml"}
{"timestamp":"2025-10-24T08:16:17.019271Z","level":"info","message":"Update Module output (stdout): manifests/compose.staging.yaml"}
{"timestamp":"2025-10-24T08:16:17.019334Z","level":"info","message":"Update Module output (stdout): manifests/compose.edge.yaml"}
{"timestamp":"2025-10-24T08:16:17.019393Z","level":"info","message":"Update Module output (stdout): manifests/.env"}
{"timestamp":"2025-10-24T08:16:17.019743Z","level":"info","message":"Update Module output (stdout): unpacking images"}
{"timestamp":"2025-10-24T08:16:17.019783Z","level":"info","message":"Update Module output (stdout): unpacking /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb"}
{"timestamp":"2025-10-24T08:16:17.026687Z","level":"info","message":"Update Module output (stdout): unpacking /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6"}
{"timestamp":"2025-10-24T08:16:17.035271Z","level":"info","message":"Update Module output (stdout): unpacking /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9"}
{"timestamp":"2025-10-24T08:16:17.043508Z","level":"info","message":"Update Module output (stdout): unpacking /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160"}
{"timestamp":"2025-10-24T08:16:17.051991Z","level":"info","message":"Update Module output (stdout): unpacking /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2"}
{"timestamp":"2025-10-24T08:16:17.060245Z","level":"info","message":"Update Module output (stdout): no previous composition found"}
{"timestamp":"2025-10-24T08:16:17.062553Z","level":"info","message":"Update Module output (stdout): mkdir: created directory '/data/mender-app'"}
{"timestamp":"2025-10-24T08:16:17.062670Z","level":"info","message":"Update Module output (stdout): mkdir: created directory '/data/mender-app/vision_v1-31-11'"}
{"timestamp":"2025-10-24T08:16:17.062745Z","level":"info","message":"Update Module output (stdout): mkdir: created directory '/data/mender-app/vision_v1-31-11/images'"}
{"timestamp":"2025-10-24T08:16:17.064626Z","level":"info","message":"Update Module output (stdout): scanning /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb"}
{"timestamp":"2025-10-24T08:18:09.420422Z","level":"info","message":"Update Module output (stdout): Loaded image: ghcr.io/robojob/vision-camera-server:edge-1.31.11"}
{"timestamp":"2025-10-24T08:18:09.674798Z","level":"info","message":"Update Module output (stdout): scanning /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6"}
{"timestamp":"2025-10-24T08:29:35.359505Z","level":"info","message":"Update Module output (stdout): Loaded image: ghcr.io/robojob/vision-backend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:29:36.822044Z","level":"info","message":"Update Module output (stdout): scanning /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9"}
{"timestamp":"2025-10-24T08:30:07.227801Z","level":"info","message":"Update Module output (stdout): Loaded image: quay.io/minio/minio:latest"}
{"timestamp":"2025-10-24T08:30:13.191918Z","level":"info","message":"Update Module output (stdout): Loaded image ID: sha256:69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9"}
{"timestamp":"2025-10-24T08:30:13.192068Z","level":"info","message":"Update Module output (stdout): Loaded image ID: sha256:c4260bcf2c25b683e4f6e6ed6474f12c5c28dc8e375bfe0ecea7a5d3fbecbc04"}
{"timestamp":"2025-10-24T08:30:13.225704Z","level":"info","message":"Update Module output (stdout): scanning /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160"}
{"timestamp":"2025-10-24T08:30:16.013304Z","level":"info","message":"Update Module output (stdout): Loaded image: ghcr.io/robojob/vision-frontend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:16.027375Z","level":"info","message":"Update Module output (stdout): scanning /var/lib/mender/modules/v3/payloads/0000/tree/tmp/images/a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2"}
{"timestamp":"2025-10-24T08:30:20.090283Z","level":"info","message":"Update Module output (stdout): Loaded image: postgres:alpine"}
{"timestamp":"2025-10-24T08:30:20.120115Z","level":"info","message":"Update Module output (stdout): -previous composition not present; nothing to stop."}
{"timestamp":"2025-10-24T08:30:20.124780Z","level":"info","message":"Update Module output (stdout): renamed '/var/lib/mender/modules/v3/payloads/0000/tree/tmp/manifests' -> '/data/mender-app/vision_v1-31-11/manifests'"}
{"timestamp":"2025-10-24T08:30:20.124872Z","level":"info","message":"Update Module output (stdout): rolling out /data/mender-app/vision_v1-31-11/manifests"}
{"timestamp":"2025-10-24T08:30:20.358748Z","level":"info","message":"Update Module output (stdout): app_rollout failed to start composition, logs follow:"}
{"timestamp":"2025-10-24T08:30:20.360097Z","level":"info","message":"Update Module output (stdout): stat /data/mender/modules/v3/payloads/0000/tree/compose.base.yaml: no such file or directory"}
{"timestamp":"2025-10-24T08:30:20.458896Z","level":"info","message":"Update Module output (stdout): unsuccessful rollout"}
{"timestamp":"2025-10-24T08:30:20.469024Z","level":"info","message":"Update Module output (stdout): docker-compose:running:docker image rm ghcr.io/robojob/vision-camera-server:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:20.883532Z","level":"info","message":"Update Module output (stdout): Untagged: ghcr.io/robojob/vision-camera-server:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:20.883583Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:1af03bc9cce79f4f4fbc3a5a365de40d6af405f60721b10d514eab767f2cfafb"}
{"timestamp":"2025-10-24T08:30:20.883594Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:9e5619e7e986391f754124a2e83283304c7ada1cffba9240a5229c5e1b66f9f8"}
{"timestamp":"2025-10-24T08:30:20.883601Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:090688d4d30d2cff74b71dd6d4d8cfcf9c2d56355f6f823bee16aee17124c391"}
{"timestamp":"2025-10-24T08:30:20.883608Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:2bf04c48747087197b721b7c19ee3d06c0c2c7d42734b78459d747d829003b03"}
{"timestamp":"2025-10-24T08:30:20.883615Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:d41c306d66943484fda10eb77bc19b659ff819b98faad7c8f03d1fb8aeb5d2c5"}
{"timestamp":"2025-10-24T08:30:20.883621Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:c938d930f74763b779630b4f71b6664abde9ea76b1f34fa698c6ef129322fbd8"}
{"timestamp":"2025-10-24T08:30:20.883627Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:b6229e4ecf278947f934d08ead2e383833aed16b612aa4f30528c3bce972ee7f"}
{"timestamp":"2025-10-24T08:30:20.883634Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:af3ff23c17a00cb64ba8575a529307f99e486265b24dc298510316e40dfd4e50"}
{"timestamp":"2025-10-24T08:30:20.883640Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:fa40d4f8c19bdcf76301eaacd346893b203370477a5f3878a27bf01b1e1aa1d3"}
{"timestamp":"2025-10-24T08:30:20.883646Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:2765d53f04e4febc7ac390d6c7860d9055aac38abbb91152f0193f03ef29760b"}
{"timestamp":"2025-10-24T08:30:20.883652Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:a2e000e2c9dd0f14ec5a577cc85bbde1be9c0b33d9b829c861848632319add09"}
{"timestamp":"2025-10-24T08:30:20.883658Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:9bc556f6344bbd91a884418183a2dc910c55d8b8607e6415fc9c4907c73b6cfd"}
{"timestamp":"2025-10-24T08:30:20.883665Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:f9f52dc133e2af9188960e5a5165cafaa51657ef740ff20219e45a561d78c591"}
{"timestamp":"2025-10-24T08:30:20.885076Z","level":"info","message":"Update Module output (stdout): docker-compose:running:docker image rm ghcr.io/robojob/vision-backend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:23.219770Z","level":"info","message":"Update Module output (stdout): Untagged: ghcr.io/robojob/vision-backend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:23.219827Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:3139e0fa08dc97ecad72f5e562210d297f08c43ee89ba0aa18f67f85c2e5f9a6"}
{"timestamp":"2025-10-24T08:30:23.219843Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:b4510747b721378be6e5c8bca0b57cef9f2a0d9dabe2fc7dd7cde9db012ee10a"}
{"timestamp":"2025-10-24T08:30:23.219858Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:86afe76d1efba88f28d1698f89617e2c63b8a811872029d8dcb9ed52a0ae91fc"}
{"timestamp":"2025-10-24T08:30:23.219864Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:c3d2deb63cf61c111d8232cf0763d809db883aea97865d65034171dfdb3ba267"}
{"timestamp":"2025-10-24T08:30:23.219870Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:0da3745ef788f4bf2c938108953f82cd1c21431c03de7518bcf15046da5afe7b"}
{"timestamp":"2025-10-24T08:30:23.219876Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:9a70770d6d2e5544ca3ae7b9d4be83e3e12a147ba693750c0c03f428fd2f5163"}
{"timestamp":"2025-10-24T08:30:23.219882Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:be855aa2f2095138e3a01399fcb7c6fe8a722974429aa9fff7e7fd7b7ca2d943"}
{"timestamp":"2025-10-24T08:30:23.219888Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:9854518acdbcd896b1d89cb89278afbd1f7c76ffffc843387f7cd1598a28baef"}
{"timestamp":"2025-10-24T08:30:23.219893Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:24b6ed9ebbc7c624237729d756d60f65a0922f60a72d27e4fc0803a883f6a616"}
{"timestamp":"2025-10-24T08:30:23.219899Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:64d26b83b0899d334ee0381298b502c66bb183c1db0b88688bb6670980975520"}
{"timestamp":"2025-10-24T08:30:23.219905Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:85bcef062da74e5e7d8376b3adbebe9e2b68899c3d1059cb7f3e2178610829ce"}
{"timestamp":"2025-10-24T08:30:23.219911Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:93e9ade8b863ea84f1219451b941df75bb52ee798a2765dcfd618a956e3a74b0"}
{"timestamp":"2025-10-24T08:30:23.219917Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:dca814babeebcb04abb423d5554965b9b756b889e1a9b68e45659254492e4aff"}
{"timestamp":"2025-10-24T08:30:23.219926Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:64ec0a729283ce7025ff74a5ddacebf47927b10907c3165d14000c9aafc24d6b"}
{"timestamp":"2025-10-24T08:30:23.219932Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:6ab9992ff9b12a5a7c70b8d6ceb9457e827b5515fd683619902b0848c4e2d6dd"}
{"timestamp":"2025-10-24T08:30:23.219938Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:d92cb86d3a51c3b9c004ac129912c2bc9a311c5e00adf8c4e6f25b1fb4c26053"}
{"timestamp":"2025-10-24T08:30:23.219944Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:9d5b488d2a3ea0d19a53e947c36880d3ec6de7edddb48021337a7a4553704478"}
{"timestamp":"2025-10-24T08:30:23.219949Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:b7ea06e04070a8fe511f850085357d22f1f8474e7f7add186d1ae48328525ad9"}
{"timestamp":"2025-10-24T08:30:23.219955Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:b1ce7f1f9b98d5fcbbdae47a4662ff2d77f4936c6ac20737023987c2bca20d11"}
{"timestamp":"2025-10-24T08:30:23.219961Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:e2703b7acd9f58d705cd950c3548a0290554375297836a4acc36c14e68ea1b0f"}
{"timestamp":"2025-10-24T08:30:23.219967Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:8901a649dd5a9284fa6206a08f3ba3b5a12fddbfd2f82c880e68cdb699d98bfb"}
{"timestamp":"2025-10-24T08:30:23.221335Z","level":"info","message":"Update Module output (stdout): docker-compose:running:docker image rm quay.io/minio/minio"}
{"timestamp":"2025-10-24T08:30:24.476830Z","level":"info","message":"Update Module output (stdout): Untagged: quay.io/minio/minio:latest"}
{"timestamp":"2025-10-24T08:30:24.476886Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:69b2ec208575b69597784255eec6fa6a2985ee9e1a47f4411a51f7f5fdd193a9"}
{"timestamp":"2025-10-24T08:30:24.476896Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:a0552d43cfab9a7f10063fe41f9cc4703d5bcf6adb0874fa4c74801a61ea289d"}
{"timestamp":"2025-10-24T08:30:24.476911Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:f787055c69a783c56fc7fd6eee6b6a11751c8f051c4fa35b2e63b6e2072280b2"}
{"timestamp":"2025-10-24T08:30:24.476918Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:81d33750addd68630dc074050cdde0f3ed343c62a2821d6b794168712b0711f6"}
{"timestamp":"2025-10-24T08:30:24.476924Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:bd8b308525b683d9152a2331b4d8ec93ca416b3de47373fbcddf2358eedfb5e2"}
{"timestamp":"2025-10-24T08:30:24.476930Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:dd23a8346735b2d4c87a1dcb68810701c48fc781b6070ed4a2ab99680226c317"}
{"timestamp":"2025-10-24T08:30:24.476936Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:55fd7bad3693d9d5225866287000a52fe054e411be78264cc16024fc61f4c52f"}
{"timestamp":"2025-10-24T08:30:24.476943Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:d9bbf03eaebf2103740fbaadc92d91ccd1f1f82bec997f4247ef9b476ce6f089"}
{"timestamp":"2025-10-24T08:30:24.476949Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:7e7d5d898982283697c80bb0631152be74f5e022a13be9d8f56004da501de59c"}
{"timestamp":"2025-10-24T08:30:24.476955Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:702bea3dc94887d2fcf1aa0d56565c57d5ca6d24659c8d0b6cbde5c58d6c182f"}
{"timestamp":"2025-10-24T08:30:24.478337Z","level":"info","message":"Update Module output (stdout): docker-compose:running:docker image rm ghcr.io/robojob/vision-frontend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:24.554132Z","level":"info","message":"Update Module output (stdout): Untagged: ghcr.io/robojob/vision-frontend:edge-1.31.11"}
{"timestamp":"2025-10-24T08:30:24.554172Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:73569992d691ffed0050e48182ae5ae109d0fb0924997edf18b3131d4c6ac160"}
{"timestamp":"2025-10-24T08:30:24.554180Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:dd9ce6a5d0a26e9bd790fdfa7585a1fc30a3464dbcb15d98240577356a7e3c4e"}
{"timestamp":"2025-10-24T08:30:24.554187Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:aa49eceb39f495ee82500fcc12644ec73610be9aa175033961583ada6554ad13"}
{"timestamp":"2025-10-24T08:30:24.554193Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:45c0b1fe3bae67bdf5dc1f6b4e356c0ea038d9ee714d751e104bab41b2c59ff3"}
{"timestamp":"2025-10-24T08:30:24.554199Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:a2f38d419d9ee50af839974f2e5e01dfb1c433c030aba18a046dedc4ab86ab5d"}
{"timestamp":"2025-10-24T08:30:24.554205Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:cf0cca8471b754e780117770f46357addb7466d757ef05610bb71ca938b7e2a6"}
{"timestamp":"2025-10-24T08:30:24.554212Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:ebfb51712bdd4a399146007ad76b5028ba70135cbc8090159912b9eb7fae2923"}
{"timestamp":"2025-10-24T08:30:24.555397Z","level":"info","message":"Update Module output (stdout): docker-compose:running:docker image rm docker.io/postgres:alpine"}
{"timestamp":"2025-10-24T08:30:24.658418Z","level":"info","message":"Update Module output (stdout): Untagged: postgres:alpine"}
{"timestamp":"2025-10-24T08:30:24.658462Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:a0f6316c2982c19f3cfc62aa56d5c6171e2a2afc2d6c3ec67c0a6e061efa41c2"}
{"timestamp":"2025-10-24T08:30:24.658472Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:b7242509c504c76ff91027f0f08f2033e8260b9ff3f864fee5b03d9696aa42dd"}
{"timestamp":"2025-10-24T08:30:24.658479Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:0dd6721566d60c0cee8f996df8d1de8a069657c5625b93770da2bce292b92899"}
{"timestamp":"2025-10-24T08:30:24.658486Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:2646db817981f1742a7f6ecdf75ded7248d3191989503a3f808eecc2a58878c0"}
{"timestamp":"2025-10-24T08:30:24.658497Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:855db1998e44222af11f698e35733b47e77d824bfd6f7489ef008c13e1636850"}
{"timestamp":"2025-10-24T08:30:24.658504Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:74cbaa9c28a1f8ff3795c3c4eea2ada0e47549a52680ff316388912a9529dac9"}
{"timestamp":"2025-10-24T08:30:24.658510Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:20c0a857bf7b80beb87332d1fb50a09a0f20ae4b77c499fc5dc7a55538a0fe2d"}
{"timestamp":"2025-10-24T08:30:24.658516Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:728d7152c545e21a376e972fbb5d4b8079db45edad78d96f931bb7569e4f46ce"}
{"timestamp":"2025-10-24T08:30:24.658522Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:85829c0f50c1d786dc1dc5b14bd106b9f5c00abd65906f37f5e8a424189ebaca"}
{"timestamp":"2025-10-24T08:30:24.658528Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:0ebbeb8b3c07a796e33db96000cbc8a379690d9a0f74dd14902b1ff56dcd3ff1"}
{"timestamp":"2025-10-24T08:30:24.658534Z","level":"info","message":"Update Module output (stdout): Deleted: sha256:418dccb7d85a63a6aa574439840f7a6fa6fd2321b3e2394568a317735e867d35"}
{"timestamp":"2025-10-24T08:30:24.927905Z","level":"info","message":"Update Module output (stderr): app_rollout failed to start composition, logs follow:"}
{"timestamp":"2025-10-24T08:30:24.928325Z","level":"info","message":"Update Module output (stderr): stat /data/mender/modules/v3/payloads/0000/tree/compose.base.yaml: no such file or directory"}
{"timestamp":"2025-10-24T08:30:25.014730Z","level":"info","message":"Update Module output (stdout): unsuccessful rollback trying to rollout the composition we saw when we started"}
{"timestamp":"2025-10-24T08:30:25.016697Z","level":"info","message":"Update Module output (stderr): /usr/share/mender/app-modules/v1/docker-compose: 243: cannot create /data/mender-app/vision_v1-31-11-previous/manifests/compose.log: Directory nonexistent"}
{"timestamp":"2025-10-24T08:30:25.016910Z","level":"info","message":"Update Module output (stderr): app_rollout failed to start composition, logs follow:"}
{"timestamp":"2025-10-24T08:30:25.018278Z","level":"info","message":"Update Module output (stderr): cat: /data/mender-app/vision_v1-31-11-previous/manifests/compose.log{...}"}
{"timestamp":"2025-10-24T08:30:25.018462Z","level":"info","message":"Update Module output (stderr): : No such file or directory"}
{"timestamp":"2025-10-24T08:30:25.259402Z","level":"info","message":"Update Module output (stdout): rollout of /data/mender-app/vision_v1-31-11-previous was unsuccessful; deployment failed; rollout attempts failed."}
{"timestamp":"2025-10-24T08:30:25.259861Z","level":"error","message":"Process returned non-zero exit status: ArtifactInstall: Process exited with status 84"}
{"timestamp":"2025-10-24T08:30:25.948811Z","level":"info","message":"Sending status update to server"}
{"timestamp":"2025-10-24T08:30:26.323232Z","level":"info","message":"Deployment with ID 672baa6d-8c91-4900-a72c-fd054679cc35 finished with status: Failure"}
5

Something else completely, I just noticed something that might explain why my artifact is so big. It seems that app-gen is including my images twice. I just saw that it pulls each image twice in the logs and then logs the image SHAs also duplicated (SHAs truncated):

declare -a images_shas=([0]="8d13d45a316c3f..." [1]="8d13d45a316c3f..." [2]="212fe9c9342..." [3]="212fe9c934230..." [4]="bf60e9fdc1a39..." [5]="bf60e9fdc1a396..." [6]="3f4989ad6f4a9d8859..." [7]="3f4989ad6f4a9d8859..." [8]="69b2ec208575b6..." [9]="69b2ec208575...")
6

Hi @JonathanC,
I was recently confused by the same, so let me share some tips and some of my findings.

The most important one is that the app-update-module runs docker load on the images from the artifact. It also saves/stores them so that previous versions are available for rollback and such things, but for running the composition, the module runs docker compose up which requires the images to be present in the Docker’s local registry, hence docker load on them.

Also, /var/lib/mender is a symlink to /data/mender in the converted images. So whatever is stored in there is actually on the /data partition and not the root partition.

And finally, --software-filesystem is merely metadata that helps Mender to determine which software (past artifacts) are removed during an update. It has no impact on where things are installed or stored, it only documents that.

The real trick that you need, I believe, is to configure docker to use something like /data/docker for the local repository. Then, all images should be in /data. We are currently working on a new Mender Update Module focusing on docker-compose which doesn’t store the images separately. You can check it out. However, the main trick remains – Docker has to be configured to have its local registry (for loaded or pulled images) outside of the root filesystem. While the Update Module could come with a custom Docker config, we find it too intrusive and potentially dangerous so this step will most likely always be on the users. Documented in a better way, though, of course!

Hope this helps.

Vratislav (Mender Client Team)

1 Like
7

Hi @vpodzime,

Thanks for your reply, this helped me figure it out. I now have tightly sized root partitions, thus optimizing for the size of my data partition. I then configured docker to use /data/docker as its storage directory by adding "data-root": "/data/docker" to /etc/docker/daemon.json (with an overlay with mender-convert). Now my storage is being utilized as I wanted.

Thanks again!

1 Like