[FATAL] [mender-convert-package] You can try adjusting the MENDER_STORAGE_TOTAL_SIZE_MB variable to increase available space, or

General Discussions
1

Hello All:

I have been building mender-artifacts for some time with help from tutorials and Mender support team.

For my raspberry pi image I have been using 64GB microSD card. I have been adding new features and necessary packages.

Now I am getting the error:

[FATAL] [mender-convert-package] You can try adjusting the MENDER_STORAGE_TOTAL_SIZE_MB variable to increase available space, or modify one of the variables IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_EXTRA_SPACE or IMAGE_OVERHEAD_FACTOR to reduce the size of the root filesystem.

Attached below is the full log file of my backup and mender-convert steps.

Eventually I will need to use 128GB microSD card. For now, how can I modify the parameters so that I can continue with 64GB microSD card (because removing the SD card is not possible for the current set of devices).

For now I am not using the data partition for anything unless it is being used for the mender itself.

Before I last-resort to remove some features and packages. I am thinking if I can fine tune the mender config parameters, but afraid of messing-up my mender-convert environment without fully understanding it.

Any guidance or help optimizing/fine tuning the parameters would be greatly appreciated.

Thanks,
Raj
-------X------------------
rajendra@SmartSquareSystems:~/Downloads/NewRPIimages-Stretch$ newbackuppistretch.sh ā€œRemoved some packages to cleanup. and freeup space in rootfs.ā€ MENDERIT=1
Mon 14 Jun 2021 01:15:23 PM CDT
[sudo] password for rajendra:
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 3.7M 1.6G 1% /run
/dev/sda2 4.6T 3.2T 1.2T 73% /
tmpfs 7.8G 4.3M 7.8G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/loop0 56M 56M 0 100% /snap/core18/2066
/dev/loop4 56M 56M 0 100% /snap/core18/1997
/dev/loop3 162M 162M 0 100% /snap/gnome-3-28-1804/128
/dev/loop2 99M 99M 0 100% /snap/core/11081
/dev/loop1 100M 100M 0 100% /snap/core/11167
/dev/loop5 163M 163M 0 100% /snap/gnome-3-28-1804/145
/dev/loop8 2.3M 2.3M 0 100% /snap/gnome-system-monitor/148
/dev/loop7 219M 219M 0 100% /snap/gnome-3-34-1804/72
/dev/loop6 219M 219M 0 100% /snap/gnome-3-34-1804/66
/dev/loop10 2.3M 2.3M 0 100% /snap/gnome-system-monitor/157
/dev/loop9 65M 65M 0 100% /snap/gtk-common-themes/1514
/dev/loop11 68M 68M 0 100% /snap/lxd/20326
/dev/loop12 5.7M 5.7M 0 100% /snap/notepad-plus-plus/260
/dev/loop13 66M 66M 0 100% /snap/gtk-common-themes/1515
/dev/loop15 51M 51M 0 100% /snap/snap-store/542
/dev/loop20 52M 52M 0 100% /snap/snap-store/518
/dev/loop14 136M 136M 0 100% /snap/skype/176
/dev/loop18 135M 135M 0 100% /snap/skype/173
/dev/loop16 338M 338M 0 100% /snap/wine-platform-runtime/216
/dev/loop17 338M 338M 0 100% /snap/wine-platform-runtime/220
/dev/loop19 304M 304M 0 100% /snap/wine-platform-5-stable/16
/dev/sda1 511M 18M 494M 4% /boot/efi
tmpfs 1.0M 0 1.0M 0% /var/snap/lxd/common/ns
tmpfs 1.6G 4.4M 1.6G 1% /run/user/1000
/dev/sdc2 15G 13G 934M 94% /media/rajendra
Mender Flag is: MENDERIT=1
RELABELED: export VERSION_NAME=NEWSSHP-202106141315.img >> /media/rajendra/home/ssqr/.smartsquare
15226+1 records in
15226+1 records out
63864569856 bytes (64 GB, 59 GiB) copied, 1589.93 s, 40.2 MB/s
Mon 14 Jun 2021 01:46:18 PM CDT
Backup completedā€¦
Moving /home/rajendra/Downloads/NewRPIimages-Stretch//NEWSSHP-202106141315.img to /home/rajendra/mender-convert//input ā€¦
-rw-rw-r-- 1 rajendra rajendra 63 Jun 14 13:15 NEWSSHP-202106141315.img.info
-rw-rw-r-- 1 rajendra rajendra 63864569856 Jun 14 13:46 NEWSSHP-202106141315.img
-rw-rw-r-- 1 rajendra rajendra 63864569856 Jun 14 13:46 /home/rajendra/mender-convert//input/NEWSSHP-202106141315.img
Image is placed in the mender-convert/input directory.
-rw-rw-r-- 1 rajendra rajendra 63864569856 Jun 14 13:46 /home/rajendra/mender-convert//input/NEWSSHP-202106141315.img
Now Gzipping /home/rajendra/mender-convert//input/NEWSSHP-202106141315.img > /home/rajendra/Downloads/NewRPIimages-Stretch//NEWSSHP-202106141315.img.gz ā€¦
cd mender-convert; MENDER_ARTIFACT_NAME=NEWSSHP-202106141315.img ./docker-mender-convert --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
Mon 14 Jun 2021 02:17:46 PM CDT
Mender Flag is: 1
To Menderize this release, run the following command.
Starting To Mendserizing in Background. Using ā€¦
ā€¦
ā€¦
ā€¦
ā€¦
cd mender-convert; MENDER_ARTIFACT_NAME=NEWSSHP-202106141315.img ./docker-mender-convert --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
Mon 14 Jun 2021 02:17:46 PM CDT
[sudo] password for rajendra:
BACKED UP THE RELABELED IMAGE AS: /media/rajendra/NEWSSHP-202106141315.img
umount: /media/rajendra: umount failed: Operation not permitted.
Mon 14 Jun 2021 02:26:03 PM CDT
rajendra@SmartSquareSystems:~/Downloads/NewRPIimages-Stretch$

rajendra@SmartSquareSystems:~/Downloads/NewRPIimages-Stretch$ cd
rajendra@SmartSquareSystems:~$ cd mender-convert; MENDER_ARTIFACT_NAME=NEWSSHP-202106141315.img ./docker-mender-convert --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
Running mender-convert --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
Running mender-convert-extract: --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-14 19:29:12 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2021-06-14 19:29:13 [INFO] [mender-convert-extract] Using configuration file: configs/raspberrypi3_config
2021-06-14 19:29:13 [INFO] [mender-convert-extract] Validating disk image
2021-06-14 19:29:13 [INFO] [mender-convert-extract] Disk parsed successfully
2021-06-14 19:29:13 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: dos
2021-06-14 19:29:13 [INFO] [mender-convert-extract] PART 1: SIZE: 42.9M TYPE: 0xc
2021-06-14 19:29:13 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2021-06-14 19:29:15 [INFO] [mender-convert-extract] PART 2: SIZE: 14.7G TYPE: 0x83
2021-06-14 19:29:15 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning upā€¦
Running mender-convert-modify: --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-14 19:39:30 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2021-06-14 19:39:30 [INFO] [mender-convert-modify] Using configuration file: configs/raspberrypi3_config
2021-06-14 19:39:43 [INFO] [mender-convert-modify] Installing Mender client and related files
2021-06-14 19:39:45 [INFO] [mender-convert-modify] Creating state scripts version file.
2021-06-14 19:39:45 [INFO] [mender-convert-modify] Installing a custom /etc/fstab (see work/convert.log for more info)
2021-06-14 19:39:45 [INFO] [mender-convert-modify] Performing platform specific modifications (if any)
2021-06-14 19:39:45 [INFO] [mender-convert-modify] Running hook: platform_modify
2021-06-14 19:39:50 [INFO] [mender-convert-modify] Certain service files have been changed to align with our /uboot boot partition mount point. See convert.log for more information
2021-06-14 19:39:50 [INFO] [mender-convert-modify] Enabling UART in U-Boot configuration
2021-06-14 19:39:50 [INFO] [mender-convert-modify] Performing user/local specific modifications (if any)
2021-06-14 19:39:50 [INFO] [mender-convert-modify] Running hook: user_local_modify
2021-06-14 19:39:50 [INFO] [mender-convert-modify] Applying rootfs overlay: ./bootstrap-rootfs-overlay-hosted-server
mender-convert-modify has finished. Cleaning upā€¦
Running mender-convert-package: --disk-image input/NEWSSHP-202106141315.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-14 19:39:51 [INFO] [mender-convert-package] Using configuration file: configs/mender_convert_config
2021-06-14 19:39:51 [INFO] [mender-convert-package] Using configuration file: configs/raspberrypi3_config
2021-06-14 19:40:24 [WARN] [mender-convert-package] The calculated rootfs partition size 18904 MiB is too small.
2021-06-14 19:40:24 [WARN] [mender-convert-package] The actual rootfs image size is 19797 MiB
2021-06-14 19:40:24 [FATAL] [mender-convert-package] You can try adjusting the MENDER_STORAGE_TOTAL_SIZE_MB variable to increase available space, or modify one of the variables IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_EXTRA_SPACE or IMAGE_OVERHEAD_FACTOR to reduce the size of the root filesystem.
mender-convert-package has finished. Cleaning upā€¦

rajendra@SmartSquareSystems:~/Downloads/NewRPIimages-Stretch$

-------X------------------

2

The big question here is how big is your input golden image? It looks like it is about 20GB:

2021-06-14 19:40:24 [WARN] [mender-convert-package] The calculated rootfs partition size 18904 MiB is too small.
2021-06-14 19:40:24 [WARN] [mender-convert-package] The actual rootfs image size is 19797 MiB

which is obviously quite large but will certainly fit within a 64GB SDCard. Iā€™d be surprised if you really needed 20GB of stuff in your golden image and even if you can get it to work with the card you are using, the amount of download and time for that large of an image will be significant.

As for the partition sizes, you specify the values MENDER_STORAGE_TOTAL_SIZE_MB, MENDER_BOOT_PART_SIZE_MB, and MENDER_DATA_PART_SIZE_MB and then the root file system partition sizes created by the mender logic will be calculated as:

(MENDER_STORAGE_TOTAL_SIZE_MB - MENDER_BOOT_PART_SIZE_MB - MENDER_DATA_PART_SIZE_MB) / 2

Hope that helps.

Drew

3

Hi Drew:

Thanks for your prompt reply.
To answer your questions:

The golden image is:
-rw-rw-r-- 1 rajendra rajendra 63864569856 Jun 14 13:46 /home/rajendra/mender-convert/input/NEWSSHP-202106141315.img

df -h on the booted image(un-mendered/development unit) is:
(sspy35_env) ssqr@SS-HealthPOD:/home/ssqr/SSquare/sspy35_env $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 13G 934M 94% /
devtmpfs 433M 0 433M 0% /dev
tmpfs 438M 308K 437M 1% /dev/shm
tmpfs 438M 12M 426M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 438M 0 438M 0% /sys/fs/cgroup
/dev/mmcblk0p1 43M 23M 20M 53% /boot
tmpfs 88M 4.0K 88M 1% /run/user/1000

You are right. Too long to download. I need to learn how the delta update for incremental update releases.

For now, how do I modify the configs parameters so that it menders it correctly and fits in 64GB SDcard?

Thanks again,

Raj

4

As well as adjusting you partitions sizes, i recommend that you audit what is consuming so much disk space on your rootfs, as there are usually lots of ways you can reduce the size of the rootfs depending on the target usage of your device.

5

Hi Dell:

There are lot of sudo apt-get packages and python packages installed to support my applications. There are no browser cache or other data from my applications.

Any command you might recommend to audit?

Thanks for your help.

Regards,
Raj

6

I tend to use, ā€˜du -h -d 1 /ā€™ to get a total of each directory in the top level of the rootfs, and then repeat and drill down from there, thereā€™s probably better tools but itā€™s been enough for me.

Areas to check off the top of my head.

  1. disable and remove package-manager caches
  2. remove all unused kernels/initrd/kernel-headers
  3. if using swap file ensure its created dynamically at boot and not a large blob on your golden image file system
  4. if headless device remove any desktop environment packages installed
  5. disable and remove man page installation by packages.
7

You will need to adjust the variables I mentioned above to make sure it all fits. You may also need to set IMAGE_ROOTFS_SIZE to ā€œ-1ā€.
Drew

8

Thanks Dell, Drew:
I ran
(sspy35_env) ssqr@SS-HealthPOD:~ $ sudo du -h -d 1 /
23M /boot
16K /lost+found
0 /proc
164K /dev
354M /var
4.0K /mnt
3.3G /usr
4.0K /srv
4.0K /media
99M /etc
0 /sys
8.0M /sbin
7.5G /home
8.3M /bin
248M /lib
212M /opt
6.3M /run
44M /root
68K /tmp
12G /

I removed old boot, /var/log/log and gz files files:
sudo rm -rf /boot.bak boot1
sudo rm -rf log.gz
sudo rm -rf log..gz
sudo rm -rf log
rm -rf messages
sudo rm -rf messages
rm *.gz
sudo rm -rf *.gz
rm *.log
sudo rm *.log
history | grep rm

Then hereā€™s the final output of du -h -d 1 /
(sspy35_env) ssqr@SS-HealthPOD:/home/ssqr/SSquare/sspy35_env $ sudo du -h -d 1 /
23M /boot
16K /lost+found
0 /proc
44K /dev
354M /var
4.0K /mnt
3.3G /usr
4.0K /srv
4.0K /media
99M /etc
0 /sys
8.0M /sbin
7.5G /home
8.3M /bin
248M /lib
212M /opt
12M /run
44M /root
68K /tmp
12G /
(sspy35_env) ssqr@SS-HealthPOD:/home/ssqr/SSquare/sspy35_env $

And, the image boots! So I didnā€™t butcher it to its death.

Is there a utility to do such cleanup to run as part of the mender-convert?

What values should I configure the mender parameters with?

Greatly appreciate your guidance.

Regards,
Raj

9

You could potentially add a hook function.

But Iā€™m most concerned now about 7.6G in /home. What is taking up so much space there? Do you really want that? If that is required for your app, is it something that should rather be in the /data partition so it is not updated over the air?
Drew

1 Like
10

Thanks Drew, Dell:

I removed 6.5G /home/ssqr/tmp_cv3 that I used for build and install.
I tested the application after removing it.

Now Iā€™m back on track. Mender-convert and the transfers should be lot faster.
I still need to do some more cleanups and automate it.

Thanks,
Raj
(sspy35_env) ssqr@SS-HealthPOD:~ $ sudo du -h -d 1 /home/ssqr
56K /home/ssqr/.ssdb
40K /home/ssqr/.mono
8.0K /home/ssqr/.gnupg
4.0K /home/ssqr/Desktop
68K /home/ssqr/opencv3-setup
3.2M /home/ssqr/imagezmq
6.5G /home/ssqr/tmp_cv3
630M /home/ssqr/SSquare
8.1M /home/ssqr/.local
4.0K /home/ssqr/Public
242M /home/ssqr/Downloads
504K /home/ssqr/Pictures
520K /home/ssqr/docs
71M /home/ssqr/.npm
12K /home/ssqr/.ssh
12K /home/ssqr/.ngrok2
4.0K /home/ssqr/.icons
32K /home/ssqr/AlsaFiles
39M /home/ssqr/bin
392K /home/ssqr/share
34M /home/ssqr/.config
12K /home/ssqr/.dbus
76K /home/ssqr/.pki
8.0K /home/ssqr/Documents
8.0K /home/ssqr/.vim
196M /home/ssqr/.cache
7.7G /home/ssqr

1 Like
11

Drew, Dell:

Now Rootfs filesystem size will be 10616 MiB.

Setting mender variables as follows:
#(MENDER_STORAGE_TOTAL_SIZE_MB - MENDER_BOOT_PART_SIZE_MB - MENDER_DATA_PART_SIZE_MB) / 2
2264MB - 44MB - 128 /2 = 11046MB = 10534.286499023 MiB
Should I also set IMAGE_ROOTFS_SIZE to ā€œ-1ā€?

Thanks,
Raj
-----X----
rajendra@SmartSquareSystems:~$ cd mender-convert; MENDER_ARTIFACT_NAME=NEWSSHP-202106160922.img ./docker-mender-convert --disk-image input/NEWSSHP-202106160922.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server

Running mender-convert --disk-image input/NEWSSHP-202106160922.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
Running mender-convert-extract: --disk-image input/NEWSSHP-202106160922.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-16 16:56:07 [INFO] [mender-convert-extract] Using configuration file: configs/mender_convert_config
2021-06-16 16:56:07 [INFO] [mender-convert-extract] Using configuration file: configs/raspberrypi3_config
2021-06-16 16:56:07 [INFO] [mender-convert-extract] Validating disk image
2021-06-16 16:56:07 [INFO] [mender-convert-extract] Disk parsed successfully
2021-06-16 16:56:07 [INFO] [mender-convert-extract] NUMBER OF PARTS: 2 TYPE: dos
2021-06-16 16:56:07 [INFO] [mender-convert-extract] PART 1: SIZE: 42.9M TYPE: 0xc
2021-06-16 16:56:07 [INFO] [mender-convert-extract] PART 1: extracting to work/part-1.fs
2021-06-16 16:56:08 [INFO] [mender-convert-extract] PART 2: SIZE: 14.7G TYPE: 0x83
2021-06-16 16:56:08 [INFO] [mender-convert-extract] PART 2: extracting to work/part-2.fs
mender-convert-extract has finished. Cleaning upā€¦
Running mender-convert-modify: --disk-image input/NEWSSHP-202106160922.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-16 17:08:55 [INFO] [mender-convert-modify] Using configuration file: configs/mender_convert_config
2021-06-16 17:08:55 [INFO] [mender-convert-modify] Using configuration file: configs/raspberrypi3_config
2021-06-16 17:09:10 [INFO] [mender-convert-modify] Installing Mender client and related files
2021-06-16 17:50:10 [INFO] [mender-convert-modify] Creating state scripts version file.
2021-06-16 17:50:10 [INFO] [mender-convert-modify] Installing a custom /etc/fstab (see work/convert.log for more info)
2021-06-16 17:50:10 [INFO] [mender-convert-modify] Performing platform specific modifications (if any)
2021-06-16 17:50:10 [INFO] [mender-convert-modify] Running hook: platform_modify
2021-06-16 17:50:14 [INFO] [mender-convert-modify] Certain service files have been changed to align with our /uboot boot partition mount point. See convert.log for more information
2021-06-16 17:50:14 [INFO] [mender-convert-modify] Enabling UART in U-Boot configuration
2021-06-16 17:50:14 [INFO] [mender-convert-modify] Performing user/local specific modifications (if any)
2021-06-16 17:50:14 [INFO] [mender-convert-modify] Running hook: user_local_modify
2021-06-16 17:50:14 [INFO] [mender-convert-modify] Applying rootfs overlay: ./bootstrap-rootfs-overlay-hosted-server
mender-convert-modify has finished. Cleaning upā€¦
Running mender-convert-package: --disk-image input/NEWSSHP-202106160922.img --config configs/raspberrypi3_config --overlay ./bootstrap-rootfs-overlay-hosted-server
2021-06-16 17:50:15 [INFO] [mender-convert-package] Using configuration file: configs/mender_convert_config
2021-06-16 17:50:15 [INFO] [mender-convert-package] Using configuration file: configs/raspberrypi3_config
2021-06-16 17:50:47 [INFO] [mender-convert-package] Rootfs filesystem size will be 10616 MiB
2021-06-16 17:50:47 [INFO] [mender-convert-package] Creating a file-system image from: work/rootfs/data/
2021-06-16 17:50:48 [INFO] [mender-convert-package] Creating a file-system image from: work/rootfs/

2021-06-16 18:00:40 [INFO] [mender-convert-package] Copying root filesystem image to deploy directory
2021-06-16 18:06:39 [INFO] [mender-convert-package] Writing Mender artifact to: deploy/NEWSSHP-202106160922-raspberrypi3-mender.mender
2021-06-16 18:06:39 [INFO] [mender-convert-package] This can take up to 20 minutes depending on which compression method is used
2021-06-16 18:17:57 [INFO] [mender-convert-package] Creating Mender compatible disk-image
2021-06-16 18:17:57 [INFO] [mender-convert-package] Total disk size: 38000 MiB
2021-06-16 18:17:57 [INFO] [mender-convert-package] Boot partition 44 MiB
2021-06-16 18:17:57 [INFO] [mender-convert-package] RootFS partitions 18960 MiB x 2
2021-06-16 18:17:57 [INFO] [mender-convert-package] Data partition 16 MiB
2021-06-16 18:17:58 [INFO] [mender-convert-package] Using input diskimage partition scheme (dos)
2021-06-16 18:17:58 [INFO] [mender-convert-package] Writing DOS (MBR) partition table
2021-06-16 18:17:59 [INFO] [mender-convert-package] Writing boot partition image
2021-06-16 18:18:00 [INFO] [mender-convert-package] Writing rootfsa partition image
2021-06-16 18:24:31 [INFO] [mender-convert-package] Writing rootfsb partition image
2021-06-16 18:32:19 [INFO] [mender-convert-package] Writing data partition image
2021-06-16 18:32:19 [INFO] [mender-convert-package] Performing platform specific package operations (if any)
2021-06-16 18:32:19 [INFO] [mender-convert-package] Running hook: platform_package
2021-06-16 18:32:19 [INFO] [mender-convert-package] Compressing deploy/NEWSSHP-202106160922-raspberrypi3-mender.img.gz
2021-06-16 18:52:59 [INFO] [mender-convert-package] Conversion has completed! \o/
mender-convert-package has finished. Cleaning upā€¦
Output Artifacts and images can be found in the deploy directory:

12

Setting IMAGE_ROOTFS_SIZE to ā€œ-1ā€ will result in the partitions being sized exactly as you calculated above. If you set it to 0 then it will be sized based on the existing contents. Your choice which is the right one for your application.
Drew