Large persistent data partition without large image file

General Discussions
1

Hi.

I’m just getting started with mender and try to convert the installation of a x64 host.
For that I followed INTEL NUC x86-64 Ubuntu 18.04 with a few small changes.

My main difference is that I use a 256GB SSD onto which I would like to have the following partition table.
512MB boot/EFI partition
20GB rootfs A partition
20GB rootfs B partition
~200GB data partition

I used the docker-mender-convert script to generate an image with this setup, but was surprised that I actually had to create an image with almost 250GB of data, as the data partition is basically empty at that time.

My next approach was to set MENDER_STORAGE_DEVICE to something like 50GB to get a smaller image file. I was hoping that during the first boot the growing of the partition would allow it to fill the rest of the SSD on its own. This didn’t work however.

So, my question is, can I somehow achieve a small disk image file and automatically resize the /data partition and regrow the filesystem on it afterwards?

I’m especially wondering how I can do this with as low of a risk as possible that a future update would try to rewrite the partition table.

2

Have a look at mender-growfs-data

1 Like
3

Yes, like @elipsion says, the data partition is grown on boot, to fill the available space, and this is enabled by default, here
and here
:grinning_face_with_smiling_eyes:

4

Thank you @elipsion and @oleorhagen .
Yes, I noticed this as well. In the log files I also noticed that the filesystem was grown to the full size of the partition. That was not really the problem.

My concern is that I want to have a data partition that will be resized (partition size and filesystem) to the full size of the SSD I have in the machine.
This is for now 256GB. So I want boot to be 512MB, root A 20GB, root B 20GB and data whatever the rest.

If I set this up like this I get an image of ~256GB of size (for some reason it was not compressed even though it is set to gzip in the convert config file). I feel that having an image with 200+GB of empty space is a waste of time and resources.
So I changed my configuration to have boot to be 512MB, root A 20GB, root B 20GB and data at 10GB. I was hoping that when I flash the resulting image that the date partition would grow beyond the 10GB I set up and would fill the whole available space. And after that update the filesystem with growfs to match the new partition size. This however, is not what I observed.

Should it actually be the case or not?

5

Check the boot log for errors, as it should expand the data partition if its the last partition and resize the file system to fill it on boot.

However saying that, if you do see errors then i have observed similar issues on my x86 projects and have needed to supplement the systemd growfs feature with our own systemd service and script to fix up any disk issues prior to the growfs feature running as on its own its not enough sometimes depending on the errors encountered.

1 Like
6

The gpt issue can be fixed up with sgdisk tool, I believe there are other discussions on the topic in the forum,

That’s basically, the reason we have our own systemd service that checks and fixes disk issues, prior to the mender/systemd growfs service

7

Hi @dellgreen .
sorry it took me so long to get back to you. I found a few more problems blocking me. :sweat:
Mainly EFI not booting into mender’s grub.cfg but another one from the original OS installation. Removing that entry with efibootmgr gave me a booting system.

So, I tried again and made sure to check the logs for growfs being called. What I do find is this line

Sep  6 16:25:47 r25 systemd-growfs[516]: Successfully resized "/data" to 1.0G bytes.

showing that it resized /data to only the size I gave it with
MENDER_DATA_PART_SIZE_MB="1024"
I do not see any other output that looks like it even attempted to resize the actual partition. I mainly checked /var/log/syslog, is there any other log I should be checking?

8

I have just taken a look back at my projects, and i can see that it clearly wasnt working for me either, so the custom systemd service that I added uses sgdisk to fix GPT issues, grow the last partition, and notify the kernel of the parition table change if it was changed, prior to systemd mounting and growfs running. I cant paste the full service as its highly custom, however the premise is:

to fix the GPT issues is was using the following in a script and customize

/sbin/sgdisk -e "$basedev"

if growpart "$basedev" "$lastpartn";then 
  partprobe
fi

then i created a custrom systemd service file


[Unit]
Description=Grow last parition on rootfs disk
DefaultDependencies=no
Before=data.mount
Before=systemd-growfs@data.service
Before=local-fs-pre.target
ConditionVirtualization=!container
After=systemd-remount-fs.service

[Service]
Type=oneshot
User=root
Group=root
ExecStart=/usr/local/bin/your-growfs-script

[Install]
WantedBy=local-fs-pre.target

then in my mender config i add extra fstabs options

MENDER_DATA_PART_FSTAB_OPTS="${MENDER_DATA_PART_FSTAB_OPTS},x-systemd.after=your-growfs.service"

1 Like
9

Thanks again for the update. I didn’t find the time to continue this project until now.
Looking at the approach mender takes to resize the partition I found what I believe to be the issue on at least my machine.

Mender uses parted to try to resize the partition, meta-mender/mender-client-resize-data-part.sh.in at 1dbb4b07a057045b4b60e26a5eefcac764abb7c4 · mendersoftware/meta-mender · GitHub , but at least on my machine this will fail. When I run this by hand the following output is generated and no change is applied to the data partition.

sudo parted -s /dev/sda resizepart 4 100%
Warning: Partition /dev/sda4 is being used. Are you sure you want to continue?

My assumption is that message is silently ignored during the system boot and meta-mender/mender-systemd-growfs-data.service at 1dbb4b07a057045b4b60e26a5eefcac764abb7c4 · mendersoftware/meta-mender · GitHub tries to expand the file system accordingly. It even reports that it successfully grew to 512MB, which was the original size anyway.

My first assumption was to just make sure to unmount /data before we try parted, but that fails as mender is actively using the partition now. It might work during boot, not sure.

sudo lsof /data/
[sudo] password for adev: 
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mender  1194 root  mem    REG    8,4    12288   16 /data/mender/mender-store
mender  1194 root  mem-r  REG    8,4     8192   15 /data/mender/mender-store-lock
mender  1194 root    7ur  REG    8,4     8192   15 /data/mender/mender-store-lock
mender  1194 root    8u   REG    8,4    12288   16 /data/mender/mender-store
mender  1194 root    9u   REG    8,4    12288   16 /data/mender/mender-store

Resorting to the tool growpart I was then able to resize the /data partition even while it was mounted.

sudo growpart /dev/sda 4
CHANGED: partition=4 start=85966848 old: size=1048576 end=87015424 new: size=414151311 end=500118159
sudo partprobe
sudo /lib/systemd/systemd-growfs /data
Successfully resized "/data" to 197.4G bytes (3584 bytes lost due to blocksize).

And now I’m ready to integrate this into mender-convert I guess.
Judging by this config mender-convert/raspberrypi_raspbian_config at 5cc9cf5a45066855800e2e939b067ed52fc24567 · mendersoftware/mender-convert · GitHub it looks to me that I can just override the grow_data_partition function with the commands I found working for me.

I guess some more testing on my machine and then I should be ready to open a PR to propose the use of growpart instead of parted. Maybe this is of some interest to others.

1 Like