Large SDIMG File

#1

Hi There,
Any suggestions on how to speed up the flashing of a large SDIMG file? I’ve got a 14GB image for my RPi3 based system, and it’s taking much too long to flash the SD cards.
If I was to reduce the root partition size, is there some way to expand the two root partitions upon first boot?
Cheers,
Donal

#2

If I was to reduce the root partition size, is there some way to expand the two root partitions upon first boot?

Typically you can only re-size and expand the last partition. So a simple way of reducing the SDIMG file size is to lower the DATA_PART_SIZE to minimum and expand that on boot.

Other optimization is to work with sparse files, e.g with:

#3

Thank you, will look into the BMAP tools.

#4

@mirzak bmap-tools has worked a treat, greatly reducing the flash time, thank you! Also, I hadn’t realised that the generation of the bmap files for the various yocto build output files was already enabled - is this a mender or yocto feature?

#5

Glad it worked out. It is a Yocto feature

#6

Dear community,

I wanted to drop my 2 cents here by reporting results.
bmap-tools is indeed faster.

Please proceed with the repo indicated or in ubuntu:
sudo apt-get install bmap-tools

Make sure that your device is connected and present in the /dev partition and both files, the raw image and the .bmap file are in the same folder.

I tested with a RaspberryPi3, and after .rpi-boot it is necessary to umount any existing partitions except for /dev/sdb (in my case)

Then proceed:

mvargasevans@X1:~//mender-builder$ sudo bmaptool copy ./development.rpi-sdimg /dev/sdb
bmaptool: info: discovered bmap file './development.rpi-sdimg.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 1661952 blocks of size 4096 (6.3 GiB), mapped 606924 blocks (2.3 GiB or 36.5%)
bmaptool: info: copying image 'development.rpi-sdimg' to block device '/dev/sdb' using bmap file 'development.rpi-sdimg.bmap'
bmaptool: WARNING: failed to enable I/O optimization, expect suboptimal speed (reason: cannot switch to the 'noop' I/O scheduler: [Errno 22] Invalid argument)
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdb'
bmaptool: info: copying time: 7m 36.4s, copying speed 5.2 MiB/sec

Some info from the documentation.
https://www.yoctoproject.org/docs/2.3/dev-manual/dev-manual.html#flashing-images-using-bmaptool

There is an issue with my I/O, but that was already faster for a 7 GB image.

1 Like
#7

bmap-tools can also directly flash a compressed image file without decompressing the entire file first.

So it’s possible to make a sdimg, then copy the compressed sdimg and bmap file into the sdimg (by loopback mounting a partition on it). Since it will compress greatly the copy of the image can fit inside the image itself.

Then the sd image with the compressed image inside it can be used to boot the device via an SD card slot and then use bmap-tool to reflash the onboard eMMC quickly.

#8

Great to see that you are sharing you experiences and how you can optimize the workflows.

Utilization of bmap and similar tools to speed provisioning (which is very important when in a production line) devices would be great content for a tutorial on our Tutorial section if someone is up for writing something a bit more detailed.