Dunfell to Kirkstone migration /boot partition on flahsed devices

General Discussions
1

Hello mender community,

First of all, I find it necessary to say that I’ve been very impressed with mender and its integration with the Yocto project, it is extremely rugged and has saved us countless hours of development work.

My problem is an extension of the comment I placed on github regarding the new Kirkstone branch of the meta-mender-community integration layers for the raspberry pi.

I have a fleet of RPis (3B and 4B) that use mender standalone in order to preform software and OS updates. We have a custom Yocto layer that makes heavy use of the meta-mender and meta-mender-community layers and we’re very excited to make the change from Dunfell to Kirkstone as it’ll open up new recipes that will help us a lot and also needless to say, update the OS to a more recent version and all the stability that comes with that.

Our problem currently is due to the change in the default size of the /boot partition. My understanding is that there is a need for this partition to be larger than the dunfell default and thus a kirkstone update applied to a dunfell update does not work. I have tested this to be true, simply changing the branch from dunfell to kirkstone (and applying the migration considerations from Yocto), I can create an update file and run a mender install over it in the dunfell image. However, after commiting and rebooting, the device will not boot properly - I get no splash screen nor do I get any particularly relevant output from the screen.

By looking at this topic provided in @TheYoctoJester 's response, I can inherit rpi-update-firmware to apply the change to the /boot partition, however the change in default boot partition size makes the update fail since the existing /boot partition is too small compared to the required /boot partition.

Now the suggestion before considering reformatting (which will inevitability be required but ideally would prefer to leave it for a later update) is to review the overlays, which I agree are not all necessary. And I guess therein lies my current question. Has anyone played around with the overlays of the /boot partition, and would it be possible to point me in the right direction to play around with them myself? Has someone here had these issues before and has had to consider the process of reformatting who could also point me in the right direction for that?

I appreciate any answers and am open to provide any needed information, configuration files and outputs necessary. I didn’t right now since this is:

  1. A fairly high level question,
  2. I am not a very talented firmware developer or implementer and am fairly dependent on third party (hence mender) tools for these kinds of situations.

Thanks all

2

Hi @Lifepowr,

Just a short heads up: I am already looking into reducing the /boot partition size, but nothing ready to share yet. Will keep you in the loop.

Greetz,
Josef

3

Hi @Lifepowr,

Managed to figure it out. :slight_smile:
You can remove a few things from the boot partition by adding

IMAGE_BOOT_FILES ?= "${BOOTFILES_DIR_NAME}/* \
                 ${@make_dtb_boot_files(d)} \
                 u-boot.bin;${SDIMG_KERNELIMAGE} boot.scr \
                 "

to your local.conf. This trims it down enough to fit into 40M again.
If you want to cut it down even more, you can additionally add

RPI_KERNEL_DEVICETREE_OVERLAYS = ""

(or adjusted to your hardware needs), but then you are losing some functionality. Therefore I would advise to only do so if absolutely required.

We will now look into adding this to meta-mender-raspberrypi in some way, but you should be good to go now.

Greetz,
Josef

4

Hi @TheYoctoJester,

Thanks a lot for checking this out. I’ll have a look at this approach today and get back to you.

We did manage to get the update working by creating a bbappend to the meta-raspberrypi bootfiles bsp-recipe. With code similar to this:

do_deploy:append() {
    rm ${DEPLOYDIR}/${BOOTFILES_DIR_NAME}/start_cd.elf
    rm ${DEPLOYDIR}/${BOOTFILES_DIR_NAME}/start_db.elf
}

I believe your solution does what we did in a more dynamic way and thus it’s probably useful to consider.

Again, thanks a lot for the extremely swift response, I’m impressed that this got traction so quickly and a solution was found just as quickly. I’ll test the solution during the following days for both models of boards we’re using and will reply here if everything works.

Cheers

António (Lifepowr)

5

@Lifepowr how did your tests go?

6

Hello all,

I apologize for the delay in the reply. Startup work removes a lot of my availability unexpectedly.

The tests have been made from my end but it did not solve the problem. For the raspberry pi 4, the firmware update finished fine. However, for the raspberry pi 3, the firmware update fails, claiming that there is no space left in the device.

In those particular devices, the boot partition is full with temporary .elf files, which I assume are part of the firmware update script. And during this temporary file creation, the partition (which is currently 40MB) does indeed reach the limit (100% according to df). All of these are assumptions as to what is happening at the firmware update level, but it sure does look like it’s what’s happening.

The way I managed to make the firmware update to work was to send out a “pre-update” update with a postinstall_ontarget script that removes the .elf files that are shared between the rpi3 and rpi4 but are unnecessary in the “opposite” CPU architectures:

pkg_postinst_ontarget:${PN}() {
  ...
  ELF = "start.elf"
  ELF:arm = "start4.elf"
  ELFX = "start_x.elf"
  ELFX:arm = "start4x.elf"
  ELFCD = "start_cd.elf"
  ELFCD:arm = "start4cd.elf"
  ELFDB = "start_db.elf"
  ELFDB:arm = "start4db.elf"
  ...
  pkg_postinst_ontarget:${PN}() {
    ...
    if test -f "/uboot/${ELFCD}"; then
      rm /uboot/${ELF}
      rm /uboot/${ELFX}
      rm /uboot/${ELFCD}
      rm /uboot/${ELFDB}
    fi
    ...
}

I’m sure this is not very eloquent code but it works for our use case. It removes these unnecessary files as a function of the architecture used and provides enough space in the boot partition for the temporary files to be correctly copied and the firmware update script to finish properly. Running this as an update BEFORE preforming the Kirkstone update should allow the firmware update to complete without issues.

Attempting to anticipate questions: I do not believe we do any alteration to the boot partition unless some of the recipes we install do so, so I’m almost certain that we don’t have any custom alterations to the boot partition (except of course for this change we now do) when this firmware update script does get triggered, so I’d say we are working with an unchanged “meta-yocto” boot partition.

I can provide our local.conf file if required. Unfortunately we are currently moving ahead with the aforementioned script so after the push we’ll be doing to production this week, our production devices will have an altered boot partition and I will no longer be able to test these solutions on a “unchanged” boot partition from a mender build.

I am available for any other questions you may have. I’ll attempt to answer this time in a speedier manner.

Cheers

António - Lifepowr