# Migrating from yocto dunfell to kirkstone causes RPI3B+ devices to hang on OTA update

**URL:** <https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483>\
**Category:** General Discussions\
**Tags:** raspberry-pi-3, dunfell, kirkstone\
**Created:** [December 9, 2022, 11:36pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483 "2022-12-09T23:36:36Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [December 9, 2022, 11:36pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/1 "2022-12-09T23:36:36Z")

</div>

I’m having issues with migrating in-field raspberry pi 3B+s over from Dunfell to Kirkstone yocto builds. They are currently running on kernel 4.14.114 and I’m trying to migrate them to 5.15.36. I’m using u-boot for my bootloader and have not had any issue when flashing a fresh pi with the Kirkstone built image.

I’ve enabled uart console output on my RPI and I find that I’m getting stuck as soon as the kernel is loaded:

> U-Boot\> boot  
> switch to partitions #0, OK  
> mmc0 is current device  
> Scanning mmc 0:1…  
> Found U-Boot script /boot.scr  
> 300 bytes read in 1 ms (293 KiB/s)  
> ## Executing script at 02400000  
> switch to partitions #0, OK  
> mmc0 is current device  
> 7253840 bytes read in 302 ms (22.9 MiB/s)  
> ## Booting kernel from Legacy Image at 00080000 …  
> Image Name: Linux-5.15.34-v7  
> Image Type: ARM Linux Kernel Image (uncompressed)  
> Data Size: 7253776 Bytes = 6.9 MiB  
> Load Address: 00008000  
> Entry Point: 00008000  
> Verifying Checksum … OK  
> ## Flattened Device Tree blob at 2eff9600  
> Booting using the fdt blob at 0x2eff9600  
> Loading Kernel Image  
> Using Device Tree in place at 2eff9600, end 2f002f6d
> 
> Starting kernel …

I believe the issue lies in some sort of uboot configuration in the Dunfell image that isn’t compatible with the kernel in the Kirkstone image. I’m not familiar with the boot sequence of u-boot and Linux so I’d like to see if anyone else has been able to reproduce this or has any insight as to what might be going wrong.

---

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [December 9, 2022, 11:40pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/2 "2022-12-09T23:40:44Z")

</div>

In case it’s relevant, I’ve also done a diff printout of the difference between the uboot environments of the Dunfell image and the Kirkstone image, respectively, when they are freshly flashed. Here’s what I found  
\< shows Dunfell uboot env, \> shows Kirkstone uboot env

> 10c10,11  
> \< boot\_efi\_binary=if fdt addr ${fdt\_addr\_r}; then bootefi bootmgr ${fdt\_addr\_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro\_bootpart} ${kernel\_addr\_r} efi/boot/bootarm.efi; if fdt addr ${fdt\_addr\_r}; then bootefi ${kernel\_addr\_r} ${fdt\_addr\_r};else bootefi ${kernel\_addr\_r} ${fdtcontroladdr};fi  
> ---  
> \> boot\_efi\_binary=load ${devtype} ${devnum}:${distro\_bootpart} ${kernel\_addr\_r} efi/boot/bootarm.efi; if fdt addr ${fdt\_addr\_r}; then bootefi ${kernel\_addr\_r} ${fdt\_addr\_r};else bootefi ${kernel\_addr\_r} ${fdtcontroladdr};fi  
> \> boot\_efi\_bootmgr=if fdt addr ${fdt\_addr\_r}; then bootefi bootmgr ${fdt\_addr\_r};else bootefi bootmgr;fi
> 
> 17,18c18  
> \< boot\_targets=mmc0 mmc1 usb0 pxe dhcp  
> \< bootargs=8250.nr\_uarts=1 bcm2708\_fb.fbwidth=656 bcm2708\_fb.fbheight=416 bcm2708\_fb.fbswap=1 vc\_mem.mem\_base=0x3ec00000 vc\_mem.mem\_size=0x40000000 dwc\_otg.lpm\_enable=0 console=ttyS0,115200 rootfstype=ext4 rootwait root=${mender\_kernel\_root}  
> ---  
> \> boot\_targets=mmc0 mmc1 mmc2 usb0 pxe dhcp
> 
> 20c20  
> \< bootcmd\_dhcp=run boot\_net\_usb\_start; if dhcp ${scriptaddr} ${boot\_script\_dhcp}; then source ${scriptaddr}; fi;setenv efi\_fdtfile ${fdtfile}; if test -z “${fdtfile}” -a -n “${soc}”; then setenv efi\_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi\_old\_vci ${bootp\_vci};setenv efi\_old\_arch ${bootp\_arch};setenv bootp\_vci PXEClient:Arch:00010:UNDI:003000;setenv bootp\_arch 0xa;if dhcp ${kernel\_addr\_r}; then tftpboot ${fdt\_addr\_r} dtb/${efi\_fdtfile};if fdt addr ${fdt\_addr\_r}; then bootefi ${kernel\_addr\_r} ${fdt\_addr\_r}; else bootefi ${kernel\_addr\_r} ${fdtcontroladdr};fi;fi;setenv bootp\_vci ${efi\_old\_vci};setenv bootp\_arch ${efi\_old\_arch};setenv efi\_fdtfile;setenv efi\_old\_arch;setenv efi\_old\_vci;  
> ---  
> \> bootcmd\_dhcp=devtype=dhcp; run boot\_net\_usb\_start; if dhcp ${scriptaddr} ${boot\_script\_dhcp}; then source ${scriptaddr}; fi;setenv efi\_fdtfile ${fdtfile}; if test -z “${fdtfile}” -a -n “${soc}”; then setenv efi\_fdtfile ${soc}-${board}${boardver}.dtb; fi; setenv efi\_old\_vci ${bootp\_vci};setenv efi\_old\_arch ${bootp\_arch};setenv bootp\_vci PXEClient:Arch:00010:UNDI:003000;setenv bootp\_arch 0xa;if dhcp ${kernel\_addr\_r}; then tftpboot ${fdt\_addr\_r} dtb/${efi\_fdtfile};if fdt addr ${fdt\_addr\_r}; then bootefi ${kernel\_addr\_r} ${fdt\_addr\_r}; else bootefi ${kernel\_addr\_r} ${fdtcontroladdr};fi;fi;setenv bootp\_vci ${efi\_old\_vci};setenv bootp\_arch ${efi\_old\_arch};setenv efi\_fdtfile;setenv efi\_old\_arch;setenv efi\_old\_vci;
> 
> 22a23  
> \> bootcmd\_mmc2=devnum=2; run mmc\_boot
> 
> 27d27  
> \< bootfstype=fat
> 
> 30d29  
> \< devplist=1
> 
> 35c34  
> \< fdt\_addr=2eff9600  
> ---  
> \> fdt\_addr=2eff7b00
> 
> 38,39c37  
> \< fdtaddr=2eff9600  
> \< fdtcontroladdr=3b3cd030  
> ---  
> \> fdtcontroladdr=3b3db5b0
> 
> 41,42d38  
> \< fileaddr=2400000  
> \< filesize=12c
> 
> 46c42  
> \< loadaddr=0x00200000  
> ---  
> \> loadaddr=0x1000000
> 
> 54d49  
> \< mender\_saveenv\_canary=1
> 
> 66,67c61,62  
> \< scan\_dev\_for\_efi=setenv efi\_fdtfile ${fdtfile}; if test -z “${fdtfile}” -a -n “${soc}”; then setenv efi\_fdtfile ${soc}-${board}${boardver}.dtb; fi; for prefix in ${efi\_dtb\_prefixes}; do if test -e ${devtype} ${devnum}:${distro\_bootpart} ${prefix}${efi\_fdtfile}; then run load\_efi\_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro\_bootpart} efi/boot/bootarm.efi; then echo Found EFI removable media binary efi/boot/bootarm.efi; run boot\_efi\_binary; echo EFI LOAD FAILED: continuing…; fi; setenv efi\_fdtfile  
> \< scan\_dev\_for\_extlinux=if test -e ${devtype} ${devnum}:${distro\_bootpart} ${prefix}${boot\_syslinux\_conf}; then echo Found ${prefix}${boot\_syslinux\_conf}; run boot\_extlinux; echo SCRIPT FAILED: continuing…; fi  
> ---  
> \> scan\_dev\_for\_efi=setenv efi\_fdtfile ${fdtfile}; if test -z “${fdtfile}” -a -n “${soc}”; then setenv efi\_fdtfile ${soc}-${board}${boardver}.dtb; fi; for prefix in ${efi\_dtb\_prefixes}; do if test -e ${devtype} ${devnum}:${distro\_bootpart} ${prefix}${efi\_fdtfile}; then run load\_efi\_dtb; fi;done;run boot\_efi\_bootmgr;if test -e ${devtype} ${devnum}:${distro\_bootpart} efi/boot/bootarm.efi; then echo Found EFI removable media binary efi/boot/bootarm.efi; run boot\_efi\_binary; echo EFI LOAD FAILED: continuing…; fi; setenv efi\_fdtfile  
> \> scan\_dev\_for\_extlinux=if test -e ${devtype} ${devnum}:${distro\_bootpart} ${prefix}${boot\_syslinux\_conf}; then echo Found ${prefix}${boot\_syslinux\_conf}; run boot\_extlinux; echo SCRIPT FAILED: continuing…; fi

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [December 12, 2022, 8:19pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/3 "2022-12-12T20:19:19Z")

</div>

Hello @ygu,

Maybe it is related to the workaround we have described [here](https://docs.mender.io/troubleshoot/yocto-project-runtime#deploying-an-update-from-dunfell-to-kirkstone-on-raspberry-pi-bl)?

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [December 12, 2022, 8:41pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/4 "2022-12-12T20:41:50Z")

</div>

Hi @TheYoctoJester,

Thanks for the response. I tried it with rpi-update-firmware and it works. However, like the disclaimer said, it’s a risky operation and when I rebooted the device before it had a chance to check into mender and commit the update, the unit could not book back into the old rootfs partition.

I was hoping that there was a way to upgrade that didn’t involve modifying u-boot but based on the link you shared it sounds like it’s unavoidable?

Best,  
Yutong

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [December 13, 2022, 8:53am UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/5 "2022-12-13T08:53:24Z")

</div>

Hi @ygu,

Yes, in this particular case it is unavoidable. This stuff being required to sit in the boot partition is one of the prime reasons why the Raspberry Pi is not exactly a platform that is well suited for industrial use cases. The CM versions that bring an eMMC can already improve things, and you can then fine tune things quite a bit - but in the end, the RasPis boot process just is not designed for updates and fallback. Sorry.

Greetz,  
Josef

---

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [December 20, 2022, 6:44pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/6 "2022-12-20T18:44:40Z")

</div>

From what I’ve gathered, there are DTBs in the boot partition of the raspberry pi that got packaged in with the original kernel it was built with. There are also DTBs the rootfs partition of the new image built that should be used with the new kernel. However, they aren’t being loaded and uboot is always going for the one in the boot partition which never gets updated.

This is where it gets unclear for me, is it not possible to tell uboot to load from the DTB in the rootfs partition instead of the boot partition?

---

<div class="post-metadata">

**Author:** ![lramirez](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/lramirez/32/1195_2.png) [@lramirez](https://hub.mender.io/u/lramirez)\
**Post date:** [December 20, 2022, 10:29pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/7 "2022-12-20T22:29:39Z")

</div>

Technically speaking I don’t see why it should not be possible (I have not tried myself) but for sure you will need to patch u-boot to get this behavior.

We avoid to differ too much from upstream as the best practice is to stick as close as the official repositories from the project are, and try to keep the patching as minimum as possible.

I was not part of the group who decided how to patch the bootloader for Dunfell or Kirkstone, still I would imagine this was the logic when it was implemented and then the subsequent advice @TheYoctoJester made here [Migrating from yocto dunfell to kirkstone causes RPI3B+ devices to hang on OTA update - #3 by TheYoctoJester](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/3)

Did you try already this flag?

---

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [December 22, 2022, 11:51pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/8 "2022-12-22T23:51:17Z")

</div>

Yes the flag works but I don’t like the idea of updating the bootloader/DTB in the boot partition since it increases the chances that the unit may brick.

For instance, when I reboot the unit before it’s had a chance to check into mender, it causes the device to revert which (I assume) causes the new DTB to no longer match the old kernel resulting in it failing to boot. Additionally, I’m worried that rebooting the device while in the middle of updating the DTB/uboot will cause it to brick too.

Is there any way around this to get the chances of bricking down to zero (or as close to zero as possible)?

---

<div class="post-metadata">

**Author:** ![Piocky](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/piocky/32/1840_2.png) [@Piocky](https://hub.mender.io/u/Piocky)\
**Post date:** [February 16, 2023, 5:11pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/9 "2023-02-16T17:11:56Z")

</div>

Hi ygu,

Did you find a way to reduce the risk of turning your RPi3 into a brick? I’m facing the exact same issue here, and have some beta clients that are using our products with RPi3 inside, so I’m a bit stressed with the idea of breaking their products… At the same time, we need to develop a new image, as we already did for main customers, for whome their products use RPi-CM4, and don’t have this problem.

---

<div class="post-metadata">

**Author:** ![ygu](https://avatars.discourse-cdn.com/v4/letter/y/6bbea6/32.png) [@ygu](https://hub.mender.io/u/ygu)\
**Post date:** [February 20, 2023, 11:44pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/10 "2023-02-20T23:44:03Z")

</div>

Unfortunately no, I haven’t. I’m in a similar position and I can certainly relate to your pain. For now we’ve decided to keep using Dunfell and manage two branches of our yocto project, one on Dunfell for an our current customers and another on Kirkstone for if we ever decide to switch over and make it available to newer customers. It’s certainly not ideal so I’m still interested in figuring out a better way to migrate.

One idea I had is to create a ArtifactInstall\_Leave script which copies over the dtb in the rootfs of the new firmware over to the boot partition under a new name such as ‘bcm2837-rpi-3-b.dtb.new.’ Then it would modify the u-boot environment scripts to attempt to load the dtb with the ‘.new’ suffix if it exists (this can be done using the fw\_setenv utility available in Linux). If there does exist a dtb with the ‘.new’ suffix, it would also set an environment variable flag to indicate that it’s in the process of migrating DTBs.

If the new kernel boots up properly, it will execute an ArtifactCommit\_Leave script that would 1) copy the bcm2837-rpi-3-b.dtb.new to the old bcm2837-rpi-3-b.dtb and delete the ‘.new’ dtb, 2) unset the environment variable flag that was previously set by the old kernel.

If uboot boots up again and it sees that the environment variable flag is already set, it unsets it and uses the original DTB since it would mean that it did not successfully boot into the new kernel. The .new dtb should also be deleted by uboot or by the original rootfs’ mender on ArtifactFailure\_Enter.

Of course, this approach makes changes to the uboot environment which can introduce risk. To mitigate, you can take extra precautionary steps by storing the original value in some of the environment scripts, checking that edits are made properly by comparing hashes, and reverting if not. I suppose that there is a risk that the device is rebooted in the middle of updating the uboot environment. I’m not sure how significant that is though, perhaps someone wiser than me can answer that one.

I’m also not sure if making such changes to the uboot environment variables would break anything in Mender so it’s best to keep changes to a minimum (which this approach takes into account).

I’m curious to get someone from Mender’s take on this workaround. @lramirez , @TheYoctoJester, Could this work?

---

<div class="post-metadata">

**Author:** ![Piocky](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/piocky/32/1840_2.png) [@Piocky](https://hub.mender.io/u/Piocky)\
**Post date:** [February 21, 2023, 5:15pm UTC](https://hub.mender.io/t/migrating-from-yocto-dunfell-to-kirkstone-causes-rpi3b-devices-to-hang-on-ota-update/5483/11 "2023-02-21T17:15:33Z")

</div>

Hi @ygu,

Thanks a lot for your input. It seems to be a complicate process to make sure that the RPi won’t turn into a brick, and as you mentioned, what if something happens during the modification of uboot files (chances are very low I think, but still). In our case, I think that I will take the risk of just adding INHERIT += ‘rpi-update-firmware’ as suggested by Mender. It only concerns 30-40 devices, and all of them are beta users that we personnaly know, so it won’t be too difficult to help them if necessary.  
Thanks for your help anyway, I hope you’ll find a good solution for your problem.