Hey there! I’m interested in using the mender-convert toolchain to integrate Mender into my existing Raspberry Pi Zero W “golden image.” However I’m skeptical of this decision, as I intend to move to Yocto in the near future.
If I deploy devices with a Debian image processed by mender-convert, will I be able to update those devices to Yocto? Any known or anticipated gotchas that I should expect along the way?
Are there any differences between U-boot for the Yocto codebase vs Debian? I assume the partition layout is the the same, but haven’t dug into this yet as I am new to Mender, Yocto, and U-boot.
I suppose that I could use the Mender 2.0 update modules to make any required boot partition changes during a migration to Yocto from Debian, but I haven’t seen any examples of this. Has anyone tried this migration before?
If I deploy devices with a Debian image processed by mender-convert, will I be able to update those devices to Yocto? Any known or anticipated gotchas that I should expect along the way?
This is not something that I have tested recently, but there is one gotcha and that would be the content of the boot partition (bootloader, devicetree and rpi boot firmware) and you should assume that you will need to update these during a Debian → Yocto migration, which is a “risk” and it might be hard to support a full rollback in case something goes wrong during this update.
Are there any differences between U-boot for the Yocto codebase vs Debian? I assume the partition layout is the the same, but haven’t dug into this yet as I am new to Mender, Yocto, and U-boot.
The U-boot integration in mender-convert, is copied from the Yocto environment and the version used is aligned with the latest Yocto stable branch (thud right now)
I suppose that I could use the Mender 2.0 update modules to make any required boot partition changes during a migration to Yocto from Debian, but I haven’t seen any examples of this.
I do not have an example using update modules, but I do have an example on how to do this using “state-scripts”
The boot partition (rpi boot firmware files) on Raspberry Pi is problematic and can even cause problems when updating Yocto -> Yocto during major releases.
Thanks for the tips! That state-script will really help.
After rereading the known issues regarding the boot partition, things make a lot more sense.
I’ve known for quite a while that I’d encounter various issues—unfixable due to proprietary firmware—while deploying the Raspberry Pi as a production system, but this issue crystalizes the limitations and lack of flexibility when attempting to scale. All the more reason to redesign with a stable SoM.
I just stumbled across this post by coincidence and it affects our planned workflow significantly. It most certainly was pretty naive to assume that transitioning from mender convert to yocto would be easy. But that was the impression I got back when I started. I think the fact that rolling out yocto images to mender-converted systems is not a seamless transition should be pointed out very prominent (bold, red, italic, underlined) in the mender-convert documentation (or did I overlook this?) as this can heavily affect the road ahead.
For us that means, that we have to most likely collect our devices deployed on the other side of the globe and provision them again. Well after all, I should have been less naive…
Given the tedious nature of OTA Updates, I’d recommend a clean yocto re-provisioning approach. We decided to skip the mender-convert step and we refactored for yocto. Though my advice won’t make the collection and re-provisioning much easier, I’m happy to answer any questions about our transition! I’m sure others in the community would appreciate it.
Thank you very much for that offer and sharing your experience!!! We are going to do as you suggest, to re-provision the devices once we are back at the installation site.
Hey again,
I spent a little time trying to produce a mender artifact that can be deployed on a mender-converted image. I’m aware that this is not recommended and it is at my own risk of bricking devices. Still I found it neat to get away from mender-convert without actually having to touch each device (RaspberryPi Zero wifi). My test device is downloading, flashing and rebooting fine at first when I run mender install <my-artifact-url>. Unfortunately, in the end of the boot procedure it switches to maintenance mode.
[ 33.271534] bcm2835-codec bcm2835-codec: Device registered as /dev/video12
[ 33.375691] bcm2835-codec bcm2835-codec: Loaded V4L2 isp
[ 33.511146] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[ 33.554823] usbcore: registered new interface driver brcmfmac
[ 33.839848] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[ 33.898261] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: May 2 2019 02:39:18 version 7.45.98.83 (r714225 CY) FWID 01-e539531f
[ 35.608599] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 77.844046] random: crng init done
[ 77.849430] random: 6 urandom warning(s) missed due to ratelimiting
Press Enter for maintenance
(or press Control-D to continue):
I assume that Volume was not properly unmounted. Some data may be corrupt. Please run fsck. points to the problem. Do you have a hint how I can tackle this problem or what to check to trace down the issue?
Any hint is highly appreciated.
All the best and stay healthy!
I did not set the root filesystem to be readonly. Maybe that hints to the root. Or do you happen to see something obvious that I missed (if you find the time)?
As suggested here. Is this the correct way to switch partitions in this case?
Funny thing is that after a reboot the device crashed entirely complaining about missing environment variables pxeuuid and bootfile): https://paste2.org/YLGD2k1G strange…
Checking /etc/fstab showed that device /dev/mmcblk0pp4 was to be mounted at /data. This device does not exist. The error stems from MENDER_DATA_PART which if not set defaults to MENDER_DATA_PART_DEFAULT=mmcblk0pp4. I corrected the line in fstab. Now, /data gets properly mounted and expanded.
The update seems to finish successfully. I will run a couple of tests and update this thread once I rolled out the update. Thanks a lot for your help!
Hey,
For some reason my device always tries to boot from the device /dev/mmcblk0pp2. That’s what the bootlog says.
My bitbake configuration states (from bitbake -e):
MENDER_ROOTFS_PART_A="/dev/mmcblk0p2"
(explicitly set in my multiconfig configuration) and
MENDER_ROOTFS_PART_A_DEFAULT="/dev/mmcblk0pp2"
Is the second taking precedence over the first one? I don’t really understand if there is a special purpose of the devices containing the “extra p”. I’m a little puzzled and hope that you can shed some light onto the rootfs devices naming here.
The place where a “double-pp” device shows up is in the mender_setup command. So I’m pasting this here also if others stumble accross this post and the paste bin is gone:
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; if test "${mender_systemd_machine_id}" != ""; then setenv bootargs systemd.machine_id=${mender_systemd_machine_id} ${bootargs}; fi; setenv mender_kernel_root /dev/mmcblk0pp${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk0p2; else setenv mender_boot_part_name /dev/mmcblk0p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 0:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_post_setup_commands; fi
The device booted from a pure yocto built image to check if the problem persists here as well. Thus, it’s not related to the transition from mender convert to pure yocto.
As you note, it is embedded in the mender_setup command which likely came from your invocation of mender-convert. Since you deployed the Yocto image OTA, the UBoot settings were not changed except for the partition numbers when switch from active to passive.
To work around this in a deployed system, you can use fw_setenv from the Linux shell to modify the mender_setup variable.
The proper fix would be to determine why in your mender-convert invocation you are getting the “double-pp”.
Just to emphasize: I did not combine mender-convert and yocto here for this test. I provisioned the SD card with a newly bitbaked image. Thus, it’s more related to the actual bootloader mixture of mender-convert and bitbake (I guess).
I have the odd feeling that the problem is related to bitbakes cache. I recompiled the bootloader with something like bitbake virtual/bootloader -C compile and the kernel with bitbake virtual/kernel -C compile before starting to bitbake the actual image. Is there any further measure I should take to ensure the necessary bits and pieces get recompiled?
Maybe I’m wring, it’s just that bitbakes cache already caught me off guard a few times before…
I know that in recent years, the sstate-cache mechanism has become much more automatic and rebuilds pretty much when it needs to. However if you want to force a full rebuild without using the cache, you can do something like bitbake -c cleansstate virtual/kernel and then your normal image build and the kernel will be full built from scratch. You can use cleanall instead if you want to wipe the downloads and re-fetch them as well.
strange…