Mender not building with yocto due to issues with u-boot

Good evening,

I have a jetson xavier agx and was able to build the tegra-demo-distro with scarthgap (LT35.5.0) without problems, with these layers:

  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-tegra \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-oe \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-python \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-networking \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-filesystems \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-virtualization \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-tegra-community \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-tegra-support \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-demo-ci \
  /home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-tegrademo \

When I then add the mender-core and -demo layer I get this error:

NOTE: Resolving any missing task queue dependencies
ERROR: Nothing PROVIDES 'u-boot'
u-boot was skipped: Either UBOOT_MACHINE or UBOOT_CONFIG must be set in the jetson-agx-xavier-devkit machine configuration.
ERROR: Required build target 'core-image-minimal' has no buildable providers.
Missing or unbuildable dependency chain was: ['core-image-minimal', 'u-boot']

I think the reason is this: mender-setup-grub

If I then add MENDER_EFI_LOADER = "" to the conf/local.conf the build starts but at the very end fails with the following error:

| DEBUG: Executing dd if=/dev/zero of=/home/yocto/yocto-tegra/tegra-demo-distro/build/tmp/work/jetson_agx_xavier_devkit-oe4t-linux/tegra-espimage/1.0/deploy-tegra-espimage-image-complete/tegra-espimage-jetson-agx-xavier-devkit-20240823205838.esp seek=65536 count=0 bs=1024
| 0+0 records in
| 0+0 records out
| 0 bytes copied, 6.2206e-05 s, 0.0 kB/s
| DEBUG: Actual Rootfs size:  4 /home/yocto/yocto-tegra/tegra-demo-distro/build/tmp/work/jetson_agx_xavier_devkit-oe4t-linux/tegra-espimage/1.0/rootfs.image_esp
| DEBUG: Actual Partition size: 67108864
| DEBUG: Executing mkfs.vfat -F 32 -I  /home/yocto/yocto-tegra/tegra-demo-distro/build/tmp/work/jetson_agx_xavier_devkit-oe4t-linux/tegra-espimage/1.0/deploy-tegra-espimage-image-complete/tegra-espimage-jetson-agx-xavier-devkit-20240823205838.esp
| mkfs.fat 4.2 (2021-01-31)
| /home/yocto/yocto-tegra/tegra-demo-distro/build/tmp/work/jetson_agx_xavier_devkit-oe4t-linux/tegra-espimage/1.0/rootfs.image_esp/*: No such file or directory
| WARNING: exit code 1 from a shell command.
ERROR: Task (/home/yocto/yocto-tegra/tegra-demo-distro/layers/meta-tegra/recipes-bsp/uefi/tegra-espimage.bb:do_image_esp) failed with exit code '1'

I am using the INHERIT += "mender-full" option.

What I am doing wrong?

Hello @y1thu,

I recommend you to follow meta-mender-community/meta-mender-tegra/meta-mender-tegra-jetpack5 at kirkstone · mendersoftware/meta-mender-community · GitHub instead.

In the KAS’s yaml file you can see we actually remove it as these devices don’t use u-boot in here:

MENDER_FEATURES_DISABLE:append = " mender-uboot"

Try adding this line in your local.conf, if it doesn’t work, please try following the README file from the repository instead.

Luis

Dear Luis,

I added this line to the layer.conf:

MENDER_FEATURES_DISABLE:append = " mender-uboot"

but still the same error message:

ERROR: Nothing PROVIDES 'u-boot' (but /home/yocto/git/yocto-heraeus/layers/meta/recipes-bsp/grub/grub-efi_2.12.bb, /home/yocto/git/yocto-heraeus/layers/meta-mender-core/recipes-bsp/grub/grub-efi-mender-precompiled_2.04.bb DEPENDS on or otherwise requires it)
u-boot was skipped: Either UBOOT_MACHINE or UBOOT_CONFIG must be set in the jetson-agx-xavier-devkit machine configuration.
NOTE: Runtime target 'grub-efi' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['grub-efi', 'u-boot']
ERROR: Required build target 'heraeus-ai-image' has no buildable providers.
Missing or unbuildable dependency chain was: ['heraeus-ai-image', 'tegra-espimage', 'grub-efi', 'u-boot']

Now I am building using the kas.yaml from here as suggested.
However this is for kirkstone. Do you have a modification that works with meta-tegra scarthgap-l4t-r35.x ?

Hello @lramirez ,

I did compile according to https://github.com/mendersoftware/meta-mender-community/tree/kirkstone/meta-mender-tegra/meta-mender-tegra-jetpack5 but the NVidia Jetson fails in the boot process with systemd-gpt-auto-generator: EFI loader partition unknown .

Also when building with yocto I normally obtain *rootfs.tegraflash.tar.gz however with kas the rootfs part is missing.

Can you maybe advice?

What artifacts are being produced? I’m curious if you are building the image suitable for the storage device you are using.

@lramirez not sure exactly what you mean. I just cloned the repo and used kas build kas/jetson-agx-xavier-devkit.yml, so essentially using the standards.

These are the standard settings in the other yaml’s:
IMAGE_FSTYPES:tegra = “tegraflash mender dataimg”

Also this is specified:
MENDER_FEATURES_ENABLE:append = " mender-growfs-data"

perhaps I need to change to something along these lines:

Mender partition settings

MENDER_STORAGE_DEVICE_BASE = “/dev/mmcblk0p”
MENDER_DATA_PART = “${MENDER_STORAGE_DEVICE_BASE}24”
MENDER_ROOTFS_PART_A = “${MENDER_STORAGE_DEVICE_BASE}1”
MENDER_ROOTFS_PART_B = “${MENDER_STORAGE_DEVICE_BASE}18”

Or can you comment what to use to flask the tar.gz? I am using this: ./doflash.sh

I just saw that there is since last week a scarthgap meta-mender-community version. I am testing this one now but also failing, see this pull request: scarthgap-l4t-r35.x kas build errors by tjhgit · Pull Request #403 · mendersoftware/meta-mender-community · GitHub

@lramirez any help with this? I progressed a lot meanwhile. Not I am stuck again. Why does A/B boot not work properly … See the pull request: scarthgap-l4t-r35.x kas build errors by tjhgit · Pull Request #403 · mendersoftware/meta-mender-community · GitHub

Hello @y1thu,

Quick question, does this PRs fix your problem or after applying them are you still stuck?

BR,
Luis

@lramirez as mentioned in my own PR, the problem is the A/B boot not working properly. Not sure how to install the nvbootctrl from NVidia in yocto or whether this is not needed.

This is the problem

root@jetson-agx-xavier-devkit:~# fw_printenv
ERR: could not identify current boot slot
mender_boot_part=UNKNOWN

And after changing the slot from 2 and back to 1 it hangs in EFI.

@lramirez , this issue with the slots also prevents yocto updates. See the logs of mender cloud instance after trying an update:

2024-09-05 17:37:24.541 +0000 UTC info: Running Mender client 4.0.4
2024-09-05 17:37:24.543 +0000 UTC info: Deployment with ID 1d1d43fa-ddd6-4ac5-abb0-0a857d50b055 started.
2024-09-05 17:37:24.549 +0000 UTC info: Sending status update to server
2024-09-05 17:37:25.342 +0000 UTC info: Installing artifact...
2024-09-05 17:37:25.568 +0000 UTC info: Update Module output (stderr): ERR: could not identify current boot slot
2024-09-05 17:37:25.57 +0000 UTC info: Update Module output (stderr): /usr/share/mender/modules/v3/rootfs-image: line 96: test: UNKNOWN: integer expression expected
2024-09-05 17:37:25.596 +0000 UTC info: Update Module output (stderr): Mounted root does not match boot loader environment (/dev/mmcblk0p2)!
2024-09-05 17:37:25.597 +0000 UTC error: Process returned non-zero exit status: Download: Update Module returned non-zero status: Process exited with status 1
2024-09-05 17:37:25.626 +0000 UTC info: Sending status update to server

@lramirez trying now this update procedure using nvidia ota tools as described thankfully by you. Nvidia Jetson Orin L4T image Update. This seems to work (with L4T 35.5), however it is real pitty that the yocto image of mender does not work with jetson.

Why is it so hard to integrate mender in yocto correctly?

Hello @y1thu,

Thanks for following up. Sadly most of the meta-tegra and mender integration are community maintained.

Let me ping @TheYoctoJester in here as he may add some light on this.

Best regards,
Luis

Hi @y1thu,

The problem is not so much Yocto nor Mender, but Nvidia Tegra. They constantly refuse to do things in an open way, or such that it aligns with somebody else. Hence if you walk off their proposed path using their Ubuntu “derivative”, you end up dealing with all their peculiarities. That’s why meta-tegra is so complex, it replicates much of that stuff.

With meta-mender and meta-mender-community being meant for general usability, combining it with meta-tegra can be somewhat painful then. Many thanks again to @zach-welch-aquabyte, @dwalkes, and everybody who contributes to this.

Concerning the given error message, I just commented over on GitHub: scarthgap-l4t-r35.x kas build errors by tjhgit · Pull Request #403 · mendersoftware/meta-mender-community · GitHub

The logic needs to make sure the partition which is about to be modified is actually the one which is expected, e.g. not in active use. Hence this check.

Please drop the output of the mount command so we can see what is actually mounted given which way, so we can figure out how to move forward.

Greets,
Josef

Hi @TheYoctoJester . I replied to your suggestion. The root cause must be someplace else. I also have issues with A/B booting with x86_64 yocto build with mender. When issuing the grub commands:

grub-mender-grubenv-set mender_boot_part 3
grub-mender-grubenv-set mender_boot_part_hex 3
reboot

from here.

It gets stuck in boot :frowning:

It is actually worse than that: yocto with mender you can boot Nvidia jetson agx xavier exactly 2x times until it gets stuck in PXE boot. First reboot switches to B (even when not told to do so). Next boot IPXE.

Same yocto without any mender layers works perfectly.

@TheYoctoJester interestingly however I did just ignore the integration test above ( that mender proposes ) using x86_64 yocto with mender and uploaded .mender yocto os build file and deployed on the device. Magic happens: it boots to root B installs and it works. This process is repeatable. So maybe the above grub commands are just wrong (or starting at the wrong offset)?

This however does not solve the NVIDIA Jetson issue. Just noticed however today that 2 days ago a new commit in meta-tegra regarding nvbootctrl. So testing this now. See if that resolves issues => NO IT DOES NOT. Still same issue 2x reboots and Nvidia jetson gets stuck in IPXE boot.

Additionally what I find strange, that is even if I do enable IMAGE_INSTALL:append = " setup-nv-boot-control" , I do not have the command available … meaning the install did not take place.
Debugging this:

root@jetson-agx-xavier-devkit:/# setup-nv-boot-control
ERR: cannot store EFI variable - ESP partition not mounted
ERR: cannot store EFI variable - ESP partition not mounted
/usr/bin/setup-nv-boot-control: line 30: /etc/nv_boot_control.conf: No such file or directory
chmod: cannot operate on dangling symlink '/etc/nv_boot_control.conf'

root@jetson-agx-xavier-devkit:/# tegra-boardspec
2888-400-0004-L.0-1-2