Jetson Orin NX & Jetpack 5 Integration Issues

Board Integrations Yocto Project
, , ,
1

Board description

Custom baseboard supporting NVIDIA Jetson Orin Nano/NX

General

Currently, I am working on integrating Mender into an existing functional Yocto configuration for a Jetson Orin Nano Board. As we all know, integrating Mender into new Tegra boards is always fun, and since I am currently stuck, I wanted to ask for advice here. I have already tried to adopt the changes from @Austriker (as seen in his thread), but I am encountering errors in my build.

Versions

Jetpack 5
Yocto Scarthgap

Yocto Config

meta-mender - branch: scarthgap GitHub - mendersoftware/meta-mender: Yocto Project meta layer for the Mender client
meta-mender-community - branch: jetson-orin-nx-support GitHub - Austriker/meta-mender-community: Community supported integration layers for Mender on various boards
meta-tegra - branch: scarthgap-l4t-r35.x GitHub - OE4T/meta-tegra: BSP layer for NVIDIA Jetson platforms, based on L4T

local.conf

require conf/machine/jetson-orin-nano-devkit-nvme.conf

CONF_VERSION = "2"
INIT_MANAGER = "systemd"

INHERIT += "mender-full"
MENDER_ARTIFACT_NAME = "NeverEndingStory"

USE_REDUNDANT_FLASH_LAYOUT_DEFAULT = "1"
EMMC_SIZE = "34359738368"

MENDER_DATA_PART_NUMBER = "17"
MENDER_STORAGE_DEVICE = "/dev/nvme0n1"

INHERIT:remove = "tegra-support-sanity distro_layer_buildinfo"
INHERIT += "tegra-mender-setup"
MENDER_FEATURES_ENABLE:append = " mender-growfs-data"
MENDER_FEATURES_DISABLE:append = " mender-uboot"
MENDER_STORAGE_TOTAL_SIZE_MB = "121440"
IMAGE_FSTYPES:tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES:pn-tegra-minimal-initramfs:tegra = "${INITRAMFS_FSTYPES}"
IMAGE_FSTYPES:pn-tegra-initrd-flash-initramfs:tegra = "${TEGRA_INITRD_FLASH_INITRAMFS_FSTYPES}"

bblayers.conf

BBLAYERS ?= " \
  ${TOPDIR}/../poky/meta \
  ${TOPDIR}/../poky/meta-yocto-bsp \
  ${TOPDIR}/../meta-openembedded/meta-oe \
  ${TOPDIR}/../meta-openembedded/meta-multimedia \
  ${TOPDIR}/../meta-openembedded/meta-networking \
  ${TOPDIR}/../meta-openembedded/meta-python \
  ${TOPDIR}/../meta-tailscale \
  ${TOPDIR}/../meta-tegra \
  ${TOPDIR}/../meta-mender/meta-mender-core \
  ${TOPDIR}/../meta-mender-community/meta-mender-tegra \
  "

Errors

When I try to start a build, I quickly receive the following error:

ERROR: No recipes in default available for:
  /workspaces/operating-system/build-tegra/../meta-mender-community/meta-mender-tegra/recipes-bsp/uefi/l4t-launcher-rootfs-ab-config_%.bbappend
  /workspaces/operating-system/build-tegra/../meta-mender-community/meta-mender-tegra/recipes-kernel/linux/linux-jammy-nvidia-tegra_5.15%.bbappend

Since its “only” for JP 6 I am going to remove those two bbappends, as the git commit states

“[JP 6 only: Use new recipe for uefi config”

After that the image successfully builds in Yocto.

However when flashing with the ./initflashscript, I receive the following errors:

./initrd-flash output

Starting at 2025-01-28T00:28:06+01:00
Machine:       p3768-0000-p3767-0001
Rootfs device: nvme0n1p1
Found Jetson device in recovery mode at USB 2-2
== Step 1: Signing binaries at 2025-01-28T00:28:06+01:00 ==
cp: cannot stat 'signed/*': No such file or directory
cp: cannot stat 'signed/flash.xml.tmp': No such file or directory
sed: can't read flash.xml.tmp: No such file or directory
== Step 2: Boot Jetson via RCM at 2025-01-28T00:28:26+01:00 ==
Found Jetson device in recovery mode at USB 2-2
== Step 3: Sending flash sequence commands at 2025-01-28T00:28:29+01:00 ==
Waiting for USB storage device flashpkg from b8440206.......[/dev/sdb]
Device size in blocks: 262144
Unmounted /dev/sdb.
== Step 4: Writing partitions on external storage device at 2025-01-28T00:28:54+01:00 ==
Waiting for USB storage device nvme0n1 from b8440206...[/dev/sdb]
Traceback (most recent call last):
  File "/home/pkoegel/tegra/nvflashxmlparse", line 483, in <module>
    ret = main()
  File "/home/pkoegel/tegra/nvflashxmlparse", line 408, in main
    rewrite_layout(args.filename, args.rewrite_contents_from.split(','), outf)
  File "/home/pkoegel/tegra/nvflashxmlparse", line 266, in rewrite_layout
    maptree = ET.parse(mapfile)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 595, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
Traceback (most recent call last):
  File "/home/pkoegel/tegra/nvflashxmlparse", line 483, in <module>
    ret = main()
  File "/home/pkoegel/tegra/nvflashxmlparse", line 426, in main
    layout = PartitionLayout(args.filename)
  File "/home/pkoegel/tegra/nvflashxmlparse", line 142, in __init__
    tree = ET.parse(configfile)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 595, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
No partition definitions found in initrd-flash.xml
ERR: write failure to external storage at 2025-01-28T00:28:57+01:00

Console

[    8.203195] nvdla 158c0000.nvdla1: deferred probe timeout, ignoring dependency
[    8.209600] nvdla: probe of 158c0000.nvdla1 failed with error -110
[    8.216876] irq: IRQ308: trimming hierarchy from :pmc@c360000
[    8.221649] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    8.229498] tegra-se-nvhost 15810000.se: Adding to iommu group 52
[    8.234974] tegra-se-nvhost 15810000.se: initialized
[    8.241106] tegra-se-nvhost 15810000.se: tegra_se_probe: complete
[    8.245706] tegra-se-nvhost 15820000.se: Adding to iommu group 53
[    8.252202] tegra-se-nvhost 15820000.se: initialized
[    8.257862] tegra-se-nvhost 15820000.se: tegra_se_probe: complete
[    8.262921] tegra-se-nvhost 15840000.se: Adding to iommu group 54
[    8.269449] tegra-se-nvhost 15840000.se: initialized
[    8.275540] tegra-se-nvhost 15840000.se: tegra_se_probe: complete
[    8.280827] tegra_actmon d230000.actmon: in actmon_register()...
[    8.286298] tegra_actmon d230000.actmon: bwmgr_disable = 1
[    8.292250] tegra_actmon d230000.actmon: initialization Completed for the device mc_all
[    8.299806] hvc_sysfs: hypervisor is not present
[    8.304216] clk: Disabling unused clocks
[    8.346025] ALSA device list:
[    8.346122]   No soundcards found.
[    8.347184] Freeing unused kernel memory: 4416K
[    8.348072] Run /init as init process
128+0 records in
128+0 records out
mke2fs 1.47.0 (5-Feb-2023)
Found /tmp/flashpkg.ext4
[    8.457929] Mass Storage Function, version: 2009/09/11
[    8.458083] LUN: removable file: (no medium)
[    8.458946] tegra-xudc 3550000.xudc: EP 0 (type: ctrl, dir: out) enabled
Exported /tmp/flashpkg.ext4 as flashpkg
Waiting for host to connect...[    8.967129] mmc1: SDHCI controller on 3400000.sdhci [3400000.sdhci] using ADMA 64-bit
[    8.978011] tegra-xudc 3550000.xudc: EP 3 (type: bulk, dir: in) enabled
[    8.978224] tegra-xudc 3550000.xudc: EP 2 (type: bulk, dir: out) enabled
[    8.996027] usb 1-2.2: new full-speed USB device number 6 using tegra-xusb
[    8.996751] usb 1-2.2: Device not responding to setup address.
[    9.204754] usb 1-2.2: Device not responding to setup address.
[    9.412177] usb 1-2.2: device not accepting address 6, error -71
[connected]
Waiting for host to disconnect...[    9.492376] usb 1-2.2: new full-speed USB device number 7 using tegra-xusb
[    9.493419] usb 1-2.2: Device not responding to setup address.
[    9.700751] usb 1-2.2: Device not responding to setup address.
[    9.908188] usb 1-2.2: device not accepting address 7, error -71
[    9.909444] usb 1-2-port2: unable to enumerate USB device
[    9.988182] usb 1-2.5: new high-speed USB device number 8 using tegra-xusb
[   10.093098] usb 1-2.5: New USB device found, idVendor=0424, idProduct=2740, bcdDevice= 2.00
[   10.093337] usb 1-2.5: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   10.093541] usb 1-2.5: Product: Hub Controller
[   10.093679] usb 1-2.5: Manufacturer: Microchip Tech
[   11.600451] tegra-xudc 3550000.xudc: ep 3 disabled
[   11.600616] tegra-xudc 3550000.xudc: ep 2 disabled
[disconnected]
[   12.471295] tegra-xudc 3550000.xudc: ep 0 disabled
[   12.478310] loop: module loaded
[   12.479461] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Processing: bootloader
Processing: extra-pre-wipe
Processing: erase-mmc
Processing: export-devices nvme0n1
[   12.488807] Mass Storage Function, version: 2009/09/11
[   12.488958] LUN: removable file: (no medium)
[   12.489604] tegra-xudc 3550000.xudc: EP 0 (type: ctrl, dir: out) enabled
[   12.930466] tegra-xudc 3550000.xudc: EP 3 (type: bulk, dir: in) enabled
[   12.930680] tegra-xudc 3550000.xudc: EP 2 (type: bulk, dir: out) enabled
[   14.888183] tegra-xudc 3550000.xudc: ep 3 disabled
[   14.888350] tegra-xudc 3550000.xudc: ep 2 disabled
[   15.499102] tegra-xudc 3550000.xudc: ep 0 disabled
Processing: extra
Processing: reboot
Waiting for boot device programming to complete

Is there anyone who faced similar problems and could resolve them?

Cheers!

2

Hey @pkgl ,

Out of curiosity why do you need jetpack 5 ?

I indeed only tested with jetpack 6.

It’s probably because you need uboot instead of uefi as a bootloader, since you removed the following below:

/workspaces/operating-system/build-tegra/../meta-mender-community/meta-mender-tegra/recipes-bsp/uefi/l4t-launcher-rootfs-ab-config_%.bbappend

Cheers

3

Hi @Austriker,

thanks for your reply :slightly_smiling_face:

Unfortunately, it is not possible for us to bump the version to Jetpack 6, as we have some dependencies that we cannot adjust quickly at the moment.

That’s really strange. Do you have any suggestions on how I can easily include uboot in my configuration?

Also, I was wondering if Nvidia provides a page to see which Nvidia L4T version is LTS or how long it will be supported. Unfortunately, I haven’t found anything about this, so maybe such a thing doesn’t exist, and the release cycle is rather arbitrary.

Thanks!

4

Hello @pkgl,
for Scarthgap and Jetpack 5 support, I would suggest to try going back some commits on meta-mender-community to at lest revert PR #427. This PR was made to support JP6 without taking jetpack 5 support into account.

I would say Jetpack 4.6 and 5.1 are LTS right now, but there is no official page showing this.

5

Hi poett1,

thank you for your response and the good idea. I have already tried to implement it. Unfortunately, the same problem persists. I noticed that as soon as I add the Mender layers (mender-core & mender-community), the initrd-flash.xml file is not included in the tegraflash.tar.gz.

I have looked through the recipes and tried to find out why it is not being copied, but so far, I haven’t been able to find anything.

6

You could try a bare kas build from meta-mender-community to make sure its not due to your distro:

kas build kas/jetson-orin-nano-devkit.yml

If that does not solve it, I would suggest starting with the AGX Orin kas build from scratch, just to make sure the Nano/NX machine config is not causing it.

7

Thanks @poett1 for your input! I have now made a few adjustments and even used the latest state of the Scarthgap branch.

renamed

meta-mender-tegra/recipes-kernel/linux/linux-jammy-nvidia-tegra_5.15%.bbappend →
meta-mender-community/meta-mender-tegra/recipes-kernel/linux/linux-tegra_%.bbappend

removed

meta-mender-tegra/recipes-bsp/uefi/l4t-launcher-rootfs-ab-config_%.bbappend

My current local.conf

require conf/machine/jetson-orin-nano-devkit-nvme.conf

CONF_VERSION = "2"
INIT_MANAGER = "systemd"
INHERIT += "rm_work"

INHERIT += "tegra-mender-setup"
INHERIT:remove = "tegra-support-sanity distro_layer_buildinfo"

MENDER_FEATURES_ENABLE:append = " mender-growfs-data"
MENDER_FEATURES_DISABLE:append = " mender-uboot"
IMAGE_FSTYPES:tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES:pn-tegra-minimal-initramfs:tegra = "${INITRAMFS_FSTYPES}"
IMAGE_FSTYPES:pn-tegra-initrd-flash-initramfs:tegra = "${TEGRA_INITRD_FLASH_INITRAMFS_FSTYPES}"

INHERIT += "mender-full"

EMMC_SIZE = "34359738368"
MENDER_STORAGE_DEVICE = "/dev/nvme0n1"

UBOOT_EXTLINUX = "1"
USE_REDUNDANT_FLASH_LAYOUT_DEFAULT = "1"

growfs-data isn’t working yet, but that’s not really a big deal for me since I’ve already defined the data partition anyway.

root@localhost:~# systemctl status mender-grow-data.service -l
× mender-grow-data.service - Mender service to grow data partition size
     Loaded: loaded (/usr/lib/systemd/system/mender-grow-data.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Thu 1970-01-01 00:00:28 UTC; 55 years 1 month ago
    Process: 439 ExecStart=/usr/bin/mender-resize-data-part (code=exited, status=1/FAILURE)
   Main PID: 439 (code=exited, status=1/FAILURE)
        CPU: 46ms

Jan 01 00:00:28 localhost systemd[1]: Starting Mender service to grow data partition size...
Jan 01 00:00:28 localhost mender-resize-data-part[451]: Error: Can't have overlapping partitions.
Jan 01 00:00:28 localhost systemd[1]: mender-grow-data.service: Main process exited, code=exited, status=1/FAILURE
Jan 01 00:00:28 localhost systemd[1]: mender-grow-data.service: Failed with result 'exit-code'.
Jan 01 00:00:28 localhost systemd[1]: Failed to start Mender service to grow data partition size.

Just tested the OTA functionality. Mender can place the .mender artifact on the unused slot without any problems and also restarts the device, but it boots from the same slot again. Mender also tells me the update was “successful”.

However, when I use

nvbootctrl set-active-boot-slot n

#or

fw_setenv upgrade_available 1
fw_setenv bootcount 0

fw_setenv mender_boot_part 2
fw_setenv mender_boot_part_hex 2

and perform a manual reboot, I can boot into the slot without any issues.

Now I don’t understand why the switch doesn’t work automatically after a Mender update. I will check the Mender update process again tomorrow, maybe I’ll find a logical error related to Jetpack 6.

I also was wondering, has anyone else had the problem with the systemd-networkd-wait-online.service timing out every time on Tegra Linux? I can only explain it because of the dummy0 interface, as it degrades every time. According to some GitHub issues, it is also safe to simply disable the interface in the kernel, which I will probably do…

root@localhost:~# networkctl
IDX LINK       TYPE     OPERATIONAL SETUP
  1 lo         loopback carrier     unmanaged
  2 dummy0     ether    degraded    configuring
  3 eth0       ether    routable    configured
  4 can0       can      off         unmanaged
  5 tailscale0 none     routable    unmanaged

5 links listed.

× systemd-networkd-wait-online.service - Wait for Network to be Configured
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-wait-online.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Sun 2025-02-02 02:38:54 UTC; 27s ago
       Docs: man:systemd-networkd-wait-online.service(8)
    Process: 557 ExecStart=/usr/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
   Main PID: 557 (code=exited, status=1/FAILURE)
        CPU: 28ms

Feb 02 02:36:54 localhost systemd[1]: Starting Wait for Network to be Configured...
Feb 02 02:38:54 localhost systemd-networkd-wait-online[557]: Timeout occurred while waiting for network connectivity.
Feb 02 02:38:54 localhostsystemd [1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Feb 02 02:38:54 localhost systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Feb 02 02:38:54 localhost systemd[1]: Failed to start Wait for Network to be Configured.

:rat:

8

Sounds good so far. It would be great if you could commit your changes to meta-mender-community after you finally got it working (please make sure to sign off your commits). Depending on the changes, it might make sense to create a Scarthgap Jetpack 5 branch in meta-mender-community or merge it into the scarthgap branch.

Regarding your systemd-networkd-wait-online.service issue: Yes, I faced the same problem. It might help to add -i INTERFACE or --ignore=INTERFACE to whitelist/blacklist the interfaces for systemd-networkd-wait-online.service. I also have an additional problem with eth0 and eth1 switching regularly when using an additional USB Ethernet interface. So I might have to add a custom script that looks for the network interface driver first…

9

Yes, I agree that we should create a separate branch for Jetpack 5, assuming the “normal” scarthgap branch is intended for Jetpack 6. Currently, in meta-tegra, we have a similar setup with the branch scarthgap-l4t-r35.x - I am sure I am not the only one using Jetpack 5.

Regarding the issue with the systemd-networkd-wait-online.service, since I disabled the dummy0 interface in the kernel, I haven’t had any problems with the service.