NVIDIA Tegra Jetson Xavier NX

dwalkes · October 4, 2020, 6:20pm

The official Mender documentation explains how Mender works. This is a board-specific complement to the official documentation.

Device description

NVIDIA Xavier NX hardware with computer vision, GPU and multimedia support.

URL: https://www.nvidia.com/en-us/autonomous-machines/embedded-systems-dev-kits-modules/
Wiki: Jetson Xavier NX - eLinux.org

Support level

Yocto Project	Build	Runtime
dunfell (3.1)

Prerequisites

A supported Linux distribution and dependencies installed on your workstation/laptop as described in the Yocto Mega Manual
- NOTE. Instructions depend on which Yocto version you intend to use.

Setup Yocto Environment

Use the tegra-demo-distro to demonstrate a working mender configuration.

Please use the setup-env script in tegra-demo-distro to setup and configure the tegrademo-mender distribution. Use the branch corresponding to the yocto version you plan to target. For instance, for the dunfell branch and NVIDIA L4T release 32.4.3, use these commands from the root of the cloned tree:

git checkout dunfell-l4t-r32.4.3
git submodule update --init

Then use

source ./setup-env --distro tegrademo-mender --machine jetson-xavier-nx-devkit

To setup your build environment for the Jetson AGX Xavier development kit.

See the OE4T/meta-tegra wiki for more information about the meta-tegra project.

Setup build environment

Initialize the build environment:

source ./setup-env --distro tegrademo-mender --machine jetson-xavier-nx-devkit

or if you’ve already setup your initial environment you can use the simpler option:

source ./setup-env

Building the image

You can now proceed with building an image:

bitbake demo-image-base

Using the build output

After a successful build, the images and build artifacts are placed in tmp/deploy/images/jetson-xavier-nx-devkit.

The disk image package for Jetson has a name like demo-image-base-jetson-xavier-nx-devkit.tegraflash.tar.gz. This file can be used along with recovery mode to initially provision the device. To enter recovery mode:

Power off
Place in recovery mode by shorting REC pin to GND. For carrier board revision A01 these are pins 9 and 10 on J14. See picture at this link.
Power on

You should see your NVIDIA device listed in the lsusb device output.

Next, extract the .tar.gz file and then run the ./doflash.sh script inside to flash the build to the device. The script at repos/meta-mender-community/meta-mender-tegra/scripts/deploy.sh can be customized to automate this step if desired. See the instructions on the OE4T wiki for additional details.

On the other hand, if you already have Mender running on your device and want to deploy a rootfs update using this build, you should use the Mender Artifact files, which have .mender suffix. You can either deploy this Artifact in managed mode with the Mender server (upload it under Releases in the server UI) or by using the Mender client standalone mode.

References

meta-tegra: The base BSP layer with support for NVIDIA tegra.
meta-mender-tegra layer in meta-mender-community: Customization for mender on tegra, including u-boot and partitioning. See OE4T layer at this link for any changes or to file issues against mender and NVIDIA support.
The official Mender documentation explains how Mender works. This is simply a board-specific complement to the official documentation.

Known issues

Let us know if you find one!

Sibert · November 12, 2020, 11:36am

I have integrated the meta-mender and meta-mender-community in my Yocto build. It works for the Jetson Nano, but when I try to build for the Xavier I get following error when flashing the build using the doflash.sh script:
End sector for APP_b, expected at: 30777310, actual: 59729703

Do you have any idea what the problem could possibly be?

dwalkes · November 12, 2020, 1:49pm

You might be seeing the same issue discussed on the meta-tegra issues forum, see Create new page · OE4T/tegra-demo-distro Wiki · GitHub and when using mender and xavier nx rootfs is too large · Issue #453 · OE4T/meta-tegra · GitHub

anishmonachan · March 15, 2021, 12:50pm

@dwalkes Thank you for this article.

I’ve managed to use this article to create my first image. I couldn’t test the mender local update because the uncompressed size of mender file is insanely high 11.5GB.

dwalkes · March 15, 2021, 2:23pm

@anishmonachan which branch are you using? What size is your rootfs partition? This is probably because the rootfs size needs to be decreased. See Customization of Storage Size and root filesystem size on Mender Builds · OE4T/tegra-demo-distro Wiki · GitHub

anishmonachan · March 15, 2021, 3:16pm

I’m using dunfell-l4t-r32.4.3 branch.And the solution suggested does the job. Thank you very much @dwalkes.

anishmonachan · March 15, 2021, 3:38pm

@dwalkes I’ve encountered another issue while attempting both the local and OTA update . Do you know what I miss here

/var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update: line 29: nvbootctrl: command not found
chroot: failed to run command '/usr/sbin/nv_update_engine': No such file or directory
ERR: bootloader update failed

INFO[0029] All bytes were successfully written to the new partition
INFO[0029] The optimized block-device writer wrote a total of 3892 frames, where 0 frames did need to be rewritten (i.e., skipped)
INFO[0030] Wrote 4080218112/4080218112 bytes to the inactive partition
INFO[0030] Enabling partition with new image installed to be a boot candidate: 33
INFO[0030] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
/usr/bin/setup-nv-boot-control: line 19: /etc/nv_boot_control.conf: No such file or directory
chmod: cannot operate on dangling symlink ‘/etc/nv_boot_control.conf’
ERR: bootloader update failed

---------- end of script output 
ERRO[0030] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 
Rolling back Artifact...
INFO[0031] Rolling back to the active partition: (1).   
ERRO[0031] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1

dwalkes · March 15, 2021, 9:27pm

anishmonachan:

/var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update: line 29: nvbootctrl: command not found
chroot: failed to run command '/usr/sbin/nv_update_engine': No such file or directory

Hmm that’s definitely a problem. It should be added from https://github.com/OE4T/meta-tegra/blob/dunfell-l4t-r32.4.3/recipes-bsp/tegra-binaries/tegra-redundant-boot-base_32.4.3.bb#L28 which should be included from tegra-bup-payload at https://github.com/OE4T/meta-tegra/blob/dunfell-l4t-r32.4.3/recipes-bsp/tegra-bup-payload/tegra-bup-payload_1.0.bb#L43 which should be included based on https://github.com/mendersoftware/meta-mender-community/blob/dunfell/meta-mender-tegra/recipes-mender/mender-client/mender-client_%25.bbappend#L1 I’ll kick of a build now to see if I can reproduce.

dwalkes · March 15, 2021, 11:26pm

I haven’t tried running the build but I didn’t reproduce the missing nv_update_engine

find tmp/work/jetson_xavier_nx_devkit-oe4t-linux/demo-image-base/1.0-r0/rootfs/ -name "nv_update_engine"
tmp/work/jetson_xavier_nx_devkit-oe4t-linux/demo-image-base/1.0-r0/rootfs/usr/sbin/nv_update_engine

Build configuration is:

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-oe4t-linux"
MACHINE              = "jetson-xavier-nx-devkit"
DISTRO               = "tegrademo-mender"
DISTRO_VERSION       = "3.1+snapshot"
TUNE_FEATURES        = "aarch64 armv8a crc"
TARGET_FPU           = ""
meta                 = "HEAD:d20ef1f5a5de7820b3e9b7f539b51f94954e9cd5"
meta-tegra
contrib              = "HEAD:19887a87d109493bf27a8c78b51e711ff282505f"
meta-oe
meta-python
meta-networking
meta-filesystems     = "HEAD:346681e7bf9c78008a845fc89031be4fd4ceb3a1"
meta-virtualization  = "HEAD:89abc62b47f6f20db9d00a8ec9b2c1b6b60ac3f9"
meta-mender-core     = "HEAD:bd5a6de7f3e79f1bb37e18d02595ff31d22584ae"
meta-mender-tegra    = "HEAD:bd63a1e6de12f7f96dfe2705599c517dac049e91"
meta-tegra-support
meta-demo-ci
meta-tegrademo       = "dunfell-l4t-r32.4.3:50c9515827c2afef1eea4a2986cb2b0dd127431c"

anishmonachan · March 16, 2021, 1:32pm

Thank you @dwalkes for directing me towards the cause of the issue. After an investigation, I’ve realized that there is a tegra-bup-payload_%.bbappend recipe that does

RDEPENDS_${PN}_remove = "tegra-redundant-boot" RDEPENDS_${PN}_append = " tegra-boot-tools-updater"

This is removed. Now the above issue is resolved. But yet I didn’t manage to install a local update. I’ve the following error:

NFO[0000] Mender running on partition: /dev/mmcblk0p1
INFO[0000] Start updating from local image file: [test-jetson-xavier.mender]
Installing Artifact of size 389575680…
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Opening device “/dev/mmcblk0p42” for writing
INFO[0000] Native sector size of block device /dev/mmcblk0p42 is 512 bytes. Mender will write in chunks of 1048576 bytes
… - 100 %
INFO[0021] All bytes were successfully written to the new partition
INFO[0021] The optimized block-device writer wrote a total of 3892 frames, where 0 frames did need to be rewritten (i.e., skipped)
INFO[0021] Wrote 4080218112/4080218112 bytes to the inactive partition
INFO[0021] Enabling partition with new image installed to be a boot candidate: 42
INFO[0022] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
mount: /tmp/nvbup.w2OC5i: special device 1800
1800 does not exist.
ERR: Missing /opt/ota_package directory in installed rootfs

---------- end of script output
ERRO[0022] ArtifactInstall_Leave script failed: statescript: error executing ‘ArtifactInstall_Leave_80_bl-update’: 1 : exit status 1
Rolling back Artifact…
INFO[0022] Rolling back to the active partition: (1).
ERRO[0022] statescript: error executing ‘ArtifactInstall_Leave_80_bl-update’: 1 : exit status 1
root@jetson-xavier:/data#

Note:
For the ERR: Missing /opt/ota_package directory in installed rootfs , directory /opt/ota_package does exist

dwalkes · March 16, 2021, 3:21pm

I don’t think you should have this on the dunfell-r32.4.3 branch. Here’s what I see:

dan@yocto:/build/tegra-demo-distro-xavernx/layers$ find -L . -name "tegra-bup-payload*"
./meta-tegra/recipes-bsp/tegra-bup-payload
./meta-tegra/recipes-bsp/tegra-bup-payload/tegra-bup-payload_1.0.bb
dan@yocto:/build/tegra-demo-distro-xavernx/layers$

This does exist on the master branch though:

dan@yocto:/build/tegra-demo-distro-xavernx/layers$ find -L . -name "tegra-bup-payload*"
./meta-tegra-support/recipes-bsp/tegra-bup-payload
./meta-tegra-support/recipes-bsp/tegra-bup-payload/tegra-bup-payload_%.bbappend
./meta-tegra/recipes-bsp/tegra-bup-payload
./meta-tegra/recipes-bsp/tegra-bup-payload/tegra-bup-payload_1.0.bb
dan@yocto:/build/tegra-demo-distro-xavernx/layers$

So I suspect you are building a branch other than dunfell-l4t-r32.4.3 of tegra-demo-distro. Please try

git checkout dunfell-l4t-r32.4.3
git submodule update
rm -rf build

Then retry with this branch. Check your build configuration with the config I shared in the message above, it should match.

anishmonachan · March 16, 2021, 3:38pm

@dwalkes I confirm that I’m using the configuration you have given. I’ve added this bbappend recipe when I had a flashing issue. But it wasn’t necessary.

Now, with the configuration you have given, I’ve this issue while I attempt a local update. Note I haven’t tested an ota update yet.

INFO[0022] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
mount: /tmp/nvbup.w2OC5i: special device 1800
1800 does not exist.
ERR: Missing /opt/ota_package directory in installed rootfs

anishmonachan · March 17, 2021, 12:56pm

Just found out that this line in the state script is causing the bug. It outputs /dev/mmcblkop42 /dev/mmcblk0p42
That means: the boot part twice. I’ll test the local update and post updates

github.com

mendersoftware/meta-mender-community/blob/bd63a1e6de12f7f96dfe2705599c517dac049e91/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script#L35


    nv_update_engine --enable-ab
    if [ ! "$?" -eq 0 ]; then
        echo "ERR: could not enable A/B update by using nv_update_engine" >&2
        exit 1
    fi
fi
current_slot=`nvbootctrl get-current-slot`
echo "Current boot slot: $current_slot"
otherslot=`expr 1 - $current_slot`
cfglbl="\"RootfsPart${LABELCHARS:$otherslot:1}\""
devnam=`grep -h "$cfglbl:" /etc/mender/mender.conf /var/lib/mender/mender.conf | cut -d: -f2 | cut -d, -f1 | tr -d '" '`
if [ -z "$devnam" ]; then
    echo "ERR: could not determine device name for boot slot $otherslot" >&2
    exit 1
fi
mnt=`mktemp -d -t nvbup.XXXXXX`
if [ -z "$mnt" -o ! -d "$mnt" ]; then
    echo "ERR: could not create directory for mounting install partition" >&2
    exit 1
fi
# Support delta updates by mounting the rootfs read only.

anishmonachan · March 17, 2021, 12:59pm

I’ve got this error now,

n (i.e., skipped) 
INFO[0022] Wrote 4080218112/4080218112 bytes to the inactive partition 
INFO[0022] Enabling partition with new image installed to be a boot candidate: 42 
INFO[0023] Collected output (stderr) while running script /var/lib/mender/scripts/ArtifactInstall_Leave_80_bl-update
ERR: bootloader update failed

---------- end of script output 
ERRO[0023] ArtifactInstall_Leave script failed: statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1 
Rolling back Artifact...
INFO[0023] Rolling back to the active partition: (1).   
ERRO[0023] statescript: error executing 'ArtifactInstall_Leave_80_bl-update': 1 : exit status 1

dwalkes · March 17, 2021, 2:45pm

What is the content of your mender.conf?
What do you see if you try running the state script yourself with bash -x? Is there any output from nv_update_engine at https://github.com/mendersoftware/meta-mender-community/blob/bd63a1e6de12f7f96dfe2705599c517dac049e91/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script#L68 which describes the error?

anishmonachan · March 17, 2021, 2:58pm

The mender.conf has following content.

{
“RootfsPartA”: “/dev/mmcblk0p1”,
“RootfsPartB”: “/dev/mmcblk0p42”
}

I’m investigating the issue. What I tried is to mount the /dev/mmcblk0p42 to nvbup.XXXXXX. Then I tried the command chroot "${mnt}" /usr/sbin/nv_update_engine --install no-reboot. I’ve following error

root@jetson-xavier:/data#  chroot "${mnt}" /usr/sbin/nv_update_engine --install no-reboot
Nvidia A/B-Redundancy Update tool Version 1.2
null input file!
Init SMD partition failed!
null input file!
A/B has been disabled. Need to enable A/B.

anishmonachan · March 17, 2021, 3:11pm

+ mnt=
+ LABELCHARS=AB
+ COPY_MACHINE_ID=yes
+ echo 'Installing NVIDIA bootloader update payload'
Installing NVIDIA bootloader update payload
++ nvbootctrl get-number-slots
+ num_slots=2
+ '[' 2 '!=' 2 ']'
++ nvbootctrl get-current-slot
+ current_slot=0
+ echo 'Current boot slot: 0'
Current boot slot: 0
++ expr 1 - 0
+ otherslot=1
+ cfglbl='"RootfsPartB"'
++ grep -h '"RootfsPartB":' /etc/mender/mender.conf
++ cut -d, -f1
++ tr -d '" '
++ cut -d: -f2
+ devnam=/dev/mmcblk0p42
+ '[' -z /dev/mmcblk0p42 ']'
++ mktemp -d -t nvbup.XXXXXX
+ mnt=/tmp/nvbup.zFV3Ko
+ '[' -z /tmp/nvbup.zFV3Ko -o '!' -d /tmp/nvbup.zFV3Ko ']'
+ mount -o ro /dev/mmcblk0p42 /tmp/nvbup.zFV3Ko
+ '[' '!' -d /tmp/nvbup.zFV3Ko/opt/ota_package ']'
+ mount --bind /sys /tmp/nvbup.zFV3Ko/sys
+ mount --bind /proc /tmp/nvbup.zFV3Ko/proc
+ mount --bind /dev /tmp/nvbup.zFV3Ko/dev
+ mount -t tmpfs tmpfs /tmp/nvbup.zFV3Ko/run
+ '[' -L /tmp/nvbup.zFV3Ko/etc/nv_boot_control.conf -a -x /tmp/nvbup.zFV3Ko/usr/bin/setup-nv-boot-control ']'
+ mkdir -p /tmp/nvbup.zFV3Ko/run/tegra-nv-bootctrl
+ chroot /tmp/nvbup.zFV3Ko /usr/bin/setup-nv-boot-control
+ chroot /tmp/nvbup.zFV3Ko /usr/sbin/nv_update_engine --install no-reboot
Nvidia A/B-Redundancy Update tool Version 1.2
current slot 0
Start updating slot 1
Tegra User Block Device: /dev/disk/by-partlabel
Tegra Boot Block Device: /dev/mmcblk0boot0
HEADER: MAGIC NVIDIA__BLOB__V2
HEX_VALUE 131072
BLOB_SIZE 44147958
HEADER_SIZE 48
NUMBER_OF_ELEMENTS 19
HEADER_TYPE 0
UNCOMP_SIZE 44147958
MB1_RATCHET_LV 0
MTS_RATCHET_LV 0
ROLLBACK_FUSE_LV 0
config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 2888-400-0001-J.0-1-2-jetson-xavier-mmcblk0p1
Can't find matching TN Spec in OTA Blob!
OTA Blob update failed. Status: 3
/usr/sbin/nv_bootloader_payload_updater --no-dependent-partition failed.
Slot 1 update failed
+ echo 'ERR: bootloader update failed'
ERR: bootloader update failed
+ cleanup
+ '[' -n /tmp/nvbup.zFV3Ko ']'
+ for d in sys proc dev run
+ mountpoint -q /tmp/nvbup.zFV3Ko/sys
+ umount /tmp/nvbup.zFV3Ko/sys
+ for d in sys proc dev run
+ mountpoint -q /tmp/nvbup.zFV3Ko/proc
+ umount /tmp/nvbup.zFV3Ko/proc
+ for d in sys proc dev run
+ mountpoint -q /tmp/nvbup.zFV3Ko/dev
+ umount /tmp/nvbup.zFV3Ko/dev
+ for d in sys proc dev run
+ mountpoint -q /tmp/nvbup.zFV3Ko/run
+ umount /tmp/nvbup.zFV3Ko/run
+ mountpoint -q /tmp/nvbup.zFV3Ko
+ umount /tmp/nvbup.zFV3Ko
+ rmdir /tmp/nvbup.zFV3Ko
+ exit 1

dwalkes · March 19, 2021, 1:58pm

@anishmonachan looks like there’s a new FAB version of the Xavier NX devkit, see related discussion in Mender update fails on Xavier NX · Issue #52 · OE4T/tegra-demo-distro · GitHub. I expect the workaround at Mender update fails on Xavier NX · Issue #52 · OE4T/tegra-demo-distro · GitHub should work if you replace the fab=300 with fab=400. It looks like we need to add this into meta-tegra/jetson-xavier-nx-devkit.conf at dunfell-l4t-r32.4.3 · OE4T/meta-tegra · GitHub. Give it a shot and if this works I’ll include in some tegra demo distro updates we need to make for mender.

anishmonachan · March 19, 2021, 2:30pm

@dwalkes thanks for suggesting another solution.

Also, I’d like to add it is statescript that caused the failure. After debugging the state script, I managed to have it working.
As I said before, the following line was causing the issue. It outputs wrong value than expected.

github.com

mendersoftware/meta-mender-community/blob/bd63a1e6de12f7f96dfe2705599c517dac049e91/meta-mender-tegra/recipes-mender/tegra-state-scripts/files/redundant-boot-install-script#L35


    nv_update_engine --enable-ab
    if [ ! "$?" -eq 0 ]; then
        echo "ERR: could not enable A/B update by using nv_update_engine" >&2
        exit 1
    fi
fi
current_slot=`nvbootctrl get-current-slot`
echo "Current boot slot: $current_slot"
otherslot=`expr 1 - $current_slot`
cfglbl="\"RootfsPart${LABELCHARS:$otherslot:1}\""
devnam=`grep -h "$cfglbl:" /etc/mender/mender.conf /var/lib/mender/mender.conf | cut -d: -f2 | cut -d, -f1 | tr -d '" '`
if [ -z "$devnam" ]; then
    echo "ERR: could not determine device name for boot slot $otherslot" >&2
    exit 1
fi
mnt=`mktemp -d -t nvbup.XXXXXX`
if [ -z "$mnt" -o ! -d "$mnt" ]; then
    echo "ERR: could not create directory for mounting install partition" >&2
    exit 1
fi
# Support delta updates by mounting the rootfs read only.

dwalkes · March 19, 2021, 3:55pm

Thanks @anishmonachan not sure I understand completely but I’m glad you have it working.

If you could share the detail of the change you made I’d appreciate it. The bash -x output above looks like it’s correctly mapping the devnam at that line:

++ grep -h '"RootfsPartB":' /etc/mender/mender.conf
++ cut -d, -f1
++ tr -d '" '
++ cut -d: -f2
+ devnam=/dev/mmcblk0p42

then failing due to the missing fab reference:

config COMPATIBLE_SPEC not found in /etc/nv_boot_control.conf
Device TN Spec: 2888-400-0001-J.0-1-2-jetson-xavier-mmcblk0p1
Can't find matching TN Spec in OTA Blob!
OTA Blob update failed. Status: 3

Topic		Replies	Views
NVIDIA Tegra AGX Xavier Yocto Project yocto , nvidia , dunfell	0	1819	October 4, 2020
NVIDIA Tegra Jetson Nano Yocto Project yocto , nvidia , warrior	33	11317	May 2, 2024
NVIDIA Tegra Jetson TX2 Yocto Project yocto , sumo , nvidia , thud , warrior , dunfell	6	11907	September 15, 2020
NXP WaRP7 Yocto Project yocto , sumo , imx7 , nxp , warp7 , freescale , thud , dunfell	0	1901	November 24, 2018
Toradex Verdin iMX8M Plus Yocto Project yocto , nxp , imx8 , dunfell , toradex	0	1520	June 15, 2022