Nvidia Jetson L4T Integration with full A/B system update tutorial This tutorial is based on Nvidia’s official documentation for flashing the devices and getting back the filesystems for the developers to customize the device properly. Then the person can replicate this board used for development to the ones already on the field thanks to Mender’s OTA technology. Please be aware this tutorial is intended for devices that are compatible with the A/B partitioning feature and it is not available for Jetson Nano devices. This tutorial will be focused on Nvidia’s Jetson Xavier NX board, but you can use it for any other compatible board with some minor tweaks. Also, this tutorial only covers upgrades without “Partition Layout Changes” You can get all the Nvidia-provided files from here. Requirements One Jetson Board for development (Xavier NX Devkit in this example) It is recommended to use a 64GB SD card or larger as A/B configuration use twice the space of the device. Ubuntu 16.04 or 18.04 as host machine (as required by Nvidia’s official tools) Mender artifact installed in your host machine An existing Hoster Mender account. If you don’t have one you can open and try it for for free (12 months up to 10 devices)" Micro USB cable to connect the Jetson board to the Host machine (Recommended) Ethernet cable for the Jetson (Recommended) HDMI cable and a screen for configuration of the development board (Optional) TTL Serial cable Setting up the Nvidia Jetson The following instructions are based on the “Setting Up the Root File System” section from the official Nvidia documentation. This procedure uses the sample file system provided by NVIDIA, but can use your own rootfs. It will assume you prefer the Unified Bootloader and Rootfs A/B Enabled approach recommended by Nvidia; otherwise, please check here. Set up the rootfs Get the package for the current release of Jetson Linux and unpack and apply its sample root filesystem Tegra-Linux-Sample-Root-Filesystem_ .tbz2 And extract it tar xpf Jetson_Linux_ _aarch64.tbz2 Get the NVIDIA-provided sample root file system Tegra-Linux-Sample-Root-Filesystem_ .tbz2 and extract it as follows: cd ./Linux_for_Tegra/rootfs sudo tar -jxpf ../../Tegra-Linux-Sample-Root-Filesystem_ .tbz2 Do the same for the Nvidia OTA tools ota_tools_ _aarch64.tbz2 cd ../.. tar xpf ./ota_tools_ _aarch64.tbz2 cd Linux_for_Tegra Run the apply_binaries.sh script to copy the NVIDIA user space libraries into the target file system. sudo ./apply_binaries.sh If the apply_binaries.sh script installs the binaries correctly, the last message output from the script is “Success!”. Flash the Jetson device You need to connect the Jetson device to your host machine now using the USB cable, you can connect the other cables as well (ethernet, HDMI and peripherals or TTL). For flashing the devices you only need to put your device in Factory Reset mode placing a jumper just like the image below and then connecting it to a power source. [jumper] And then, from there run sudo ROOTFS_AB=1 ./flash.sh [options] Where: ROOTFS_AB=1 enables the rootfs redundancy. indicates the type of device: For Jetson Xavier NX: jetson-xavier-nx-devkit-emmc or jetson-xavier-nx-devkit. For Jetson AGX Xavier series: jetson-agx-xavier-devkit. For Jetson TX2 series: jetson-tx2-devkit. specifies the location of the root file systems. For the Jetson Xavier NX devkit, we are running the following command: sudo ROOTFS_AB=1 ./flash.sh jetson-xavier-nx-devkit mmcblk0p1 You can remove the jumper now. Power up the Jetson board After ensuring the jumper is no longer inserted into the device, you can power cycle your Nvidia board and configure it using the Serial cable or a screen with a mouse and keyboard. Proceed to configure your username, passwords and accept Nvidia’s license. Then, you can modify the system to fit your needs. Taking advantage of the UDA partition Depending on the partition schema used in your board, you can use the UDA partition as described by official Nvidia docs to store your information. For more information, please check this website. We are going to use this UDA partition to store information that will be shared between root filesystems A and B. Let’s format it as it is just a placeholder at this moment. In the default Jetson Xavier NX partition schema, it can be found as mmcblk0p12. But depending on your device, it could be sudo mkfs.ext4 /dev/mmcblk0p12 And use it as the /data partition as required by Mender. sudo mkdir -p /data sudo su -c "echo '/dev/mmcblk0p12 /data ext4 defaults 0 0' >> /etc/fstab" sudo mount -a Mender installation Download the Mender’s application. In this tutorial we are using the Express installation, but you can choose a different method from here. If you follow this one, you may need to install curl first. curl -fLsS https://get.mender.io -o get-mender.sh sudo sh get-mender.sh Mender configuration Configure the mender client, you will be asked for your Mender’s credentials and other parameters. sudo mender setup We strongly recommend to use the same official nomenclature from Nvidia for setting the “device_type” variable as described previously in Flash the Jetson device. After installing the Mender client with get.mender.io, the mender-client package is maintained by the package manager. Some Mender files have to live in the /data partition. Move them as follows: sudo mv /var/lib/mender /data sudo ln -s /data/mender /var/lib/mender Creating the update module First create the following directories sudo mkdir -p /usr/share/mender/modules/v3 sudo touch /usr/share/mender/modules/v3/rootfs-image-jetson And create the rootfs-image-jetson file by copying the following code #!/bin/sh set -ue STATE="$1" FILES="$2" ota_payload_package="$FILES"/files/ota_payload_package.tar.gz nvidia_tools="$FILES"/files/ota_tools_aarch64.tbz2 # Nvidia uses the concept of "slots", they will # be always 0 and 1 values for A and B partitions MENDER_ROOTFS_PART_A_NUMBER="0" MENDER_ROOTFS_PART_B_NUMBER="1" # Some useful information for integrity check mkdir -p "/data/fs_update/" mender_boot_part="/data/fs_update/mender_boot_part" active_num="$(nvbootctrl get-current-slot)" if test $active_num -eq $MENDER_ROOTFS_PART_A_NUMBER; then passive_num=$MENDER_ROOTFS_PART_B_NUMBER else passive_num=$MENDER_ROOTFS_PART_A_NUMBER fi case "$STATE" in Download) if [ "$(nvbootctrl get-number-slots)" != "2" ]; then echo "Your device need to be configured to use A/B partitioning" exit 1 fi ;; ArtifactInstall) # Let's record what slot we expect to boot next time echo $passive_num > $mender_boot_part # Let's enable Nvidia's scripts mkdir -p "$HOME"/workdir export WORKDIR="$HOME"/workdir tar -jxvf $nvidia_tools -C $WORKDIR # UDA partition as temp location disabled initially UDA="" # If you have a big enough UDA partition or a small rootfs # partition size, uncomment the following line #UDA="data/" # Let's move the payload mkdir -p "/${UDA}ota/" mv $ota_payload_package "/${UDA}ota/" mkdir -p "/${UDA}ota_work" if [ ! -z "$UDA" ] then ln -sfn "/${UDA}ota_work" "/ota_work" fi # Let's run the upgrade process cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade ./nv_ota_start.sh /dev/mmcblk0 "/${UDA}ota/ota_payload_package.tar.gz" # nv_ota_start.sh set the unused slot to active for next reboot >&2 echo "Next boot will load Slot $passive_num" #cleaning up the disk if [ ! -z "$UDA" ]; then rm -rf "/${UDA}ota_work" rm -rf "/${UDA}ota/" fi ;; PerformsFullUpdate) echo "Yes" ;; NeedsArtifactReboot) echo "Automatic" ;; SupportsRollback) echo "Yes" ;; ArtifactVerifyReboot) # We use stderr for logging as Mender protocol uses stdout for exchanging messages with the server. >&2 echo "ArtifactVerifyReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)" if test "$(cat $mender_boot_part)" != "$active_num"; then exit 1 fi # Recommend calling sync at the end here as well sync ;; ArtifactVerifyRollbackReboot) >&2 echo "ArtifactVerifyRollbackReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)" if test "$(cat $mender_boot_part)" = "$active_num"; then exit 1 fi # Recommend calling sync at the end here as well sync ;; ArtifactCommit) >&2 echo "ArtifactCommit: The active partition is $active_num while the last passive was $(cat $mender_boot_part)" if test "$(cat $mender_boot_part)" = "$active_num"; then nvbootctrl mark-boot-successful else # If we get here, an upgrade in standalone mode failed to # boot and the user is trying to commit from the old OS. # This communicates to the user that the upgrade failed. echo "Upgrade failed and was reverted: refusing to commit!" exit 1 fi ;; ArtifactRollback) >&2 echo "ArtifactRollback: The active partition is $active_num while the last passive was $(cat $mender_boot_part)" if test "$(cat $mender_boot_part)" = "$active_num"; then nvbootctrl set-active-boot-slot $passive_num fi sync ;; esac exit 0 And don’t forget to make it executable chmod +x /usr/share/mender/modules/v3/rootfs-image-jetson You can modify it if it makes sense to your workflow, but we encourage to use the state scripts to modify the default behavior instead. Please notice this tutorial only covers the rootfs update; if you need to update your bootloader or the kernel partitions, please modify your update module accordingly. Getting the Golden image After customizing your system as you like, we can snapshot this state in order to replicate it in the other devices of the fleet by OTA updates. You can choose to follow the Nvidia way of getting the rootfs or the mender way of doing it. The Mender way The Mender way of getting the rootfs image relays in dumping the filesystem from the running target and with that creating the final artifact. If you don’t have SSH enabled, don’t forget to install it in your host machine first with “sudo apt install openssh-server”. Don’t forget to check the firewall to allow SSH connections. First of all, let’s grab the rootfs from our reference board, the rootfs will be located in the home directory of our host machine. Run the following command on your Jetson device: USER="user" #host username HOST="host-ip" #check it is reachable from the Jetson sudo mender snapshot dump | ssh $USER@$HOST /bin/sh -c 'cat > $HOME/my-jetson-fs.raw' If you prefer to stick with the Mender way, you can check for additional information from here The Nvidia way The Nvidia way of extracting the image is documented here. For this tutorial, we are going to relay in the filesystem from the running target and with that creating the final artifact. First of all, let’s grab the rootfs from our reference board, so we need to put our device in factory reset mode again and run from the host the following command: $ sudo ./flash.sh -r -k APP -G mmcblk0p1 Where: determines the names of the copies. specifies the configuration of the target device. This step creates two copies of in the directory: a sparse image (smaller than the original) named , and an exact copy named .raw. In this example, the command will look like this: sudo ./flash.sh -r -k APP -G my-jetson-fs jetson-xavier-nx-devkit mmcblk0p1 Generating the Nvidia’s OTA package As we are not moving from versions of the system but we are deploying our modifications, we are going to set the following variables to the same value. In the case you need to upgrade from different versions of L4T, please check official Nvidia’s docs TARGET_BSP=$HOME/Linux_for_Tegra BASE_BSP=$HOME/Linux_for_Tegra To generate our OTA package, we will need to create first a recovery image as follows (please notice we specify R32-6 as our version but you can change it for new releases): sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP} Due to Nvidia’s bootloader configuration and the updater script we are going to use, we will pass the captured rootfs as a tarball. So we need to mount it, create the tarball and finally create the Nvidia’s OTA payload. mkdir -p tmp-fs sudo mount -o loop my-jetson-fs.raw tmp-fs cd tmp-fs sudo tar -cvpzf ../image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./ cd .. sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f image_fs.tar.gz jetson-xavier-nx-devkit R32-6 Please notice the -f image_fs.tar.gz we generated in the previous step. The rootfs_updater.sh is an example script provided as a base by Nvidia in tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh and renamed later and we are going to use it in this example. But you can change it as needed. For other configurations and options, please review this. Generating a mender artifact For creating the artifact, you will need to set the mender artifact application in your host machine following this link. Ensure the “DEVICE_TYPE” value is the same specified when configuring the Mender client. Please notice the device_type in the IMAGE variable path and in the DEVICE_TYPE variable and change it as needed. ARTIFACT_NAME="my-update-1.0" DEVICE_TYPE="jetson-xavier-nx" OUTPUT_PATH="my-update-1.0.mender" IMAGE="./bootloader/jetson-xavier-nx-devkit/ota_payload_package.tar.gz" OTA_TOOLS="./ota_tools_aarch64.tbz2" #Please modify the path and name like this mender-artifact write module-image -T rootfs-image-jetson -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -o ${OUTPUT_PATH} -f ${IMAGE} -f ${OTA_TOOLS} Where ARTIFACT_NAME - The name of the Mender Artifact DEVICE_TYPE - The compatible device type of this Mender Artifact OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix IMAGE - The path to the nvidia’s package to be bundled in the Artifact The Mender Artifact used by this Update Module has the root filesystem image as the only payload. Deploying the full system update to your devices You can use Mender’s API to upload you artifact to the OTA server or to take advantage of the Mender’s website as described in the official Mender documentation to deploy your update to your whole fleet. Final comments Don’t forget this process is highly scalable, so you can use it to upgrade your whole fleet at once with many other options Mender provides and you can learn from the official documentation website and from the official forum Mender Hub. Also, don’t forget to check the official Nvidia’s documentation website for extra parameters.

Hi @oscarthorn, To clarify, are you using the L4T rather than Yocto, right? If so, you need to get a device for development purposes (not part of your fleet) that you will use it to generate Golden Images. So let’s imagine you have your fleet deployed with L4T and Mender, so you have to update and modify your development board with the new changes and then you can generate a new artifact from it. But how? Just follow this guide from this point: [image] lramirez: Getting the Golden image After customizing your system as you like, we can snapshot this state in order to replicate it in the other devices of the fleet by OTA updates. To understand the concept of a Golden image you can take a look here: Create an Artifact with system snapshot | Mender documentation. Hope it helps :slight_smile: Luis

@lramirez Thanks for the response! But I’m still not certain about what’s the best workflow for adding a new device, an artifact can be used to update an existing device but not flash a blank sd card right? So is the best way to take the generated my-jetson-fs.raw file and flash that using the flash script from nvidia? Or is there a better way?

@oscarthorn, correct. The first time we flash the device, it is recommended to do it using NVidia tools as their system is pretty custom, and we want to add value with Mender to their workflow without breaking anything. I think it is a good idea to follow their best practices. After we have the device ready the first time, then we can use Mender easily. As a side comment, another option you could contemplate is to use a different filesystem instead of the sample NVidia provides. I haven’t tried it myself, but it could save some time if it makes more sense in your workflow :slight_smile:

Thanks a lot for this clean and nice tutorial. Managed to apply it to an AGX Xavier and almost got it to work. Still an issue when booting the new partition, but it seems more Nvidia than Mender related. Just a question, it there a special reason you # Let's move the payload mkdir -p "/${UDA}ota/" mv $ota_payload_package "/${UDA}ota/" and then ./nv_ota_start.sh /dev/mmcblk0 "/${UDA}ota/ota_payload_package.tar.gz" Wouldn’t it be simpler to just ./nv_ota_start.sh /dev/mmcblk0 "$ota_payload_package" Do we need this move ?

Hello @MartinHerren, It is possible to do it as you mentioned. However, in my tests, (once) the payload started to unpack, and the partition did not have enough space to continue, so the update failed. So using the UDA partition (big enough) was a fail-proof option for my tests. But in the end, it will depend on how big your partitions are configured and how big your payload is. As a side note, I faced this issue on an early release version of Jetpack, and I don’t know if that was fixed, but I can tell that the version I shared works as expected at least. My recommendation is to test it as you described and if it works for you, it should be ok. Luis

Hi, has anyone tried this with Nvidia R35.1? It does not seem to support creating OTA packages without moving between versions. If I run sudo -E ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit-emmc R35-1 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}, i get Target board(JETSON_XAVIER_NX_DEVKIT_EMMC) and base version(R35_1) is not supported

Hello @lramirez has this experiment been tried with OS OTA on external drive on a Xavier NX emmc using Jetpack 5.1 (R35.2.1)? We have successfully been able to create the OTA Payload package, deploy to target Jetson device with changes in some parts of the scripts to utilize /dev/nvme0n1 instead of mmcblk0p1. The Mender deployment works until installation stage & then stays in reboot stage. However, when we check the Jetson device, we see that it has rebooted with the new rootfs image, but on internal drive instead of external even though we have updated the /usr/share/mender/modules/v3/rootfs-image-jetson to consider external drive instead of internal. We also made this change in the above script: # Let's run the upgrade process cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade ./nv_ota_start.sh /dev/nvme0n1 "/${UDA}ota/ota_payload_package.tar.gz" We think this bug could be a part being enforced by the mender binary object. How do we tell Mender that the reboot needs to happen on external drive instead? Could you let us know if the hypothesis is correct? Edit 1: Adding more context OTA from Jetpack 4.6 R32.7.1 to Jetpack 5.1 R35.2.1 OTA Payload package created using the sample rootfs image & BSP downloaded from the release page of Jetpack 5.1. Command used for OTA Payload package creation from NVIDIA Documentation sudo -E ROOTFS_AB=1 ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -o rootfs_updater.sh --external-device nvme0n1 -S 40GiB jetson-xavier-nx-devkit-emmc R32-7

@KlausPopp, as far as I am aware, NVIDIA is choosing to provide support for performing OTA from older BSP versions to the latest as the support for the older versions is much more urgent. I think OTA from 35.x → 35.y versions might be supported in Jetpack 5.1.1.

@KlausPopp & @user_1, I am in touch with the OE4T people who are currently also working on understanding NVidias stance on their own so-called OTA process and the usecases it supports, and which it does not. As far as I can see, there is still a lot of moving parts there. My suggestion is to join their [project chat|(- Gitter) and try to find a way forward there. Feel free to ping me there too in order to hook you up. Greetz, Josef

Hello guys. I am trying to update the version R32.7.1 to R32.7.2 on AGX Xavier 32GB using Mender and everything went well until the end. I followed all the instructions above. I get this error on the Mender server: Artifact install failed: Payload: can not install Payload: ota_payload_package.tar.gz: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/files/ota_payload_package.tar.gz: write /var/lib/mender/modules/v3/payloads/0000/tree/files/ota_payload_package.tar.gz: no space left on device I think that there is enough space on bot the partitions: [Screenshot from 2023-04-06 18-59-50] mmcblk0p1 → 40% free mmcblk0p2 → 100% free Thanks in advance

Nvidia Jetson L4T Integration

Board Integrations Debian family

Topic		Replies	Views
Nvidia Jetson Orin L4T image Update Board Integrations nvidia , jetson , l4t	3	2939	April 1, 2026
NVIDIA Tegra Jetson Xavier NX Yocto Project nvidia , dunfell	25	7166	May 4, 2022
NVIDIA Tegra Jetson Nano Yocto Project yocto , nvidia , warrior	31	12556	May 2, 2024
Jetson Orin NX and Jetpack 6 Mender & Intergration issues Yocto Project yocto , nvidia , jetson , scarthgap , jetpack-6	2	741	December 12, 2024
NXP i.MX6 DL - Ubuntu 20.04 LTS Other nxp , imx6dl	17	2793	March 10, 2021

Nvidia Jetson L4T Integration

Related topics