Nvidia Jetson L4T Integration

Nvidia Jetson L4T Integration with full A/B system update tutorial

This tutorial is based on Nvidia’s official documentation for flashing the devices and getting back the filesystems for the developers to customize the device properly. Then the person can replicate this board used for development to the ones already on the field thanks to Mender’s OTA technology.

Please be aware this tutorial is intended for devices that are compatible with the A/B partitioning feature and it is not available for Jetson Nano devices. This tutorial will be focused on Nvidia’s Jetson Xavier NX board, but you can use it for any other compatible board with some minor tweaks. Also, this tutorial only covers upgrades without “Partition Layout Changes”

You can get all the Nvidia-provided files from here.

Requirements

  • One Jetson Board for development (Xavier NX Devkit in this example)
    • It is recommended to use a 64GB SD card or larger as A/B configuration use twice the space of the device.
  • Ubuntu 16.04 or 18.04 as host machine (as required by Nvidia’s official tools)
  • Mender artifact installed in your host machine
  • A Hosted Mender account you can get from mender.io.
    • It is recommended to use email+password instead of Google or Github SSO for this tutorial. If your account was made by SSO, you can add an extra user to follow this tutorial in an easier way.
  • Micro USB cable to connect the Jetson board to the Host machine
  • (Recommended) Ethernet cable for the Jetson
  • (Recommended) HDMI cable and a screen for configuration of the development board
  • (Optional) TTL Serial cable

Setting up the Nvidia Jetson

The following instructions are based on the “Setting Up the Root File System” section from the official Nvidia documentation. This procedure uses the sample file system provided by NVIDIA, but can use your own rootfs. It will assume you prefer the Unified Bootloader and Rootfs A/B Enabled approach recommended by Nvidia; otherwise, please check here.

Set up the rootfs

  • Get the package for the current release of Jetson Linux and unpack and apply its sample root filesystem Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2

And extract it

tar xpf Jetson_Linux_<rel>_aarch64.tbz2
  • Get the NVIDIA-provided sample root file system Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2 and extract it as follows:
cd ./Linux_for_Tegra/rootfs
sudo tar -jxpf ../../Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2

Do the same for the Nvidia OTA tools ota_tools_<rel>_aarch64.tbz2

cd ../..
tar xpf ./ota_tools_<rel>_aarch64.tbz2
cd Linux_for_Tegra
  • Run the apply_binaries.sh script to copy the NVIDIA user space libraries into the target file system.
sudo ./apply_binaries.sh

If the apply_binaries.sh script installs the binaries correctly, the last message output from the script is “Success!”.

Flash the Jetson device

You need to connect the Jetson device to your host machine now using the USB cable, you can connect the other cables as well (ethernet, HDMI and peripherals or TTL). For flashing the devices you only need to put your device in Factory Reset mode placing a jumper just like the image below and then connecting it to a power source.

And then, from there run

sudo ROOTFS_AB=1 ./flash.sh [options] <target_board> <rootdev>

Where:

  • ROOTFS_AB=1 enables the rootfs redundancy.
  • <target_board> indicates the type of device:
    • For Jetson Xavier NX: jetson-xavier-nx-devkit-emmc or jetson-xavier-nx-devkit.
    • For Jetson AGX Xavier series: jetson-agx-xavier-devkit.
    • For Jetson TX2 series: jetson-tx2-devkit.
  • <rootdev> specifies the location of the root file systems.

For the Jetson Xavier NX devkit, we are running the following command:

sudo ROOTFS_AB=1 ./flash.sh jetson-xavier-nx-devkit mmcblk0p1

You can remove the jumper now.

Power up the Jetson board

After ensuring the jumper is no longer inserted into the device, you can power cycle your Nvidia board and configure it using the Serial cable or a screen with a mouse and keyboard.

Proceed to configure your username, passwords and accept Nvidia’s license. Then, you can modify the system to fit your needs.

Taking advantage of the UDA partition

Depending on the partition schema used in your board, you can use the UDA partition as described by official Nvidia docs to store your information. For more information, please check this website.

We are going to use this UDA partition to store information that will be shared between root filesystems A and B. Let’s format it as it is just a placeholder at this moment. In the default Jetson Xavier NX partition schema, it can be found as mmcblk0p12. But depending on your device, it could be

sudo mkfs.ext4 /dev/mmcblk0p12

And use it as the /data partition as required by Mender.

sudo mkdir -p /data
sudo su -c "echo '/dev/mmcblk0p12 /data ext4 defaults 0 0' >> /etc/fstab"
sudo mount -a

Mender installation

Download the Mender’s application. In this tutorial we are using the Express installation, but you can choose a different method from here. If you follow this one, you may need to install curl first.

curl -fLsS https://get.mender.io -o get-mender.sh
sudo sh get-mender.sh

Mender configuration

Configure the mender client, you will be asked for your Mender’s credentials and other parameters.

sudo mender setup

We strongly recommend to use the same official nomenclature from Nvidia for setting the “device_type” variable as described previously in Flash the Jetson device.

After installing the Mender client with get.mender.io, the mender-client package is maintained by the package manager.

Some Mender files have to live in the /data partition. Move them as follows:

sudo mv /var/lib/mender /data
sudo ln -s /data/mender /var/lib/mender

Creating the update module

First create the following directories

sudo mkdir -p /usr/share/mender/modules/v3
sudo touch  /usr/share/mender/modules/v3/rootfs-image-jetson

And create the rootfs-image-jetson file by copying the following code

#!/bin/sh

set -ue

STATE="$1"
FILES="$2"

ota_payload_package="$FILES"/files/ota_payload_package.tar.gz
nvidia_tools="$FILES"/files/ota_tools_aarch64.tbz2

# Nvidia uses the concept of "slots", they will
# be always 0 and 1 values for A and B partitions

MENDER_ROOTFS_PART_A_NUMBER="0"
MENDER_ROOTFS_PART_B_NUMBER="1"

# Some useful information for integrity check
mkdir -p "/data/fs_update/"
mender_boot_part="/data/fs_update/mender_boot_part"
active_num="$(nvbootctrl get-current-slot)"

if test $active_num -eq $MENDER_ROOTFS_PART_A_NUMBER; then
    passive_num=$MENDER_ROOTFS_PART_B_NUMBER
else
    passive_num=$MENDER_ROOTFS_PART_A_NUMBER
fi

case "$STATE" in
      Download)
        if [ "$(nvbootctrl get-number-slots)" != "2" ]; then
            echo "Your device need to be configured to use A/B partitioning"
            exit 1
        fi
        ;;

    ArtifactInstall)
        # Let's record what slot we expect to boot next time
        echo $passive_num > $mender_boot_part
        # Let's enable Nvidia's scripts
        mkdir -p "$HOME"/workdir
        export WORKDIR="$HOME"/workdir
        tar -jxvf $nvidia_tools -C $WORKDIR
        # UDA partition as temp location disabled initially
        UDA=""
        # If you have a big enough UDA partition or a small rootfs
        # partition size, uncomment the following line  
        #UDA="data/"
        # Let's move the payload
        mkdir -p "/${UDA}ota/"
        mv $ota_payload_package "/${UDA}ota/"
        mkdir -p "/${UDA}ota_work"
        if [ ! -z "$UDA" ]
        then
            ln -sfn "/${UDA}ota_work" "/ota_work"
        fi
        # Let's run the upgrade process
        cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade
        ./nv_ota_start.sh /dev/mmcblk0 "/${UDA}ota/ota_payload_package.tar.gz"
        # nv_ota_start.sh set the unused slot to active for next reboot
        >&2 echo "Next boot will load Slot $passive_num" 
        #cleaning up the disk
        if [ ! -z "$UDA" ]; then
            rm -rf "/${UDA}ota_work"
            rm -rf "/${UDA}ota/"
        fi   
        ;;

    PerformsFullUpdate)
        echo "Yes"
        ;;

    NeedsArtifactReboot)
        echo "Automatic"
        ;;

    SupportsRollback)
        echo "Yes"
        ;;

    ArtifactVerifyReboot)
        # We use stderr for logging as Mender protocol uses stdout for exchanging messages with the server.
        >&2 echo "ArtifactVerifyReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" != "$active_num"; then
            exit 1
        fi
        # Recommend calling sync at the end here as well
        sync
        ;;

    ArtifactVerifyRollbackReboot)
        >&2 echo "ArtifactVerifyRollbackReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            exit 1
        fi
        # Recommend calling sync at the end here as well
        sync
        ;;

    ArtifactCommit)
        >&2 echo "ArtifactCommit: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            nvbootctrl mark-boot-successful
        else
            # If we get here, an upgrade in standalone mode failed to  
            # boot and the user is trying to commit from the old OS.
            # This communicates to the user that the upgrade failed.
            echo "Upgrade failed and was reverted: refusing to commit!"
            exit 1
        fi
        ;;

    ArtifactRollback)
        >&2 echo "ArtifactRollback: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            nvbootctrl set-active-boot-slot $passive_num
        fi
        sync
        ;;

esac
exit 0

And don’t forget to make it executable

chmod +x /usr/share/mender/modules/v3/rootfs-image-jetson

You can modify it if it makes sense to your workflow, but we encourage to use the state scripts to modify the default behavior instead.

Please notice this tutorial only covers the rootfs update; if you need to update your bootloader or the kernel partitions, please modify your update module accordingly.

Getting the Golden image

After customizing your system as you like, we can snapshot this state in order to replicate it in the other devices of the fleet by OTA updates.

You can choose to follow the Nvidia way of getting the rootfs or the mender way of doing it.

The Mender way

The Mender way of getting the rootfs image relays in dumping the filesystem from the running target and with that creating the final artifact. If you don’t have SSH enabled, don’t forget to install it in your host machine first with “sudo apt install openssh-server”. Don’t forget to check the firewall to allow SSH connections.

First of all, let’s grab the rootfs from our reference board, the rootfs will be located in the home directory of our host machine.

Run the following command on your Jetson device:

USER="user" #host username
HOST="host-ip" #check it is reachable from the Jetson
sudo mender snapshot dump | ssh $USER@$HOST /bin/sh -c 'cat > $HOME/my-jetson-fs.raw'

If you prefer to stick with the Mender way, you can check for additional information from here

The Nvidia way

The Nvidia way of extracting the image is documented here. For this tutorial, we are going to relay in the filesystem from the running target and with that creating the final artifact.

First of all, let’s grab the rootfs from our reference board, so we need to put our device in factory reset mode again and run from the host the following command:

$ sudo ./flash.sh -r -k APP -G <clone> <board> mmcblk0p1

Where:

  • <clone> determines the names of the copies.
  • <board> specifies the configuration of the target device.

This step creates two copies of in the directory: a sparse image (smaller than the original) named , and an exact copy named .raw.

In this example, the command will look like this:

sudo ./flash.sh -r -k APP -G my-jetson-fs jetson-xavier-nx-devkit mmcblk0p1

Generating the Nvidia’s OTA package

As we are not moving from versions of the system but we are deploying our modifications, we are going to set the following variables to the same value. In the case you need to upgrade from different versions of L4T, please check official Nvidia’s docs

TARGET_BSP=$HOME/Linux_for_Tegra
BASE_BSP=$HOME/Linux_for_Tegra

To generate our OTA package, we will need to create first a recovery image as follows (please notice we specify R32-6 as our version but you can change it for new releases):

sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}

Due to Nvidia’s bootloader configuration and the updater script we are going to use, we will pass the captured rootfs as a tarball. So we need to mount it, create the tarball and finally create the Nvidia’s OTA payload.

mkdir -p tmp-fs
sudo mount -o loop my-jetson-fs.raw tmp-fs
cd tmp-fs
sudo tar -cvpzf ../image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./
cd ..
sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f image_fs.tar.gz jetson-xavier-nx-devkit R32-6

Please notice the -f image_fs.tar.gz we generated in the previous step. The rootfs_updater.sh is an example script provided as a base by Nvidia in tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh and renamed later and we are going to use it in this example. But you can change it as needed.

For other configurations and options, please review this.

Generating a mender artifact

For creating the artifact, you will need to set the mender artifact application in your host machine following this link. Ensure the “DEVICE_TYPE” value is the same specified when configuring the Mender client.

Please notice the device_type in the IMAGE variable path and in the DEVICE_TYPE variable and change it as needed.

ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="jetson-xavier-nx"
OUTPUT_PATH="my-update-1.0.mender"
IMAGE="./bootloader/jetson-xavier-nx-devkit/ota_payload_package.tar.gz"
OTA_TOOLS="./ota_tools_aarch64.tbz2" #Please modify the path and name like this
mender-artifact write module-image -T rootfs-image-jetson -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -o ${OUTPUT_PATH} -f ${IMAGE} -f ${OTA_TOOLS}

Where

  • ARTIFACT_NAME - The name of the Mender Artifact
  • DEVICE_TYPE - The compatible device type of this Mender Artifact
  • OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
  • IMAGE - The path to the nvidia’s package to be bundled in the Artifact

The Mender Artifact used by this Update Module has the root filesystem image as the only payload.

Deploying the full system update to your devices

You can use Mender’s API to upload you artifact to the OTA server or to take advantage of the Mender’s website as described in the official Mender documentation to deploy your update to your whole fleet.

Final comments

Don’t forget this process is highly scalable, so you can use it to upgrade your whole fleet at once with many other options Mender provides and you can learn from the official documentation website and from the official forum Mender Hub. Also, don’t forget to check the official Nvidia’s documentation website for extra parameters.

2 Likes