Nvidia Jetson L4T Integration with full A/B system update tutorial
This tutorial is based on Nvidia’s official documentation for flashing the devices and getting back the filesystems for the developers to customize the device properly. Then the person can replicate this board used for development to the ones already on the field thanks to Mender’s OTA technology.
Please be aware this tutorial is intended for devices that are compatible with the A/B partitioning feature and it is not available for Jetson Nano devices. This tutorial will be focused on Nvidia’s Jetson Xavier NX board, but you can use it for any other compatible board with some minor tweaks. Also, this tutorial only covers upgrades without “Partition Layout Changes”
You can get all the Nvidia-provided files from here.
Requirements
- One Jetson Board for development (Xavier NX Devkit in this example)
- It is recommended to use a 64GB SD card or larger as A/B configuration use twice the space of the device.
- Ubuntu 16.04 or 18.04 as host machine (as required by Nvidia’s official tools)
- Mender artifact installed in your host machine
- An existing Hoster Mender account. If you don’t have one you can open and try it for for free (12 months up to 10 devices)"
- Micro USB cable to connect the Jetson board to the Host machine
- (Recommended) Ethernet cable for the Jetson
- (Recommended) HDMI cable and a screen for configuration of the development board
- (Optional) TTL Serial cable
Setting up the Nvidia Jetson
The following instructions are based on the “Setting Up the Root File System” section from the official Nvidia documentation. This procedure uses the sample file system provided by NVIDIA, but can use your own rootfs. It will assume you prefer the Unified Bootloader and Rootfs A/B Enabled approach recommended by Nvidia; otherwise, please check here.
Set up the rootfs
- Get the package for the current release of Jetson Linux and unpack and apply its sample root filesystem
Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2
And extract it
tar xpf Jetson_Linux_<rel>_aarch64.tbz2
- Get the NVIDIA-provided sample root file system
Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2and extract it as follows:
cd ./Linux_for_Tegra/rootfs
sudo tar -jxpf ../../Tegra-Linux-Sample-Root-Filesystem_<release_type>.tbz2
Do the same for the Nvidia OTA tools ota_tools_<rel>_aarch64.tbz2
cd ../..
tar xpf ./ota_tools_<rel>_aarch64.tbz2
cd Linux_for_Tegra
- Run the
apply_binaries.shscript to copy the NVIDIA user space libraries into the target file system.
sudo ./apply_binaries.sh
If the apply_binaries.sh script installs the binaries correctly, the last message output from the script is “Success!”.
Flash the Jetson device
You need to connect the Jetson device to your host machine now using the USB cable, you can connect the other cables as well (ethernet, HDMI and peripherals or TTL). For flashing the devices you only need to put your device in Factory Reset mode placing a jumper just like the image below and then connecting it to a power source.
And then, from there run
sudo ROOTFS_AB=1 ./flash.sh [options] <target_board> <rootdev>
Where:
-
ROOTFS_AB=1enables the rootfs redundancy. -
<target_board>indicates the type of device:- For Jetson Xavier NX:
jetson-xavier-nx-devkit-emmcorjetson-xavier-nx-devkit. - For Jetson AGX Xavier series:
jetson-agx-xavier-devkit. - For Jetson TX2 series:
jetson-tx2-devkit.
- For Jetson Xavier NX:
-
<rootdev>specifies the location of the root file systems.
For the Jetson Xavier NX devkit, we are running the following command:
sudo ROOTFS_AB=1 ./flash.sh jetson-xavier-nx-devkit mmcblk0p1
You can remove the jumper now.
Power up the Jetson board
After ensuring the jumper is no longer inserted into the device, you can power cycle your Nvidia board and configure it using the Serial cable or a screen with a mouse and keyboard.
Proceed to configure your username, passwords and accept Nvidia’s license. Then, you can modify the system to fit your needs.
Taking advantage of the UDA partition
Depending on the partition schema used in your board, you can use the UDA partition as described by official Nvidia docs to store your information. For more information, please check this website.
We are going to use this UDA partition to store information that will be shared between root filesystems A and B. Let’s format it as it is just a placeholder at this moment. In the default Jetson Xavier NX partition schema, it can be found as mmcblk0p12. But depending on your device, it could be
sudo mkfs.ext4 /dev/mmcblk0p12
And use it as the /data partition as required by Mender.
sudo mkdir -p /data
sudo su -c "echo '/dev/mmcblk0p12 /data ext4 defaults 0 0' >> /etc/fstab"
sudo mount -a
Mender installation
Download the Mender’s application. In this tutorial we are using the Express installation, but you can choose a different method from here. If you follow this one, you may need to install curl first.
curl -fLsS https://get.mender.io -o get-mender.sh
sudo sh get-mender.sh
Mender configuration
Configure the mender client, you will be asked for your Mender’s credentials and other parameters.
sudo mender setup
We strongly recommend to use the same official nomenclature from Nvidia for setting the “device_type” variable as described previously in Flash the Jetson device.
After installing the Mender client with get.mender.io, the mender-client package is maintained by the package manager.
Some Mender files have to live in the /data partition. Move them as follows:
sudo mv /var/lib/mender /data
sudo ln -s /data/mender /var/lib/mender
Creating the update module
First create the following directories
sudo mkdir -p /usr/share/mender/modules/v3
sudo touch /usr/share/mender/modules/v3/rootfs-image-jetson
And create the rootfs-image-jetson file by copying the following code
#!/bin/sh
set -ue
STATE="$1"
FILES="$2"
ota_payload_package="$FILES"/files/ota_payload_package.tar.gz
nvidia_tools="$FILES"/files/ota_tools_aarch64.tbz2
# Nvidia uses the concept of "slots", they will
# be always 0 and 1 values for A and B partitions
MENDER_ROOTFS_PART_A_NUMBER="0"
MENDER_ROOTFS_PART_B_NUMBER="1"
# Some useful information for integrity check
mkdir -p "/data/fs_update/"
mender_boot_part="/data/fs_update/mender_boot_part"
active_num="$(nvbootctrl get-current-slot)"
if test $active_num -eq $MENDER_ROOTFS_PART_A_NUMBER; then
passive_num=$MENDER_ROOTFS_PART_B_NUMBER
else
passive_num=$MENDER_ROOTFS_PART_A_NUMBER
fi
case "$STATE" in
Download)
if [ "$(nvbootctrl get-number-slots)" != "2" ]; then
echo "Your device need to be configured to use A/B partitioning"
exit 1
fi
;;
ArtifactInstall)
# Let's record what slot we expect to boot next time
echo $passive_num > $mender_boot_part
# Let's enable Nvidia's scripts
mkdir -p "$HOME"/workdir
export WORKDIR="$HOME"/workdir
tar -jxvf $nvidia_tools -C $WORKDIR
# UDA partition as temp location disabled initially
UDA=""
# If you have a big enough UDA partition or a small rootfs
# partition size, uncomment the following line
#UDA="data/"
# Let's move the payload
mkdir -p "/${UDA}ota/"
mv $ota_payload_package "/${UDA}ota/"
mkdir -p "/${UDA}ota_work"
if [ ! -z "$UDA" ]
then
ln -sfn "/${UDA}ota_work" "/ota_work"
fi
# Let's run the upgrade process
cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade
./nv_ota_start.sh /dev/mmcblk0 "/${UDA}ota/ota_payload_package.tar.gz"
# nv_ota_start.sh set the unused slot to active for next reboot
>&2 echo "Next boot will load Slot $passive_num"
#cleaning up the disk
if [ ! -z "$UDA" ]; then
rm -rf "/${UDA}ota_work"
rm -rf "/${UDA}ota/"
fi
;;
PerformsFullUpdate)
echo "Yes"
;;
NeedsArtifactReboot)
echo "Automatic"
;;
SupportsRollback)
echo "Yes"
;;
ArtifactVerifyReboot)
# We use stderr for logging as Mender protocol uses stdout for exchanging messages with the server.
>&2 echo "ArtifactVerifyReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
if test "$(cat $mender_boot_part)" != "$active_num"; then
exit 1
fi
# Recommend calling sync at the end here as well
sync
;;
ArtifactVerifyRollbackReboot)
>&2 echo "ArtifactVerifyRollbackReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
if test "$(cat $mender_boot_part)" = "$active_num"; then
exit 1
fi
# Recommend calling sync at the end here as well
sync
;;
ArtifactCommit)
>&2 echo "ArtifactCommit: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
if test "$(cat $mender_boot_part)" = "$active_num"; then
nvbootctrl mark-boot-successful
else
# If we get here, an upgrade in standalone mode failed to
# boot and the user is trying to commit from the old OS.
# This communicates to the user that the upgrade failed.
echo "Upgrade failed and was reverted: refusing to commit!"
exit 1
fi
;;
ArtifactRollback)
>&2 echo "ArtifactRollback: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
if test "$(cat $mender_boot_part)" = "$active_num"; then
nvbootctrl set-active-boot-slot $passive_num
fi
sync
;;
esac
exit 0
And don’t forget to make it executable
chmod +x /usr/share/mender/modules/v3/rootfs-image-jetson
You can modify it if it makes sense to your workflow, but we encourage to use the state scripts to modify the default behavior instead.
Please notice this tutorial only covers the rootfs update; if you need to update your bootloader or the kernel partitions, please modify your update module accordingly.
Getting the Golden image
After customizing your system as you like, we can snapshot this state in order to replicate it in the other devices of the fleet by OTA updates.
You can choose to follow the Nvidia way of getting the rootfs or the mender way of doing it.
The Mender way
The Mender way of getting the rootfs image relays in dumping the filesystem from the running target and with that creating the final artifact. If you don’t have SSH enabled, don’t forget to install it in your host machine first with “sudo apt install openssh-server”. Don’t forget to check the firewall to allow SSH connections.
First of all, let’s grab the rootfs from our reference board, the rootfs will be located in the home directory of our host machine.
Run the following command on your Jetson device:
USER="user" #host username
HOST="host-ip" #check it is reachable from the Jetson
sudo mender snapshot dump | ssh $USER@$HOST /bin/sh -c 'cat > $HOME/my-jetson-fs.raw'
If you prefer to stick with the Mender way, you can check for additional information from here
The Nvidia way
The Nvidia way of extracting the image is documented here. For this tutorial, we are going to relay in the filesystem from the running target and with that creating the final artifact.
First of all, let’s grab the rootfs from our reference board, so we need to put our device in factory reset mode again and run from the host the following command:
$ sudo ./flash.sh -r -k APP -G <clone> <board> mmcblk0p1
Where:
-
<clone>determines the names of the copies. -
<board>specifies the configuration of the target device.
This step creates two copies of in the directory: a sparse image (smaller than the original) named , and an exact copy named .raw.
In this example, the command will look like this:
sudo ./flash.sh -r -k APP -G my-jetson-fs jetson-xavier-nx-devkit mmcblk0p1
Generating the Nvidia’s OTA package
As we are not moving from versions of the system but we are deploying our modifications, we are going to set the following variables to the same value. In the case you need to upgrade from different versions of L4T, please check official Nvidia’s docs
TARGET_BSP=$HOME/Linux_for_Tegra
BASE_BSP=$HOME/Linux_for_Tegra
To generate our OTA package, we will need to create first a recovery image as follows (please notice we specify R32-6 as our version but you can change it for new releases):
sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}
Due to Nvidia’s bootloader configuration and the updater script we are going to use, we will pass the captured rootfs as a tarball. So we need to mount it, create the tarball and finally create the Nvidia’s OTA payload.
mkdir -p tmp-fs
sudo mount -o loop my-jetson-fs.raw tmp-fs
cd tmp-fs
sudo tar -cvpzf ../image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./
cd ..
sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f image_fs.tar.gz jetson-xavier-nx-devkit R32-6
Please notice the -f image_fs.tar.gz we generated in the previous step. The rootfs_updater.sh is an example script provided as a base by Nvidia in tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh and renamed later and we are going to use it in this example. But you can change it as needed.
For other configurations and options, please review this.
Generating a mender artifact
For creating the artifact, you will need to set the mender artifact application in your host machine following this link. Ensure the “DEVICE_TYPE” value is the same specified when configuring the Mender client.
Please notice the device_type in the IMAGE variable path and in the DEVICE_TYPE variable and change it as needed.
ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="jetson-xavier-nx"
OUTPUT_PATH="my-update-1.0.mender"
IMAGE="./bootloader/jetson-xavier-nx-devkit/ota_payload_package.tar.gz"
OTA_TOOLS="./ota_tools_aarch64.tbz2" #Please modify the path and name like this
mender-artifact write module-image -T rootfs-image-jetson -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -o ${OUTPUT_PATH} -f ${IMAGE} -f ${OTA_TOOLS}
Where
- ARTIFACT_NAME - The name of the Mender Artifact
- DEVICE_TYPE - The compatible device type of this Mender Artifact
- OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
- IMAGE - The path to the nvidia’s package to be bundled in the Artifact
The Mender Artifact used by this Update Module has the root filesystem image as the only payload.
Deploying the full system update to your devices
You can use Mender’s API to upload you artifact to the OTA server or to take advantage of the Mender’s website as described in the official Mender documentation to deploy your update to your whole fleet.
Final comments
Don’t forget this process is highly scalable, so you can use it to upgrade your whole fleet at once with many other options Mender provides and you can learn from the official documentation website and from the official forum Mender Hub. Also, don’t forget to check the official Nvidia’s documentation website for extra parameters.
C’est la vie 
