Nvidia Jetson L4T Integration

Hi @oscarthorn,

To clarify, are you using the L4T rather than Yocto, right?

If so, you need to get a device for development purposes (not part of your fleet) that you will use it to generate Golden Images.

So let’s imagine you have your fleet deployed with L4T and Mender, so you have to update and modify your development board with the new changes and then you can generate a new artifact from it. But how? Just follow this guide from this point:

To understand the concept of a Golden image you can take a look here: Create an Artifact with system snapshot | Mender documentation.

Hope it helps :slight_smile:
Luis

@lramirez Thanks for the response! But I’m still not certain about what’s the best workflow for adding a new device, an artifact can be used to update an existing device but not flash a blank sd card right?

So is the best way to take the generated my-jetson-fs.raw file and flash that using the flash script from nvidia? Or is there a better way?

@oscarthorn, correct. The first time we flash the device, it is recommended to do it using NVidia tools as their system is pretty custom, and we want to add value with Mender to their workflow without breaking anything. I think it is a good idea to follow their best practices.

After we have the device ready the first time, then we can use Mender easily.
As a side comment, another option you could contemplate is to use a different filesystem instead of the sample NVidia provides. I haven’t tried it myself, but it could save some time if it makes more sense in your workflow :slight_smile:

Thanks a lot for this clean and nice tutorial.

Managed to apply it to an AGX Xavier and almost got it to work. Still an issue when booting the new partition, but it seems more Nvidia than Mender related.

Just a question, it there a special reason you

        # Let's move the payload
        mkdir -p "/${UDA}ota/"
        mv $ota_payload_package "/${UDA}ota/"

and then

./nv_ota_start.sh /dev/mmcblk0 "/${UDA}ota/ota_payload_package.tar.gz"

Wouldn’t it be simpler to just

./nv_ota_start.sh /dev/mmcblk0 "$ota_payload_package"

Do we need this move ?

Hello @MartinHerren,

It is possible to do it as you mentioned. However, in my tests, (once) the payload started to unpack, and the partition did not have enough space to continue, so the update failed.
So using the UDA partition (big enough) was a fail-proof option for my tests. But in the end, it will depend on how big your partitions are configured and how big your payload is.

As a side note, I faced this issue on an early release version of Jetpack, and I don’t know if that was fixed, but I can tell that the version I shared works as expected at least.

My recommendation is to test it as you described and if it works for you, it should be ok.

Luis

A post was split to a new topic: Problem flashing NVidia Jetson/Xavier, Jetpack 5.1

Hi, has anyone tried this with Nvidia R35.1?

It does not seem to support creating OTA packages without moving between versions. If I run sudo -E ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-xavier-nx-devkit-emmc R35-1 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}, i get Target board(JETSON_XAVIER_NX_DEVKIT_EMMC) and base version(R35_1) is not supported

1 Like

Hello @lramirez has this experiment been tried with OS OTA on external drive on a Xavier NX emmc using Jetpack 5.1 (R35.2.1)? We have successfully been able to create the OTA Payload package, deploy to target Jetson device with changes in some parts of the scripts to utilize /dev/nvme0n1 instead of mmcblk0p1.
The Mender deployment works until installation stage & then stays in reboot stage. However, when we check the Jetson device, we see that it has rebooted with the new rootfs image, but on internal drive instead of external even though we have updated the /usr/share/mender/modules/v3/rootfs-image-jetson to consider external drive instead of internal. We also made this change in the above script:

# Let's run the upgrade process
        cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade
        ./nv_ota_start.sh /dev/nvme0n1 "/${UDA}ota/ota_payload_package.tar.gz"

We think this bug could be a part being enforced by the mender binary object. How do we tell Mender that the reboot needs to happen on external drive instead?
Could you let us know if the hypothesis is correct?

Edit 1: Adding more context

  • OTA from Jetpack 4.6 R32.7.1 to Jetpack 5.1 R35.2.1
  • OTA Payload package created using the sample rootfs image & BSP downloaded from the release page of Jetpack 5.1.
  • Command used for OTA Payload package creation from NVIDIA Documentation
sudo -E ROOTFS_AB=1 ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -o rootfs_updater.sh --external-device nvme0n1 -S 40GiB jetson-xavier-nx-devkit-emmc R32-7
1 Like

@KlausPopp, as far as I am aware, NVIDIA is choosing to provide support for performing OTA from older BSP versions to the latest as the support for the older versions is much more urgent. I think OTA from 35.x → 35.y versions might be supported in Jetpack 5.1.1.

1 Like

@KlausPopp & @user_1,

I am in touch with the OE4T people who are currently also working on understanding NVidias stance on their own so-called OTA process and the usecases it supports, and which it does not. As far as I can see, there is still a lot of moving parts there. My suggestion is to join their [project chat|(- Gitter) and try to find a way forward there. Feel free to ping me there too in order to hook you up.

Greetz,
Josef

1 Like

Hello guys. I am trying to update the version R32.7.1 to R32.7.2 on AGX Xavier 32GB using Mender and everything went well until the end. I followed all the instructions above.
I get this error on the Mender server:

Artifact install failed: Payload: can not install Payload: ota_payload_package.tar.gz: Unable to stream into /var/lib/mender/modules/v3/payloads/0000/tree/files/ota_payload_package.tar.gz: write /var/lib/mender/modules/v3/payloads/0000/tree/files/ota_payload_package.tar.gz: no space left on device

I think that there is enough space on bot the partitions:

mmcblk0p1 → 40% free
mmcblk0p2 → 100% free

Thanks in advance

Hello @petarkalafatic,

Can you execute a realpath to that directory? I am almost sure it still lives in mmcblk0p1. In that case, remember that the payload containing the new image could be as big as the current content of mmcblk0p1. Meaning that 60% + 60% will cause to get out of disk.
You can mount /var/lib/mender in any partition if it makes sense and as long as it is persistent.

Otherwise, you will need to have bigger partitions at a point that you will always have 50%+ free.

Hope it helps, have a nice day!
Luis

Thank you for your response. Yes it does make sense.

I am now trying to switch the boot sequence to the M.2 NVMe SSD I bought as first boot device but also am not having any luck.

As I understand, the Jetson Xavier AGX uses Cboot that should “pop up” by pressing “any key” during the restart process? It does not appear to me?

I also found that there are many workarounds by using some scripts on github and I am not sure if that would work for the A/B system update using mender?

Is there a way to flash the AGX with A/B partitioning on the external NVMe SSD and being able to do the OTA update on the same drive?

Thank you

Hello @petarkalafatic,

I only have the Jetson Xavier NX at hand, I am not aware of any particular workflow in the AGX bootloader to be honest. Probably Nvidia’s support is a better fit for this question in particular.

flash.sh utility from Nvidia makes the redundant configuration on the device side. You can check their docs here.

Hope it helps,
Luis

Thanks :slight_smile:

And sorry for the many questions. I am trying to get all of this running and every time I try something using the tutorial or by the docs a new error message appears :smiley: C’est la vie :smiley:
I got the (A/B update) running too but am a bit stuck at the OTA via Mender.
Followed the tutorial above and created the Mender Artifact with the update, uploaded the Artifact to the hosted server and when I click Deploy it’s just stuck at downloading. Not moving from that 1%?
Am I doing something wrong? No error just stuck

I created a script type Artifact to test the installation of VLC Media player and it went without any problems.

Hello @petarkalafatic,

No worries, and thanks for using Mender Hub, this could be useful for someone else in the future :slight_smile:

Mender server doesn’t know anything about “how much time is left in the current state” so these percentages are approximations but the actual critical part is when Mender moves from Download to Install, from Install to Reboot and so on. Depending on the network the download state could take plenty of time, but if you are sure it actually got stuck, I recommend you to use the Troubleshoot add-on’s remote terminal feature (or any other kind of connection like Serial or SSH) to check the logs from the Mender client. It would be as simple as journalctl -u mender-client to understand what is happening. You can also increase the verbosity of the logs by following this tutorial.

Hope it helps,
Luis

Hello @lramirez :slight_smile:

Tnx for the response. It was just as you said. After 20mins or so the process went from downloading to installation and then reboot. I taught that there would be some feedback on the server during the whole process. It was just my impatience and my poor download speed :sweat_smile:

Have a nice day!

Hi :slight_smile:

Is there maybe a way we can use the delta update instead of the whole system update for Jetson Linux?

Hi @lramirez

I have a question regarding the OTA update.
Following the above steps I encounter a small issue. Everything goes fine and there is a partition swap with all my data exchanged between partitions A and B after the artifact update but there is no system update. Before and after the update when I do cat / etc / nv_tegra_release I get the same 32.6.1 version. I wanted to update from version 32.6.1 to version 32.7.3

Here are all my steps that I followed through the installation process:

Install a fresh Jetpack Linux version on AGX that supports A/B update ( I used 32.6.1 ):

On Host computer:

Download version from here:

tar xpf Jetson_Linux_aarch64.tbz2
cd ./Linux_for_Tegra/rootfs
sudo tar -jxpf …/…/Tegra-Linux-Sample-Root-Filesystem
<release_type>.tbz2
cd …/…
tar xpf ./ota_tools_aarch64.tbz2
tar xpf ./secureboot
_aarch64.tbz2
cd Linux_for_Tegra

Check the Sector size and No. of sectors of the hard drive you want to flash:
sector size: cat /sys/block/nvme0n1/queue/hw_sector_size
num_sectors: cat /sys/block/nvme0n1/size
In the Linux_for_Tegra/tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml <device type="nvme" instance="0" sector_size="512" num_sectors="1000215216">

sudo ./apply_binaries.sh

sudo ROOTFS_AB=1 ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1 -c ./tools/kernel_flash/flash_l4t_nvme_rootfs_ab.xml --external-only -S 32GiB --showlogs jetson-agx-xavier-devkit external

On Jetson AGX Xavier:
sudo mkfs.ext4 /dev/nvme0n1p12

sudo mkdir -p /data
sudo su -c “echo ‘/dev/nvme0n1p12 /data ext4 defaults 0 0’ >> /etc/fstab”
sudo mount -a

sudo apt update && sudo apt upgrade
sudo apt install curl
sudo apt install nano

curl -fLsS https://get.mender.io -o get-mender.sh
sudo bash get-mender.sh mender-client

configure mender client → hosted mender server credentials.

sudo mv /var/lib/mender /data
sudo ln -s /data/mender /var/lib/mender
sudo mkdir -p /usr/share/mender/modules/v3
sudo touch /usr/share/mender/modules/v3/rootfs-image-jetson
cd /usr/share/mender/modules/v3
sudo nano rootfs-image-jetson

#!/bin/sh

set -ue

STATE="$1"
FILES="$2"

ota_payload_package="$FILES"/files/ota_payload_package.tar.gz
nvidia_tools="$FILES"/files/ota_tools_aarch64.tbz2

# Nvidia uses the concept of "slots", they will
# be always 0 and 1 values for A and B partitions

MENDER_ROOTFS_PART_A_NUMBER="0"
MENDER_ROOTFS_PART_B_NUMBER="1"

# Some useful information for integrity check
mkdir -p "/data/fs_update/"
mender_boot_part="/data/fs_update/mender_boot_part"
active_num="$(nvbootctrl get-current-slot)"

if test $active_num -eq $MENDER_ROOTFS_PART_A_NUMBER; then
    passive_num=$MENDER_ROOTFS_PART_B_NUMBER
else
    passive_num=$MENDER_ROOTFS_PART_A_NUMBER
fi

case "$STATE" in
      Download)
        if [ "$(nvbootctrl get-number-slots)" != "2" ]; then
            echo "Your device need to be configured to use A/B partitioning"
            exit 1
        fi
        ;;

    ArtifactInstall)
        # Let's record what slot we expect to boot next time
        echo $passive_num > $mender_boot_part
        # Let's enable Nvidia's scripts
        mkdir -p "$HOME"/workdir
        export WORKDIR="$HOME"/workdir
        tar -jxvf $nvidia_tools -C $WORKDIR
        # UDA partition as temp location disabled initially
        UDA=""
        # If you have a big enough UDA partition or a small rootfs
        # partition size, uncomment the following line  
        #UDA="data/"
        # Let's move the payload
        mkdir -p "/${UDA}ota/"
        mv $ota_payload_package "/${UDA}ota/"
        mkdir -p "/${UDA}ota_work"
        if [ ! -z "$UDA" ]
        then
            ln -sfn "/${UDA}ota_work" "/ota_work"
        fi
        # Let's run the upgrade process
        cd ${WORKDIR}/Linux_for_Tegra/tools/ota_tools/version_upgrade
        ./nv_ota_start.sh /dev/nvme0n1 "/${UDA}ota/ota_payload_package.tar.gz"
        # nv_ota_start.sh set the unused slot to active for next reboot
        >&2 echo "Next boot will load Slot $passive_num" 
        #cleaning up the disk
        if [ ! -z "$UDA" ]; then
            rm -rf "/${UDA}ota_work"
            rm -rf "/${UDA}ota/"
        fi   
        ;;

    PerformsFullUpdate)
        echo "Yes"
        ;;

    NeedsArtifactReboot)
        echo "Automatic"
        ;;

    SupportsRollback)
        echo "Yes"
        ;;

    ArtifactVerifyReboot)
        # We use stderr for logging as Mender protocol uses stdout for exchanging messages with the server.
        >&2 echo "ArtifactVerifyReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" != "$active_num"; then
            exit 1
        fi
        # Recommend calling sync at the end here as well
        sync
        ;;

    ArtifactVerifyRollbackReboot)
        >&2 echo "ArtifactVerifyRollbackReboot: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            exit 1
        fi
        # Recommend calling sync at the end here as well
        sync
        ;;

    ArtifactCommit)
        >&2 echo "ArtifactCommit: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            nvbootctrl mark-boot-successful
        else
            # If we get here, an upgrade in standalone mode failed to  
            # boot and the user is trying to commit from the old OS.
            # This communicates to the user that the upgrade failed.
            echo "Upgrade failed and was reverted: refusing to commit!"
            exit 1
        fi
        ;;

    ArtifactRollback)
        >&2 echo "ArtifactRollback: The active partition is $active_num while the last passive was $(cat $mender_boot_part)"
        if test "$(cat $mender_boot_part)" = "$active_num"; then
            nvbootctrl set-active-boot-slot $passive_num
        fi
        sync
        ;;

esac
exit 0

a small note-----> I replaced in this part of the script the internal with my external drive: ./nv_ota_start.sh /dev/nvme0n1 "/${UDA}ota/ota_payload_package.tar.gz"

sudo chmod +x /usr/share/mender/modules/v3/rootfs-image-jetson
sudo systemctl restart mender-client

sudo su
sudo mender snapshot dump | ssh $USER@$HOST /bin/sh -c ‘cat > $HOME/my-jetson-fs.raw’

on Host computer:

New Jetpack version ( I used 32.7.3 ):

tar xpf Jetson_Linux_aarch64.tbz2
cd ./Linux_for_Tegra/rootfs
sudo tar -jxpf …/…/Tegra-Linux-Sample-Root-Filesystem
<release_type>.tbz2
cd …/…
tar xpf ./ota_tools__aarch64.tbz2
cd Linux_for_Tegra

BASE_BSP=/tmp/Jetpack_Linux/32_6_1/Linux_for_Tegra
TARGET_BSP=/tmp/Jetpack_Linux/32_7_3/Linux_for_Tegra

check:
echo $TARGET_BSP
echo $BASE_BSP

sudo ./tools/ota_tools/version_upgrade/build_base_recovery_image.sh jetson-agx-xavier-devkit R32-6 ${BASE_BSP} ${BASE_BSP}/rootfs ${TARGET_BSP}

go to the Mender snapshot file and run this commands to mount it as a filesystem image:
mkdir -p tmp-fs
sudo mount -o loop my-jetson-fs.raw tmp-fs
cd tmp-fs
sudo tar -cvpzf …/image_fs.tar.gz --exclude=./data --exclude=./image_fs.tar.gz --one-file-system ./
cd …

cp tools/ota_tools/version_upgrade/nv_ota_rootfs_updater.sh rootfs_updater.sh
sudo ./tools/ota_tools/version_upgrade/l4t_generate_ota_package.sh -sr -o rootfs_updater.sh -f /home/petar/jetson/image_fs.tar.gz jetson-agx-xavier-devkit R32-6

Create Mender Arthefact:
ARTIFACT_NAME=“R37_2_3_update”
DEVICE_TYPE=“jetson-agx-xavier”
OUTPUT_PATH=“/home/petar/R37_2_3_update.mender”
IMAGE=“/tmp/Jetpack_Linux/32_7_3/Linux_for_Tegra/bootloader/jetson-agx-xavier-devkit/ota_payload_package.tar.gz”
OTA_TOOLS=“/tmp/Jetpack_Linux/32_7_3/ota_tools_aarch64.tbz2”
mender-artifact write module-image -T rootfs-image-jetson -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -o ${OUTPUT_PATH} -f ${IMAGE} -f ${OTA_TOOLS}

upload to mender server using GUI or CLI

check jetpack version:
cat / etc / nv_tegra_release

Hello @petarkalafatic,

Regarding the delta feature this is not possible. In another thread I shared a workaround and the technical challenge on why this is not possible. I hope it helps :slight_smile: