Toradex integration updates

There have been a number of posts recently regarding issues building Yocti with Mender for various Toradex boards. I took a quick look and it seems our integrations for the Toradex boards are a bit inconsistent; some of them use the manifests from the meta-mender-community repo and some use the Toradex manifests. I think the right approach is to use the Toradex manifests as that makes it easier to keep up to date with any updates that Toradex posts. I’m going to take a look and see about cleaning this up and get the builds working again for the boards that I have access to. In the process, I’m going to try and validate both U-Boot and Grub style integrations.

For those that are already working with these boards and trying to fix issues, we should definitely pool our efforts. Let me know if you have fixes locally that should be included.

Drew

cc: @riktw, @ABTP, @pgiangrossi

2 Likes

Hi Drew,

What boards do you have access to? If you have an Apalis IMX8, please see my local changes at Toradex Apalis IMX8 Yocto with Mender. These changes get it to build OK, but it does not run yet.

Paul

Hi Drew,

Thanks a lot for looking into this, its really appreciated :slight_smile:

So on my side, I am trying to integrate Mender on:

1-Apalis-iMX8QM running Boot2Qt based on Zeus (uboot-toradex_2018.3) on Apalis Evaluation Board V1.1C
2-Colibri iMX6ULL running Boot2Qt based on Zeus (uboot-toradex_ 2019.7) on Colibri Evaluation Board V3.2B

For the Aplis-iMX8, I have the image compiled and it no longer complained about mender-grow-data.service but I still have two issues:

1- When I run mender daemon, I get this error:

ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) ‘mender_setup’ is not run by the U-Boot boot script: exit status 1

The reason could be either that fw_env.config is set with the correct Device name , Device offset , and Env. size so that they match those U-Boot is using or (what is think is the case) that mender_setup’ is not run by the U-Boot boot script.

2- The 2nd issue is that after the board finishes booting, I get this message repeatedly:

mmcblk1: response CRC error sending SET_BLOCK_COUNT command, card status 0xc00b00

To get to this point, I made modifications to the mender-community patches:

0001-configs-apalis-imx8-mender-integration.patch

diff --git a/configs/apalis-imx8_defconfig b/configs/apalis-imx8_defconfig
index 4f1266ed6f..0b93d0935a 100644
--- a/configs/apalis-imx8_defconfig
+++ b/configs/apalis-imx8_defconfig
@@ -71,3 +71,7 @@ CONFIG_VIDEO=y
 CONFIG_VIDEO_IMX_HDP_LOAD=y
 CONFIG_OF_LIBFDT_OVERLAY=y
 # CONFIG_EFI_LOADER is not set
+CONFIG_ENV_SIZE=0x2000
+CONFIG_ENV_OFFSET=0x2000
+CONFIG_ENV_OFFSET_REDUND=0x4000
+
diff --git a/include/configs/apalis-imx8.h b/include/configs/apalis-imx8.h
index 0d81069394..84802548c8 100644
--- a/include/configs/apalis-imx8.h
+++ b/include/configs/apalis-imx8.h
@@ -149,15 +149,15 @@
 
 /* Environment in eMMC, before config block at the end of 1st "boot sector" */
 #define CONFIG_ENV_SIZE			0x2000
-#define CONFIG_ENV_OFFSET		(-CONFIG_ENV_SIZE + \
-					 CONFIG_TDX_CFG_BLOCK_OFFSET)
-#define CONFIG_SYS_MMC_ENV_DEV		0
-#define CONFIG_SYS_MMC_ENV_PART		1
 
+
+#define CONFIG_SYS_REDUNDAND_ENVIRONMENT
+#define CONFIG_SYS_MMC_ENV_PART		1
+#define CONFIG_SYS_MMC_ENV_DEV		1   /* USDHC2 is 8-bit */
 #define CONFIG_SYS_MMC_IMG_LOAD_PART	1
 
 /* On Apalis iMX8 USDHC1 is eMMC, USDHC2 is 8-bit SD and USDHC3 is 4-bit SD */
-#define CONFIG_MMCROOT			"/dev/mmcblk0p2" /* USDHC1 eMMC */
+#define CONFIG_MMCROOT			"/dev/mmcblk1p2"  /* USDHC1 eMMC */
 #define CONFIG_SYS_FSL_USDHC_NUM	3
 
 #define CONFIG_SYS_BOOTM_LEN		(64 << 20) /* Increase max gunzip size */

0001-toradex-Integration-of-Mender-boot-code-into-U-Boot.patch

diff --git a/include/env_default.h b/include/env_default.h
index 86b639d3e2..f56a1135f8 100644
--- a/include/env_default.h
+++ b/include/env_default.h
@@ -9,7 +9,7 @@
  */
 
 #include <env_callback.h>
-
+#include <env_mender.h>
 #ifdef DEFAULT_ENV_INSTANCE_EMBEDDED
 env_t environment __UBOOT_ENV_SECTION__ = {
 	ENV_CRC,	/* CRC Sum */
@@ -22,6 +22,7 @@ static char default_environment[] = {
 #else
 const uchar default_environment[] = {
 #endif
+	MENDER_ENV_SETTINGS
 #ifdef	CONFIG_ENV_CALLBACK_LIST_DEFAULT
 	ENV_CALLBACK_VAR "=" CONFIG_ENV_CALLBACK_LIST_DEFAULT "\0"
 #endif
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index 00b8fb34aa..e312c8037d 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -109,7 +109,8 @@ define filechk_config_h
 	echo \#include \<configs/$(CONFIG_SYS_CONFIG_NAME).h\>;		\
 	echo \#include \<asm/config.h\>;				\
 	echo \#include \<linux/kconfig.h\>;				\
-	echo \#include \<config_fallbacks.h\>;)
+	echo \#include \<config_fallbacks.h\>;				\
+	echo \#include \<config_mender.h\>;)
 endef
 
 include/config.h: scripts/Makefile.autoconf create_symlink FORCE

local.conf

ACCEPT_FSL_EULA = "1" 
IMAGE_POSTPROCESS_COMMAND_remove = "do_qbsp_image;"

MENDER_ARTIFACT_NAME = "Test-1"
MENDER_DEVICE_TYPE = "Test"
INHERIT += "mender-full"

PREFERRED_PROVIDER_u-boot_mender-uboot = "u-boot-toradex"
PREFERRED_PROVIDER_virtual/bootloader_mender-uboot = "u-boot-toradex"

DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
ARTIFACTIMG_FSTYPE = "ext4"

MENDER_SERVER_URL = "https://hosted.mender.io"
MENDER_TENANT_TOKEN = "Removed token for posting on Mender Hub"
 
MENDER_FEATURES_ENABLE_append = " mender-uboot mender-image-sd"
MENDER_FEATURES_DISABLE_append = " mender-grub mender-image-uefi"

IMAGE_FSTYPES_remove = "tar.gz wic.gz wic.bmap mender.bmap sdimg.bmap sdcard.gz wic"

MENDER_STORAGE_DEVICE = "/dev/mmcblk1"
MENDER_STORAGE_TOTAL_SIZE_MB = "12288"
MENDER_IMAGE_BOOTLOADER_BOOTSECTOR_OFFSET = "66"
MENDER_BOOT_PART_SIZE_MB="32"
UBOOT_CONFIG_aplis-imx8 = "sd"
MENDER_UBOOT_STORAGE_DEVICE = "1"
MENDER_UBOOT_STORAGE_INTERFACE = "mmc"

#MENDER_BOOT_PART = "${MENDER_STORAGE_DEVICE}1"
#MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE}4"
#MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE}2"
#MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE}3"
#MENDER_DATA_PART_SIZE_MB = "12288"
#MENDER_FEATURES_DISABLE_append = " mender-growfs-data"

With this conf file and patches I have the following observations:

1- if I set:

MENDER_BOOT_PART_SIZE_MB=“0”

I get 3 partitions inseated of 4 (no boot partition), but the board doesnt boot. During compiling, i get this warning message:

WARNING: b2qt-embedded-qt5-image-1.0-r0 do_image_sdimg: MENDER_BOOT_PART_SIZE_MB is set to zero, but IMAGE_BOOT_FILES is not empty. The files are being omitted from the image.
Which is strange because IMAGE_BOOT_FILES is empty in local.conf. Any ideas where else it is getting set ?
2- if I print the uboot environemnt variables, I dont find any Mender variables even though those variables exist in the env.txt file produced by compiling u-boot. Is it possible that the default variables are loaded instead? I believe this is the reason I get the first issue I mentioned above, since mender-setup is not called during booting.

arch=arm
baudrate=115200
board=apalis-imx8
board_name=Apalis iMX8QM
board_rev=v1.0
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf
boot_fdt=try
boot_net_usb_start=usb start
boot_prefixes=/ /boot/
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_targets=mmc1 mmc2 mmc0 usb0 dhcp 
bootcmd=run distro_bootcmd
bootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;
bootcmd_mfg=select_dt_from_module_version && fastboot 0
bootcmd_mmc0=setenv devnum 0; run mmc_boot
bootcmd_mmc1=setenv devnum 1; run mmc_boot
bootcmd_mmc2=setenv devnum 2; run mmc_boot
bootcmd_usb0=setenv devnum 0; run usb_boot
bootcount=1
bootdelay=1
commit_atf=bb209a0
commit_mkimage=d7f9440d
commit_scfw=b929edfe
commit_secofw=27167ff2
console=ttyLP1,115200 earlycon=lpuart32,0x5a070000,115200
cpu=armv8
defargs=pci=nomsi
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
ethaddr=00:14:2d:68:8e:4c
ethprime=FEC
fastboot_dev=mmc0
fdt_addr=0x83000000
fdt_addr_r=0x84000000
fdtcontroladdr=fd67e368
fdtfile=fsl-imx8qm-apalis-v1.1-eval.dtb
hdp_addr=0x84000000
hdp_file=hdmitxfw.bin
image=Image
initrd_addr=0x83800000
initrd_high=0xffffffffffffffff
ipaddr=192.168.10.2
kernel_addr_r=0x82000000
kernel_image=Image
loadaddr=0x80280000
loadhdp=fatload mmc ${mmcdev}:${mmcpart} ${hdp_addr} ${hdp_file}
loadm4image_0=fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} ${m4_0_image}
loadm4image_1=fatload mmc ${mmcdev}:${mmcpart} ${loadaddr} ${m4_1_image}
m4_0_image=m4_0.bin
m4_1_image=m4_1.bin
m4boot_0=run loadm4image_0; dcache flush; bootaux ${loadaddr} 0
m4boot_1=run loadm4image_1; dcache flush; bootaux ${loadaddr} 1
mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi
mmcargs=setenv bootargs console=${console},${baudrate} root=${mmcroot} video=imxdpufb5:off video=imxdpufb6:off video=imxdpufb7:off
mmcautodetect=yes
mmcdev=0
mmcpart=1
mmcroot=/dev/mmcblk0p2 rootwait rw
netmask=255.255.255.0
panel=NULL
ramdisk_addr_r=0x86400000
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
script=boot.scr
scriptaddr=0x86000000
sec_boot=no
serial#=06852172
serverip=192.168.10.1
setup=run loadhdp; hdp load ${hdp_addr}; run mmcargs
soc=imx8
soc_type=imx8qm
usb_boot=usb start; if usb dev ${devnum}; then setenv devtype usb; run scan_dev_for_boot_part; fi
vendor=toradex
ver=U-Boot 2018.03-toradex_imx_v2018.03_4.14.98_2.3.0_bringup+g52f89f8382 (Apr 15 2021 - 17:12:01 +0000)

Which Toradex BSP branch are you guys using? I’m planning to use 5.2.0 which is the currently released branch.

Drew

The first pass is available. This only gets verdin-imx8mm and colibri-imx6ull working since they already existed in Dunfell. I tested against the Toradex 5.2.0 BSP.

1 Like

Hi Drew,

Thank you very much for this integration. I also tested it ( using Mender Integration Checklist ) on a Colibri-imx6ulll running “tdx-reference-minimal-image” based on Dunfell and Toradex BSP 5.2.0 and was able to successfully make a rootfs update through mender server :slight_smile:

Hi Drew,

Thanks for the updates. I have tested this on an apalis-imx8, and I get some errors, probably expected since, as you mention, it only works for verdin-imx8mm and colibri-imx6ull. Any hints on how to get this working for apalis-imx8?

I added files/apalis-imx8/0001-configs-toradex-board-specific-mender-integration.patch:

diff --git a/configs/apalis-imx8_defconfig b/configs/apalis-imx8_defconfig
index e213b6b42d..b72f651697 100644
--- a/configs/apalis-imx8_defconfig
+++ b/configs/apalis-imx8_defconfig
@@ -3,7 +3,9 @@ CONFIG_ARCH_IMX8=y
 CONFIG_SYS_TEXT_BASE=0x80020000
 CONFIG_SYS_MALLOC_F_LEN=0x8000
 CONFIG_ENV_SIZE=0x2000
-CONFIG_ENV_OFFSET=0xFFFFDE00
+CONFIG_ENV_OFFSET=0x800000
+CONFIG_ENV_OFFSET_REDUND=0x1000000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
 CONFIG_DM_GPIO=y
 CONFIG_BOOTAUX_RESERVED_MEM_BASE=0x88000000
 CONFIG_BOOTAUX_RESERVED_MEM_SIZE=0x08000000
diff --git a/include/configs/apalis-imx8.h b/include/configs/apalis-imx8.h
index 7c04a4e3e7..224d895cd1 100644
--- a/include/configs/apalis-imx8.h
+++ b/include/configs/apalis-imx8.h
@@ -126,10 +126,6 @@
 #define CONFIG_SYS_MEMTEST_START	0x90000000
 #define CONFIG_SYS_MEMTEST_END		0xc0000000
 
-/* Environment in eMMC, before config block at the end of 1st "boot sector" */
-#define CONFIG_SYS_MMC_ENV_DEV		0	/* USDHC1 eMMC */
-#define CONFIG_SYS_MMC_ENV_PART		1
-
 #define CONFIG_SYS_MMC_IMG_LOAD_PART	1
 
 /* On Apalis iMX8 USDHC1 is eMMC, USDHC2 is 8-bit and USDHC3 is 4-bit MMC/SD */

U-boot boot args:

Bootargs: pci=nomsi root=/dev/mmcblk0p2 ro rootwait

Mender grow size error:

[FAILED] Failed to start Mender service to grow data partition size.
[DEPEND] Dependency failed for File System Check on /dev/mmcblk0p4.
[DEPEND] Dependency failed for /data.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Grow File System on /data.

Nothing off the top of my head. I’ll try to give it a shot sometime this week and see if I can figure out anything more than you already have.

Drew

My problems were with the Toradex easy installer, I was not using the mender version of the image.json file, so the partitions were not being created. I fixed that yesterday, and was able to update via the mender web interface.

Now I am cleaning up, and will post my changes (I guess through a pull request?). One immediate question is about the MENDER_STORAGE_TOTAL_SIZE_MB variable. I see in an earlier post, this is set to 12228. I tried setting it to 16384 (16 GB of MMC), but the easy installer failed to flash the device. Should it be 12228 for a device with 16 GB of flash?

That’s great news. Yes, please do submit a pull request.

I am not sure what the max supported total size is. It will likely be at least a bit less than the total amount on the board since the boot loader and the Toradex Boot Configuration Block take up some space at the start of the device.

What I normally do is set that value pretty small, and set small values for the data partition size. This allows the image to be small, transfer and install times to be reasonable, and then let the mender-growfs-data feature fill the rest of the space.

Drew

I cannot seem to change MENDER_STORAGE_TOTAL_SIZE_MB to anything but it’s default (1024). I have tried large values (10240) and small values (2048), but then I get errors running the Toradex Easy Installer to flash the image. Are there other variables that I also need to change to increase the storage size?
teziError

The first thing to try is to verify that your modification of that size is taking effect.

bitbake -e <image-name> | grep ^MENDER_STORAGE_TOTAL_SIZE_MB=

will show you what the value is. It’s possible that it is being set in a layer somewhere without using a soft-default to allow overrides.

You also mentioned not using the Mender image.json. The build should be creating a full mender_tezi image in a tar file in the deploy directory. You just need to untar that into your USB key and it should all work.

Drew

I verified the size is taking effect via the method you mention.

The mender_tezi tar file contains:

├── image.json
├── imx-boot
├── marketing_mender_toradex.tar
├── mender_toradex_linux.png
├── prepare.sh
├── Reference-Minimal-Image-apalis-imx8.sdimg.bz2
├── uboot.env
└── wrapup.sh

What I use to flash the MMC memory from my local Ubuntu host is the Toradex easy installer from their jfrog artifactory, with connections to the target USB OTG and Ethernet (Linux host serving nginx, DNS and DHCP). It contains (I just run recovery-linux.sh):

├── boot.scr
├── dpfw.bin
├── hdmitxfw.bin
├── image.json
├── imx-boot
├── recovery
│   ├── uuu
│   ├── uuu.auto
│   └── uuu.exe
├── recovery-linux.sh
├── recovery-windows.bat
├── tezi.itb
├── tezi.png
├── u-boot.bin
└── wrapup.sh

So I don’t really see how I can use the Yocto generated mender_tezi image. Maybe there is a mender variable that I am setting wrong?

After running recovery-linux.sh, you should be able to put a USB key into the board containing the extracted contents of the mender_tezi tar file. It will show up in the list with a USB key icon and can be installed from there. Let me know if that is not what you are doing.

Drew

No, that is not what I am doing. I am using unattended flashing over ethernet. I put the board into recovery mode, and supply a DNS entry to tezi.toradex.com. The machine at that address has nginx running, and the image is stored in the web server location. This is described at Toradex Easy Installer - Detailed Manual

I haven’t tried with a USB stick. I will do so.

Yeah, the images provided by the Toradex feeds generally don’t have Mender integrated and certainly not for the apalis-imx8.
Drew

I got it working with the USB stick, so the issue appears to be with the ethernet flashing of the toradex easy installer, or the way I am using it. I took the mender_tezi tar file and put it on a USB stick, no problem flashing at large sizes. Using the exact same contents for the flashing over ethernet, and I get the error listed above.

I have raised this issue on the Toradex community:
https://www.toradex.com/community/questions/65447/easy-installer-fails-with-large-images.html

Now I will focus on getting my changes in order for the apalis-imx8 in the meta-mender-toradex-nxp layer.

Except I cannot seem to create a branch in this repo. Should I just paste the patch here?

You need to create a fork of meta-mender-community which you will then have write access to. You push your changes to a branch of your fork and then use that to submit a PR against the primary meta-mender-community repo. See this link for more details.

(FWIW, the ticket you created in the Toradex community is assigned to me; I just have not had a chance to get to it yet.)

Drew