i.MX8M Mini (Nitrogen 8) - Mender and Scarthgap Integration

Hi all,

I am currently working on the Nitrogen8M Mini, an i.MX8 based board with Yocto Scarthgap.
I have made several test and attempts to integrate Mender with my board, with no luck so far. My image compiles, however it does not boot into the kernel.

The changes and tests I have made are based on Mender’s Manual U-Boot Integration, as well as the patches previously applied to the board support in the community layer for NXP boards.

Below is the Mender-related configuration from my local.conf

# Mender parameters

# MENDER_UBOOT_STORAGE_DEVICE = "1"
# According to uboot's variables with mmc list:
#
# FSL_SDHC: 0 -> eMMC
# FSL_SDHC: 1 -> SD
# Using storage device 1
MENDER_STORAGE_DEVICE = "/dev/mmcblk1" 
# Patches must be in alignment with Mender storage Device. 
# Set CONFIG_SYS_MMC_ENV_DEV = 1
#
# Memory card with 256GiB of storage.
MENDER_STORAGE_TOTAL_SIZE_MB = "5000"
MENDER_BOOT_PART_SIZE_MB ?= "128"
MENDER_DATA_PART_SIZE_MB ?= "2048"

MENDER_FEATURES_ENABLE:append = " mender-uboot mender-image-sd"
MENDER_FEATURES_DISABLE:append = " mender-grub mender-image-uefi"

MENDER_ARTIFACT_NAME = "release-1"
INHERIT += "mender-full"

DISTRO_FEATURES:append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED =" sysvinit"
VIRTUAL-RUNTIME_initscripts = ""

MENDER_IMAGE_BOOTLOADER_BOOTSECTOR_OFFSET = "33"

Applying the vendor’s patches along with my own modifications, I have set the required variables by Mender:

diff --git a/configs/imx8mm_nitrogen_smarc_4g_defconfig b/configs/imx8mm_nitrogen_smarc_4g_defconfig
index e29d2cc8aa1..fccb8e2e098 100644
--- a/configs/imx8mm_nitrogen_smarc_4g_defconfig
+++ b/configs/imx8mm_nitrogen_smarc_4g_defconfig
@@ -12,7 +12,7 @@ CONFIG_NR_DRAM_BANKS=2
 CONFIG_SYS_MEMTEST_START=0x40000000
 CONFIG_SYS_MEMTEST_END=0x40010000
 CONFIG_ENV_SIZE=0x2000
-CONFIG_ENV_OFFSET=0xffffe000
+CONFIG_ENV_OFFSET=0x3fe000
 CONFIG_IMX_CONFIG="arch/arm/mach-imx/imx8m/imximage.cfg"
 # CONFIG_SPLASH_SCREEN_PREPARE is not set
 CONFIG_DDR_MB=4096
@@ -92,9 +92,12 @@ CONFIG_OF_LIVE=y
 CONFIG_ENV_OVERWRITE=y
 CONFIG_ENV_IS_IN_MMC=y
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
+CONFIG_SYS_MMC_ENV_DEV=1
 CONFIG_SYS_MMC_ENV_PART=1
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_NET_RANDOM_ETHADDR=y
+CONFIG_BOOTCOUNT_LIMIT=y
+CONFIG_BOOTCOUNT_ENV=y
 CONFIG_NETCONSOLE=y
 CONFIG_SPL_DM=y
 CONFIG_REGMAP=y
@@ -180,3 +183,5 @@ CONFIG_SPLASH_SCREEN_ALIGN=y
 CONFIG_VIDEO_BMP_GZIP=y
 CONFIG_IMX_WATCHDOG=y
 CONFIG_LIBAVB=y
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
+CONFIG_ENV_OFFSET_REDUND=0x3fc000

The boot.scr is also modified to include the command run mender_setup and load the Image and DT from the /boot partition:

run mender_setup
mmc dev ${mender_uboot_dev}
load ${mender_uboot_root} ${a_zImage} /boot/Image
load ${mender_uboot_root} ${a_fdt} /boot/${fdt_file}
fdt addr ${a_fdt}

With this, I am able to build an image, but the device is unable to boot. I see two errors that catch my attention: “mender_setup” not defined and then the FDT address problem.

switch to partitions #0, OK
mmc1 is current device
Scanning mmc 1:1...
Found U-Boot script /boot.scr
5314 bytes read in 2 ms (2.5 MiB/s)
## Executing script at 40480000
cpu2=8M
cpu3=8MM
Failed to load 'uEnv.txt'
## Error: "mender_setup" not defined
switch to partitions #0, OK
mmc1 is current device
** Bad device specification 40800000 /boot/Image **
Couldn't find partition 40800000 /boot/Image
Can't set block device
** Bad device specification 43000000 /boot/imx8mm-nitrogen_smarc **
Couldn't find partition 43000000 /boot/imx8mm-nitrogen_smarc.dtb
Can't set block device
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!

I am not sure if mender_setup failing to be executed is not setting up the following variables that are in the script, which is why the Image nor the DT are found.

load ${mender_uboot_root} ${a_zImage} /boot/Image

In terms of the address for the DT not being recognized, I’m not entirely sure, as it is declared in u-boot’s variables (a_fdt):

a_fdt=43000000
a_initrd=43100000
a_reset_cause=40000084
a_reset_cause_marker=40000080
a_script=40800000
a_zImage=40800000
arch=arm
baudrate=115200
board=nitrogen_smarc
board_name=nitrogen_smarc
boot_a_script=setenv disk ${devnum}; setenv dtype ${devtype}; setenv bootpart ${distro_bootpart}; load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
boot_net_usb_start=usb start
boot_prefixes=/boot/ /
boot_script_dhcp=boot.scr.uimg
boot_scripts=boot.scr.uimg boot.scr
boot_syslinux_conf=extlinux/extlinux.conf
boot_targets=mmc1 mmc0 usb0
bootargs=console=ttymxc1,115200 consoleblank=0 rootwait fixrtc root=${mender_kernel_root} snd.slots=,snd-soc-imx-hdmi
bootcmd=run distro_bootcmd
bootcmd_mmc0=devnum=0; run mmc_boot
bootcmd_mmc1=devnum=1; run mmc_boot
bootcmd_usb0=devnum=0; run usb_boot
bootdelay=2
bootfstype=fat
bootpart=1
cmd_board=fdt set pcie-phy fsl,refclk-pad-mode <2>
cmd_mipi=fdt set backlight_mipi status okay;fdt set lcdif status okay;fdt set mipi_dsi status okay;fdt set fb_mipi status okay;fdt set mipi_to_lvds status okay;fdt set ts_goodix status okay;fdt set fb_mipi dsi-format rgb888;fdt set mipi mode-skip-eot;fdt rm mipi_cmds_m101nwwb mode-video-hbp-disable;fdt rm mipi_cmds_m101nwwb mode-video-hfp-disable;fdt rm mipi_cmds_m101nwwb mode-video-hsa-disable;fdt set mipi mode-video;fdt rm mipi mode-video-burst;fdt set mipi mode-video-sync-pulse;fdt get value cmds mipi_cmds_m101nwwb phandle;fdt set mipi mipi-cmds <${cmds}>;fdt set mipi dsi-lanes <4>;fdt get value pwm pwm_mipi phandle;fdt set backlight_mipi pwms <${pwm} 0 32000 0>;fdt set t_mipi clock-frequency <71700000>;fdt set t_mipi hactive <1280>;fdt set t_mipi vactive <800>;fdt set t_mipi hback-porch <5>;fdt set t_mipi hfront-porch <151>;fdt set t_mipi vback-porch <2>;fdt set t_mipi vfront-porch <28>;fdt set t_mipi hsync-len <4>;fdt set t_mipi vsync-len <1>;
console=ttymxc1
cpu=armv8
cpu2=8M
cpu3=8MM
disk=1
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
dtype=mmc
env_dev=0
env_part=1
eth1addr=00:19:b8:00:00:02
ethact=FEC
ethaddr=00:19:b8:10:3c:bc
ethprime=FEC
fastboot_raw_partition_bootloader=0x42 0x1fae mmcpart 1
fastboot_raw_partition_bootloader-env=0x1ff0 0x10 mmcpart 1
fb_mipi_name=tm070jdhg30-1
fdt_addr=0x43000000
fdt_high=0xffffffffffffffff
fdtcontroladdr=f6d112d0
fileaddr=40480000
filesize=14c2
fuse1=1 3
fuse1_val=10002022
fuse_mac1a=9 1
fuse_mac1a_val=00000019
fuse_mac1b=9 0
imx_cpu=8MMQ
initrd_high=0xffffffffffffffff
loadaddr=0x40480000
m4boot=load ${devtype} ${devnum}:1 ${m4loadaddr} ${m4image}; dcache flush; bootaux ${m4loadaddr}
m4image=m4_fw.bin
m4loadaddr=0x007E0000
mcore_bootargs=clk-imx8mm.mcore_booted
mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fi
net_upgradeu=dhcp 40020000 net_upgradeu.scr && source 40020000
netargs=setenv bootargs console=${console},115200 root=/dev/nfs rw ip=dhcp nfsroot=${tftpserverip}:${nfsroot},v3,tcp
netboot=run netargs; if test -z "${fdt_file}" -a -n "${soc}"; then setenv fdt_file ${soc}-${board}${boardver}.dtb; fi; if test ${ip_dyn} = yes; then setenv get_cmd dhcp; else setenv get_cmd tftp; fi; ${get_cmd} ${loadaddr} ${tftpserverip}:Image; if ${get_cmd} ${fdt_addr} ${tftpserverip}:${fdt_file}; then booti ${loadaddr} - ${fdt_addr}; else echo WARN: Cannot load the DT; fi;
otg_upgradeu=run usbnetwork; tftp 40020000 net_upgradeu.scr && source 40020000
reset_cause=1
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplist
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo script exited: continuing...; fi; done
scriptaddr=0x40480000
serial#=0019b8103cbc
soc=imx8m
uboot_defconfig=imx8mm_nitrogen_smarc_4g
uboot_release=2022.04-63944-gab4b98364b6
upgradeu=setenv boot_scripts upgrade.scr; boot;echo Upgrade failed!; setenv boot_scripts boot.scr
usb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fi
usbnet_devaddr=00:19:b8:00:00:02
usbnet_hostaddr=00:19:b8:00:00:01
usbnetwork=setenv ethact usb_ether; setenv ipaddr 10.0.0.2; setenv netmask 255.255.255.0; setenv serverip 10.0.0.1;
vendor=boundary
vidconsole=vidconsole

I am not exactly sure what could be missing to be integrated so that mender_setup is recognized, as I believe that to be my main issue.

I would appreaciate any advice or guidance you can give.

Thanks!

Hi @JCori,

By looking at the environment, I would guess that the patch here is missing, or not being properly pulled in as via this. Can you check that both have been properly applied?

Greetz,
Josef

Hi Josef,

Thanks for the quick response! Sure, here is an excerpt from the logs when building.
I can also see both files in the include/ folder in my working tree.

NOTE: Applying patch '0002-Integration-of-Mender-boot-code-into-U-Boot-no-fuzz.patch' (../meta-sgl-spec/recipes-bsp/u-boot/u-boot-boundary/0002-Integration-of-Mender-boot-code-into-U-Boot-no-fuzz.patch)
NOTE: Applying patch '0005-modify-mmx-dev-part-in-imx8mm-nitrogen-smarc-and-som.patch' (../meta-sgl-spec/recipes-bsp/u-boot/u-boot-boundary/0005-modify-mmx-dev-part-in-imx8mm-nitrogen-smarc-and-som.patch)
NOTE: Applying patch '0006-Add-remaining-nitrogen8mm-configs-mender-support.patch' (../meta-sgl-spec/recipes-bsp/u-boot/u-boot-boundary/0006-Add-remaining-nitrogen8mm-configs-mender-support.patch)
NOTE: Applying patch '0001-Generic-boot-code-for-Mender.patch' (../upstream/meta-mender/meta-mender-core/recipes-bsp/u-boot/patches/0001-Generic-boot-code-for-Mender.patch)
DEBUG: Python function patch_do_patch finished
DEBUG: Python function do_patch finished

I reworked this patch because it reported Fuzz issues with my versions of env_default and Makefile.autoconf , but I’ve ensured that the patches apply the same changes.

 include/env_default.h     | 3 +++
 scripts/Makefile.autoconf | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/env_default.h b/include/env_default.h
index 21afd7f7dcf..4b6426bc5f7 100644
--- a/include/env_default.h
+++ b/include/env_default.h
@@ -14,6 +14,8 @@
 #include <generated/environment.h>
 #endif
 
+#include <env_mender.h>
+
 #ifdef DEFAULT_ENV_INSTANCE_EMBEDDED
 env_t embedded_environment __UBOOT_ENV_SECTION__(environment) = {
 	ENV_CRC,	/* CRC Sum */
@@ -28,6 +30,7 @@ char default_environment[] = {
 #else
 const char default_environment[] = {
 #endif
+	MENDER_ENV_SETTINGS
 #ifndef CONFIG_USE_DEFAULT_ENV_FILE
 #ifdef	CONFIG_ENV_CALLBACK_LIST_DEFAULT
 	ENV_CALLBACK_VAR "=" CONFIG_ENV_CALLBACK_LIST_DEFAULT "\0"
diff --git a/scripts/Makefile.autoconf b/scripts/Makefile.autoconf
index 0b3ffa08bfa..6b97a2945d8 100644
--- a/scripts/Makefile.autoconf
+++ b/scripts/Makefile.autoconf
@@ -109,7 +109,8 @@ define filechk_config_h
 	echo \#include \<configs/$(CONFIG_SYS_CONFIG_NAME).h\>;		\
 	echo \#include \<asm/config.h\>;				\
 	echo \#include \<linux/kconfig.h\>;				\
-	echo \#include \<config_fallbacks.h\>;)
+	echo \#include \<config_fallbacks.h\>;			\
+	echo \#include \<config_mender.h\>;)
 endef
 
 include/config.h: scripts/Makefile.autoconf create_symlink FORCE

Another detail that might be important but that I cannot seem to trace back is that, in my generated images folder, I see some u-boot-boundary-initial-env files that seem to have the mender variables defined; this leads me to believe the patches are working, but the variables get lost along the way.

mender_boot_part=2
mender_boot_part_hex=2
mender_uboot_boot=mmc 1:1
mender_uboot_if=mmc
mender_uboot_dev=1
mender_boot_kernel_type=booti
mender_kernel_name=Image
mender_dtb_name=imx8mm-nitrogen_smarc-tevs.dtb
mender_pre_setup_commands=
mender_post_setup_commands=
mender_check_saveenv_canary=1
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; if test "${mender_systemd_machine_id}" != ""; then setenv bootargs "systemd.machine_id=${mender_systemd_machine_id} ${bootargs}"; fi; setenv mender_kernel_root /dev/mmcblk1p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk1p2; else setenv mender_boot_part_name /dev/mmcblk1p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 1:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \\"${bootargs}\\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_post_setup_commands; fi
mender_altbootcmd=if test ${mender_boot_part} = 2; then setenv mender_boot_part 3; setenv mender_boot_part_hex 3; else setenv mender_boot_part 2; setenv mender_boot_part_hex 2; fi; setenv upgrade_available 0; saveenv; run mender_setup
mender_try_to_recover=if test ${upgrade_available} = 1; then reset; fi

Thanks,

Hi @JCori

So are you building/patching u-boot-boundary? Or are two u-boot incarnations getting mixed up here?

Greetz,
Josef

Hi Josef,

Yes, I’m only working with u-boot-boundary.

Thanks,

Hi @JCori,

A long shot, but: is there possibly a pre-integration environment of u-boot still present and in effect? What happens if you reset the environment to default, like env default -a and then print it?

Greetz,
Josef

Hi Josef,

Partially, yes, that is what was happening. This boards u-boot needs to be upgraded via the update script, which in turn will provide with the definition of the Mender variables. The issue was in Boundary Device’s upstream, since apparently there was no u-boot configuration for the model of my board at the moment, therefore the upgrade could not be done.
So, in case anyone stumbles onto this later, adding the UBOOT_CONFIG for my model was necessary.

UBOOT_CONFIG = "2g 2gr0 4g rev2_2g rev2_2gr0 rev2_4g som_1gr0 som_2g som_2gr0 som_4g smarc_2gr0 smarc_4g"
UBOOT_CONFIG[smarc_4g]   = "imx8mm_nitrogen_smarc_4g_defconfig,sdcard"

IMAGE_BOOT_FILES:append = " \
	flash.bin-${MACHINE}-smarc_4g;u-boot.imx8mm_nitrogen_smarc_4g \
"

After flashing, interrupt the automatic booting and run

run upgradeu

This now produces the correct set of variables and my board boots.

Now there is a new issue: after booting, I can see in dmesg

[FAILED] Failed to start Mender service to grow data partition size.
See 'systemctl status mender-grow-data.service' for details.

Following the integration checklist, I find that the bootloader environment tools are not correctly set up, since I see the following output.

root@nitrogen8mm:~# fw_printenv
Cannot initialize environment
root@nitrogen8mm:~# fw_setenv
Cannot initialize environment

Inspecting the fw_env.config:

/dev/mmcblk1boot0 0x3fe000 0x2000
/dev/mmcblk1boot0 0x3fc000 0x2000

The other Mender services and A/B partitions look right:

root@nitrogen8mm:~# systemctl is-active mender-authd
active
root@nitrogen8mm:~# systemctl is-enabled mender-authd
enabled
root@nitrogen8mm:~# systemctl is-active mender-updated
active
root@nitrogen8mm:~# systemctl is-enabled mender-updated
enabled
cat /var/lib/mender/mender.conf | grep RootfsPart
    "RootfsPartA": "/dev/mmcblk1p2",
    "RootfsPartB": "/dev/mmcblk1p3"

Any ideas on this? I’d really appreciate your input.

Thanks,

Hi @JCori,

Ok, that’s some progress! Looking at it, I’m almost certain that boot.scr bites you. Probably you also need to adjust it, something like meta-mender-community/meta-mender-nxp/recipes-bsp/u-boot/u-boot-script-boundary/0001-boundary-bootscripts-bootscript-yocto-add-Mender-sup.patch at kirkstone · mendersoftware/meta-mender-community · GitHub

Greetz,
Josef

Hi Josef,

Thanks for the info! In my original setup I had also used Boundary’s patch from Kirkstone to try the integration with Scarthgap.

And yes it seems there is something going on during the boot.src, specifically during the run_mender setup. I followed this thread, as it seemed relevant to the errors I was getting. In doing so, I saw that there is a failure when switching partitions.

=> print mender_saveenv_canary
## Error: "mender_saveenv_canary" not defined
=> run mender_setup
Saving Environment to MMC... MMC partition switch failed
MMC partition switch failed
Failed (1)
=> print mender_saveenv_canary
mender_saveenv_canary=1

So, mender_setup fails because it is not able to switch the partitions, more likely at the saveenv command.

Hi @JCori,

So the obvious question: is saveenv in itself functional? If not, that’s the thing to figure out. Are you sure that the environment offsets are valid, for example?

Greetz,
Josef