Mender on IMX8 + Yocto

I’m working on getting Mender going on an IMX8 setup. I’m by no means an expert on Yocto, IMX8, or Mender. :slight_smile:

I have an image building that, when loaded, does appear to have Mender running on it (systemctl status mender-client).

However, from the “Integration Checklist”, I can’t get fw_printenv to work.

I think it’s conflicting with upstream board stuff, like the location of the u-boot environment variables.

In general, does it make sense for the upstream board stuff to conform to Mender, or Mender to conform to the upstream board stuff?

I am using these which is where things were stored BEFORE going to Mender:

MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 = “0x3F0000”
BOOTENV_SIZE = “0x2000”

However, I still see a /etc/fw_env.config of:

/dev/mmcblk0 0x3F0000 0x2000
/dev/mmcblk0 0x1000000 0x2000

Where is that second one coming from, and how do I turn that off? I’m not sure my board supports it.

Thank you for your help! Tagging @TheYoctoJester as he’s been a large help in these forums and is def. a guru of Mender in Yocto! :slight_smile:

Hi @zapados, YoctoJester is on vacation at this time, so I will try to help you. I have a couple of questions for you:

  1. Have you taken a look at this thread? Is it something that could be helpful to you?
  2. What error do you get when you run fw_printenv?
  3. Which steps did you follow to integrate u-boot with Yocto and Mender?

Cheers,
Jana

1 Like

Hi Jana,

Thank you for your help!

  1. I’m not to that step yet - just trying to get fw_printenv working reliably for the integration checklist.
  2. An update here - I can get fw_printenv to work, but I need to manually comment out the second line (the u-boot environment redundant location I think?).

root@xxx:~# cat /etc/fw_env.config
/dev/mmcblk0 0x3F0000 0x2000
#/dev/mmcblk0 0x1000000 0x2000
root@xxx:~#

When that’s not commented out, I get:

root@xxx:~# fw_printenv
Cannot read environment, using default
Cannot read default environment from file
root@xxx:~#

  1. I’ve been following the Mender integration documentation, but focused on standalone full-disk deployments for now.

u-boot-imx_%.bbappend:

FILESEXTRAPATHS:prepend := “${THISDIR}/${PN}:”

# Needed for Mender to work on a u-boot fork.
require recipes-bsp/u-boot/u-boot-mender.inc
PROVIDES += " u-boot"
RPROVIDES_${PN} += “u-boot”

SRC_URI += “file://enable-pxe-xxx-0001.patch”

local.conf stuff (may move around to more appropriate places later):

   INHERIT += "mender-full"
   MENDER_BOOT_PART_SIZE_MB = "512"
   MENDER_STORAGE_DEVICE = "/dev/mmcblk0"
   MENDER_STORAGE_TOTAL_SIZE_MB = "15028"
   MENDER_FEATURES_ENABLE:append = " mender-uboot mender-image-sd"
   MENDER_FEATURES_DISABLE:append = " mender-grub mender-image-uefi"
   MENDER_ARTIFACT_NAME = "xxx"
   IMAGE_FSTYPES:append = " sdimg.bmap"
   MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 = "0x3F0000"
   BOOTENV_SIZE = "0x2000"

Cheers,
Adam L.

I don’t see either of these cases: “Unfortunately, because of the great variation in U-Boot board code, the automatic patching process does not always succeed or produce a working boot loader, even if the above criteria are met. The symptoms are either that one of the u-boot recipes produce compile errors, or that the board does not boot using the modified bootloader. If this happens, or if you are using an older Yocto Project branch, there will be some manual work required in order to produce a working integration patch.”

I don’t get a compile error, and the board does boot, but I’m starting to wonder if it’s not booting with the modified u-boot. For example, further in the integration checklist is the step of setting it to boot off the alternate partition:

fw_setenv mender_boot_part 3
fw_setenv mender_boot_part_hex 3
reboot

However, when I inspect my u-boot environment variables with fw_printenv, I don’t see any mender_ contents!

root@xxx:~# fw_printenv | grep mender
root@xxx:~#

Does this mean something already went wrong?

Hi,

However, when I inspect my u-boot environment variables with fw_printenv, I don’t see any mender_ contents!
Does this mean something already went wrong?

Yes, I think this would be a clear indication that Mender was not integrated properly.
Can you tell me following things:

  1. Which yocto release are you using?
  2. Which layers do you have configured in you bblayers.conf? Please verify that all necessary yocto layers (for your board and mender) are included
  3. Is there anything useful in mender logs? You can check them by running journalctl | grep mender*
  4. Which imx8 board are you using? (What’s the value of your MACHINE variable?)

If the issue was wrong fw_env.config only - there would be a quick (but not pretty) workaround for it: manually adding fw_env.config using a bbappend file. But in your case the issue seems to go deeper.

Cheers,
Jana

1 Like

DISTRO = “poky”
DISTRO_NAME = “Poky (Yocto Project Reference Distro)”
DISTRO_VERSION = “4.0.4”
DISTRO_CODENAME = “kirkstone”

  1. I’m using kas to manage layers. What Mender layers should I be including?

// a-sm2s-imx8
LCONF_VERSION = “7”
BBPATH = “${TOPDIR}”
BBFILES ?= “”
BBLAYERS += "
/work/sources/yocto.git/meta-poky
/work/sources/yocto.git/meta-yocto-bsp
/work/sources/01032/msc-ldk-bsp-recipes.git/meta
/work/sources/yocto.git/meta
/work/sources/meta-openembedded.git/meta-oe
/work/sources/meta-openembedded.git/meta-networking
/work/sources/meta-openembedded.git/meta-python
/work/sources/meta-python2.git
/work/sources/meta-qt5.git
/work/sources/meta-secure-core.git/meta
/work/sources/meta-secure-core.git/meta-integrity
/work/sources/meta-secure-core.git/meta-signing-key
/work/sources/meta-secure-core.git/meta-tpm
/work/sources/meta-secure-core.git/meta-tpm2
/work/sources/meta-msc-ldk-core-recipes.git
/work/sources/meta-msc-ldk-core.git
/work/sources/meta-msc-ldk-mscio.git
/work/sources/meta-freescale.git
/work/sources/meta-freescale-distro.git
/work/sources/meta-imx.git/meta-bsp
/work/sources/meta-imx.git/meta-sdk
/work/sources/meta-imx.git/meta-ml
/work/sources/meta-nxp-demo-experience.git
/work/sources/meta-msc-arm-extensions.git
/work/sources/meta-msc-ldk-marvell.git
/work/sources/meta-openembedded.git/meta-multimedia
/work/sources/meta-openembedded.git/meta-gnome
/work/sources/meta-openembedded.git/meta-filesystems
/work/sources/meta-openembedded.git/meta-perl
/work/sources/meta-openembedded.git/meta-webserver
/work/sources/meta-msc-ldk-rt.git
/work/sources/meta-virtualization.git
/work/sources/meta-mender.git/meta-mender-core
"
BBLAYERS_NON_REMOVABLE ?= "
/work/sources/yocto.git/meta
/work/sources/yocto.git/meta-poky
"
BBLAYERS ?= "
/repo/meta-xxx
/work/sources/01032/msc-ldk-bsp-recipes.git/meta
/work/sources/meta-freescale-distro.git
/work/sources/meta-freescale.git
/work/sources/meta-imx.git/meta-bsp
/work/sources/meta-imx.git/meta-sdk
/work/sources/meta-mender.git/meta-mender-core
/work/sources/meta-msc-arm-extensions.git
/work/sources/meta-msc-ldk-core-recipes.git
/work/sources/meta-msc-ldk-core.git
/work/sources/meta-msc-ldk-marvell.git
/work/sources/meta-msc-ldk-mscio.git
/work/sources/meta-msc-ldk-rt.git
/work/sources/meta-nxp-demo-experience.git
/work/sources/meta-openembedded.git/meta-filesystems
/work/sources/meta-openembedded.git/meta-gnome
/work/sources/meta-openembedded.git/meta-multimedia
/work/sources/meta-openembedded.git/meta-networking
/work/sources/meta-openembedded.git/meta-oe
/work/sources/meta-openembedded.git/meta-perl
/work/sources/meta-openembedded.git/meta-python
/work/sources/meta-openembedded.git/meta-webserver
/work/sources/meta-python2.git
/work/sources/meta-qt5.git
/work/sources/meta-secure-core.git/meta
/work/sources/meta-secure-core.git/meta-integrity
/work/sources/meta-secure-core.git/meta-signing-key
/work/sources/meta-secure-core.git/meta-tpm
/work/sources/meta-secure-core.git/meta-tpm2
/work/sources/meta-virtualization.git
/work/sources/yocto.git/meta
/work/sources/yocto.git/meta-poky
/work/sources/yocto.git/meta-yocto-bsp"
BBPATH ?= “${TOPDIR}”
BBFILES ??= “”

  1. Lots of authorization failures, but I’m trying to get a standalone Mender going first, so I expect this:

Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=warning msg=“Reauthorization failed with error: transient error: authorization request failed”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=error msg=“Error receiving scheduled update data: update check request failed: transient error: authorization request failed”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=error msg=“Update check failed: transient error: update check request failed: transient error: authorization request failed”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=info msg=“State transition: update-check [Sync] → error [Error]”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=info msg=“Handling error state, current error: transient error: update check request failed: transient error: authorization request failed”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=info msg=“State transition: error [Error] → idle [Idle]”
Aug 26 18:53:36 xxx mender[1174]: time=“2024-08-26T18:53:36+02:00” level=info msg=“State transition: idle [Idle] → check-wait [Idle]”

Some startup stuff, but this looks good to me as I think the issue to fix first is u-boot related:

Aug 23 18:14:14 xxx mender-client-resize-data-part[339]: Disk has already been resized.
Aug 23 18:14:16 xxx systemd[1]: Mender persistent data dir was skipped because of a failed condition check (ConditionPathExists=!/data/mender).
Aug 23 18:16:17 xxx mender[1174]: time=“2024-08-23T18:16:17+02:00” level=info msg=“Loaded configuration file: /var/lib/mender/mender.conf”
Aug 23 18:16:17 xxx mender[1174]: time=“2024-08-23T18:16:17+02:00” level=info msg=“Loaded configuration file: /etc/mender/mender.conf”
Aug 23 18:16:17 xxx mender[1174]: time=“2024-08-23T18:16:17+02:00” level=info msg=“‘UpdateControlMapExpirationTimeSeconds’ is not set in the Mender configuration file. Falling back to the default of 2*UpdatePollIntervalSeconds”
Aug 23 18:16:17 xxx mender[1174]: time=“2024-08-23T18:16:17+02:00” level=info msg=“‘UpdateControlMapBootExpirationTimeSeconds’ is not set in the Mender configuration file. Falling back to the default of 600 seconds”
Aug 23 18:16:19 xxx mender[1174]: time=“2024-08-23T18:16:19+02:00” level=info msg=“Mender running on partition: /dev/mmcblk0p2”
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=info msg=“State transition: init [none] → init [none]”
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=info msg=“State transition: init [none] → idle [Idle]”
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=info msg=“State transition: idle [Idle] → check-wait [Idle]”
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=info msg=“State transition: check-wait [Idle] → inventory-update [Sync]”
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=info msg=“Output (stderr) from command "/usr/share/mender/inventory/mender-inventory-geo": /usr/share/mender/inventory/mender-inventory-geo: Unable to get IP info from ipinfo.io
Aug 23 18:16:21 xxx mender[1174]: time=“2024-08-23T18:16:21+02:00” level=warning msg=“Inventory tool /usr/share/mender/inventory/mender-inventory-geo wait failed: exit status 2”
Aug 23 18:16:24 xxx mender[1174]: time=“2024-08-23T18:16:24+02:00” level=info msg=“Device unauthorized; attempting reauthorization”

  1. Currently working on an sm2s-imx8 from Avnet. https://embedded.avnet.com/product/msc-sm2s-imx8/

Do you think I should start tackling the manual u-boot integration based on the fact that there is no mender_ stuff in my u-boot environment? I was hoping to avoid it so it’s more future-proofed against Yocto and Mender updates.

Hmm when I look at build/tmp/work/xxx/u-boot-imx/2022.04-r0/mender_auto_configured.patch, I don’t see it adding any mender_ stuff there!

When does that come in to msc_sm2s_imx8.h?

I don’t think this is the EXTERNALSRC issue discussed here: https://hub.mender.io/t/mender-u-boot-env-vars-missing-from-manual-patch/3209/12:

builder@f54f8a31e876:/build$ bitbake -e u-boot-imx | grep --ignore-case “EXTERNALSRC”
# $EXTERNALSRC
# [doc] “If externalsrc.bbclass is inherited, this variable points to the source tree, which is outside of the OpenEmbedded build system.”
# $EXTERNALSRC_BUILD
# [doc] “If externalsrc.bbclass is inherited, this variable points to the directory in which the recipe’s source code is built, which is outside of the OpenEmbedded build system.”
if [ -n “${EXTERNALSRC}” ]; then
bbwarn “Skipping U-Boot auto-configuration when >EXTERNALSRC is set. Would change the source files. If you need the patch, turn off EXTERNALSRC, call ‘bitbake -c save_mender_auto_configured_patch u-boot’ to generate a patch, and then apply that to your work tree and reenable EXTERNALSRC.”
# e.g. using externalsrc; use S as base for our dirs
# debugsources.list may be polluted from the host if we used externalsrc,
# Same check as above for externalsrc
builder@f54f8a31e876:/build$

Hi @zapados,

To be honest I’ve never used kas before - and am not a big fan of it - but I believe your issue will require some deeper digging into Yocto which might not be possible with kas.

Second, I see you have a long list of layers in your bblayers.conf. I’d suggest first making a build with only necessary layers required (mender, bsp and poky related layers). After you set up the most simple build, and get Mender to work on it, I’d suggest adding more stuff to it. This way not only does the build finish faster, but debugging it is easier and it’s more clear what’s going on.

What Mender layers should I be including?

I believe meta-mender-core is sufficient in your case.

Do you think I should start tackling the manual u-boot integration based on the fact that there is no mender_ stuff in my u-boot environment?

That’s exactly what I would do next.

Lastly, I want to point out that the board you’re using has not been tested with Mender before. (source) Because of it, integrating Mender on it could be more difficult than it seems. Sorry I couldn’t be of more help.

Best of luck,
Jana

1 Like

The manual u-boot integration steps don’t talk about how to add the mender_* variables - is it expected those are already present in the automatically generated patch? What if they are not, as is my case?

Sadly this project has been shut down.

Thank you for all your help along the way!

1 Like