Beaglebone missing default u-boot environment in MMC

Hello, it appears that the beaglebone sdimg is missing the default u-boot environment, causing mender to complain.

Background

I followed the instructions here on the Mender Hub for the beaglebone reference platform. I then made these additions to local.conf to switch to u-boot and to support my beaglebone black running from the internal mmc flash (which is /dev/mmcblk1):

MENDER_FEATURES_ENABLE_append = " mender-uboot mender-image-sd"
MENDER_FEATURES_DISABLE_append = " mender-grub mender-image-uefi"
MENDER_UBOOT_STORAGE_INTERFACE = "mmc"
MENDER_UBOOT_STORAGE_DEVICE = "1"
MENDER_STORAGE_TOTAL_SIZE_MB = "3072"
MENDER_BOOT_PART_SIZE_MB = "16"
MENDER_DATA_PART_SIZE_MB = "512"
MENDER_STORAGE_DEVICE = "/dev/mmcblk1"
MENDER_DTB_NAME_FORCE = "am335x-boneblack.dtb"

Then bitbake core-image-base

Observations, build machine

The generated uboot.env file is all-zeros, and does not contain the default environment:

$ hexdump -C build/tmp/deploy/images/beaglebone-yocto/uboot.env 
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01000000

Here’s the deployed /etc/fw_env.config:

/dev/mmcblk1 0x800000 0x20000
/dev/mmcblk1 0x1000000 0x20000

Based on that, we can check our sdimg to confirm that there are only zeros in both those locations:

$ hexdump -C -s 8388608 build/tmp/deploy/images/beaglebone-yocto/core-image-base-beaglebone-yocto-20200205190237.sdimg | head
00800000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01800000  eb 3c 90 6d 6b 66 73 2e  66 61 74 00 02 04 04 00  |.<.mkfs.fat.....|

(only zeros between 0x80_0000 and 0x180_0000, which includes both locations of u-boot env, per /etc/fw_env.config above.)

Observations, deployed machine

Let’s check our deployed root filesystem to see if somehow the non-generated default env made it where we expect:

# hexdump -C -s 8388608 /dev/mmcblk1 | head
00800000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01800000  eb 3c 90 6d 6b 66 73 2e  66 61 74 00 02 04 04 00  |.<.mkfs.fat.....|

(No u-boot env at either 0x80_0000 or 0x100_0000, but it matches the sdimg file.)

Let’s check the deployed fw_env.config:

# cat /etc/fw_env.config 
/dev/mmcblk1 0x800000 0x20000
/dev/mmcblk1 0x1000000 0x20000

(As expected)

Let’s check to see if the userspace fw_printenv can read the nonexistent environment from MMC flash:

# fw_printenv | head -n 1
Warning: Bad CRC, using default environment
altbootcmd=run mender_altbootcmd; run bootcmd

Using | head to ensure we see the CRC warning at the top. Presumably this indicates the absence of valid u-boot env data at 0x80_0000 or 0x100_0000.

As a final check, let’s see what mender says:

# mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf  module=config
INFO[0000] Loaded configuration file: /etc/mender/mender.conf  module=config
ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: Failed mender_saveenv_canary check. There is an error in the U-Boot setup. Likely causes are: 1) Mismatch between the U-Boot boot loader environment location and the location specified in /etc/fw_env.config. 2) 'mender_setup' is not run by the U-Boot boot script: exit status 1  module=main
ERRO[0000] Must give one of -install, -commit, -rollback, -daemon, -bootstrap, -version -check-update,-send-inventory or -show-artifact arguments  module=main

(Mender also fails out with the same mender-saveenv_canary check error even if I provide a valid .mender file with the -install option.)

Workaround
If I use fw_setenv to set a dummy variable, that utility will use the compiled-in default env variables to populate the flash, and eliminate future CRC warnings:

# fw_printenv | head -n 1
Warning: Bad CRC, using default environment
altbootcmd=run mender_altbootcmd; run bootcmd

# fw_setenv foo bar
Warning: Bad CRC, using default environment

# fw_setenv foo    

# fw_printenv | head -n 1
altbootcmd=run mender_altbootcmd; run bootcmd

From here on out mender -install foo.mender works great, mender -commit works great, the fallback behavior works great. But only once I awkwardly use fw_setenv to load a valid set of u-boot env vars into the MMC flash.

We can even see that our flash has been populated with the env vars (compare with the hexdump of /dev/mmcblk1 above).

# hexdump -C -s 8388608 /dev/mmcblk1 | head
00800000  c5 3e 27 5d 02 61 6c 74  62 6f 6f 74 63 6d 64 3d  |.>'].altbootcmd=|
00800010  72 75 6e 20 6d 65 6e 64  65 72 5f 61 6c 74 62 6f  |run mender_altbo|
00800020  6f 74 63 6d 64 3b 20 72  75 6e 20 62 6f 6f 74 63  |otcmd; run bootc|
00800030  6d 64 00 62 6f 6f 74 6c  69 6d 69 74 3d 31 00 62  |md.bootlimit=1.b|
00800040  6f 6f 74 63 6f 75 6e 74  3d 30 00 75 70 67 72 61  |ootcount=0.upgra|
00800050  64 65 5f 61 76 61 69 6c  61 62 6c 65 3d 30 00 6d  |de_available=0.m|
00800060  65 6e 64 65 72 5f 62 6f  6f 74 5f 70 61 72 74 3d  |ender_boot_part=|
00800070  32 00 6d 65 6e 64 65 72  5f 62 6f 6f 74 5f 70 61  |2.mender_boot_pa|
00800080  72 74 5f 68 65 78 3d 32  00 6d 65 6e 64 65 72 5f  |rt_hex=2.mender_|
00800090  75 62 6f 6f 74 5f 62 6f  6f 74 3d 6d 6d 63 20 31  |uboot_boot=mmc 1|

(The flash is now obviously populated with the “binary wrapped” uboot env variables at the expected 0x80_0000 location.)

We certainly could do this awkward step as part of our device provisioning, but it seems suboptimal.

Overall, am I doing something wrong here? Or expecting something wrong? I’ll post a potential solution as a follow-up, but I am hoping for a quick sanity/expectations check before I dive in too deep :slight_smile: Thanks!

Possible solution
This proposed patch looked promising. It uses the get_default_envs.sh and mkenvimage tools (from u-boot) to generate the default u-boot envs during the build process. I manually edited meta-mender/meta-mender-core/recipes-bsp/u-boot/u-boot-mender.inc with this diff:

diff --git a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender.inc b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender.inc
index 5ce163e..4ed4cdc 100644
--- a/meta-mender-core/recipes-bsp/u-boot/u-boot-mender.inc
+++ b/meta-mender-core/recipes-bsp/u-boot/u-boot-mender.inc
@@ -49,7 +49,9 @@ do_deploy_append_mender-uboot() {
     # Note: We cannot use a sparse file here, even though it's filled with
     # zeros, because the build process uses "du" to determine the size of this
     # file.
-    dd if=/dev/zero of=${WORKDIR}/uboot.env bs=${MENDER_BOOTENV_TOTAL_ALIGNED_SIZE} count=1
+    #dd if=/dev/zero of=${WORKDIR}/uboot.env bs=${MENDER_BOOTENV_TOTAL_ALIGNED_SIZE} count=1
+    ${WORKDIR}/build/source/scripts/get_default_envs.sh ${WORKDIR}/build | \
+        ${WORKDIR}/build/tools/mkenvimage -s ${MENDER_BOOTENV_TOTAL_ALIGNED_SIZE} -r -o ${WORKDIR}/uboot.env -
     install -m 644 ${WORKDIR}/uboot.env ${DEPLOYDIR}/uboot.env
 }
 addtask deploy after do_compile before do_package

(I would eventually move this change into a .bbappend file in my own layer if it turns out to work.)

After another bitbake core-image-base we can see the default u-boot variables are now showing up in uboot.env and the .sdimg:

$ hexdump -C build/tmp/deploy/images/beaglebone-yocto/uboot.env | head -n 1
00000000  01 80 32 e2 01 61 6c 74  62 6f 6f 74 63 6d 64 3d  |..2..altbootcmd=|
$ hexdump -C -s 8388608 build/tmp/deploy/images/beaglebone-yocto/core-image-base-beaglebone-yocto-20200205204014.sdimg | head -n 1
00800000  01 80 32 e2 01 61 6c 74  62 6f 6f 74 63 6d 64 3d  |..2..altbootcmd=|

Everything looks ok in the uboot.env and sdimg files. Let’s flash them to the device and boot the deployed image.

In the deployed image, let’s check the presence of the u-boot env vars in our flash:

# hexdump -C -s 8388608 /dev/mmcblk1 | head
00800000  01 80 32 e2 01 61 6c 74  62 6f 6f 74 63 6d 64 3d  |..2..altbootcmd=|
00800010  72 75 6e 20 6d 65 6e 64  65 72 5f 61 6c 74 62 6f  |run mender_altbo|
00800020  6f 74 63 6d 64 3b 20 72  75 6e 20 62 6f 6f 74 63  |otcmd; run bootc|
00800030  6d 64 00 61 72 63 68 3d  61 72 6d 00 61 72 67 73  |md.arch=arm.args|
00800040  5f 6d 6d 63 3d 72 75 6e  20 66 69 6e 64 75 75 69  |_mmc=run finduui|
00800050  64 3b 73 65 74 65 6e 76  20 62 6f 6f 74 61 72 67  |d;setenv bootarg|
00800060  73 20 63 6f 6e 73 6f 6c  65 3d 24 7b 63 6f 6e 73  |s console=${cons|
00800070  6f 6c 65 7d 20 24 7b 6f  70 74 61 72 67 73 7d 20  |ole} ${optargs} |
00800080  72 77 20 72 6f 6f 74 66  73 74 79 70 65 3d 24 7b  |rw rootfstype=${|
00800090  6d 6d 63 72 6f 6f 74 66  73 74 79 70 65 7d 00 62  |mmcrootfstype}.b|

That seems correct, and matches the bytes we saw in uboot.env / sdimg on the build machine. Next, let’s check if fw_printenv can read those bytes:

# fw_printenv | head -n 1
Warning: Bad CRC, using default environment
altbootcmd=run mender_altbootcmd; run bootcmd

Oh no! Something is still not quite right.

If we use the fw_setenv foo bar; fw_setenv foo trick, we get a slightly-different set of bytes:

# hexdump -C -s 8388608 /dev/mmcblk1 | head
00800000  5b 6b ca 3a 03 61 6c 74  62 6f 6f 74 63 6d 64 3d  |[k.:.altbootcmd=|
00800010  72 75 6e 20 6d 65 6e 64  65 72 5f 61 6c 74 62 6f  |run mender_altbo|
00800020  6f 74 63 6d 64 3b 20 72  75 6e 20 62 6f 6f 74 63  |otcmd; run bootc|
00800030  6d 64 00 62 6f 6f 74 6c  69 6d 69 74 3d 31 00 62  |md.bootlimit=1.b|
00800040  6f 6f 74 63 6f 75 6e 74  3d 30 00 75 70 67 72 61  |ootcount=0.upgra|
00800050  64 65 5f 61 76 61 69 6c  61 62 6c 65 3d 30 00 6d  |de_available=0.m|
00800060  65 6e 64 65 72 5f 62 6f  6f 74 5f 70 61 72 74 3d  |ender_boot_part=|
00800070  32 00 6d 65 6e 64 65 72  5f 62 6f 6f 74 5f 70 61  |2.mender_boot_pa|
00800080  72 74 5f 68 65 78 3d 32  00 6d 65 6e 64 65 72 5f  |rt_hex=2.mender_|
00800090  75 62 6f 6f 74 5f 62 6f  6f 74 3d 6d 6d 63 20 31  |uboot_boot=mmc 1|

Looks like the four CRC bytes at the start have changed, the fifth byte changed for 01 to 03 (this contains flags related to redundant copies?), and some of the key=value pairs have changed. For example, the original as-flashed bytes contained arch=arm as the second pair, but is missing (or at least not in the first 0x100 bytes) from the values written by fw_setenv. We can use fw_printenv arch to confirm that the pair is indeed present, but not written at the start of the region.

Any idea why my pre-flashed u-boot env vars are not being properly recognized by fw_printenv? Thanks!

Out of curiosity, why are you switching to U-Boot?

This is expected, and should have no effect. The file is only there to make sure it gets overwritten with zeros, and doesn’t contain some old environment. U-Boot comes with the default environment built-in, and during boot there is code to make sure that the environment is initialized on disk as well; if this is not happening it’s an indication that your U-Boot installation is not working correctly.

Stop the boot at the U-Boot prompt and save some value with setenv. This value should be visible from Linux with fw_printenv.

Thanks for your help! I appreciate your responses!

We are switching to U-Boot solely because it’s what we’ve been using on the prior generation of our product. If GRUB will work better and avoid this issue, I will look into that.

I do not observe U-Boot making sure that the env is initialized on disk, as I observe an all-zero block at both locations after booting. I have been relying on the auto patching of u-boot to work with Mender, but maybe this indicates that that process has broken for beaglebone?

Stop the boot at the U-Boot prompt and save some value with setenv . This value should be visible from Linux with fw_printenv .

That does work, but it’s not ideal for our situation. During production I’m trying to limit production steps to “flash sdming to internal MMC”, and would like to avoid having to reboot each device to use the uboot prompt or the linux prompt to initialize the u-boot env vars so that mender will actually work.

A few follow-up questions:

  • Is it expected that u-boot, upon detecting no valid env on disk, will write its compiled-in default env to disk? I do not observe this happening.

  • Is it expected that the mender binary only works if the env is properly stored on disk? I observe mender erroring-out if there is no env on disk.

Don’t worry, this was only for testing purposes. See below.

We tend to recommend GRUB because it uses the standardized UEFI boot method, which is the same on all boards, as opposed to U-Boot, whose source we have to change for every board. Not all boards support this kind of booting, but we know Beaglebone is among them, so we recommend the standard whenever we can. This also means that we don’t test U-Boot on Beaglebone.

Yes, it is expected, and required. Check out the mender_saveenv_canary. This variable does not preexist in the environment, and is written on the first boot, through the mender_setup call.

Yes, the mender_saveenv_canary from above needs to have been saved in the environment for Mender to work. This is exactly to protect people from situations like the one you are in, that someone ships a device by mistake that has incorrect U-Boot setup.

You should probably check that mender_setup and in particular the mender_saveenv_canary code is executed as expected.

Perhaps we should remove “For U-Boot-based boards, the integration checklist has been verified.” from the Beaglebone Black integration page?

From a freshly-flashed Beaglebone (using .sdimg file) I observe

U-Boot SPL 2019.01 (Feb 05 2020 - 16:04:48 +0000)
Trying to boot from MMC2
Loading Environment from FAT... Card did not respond to voltage select!
Loading Environment from MMC... *** Warning - bad CRC, using default environment



U-Boot 2019.01 (Feb 05 2020 - 16:04:48 +0000)

CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from FAT... Card did not respond to voltage select!
Loading Environment from MMC... *** Warning - bad CRC, using default environment

<ethaddr> not set. Validating first E-fuse MAC
Net:   cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds

So we see that it cannot load the environment from MMC due to bad CRC, and that it uses the default env instead. (as expected)

So I experimented with manually running mender_setup:

=> printenv bootcmd
bootcmd=run mender_setup; setenv bootargs root=${mender_kernel_root} ${bootargs}; if test "${fdt_addr_r}" != ""; then load ${mender_uboot_root} ${fdt_addr_r} /boot/${mender_dtb_name}; fi; load ${mender_uboot_root} ${kernel_addr_r} /boot/${mender_kernel_name}; ${mender_boot_kernel_type} ${kernel_addr_r} - ${fdt_addr_r}; run mender_try_to_recover

=> printenv mender_setup
mender_setup=if test "${mender_saveenv_canary}" != "1"; then setenv mender_saveenv_canary 1; saveenv; fi; if test "${mender_pre_setup_commands}" != ""; then run mender_pre_setup_commands; fi; if test "${mender_systemd_machine_id}" != ""; then setenv bootargs systemd.machine_id=${mender_systemd_machine_id} ${bootargs}; fi; setenv mender_kernel_root /dev/mmcblk1p${mender_boot_part}; if test ${mender_boot_part} = 2; then setenv mender_boot_part_name /dev/mmcblk1p2; else setenv mender_boot_part_name /dev/mmcblk1p3; fi; setenv mender_kernel_root_name ${mender_boot_part_name}; setenv mender_uboot_root mmc 1:${mender_boot_part_hex}; setenv mender_uboot_root_name ${mender_boot_part_name}; setenv expand_bootargs "setenv bootargs \"${bootargs}\""; run expand_bootargs; setenv expand_bootargs; if test "${mender_post_setup_commands}" != ""; then run mender_post_setup_commands; fi

=> print mender_saveenv_canary
## Error: "mender_saveenv_canary" not defined

=> run mender_setup
Saving Environment to FAT... Card did not respond to voltage select!
Failed (1)

=> print mender_saveenv_canary
mender_saveenv_canary=1

=> reset
resetting ...

U-Boot SPL 2019.01 (Feb 05 2020 - 16:04:48 +0000)
Trying to boot from MMC2
Loading Environment from FAT... Card did not respond to voltage select!
Loading Environment from MMC... *** Warning - bad CRC, using default environment



U-Boot 2019.01 (Feb 05 2020 - 16:04:48 +0000)

CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from FAT... Card did not respond to voltage select!
Loading Environment from MMC... *** Warning - bad CRC, using default environment

<ethaddr> not set. Validating first E-fuse MAC
Net:   cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
=> printenv mender_saveenv_canary
## Error: "mender_saveenv_canary" not defined

So it appears that the saveenv inside the mender_setup command did not actually save the env to MMC flash?

Perhaps I should configure u-boot to disable env FAT support, as it should only be using MMC for my environment? Perhaps that is what is causing the problem with saveenv?

Or maybe I should just switch to GRUB as that seems to be the favored approach going forward.

Thanks for your help!

Well spotted, I have removed this, as it was outdated.

This is very strange, mender_setup exists in your U-Boot environment, but mender_saveenv_canary does not? They are part of the same patch. Can you run:

bitbake -c devshell u-boot

And then verify that include/env_mender.h contains the canary, and that no other file contains it?

Yes, or that, which is what I would do, as it should not require any configuration at all. :slight_smile:

Using the devshell trick, I looked at include/env_mender.h and see three lines containing canary:

    "mender_check_saveenv_canary=1\0"                                   \
                                                                        \
    "mender_setup="                                                     \
    "if test \"${mender_saveenv_canary}\" != \"1\"; then "              \
    "setenv mender_saveenv_canary 1; "                                  \
    "saveenv; "                                                         \
    "fi; "

Are we possibly confusing ourselves between mender_check_saveenv_canary and mender_saveenv_canary? It looks like mender_setup will set mender_saveenv_canary = 1 if it’s not set, then calls saveenv. I’m not sure what mender_check_saveenv_canary is used for.

What I was trying to show in my previous post is that it looks like u-boot is not able to save env to MMC flash, presumably because it tries FAT first and errors-out?

Grub does not seem to work by default when running from BBB internal MMC flash. For u-boot, I had to add these lines to my local.conf:

MENDER_UBOOT_STORAGE_INTERFACE = "mmc"
MENDER_UBOOT_STORAGE_DEVICE = "1"

So that u-boot was able to boot from the right device (IIRC it defaulted to mmc and 0, which is the external SD card slot on the Beaglebone, which works fine if you’re running from the SD card slot, but I wanted to run from the internal MMC flash). Maybe there’s a similar tweak I need to make for grub… MENDER_GRUB_STORAGE_DEVICE looks promising!

Thanks for your help!

I used a u-boot_%.bbappend file to append this .cfg file (which bitbake somehow magically knows that it is a KCONF fragment for the u-boot .config file):

CONFIG_ENV_IS_IN_FAT=n

And now I observe u-boot properly able to save the env to MMC (and the redundant copy further down in the MMC). When it boots to linux, fw_printenv | head shows no CRC error.

U-Boot SPL 2019.01 (Feb 12 2020 - 17:04:22 +0000)
Trying to boot from MMC2
Loading Environment from MMC... *** Warning - bad CRC, using default environment



U-Boot 2019.01 (Feb 12 2020 - 17:04:22 +0000)

CPU  : AM335X-GP rev 2.1
I2C:   ready
DRAM:  512 MiB
No match for driver 'omap_hsmmc'
No match for driver 'omap_hsmmc'
Some drivers were not found
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
Loading Environment from MMC... *** Warning - bad CRC, using default environment

<ethaddr> not set. Validating first E-fuse MAC
Net:   cpsw, usb_ether
Press SPACE to abort autoboot in 2 seconds
Saving Environment to MMC... Writing to redundant MMC(1)... OK
58748 bytes read in 14 ms (4 MiB/s)
6659584 bytes read in 431 ms (14.7 MiB/s)
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 8ffee000, end 8ffff57b ... OK

Starting kernel ...
(messages trimmed here)
~# fw_printenv | head
altbootcmd=run mender_altbootcmd; run bootcmd
arch=arm
args_mmc=run finduuid;setenv bootargs console=${console} ${optargs} rw rootfstype=${mmcrootfstype}
baudrate=115200
board=am335x

So that seems to be working correctly. Thanks again for your help!

1 Like

Glad you got it working, and yes, I think you’re right that I was confusing the two canary variables.

Grub does not seem to work by default when running from BBB internal MMC flash.

This is interesting, and something I will try to look into. Thanks for your detailed findings!

Actually I think I was running into this innocuous error message. It boots fine with grub after a few seconds if I just ignore that error.

Ah, ok, that’s good to hear!