Update not working - reporting incorrect artifact name and reboot required

General Discussions
, ,
1

I have a very strange issue. I have a fleet of Raspberry Pi 3B+'s running a Yocto build. I’ve recently tried making a new build based on Zeus, and I’ve got that build working, but the updates are very unreliable and I’m seeing some strange behaviour.

The update will technically success according to the GUI, however some of the time it isn’t actually doing the update correctly. It’s reporting that the artifact version has updated, but when I SSH into the device I can see that it’s still running the old version.

When I run ‘mender -show-artifact’ it gives the name of the new artifact. However, nothing else has changed. The kernel, app, files, etc are all untouched.

Most of the time if I force the device to reboot, it will then boot into the correct, new, version.

I have no idea really where to start investigating, so any pointers would be very much appreciated. Please let me know if you need any more information and I’ll see what I can provide

The original build was Thud, with Mender client 2.1.0 running. I seemed to be able to do the update from thud to zeus without any issues, it’s just an issue when I then try to do a further update from zeus to zeus.

Current build:

  • kernel: “Linux version 4.19.93 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 SMP Mon Aug 31 22:29:52 UTC 2020”
  • mender_client_version: “2.3.0”
  • os: “Poky (Yocto Project Reference Distro) 3.0.4 (zeus)”

Thanks in advance,
James

2

Would be interesting to know:

  • When you update thud -> zeus do you also update the bootloader?
  • What is the U-Boot version on the device experiencing problems?
  • What is the result of cat /etc/fw_env.config ?
3

Hi Mirza,

  1. When I update from Thud to Zeus, I don’t update the bootloader. It stays on 2018.07

  2. The U-Boot version on the device experiencing problems is U-Boot 2020.01

  3. Output of /etc/fw_env.config is below. I get the same output on the old Thud and the new Zeus builds
    /dev/mmcblk0 0x400000 0x4000
    /dev/mmcblk0 0x800000 0x4000

Some more information. I have two Zeus builds, one that includes “rpi-update-firmware” (B) and one that doesn’t (A). So B is updating U-Boot to 2020.01.

I’m not able to update to Zeus directly from Thud and include rpi-update-firmware in the same update because of incompatibilities with Ethernet. So I have the following update process

  1. Start with Thud
  2. Install update A, which updates to Zeus
  3. Install update B, which is also Zeus but now includes rpi-update-firmware, and U-Boot 2020.01

It is only once I do update B that I begin to have problems with the incorrect artifact being reported and it booting into the incorrect partitions.

Today I did repo sync so that I had the latest version of everything and I deleted the tmp folder and re-built the images, and the issue is still there.

Let me know what other information I can provide

Hope that helps :slight_smile:

4

Thanks for the clarification. That helped a lot and I think I know what is the problem.

I believe that you need to revert this change, for your zeus builds. This change changes where the U-Boot environment is stored and as soon as you update the bootloader on your device (with rpi-update-firmware included), the user-space configuration does not align anymore.

You can do this with a bbappend contaning somethine like this:

FILESEXTRAPATHS_prepend_rpi := "${THISDIR}/files:"

# Remove the patch applied in meta-mender/meta-mender-raspberrypi
SRC_URI_remove = "file://0001-configs-rpi-enable-mender-requirements.patch"

# Apply custom rpi integration patch to keep the 4MB alignment.
SRC_URI_append = " file://0001-configs-rpi-enable-mender-requirements-custom.patch"

And the patch file:

From 371400a8772d5f6ae4b819ef91cd36b883a8d313 Mon Sep 17 00:00:00 2001
From: Mirza Krak <mirza.krak@gmail.com>
Date: Tue, 26 Sep 2017 06:23:52 -0400
Subject: [PATCH 1/1] configs: rpi: enable mender requirements

Which are CONFIG_BOOTCOUNT_ENV and CONFIG_BOOTCOUNT_LIMIT.

Mender also requires that the environment is on MMC
(CONFIG_ENV_IS_IN_MMC)

Signed-off-by: Mirza Krak <mirza.krak@gmail.com>
Signed-off-by: Drew Moseley <drew.moseley@northern.tech>
---
 configs/rpi_0_w_defconfig      | 3 +--
 configs/rpi_2_defconfig        | 3 +--
 configs/rpi_3_32b_defconfig    | 3 +--
 configs/rpi_3_b_plus_defconfig | 3 +--
 configs/rpi_3_defconfig        | 3 +--
 configs/rpi_4_32b_defconfig    | 3 +--
 configs/rpi_4_defconfig        | 3 +--
 configs/rpi_arm64_defconfig    | 3 +--
 configs/rpi_defconfig          | 3 +--
 env/Kconfig                    | 1 -
 include/configs/rpi.h          | 3 +++
 11 files changed, 12 insertions(+), 19 deletions(-)

diff --git a/configs/rpi_0_w_defconfig b/configs/rpi_0_w_defconfig
index 75c6c9c447..493a99a26d 100644
--- a/configs/rpi_0_w_defconfig
+++ b/configs/rpi_0_w_defconfig
@@ -19,8 +19,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2835-rpi-zero-w"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -44,3 +42,7 @@ CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
 CONFIG_EFI_LOADER=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_2_defconfig b/configs/rpi_2_defconfig
index 4e8204ef88..4f4029e006 100644
--- a/configs/rpi_2_defconfig
+++ b/configs/rpi_2_defconfig
@@ -19,8 +19,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2836-rpi-2-b"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -43,3 +41,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_3_32b_defconfig b/configs/rpi_3_32b_defconfig
index d50953287c..76d3abd54d 100644
--- a/configs/rpi_3_32b_defconfig
+++ b/configs/rpi_3_32b_defconfig
@@ -20,8 +20,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2837-rpi-3-b"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -46,3 +44,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_3_b_plus_defconfig b/configs/rpi_3_b_plus_defconfig
index e76821450f..a06ccf2d19 100644
--- a/configs/rpi_3_b_plus_defconfig
+++ b/configs/rpi_3_b_plus_defconfig
@@ -20,8 +20,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2837-rpi-3-b-plus"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -46,3 +44,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_3_defconfig b/configs/rpi_3_defconfig
index c0c0955131..fd617ae0c2 100644
--- a/configs/rpi_3_defconfig
+++ b/configs/rpi_3_defconfig
@@ -20,8 +20,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2837-rpi-3-b"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -46,3 +44,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_4_32b_defconfig b/configs/rpi_4_32b_defconfig
index 00f80f71ad..e0e7795952 100644
--- a/configs/rpi_4_32b_defconfig
+++ b/configs/rpi_4_32b_defconfig
@@ -16,8 +16,6 @@ CONFIG_CMD_GPIO=y
 CONFIG_CMD_MMC=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_BOARD=y
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -33,3 +31,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_4_defconfig b/configs/rpi_4_defconfig
index 8cf1bb81ff..a18eba2884 100644
--- a/configs/rpi_4_defconfig
+++ b/configs/rpi_4_defconfig
@@ -16,8 +16,6 @@ CONFIG_CMD_GPIO=y
 CONFIG_CMD_MMC=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_BOARD=y
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -33,3 +31,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_arm64_defconfig b/configs/rpi_arm64_defconfig
index 10fbe0db92..ce720769f5 100644
--- a/configs/rpi_arm64_defconfig
+++ b/configs/rpi_arm64_defconfig
@@ -17,8 +17,6 @@ CONFIG_CMD_MMC=y
 CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_BOARD=y
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
 CONFIG_DM_MMC=y
@@ -42,3 +40,7 @@ CONFIG_SYS_WHITE_ON_BLACK=y
 CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/configs/rpi_defconfig b/configs/rpi_defconfig
index 2f4c7da6dc..de498d0652 100644
--- a/configs/rpi_defconfig
+++ b/configs/rpi_defconfig
@@ -19,8 +19,6 @@ CONFIG_CMD_USB=y
 CONFIG_CMD_FS_UUID=y
 CONFIG_OF_EMBED=y
 CONFIG_DEFAULT_DEVICE_TREE="bcm2835-rpi-b"
-CONFIG_ENV_FAT_INTERFACE="mmc"
-CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
 CONFIG_SYS_RELOC_GD_ENV_ADDR=y
 CONFIG_ENV_VARS_UBOOT_RUNTIME_CONFIG=y
 CONFIG_DM_KEYBOARD=y
@@ -44,3 +42,7 @@ CONFIG_CONSOLE_SCROLL_LINES=10
 CONFIG_PHYS_TO_BUS=y
 CONFIG_OF_LIBFDT_OVERLAY=y
 CONFIG_EFI_LOADER=y
+CONFIG_ENV_IS_IN_MMC=y
+CONFIG_ENV_OFFSET=0x400000
+CONFIG_ENV_OFFSET_REDUND=0x800000
+CONFIG_SYS_REDUNDAND_ENVIRONMENT=y
diff --git a/env/Kconfig b/env/Kconfig
index ed12609f6a..e83c62f6f6 100644
--- a/env/Kconfig
+++ b/env/Kconfig
@@ -49,7 +49,6 @@ config ENV_IS_IN_EEPROM
 config ENV_IS_IN_FAT
 	bool "Environment is in a FAT filesystem"
 	depends on !CHAIN_OF_TRUST
-	default y if ARCH_BCM283X
 	default y if ARCH_SUNXI && MMC
 	default y if MMC_OMAP_HS && TI_COMMON_CMD_OPTIONS
 	select FS_FAT
diff --git a/include/configs/rpi.h b/include/configs/rpi.h
index 83e258a6b9..cf05fbc0d0 100644
--- a/include/configs/rpi.h
+++ b/include/configs/rpi.h
@@ -80,6 +80,9 @@
 /* Environment */
 #define CONFIG_SYS_LOAD_ADDR		0x1000000
 
+#define CONFIG_BOOTCOUNT_ENV
+#define CONFIG_BOOTCOUNT_LIMIT
+
 /* Shell */
 
 /* ATAGs support for bootm/bootz */
-- 
2.17.1
5

Thanks for this information @mirzak. I’ve just tried to apply the patch, however I got the following error:

ERROR: u-boot-fw-utils-mender-auto-provided-1.0-r0 do_configure: U-Boot configuration rpi_3_32b_config has setting: 
CONFIG_ENV_OFFSET=0x400000 
CONFIG_ENV_OFFSET_REDUND=0x800000 
but Mender expects: 
CONFIG_ENV_OFFSET=0x800000 
Please fix U-Boot's configuration file.

I then found the following values and set them, and it seems to have fixed the issue and the updates now seem to work (more testing to happen tomorrow though). I wanted to check though whether setting these values makes sense or if it might cause other issues?

MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1 = "0x400000" 
MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_2 = "0x800000"
6

Ah sorry, forgot. You also need to set this in e.g local.conf,

MENDER_PARTITION_ALIGNMENT = "4194304"

Then you should be able to drop the MENDER_UBOOT_ENV_STORAGE_DEVICE_OFFSET_1/2 changes

With the patch and above change, you will be in a similar state as the configuration in earlier branches of meta-mender.

7

@mirzak thanks very much! This seems to be working nicely now

Slightly unrelated - any idea when the Raspberry Pi 3 Model B/B+ and Mender will support Dunfell? I think I’ve seen some people on the forums mention that they’re using it already but I was just wondering if there will be official support?

8

Glad it worked out.

Dunfell support is already in place, we did an official release a while back

9

Ah ok, that’s good news. I just wasn’t sure because this page (Raspberry Pi 3 Model B/B+) doesn’t mention Dunfell or have a Dunfell tag. I’ll try it out soon though

10

Ah, think we just missed updating it. Thanks for pointing that out, I will update it right now.