Getting "error: fw_setenv returned failure: exit status 2" after "Committing update" OTA update

@kacf
I have followed the steps as told above and output as below:

$ fw_printenv # o/p before install
bootcount=1
mender_boot_part=2
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=2

$ sudo mender install alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/sda2       
INFO[0000] Start updating from local image file: [alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender] 
INFO[0000] No public key was provided for authenticating the artifact 
INFO[0000] Opening device "/dev/sda3" for writing       
INFO[0000] Native sector size of block device /dev/sda3 is 512 bytes. Mender will write in chunks of 1048576 bytes 
.............................................................. - 100 %
INFO[1483] All bytes were successfully written to the new partition 
INFO[1483] The optimized block-device writer wrote a total of 49729 frames, where 23694 frames did need to be rewritten (i.e., skipped) 
INFO[1485] Wrote 52143587328/52143587328 bytes to the inactive partition 
INFO[1485] Enabling partition with new image installed to be a boot candidate: 3 

$ fw_printenv # After reboot,  o/p as below
bootcount=1
mender_boot_part=3
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=3

$ mender commit
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/sda3
ERRO[0000] Could not commit Artifact: There is nothing to commit
WARN[0000] There is nothing to commit

Refer post: Fixed wrong error produced by rootfs-image commit by hacpa · Pull Request #607 · mendersoftware/mender · GitHub

In this scenario, what does “fw_printenv” show after the install but before the reboot?

Drew

What @drewmoseley said!

And also, what does mount say, both before and after the reboot?

@drewmoseley
“fw_printenv” o/p after the install but before the reboot as below

$ fw_printenv
bootcount=0
mender_boot_part=3
upgrade_available=1
mender_uboot_separator=
mender_boot_part_hex=3

@kacf @drewmoseley

mount output, both before and after the reboot as below in attached file.
mount_log.yml (16.8 KB)

$ sudo mender commit

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/sda2       
ERRO[0000] Could not commit Artifact: There is nothing to commit 
WARN[0000] There is nothing to commit

Second time attempt logs with new flash image on target device for reference as
mount-log-attempt-2.yml (12.2 KB)

Could you please suggest me to crack this issue.

I have to admit that I’m starting to run out of ideas. Something is modifying the boot loader environment behind your back, it’s definitely not behaving as it should. Are you using any state scripts, or modifying the sda1 partition in any way?

@kacf
Thanks for the quick response.

I am not using state scripts as below
image

I have used the directory update module i.e. directory-artifact-gen as below to modify the Home partition to create TEST folder.

ARTIFACT_NAME=“HOME_TEST_v1.0”
DEVICE_TYPE=“x86_64”
OUTPUT_PATH=“HOME_TEST_v1.0.mender”
DEST_DIR="/home/alto/TEST/"
FILE_TREE=“Test_local”

./directory-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${FILE_TREE}

Note: I have attached grub.cfg file here for reference to get any clue.
grub.cfg.yml (7.8 KB)

How are you generating the initial UEFIIMG? I know you are using mender-convert from the earlier discussion but where are you getting the input image? Is it an image that already has Mender installed?

Drew

@drewmoseley
I am using the latest mender_convert v2.4 here. I am using the input image as custom ubuntu18.04 that I am using last ~1 year without any issue.
Here Mender is not installed in an image. mender is installing through mender_convert utility only.

@drewmoseley @kacf @mirzak

Could anyone please suggest that why the statement “fw_setenv failure” (OS x86 Ubuntu18.04 as below error logs) is sometimes is failing & sometimes is working in the case of the self-hosted AWS S3 bucket storage instead of minio with a full OS filesystem update & application update (using directory-artifact-gen & single-file-artifact-gen scripts) as below screenshots in both versions v2.6 & v2.7?

2021-06-08 15:40:29 +0000 UTC info: Committing update
2021-06-08 15:40:30 +0000 UTC error: fw_setenv returned failure: exit status 2
2021-06-08 15:40:30 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 2
2021-06-08 15:40:30 +0000 UTC error: transient error: update commit failed: exit status 2
2021-06-08 15:40:30 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] → rollback [ArtifactRollback]
2021-06-08 15:40:30 +0000 UTC info: Performing rollback
2021-06-08 15:40:31 +0000 UTC info: Rolling back to the inactive partition (3).
2021-06-08 15:40:32 +0000 UTC error: fw_setenv returned failure: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Rollback failed: exit status 1
2021-06-08 15:40:32 +0000 UTC error: fatal error: exit status 1


@Rohita83: I’m working on making it easier to get information about failures like this by logging output from child processes better. See this pull request. Perhaps you can retry with a development build once that has been merged.

The exit status 1 by itself is not giving a lot of information.

@kacf
Thanks for this valuable update.
Please let me know, whenever this package available to test above mention issue. i will wait for this.

Can you try the package at:

If you are using mender-convert to prepare an image, you can use something like this patch to temporarily use the development package:

diff --git a/mender-convert-modify b/mender-convert-modify
index cf8f215..925f517 100755
--- a/mender-convert-modify
+++ b/mender-convert-modify
@@ -103,14 +103,16 @@ log_info "Installing Mender client and related files"
 
 deb_arch=$(probe_debian_arch_name)
 
-if [ "${MENDER_CLIENT_VERSION}" = "latest" ]; then
-    deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "stable" "mender-client")
-elif [ "${MENDER_CLIENT_VERSION}" = "master" ]; then
-    deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "experimental" "mender-client")
-else
-    DEBIAN_REVISION="-1"
-    deb_name=$(deb_from_repo_pool_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "mender-client" "${MENDER_CLIENT_VERSION}${DEBIAN_REVISION}")
-fi
+# if [ "${MENDER_CLIENT_VERSION}" = "latest" ]; then
+#     deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "stable" "mender-client")
+# elif [ "${MENDER_CLIENT_VERSION}" = "master" ]; then
+#     deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "experimental" "mender-client")
+# else
+#     DEBIAN_REVISION="-1"
+#     deb_name=$(deb_from_repo_pool_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "mender-client" "${MENDER_CLIENT_VERSION}${DEBIAN_REVISION}")
+# fi
+wget https://mender.s3.amazonaws.com/kacf-tmp/mender-client_2.6.1~git20210611.fd57c39-1%2BbLOCAL_amd64.deb -P work/deb-packages/
+deb_name=mender-client_2.6.1~git20210611.fd57c39-1+bLOCAL_amd64.deb
 
 deb_extract_package "work/deb-packages/${deb_name}" "work/rootfs/"
 

(I didn’t test that last part, I hope I didn’t make a mistake…)

@kacf Thanks for providing mender client binary for the test.

Here I have created new mender artifacts (let’s say V1 & V2) & OS image (through mender_convert utility) by using your provided mender client binary (attached mender_convert logs for reference
convert.log.uMuR.yml (26.3 KB)
).

I have flashed a new created OS image (having v2 artifact) on the target X86 device & triggered full OS OTA four times but it has failed on the 4th OTA as below screenshot. Attached failed OTA log for your reference.
ErrorLog11June.txt.yml (3.7 KB)

For any further details for debugging, Please let me know.

@kacf @drewmoseley

Continue I have triggered 2 more full OS OTA deployment (as below screenshot) but still :frowning: they also both have failed.

Note: observed here “_INCONSISTENT” appended into artifact name. [testImage_20210611_Testserver_v1_INCONSISTENT]

Note: I have also tried on another different device with a new flash the same image on this device but here all three OTA OS updates have failed as below screenshot.

This is all consistent with the issue you have been seeing. The _INCONSISTENT tag is used when something happens that the Mender client cannot understand. Usually, it is when a rollback fails for some reason outside the scope of Mender control, and everything you have described so far seems to indicate something like that is happening.

At this point, I would suggest you start with a simpler input image to help rule out some bad interaction with the software installed in your golden master.

Drew

@kacf
have you received any additional information from new log statements, generated by your shared mender client binary?

@Rohita83: Sorry for the delay. After having analyzed, I’m still not completely sure what’s going on, but I have a theory. Can you add the following script as state scripts:

#!/bin/sh
set -x
ENV_DIR=/boot/efi/EFI/BOOT/
which grub-editenv
which grub2-editenv
grub-editenv
grub2-editenv
for i in $ENV_DIR/mender_grubenv1/env ENV_DIR/mender_grubenv1/lock $ENV_DIR/mender_grubenv2/env ENV_DIR/mender_grubenv2/lock; do
    grub-editenv $i list
    grub2-editenv $i list
done
exit 0

Add the same script as state scripts for all of these states:

  • ArtifactInstall_Enter_00
  • ArtifactInstall_Leave_00
  • ArtifactReboot_Enter_00
  • ArtifactReboot_Leave_00
  • ArtifactCommit_Enter_00
  • ArtifactCommit_Leave_00
  • ArtifactRollback_Enter_00
  • ArtifactRollback_Leave_00
  • ArtifactRollbackReboot_Enter_00
  • ArtifactRollbackReboot_Leave_00
  • ArtifactFailure_Enter_00
  • ArtifactFailure_Leave_00

You might also want to double check that ENV_DIR is correct, and that it exists on the device. After you’ve run another deployment with this, please post the output.

these scripts should add at path “/etc/mender/scripts/” before mender_convert execution. right? any permission should apply?

No, they are added using mender-artifact. This section describes how to add state scripts to artifacts using mender-convert. You will then need to include the custom_config file with -c.