Getting "error: fw_setenv returned failure: exit status 2" after "Committing update" OTA update

Rohita83 · June 3, 2021, 1:50pm

@kacf
I have followed the steps as told above and output as below:

$ fw_printenv # o/p before install
bootcount=1
mender_boot_part=2
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=2

$ sudo mender install alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender 
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/sda2       
INFO[0000] Start updating from local image file: [alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender] 
INFO[0000] No public key was provided for authenticating the artifact 
INFO[0000] Opening device "/dev/sda3" for writing       
INFO[0000] Native sector size of block device /dev/sda3 is 512 bytes. Mender will write in chunks of 1048576 bytes 
.............................................................. - 100 %
INFO[1483] All bytes were successfully written to the new partition 
INFO[1483] The optimized block-device writer wrote a total of 49729 frames, where 23694 frames did need to be rewritten (i.e., skipped) 
INFO[1485] Wrote 52143587328/52143587328 bytes to the inactive partition 
INFO[1485] Enabling partition with new image installed to be a boot candidate: 3 

$ fw_printenv # After reboot,  o/p as below
bootcount=1
mender_boot_part=3
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=3

$ mender commit
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/sda3
ERRO[0000] Could not commit Artifact: There is nothing to commit
WARN[0000] There is nothing to commit

Refer post: Fixed wrong error produced by rootfs-image commit by hacpa · Pull Request #607 · mendersoftware/mender · GitHub

drewmoseley · June 4, 2021, 3:57pm

In this scenario, what does “fw_printenv” show after the install but before the reboot?

Drew

kacf · June 4, 2021, 4:06pm

What @drewmoseley said!

And also, what does mount say, both before and after the reboot?

Rohita83 · June 6, 2021, 1:19pm

@drewmoseley
“fw_printenv” o/p after the install but before the reboot as below

$ fw_printenv
bootcount=0
mender_boot_part=3
upgrade_available=1
mender_uboot_separator=
mender_boot_part_hex=3

Rohita83 · June 6, 2021, 1:45pm

@kacf @drewmoseley

mount output, both before and after the reboot as below in attached file.
mount_log.yml (16.8 KB)

$ sudo mender commit

INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] Mender running on partition: /dev/sda2       
ERRO[0000] Could not commit Artifact: There is nothing to commit 
WARN[0000] There is nothing to commit

Second time attempt logs with new flash image on target device for reference as
mount-log-attempt-2.yml (12.2 KB)

Could you please suggest me to crack this issue.

kacf · June 7, 2021, 7:26am

I have to admit that I’m starting to run out of ideas. Something is modifying the boot loader environment behind your back, it’s definitely not behaving as it should. Are you using any state scripts, or modifying the sda1 partition in any way?

Rohita83 · June 7, 2021, 7:46am

@kacf
Thanks for the quick response.

I am not using state scripts as below

I have used the directory update module i.e. directory-artifact-gen as below to modify the Home partition to create TEST folder.

ARTIFACT_NAME=“HOME_TEST_v1.0”
DEVICE_TYPE=“x86_64”
OUTPUT_PATH=“HOME_TEST_v1.0.mender”
DEST_DIR=“/home/alto/TEST/”
FILE_TREE=“Test_local”

./directory-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${FILE_TREE}

Note: I have attached grub.cfg file here for reference to get any clue.
grub.cfg.yml (7.8 KB)

drewmoseley · June 7, 2021, 5:55pm

How are you generating the initial UEFIIMG? I know you are using mender-convert from the earlier discussion but where are you getting the input image? Is it an image that already has Mender installed?

Drew

Rohita83 · June 8, 2021, 4:56am

@drewmoseley
I am using the latest mender_convert v2.4 here. I am using the input image as custom ubuntu18.04 that I am using last ~1 year without any issue.
Here Mender is not installed in an image. mender is installing through mender_convert utility only.

Rohita83 · June 9, 2021, 5:21am

@drewmoseley @kacf @mirzak

Could anyone please suggest that why the statement “fw_setenv failure” (OS x86 Ubuntu18.04 as below error logs) is sometimes is failing & sometimes is working in the case of the self-hosted AWS S3 bucket storage instead of minio with a full OS filesystem update & application update (using directory-artifact-gen & single-file-artifact-gen scripts) as below screenshots in both versions v2.6 & v2.7?

2021-06-08 15:40:29 +0000 UTC info: Committing update
2021-06-08 15:40:30 +0000 UTC error: fw_setenv returned failure: exit status 2
2021-06-08 15:40:30 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 2
2021-06-08 15:40:30 +0000 UTC error: transient error: update commit failed: exit status 2
2021-06-08 15:40:30 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] → rollback [ArtifactRollback]
2021-06-08 15:40:30 +0000 UTC info: Performing rollback
2021-06-08 15:40:31 +0000 UTC info: Rolling back to the inactive partition (3).
2021-06-08 15:40:32 +0000 UTC error: fw_setenv returned failure: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Rollback failed: exit status 1
2021-06-08 15:40:32 +0000 UTC error: fatal error: exit status 1

kacf · June 9, 2021, 11:53am

@Rohita83: I’m working on making it easier to get information about failures like this by logging output from child processes better. See this pull request. Perhaps you can retry with a development build once that has been merged.

The exit status 1 by itself is not giving a lot of information.

Rohita83 · June 10, 2021, 12:12pm

@kacf
Thanks for this valuable update.
Please let me know, whenever this package available to test above mention issue. i will wait for this.

kacf · June 11, 2021, 8:19am

Can you try the package at:

https://mender.s3.amazonaws.com/kacf-tmp/mender-client_2.6.1~git20210611.fd57c39-1%2BbLOCAL_amd64.deb

If you are using mender-convert to prepare an image, you can use something like this patch to temporarily use the development package:

diff --git a/mender-convert-modify b/mender-convert-modify
index cf8f215..925f517 100755
--- a/mender-convert-modify
+++ b/mender-convert-modify
@@ -103,14 +103,16 @@ log_info "Installing Mender client and related files"
 
 deb_arch=$(probe_debian_arch_name)
 
-if [ "${MENDER_CLIENT_VERSION}" = "latest" ]; then
-    deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "stable" "mender-client")
-elif [ "${MENDER_CLIENT_VERSION}" = "master" ]; then
-    deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "experimental" "mender-client")
-else
-    DEBIAN_REVISION="-1"
-    deb_name=$(deb_from_repo_pool_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "mender-client" "${MENDER_CLIENT_VERSION}${DEBIAN_REVISION}")
-fi
+# if [ "${MENDER_CLIENT_VERSION}" = "latest" ]; then
+#     deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "stable" "mender-client")
+# elif [ "${MENDER_CLIENT_VERSION}" = "master" ]; then
+#     deb_name=$(deb_from_repo_dist_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "experimental" "mender-client")
+# else
+#     DEBIAN_REVISION="-1"
+#     deb_name=$(deb_from_repo_pool_get "work/deb-packages" ${MENDER_APT_REPO_URL} ${deb_arch} "mender-client" "${MENDER_CLIENT_VERSION}${DEBIAN_REVISION}")
+# fi
+wget https://mender.s3.amazonaws.com/kacf-tmp/mender-client_2.6.1~git20210611.fd57c39-1%2BbLOCAL_amd64.deb -P work/deb-packages/
+deb_name=mender-client_2.6.1~git20210611.fd57c39-1+bLOCAL_amd64.deb
 
 deb_extract_package "work/deb-packages/${deb_name}" "work/rootfs/"

(I didn’t test that last part, I hope I didn’t make a mistake…)

Rohita83 · June 11, 2021, 1:24pm

@kacf Thanks for providing mender client binary for the test.

Here I have created new mender artifacts (let’s say V1 & V2) & OS image (through mender_convert utility) by using your provided mender client binary (attached mender_convert logs for reference
convert.log.uMuR.yml (26.3 KB)
).

I have flashed a new created OS image (having v2 artifact) on the target X86 device & triggered full OS OTA four times but it has failed on the 4th OTA as below screenshot. Attached failed OTA log for your reference.
ErrorLog11June.txt.yml (3.7 KB)

For any further details for debugging, Please let me know.

Rohita83 · June 14, 2021, 3:50am

@kacf @drewmoseley

Continue I have triggered 2 more full OS OTA deployment (as below screenshot) but still they also both have failed.

Note: observed here “_INCONSISTENT” appended into artifact name. [testImage_20210611_Testserver_v1_INCONSISTENT]

Note: I have also tried on another different device with a new flash the same image on this device but here all three OTA OS updates have failed as below screenshot.

drewmoseley · June 14, 2021, 2:00pm

This is all consistent with the issue you have been seeing. The _INCONSISTENT tag is used when something happens that the Mender client cannot understand. Usually, it is when a rollback fails for some reason outside the scope of Mender control, and everything you have described so far seems to indicate something like that is happening.

At this point, I would suggest you start with a simpler input image to help rule out some bad interaction with the software installed in your golden master.

Drew

Rohita83 · June 15, 2021, 12:26pm

@kacf
have you received any additional information from new log statements, generated by your shared mender client binary?

kacf · June 16, 2021, 11:45am

@Rohita83: Sorry for the delay. After having analyzed, I’m still not completely sure what’s going on, but I have a theory. Can you add the following script as state scripts:

#!/bin/sh
set -x
ENV_DIR=/boot/efi/EFI/BOOT/
which grub-editenv
which grub2-editenv
grub-editenv
grub2-editenv
for i in $ENV_DIR/mender_grubenv1/env ENV_DIR/mender_grubenv1/lock $ENV_DIR/mender_grubenv2/env ENV_DIR/mender_grubenv2/lock; do
    grub-editenv $i list
    grub2-editenv $i list
done
exit 0

Add the same script as state scripts for all of these states:

ArtifactInstall_Enter_00
ArtifactInstall_Leave_00
ArtifactReboot_Enter_00
ArtifactReboot_Leave_00
ArtifactCommit_Enter_00
ArtifactCommit_Leave_00
ArtifactRollback_Enter_00
ArtifactRollback_Leave_00
ArtifactRollbackReboot_Enter_00
ArtifactRollbackReboot_Leave_00
ArtifactFailure_Enter_00
ArtifactFailure_Leave_00

You might also want to double check that ENV_DIR is correct, and that it exists on the device. After you’ve run another deployment with this, please post the output.

Rohita83 · June 16, 2021, 11:50am

these scripts should add at path “/etc/mender/scripts/” before mender_convert execution. right? any permission should apply?

kacf · June 16, 2021, 12:09pm

No, they are added using mender-artifact. This section describes how to add state scripts to artifacts using mender-convert. You will then need to include the custom_config file with -c.

Topic		Replies	Views
Update fails with "The deployment was aborted from the server" General Discussions	2	981	February 11, 2022
Update is succesfull but mender announces to the server that it failed General Discussions	5	754	November 11, 2019
Mender OTA Update successful but reported by Enterprise server as a failure General Discussions	5	541	February 7, 2022
Transient error: Forced a failed update General Discussions	2	570	October 21, 2021
Unable to switch A/B partition between Ubuntu 18 and Ubuntu 20 for OTA General Discussions	1	194	June 19, 2023

Getting "error: fw_setenv returned failure: exit status 2" after "Committing update" OTA update

Related Topics