@kacf
I have followed the steps as told above and output as below:
$ fw_printenv # o/p before install
bootcount=1
mender_boot_part=2
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=2
$ sudo mender install alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/sda2
INFO[0000] Start updating from local image file: [alto_p3_upgV2.6_20210529_Testserver_v2-x86_64-mender.mender]
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Opening device "/dev/sda3" for writing
INFO[0000] Native sector size of block device /dev/sda3 is 512 bytes. Mender will write in chunks of 1048576 bytes
.............................................................. - 100 %
INFO[1483] All bytes were successfully written to the new partition
INFO[1483] The optimized block-device writer wrote a total of 49729 frames, where 23694 frames did need to be rewritten (i.e., skipped)
INFO[1485] Wrote 52143587328/52143587328 bytes to the inactive partition
INFO[1485] Enabling partition with new image installed to be a boot candidate: 3
$ fw_printenv # After reboot, o/p as below
bootcount=1
mender_boot_part=3
upgrade_available=0
mender_uboot_separator=
mender_boot_part_hex=3
$ mender commit
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/sda3 ERRO[0000] Could not commit Artifact: There is nothing to commit WARN[0000] There is nothing to commit
mount output, both before and after the reboot as below in attached file. mount_log.yml (16.8 KB)
$ sudo mender commit
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
INFO[0000] Mender running on partition: /dev/sda2
ERRO[0000] Could not commit Artifact: There is nothing to commit
WARN[0000] There is nothing to commit
Second time attempt logs with new flash image on target device for reference as mount-log-attempt-2.yml (12.2 KB)
I have to admit that I’m starting to run out of ideas. Something is modifying the boot loader environment behind your back, it’s definitely not behaving as it should. Are you using any state scripts, or modifying the sda1 partition in any way?
How are you generating the initial UEFIIMG? I know you are using mender-convert from the earlier discussion but where are you getting the input image? Is it an image that already has Mender installed?
@drewmoseley
I am using the latest mender_convert v2.4 here. I am using the input image as custom ubuntu18.04 that I am using last ~1 year without any issue.
Here Mender is not installed in an image. mender is installing through mender_convert utility only.
Could anyone please suggest that why the statement “fw_setenv failure” (OS x86 Ubuntu18.04 as below error logs) is sometimes is failing & sometimes is working in the case of the self-hosted AWS S3 bucket storage instead of minio with a full OS filesystem update & application update (using directory-artifact-gen & single-file-artifact-gen scripts) as below screenshots in both versions v2.6 & v2.7?
2021-06-08 15:40:29 +0000 UTC info: Committing update
2021-06-08 15:40:30 +0000 UTC error: fw_setenv returned failure: exit status 2
2021-06-08 15:40:30 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 2
2021-06-08 15:40:30 +0000 UTC error: transient error: update commit failed: exit status 2
2021-06-08 15:40:30 +0000 UTC info: State transition: update-commit [ArtifactCommit_Enter] → rollback [ArtifactRollback]
2021-06-08 15:40:30 +0000 UTC info: Performing rollback
2021-06-08 15:40:31 +0000 UTC info: Rolling back to the inactive partition (3).
2021-06-08 15:40:32 +0000 UTC error: fw_setenv returned failure: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Failed to probe the U-Boot environment for which separator to use. Got error: exit status 1
2021-06-08 15:40:32 +0000 UTC error: Rollback failed: exit status 1
2021-06-08 15:40:32 +0000 UTC error: fatal error: exit status 1
@Rohita83: I’m working on making it easier to get information about failures like this by logging output from child processes better. See this pull request. Perhaps you can retry with a development build once that has been merged.
The exit status 1 by itself is not giving a lot of information.
@kacf Thanks for providing mender client binary for the test.
Here I have created new mender artifacts (let’s say V1 & V2) & OS image (through mender_convert utility) by using your provided mender client binary (attached mender_convert logs for reference convert.log.uMuR.yml (26.3 KB)
).
I have flashed a new created OS image (having v2 artifact) on the target X86 device & triggered full OS OTA four times but it has failed on the 4th OTA as below screenshot. Attached failed OTA log for your reference. ErrorLog11June.txt.yml (3.7 KB)
Note: I have also tried on another different device with a new flash the same image on this device but here all three OTA OS updates have failed as below screenshot.
This is all consistent with the issue you have been seeing. The _INCONSISTENT tag is used when something happens that the Mender client cannot understand. Usually, it is when a rollback fails for some reason outside the scope of Mender control, and everything you have described so far seems to indicate something like that is happening.
At this point, I would suggest you start with a simpler input image to help rule out some bad interaction with the software installed in your golden master.
@Rohita83: Sorry for the delay. After having analyzed, I’m still not completely sure what’s going on, but I have a theory. Can you add the following script as state scripts:
#!/bin/sh
set -x
ENV_DIR=/boot/efi/EFI/BOOT/
which grub-editenv
which grub2-editenv
grub-editenv
grub2-editenv
for i in $ENV_DIR/mender_grubenv1/env ENV_DIR/mender_grubenv1/lock $ENV_DIR/mender_grubenv2/env ENV_DIR/mender_grubenv2/lock; do
grub-editenv $i list
grub2-editenv $i list
done
exit 0
Add the same script as state scripts for all of these states:
ArtifactInstall_Enter_00
ArtifactInstall_Leave_00
ArtifactReboot_Enter_00
ArtifactReboot_Leave_00
ArtifactCommit_Enter_00
ArtifactCommit_Leave_00
ArtifactRollback_Enter_00
ArtifactRollback_Leave_00
ArtifactRollbackReboot_Enter_00
ArtifactRollbackReboot_Leave_00
ArtifactFailure_Enter_00
ArtifactFailure_Leave_00
You might also want to double check that ENV_DIR is correct, and that it exists on the device. After you’ve run another deployment with this, please post the output.
No, they are added using mender-artifact. This section describes how to add state scripts to artifacts using mender-convert. You will then need to include the custom_config file with -c.
they also both have failed.
