# Mender 1.7 standalone - Power failure case failing

**URL:** <https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462>\
**Category:** General Discussions\
**Tags:** sumo, nxp, standalone\
**Created:** [April 8, 2019, 12:42pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462 "2019-04-08T12:42:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 8, 2019, 12:42pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/1 "2019-04-08T12:42:57Z")

</div>

I’m testing with mender 1.7 in standalone mode. The procedure is given below:

- Using mender -rootfs “HTTPS local server path of mender artifact”
- While downloading the artifact, remove the power plug when around 80% download completed.
- Connect the power plug again and boot the system.

I expected the fail safe mechanism here but, I’m getting the below error when performing the above procedure:

> [1.859484] EXT4-fs (mmcblk0p1): mounted filesystem with ordered data mode. Opts: (null)  
> [1.868841] VFS: Mounted root (ext4 filesystem) readonly on device 179:1.  
> [1.876893] devtmpfs: mounted  
> [1.880023] Freeing unused kernel memory: 448K  
> [1.885645] EXT4-fs warning (device mmcblk0p1): dx\_probe:792: inode #3451: comm swapper/0: dx entry: limit 0 != root limit 125  
> [1.897057] EXT4-fs warning (device mmcblk0p1): dx\_probe:864: inode #3451: comm swapper/0: Corrupt directory, running e2fsck is recommended  
> **[1.909711] Starting init: /sbin/init exists but couldn’t execute it (error -4094)**  
> [1.918429] EXT4-fs warning (device mmcblk0p1): dx\_probe:792: inode #2868: comm swapper/0: dx entry: limit 0 != root limit 125  
> [1.929845] EXT4-fs warning (device mmcblk0p1): dx\_probe:864: inode #2868: comm swapper/0: Corrupt directory, running e2fsck is recommended  
> **[1.942467] Starting init: /etc/init exists but couldn’t execute it (error -4094)**  
> **[1.951434] Starting init: /bin/init exists but couldn’t execute it (error -4094)**  
> **[1.958954] Starting init: /bin/sh exists but couldn’t execute it (error -4094)**  
> [1.966279] Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.  
> [1.980454] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.14.62-imx\_4.14.62\_1.0.0\_beta+g1907fe4 #1  
> [1.994287] Call trace:  
> [1.996740] [ffff000008089c08] dump\_backtrace+0x0/0x3c8  
> [2.002145] [ffff000008089fe4] show\_stack+0x14/0x20  
> [2.007202] [ffff000008757620] dump\_stack+0x9c/0xbc  
> [2.012257] [ffff0000080cb0b8] panic+0x124/0x29c  
> [2.017053] [ffff000008769cfc] kernel\_init+0xec/0x100  
> [2.022282] [ffff000008084ed8] ret\_from\_fork+0x10/0x18  
> [2.027598] SMP: stopping secondary CPUs  
> [2.031524] Kernel Offset: disabled  
> [2.035015] CPU features: 0x0802008  
> [2.038497] Memory Limit: none  
> [2.041552] Rebooting in 10 seconds…

On further debugging, I have found that:

- When started the artifact download, the partitions from **file system** were:

> mender\_boot\_part=2  
> mender\_boot\_part\_hex=2

- But, after power failure and switch ON, the partitions from **U-boot** become:

> mender\_boot\_part=1  
> mender\_boot\_part\_hex=1

I believe this is causing the above kernel panic since, the inactive partition is already streamed and corrupted during the power failure. Please correct me if I’m wrong.

But, how to set it properly? Why it is switching to inactive partition on U-boot during the power failure? Am I missing anything here?

PS: If I didn’t update using mender artifact and changed the partitions manually (by setting _mender\_boot\_part_ and _mender\_boot\_part\_hex_), then, there is no issues and expected partition only marked as active partition.

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 6:26am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/2 "2019-04-10T06:26:19Z")

</div>

Could anyone knows what is missing here? Should I give any specific details?

Please note that, the [integration checklist](https://docs.mender.io/1.7/devices/yocto-project/bootloader-support/u-boot/integration-checklist) is already verified and is working fine for the platform.

Any help would be really appreciated.

---

<div class="post-metadata">

**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\
**Post date:** [April 10, 2019, 6:51am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/3 "2019-04-10T06:51:59Z")

</div>

This is worrying. Whilst I don’t know the answer the first thing I would do is look though the source code on github to see when the uboot variables get changed when using standalone mode.

I myself use standalone mode, but have not checked this scenario yet. I have now added it to the list.

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 6:56am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/4 "2019-04-10T06:56:15Z")

</div>

@dellgreen Thank you and agreed on the point of checking on the source code. Since, this is a power failure test and that can happen at any point of time during the update, I thought of asking the experts 🙂

---

<div class="post-metadata">

**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\
**Post date:** [April 10, 2019, 8:13am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/5 "2019-04-10T08:13:03Z")

</div>

Unfortunately I am not going to be able to check this scenario this week as working on other things, maybe one of the go-lang guys @mirzak can confirm whether the uboot variables get changed prior to download in standalone mode under some conditions, because my understanding was that they didn’t get changed until you ran mender-client with the -commit argument after doing a mender-client -rootfs. And this seems to certainly be the case as several times i have forgotten to commit the changes and rebooted to end up on the old partition by mistake.

In your case this doesn’t sound like it is the case. Could it be something else you have going on? i.e. do you have your own systemd service that touches mender-client on boot or touches uboot variables?

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 10:26am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/6 "2019-04-10T10:26:49Z")

</div>

I’m not sure whether this is a mender standalone issue or something with my set-up. I’m disabling the mender client daemon during the boot-up as below:

> sudo systemctl stop mender  
> sudo systemctl disable mender

AFAIK, mender standalone will change the boot partitions once, the client runs _-rootfs_ and finished the download but, not before that. The _-commit_ will make the partition marked as active once, we rebooted and verified.

I found the below observation during further testing:

- I have following rootfs partitions (please note that, this is different from the standard mender partition where an additional (optional) boot partition is present)

> fw\_setenv mender\_boot\_part 1  
> fw\_setenv mender\_boot\_part\_hex 1

> fw\_setenv mender\_boot\_part 2  
> fw\_setenv mender\_boot\_part\_hex 2

- If I’m in the first partition (i.e. mender\_boot\_part 1) and doing the update, then power failure test case is passing and getting the same partition (i.e. mender\_boot\_part 1) after power on and rebooted.
- If I’m in the second partition (i.e. mender\_boot\_part 2) and doing the update, then power failure test is failing and getting other partition (i.e. mender\_boot\_part 1) after power on and rebooted!
- I’m not getting why it is always set to first partition (i.e. mender\_boot\_part 1) during the power failure and reboot scenario.

I think @mirzak can give more details regarding this.

---

<div class="post-metadata">

**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\
**Post date:** [April 10, 2019, 11:03am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/7 "2019-04-10T11:03:15Z")

</div>

I can confirm that the client does not modify the U-Boot environment at all before the download has finished completely. Are you sure that the rootfs partitions are set correctly in `/etc/mender/mender.conf`?

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 12:09pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/8 "2019-04-10T12:09:22Z")

</div>

Please see the mender.conf details of the platform below:

> cat /etc/mender/mender.conf  
> {  
> “InventoryPollIntervalSeconds”: 5,  
> “RetryPollIntervalSeconds”: 30,  
> “RootfsPartA”: “/dev/mmcblk0p1”,  
> “RootfsPartB”: “/dev/mmcblk0p2”,  
> “ServerCertificate”: “/etc/mender/server.crt”,  
> “ServerURL”: “[https://docker.mender.io](https://docker.mender.io)”,  
> “TenantToken”: “dummy”,  
> “UpdatePollIntervalSeconds”: 5  
> }

Do you suspecting anything else which causing the switch?

---

<div class="post-metadata">

**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\
**Post date:** [April 10, 2019, 12:22pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/9 "2019-04-10T12:22:18Z")

</div>

It looks correct. Can you post the whole output from `fw_printenv`?

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 12:29pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/10 "2019-04-10T12:29:12Z")

</div>

Please see the _fw\_printenv_ when the partition is in _/dev/mmcblk0p2_ below:

> fw\_printenv  
> altbootcmd=run mender\_altbootcmd; run bootcmd  
> bootlimit=1  
> bootcount=0  
> upgrade\_available=0  
> mender\_uboot\_boot=mmc 0:1  
> mender\_uboot\_if=mmc  
> mender\_uboot\_dev=0  
> mender\_boot\_kernel\_type=booti  
> mender\_kernel\_name=Image  
> mender\_dtb\_name=fsl-imx8dx-ccu.dtb  
> mender\_pre\_setup\_commands=run m4boot\_0  
> mender\_post\_setup\_commands=  
> mender\_setup=if test “${mender\_pre\_setup\_commands}” != “”; then run mender\_pre\_setup\_commands; fi; setenv mender\_kernel\_root /dev/mmcblk0p${mender\_boot\_part}; if test ${mender\_boot\_part} = 1; then setenv mender\_boot\_part\_name /dev/mmcblk0p1; else setenv mender\_boot\_part\_name /dev/mmcblk0p2; fi; setenv mender\_kernel\_root\_name ${mender\_boot\_part\_name}; setenv mender\_uboot\_root mmc 0:${mender\_boot\_part\_hex}; setenv mender\_uboot\_root\_name ${mender\_boot\_part\_name}; setenv expand\_bootargs “setenv bootargs \”${bootargs}\“”; run expand\_bootargs; setenv expand\_bootargs; if test “${mender\_post\_setup\_commands}” != “”; then run mender\_post\_setup\_commands; fi  
> mender\_altbootcmd=if test ${mender\_boot\_part} = 1; then setenv mender\_boot\_part 2; setenv mender\_boot\_part\_hex 2; else setenv mender\_boot\_part 1; setenv mender\_boot\_part\_hex 1; fi; setenv upgrade\_available 0; saveenv; run mender\_setup  
> mender\_try\_to\_recover=if test ${upgrade\_available} = 1; then reset; fi  
> bootcmd=run mender\_setup; setenv bootargs root=${mender\_kernel\_root} ${bootargs}; if test “${fdt\_addr\_r}” != “”; then load ${mender\_uboot\_root} ${fdt\_addr\_r} /boot/${mender\_dtb\_name}; fi; load ${mender\_uboot\_root} ${loadaddr} /boot/${mender\_kernel\_name}; ${mender\_boot\_kernel\_type} ${loadaddr} - ${fdt\_addr\_r}; run mender\_try\_to\_recover  
> bootdelay=3  
> baudrate=115200  
> ethprime=eth0  
> loadaddr=0x80280000  
> mfgtool\_args=setenv bootargs console=${console},${baudrate} rdinit=/linuxrc clk\_ignore\_unused  
> kboot=booti  
> bootcmd\_mfg=run mfgtool\_args;if iminfo ${initrd\_addr}; then if test ${tee} = yes; then bootm ${tee\_addr} ${initrd\_addr} ${fdt\_addr}; else booti ${loadaddr} ${initrd\_addr} ${fdt\_addr}; fi; else echo “Run fastboot …”; fastboot 0; fi;  
> initrd\_addr=0x83100000  
> initrd\_high=0xffffffffffffffff  
> m4\_0\_image=m4\_ccu\_app.bin  
> loadm4image\_0=load mmc 0:5 ${loadaddr} ${m4\_0\_image}  
> m4boot\_0=run loadm4image\_0; dcache flush; bootaux ${loadaddr} 0  
> fdt\_file=fsl-imx8dx-ccu.dtb  
> fdt\_addr\_r=0x83000000  
> fdt\_high=0xffffffffffffffff  
> ethaddr=00:01:02:03:04:05  
> loadaddr=0x80280000  
> kernel=Image  
> bootargs=console=ttyLP0,115200 earlycon=lpuart32,0x5a060000,115200 rootwait rootfstype=ext4 rw  
> mender\_boot\_part=2  
> mender\_boot\_part\_hex=2

The issue is coming only when the system is in RootfsPartB (i.e /dev/mmcblk0p2 or mender\_boot\_part 2).

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 10, 2019, 12:40pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/11 "2019-04-10T12:40:01Z")

</div>

I have not been able to reproduce the reported issues on one of my boards.

If this is only happening when you are on running on `RootfsPartB` it could mean that you are somehow corrupting the U-boot environment and it reverts to the default one, which would put you on `mender_boot_part=1` again.

Can you list output of `fdisk -l <your disk on device>` and also where the U-boot environment is stored

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 1:33pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/12 "2019-04-10T13:33:40Z")

</div>

Thank you @mirzak for confirming that, this issue is not coming by default in standalone mode.

Please see the `fdisk -l <your disk on device>` details below:

```
fdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 7456 MB, 7818182656 bytes, 15269888 sectors
238592 cylinders, 4 heads, 16 sectors/track
Units: cylinders of 64 * 512 = 32768 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/mmcblk0p1 32,0,1 31,3,16 2048 657407 655360 320M 83 Linux
/dev/mmcblk0p2 32,0,1 31,3,16 657408 1312767 655360 320M 83 Linux
/dev/mmcblk0p3 32,0,1 287,3,16 1312768 1329151 16384 8192K 83 Linux
/dev/mmcblk0p4 288,0,1 1023,3,16 1329152 15269887 13940736 6807M 5 Extended
/dev/mmcblk0p5 320,0,1 383,3,16 1331200 1335295 4096 2048K 83 Linux
/dev/mmcblk0p6 416,0,1 415,3,16 1337344 5531647 4194304 2048M 83 Linux
/dev/mmcblk0p7 448,0,1 447,3,16 5533696 6582271 1048576 512M 83 Linux
/dev/mmcblk0p8 480,0,1 991,3,16 6584320 6617087 32768 16.0M 83 Linux
/dev/mmcblk0p9 0,0,1 1023,3,16 6619136 15269887 8650752 4224M 83 Linux

```

We are storing the U-boot environment at _0x400000_ location of _/dev/mmcblk0_.

Please let me know whether anything missing or wrong here.

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 10, 2019, 1:39pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/13 "2019-04-10T13:39:42Z")

</div>

Yeah as suspected 🙂

> /dev/mmcblk0p1 32,0,1 31,3,16 2048 657407 655360 320M 83 Linux

Above is equal to a starting address of _0x100000_ which overlaps with your U-boot environment. Meaning that when you write data to `/dev/mmcblk0p1` you will overwrite what is at _0x400000_

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 10, 2019, 1:57pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/14 "2019-04-10T13:57:46Z")

</div>

Sorry, I didn’t get it 🤔 Could you please elaborate a little more?

> [@mirzak](#):
>
> Meaning that when you write data to `/dev/mmcblk0p1` you will overwrite what is at _0x400000_

Does this means that, when I’m in _/dev/mmcblk0p2_ and doing the update (streaming to _/dev/mmcblk0p1_) this will overwrite what is at _0x400000_?

How to resolve this problem?

---

<div class="post-metadata">

**Author:** ![TimFroehlich](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/timfroehlich/32/164_2.png) [@TimFroehlich](https://hub.mender.io/u/TimFroehlich)\
**Post date:** [April 10, 2019, 5:31pm UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/15 "2019-04-10T17:31:51Z")

</div>

I believe the issue is that you have the U-boot environment at _0x400000_ but your /dev/mmcblk0p1 partition extends from the beginning of the mmc device to after the location that you’re putting your U-boot environment, completely overlapping it. Because your partition table has no idea that you put your U-boot there, your Linux has no idea that there’s data there that it shouldn’t overwrite when performing an update.

I’d suggest enhancing whatever method you’re using to generate your partition table to account for the fact that you have essentially a shadow partition on the disk that needs to be avoided.

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 11, 2019, 6:23am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/16 "2019-04-11T06:23:54Z")

</div>

Thank you very much @dellgreen @TimFroehlich @kacf and @mirzak for your support. I got the point 👍

---

<div class="post-metadata">

**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\
**Post date:** [April 11, 2019, 6:47am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/17 "2019-04-11T06:47:46Z")

</div>

Should/could we add something to meta-mender to detect such situations and warn/error user during build as it was nit immediately obvious what the cause of this problem was for an average user.

---

<div class="post-metadata">

**Author:** ![mterwoord](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mterwoord/32/100_2.png) [@mterwoord](https://hub.mender.io/u/mterwoord)\
**Post date:** [April 11, 2019, 7:02am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/18 "2019-04-11T07:02:28Z")

</div>

Sorry to jump in, but imo, if its low impact for everybody, and reasonably doable, I’m all for making meta-mender detect such situations.

(Just my 2 cents…)

---

<div class="post-metadata">

**Author:** ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)\
**Post date:** [April 11, 2019, 8:34am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/19 "2019-04-11T08:34:09Z")

</div>

If one is using the provided `sdimg` image type one is protected from this happening with this,

> <https://github.com/mendersoftware/meta-mender/blob/68219086afdd6c7c1b6675753b7a7acefb0e457e/meta-mender-core/classes/mender-part-images.bbclass#L145>

and there are more sanity checks that make sure things do not overlap.

But I believe @ajithpv is using a custom image type and in that case there is nothing we could add to meta-mender to check this as we have no insight in how the image is generated. At least nothing that I am aware of and if someone has creative ideas please feel free 😃

---

<div class="post-metadata">

**Author:** ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)\
**Post date:** [April 11, 2019, 9:47am UTC](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462/20 "2019-04-11T09:47:51Z")

</div>

Yes, I agreed to @mirzak’s point . I’m using a custom partition layout which is different from the Mender standard partition layout. Hence, I believe that, the one who is making the custom changes should be take care of the sanity checks 🙂

[Next page](https://hub.mender.io/t/mender-1-7-standalone-power-failure-case-failing/462.md?page=2)