# Mender update with initramfs

**URL:** <https://hub.mender.io/t/mender-update-with-initramfs/6627>\
**Category:** General Discussions\
**Tags:** mender-client, imx6ull, kirkstone\
**Created:** [March 14, 2024, 4:05pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627 "2024-03-14T16:05:16Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 14, 2024, 4:05pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/1 "2024-03-14T16:05:16Z")

</div>

Hello Mender community,

Currently I’m test driving Mender as a possible update solution for our distribution.  
We have a compute-module which includes the imx6ull with some onboard flash memory (NAND).  
The system uses u-boot, which loads our fitImage that includes the device tree and kernel.

I took inspiration from the community meta layer which included somethings that where required  
to get it to run. I’m quite impressed in how quickly I got everything up and running, so far, so good you  
might think.

The real hurdle started today where I was testing the system with an initramfs implementation which  
we need to support secure boot. I got the following message from the mender client:

```auto
mender[1165]: time="2024-03-14T15:33:30Z" level=error msg="Artifact install failed: Payload: can not install Payload: my-image.ubifs: Active root partition matches neither RootfsPartA nor RootfsPartB."

```

So I dove into the console and verified that all settings where correct, at-least that is what I think ;-).

```auto
cat /data/mender/mender.conf 
{
    "RootfsPartA": "ubi0_0",
    "RootfsPartB": "ubi0_1"
}
cat /etc/mender/mender.conf
{
    "InventoryPollIntervalSeconds": 300,
    "RetryPollIntervalSeconds": 300,
    "ServerURL": "perhaps-less-secret",
    "TenantToken": "very-secret",
    "UpdatePollIntervalSeconds": 180
}

```

When I remove the initramfs from the fitImage and adjust the ubiargs then everything boots fine and  
the device can be updated by means of Mender. If reinsert the initramfs then I got the previously  
mentioned error.

I believe it has something to do with the changes that are different between the initramfs enabled boot  
process and the one without initramfs, but I can’t figure out why the mender-client fails. The mount  
command seems OK to me (I only pasted what I think was needed).

```auto
/dev/ubi0_1 on / type ubifs (ro,relatime,assert=read-only,ubi=0,vol=1)
/dev/ubi0_2 on /data type ubifs (rw,relatime,assert=read-only,ubi=0,vol=2)
/data/overlay-etc/upper on /etc type overlay (rw,relatime,lowerdir=/etc,upperdir=/data/overlay-etc/upper,workdir=/data/overlay-etc/work)

```

I use the trial version of the Mender server, the mender-client is version 3.5.2 (runtime: go1.17.13). The  
meta-mender repository is fixed to tag “kirkstone-v2023.12”.

Could anyone point me in the correct direction into solving this issue.

Thank you advance, with kind regards,

Jeffrey Simons

Software Engineer

- Royal Boon Edam International B.V.

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [March 15, 2024, 7:50pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/2 "2024-03-15T19:50:25Z")

</div>

Hi @JSimons,

The error message definitely sounds like it should work, with `/dev/ubi0_1` as the matching root partition. Can you please post the kernel command line in effect (`/proc/cmdline`) for both cases, maybe this gives a useful hint.

Greets,  
Josef

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 18, 2024, 6:31am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/3 "2024-03-18T06:31:09Z")

</div>

Hi @TheYoctoJester,

Thank you for your response.  
I had to swap hardware modules because I’m currently not working at home, so this devices is still  
pointing to ubi0\_0 as the primary rootfs.

```auto
cat /proc/cmdline
user_debug=30 ubi.mtd=ubi root=/dev/ubi0_0 rw ubi.fm_autoconvert=1 console=tty1 console=ttymxc0,115200n8 consoleblank=0

```

I hope this gives more insight in my issue.

Jeffrey Simons

- Royal Boon Edam International B.V.

---

<div class="post-metadata">

**Author:** ![ruben](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ruben/32/565_2.png) [@ruben](https://hub.mender.io/u/ruben)\
**Post date:** [March 18, 2024, 6:42am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/4 "2024-03-18T06:42:01Z")

</div>

I had a similar problem in the past. For me, when booting from initramfs the kernel command line changed to `root=/dev/loop0` which was not any of the configured A or B partitions.

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 18, 2024, 1:26pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/5 "2024-03-18T13:26:56Z")

</div>

Hi @ruben ,

Thank you for your suggestion, I validated the cmdline between the initramfs and rootfs switch  
but can’t find anything out of the ordinary. I stepped through the mounting process and all looks  
OK to me up to the point where it does the switch\_root.

Jeffrey

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 18, 2024, 2:19pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/6 "2024-03-18T14:19:57Z")

</div>

Hi @TheYoctoJester ,

I just had an epiphany ;-). I changed the file mender.conf in /data/mender.

```auto
cat mender.conf 
{
    "RootfsPartA": "/dev/ubi0_0",
    "RootfsPartB": "/dev/ubi0_1"
}

```

Notice the /dev/ that I have inserted before the ubi0\_x. That gave me a different message but atleast  
I think that I’m on to something (see also the log that I got from mender-client).

```auto
Mar 18 13:47:23 cg-6cebdf mender[6535]: time="2024-03-18T13:47:23Z" level=info msg="Mender running on partition: /dev/ubi0_0"
... removed some info
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="successfully received new authorization data from server <REMOVED_INFO>"
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="State transition: inventory-update [Sync] -> check-wait [Idle]"
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="State transition: check-wait [Idle] -> update-check [Sync]"
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="Validating the Update Info: <REMOVED_INFO>"
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="State transition: update-check [Sync] -> update-fetch [Download_Enter]"
Mar 18 13:47:35 cg-6cebdf mender[6535]: time="2024-03-18T13:47:35Z" level=info msg="Running Mender client version: 3.5.2"
Mar 18 13:47:36 cg-6cebdf mender[6535]: time="2024-03-18T13:47:36Z" level=info msg="State transition: update-fetch [Download_Enter] -> update-store [Download_Enter]"
... removed some info
Mar 18 13:47:36 cg-6cebdf mender[6535]: time="2024-03-18T13:47:36Z" level=info msg="Opening device \"/dev/ubi0_1\" for writing"
Mar 18 13:47:36 cg-6cebdf mender[6535]: time="2024-03-18T13:47:36Z" level=info msg="Native sector size of block device /dev/ubi0_1 is 126976 bytes. Mender will write in chunks of 2031616 bytes"
Mar 18 13:47:37 cg-6cebdf mender[6535]: time="2024-03-18T13:47:37Z" level=error msg="Failed to write 172687872 bytes to the new partition"
Mar 18 13:47:37 cg-6cebdf mender[6535]: time="2024-03-18T13:47:37Z" level=info msg="The optimized block-device writer wrote a total of 2 frames, where 2 frames did need to be rewritten (i.e., skipped)"
Mar 18 13:47:37 cg-6cebdf mender[6535]: time="2024-03-18T13:47:37Z" level=error msg="Artifact install failed: Payload: can not install Payload: my-image.ubifs: write /dev/ubi0_1: operation not permitted"

```

Can I safely assume that mender-client is a bit picky in how it is given the information regarding the  
root partition, because that has slightly changed with the initramfs change. (perhaps that was the  
thing you referred to @ruben?)

What is your advice into solving this issue?

- I could replace the rootfs init script from Yocto to match the desired outcome by pre-pending /dev/.  
– or
- Modifying an option in Mender during the build process.

Thank you in advance.

Jeffrey

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 21, 2024, 8:35am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/7 "2024-03-21T08:35:38Z")

</div>

Hi @TheYoctoJester ,

I build two setups one running with initramfs and one without. The key difference that I can observe with  
my limited knowledge is that the Mender running on partition differs between the two.

```auto
Unit with initramfs:
Mar 21 07:42:17 cg-6ce919 mender[1992]: time="2024-03-21T07:42:17Z" level=info msg="Mender running on partition: /dev/ubi0_0"

```

```auto
Unit without initramfs:
Mar 21 08:15:48 cg-6ced19 mender[594]: time="2024-03-21T08:15:48Z" level=info msg="Mender running on partition: ubi0_0"

```

Adjusting the RootfsPartA/B in the mender.conf most likely changes the point in the filesystem which  
denies it to write to that location (not sure here).

Both units have the same kernel cmdline:

```auto
user_debug=30 ubi.mtd=ubi root=ubi0_0 rw rootfstype=ubifs ubi.fm_autoconvert=1 console=tty1 console=ttymxc0,115200n8 consoleblank=0

```

The only difference is the way they are mounted the following has been modified in the rootfs script from initramfs-framework.

```auto
elif ["`echo ${bootparam_root} | cut -c1-5`" != "/dev/"] && \
		[! -e "$bootparam_root"]; then
	debug "No exact path is supplied, prepending /dev/ to $bootparam_root"
	bootparam_root="/dev/${bootparam_root}"
fi

```

```auto
Debug from shell:
DEBUG: No e2fs compatible filesystem has been mounted, mounting ubi0_0...
DEBUG: No exact path is supplied, prepending /dev/ to ubi0_0
<some ubi mounting messages>
DEBUG: Loading module finish
DEBUG: Running finish_run
Switching root to '/rootfs'...
DEBUG: Moving basic mounts onto rootfs
DEBUG: Moving /dev, /proc and /sys onto rootfs...
PREINIT: Start

```

Perhaps this will give some insights.

Thank you,

Jeffrey

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [March 21, 2024, 8:58am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/8 "2024-03-21T08:58:23Z")

</div>

Hi @JSimons,

I gave it a cursory inspection, and indeed UBIFS requires special handling. That’s also why we have the two main Yocto classes `mender-full` and `mender-full-ubi`. 😅  
Yet as far as I can tell, none of the ready made setups that we have is initramfs-aware. So my advice would be to start with Mender Client 4.0, because there the Update Module for the root filesystem is factored out and can easily be modified. Have a look at [mender/support/modules/rootfs-image at master · mendersoftware/mender · GitHub](https://github.com/mendersoftware/mender/blob/master/support/modules/rootfs-image#L61) for starters, and you probably can adjust it to your integration straight away.

Greets,  
Josef

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 21, 2024, 9:55am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/9 "2024-03-21T09:55:57Z")

</div>

Hi @TheYoctoJester ,

Thank you for the quick response.  
I will take a look at Mender 4.0, I will let you know if that worked.

Kind regards,

Jeffrey

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 22, 2024, 1:57pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/10 "2024-03-22T13:57:58Z")

</div>

Hi @TheYoctoJester ,

I got the 4.0.1 version of Mender integrated and it appears that the message regarding the Rootfs has  
disappeared, which is good. Also it seems to me that it wants to start downloading the new update.

Nevermind what I wrote here, the update works!!!

After taking a close look at the rootfs install file I saw mender-flash being included, removed it from the  
bin-dir and boom it worked! Now I only need to whisper to Yocto and meta-mender to not include the  
mender-flash tool.

Thank you for your support @TheYoctoJester, I can now move on.

Have a nice weekend,

Jeffrey

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [March 22, 2024, 3:02pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/11 "2024-03-22T15:02:50Z")

</div>

Hi @JSimons,

Thanks for the report! Just so I understand correctly:

- with Mender Client 4.0 the procedure works out of the box, except,
- `mender-flash` needs to be removed…where? From `/bin` of the root filesystem?  
That definitely sounds strange.

Greets,  
Josef

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 22, 2024, 3:21pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/12 "2024-03-22T15:21:00Z")

</div>

Hi @TheYoctoJester ,

Yes (somewhat out-of-the box), I had to remove mender-flash from some bindir (I just ran rm `which mender-flash`). So I’m not sure from which directory I removed it 😉 .  
Remember I use ubifs which needs the ubiupdatevol command and there is a test for mender-flash,  
which prevents the ubiupdatevol to run(see also @line 171 from the rootfs-update script).

I will take a closer look on Monday and I’m happy to provide more information if you want that.

Have a nice weekend.

Jeffrey

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [March 22, 2024, 3:22pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/13 "2024-03-22T15:22:54Z")

</div>

@JSimons ah right, the ubi thing. Yes let’s try to dig into it on Monday.👍

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 27, 2024, 11:06am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/14 "2024-03-27T11:06:34Z")

</div>

Hello @TheYoctoJester ,

Sorry it took me bit longer then expected to have some more information regarding the update.

When an update is initiated from the Mender server instance, then mender-flash reports:

```auto
Mar 27 10:56:26 cg-6ced19 mender-update[1676]: record_id=9 severity=error time="2024-Mar-27 10:56:26.696076" name="Global" msg="Broken pipe: AsyncWrite failed"

```

It keeps failing even after executing a rollback followed by a reboot. It just does not want to write the  
data towards the flash device.  
When I update the already existing version then I get somewhat of a verify that executes fine, that leads me to believe that it only fails when writing (reading is fine). Important, this has been done by  
means of an USB flash drive that holds the mender update file.

```auto
Installing artifact...
100%record_id=1 severity=info time="2024-Mar-27 11:13:05.619920" name="Global" msg="Update Module output (stdout): ================ STATISTICS ================" 
record_id=2 severity=info time="2024-Mar-27 11:13:05.622764" name="Global" msg="Update Module output (stdout): Blocks written: 0" 
record_id=3 severity=info time="2024-Mar-27 11:13:05.623276" name="Global" msg="Update Module output (stdout): Blocks omitted: 165" 
record_id=4 severity=info time="2024-Mar-27 11:13:05.623659" name="Global" msg="Update Module output (stdout): Bytes written: 0" 
record_id=5 severity=info time="2024-Mar-27 11:13:05.623990" name="Global" msg="Update Module output (stdout): ============================================" 

```

I took a quick peek at mender-flash source and it appears to be ubi aware, so I assume that it should work.

Kind regards,

Jeffrey

PS. sorry for the edit post, my finger when to quick towards the enter…

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 27, 2024, 2:53pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/15 "2024-03-27T14:53:38Z")

</div>

Hi @TheYoctoJester ,

I tiptoed into the C++ implementation (important, I’m no C++ expert) and printed some values to the  
stderr to validate what is happening. I think I have found some reason why the mender-flash fails to  
program.

I added this check to verify if mender-flash figures out if it is an UBI filesystem.

```auto
main.cpp @ line 111
	if (isUBI == 0) {
		std::cerr << "Not UBI!" << std::endl;
	} else {
		std::cerr << "Is UBI!" << std::endl;
	}

```

Reply:

```auto
record_id=6 severity=info time="2024-Mar-27 14:20:11.018863" name="Global" msg="Update Module output (stderr): Not UBI!"

```

So I dove a bit deeper into the source code and added another debug print.

```auto
platformfs.cpp @ line 229
std::cerr << "st_rdev: " << statbuf.st_rdev << UBIMajorDevNo << (S_ISCHR(statbuf.st_mode) && major(statbuf.st_rdev)) << "Printed.";

```

Reply:

```auto
record_id=1 severity=info time="2024-Mar-27 14:20:11.011341" name="Global" msg="Update Module output (stderr): st_rdev: {...}" 
record_id=2 severity=info time="2024-Mar-27 14:20:11.012118" name="Global" msg="Update Module output (stderr): 62722{...}" 
record_id=3 severity=info time="2024-Mar-27 14:20:11.012566" name="Global" msg="Update Module output (stderr): 250{...}" 
record_id=4 severity=info time="2024-Mar-27 14:20:11.012915" name="Global" msg="Update Module output (stderr): 1{...}" 
record_id=5 severity=info time="2024-Mar-27 14:20:11.013234" name="Global" msg="Update Module output (stderr): Printed.{...}" 

```

The values returned are quite different, so the comparison

```auto
return S_ISCHR(statbuf.st_mode) && major(statbuf.st_rdev) == UBIMajorDevNo;

```

will always be false which leads to the situation where the wrong access is done towards the filesystem.

If I force true to be returned then the implementation appears to be working, so I guess that the  
IsUBIDevice mechanism does not work as intended (the path being passed to the method is correct  
“/dev/ubi0\_1”).

Could one of your experts take a look at this and verify the behaviour?

Thank you,

Jeffrey

---

<div class="post-metadata">

**Author:** ![TheYoctoJester](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/theyoctojester/32/1444_2.png) [@TheYoctoJester](https://hub.mender.io/u/TheYoctoJester)\
**Post date:** [March 27, 2024, 9:07pm UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/16 "2024-03-27T21:07:13Z")

</div>

Hi @JSimons,

Thanks a lot for the deep dive on this! Pinging @kacf for additional input.  
EDIT: the long Easter weekend is right ahead, so please allow for some delay.

Greets,  
Josef

---

<div class="post-metadata">

**Author:** ![JSimons](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/jsimons/32/1936_2.png) [@JSimons](https://hub.mender.io/u/JSimons)\
**Post date:** [March 29, 2024, 7:04am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/17 "2024-03-29T07:04:29Z")

</div>

Hi @TheYoctoJester ,

no worries, I’ll be having a short Easter break as well (back at the 8th of April).

I do have some additional input that I will share, I dove further into the issue together with a colleague  
and we found something that might be interesting. The Major number within the mender-flash tool  
that is used to determine if it is a UBI fs (250 in your case) differs from ours which is 245.

I printed the devices list from the kernel and it showed me the following (where ubi0 resided at major number 245):

```auto
cat /proc/devices 
Character devices:
  1 mem
  4 /dev/vc/0
  4 tty
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  7 vcs
 10 misc
 13 input
 29 fb
 81 video4linux
 89 i2c
 90 mtd
116 alsa
128 ptm
136 pts
153 spi
180 usb
189 usb_device
207 ttymxc
244 ttyGS
245 ubi0
246 pxp_device
247 hidraw
248 rpmb
249 watchdog
250 iio
251 ptp
252 pps
253 rtc
254 gpiochip

```

The kernel documentation states following:  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/3/339d27e98302cdb866ecc47d0f6da4a21d510ada.png)

So I think that the assumption of 250 is not correct and should be changed depending on the platform  
it is running on, do you agree on this or did we made a wrong assumption?

Enjoy the Easter holidays,

Kind regards,

Jeffrey

---

<div class="post-metadata">

**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\
**Post date:** [April 2, 2024, 7:24am UTC](https://hub.mender.io/t/mender-update-with-initramfs/6627/18 "2024-04-02T07:24:57Z")

</div>

Great investigation, @JSimons! Looks like this is a bug, and I have reported it [here](https://northerntech.atlassian.net/browse/MEN-7156). I will ask to get this prioritized too.