# Mender 1.7 - Standalone mode - Kernel read time get difference before and after mender update

**URL:** https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487
**Category:** General Discussions
**Tags:** yocto, sumo, nxp, standalone
**Created:** [April 17, 2019, 7:37am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487 "2019-04-17T07:37:24Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 17, 2019, 7:37am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/1 "2019-04-17T07:37:24Z")

</div>

Hi,  
I have a strange problem which is described below:

- I have a platform where the dtb and kernel image are stored in the eMMC partition.
- If, I flashed the partitions and booted the platform, then the device tree and kernel image read time from Partition A (This is same for Partition B too) is as follows:

> 67376 bytes read in 38 ms (1.7 MiB/s)  
> 10629632 bytes read in **212 ms** ( **47.8 MiB/s** )  
> Flattened Device Tree blob at 83000000  
> Booting using the fdt blob at 0x83000000  
> Using Device Tree in place at 0000000083000000, end 000000008301372f

- If I switch the partition to Partition B using fw\_setenv and reboot, then also the above read time is same. In other words, both partitions are giving the same read time for device tree and kernel image.
- Now, If I’m in partition A and do `mender -rootfs <URI>` update, completes it and rebooted, then the Partition B takes different read time for kernel image as given below:

> 67376 bytes read in 38 ms (1.7 MiB/s)  
> 10629632 bytes read in **23059 ms** ( **449.2 KiB/s** )  
> Flattened Device Tree blob at 83000000  
> Booting using the fdt blob at 0x83000000  
> Using Device Tree in place at 0000000083000000, end 000000008301372f

- But, If I interrupt the update and rebooted, then the partition A will show expected read time (i.e 212 ms).
- This is same for Partition B to Partition A. In other words, If both partitions are updated via `mender -rootfs`, then both partitions will take more time (around 24 seconds) to load the kernel (which as loading within 212ms earlier)
- I’m not doing any change in the boot environments and in all the cases, the dtb and kernel images are inside /boot of rootfs.

I would like to know anyone faced this issue? Is there any fix for this problem?

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 7:49am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/2 "2019-04-17T07:49:19Z")

</div>

Interesting, you are using an iMX8 right?

There are known performance problems with `aarch64` and `ext4` in U-boot.

A couple of links,

- [https://community.nxp.com/thread/472241](https://community.nxp.com/thread/472241)
- [https://github.com/madisongh/meta-tegra/issues/42](https://github.com/madisongh/meta-tegra/issues/42)

The simplest workaround is to switch your root filesystem to ext3 using the _ARTIFACTIMG\_FSTYPE_ variable. But I have also tried some of the patches mentioned in above links which do seem to work.

I am also currently researching this right now to figure out the extent of it and what our recommendations will be.

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 9:35am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/3 "2019-04-17T09:35:07Z")

</div>

Here is a summary of my test results on a IMX8 and 2017.03 U-boot (the one from NXP BSP), and in my case I used an uSD card and would expect the performance to be better on a eMMC.

Stock U-boot (u-boot-imx\_2017.03):

```
u-boot=> ext4load mmc 1:2 $loadaddr /boot/${image}
23065088 bytes read in 79537 ms (282.2 KiB/s)

```

With “fs: ext4: cache extent data” patch, [https://github.com/madisongh/u-boot-tegra/commit/059ace6e05df0bd6842946327119ac0750766580](https://github.com/madisongh/u-boot-tegra/commit/059ace6e05df0bd6842946327119ac0750766580)

```
u-boot=> ext4load mmc 1:2 $loadaddr /boot/${image}
23065088 bytes read in 1403 ms (15.7 MiB/s)

```

With ext3 as filesystem (ARTIFACTIMG\_FSTYPE = “ext3”)

```
u-boot=> ext4load mmc 1:2 $loadaddr /boot/${image}
23065088 bytes read in 1938 ms (11.3 MiB/s)

```

With “-O ^extent”/extent disabled on a “ext4” image, note that `1:3` is the file system without `extent` feature:

```
u-boot=> ext4load mmc 1:2 $loadaddr /boot/${image}
23065088 bytes read in 82223 ms (273.4 KiB/s)
u-boot=> ext4load mmc 1:3 $loadaddr /boot/${image}
23065088 bytes read in 1927 ms (11.4 MiB/s)

```

The ext4 extent performance problem has been fixed in upstream, though it seems to just have missed the 2019.04 release (by one day!), meaning it will not be included until the next release which should be 2019.07.

> <https://github.com/u-boot/u-boot/commit/d5aee659f217746395ff58adf3a863627ff02ec1#diff-7b9d2f880c48c8fd6ec9c29d230f1fee>

---

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 17, 2019, 9:47am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/4 "2019-04-17T09:47:49Z")

</div>

Thank you @mirzak for providing me the links and references. Yes, I’m using i.MX8 device.

But, I would expect the slow read should be always present if the issue is related to the aarch64 and ext4 with U-boot. Why this is happening only when I update the rootfs? If I flashed it using MFG/UUU tools, then the kernel read is faster with the same aarch64 and ext4 !

Whether you are getting the slow read all the time or only once the rootfs got updated with an artifact?

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 9:54am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/5 "2019-04-17T09:54:00Z")

</div>

Can you please run `dumpe2fs <block device>` before update (when it is fast) and after update (when it is slow)

---

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 17, 2019, 10:21am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/6 "2019-04-17T10:21:59Z")

</div>

Unfortunately, I do not have `dumpe2fs` in my rootfs to check the block device.  
But, when I use `mount` command, then I’m getting the ext4 as given below:

- Before update (when it is fast):

> #mount  
> /dev/mmcblk0p2 on / type ext4 (rw,relatime,data=ordered)

- After update (when it is slow):

> #mount  
> /dev/mmcblk0p2 on / type ext4 (rw,relatime,data=ordered)

Moreover, I’m using `sfdisk` with type 83 (Linux) and `mkfs.ext4` for the partition at the initial case.

Could you please let me know whether you are facing the same problem? 🤔

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 10:44am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/7 "2019-04-17T10:44:34Z")

</div>

Yeah, the `mount` output is not that interesting and I am interested of which features of `ext4` are enabled. I would recommend to try to install `dumpe2fs`.

> Could you please let me know whether you are facing the same problem?

I do not have the same behavior that it it is “fast” initially and gets slow after update. It is always slow 😛.

But then again I am using the Yocto build that is `sdimg` and I suspect that the `mfg` applies different settings and that is why I was interested in the `dumpe2fs` output.

---

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 17, 2019, 11:52am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/8 "2019-04-17T11:52:06Z")

</div>

It will be difficult to add `dumpe2fs` now and try because we have a release where we need to reduce the rootfs from 250+MB to 80MB 🤯 . I will try it after the release 👍

Meanwhile, I can able to reduce the delay by switching the root filesystem to ext3 using the _ARTIFACTIMG\_FSTYPE_ variable. I’m getting the below figures:

- Before update (when it is fast):\>

> 67025 bytes read in 38 ms (1.7 MiB/s)  
> 10786824 bytes read in **216 ms** ( **47.6 MiB/s** )

- After update (when it is slow):

> 67025 bytes read in 42 ms (1.5 MiB/s)  
> 10786824 bytes read in **504 ms** ( **20.4 MiB/s** )

PS: I’m using NXP’s `UUU` tool on Linux (not `MFG` tool).

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 12:38pm UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/9 "2019-04-17T12:38:21Z")

</div>

For reference, and what I look at in the `dumpe2fs` output is this,

```auto
$ dumpe2fs /dev/sda2
dumpe2fs 1.45.0 (6-Mar-2019)
Filesystem volume name: primary
Last mounted on: /
Filesystem UUID: d38e1f82-c02c-4ab2-9cb4-42168e73a55e
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize

```

If the `extent` feature is present in `Filesystem features`, that is when the slowdown is occurring. The `extent` feature is part of the “defaults” when you run `mkfs.ext4` which is done in Yocto when it generates the file system image.

This also explains why it works better with `ext3` because this feature is not supported.

From man page of `mkfs.ext4`:

> **extent**  
> Instead of using the indirect block scheme for storing the location of data blocks in an inode, use extents instead. This is a much more efficient encoding which speeds up filesystem access, especially for large files.

But not in U-boot 😛

---

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 17, 2019, 12:58pm UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/10 "2019-04-17T12:58:46Z")

</div>

This is quiet interesting. Indeed, I have to learn a lot 👨‍🚀  
I have enabled the `dumpe2fs` in my rootfs and following are the result:

- Before update (when it is faster):

> **#dumpe2fs /dev/mmcblk0p1**  
> dumpe2fs 1.43.8 (1-Jan-2018)  
> Filesystem volume name:   
> Last mounted on: /  
> Filesystem UUID: f7ffcc14-e22b-4a04-8da0-97c1f39f8cf3  
> Filesystem magic number: 0xEF53  
> Filesystem revision #: 1 (dynamic)  
> Filesystem features: has\_journal ext\_attr resize\_inode dir\_index filetype needs\_recovery **extent** flex\_bg sparse\_super large\_file huge\_file uninit\_bg dir\_nlink extra\_isize  
> Filesystem flags: unsigned\_directory\_hash  
> Default mount options: user\_xattr acl  
> Filesystem state: clean  
> Errors behavior: Continue  
> Filesystem OS type: Linux  
> Inode count: 81920  
> Block count: 327680  
> Reserved block count: 16384  
> Free blocks: 248598  
> Free inodes: 80398  
> First block: 1  
> Block size: 1024  
> Fragment size: 1024  
> Reserved GDT blocks: 256  
> Blocks per group: 8192  
> Fragments per group: 8192  
> Inodes per group: 2048  
> Inode blocks per group: 256  
> Flex block group size: 16  
> Filesystem created: Thu Jan 1 00:00:04 1970  
> Last mount time: Wed Apr 17 12:32:08 2019  
> Last write time: Wed Apr 17 12:32:08 2019  
> Mount count: 3  
> Maximum mount count: -1  
> Last checked: Thu Jan 1 00:00:04 1970  
> Check interval: 0 ()  
> Lifetime writes: 84 MB  
> Reserved blocks uid: 0 (user root)  
> Reserved blocks gid: 0 (group root)  
> First inode: 11  
> Inode size: 128  
> Journal inode: 8  
> Default directory hash: half\_md4  
> Directory Hash Seed: bb0691da-72e7-415f-b7fc-993b0f533ad5  
> Journal backup: inode blocks  
> Journal features: (none)  
> Journal size: 8M  
> Journal length: 8192  
> Journal sequence: 0x00000012  
> Journal start: 1

> **#dumpe2fs /dev/mmcblk0p2**  
> dumpe2fs 1.43.8 (1-Jan-2018)  
> Filesystem volume name:   
> Last mounted on: /mnt/mmcblk0p2  
> Filesystem UUID: 7961c8f3-35e1-4f30-beaf-4ece0ae23171  
> Filesystem magic number: 0xEF53  
> Filesystem revision #: 1 (dynamic)  
> Filesystem features: has\_journal ext\_attr resize\_inode dir\_index filetype **extent** flex\_bg sparse\_super large\_file huge\_file uninit\_bg dir\_nlink extra\_isize  
> Filesystem flags: unsigned\_directory\_hash  
> Default mount options: user\_xattr acl  
> Filesystem state: clean  
> Errors behavior: Continue  
> Filesystem OS type: Linux  
> Inode count: 81920  
> Block count: 327680  
> Reserved block count: 16384  
> Free blocks: 250178  
> Free inodes: 80418  
> First block: 1  
> Block size: 1024  
> Fragment size: 1024  
> Reserved GDT blocks: 256  
> Blocks per group: 8192  
> Fragments per group: 8192  
> Inodes per group: 2048  
> Inode blocks per group: 256  
> Flex block group size: 16  
> Filesystem created: Thu Jan 1 00:00:05 1970  
> Last mount time: Thu Jan 1 00:00:39 1970  
> Last write time: Thu Jan 1 00:00:57 1970  
> Mount count: 1  
> Maximum mount count: -1  
> Last checked: Thu Jan 1 00:00:05 1970  
> Check interval: 0 ()  
> Lifetime writes: 75 MB  
> Reserved blocks uid: 0 (user root)  
> Reserved blocks gid: 0 (group root)  
> First inode: 11  
> Inode size: 128  
> Journal inode: 8  
> Default directory hash: half\_md4  
> Directory Hash Seed: c782444a-686e-4837-8620-82e18ba8edfe  
> Journal backup: inode blocks  
> Journal features: (none)  
> Journal size: 8M  
> Journal length: 8192  
> Journal sequence: 0x00000006  
> Journal start: 0

- After the update (when it is slower):

> **#dumpe2fs /dev/mmcblk0p1**  
> dumpe2fs 1.43.8 (1-Jan-2018)  
> Filesystem volume name:   
> Last mounted on: /  
> Filesystem UUID: f7ffcc14-e22b-4a04-8da0-97c1f39f8cf3  
> Filesystem magic number: 0xEF53  
> Filesystem revision #: 1 (dynamic)  
> Filesystem features: has\_journal ext\_attr resize\_inode dir\_index filetype **extent** flex\_bg sparse\_super large\_file huge\_file uninit\_bg dir\_nlink extra\_isize  
> Filesystem flags: unsigned\_directory\_hash  
> Default mount options: user\_xattr acl  
> Filesystem state: clean  
> Errors behavior: Continue  
> Filesystem OS type: Linux  
> Inode count: 81920  
> Block count: 327680  
> Reserved block count: 16384  
> Free blocks: 248363  
> Free inodes: 80394  
> First block: 1  
> Block size: 1024  
> Fragment size: 1024  
> Reserved GDT blocks: 256  
> Blocks per group: 8192  
> Fragments per group: 8192  
> Inodes per group: 2048  
> Inode blocks per group: 256  
> Flex block group size: 16  
> Filesystem created: Thu Jan 1 00:00:04 1970  
> Last mount time: Wed Apr 17 12:32:08 2019  
> Last write time: Wed Apr 17 12:32:08 2019  
> Mount count: 3  
> Maximum mount count: -1  
> Last checked: Thu Jan 1 00:00:04 1970  
> Check interval: 0 ()  
> Lifetime writes: 94 MB  
> Reserved blocks uid: 0 (user root)  
> Reserved blocks gid: 0 (group root)  
> First inode: 11  
> Inode size: 128  
> Journal inode: 8  
> Default directory hash: half\_md4  
> Directory Hash Seed: bb0691da-72e7-415f-b7fc-993b0f533ad5  
> Journal backup: inode blocks  
> Journal features: (none)  
> Journal size: 8M  
> Journal length: 8192  
> Journal sequence: 0x00000029  
> Journal start: 0

> **#dumpe2fs /dev/mmcblk0p2**  
> dumpe2fs 1.43.8 (1-Jan-2018)  
> Filesystem volume name:   
> Last mounted on: /  
> Filesystem UUID: 3b7c7b4d-301a-4fe9-9808-630314f8cf6f  
> Filesystem magic number: 0xEF53  
> Filesystem revision #: 1 (dynamic)  
> Filesystem features: has\_journal ext\_attr resize\_inode dir\_index filetype needs\_recovery **extent** flex\_bg sparse\_super large\_file huge\_file uninit\_bg dir\_nlink extra\_isize  
> Filesystem flags: signed\_directory\_hash  
> Default mount options: user\_xattr acl  
> Filesystem state: clean  
> Errors behavior: Continue  
> Filesystem OS type: Linux  
> Inode count: 77824  
> Block count: 311296  
> Reserved block count: 15564  
> Free blocks: 235662  
> Free inodes: 76327  
> First block: 1  
> Block size: 1024  
> Fragment size: 1024  
> Reserved GDT blocks: 256  
> Blocks per group: 8192  
> Fragments per group: 8192  
> Inodes per group: 2048  
> Inode blocks per group: 256  
> Flex block group size: 16  
> Filesystem created: Wed Apr 17 12:20:03 2019  
> Last mount time: Wed Apr 17 12:36:46 2019  
> Last write time: Wed Apr 17 12:36:46 2019  
> Mount count: 1  
> Maximum mount count: -1  
> Last checked: Wed Apr 17 12:20:04 2019  
> Check interval: 0 ()  
> Lifetime writes: 56 MB  
> Reserved blocks uid: 0 (user root)  
> Reserved blocks gid: 0 (group root)  
> First inode: 11  
> Inode size: 128  
> Journal inode: 8  
> Default directory hash: half\_md4  
> Directory Hash Seed: d6a6319d-e269-47bd-9923-cc176ae360d5  
> Journal backup: inode blocks  
> Journal features: (none)  
> Journal size: 8M  
> Journal length: 8192  
> Journal sequence: 0x00000002  
> Journal start: 1

I’m getting `extend` in all the cases 😕 . This is because I’m using `mkfs.ext4` in the `UUU` tool for partitioning.

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 17, 2019, 1:17pm UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/11 "2019-04-17T13:17:24Z")

</div>

> I’m getting `extend` in all the cases 😕

For this I have no explanation.

It has been very clear in my tests that it is related to `extent`, and by using the following in `local.conf`,

```auto
ARTIFACTIMG_FSTYPE = "ext4"
EXTRA_IMAGECMD_ext4 = "-O ^extent"

```

Which removes the `extent`feature from the `ext4`image, and improves the performance significantly.

> This is because I’m using `mkfs.ext4` in the `UUU` tool for partitioning.

Can you share the full command you are using?

---

<div class="post-metadata">

### Author: ![ajithpv](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/ajithpv/32/728_2.png) [@ajithpv](https://hub.mender.io/u/ajithpv)
#### Post date: [April 22, 2019, 7:32am UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/12 "2019-04-22T07:32:50Z")

</div>

@mirzak, I think I have figured out the reason why UUU loads kernel faster.

The UUU do partitioning as follows:

- Initially partition the eMMC using `sfdisk` with `83 (Linux)` type
- Then UUU do the mkfs.ext4 as below on the partition(s)

> FBK: ucmd mkfs.ext4 -F **-j** /dev/mmcblk0p1

Here, -j option is the reason why UUU partitioned image loads the kernel faster as compared to mender updated one.

According to the man page of `mkfs.ext4`

> **-j** : Create the filesystem with an ext3 journal. If the -J option is not specified, the default journal parameters will be used to create an appropriately sized journal (given the size of the filesystem) stored within the filesystem. Note that you must be using a kernel which has ext3 support in order to actually make use of the journal.

Please let me know your thoughts 🙂

---

<div class="post-metadata">

### Author: ![mirzak](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mirzak/32/2056_2.png) [@mirzak](https://hub.mender.io/u/mirzak)
#### Post date: [April 22, 2019, 6:26pm UTC](https://hub.mender.io/t/mender-1-7-standalone-mode-kernel-read-time-get-difference-before-and-after-mender-update/487/13 "2019-04-22T18:26:44Z")

</div>

> Please let me know your thoughts 🙂

At least it is somewhat consistent as we know already that it works just fine with `ext3`.

I will also give it a try with `-j` flag to confirm on my side but my guess here is that `extens` are not used when the journal is `ext3` compatible and hence why there is no performance drop.