# Problem with rootfs update with \*.mender **URL:** \ **Category:** General Discussions\ **Created:** [June 14, 2021, 7:40am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742 "2021-06-14T07:40:26Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 14, 2021, 7:40am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/1 "2021-06-14T07:40:26Z")
Hi Drew and crew 🙂 I think I am almost there with the mender integration in our custom hardware. The last step is still to go, though, as my rootfs update via mender keeps on failing. Here is the situation in detail: 1. I managed to build my ubimg with the five volumes as intended (rootfs x2, uboot-env x2, data), flashed that into my NAND and I am able to check the [integration list](https://docs.mender.io/client-installation/integration-checklist) , so I am really happy so far 2. When trying to do a rootfs update, it fails via server deployment[1] and via local update. As a first step, I can access the two ubifs volumes with no problems (ubi0\_0 and ubi0\_1) after booting. I read this [topic](https://hub.mender.io/t/issues-with-ubifs-rebooting-and-dashboard/2607/5) and followed Mirzak’s steps for local installation, leading to: ![update_local_installation](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/7/73b5b4468d1204cbb510fb6183f7895e6ba5f9c6.png) And finally to a successful update: ![update_local_success](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/b/b9e4dc86634236adf08c21f1a174aab4e544555f.png) Horray! But after rebooting, the ubifs is corrupted: ![ubierror](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/f/f5507a818d456e18ca3aeedb4aaae35505d2340a.png) and leads to a stack trace and a unfortunately not to a reboot: ![stacktrace](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/5/55121ab345018814b0507d4d84bfaa253480339c.png) The \*.mender and the \*.ubimg are results from the same build process via Yocto, extract from my local.conf: ![localconf](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/9/992a5c798f2741648a8b903c34b54e29e028f610.png) My question is now: are there any further variables to set to get the \*.mender artifact to work to flash the rootfs correctly? Thanks! [1] in fact there is another problem now and then with the server deployment which needs further investigation but is not related to this topic here ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [June 14, 2021, 2:36pm UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/2 "2021-06-14T14:36:50Z")
I don’t have any ideas at the moment. Perhaps @kacf can spot something. Drew ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [June 16, 2021, 8:47am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/3 "2021-06-16T08:47:09Z")
After the `mender install` step, but before rebooting, are you able to mount the volume manually? ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 16, 2021, 8:55am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/4 "2021-06-16T08:55:55Z")
Thanks for answering @kacf . After updating and before booting, the effect is the same: ![ubi_mount](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/6b4381369b16502e2683c74d9f9c025e48e74ac0.png) ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 17, 2021, 12:24pm UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/5 "2021-06-17T12:24:46Z")
Can you give me some more information on how I can investigate the problem further? Overall there are several places where the problem could be. Is there a possibility to test the update process manually? Say perhaps untar the \*.mender file and exchange the ubi0\_1 volume on my own? Another question is: when defining the MENDER\_x variables at my local.conf file, these values are available for the bootloader and the kernel implementation, arent they? ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [June 18, 2021, 7:14am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/6 "2021-06-18T07:14:17Z")
You can try to untar and dd it to the device manually, but I suspect that it won’t work because ubi devices are not regular block devices. But you can try some other things: 1. Before running an update, can you do ~~`md5sum /dev/ubi0_1`~~ `sha256sum /dev/ubi0_1`. Compare this to the checksum reported by doing `mender-artifact read ARTIFACT.mender`. 2. Can you mount the image on `ubi0_1` before you run `mender install`? At compile time they are filled with the same content, so it should be possible to mount it. 3. What about the size of the device. Does `blockdev --getsize64 /dev/ubi0_1` work? Does it match what `mender-artifact read` reports? 4. What does `/etc/mender/mender.conf` contain? 5. What does `/etc/fw_env.config` contain? > [@buffo](#): > > Another question is: when defining the MENDER\_x variables at my local.conf file, these values are available for the bootloader and the kernel implementation, arent they? Yes. **Update:** You need to use `sha256sum` of course, not `md5sum`. ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 21, 2021, 7:57am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/7 "2021-06-21T07:57:45Z")
Hi Kristian, thanks for your reply. 1. Regarding the sha256 checksum I found out, that the `ubi0_1` checksum and the `*.mender` checksum do not match. However, this is the case for the working version on my device vs. the mender artifact before installing the update. AFTER doing mender install, they do match (but the ubifs is corrupted). 2. It is totally possible to mount the `ubi0_1` volume before updating. As a result, the mender integration list works well, as switching between the volumes with a reboot also does the thing. 3. I did not manage the get a valid output of the blockdev call. This is the case for any ubi volume. As far as I understand, the mender client can handle the volume correctly (see matching checksum). ![blockdev_error](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/9/94416f24acac8a0c5bd35ad9f11e90894383f50a.png) ![update+sha](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/1/1e34b64c76d56e14c208c2802318bf948ef9b2a2.png) 4. My `mender.conf` (with cut token) ![menderconf](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/9/970f80b34a529fec299b516a7cff4d57c24302ff.png) 5. My fw\_env: ![fwenv](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/7/7cc1a25c1ddf186539df92f4b20d354ce6ac8aba.png) So according to the checksum comparison, writing to the UBI volume works as intended, the not-matching checksum is strange. Or are there any hardware dependent things that must be accessible for the build process besides PEB/LEB? When flashing my `*.ubimg` file to the NAND, the u-boot takes care of that. Appreciate your help! ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [June 22, 2021, 5:57am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/8 "2021-06-22T05:57:47Z")
> [@buffo](#): > > So according to the checksum comparison, writing to the UBI volume works as intended, the not-matching checksum is strange. Yes, this is indeed strange. If we look at the code which builds the ubimg, we can see that it uses [the raw filesystem blob as input](https://github.com/mendersoftware/meta-mender/blob/daf755df27d142556e43620c6550b94538757c6a/meta-mender-core/classes/mender-ubimg.bbclass#L67). It’s not rebuilding the filesystem for the artifact, and then for the ubimg. They are literally the same filesystem. So why is the checksum different? Is U-Boot changing it? It might be helpful if you could dump the U-Boot flashed content on `ubi0_1` somewhere and compare it to the one in the original ubifs. For example, use `ubireader_display_info -v ` on both. ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 23, 2021, 7:54am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/9 "2021-06-23T07:54:24Z")
Okay, I found out some more things. When flashing my NAND via u-boot I have two different possibilities: either I use the `nand write` or the `nand write.trimffs` function (which is also handled here: [clicky](https://hub.mender.io/t/ubifs-ecc-errors-after-adding-mender-to-toradex-imx6ull-image/1767/6), more infos: [clicky](https://microchipsupport.force.com/s/article/How-to-program-the-rootfs-into-nandflash-through-u-boot-for-SAM9x5-or-SAMA5D3x)). The first one will lead to a successful boot, but I cannot reboot then due to a `ubi_io_read error.` with lots of ECC errors. BUT: the checksum of the `ubi0_1` volume matches then the one from my host build. On the other side, the second command will lead to a perfectly fine reboot, but the checksum differs (as 0xff data at the end of pages will not be written to the NAND flash). As a result (and as far as I can see) I either have to fix my reboot issue with the normal `nand write` OR ensure that the kernel writes in a `nand write.trimffs` mode … ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [June 23, 2021, 8:24am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/10 "2021-06-23T08:24:51Z")
This is a very interesting finding, @buffo. I don’t think the client takes this into account. But it may not need to take it into account if the filesystem size is an exact multiple of the `MENDER_UBI_LEB_SIZE` (which it should be). Can you check what it is with: ```bash bitbake -e core-image-minimal | grep ROOTFS_SIZE= ``` You’ll get several numbers, they might all be interesting. ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 23, 2021, 8:52am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/11 "2021-06-23T08:52:48Z")
Thanks for the information! Currently I am building with my own image, nevertheless I also checked the minimal-image: ![rootfs_size](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/d/d5bfabf3d4bc1b981d29d0cd5a3cf5acc3573ff2.png) As a result, the `IMAGE_ROOTFS_SIZE` must be a multiple of `MENDER_UBI_LEB_SIZE` (126976 in my case)? ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [June 24, 2021, 6:25am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/12 "2021-06-24T06:25:13Z")
And the file size inside the artifact, it matches this? (maybe you can just post the output from `mender-artifact read`) ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 24, 2021, 7:48am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/13 "2021-06-24T07:48:31Z")
Ya, the rootfs size matches this number. On the one side the size of the direct build output and on the other side the one inside the mender artifact, as 231,508 \* 1,024 = 237,064,192. ![images](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/64adb594e2a9d79bc22458b92c6fba3805ee40da.png) ![ext4](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/0/0ab777536708aada7e91adaab281b3030ebc0ac1.png) ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [June 24, 2021, 9:23am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/14 "2021-06-24T09:23:05Z")
I played a little bit with the `mtd-utils-ubifs` tools on my device. I don’t know if this information is helpful, as I am not aware of the mender-client mechanics. What I did was: 1. (Skipped attaching ubi\_ctrl as its already attached) 2. Removed the erroneous ubi volume with `ubirmvol /dev/ubi0 -n 1` 3. Checked available eraseblocks and their size. Please note the available blocks differ by 1 to my expectation (1868 vs. 1867): ![eraseblocks](https://canada1.discourse-cdn.com/flex036/uploads/mender/original/2X/6/62308d621220dcad6ca379bf0ba57c8d0843976e.png) 4. Create new volume: `ubimkvol /dev/ubi0 -N rootfsb_alt -s 237191168` 5. Mount it and untar the `*.tar.gz` to the volume. That did the “light update” for me. ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [July 28, 2021, 1:37pm UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/15 "2021-07-28T13:37:56Z")
Hi @Alan, as discussed I tried the following items: Usage of the update module rootfs-v2: First I created a .conf file at `/etc/mender/rootfs-image-v2.conf` with the following content: ```auto #!/bin/sh MENDER_ROOTFS_PART_A="/dev/ubi0_0" MENDER_ROOTFS_PART_B="/dev/ubi0_1" ``` Second I build my related artifact, but when trying to install the artifact, I run in the following problem, that I cannot cat to the inactive ubifs: ```auto root@device:/mnt# mender install my-rootfs-update-1.0.mender INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf INFO[0000] Loaded configuration file: /etc/mender/mender.conf WARN[0000] Could not resolve path link: ubi0_0 Attempting to continue WARN[0000] Could not resolve path link: ubi0_1 Attempting to continue INFO[0000] Mender running on partition: ubi0_0 INFO[0000] Start updating from local image file: [my-rootfs-update-1.0.mender] Installing Artifact of size 49954816... INFO[0000] No public key was provided for authenticating the artifact INFO[0000] Update module output: /dev/ubi0_1 INFO[0000] Update module output: streams/swarco-image-scc-air-v2.ext4 INFO[0000] Update module output: cat: write error: Operation not permitted ERRO[0000] Download failed: Payload: can not install Payload: image-device.ext4: Update module terminated abnormally: exit status 1 ERRO[0000] Payload: can not install Payload: image-device.ext4: Update module terminated abnormally: exit status 1 ``` This is the related snippet (I added some debug output): ```auto case "$1" in Download) file="$(cat stream-next)" echo $passive echo $file cat "$file" > $passive if ["$(cat stream-next)" != ""]; then echo "More than one file in payload" exit 1 fi ;; ``` What am I doing wrong here, have I to set some permissons first? ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [July 28, 2021, 1:42pm UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/16 "2021-07-28T13:42:08Z")
> Was there a successful manual update (by extracting the content of the mender artifact and flashing it to the inactive partition)? Trying to flash the `ext4` file manually into the **inactive** partition ends up in the same result as using the rootfs-v2 update module. I cannot write to `ubi0_1` due to writing permissions. When I mount it first and untar it to there, it totally works fine. ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [August 2, 2021, 7:02am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/17 "2021-08-02T07:02:20Z")
Are you using ext4? On UBI only ubifs is supported, as far as I know. This could explain why it can’t be mounted, and why creating it manually and untaring on top of it works. I think you cannot do direct writing to UBI devices like `rootfs-image-v2` does. It was only ever made to work with regular block devices. According to [this thread](https://linux-mtd.infradead.narkive.com/Si2GEqkc/cannot-write-to-ubi-volume) you can use this to flash the volume: ```bash ubiupdatevol /dev/ubi1_0 UPDATEFILE ``` ---
**Author:** ![buffo](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/buffo/32/909_2.png) [@buffo](https://hub.mender.io/u/buffo)\ **Post date:** [August 9, 2021, 12:06pm UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/18 "2021-08-09T12:06:23Z")
Okay, so I cannot use the `rootfs-image-v2` update module to update my pre-made ubifs? As suggested, the `ubiupdatevol` method works with no problem, but the update via the mender client still fails. ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [August 10, 2021, 8:49am UTC](https://hub.mender.io/t/problem-with-rootfs-update-with-mender/3742/19 "2021-08-10T08:49:22Z")
Ok, I don’t know why that is. You can certainly adapt `rootfs-image-v2` to work with `ubiupdatevol` though. Just look for whether the device matches the `/dev/ubi*` pattern, and if so, use `ubiupdatevol` instead of `cat`. I suspect though, that `ubiupdatevol` will not work with a pipe, which is what is given to `cat` (the `file` variable in the `rootfs-image-v2` script). So it may be better to make a new Update Module with the `Download` state removed, and rely on calling `ubiupdatevol` with a file instead, as per [the file tree API](https://github.com/mendersoftware/mender/blob/3.0.x/Documentation/update-modules-v3-file-api.md#file-tree). If you do make this work, it would be a worthwhile addition to [Mender’s Update Modules](https://hub.mender.io/c/update-modules/13).