I think I am almost there with the mender integration in our custom hardware. The last step is still to go, though, as my rootfs update via mender keeps on failing. Here is the situation in detail:
I managed to build my ubimg with the five volumes as intended (rootfs x2, uboot-env x2, data), flashed that into my NAND and I am able to check the integration list , so I am really happy so far
When trying to do a rootfs update, it fails via server deployment[1] and via local update. As a first step, I can access the two ubifs volumes with no problems (ubi0_0 and ubi0_1) after booting. I read this topic and followed Mirzak’s steps for local installation, leading to:
Can you give me some more information on how I can investigate the problem further? Overall there are several places where the problem could be. Is there a possibility to test the update process manually? Say perhaps untar the *.mender file and exchange the ubi0_1 volume on my own? Another question is: when defining the MENDER_x variables at my local.conf file, these values are available for the bootloader and the kernel implementation, arent they?
You can try to untar and dd it to the device manually, but I suspect that it won’t work because ubi devices are not regular block devices. But you can try some other things:
Before running an update, can you do md5sum /dev/ubi0_1sha256sum /dev/ubi0_1. Compare this to the checksum reported by doing mender-artifact read ARTIFACT.mender.
Can you mount the image on ubi0_1 before you run mender install? At compile time they are filled with the same content, so it should be possible to mount it.
What about the size of the device. Does blockdev --getsize64 /dev/ubi0_1 work? Does it match what mender-artifact read reports?
What does /etc/mender/mender.conf contain?
What does /etc/fw_env.config contain?
Yes.
Update: You need to use sha256sum of course, not md5sum.
Regarding the sha256 checksum I found out, that the ubi0_1 checksum and the *.mender checksum do not match. However, this is the case for the working version on my device vs. the mender artifact before installing the update. AFTER doing mender install, they do match (but the ubifs is corrupted).
It is totally possible to mount the ubi0_1 volume before updating. As a result, the mender integration list works well, as switching between the volumes with a reboot also does the thing.
I did not manage the get a valid output of the blockdev call. This is the case for any ubi volume. As far as I understand, the mender client can handle the volume correctly (see matching checksum).
So according to the checksum comparison, writing to the UBI volume works as intended, the not-matching checksum is strange. Or are there any hardware dependent things that must be accessible for the build process besides PEB/LEB? When flashing my *.ubimg file to the NAND, the u-boot takes care of that.
Yes, this is indeed strange. If we look at the code which builds the ubimg, we can see that it uses the raw filesystem blob as input. It’s not rebuilding the filesystem for the artifact, and then for the ubimg. They are literally the same filesystem. So why is the checksum different? Is U-Boot changing it?
It might be helpful if you could dump the U-Boot flashed content on ubi0_1 somewhere and compare it to the one in the original ubifs. For example, use ubireader_display_info -v <FILE> on both.
Okay, I found out some more things. When flashing my NAND via u-boot I have two different possibilities: either I use the nand write or the nand write.trimffs function (which is also handled here: clicky, more infos: clicky). The first one will lead to a successful boot, but I cannot reboot then due to a ubi_io_read error. with lots of ECC errors. BUT: the checksum of the ubi0_1 volume matches then the one from my host build. On the other side, the second command will lead to a perfectly fine reboot, but the checksum differs (as 0xff data at the end of pages will not be written to the NAND flash). As a result (and as far as I can see) I either have to fix my reboot issue with the normal nand write OR ensure that the kernel writes in a nand write.trimffs mode …
This is a very interesting finding, @buffo. I don’t think the client takes this into account. But it may not need to take it into account if the filesystem size is an exact multiple of the MENDER_UBI_LEB_SIZE (which it should be). Can you check what it is with:
bitbake -e core-image-minimal | grep ROOTFS_SIZE=
You’ll get several numbers, they might all be interesting.
Ya, the rootfs size matches this number. On the one side the size of the direct build output and on the other side the one inside the mender artifact, as 231,508 * 1,024 = 237,064,192.
I played a little bit with the mtd-utils-ubifs tools on my device. I don’t know if this information is helpful, as I am not aware of the mender-client mechanics. What I did was:
(Skipped attaching ubi_ctrl as its already attached)
Removed the erroneous ubi volume with ubirmvol /dev/ubi0 -n 1
Checked available eraseblocks and their size. Please note the available blocks differ by 1 to my expectation (1868 vs. 1867):
Create new volume: ubimkvol /dev/ubi0 -N rootfsb_alt -s 237191168
Second I build my related artifact, but when trying to install the artifact, I run in the following problem, that I cannot cat to the inactive ubifs:
root@device:/mnt# mender install my-rootfs-update-1.0.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf
INFO[0000] Loaded configuration file: /etc/mender/mender.conf
WARN[0000] Could not resolve path link: ubi0_0 Attempting to continue
WARN[0000] Could not resolve path link: ubi0_1 Attempting to continue
INFO[0000] Mender running on partition: ubi0_0
INFO[0000] Start updating from local image file: [my-rootfs-update-1.0.mender]
Installing Artifact of size 49954816...
INFO[0000] No public key was provided for authenticating the artifact
INFO[0000] Update module output: /dev/ubi0_1
INFO[0000] Update module output: streams/swarco-image-scc-air-v2.ext4
INFO[0000] Update module output: cat: write error: Operation not permitted
ERRO[0000] Download failed: Payload: can not install Payload: image-device.ext4: Update module terminated abnormally: exit status 1
ERRO[0000] Payload: can not install Payload: image-device.ext4: Update module terminated abnormally: exit status 1
This is the related snippet (I added some debug output):
case "$1" in
Download)
file="$(cat stream-next)"
echo $passive
echo $file
cat "$file" > $passive
if [ "$(cat stream-next)" != "" ]; then
echo "More than one file in payload"
exit 1
fi
;;
What am I doing wrong here, have I to set some permissons first?
Was there a successful manual update (by extracting the content of the mender artifact and flashing it to the inactive partition)?
Trying to flash the ext4 file manually into the inactive partition ends up in the same result as using the rootfs-v2 update module. I cannot write to ubi0_1 due to writing permissions. When I mount it first and untar it to there, it totally works fine.
Are you using ext4? On UBI only ubifs is supported, as far as I know. This could explain why it can’t be mounted, and why creating it manually and untaring on top of it works.
I think you cannot do direct writing to UBI devices like rootfs-image-v2 does. It was only ever made to work with regular block devices. According to this thread you can use this to flash the volume:
Okay, so I cannot use the rootfs-image-v2 update module to update my pre-made ubifs? As suggested, the ubiupdatevol method works with no problem, but the update via the mender client still fails.
Ok, I don’t know why that is. You can certainly adapt rootfs-image-v2 to work with ubiupdatevol though. Just look for whether the device matches the /dev/ubi* pattern, and if so, use ubiupdatevol instead of cat.
I suspect though, that ubiupdatevol will not work with a pipe, which is what is given to cat (the file variable in the rootfs-image-v2 script). So it may be better to make a new Update Module with the Download state removed, and rely on calling ubiupdatevol with a file instead, as per the file tree API. If you do make this work, it would be a worthwhile addition to Mender’s Update Modules.
