Hi,
I am trying to integrate Mender with an Owasys-5 which is running an iMX8M Plus with 1GB NAND flash and 8GB eMMC. Ideally, I would like the data partition on eMMC, but for now am ignoring the data partition.
Here are my Mender definitions:
MENDER_ARTIFACT_NAME = “release-1”
MENDER_STORAGE_TOTAL_SIZE_MB = “1024”
MENDER_STORAGE_PEB_SIZE = “262144”
MENDER_BOOT_PART_SIZE_MB = “8”
MENDER_DATA_PART_SIZE_MB = “0”
MENDER_MTDPARTS = “nand.0:8m(NAND.u-boot),8m(NAND.u-boot-bckp),1m(NAND.u-boot-env),1m(NAND.u-boot-env.backup),503m(NAND.file-systemA),-(NAND.file-systemB)”
MENDER_FEATURES_ENABLE:append = " mender-uboot mender-image mender-client-install"
When I build, I get the following error:
do_image_mender: Size of rootfs is greater than the calculated partition space (540014592 > 532152320). This image won’t fit on a device with the current storage configuration. Try reducing IMAGE_OVERHEAD_FACTOR if it is higher than 1.0, or raise MENDER_STORAGE_TOTAL_SIZE_MB if the device in fact has more storage.
Can you please explain how I resolve this error?
Hello @smartin,
Thanks for reaching out! Two immediate observations:
- the error message indicates that your resulting build (including the overhead factor) is bigger than the 503MBytes you have allocated for the root filesystem partitions. Thoughts on this:
- While in borderline cases setting the
IMAGE_OVERHEAD_FACTOR might be a temporary fix for evaluation, it is hardly a real solution in the long term because it massively hinders software extensions and maintenance.
- for getting started, try to build a smaller image. 500MBytes is quite hefty for a Yocto-based build, so I can only guess that you are starting out with a quite bloated demo image setup by the vendor. If you can share the BSP/setup you’re using, I can try to replicated and find a suitable target
- “ignoring the data partition” is not an option, because Mender needs a persistent storage area for some things. For getting started, you need to add at least a somewhat small partition.
If you can share the build digest that bitbake shows upon startup, I can have a look and try to find a way forwards.
Greetz,
Josef
Thanks Josef, I have managed to resolve that issue. I think it was because I hadn’t included the mender-ubi feature. I have progressed a little further now and can build the image. I am using the vendor’s code to build a ubi image rather than using the mender-image-ubi feature. Is this a problem?
The issue I’m experiencing now is when the bootloader attempts to mount the filesystem. The nand_mount_fs command is returning the following:
ubi0: default fastmap pool size: 100
ubi0: default fastmap WL pool size: 50
ubi0: attaching mtd5
ubi0 error: validate_ec_hdr: bad VID header offset 2048, expected 4096
ubi0 error: validate_ec_hdr: bad EC header
Erase counter header dump:
magic 0x55424923
version 1
ec 0
vid_hdr_offset 2048
data_offset 4096
image_seq 1449862604
hdr_crc 0x1eb98fc2
erase counter header hexdump:
ubi0 error: ubi_io_read_ec_hdr: validation failed for PEB 0
ubi0 error: ubi_attach_mtd_dev: failed to attach mtd5, error -22
UBI error: cannot attach mtd5
UBI error: cannot initialize UBI, error -22
UBI init error 22
Please check, if the correct MTD partition is used (size big enough?)
Is the mtdparts definition problematic? I have also added the following definition to local.conf:
MENDER_MTD_UBI_DEVICE_NAME = “NAND.file-systemA”
I will also note that the vendor’s image works fine before the Mender integration.
Hi @smartin,
Yes indeed, using the vendor mechanism is quite likely to cause issues. Can you point me to their BSP maybe so I can have a look? Mixing their customizations with parts of the Mender board integration might result in boot issues just as you are describing.
Greetz,
Josef
Here is their BSP layer:
GitHub - Owasys/meta-owasys: Owasys yocto layer for the owa5X device
Here is their project root:
GitHub - Owasys/owa5x_yocto_kirkstone
I have removed the meta-rauc layer. I will try removing their bootloader patches and use the imx version instead.
Hi @smartin,
Just a few notes, as I have not finished my evaluation yet.
- the
cetus image has a lot of hidden things mangled into it, especially the UBI generation is manipulated. I guess that this works to some extent, but does the unmodified setup even boot if you build core-image-minimal or such?
- their
u-boot fork seems to be very minimal, just a slighly bumped uboot-imx. I wouldn’t expect major problems replacing it and modifying as needed.
Concerning the UBI setup, I can’t verify it at the moment but my interpretation of the documentation would be that you need
MENDER_MTD_UBI_DEVICE_NAME = "nand.0"
MENDER_MTDIDS = "..." (see [here](https://docs.mender.io/operating-system-updates-yocto-project/variables#mender_mtdids)
MENDER_MTDPARTS = "..." (you have that already)
MENDER_ROOTFS_PART_A_NAME = "nand.0:NAND.file-systemA"
MENDER_ROOTFS_PART_B_NAME = "nand.0:NAND.file-systemB"
as you have only one flash area, I think MENDER_IS_ON_MTDID does not a need manual setting.
and check if the following variable defaults are okay for you
MENDER_UBI_LEB_PEB_BLOCK_OVERHEAD
MENDER_UBI_LEB_SIZE
MENDER_UBI_TOTAL_BAD_PEB_OVERHEAD
MENDER_NAND_FLASH_PAGE_SIZE
MENDER_PARTITION_ALIGNMENT
Hope this helps, I’ll try to get a proper hold on it in the meantime.
Greetz,
Josef
Thanks Josef, I have made a little progress today. I needed to set the following:
MENDER_NAND_FLASH_PAGE_SIZE = “4096”
I am now running into a problem with the data partition. The vendor had set up eMMC for data, which is very convenient since it gives us an additional 8GB. However, I’m not sure if Mender can handle a combination of NAND and eMMC. Is there some way to support this?
Hi @smartin,
Great to hear!
According to the documentation, you can set MENDER_DATA_PART to basically anything. So in your case, it might be /dev/mmcblk1 as the eMMC block device, therefore
MENDER_DATA_PART = "/dev/mmcblk1p1"
Alternatively, you can just carve out a small data partition, like 32MB, to hold the persistent core configuration and data, and use the eMMC manually to hold your application/usecase data. Either should work.
Greetz,
Josef
I had the following definition:
MENDER_DATA_PART = “/dev/mmcblk2p1”
But I am seeing the following error:
[ 2.909302] UBIFS error (pid: 338): cannot open “/dev/mmcblk2p1”, error -22
I don’t think this should be handled by UBIFS since it is a block-based device. Is there some way of setting an alternate type for the data partition?
Your suggestion of setting aside 32MB sounds logical. Do I need to redefine MTDPARTS if I take this route?
@smartin yes its possible that there is an implicit assumption about all partitions using the same storage subsystem somewhere. 
My personal suggestion would be to yes, redefine the MTDPARTS for a small data partition, make it work for the initial case, and then add things one by one, specifically the second data partition on eMMC.
Greetz,
Josef
Josef,
I’m still a bit stuck with this integration. I’m building the rootfs image without problems and the client is integrated. But I’m having problems generating the bootloader image. I get the following error:
| DEBUG: Executing shell function do_image_bootimg
| 0+0 records in
| 0+0 records out
| 0 bytes copied, 2.0458e-05 s, 0.0 kB/s
| mkfs.fat: Warning: lowercase labels might not work properly on some systems
| mkfs.fat 4.2 (2021-01-31)
| Disk full
| WARNING: exit code 1 from a shell command.
ERROR: Task (/home/smartin/Owa5xYocto/sources/meta-wenco/recipes-core/images/asset-health.bb:do_image_bootimg) failed with exit code ‘1’
It seems to be trying to build the wrong image type. I noted that there is a MENDER_BOOT_PART_FSTYPE variable which has the following options:
- vfat
- btrfs
- ext4
- ext3
- ext2
There doesn’t seem to be an option for ubifs. How do I build the boot image for NAND flash? I can use the vendor’s boot image (it is still being generated together with the Mender patches) but I am then running into problems with the environment variables. The vendor has these in a separate partition whereas Mender expects these to be on a separate volume within the same partition.
Sorry, I take some of this back. The environment variables are on the same partition as the rootfs with two separate volumes. So I think I can just use the vendor’s bootloader with the Mender patches, although I still need to disable the boot image generation in Mender because of the above errors. Does that sound correct?
Hi @smartin,
The original error just indicates that the boot partition is too small. You could address this by setting MENDER_BOOT_PART_SIZE_MB = "32" or whatever matches your requirements, respectively adjusting the mtdparts accordingly. The default is 16 MBytes, which is too small in a number of cases.
Greetz,
Josef
Josef, I’m still stuck on this issue. The problem isn’t with the BOOT_PART_SIZE_MB but rather with the artifacts in the bootfs directory. Our build is yielding the following artifacts:
lrwxrwxrwx 1 smartin smartin 16 Mar 9 2018 devicetree.dtb → imx8mp-owa5x.dtb
-rw-r–r-- 1 smartin smartin 1589248 Mar 9 2018 flash.bin
lrwxrwxrwx 1 smartin smartin 24 Mar 9 2018 Image → Image-5.10.72-1.0.8-rc1+
-rw-r–r-- 1 smartin smartin 31623680 Mar 9 2018 Image-5.10.72-1.0.8-rc1+
-rw-r–r-- 1 smartin smartin 74007 Mar 9 2018 imx8mp-owa5x.dtb
lrwxrwxrwx 1 smartin smartin 30 Mar 9 2018 u-boot.bin → u-boot-imx8mp_owa5x-1.0-r0.bin
lrwxrwxrwx 1 smartin smartin 30 Mar 9 2018 u-boot.bin-imx8mp_owa5x → u-boot-imx8mp_owa5x-1.0-r0.bin
-rw-r–r-- 1 smartin smartin 1167015 Mar 9 2018 u-boot-imx8mp_owa5x-1.0-r0.bin
lrwxrwxrwx 1 smartin smartin 34 Mar 9 2018 u-boot-spl.bin → u-boot-spl.bin-imx8mp_owa5x-1.0-r0
lrwxrwxrwx 1 smartin smartin 34 Mar 9 2018 u-boot-spl.bin-imx8mp_owa5x → u-boot-spl.bin-imx8mp_owa5x-1.0-r0
-rw-r–r-- 1 smartin smartin 64800 Mar 9 2018 u-boot-spl.bin-imx8mp_owa5x-1.0-r0
The Image-5.10.72-1.0.8-rc1+ artifact is 31MB by itself. Which artifacts should be in this directory?
I thought I could bypass this image generation and simply use the image generated by the vendor’s recipe. Actually, the flash.bin file is what is being used. I still applied the Mender patches by requiring recipes-bsp/u-boot/u-boot-mender.inc in the recipe. When I do so and when I attempt to run the u-boot image, I can see a few messages in the console:
U-Boot SPL 2021.04-1.0.5 (Jun 10 2022 - 06:24:02 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 3000MTS
DDRINFO: DRAM rate 3000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
Normal Boot
Trying to boot from BOOTROM
Find img info 0x&48031ba0, size 880
Need continue download 1024
Download 1227680, Total size 1228800
NOTICE: BL31: v2.4(release):lf-5.10.72-2.2.0-0-g5782363f9-dirty
NOTICE: BL31: Built : 12:17:17, Nov 18 2021
It then just hangs and doesn’t proceed any further. I don’t have these problems when I comment out the recipes-bsp/u-boot/u-boot-mender.inc line in the recipe. But then I’m failing to get the Mender patches.
1 Like
I have discovered the code which is causing the lockup but have no idea why. The vendor’s config (configs/imx8mp_owa5x_defconfig) has the following entries:
CONFIG_ENV_IS_IN_MMC=y
CONFIG_ENV_IS_IN_SPI_FLASH=y
CONFIG_ENV_IS_IN_NAND=y
CONFIG_ENV_IS_NOWHERE=y
The Mender patches removes these lines and replaces them with:
CONFIG_ENV_IS_IN_UBI=y
When I revert these changes, the bootloader is running properly. But I had to hack config_mender.h which specifically tests that CONFIG_ENV_IS_NOWHERE is not set. If I don’t set this variable, then the bootloader doesn’t start up. What are the implications of having this variable set? Why would the bootloader be hanging for if the variable isn’t set?
Hi @smartin,
Sorry for the belated response. I tried to set up a corresponding build, but for some reason the documented Owasys stuff does not even build for me.
Looking at the description, it seems that the U-Boot SPL loads, but then doesn’t manage to take the next step to U-Boot proper, right? I think it might be that the overall U-Boot configuration might need additional modifications to switch to UBI, respectively for the the SPL to properly load from NAND. Really just guessing - or is their default configuration using UBI already?
Greetz,
Josef
Josef,
There seems to be a problem with storing the u-boot environment in volumes on the ubi partition. I got around this for the time being by reverting to the Owasys strategy of storing the environment in separate partitions. It is using raw NAND instead of UBI for the environments. This is working for me now, but I am running into similar problems when it comes to launching the OS. What command should I execute from u-boot to launch the OS? Currently, I am seeing the kernel start up but am then getting the following error:
end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
My current nandboot command (which I am running to launch the OS) looks like this:
nandboot=echo Booting from nand ...; run nandargs; ubi part ubi; ubifsmount ubi0:rootfsa; ubifsload ${fdt_addr_r} /boot/${devicetreefile}; ubifsload ${loadaddr} /boot/Image; booti ${loadaddr} - ${fdt_addr_r}
My nandargs command looks like this:
nandargs=setenv bootargs console=${console} root=${mender_boot_part_name} ${optargs} rootfstype=${nandrootfstype}
I’m guessing that I have flashed the correct image to the correct partition, otherwise I wouldn’t expect the kernel to launch at all.
HI @smartin,
But that sounds like you’re really, really close. If the kernel boots and just can’t mount the root partition, that feels like you’re 90% there.
Two next steps:
- Have you checked if your boot script calls to
mender_setup?
- change
{mender_boot_part_name} to {mender_kernel_root}
For more details, see the integration documentation.
Greetz,
Josef