Issue about mender-convert from 2 different input image

[xiaoxiaoyuwts]

As we known, mender-convert can be used to convert existing disk images for use with Mender.
For more info, pls see: https://docs.mender.io/2.0/artifacts/debian-family
And the code is: git clone -b 1.1.1 https://github.com/mendersoftware/mender-convert.git

Here we use the following mender-convert cmd to generate the mender image:

DEVICE_TYPE="raspberrypi3"
# RAW_DISK_IMAGE="input/2019-04-09-raspbian-stretch-lite.img "
RAW_DISK_IMAGE="input/ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img"
ARTIFACT_NAME="test"
MENDER_DISK_IMAGE="test.sdimg"
DEMO_HOST_IP="10.80.17.256"

./docker-mender-convert from-raw-disk-image                             \
                    --raw-disk-image $RAW_DISK_IMAGE                    \
                    --mender-disk-image $MENDER_DISK_IMAGE      \
                    --device-type $DEVICE_TYPE                                 \
                    --artifact-name $ARTIFACT_NAME                          \
                    --bootloader-toolchain arm-buildroot-linux-gnueabihf              \
                    --demo                                             \
                    --demo-host-ip $DEMO_HOST_IP

Then the output test.sding image will be:

Part	Type	Purpose	Size	Configure size
mmcblk0p1	vfat	Store the bootloader	44MB	Extracted from input image
mmcblk0p2	ext4	Store the root file system and kernel (active)	3908MB	--storage-total-size-mb 1

mmcblk0p3	ext4	Store the root file system and kernel (inactive)	3908MB	--storage-total-size-mb 1

mmcblk0p4	ext4	Store persistent data, preserved during Mender updates.	128MB	--data-part-size-mb 2

Here are two kinds of input image.
The 1st input image:

root@kvm:/home/aaron/Work/mender-convert/input# fdisk -l 2019-04-09-raspbian-stretch-lite.img 
Disk 2019-04-09-raspbian-stretch-lite.img: 1.1 GiB, 1174405120 bytes, 2293760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc1dc39e5
Device                                Boot Start     End Sectors  Size Id Type
2019-04-09-raspbian-stretch-lite.img1       8192   96042   87851 42.9M  c W95 FAT32 (LBA)
2019-04-09-raspbian-stretch-lite.img2      98304 2293759 2195456    1G 83 Linux

root@kvm:/home/aaron/Work/mender-convert/input# mount -o loop,offset=4194304 2019-04-09-raspbian-stretch-lite.img /home/aaron/Mnt/
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/                           
bcm2708-rpi-0-w.dtb     bcm2709-rpi-2-b.dtb       bootcode.bin   fixup_cd.dat  issue.txt         LICENSE.oracle  start.elf
bcm2708-rpi-b.dtb       bcm2710-rpi-3-b.dtb       cmdline.txt    fixup.dat     kernel7.img       overlays        start_x.elf
bcm2708-rpi-b-plus.dtb  bcm2710-rpi-3-b-plus.dtb  config.txt     fixup_db.dat  kernel.img        start_cd.elf
bcm2708-rpi-cm.dtb      bcm2710-rpi-cm3.dtb       COPYING.linux  fixup_x.dat   LICENCE.broadcom  start_db.elf

root@kvm:/home/aaron/Work/mender-convert/input# umount /home/aaron/Mnt/
root@kvm:/home/aaron/Work/mender-convert/input# mount -o loop,offset=50331648 2019-04-09-raspbian-stretch-lite.img /home/aaron/Mnt/    
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/boot/
root@kvm:/home/aaron/Work/mender-convert/input# 
root@kvm:/home/aaron/Work/mender-convert/input# 
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/boot/
root@kvm:/home/aaron/Work/mender-convert/input# 
root@kvm:/home/aaron/Work/mender-convert/input#

The 2nd input image:

root@kvm:/home/aaron/Work/mender-convert/input# fdisk -l ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img 
Disk ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img: 1.9 GiB, 2056257536 bytes, 4016128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x85baa6c1
Device                                               Boot  Start     End Sectors  Size Id Type
ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img1 *      2048  526335  524288  256M  c W95 FAT32 (LBA)
ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img2      526336 4016127 3489792  1.7G 83 Linux
root@kvm:/home/aaron/Work/mender-convert/input# 

root@kvm:/home/aaron/Work/mender-convert/input# mount -o loop,offset=1048576 ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img /home/aaron/Mnt/
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/
bcm2710-rpi-3-b.dtb       bcm2837-rpi-3-b.dtb  broadcom     fixup_cd.dat  fixup_x.dat  overlays      start.elf
bcm2710-rpi-3-b-plus.dtb  bootcode.bin         cmdline.txt  fixup.dat     initrd.img   start_cd.elf  start_x.elf
bcm2710-rpi-cm3.dtb       boot.scr             config.txt   fixup_db.dat  kernel8.img  start_db.elf  vmlinuz

root@kvm:/home/aaron/Work/mender-convert/input# umount /home/aaron/Mnt/
root@kvm:/home/aaron/Work/mender-convert/input# mount -o loop,offset=269484032 ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img /home/aaron/Mnt/
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/    
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  snap  srv  sys  tmp  usr  var
root@kvm:/home/aaron/Work/mender-convert/input# ls /home/aaron/Mnt/boot/
config-4.15.0-1031-raspi2  grub        initrd.img-4.15.0-1031-raspi2  System.map-4.15.0-1031-raspi2  vmlinuz-4.15.0-1031-raspi2
firmware               initrd.img    initrd.img.old                 vmlinuz                        vmlinuz.old

The result:
the same mender-convert cmd to generate mender image from two different input image above,
Then the mender image from 2019-04-09-raspbian-stretch-lite.img will run and update in normal way,
But the mender image from ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img will run in abnormal way when boot, it means that the boot proc is abnormal. Why ??

Or, how to generate Mender image from ubuntu 18.04 image with mender-convert?

Edit: @mirzak: formating

[mirzak]

But the mender image from ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img will run in abnormal way when boot, it means that the boot proc is abnormal. Why ??

We have never tested converting a ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img. Would expect it to be similar to a Raspbian image, but might require some tweaks.

If you can post some logs, showing what is wrong maybe I can help debug the problem.

I would recommend testing with 1.2.0b1 version of mender-convert,

https://docs.mender.io/2.1/artifacts/debian-family#download-mender-convert

[mirzak]

You can also try with this,

New iteration of the mender-convert tool

Hi, As you might be aware of we have a tool we call mender-convert to enable support for Mender on binary distributions. We are happy that this has worked out, enabling people to securely update their devices on non-Yocto devices. Even though we are happy with the functionality of the current tool, there are a couple drawbacks in the current design of the tool, which we want to improve. We looked hard at the current code and decided, to achieve the desired design we would need to do a re-wri…

[xiaoxiaoyuwts]

I have tested with the 1.2.0b1 version of mender-convert instead of 1.1.1 version. When generating mender image from ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img, there is the same result of the boot process.

The following is the logs of generating the mender image:

*Running mender-convert from-raw-disk-image --raw-disk-image input/ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img --mender-disk-image golden-image-1-mender-integ.sdimg --device-type raspberrypi3 --artifact-name golden-image-1-mender-integ --bootloader-toolchain arm-buildroot-linux-gnueabihf --server-url 10.80.17.236 --server-cert server.crt*
**** Data partition size set to default value: 128MB ****
**** Total storage size set to default value: 8GB ****
*1/9 Shrinking raw disk image root filesystem...*
*dumpe2fs 1.44.1 (24-Mar-2018)*
*resize2fs 1.44.1 (24-Mar-2018)*
*        Root filesystem size:       *
*…*
*9/9 Installing Bootloader to Mender disk image...*
*        Building U-Boot related files.*
*        Installing U-Boot related files.*
*cp: cannot stat '/mender-convert/output/sdimg/boot/kernel7.img': No such file or directory*
*install: cannot create regular file '/mender-convert/output/sdimg/primary/usr/lib/raspi-config/init_resize.sh': No such file or directory*
*unlink: cannot unlink '/mender-convert/output/sdimg/primary/etc/rc3.d/S01resize2fs_once': No such file or directory*
*rm: cannot remove '/mender-convert/output/sdimg/primary/etc/init.d/resize2fs_once': No such file or directory*
*        Done.*
*10/9 Creating Mender Artifact...*
*        Rootfs partition id not set - 'primary' will be used by default.*
*        Storing data in golden-image-1-mender-integ.ext4.*
*        Writing Mender artifact to: /mender-convert/output/golden-image-1-mender-integ.mender*
*        This may take 10-20 minutes (using LZMA)...*

*        Creating Mender Artifact succeeded.*
*Conversion complete!*
*The Mender disk image you can provision your device storage with is at:         *
*        /mender-convert/output/golden-image-1-mender-integ.sdimg*
*The Mender root file system partition is at:*
*        /mender-convert/output/golden-image-1-mender-integ.ext4*
*The Mender Artifact you can upload to your Mender server to deploy to your devices is at:         *
*        /mender-convert/output/golden-image-1-mender-integ.mender*

The following picture shows the logs when Raspberrypi3 boots after the cmd “sudo dd if=golden-image-1-mender-integ.sdimg of=/dev/sdb bs=1M && sudo sync” :

edit: @mirzak: formating

[mirzak]

Ok, now I see what is going on. It seems that the Ubuntu image (now that I look at the name it is obvious as well), is running Raspberry Pi 3 in 64-bit mode, which is not yet supported in mender-convert.

This is also indicated by this error from your log:

cp: cannot stat '/mender-convert/output/sdimg/boot/kernel7.img': No such file or directory*

[mirzak]

I would try with,

http://cdimage.ubuntu.com/ubuntu/releases/bionic/release/ubuntu-18.04.2-preinstalled-server-armhf+raspi3.img.xz

which is the 32-bit version.

[xiaoxiaoyuwts]

Hi, I use mender-convert with the input img named “ubuntu-18.04.2-preinstalled-server-arm64+raspi3.img” and output img named “test.sdimg”, but the RaspberryPi3B run in abnormal way when boot as above described. Then I copy the right *.dtb file and boot.scr to the 1st disk of test.sdimg, and the RaspberryPi3B will boot OK. When I run the cmd “mender -daemon”, new error is shown as following:

root@ubuntu:/home/ubuntu# uname -a
Linux ubuntu 4.15.0-1031-raspi2 #33-Ubuntu SMP PREEMPT Wed Jan 16 09:52:45 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:/home/ubuntu#
root@ubuntu:/home/ubuntu# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
root@ubuntu:/home/ubuntu#
root@ubuntu:/home/ubuntu# mender -daemon
INFO[0000] Loaded configuration file: /etc/mender/mender.conf module=config
ERRO[0000] Failed to read the current active partition: No match between boot and root partitions.: exit status 1 module=main
INFO[0000] State transition: init [none] -> init [none] module=mender
INFO[0000] State transition: init [none] -> idle [Idle] module=mender
INFO[0000] State transition: idle [Idle] -> authorize-wait [Idle] module=mender
INFO[0000] State transition: authorize-wait [Idle] -> authorize [Sync] module=mender

My question is: how to read the current active partition in right way? Any other file or parameter need to be revised? Thank you for your replay.

[mirzak]

Then I copy the right *.dtb file and boot.scr to the 1st disk of test.sdimg, and the RaspberryPi3B will boot OK

This is not the right way to fix it. Yes your device has booted but it is missing a Mender integration in the bootloader. So this would need to be resolved first before moving on trying things out in user-space.

I guess best resource we have on U-boot integrations are:

• Mender from scratch
• Manual U-Boot integration | Mender documentation

[xiaoxiaoyuwts]

I have read the guide document and I want to use Yocto to build a ubuntu-18.04.2-arm64+raspi3.sdimg, the following is my output sdimg:

aaron@kvm:~/Work/mender-raspberrypi/build$ fdisk -l tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg
Disk tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg: 604 MiB, 633339904 bytes, 1236992 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa193a017

Device Boot Start End Sectors Size Id Type
tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg1 * 24576 106495 81920 40M c W95 FAT32 (LBA)
tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg2 106496 540671 434176 212M 83 Linux
tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg3 540672 974847 434176 212M 83 Linux
tmp/deploy/images/raspberrypi3/core-image-base-raspberrypi3–20190904030849.sdimg4 974848 1236991 262144 128M 83 Linux

The output image above is 32bit and the kernel is not what I wanted. How to cfg Yocto Project to generate ubuntu-18.04.2-arm64+raspi3.sdimg? Thanks.

[mirzak]

@xiaoxiaoyuwts

The output image above is 32bit and the kernel is not what I wanted. How to cfg Yocto Project to generate ubuntu-18.04.2-arm64+raspi3.sdimg?

Unfortunately this is not how the Yocto Project works. It allows you to build custom Linux distributions but it will not have anything to do with Ubuntu.