Mender-convert: trouble booting raspbian

[genedupre]

Hi!
first of i would like to introduce myself since i’m new here.
i’m gene from belgium and iam following a bach in IT atm, i’m working as a student for a IT company and they gave me mender as an idea to help maintain raspbian pi’s for a client.
And so iam to make a proof of concept concerning mender, for if we will use mender, to help maintain the pi’s.

i’ve followed the documentation to get a demo mender-server running with some virtual devices & real pi’s.
The problem arises when i want to use mender-convert on a production image and get this device to work with the mender-server.
After using mender-convert and dd the .sdimg image on the sd card, the pi simply does not start, no boot message or anything, just the 4 raspberry logo’s.

by doing a fdisk, i can clearly see the 4 partitions

Device     Boot    Start      End Sectors  Size Id Type
/dev/sdb1  *       24576   155647  131072   64M  c W95 FAT32 (LBA)
/dev/sdb2         155648  8134655 7979008  3.8G 83 Linux
/dev/sdb3        8134656 16113663 7979008  3.8G 83 Linux
/dev/sdb4       16113664 16375807  262144  128M 83 Linux

these where the options i used for mender-convert

DEVICE_TYPE="raspberrypi3"
ARTIFACT_NAME="2018-11-13-raspbian-stretch"
RAW_DISK_IMAGE="input/2016-11-25-raspbian-jessie-lite.img"
ARTIFACT_NAME="2019-07-03-raspbian-jessie-lite"
MENDER_DISK_IMAGE="3-7-19_rapbian-jessie-lite.sdimg"
IP="192.168.20.1"

./docker-mender-convert from-raw-disk-image 
--raw-disk-image $RAW_DISK_IMAGE 
--mender-disk-image $MENDER_DISK_IMAGE 
--device-type $DEVICE_TYPE 
--artifact-name $ARTIFACT_NAME 
--bootloader-toolchain arm-buildroot-linux-gnueabihf 
--demo-host-ip $IP

the mender-convert log can be found here
mender-convert-output-log

(the raspbian-jessie image i use, works ofcourse without the conversion)

thanks in advance!

[mirzak]

Hi @genedupre,

Welcome to Mender Hub!

The fact that you get 4 raspberrypis on you screen actually hints that it got past the bootloader and is booting the Linux kernel. So something must be going wrong during Linux kernel or user-space boot.

Can you please share which version of mender-convert you use, and what type of Raspberry Pi board you are trying to boot on.

The image you are trying to convert is a bit older as not part of our “tested images”, so there might be issues.

You can see which we have tested here,

Raspberry Pi 3 Model B/B+ Raspbian

Board description The Raspberry Pi 3 Model B/B+ is a popular single board computer based on Broadcom SoCs. It is the most powerful board within the Raspberry Pi family and probably the most popular with “makers”. [770A5842-462x322] URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b Wiki: https://elinux.org/RPi_Hub Test results The Raspberry Pi OS releases in the table below have been tested by the Mender…

Can you try just using one of the tested images to verify the process.

[genedupre]

Hello @mirzak, thank you for your answer!
i use the raspberry 3 model B,
as for the the mender-convert version, it seem to give an error asking for it, most been something wrong here

but it was the latest pull for the git repo on as from 5-07-19 which is probably still version 1.1.x

This time i followed the raspberry-pi-3-model-b-raspbian guide, and tried to get a bare-raspbian_buster_lite-client running, but this also was no success for me.

after downloading the fresh .img & schrinking it with raw-disk-image-schrink
i used the mender-converter with these options

DEVICE_TYPE="raspberrypi3"
ARTIFACT_NAME="2019-07-08-raspbian-buster"
RAW_DISK_IMAGE="input/2019-06-20-raspbian-buster-lite.img"
ARTIFACT_NAME="2019-07-08-raspbian-buster-lite"
MENDER_DISK_IMAGE="8-7-19_rapbian-buster-lite.sdimg"
IP="192.168.30.11"

and

./docker-mender-convert from-raw-disk-image --raw-disk-image $RAW_DISK_IMAGE --storage-total-size-mb 3040 --mender-disk-image $MENDER_DISK_IMAGE --device-type $DEVICE_TYPE --artifact-name $ARTIFACT_NAME --bootloader-toolchain arm-buildroot-linux-gnueabihf --demo --demo-host-ip $IP

i"ve put a --storage-total-size-mb 3040 (minimum required according to the mender-converter) because the default 8000mb did not fit on the 8gb sd card that came with it! (its 7.4gb when partitioned)

after putting the .sdimg with an dd as described in provisioning-a-new-device

This time, the pi booted up, showed the colorful screen, then the 4 raspberrys for like 0.5 second went black for a while, then rebooted again the the colorful screen and 0.5sec and then nothing.

so probably it found the boot, went into kernel and got stuck

the log can be found here build.log

thanks in advance! & i enjoy reading in the community

[mirzak]

@genedupre

This time, the pi booted up, showed the colorful screen, then the 4 raspberrys for like 0.5 second went black for a while, then rebooted again the the colorful screen and 0.5sec and then nothing.

This is expected behavior until the “nothing” . Because at first boot it attempts to expand the data partition and that is why there is one extra boot.

For the version can you just run:

git log 

In the mender-convert directory. This does sound similar to this issue,

https://hub.mender.io/t/raspberry-pi-3-model-b-b-raspbian/140/26?u=mirzak

Which was resolved by updating to the latest version of the 1.1.x

[genedupre]

hi @mirzak
thanks for the answer i was not aware of this.
With a git log & git pull, was i sure that i was using the latest version.

i used a Raspbian Stretch 2019-04-08 img this time with

DEVICE_TYPE=“raspberrypi3”
ARTIFACT_NAME=“2019-09-08-raspbian-buster”
RAW_DISK_IMAGE=“input/2019-04-08-raspbian-stretch-lite.img”
ARTIFACT_NAME=“2019-09-08-raspbian-stretch-lite”
MENDER_DISK_IMAGE=“9-7-19_rapbian-stretch-lite.sdimg”
IP=“192.168.30.11”

and

./docker-mender-convert from-raw-disk-image --raw-disk-image $RAW_DISK_IMAGE --storage-total-size-mb 3600 --mender-disk-image $MENDER_DISK_IMAGE --device-type $DEVICE_TYPE --artifact-name $ARTIFACT_NAME --bootloader-toolchain arm-buildroot-linux-gnueabihf --demo --demo-host-ip $IP

i was probably using to less storage space the previous time, so now i maxed it out for the 7.4gb sd-card

This time i got the pi with mender-client working, its not showing up in pending devices though, after a systemctl status i was informed that the certificates where not in order, nevertheless my first reaction was the clock, which where indeed not correct on both client & server.
After getting the clocks straight on both devices it still seems to ask for a certificate, would it be possible to temporary disable this behaviour? Im not that familiar yet with API’s, and would need some time to know how to use it as described in the chapter 8 & 9 of the documentation.

the log of the systemctl for client can be found here systemctl mender log
As always thanks in advance!

[mirzak]

After getting the clocks straight on both devices it still seems to ask for a certificate, would it be possible to temporary disable this behaviour?

I am not aware of a way of disabling HTTPS (which is why certificates are needed)

Can you share the content of /etc/mender/mender.conf and /etc/hosts?

[genedupre]

hi @mirzak
the content of mender.conf & hosts are:

root@raspberrypi:/etc/mender# cat mender.conf
{
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“ServerURL”: “https://docker.mender.io”,
“TenantToken”: “Paste your Hosted Mender token here”,
“UpdatePollIntervalSeconds”: 5,
“RootfsPartA”: “/dev/mmcblk0p2”,
“RootfsPartB”: “/dev/mmcblk0p3”
}
root@raspberrypi:/etc/mender# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

127.0.1.1 raspberrypi
192.168.30.11 docker.mender.io s3.docker.mender.io
root@raspberrypi:/etc/mender#

the device is ofcourse able to ping to the server, there is just routing between the two, no firewall.

[mirzak]

Hm it seems that mender.conf is missing this entry:

"ServerCertificate": "/etc/mender/server.crt"

[genedupre]

hi @mirzak
yes this fixed the situation, i could successfully add the device now,
Thanks!

[genedupre]

----- to be added on why mender-clients could not authenticate to the mender-server -----

After making new .sdimg’s & .mender’s (artifacts) this problem seems to keep reoccurring.
And thus also prevented my working mender-clients from completing the mender-artifact i push via an deployment.
Here u can see the missing path inside the artifact’s mender.conf

root@ubuntu:/home/gene/Downloads/mender_artifact# ./mender-artifact cat input/10-7-19_rapbian-stretch-lite-artifact-2-test.mender:/etc/mender/mender.conf
{
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“ServerURL”: “https://docker.mender.io”,
“TenantToken”: “Paste your Hosted Mender token here”,
“UpdatePollIntervalSeconds”: 5,
“RootfsPartA”: “/dev/mmcblk0p2”,
“RootfsPartB”: “/dev/mmcblk0p3”
}

U can just use the mender ./artifact to add the missing cert path using a correct mender.conf for example

{
“InventoryPollIntervalSeconds”: 5,
“RetryPollIntervalSeconds”: 30,
“ServerURL”: “https://docker.mender.io”,
“ServerCertificate”: “/etc/mender/server.crt”,
“TenantToken”: “Paste your Hosted Mender token here”,
“UpdatePollIntervalSeconds”: 5,
“RootfsPartA”: “/dev/mmcblk0p2”,
“RootfsPartB”: “/dev/mmcblk0p3”
}

and by using

./mender-artifact cp /home/gene/Downloads/working-example-mender.conf input/10-7-19_rapbian-stretch-lite-artifact-2-test.mender:/etc/mender/mender.conf

to replace the faulty mender.conf inside the artifact

[mirzak]

This is probably a bug in the mender-convert tool, as it should be updating the mender.conf accordingly so that the output artifacts are valid.

[genedupre]

hi @mirzak
would it also be possible to modify a .sdimg?
so i dont have to manually edit each new pi that has this problem

i have tried to mount the .sdimg with offset but this does not seem to work

Device Boot Start End Sectors Size Id Type
/home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg1 * 24576 114687 90112 44M c W95 FAT32 (LBA)
/home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg2 114688 3612671 3497984 1.7G 83 Linux
/home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg3 3612672 7110655 3497984 1.7G 83 Linux
/home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg4 7110656 7372799 262144 128M 83 Linux

root@ubuntu:/home/gene/Downloads/mender_artifact# mount -o loop,offset=114688 /home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg /tmp/artifact/
mount: /tmp/artifact: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.

Thanks!

[mirzak]

I have created a bug for this,

https://tracker.mender.io/browse/MEN-2640

mount -o loop,offset=114688 /home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg /tmp/artifact/

I think this should be,

mount -o loop,offset=$((114688*512)) /home/gene/Documents/working-mender-client_raspbian-stretch-lite_V1_09-07-19/9-7-19_rapbian-stretch-lite.sdimg /tmp/artifact/

[mirzak]

Also mender-artifact cp should work on sdimg as well.

[genedupre]

yes, using the ./mender-artifact to modify a .sdimg also works like a charm!

[ster]

@mirzak

This is probably a bug in the mender-convert tool, as it should be updating the mender.conf accordingly so that the output artifacts are valid.

By the way I noticed this as well, the necessary line with the path to .cert file is not getting added to mender.conf, which was confusing and made me spend some time debugging this. I ended up changing the code of the script in mender-convert responsible for this, I should probably create a PR

[MarekBelisko]

@ster pls create it we can review and merge. Thanks.

[ster]

here: https://github.com/mendersoftware/mender-convert/pull/104

[agniiyer]

Hi @mirzak can you explain how this correction can be made on a Mac?

I have added “ServerCertificate”: “/etc/mender/server.crt”, “RootfsPartA”: “/dev/mmcblk0p2” and “RootfsPartB”: “/dev/mmcblk0p3” which was missing in my mender.conf file. I now need to update the artifact and image with this new conf file.

I tried using mender-artifact cp rootfs_overlay_demo/etc/mender/mender.conf deploy/golden-image-1-raspberrypi4-mender.mender:/etc/mender/mender.conf to update the artifact but got the following error:

fsck command not found: exec: “/usr/local/sbin/fsck.ext4”: stat /usr/local/sbin/fsck.ext4: no such file or directory