Mender raspberryPI 3 b+ (mender service failed )

[systeam]

Raspberry pi 3 b+ Yocto poky

Yocto version

thud 2.6

Mender version

1.7

Known issues

Hello everyone i integrated mender into my custom system created using Yocto thud the system is very basic the problem that i’m facing is that mender.service stopped working ,
here is the error :

systemd[1]: mender.service: Service RestartSec=100ms expired, scheduling restart.
systemd[1]: mender.service: Scheduled restart job, restart counter is at 5.
systemd[1]: Stopped Mender OTA update service.
systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39mmender.service: Start request repeated too quickly.[[0m
systemd[1]: [[0;1;39m[[0;1;31m[[0;1;39mmender.service: Failed with result ‘signal’.[[0m
systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to start Mender OTA update service.[[0m

Thank’s in advance for your solutions

[mirzak]

Hello @systeam,

Welcome to Mender Hub :).

Unfortunately the log does not say that much what could be wrong, could you try running:

mender -daemon -debug

And paste the output?

[systeam]

Hello @mirzak
Thank you so much for your replay
i executed the commend
mender -daemon -debug
i got this message : Segmentation fault
thank you

[mirzak]

Hm, that is a new one. Seems to be issues with binary compatibility.

We do have a reference integration for Raspberry Pi 3, where I do not see these problems. Maybe you can try it out and compare with your environment.

Raspberry Pi 3 Model B/B+

Board description The Raspberry Pi 3 Model B/B+ is a popular single board computer based on Broadcom SoCs. It is the most powerful board within the Raspberry Pi family and probably the most popular with “makers”. [770A5842-462x322] URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b Wiki: https://elinux.org/RPi_Hub Test results Raspberry Pi 3 is one of the official reference devices of Mender and is easy to g…

[systeam]

i already used mender 1.6 and yocto sumo it work fine , everything is ok i’ve a problem only with yocto thud and mender 1.7

[MarekBelisko]

@systeam would be possible to share build log for mender from yocto? As @mirzak mentioned if you get segmentation fault there is issue with binary compatibility. Also as reference pls use mentioned topic. Thanks.

[systeam]

here is the log of all my image but i used a grep to get only mender log

meta-mender-core     
meta-mender-raspberrypi = "thud:94cb38aae9fc5bb7ad8c2d602c61187d215238aa"
NOTE: Running setscene task 469 of 767 (virtual:native:/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact/mender-artifact_2.4.0.bb:do_populate_sysroot_setscene)
NOTE: Running setscene task 470 of 767 (virtual:native:/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact/mender-artifact_2.4.0.bb:do_populate_lic_setscene)
NOTE: recipe mender-artifact-native-2.4.0-r0: task do_populate_sysroot_setscene: Started
NOTE: recipe mender-artifact-native-2.4.0-r0: task do_populate_lic_setscene: Started
NOTE: recipe mender-artifact-native-2.4.0-r0: task do_populate_lic_setscene: Succeeded
NOTE: recipe mender-artifact-native-2.4.0-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: Running setscene task 592 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender-auto-provided_1.0.bb:do_populate_sysroot_setscene)
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_populate_sysroot_setscene: Started
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: Running setscene task 594 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender-auto-provided_1.0.bb:do_package_qa_setscene)
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_package_qa_setscene: Started
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_package_qa_setscene: Succeeded
NOTE: Running setscene task 596 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender-auto-provided_1.0.bb:do_populate_lic_setscene)
NOTE: Running setscene task 597 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact-info/mender-artifact-info.bb:do_populate_sysroot_setscene)
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_populate_lic_setscene: Started
NOTE: recipe mender-artifact-info-0.1-r0: task do_populate_sysroot_setscene: Started
NOTE: recipe mender-artifact-info-0.1-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: recipe u-boot-fw-utils-mender-auto-provided-1.0-r0: task do_populate_lic_setscene: Succeeded
NOTE: Running setscene task 598 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact-info/mender-artifact-info.bb:do_package_qa_setscene)
NOTE: Running setscene task 599 of 767 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact-info/mender-artifact-info.bb:do_populate_lic_setscene)
NOTE: recipe mender-artifact-info-0.1-r0: task do_package_qa_setscene: Started
NOTE: recipe mender-artifact-info-0.1-r0: task do_populate_lic_setscene: Started
NOTE: recipe mender-artifact-info-0.1-r0: task do_package_qa_setscene: Succeeded
NOTE: recipe mender-artifact-info-0.1-r0: task do_populate_lic_setscene: Succeeded
NOTE: Running noexec task 2345 of 2484 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact-info/mender-artifact-info.bb:do_build)
NOTE: Running noexec task 2346 of 2484 (/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-bsp/u-boot/u-boot-fw-utils-mender-auto-provided_1.0.bb:do_build)
NOTE: Running noexec task 2452 of 2484 (virtual:native:/home/hakim/sys-workspace/poky/meta-mender/meta-mender-core/recipes-mender/mender-artifact/mender-artifact_2.4.0.bb:do_build)
NOTE: Running task 2481 of 2484 (/home/hakim/sys-workspace/poky/meta/recipes-core/images/core-image-minimal.bb:do_image_mender)
NOTE: recipe core-image-minimal-1.0-r0: task do_image_mender: Started
NOTE: recipe core-image-minimal-1.0-r0: task do_image_mender: Succeeded

Edit: @mirzak: Prettify log

[mirzak]

The log does not indicate that the Mender client has been built. Do you share your Yocto environment between different boards with different architectures?

I would at least try:

bitbake -c cleansstate mender

and then re-build to make sure that the Mender client is re-built.

[systeam]

OK i will try to rebuild it , and no i don’t share yocto environment between diffrenet borads i keep them separated the only folder shared between them is the downloads folder

[systeam]

@mirzak
after executing the command

bitbake -c cleansstate mender

after that i tried to rebuild my system using the commend

bitbake core-image-minimal

i got this error

WARNING: mender-1.7.0-r0 do_package_qa: QA Issue: mender rdepends on u-boot-fw-utils, but it isn’t a build dependency, missing u-boot-fw-utils in DEPENDS or PACKAGECONFIG? [build-deps]

thanks in advance for your replay

[mirzak]

WARNING: mender-1.7.0-r0 do_package_qa: QA Issue: mender rdepends on u-boot-fw-utils, but it isn’t a build dependency, missing u-boot-fw-utils in DEPENDS or PACKAGECONFIG? [build-deps]

Ignoring this for now, as it should not have any impact on the reported error.

Did the re-build of the Mender client resolve the segfault problem?

[systeam]

Hello @mirzak
unfortunately the re-build of mender didn’t resolve the problem

[mirzak]

Can you try running this one,

Raspberry Pi 3 Model B/B+

Board description The Raspberry Pi 3 Model B/B+ is a popular single board computer based on Broadcom SoCs. It is the most powerful board within the Raspberry Pi family and probably the most popular with “makers”. [770A5842-462x322] URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus URL: https://www.raspberrypi.org/products/raspberry-pi-3-model-b Wiki: https://elinux.org/RPi_Hub Test results Raspberry Pi 3 is one of the official reference devices of Mender and is easy to g…

This might help isolate the issue further.

Also there was a similar issue reported here,

Segmentation fault while executing mender

Hello Mirzak, Thank you for replying. I am producing the output you requested, $ readelf -A /usr/bin/mender Attribute Section: aeabi File Attributes Tag_CPU_name: “7-A” Tag_CPU_arch: v7 Tag_CPU_arch_profile: Application Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-2 Tag_FP_arch: VFPv3 Tag_Advanced_SIMD_arch: NEONv1 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_rounding: Needed Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align_needed:…

But the environments are so different (different Yocto version, different hardware) and I do not think that they are related but wanted to link anyway

[systeam]

Hi @mirzak

finally it’s work , i used google repo as in the tutorial , everything works fine now , and i think i found the real problem , when i compile my distribution using musl ,

TCLIBC=“musl”

i get the error Segmentation fault, is there any relation between c libraries an mender. because when i use glibc like i said everything works fine

and thank you so much @mirzak for all your replies

[mirzak]

Thank you @systeam for reporting back your findings. This will definitely be helpful to our community.

is there any relation between c libraries an mender

We need to investigate this as we have not much testing using musl libc. This could be a golang thing or yocto + muslc + mender. Quick look does hint that muls libc should work with golang.

I have created a bug on tracker,

https://tracker.mender.io/browse/MEN-2443

Where you can follow progress, and feel free to add any comments or additional information.

[systeam]

Thank you @mirzak for all the exchanged information it’s helped me a lot in my project and working with mender is real pleasure it’s a very good solution for OTA update , and thank you for creating bug on tracker.

[lluiscampos]

Hi @systeam

I’ve been investigating a bit your issue. I am able to reproduce the error when using musl libc, so it is clear that it is not a problem on your setup only.

To make the story short: there must be some of our C/C++ code in the mender client that triggers this segfault. However, disabling the PIC option on gcc seems to work.

Try adding these lines on your local.conf:

SECURITY_CFLAGS_pn-mender = "${SECURITY_NOPIE_CFLAGS}"
SECURITY_LDFLAGS_pn-mender = "" 

Please report back if it works

[systeam]

Hello @lluiscampos @mirzak

I tried what you told me to do i mean adding

SECURITY_CFLAGS_pn-mender = “${SECURITY_NOPIE_CFLAGS}”
SECURITY_LDFLAGS_pn-mender = “”

to my local.conf and finally it’s work no segmentation fault message at all and the service is running

thank you so much for your replay it’s helped me a lot

[lluiscampos]

You are welcome, it was fun debugging it.

This is just a workaround. We will keep the ticket open in Mender and try to find the root cause that triggers the bug in musl runtime.

[systeam]

Thank you so much @lluiscampos for keeping the ticket open