Mender-monitor and intended shutdowns

General Discussions
1

I’m trying to use mender-monitor to determine when some services are running or not. I’m seeing monitor triggers both at initial boot and on reboot, which feels like a race condition between intentional service shutdowns and mender-monitor detection. Is there a way to have mender-monitor ignore these service shutdowns as intentional? In my case I’m using “shutdown -r now” to force the reboot so it seems like it should be possible to detect this situation and not trigger an alert.

Drew

2

@lluiscampos @kacf any thoughts on this?

3

@merlin: Maybe this could be detected by calling something like systemctl is-system-running and see if it is stopping, before considering the alert triggered?

4

Interesting. I didn’t know that was a command. I’ll experiment with it.

5

OK. Changing /usr/share/mender-monitor/lib/service-lib.sh to the following seems to work for me. At least in the limit of my minimal testing. I don’t know if there is an equivalent for sysvinit.

Drew

#!/usr/bin/env bash
#
# Copyright 2021 Northern.tech AS
#
#    All Rights Reserved

function service_systemd_is_running() {
    case $(/bin/systemctl is-system-running) in
        running | degraded )
            # In these states we can go ahead and check service status 
            systemctl is-active --quiet "$1"
            ;;
        * )
            # In any other state, the service status is not necessarily correct
            # so we will just return true to bypass the check for now.
            true
            ;;
    esac
}

function service_sysv_is_running() {
    /etc/init.d/${1} status > /dev/null 2>&1
}
6

Thanks @drewmoseley! I have submitted this internally, I’ll try to get this into the next release!

7

looks good, we are already discussing in the PR

pg

8

this is all strange, @drewmoseley do you see alerts coming in on every shutdown? we do have an explicit check for that: monitor.d/service.sh:14 where we check if daemon is not in the process of shutting down. it however relies on the SIGTERM being delivered to the mender-monitord, is it possible that your systemd services are being shutdown first?

peter

9

Hi @peter

I’m pretty sure this happens every time. I don’t see any such check in my service.sh file. Line 14 is SERVICE_RC_OK=0. Maybe I have an older version? I am using mender-monitor-1.1.0.tar.gz as downloaded from here.

Drew

10

Hey Drew,

ok, I am sorry, it has been fixed in master, and not yet released.
it will be part of the next release. are you able to access master?

peter

11

I cannot access that but I don’t really need it right now. I have just patched the file locally in my Yocto configuration so I have it working. I’ll take a look when the next release comes out and revert my local change.

Drew