# Mender-monitor does not work

**URL:** <https://hub.mender.io/t/mender-monitor-does-not-work/4364>\
**Category:** General Discussions\
**Created:** [December 1, 2021, 9:09am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364 "2021-12-01T09:09:29Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![mfloiot](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mfloiot/32/1438_2.png) [@mfloiot](https://hub.mender.io/u/mfloiot)\
**Post date:** [December 1, 2021, 9:09am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/1 "2021-12-01T09:09:29Z")

</div>

Installing mender-monitor by the cmdline provieded by the “Conncet a device” wizard does not work as expected.  
The cmdline installs the mender-monitor version:  
mender-monitor is already the newest version (1.0.0-1).

The systemd service always reports the following failure.  
Nov 23 08:24:39 iot2050-debian mender-monitord[177828]: mdb\_env\_open failed, error 2 No such file or directory

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 9:12am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/2 "2021-12-01T09:12:35Z")

</div>

Hi @mfloiot

The error can be ignored, and is wrongly reported, this is just a result of accessing an empty database.

So does it still not work for you?

Like does the systemd service die, and the monitor service is restarted in a loop?

---

<div class="post-metadata">

**Author:** ![mfloiot](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mfloiot/32/1438_2.png) [@mfloiot](https://hub.mender.io/u/mfloiot)\
**Post date:** [December 1, 2021, 9:14am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/3 "2021-12-01T09:14:52Z")

</div>

I would have expected some monitor items (like CPU usage). But I do not see any monitoring data in mender.  
Is there tutorial for that feature?

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 9:21am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/4 "2021-12-01T09:21:53Z")

</div>

There are the docs:

> [@Mender 3.1 release: Monitor using edge processing | Mender](https://hub.mender.io/t/mender-3-1-release-monitor-using-edge-processing-mender/4128):
>
> We are excited to announce the Mender 3.1 release! Mender 3.1 introduces the brand new Monitor add-on package. Unlike traditional monitoring tools, Mender’s Monitor add-on is built for connected devices. It uses edge processing, is designed for intermittent connectivity and requires much less bandwidth. This is a companion discussion topic for the original entry at [https://mender.io/blog/mender-3-1-release-monitor-using-edge-processing](https://mender.io/blog/mender-3-1-release-monitor-using-edge-processing)

[https://docs.mender.io/add-ons/monitor](https://docs.mender.io/add-ons/monitor)

Which shows you how to use the currently supported monitoring subsystems:

- log
- services
- dbus

However, it does not seem we have any good tutorials for writing custom subsystems present (this should be amended imo):

In the meantime, I am actually writing some demo code that we are about to ship in our demo setup, which will be some variation of [this](https://github.com/mendersoftware/monitor-client/pull/99) Pr.

Here you can see how I have created a `docker events` subsystem, and added a log-monitor for `USB disconnects`.

Have a look at this, and tell me if it is helpful or not. I will help you out here, then you can help me out improving our documentation on the subject for future users 😃

---

<div class="post-metadata">

**Author:** ![mfloiot](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mfloiot/32/1438_2.png) [@mfloiot](https://hub.mender.io/u/mfloiot)\
**Post date:** [December 1, 2021, 11:21am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/5 "2021-12-01T11:21:45Z")

</div>

Do I need to configure the device in some manner to collect data?  
I do not see any hint in docs.

[https://docs.mender.io/add-ons/monitor/30.Advanced-configuration/docs.md](https://docs.mender.io/add-ons/monitor/30.Advanced-configuration/docs.md) gives an error 404

[https://github.com/mendersoftware/monitor-client/pull/99](https://github.com/mendersoftware/monitor-client/pull/99) gives an error 404

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 12:02pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/6 "2021-12-01T12:02:34Z")

</div>

Oh, sorry, github private repository, I’ll share the code with you here in the next post.

Doc link:  
[https://docs.mender.io/add-ons/monitor/advanced-configuration](https://docs.mender.io/add-ons/monitor/advanced-configuration)

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 12:24pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/7 "2021-12-01T12:24:58Z")

</div>

So short tutorial:

Mender monitor (monitor client) is a Mender add-on, which is loosely coupled  
with the Mender client to provide monitoring of device events related to three  
subsystems:

- D-Bus
- Logs
- init system services (systemd)

The daemon is designed to be a lightweight edge processing oriented monitoring solution, which

The functionality is mostly tied up into three files:

## mender-monitorctl

`mender-monitorctl` provides the user interface for enabling/disabling monitor  
services, run `mender-monitorctl -h` for more information.

## mender-monitord

`mender-monitord` is the main entrance to the monitor client, and is essentially  
a wrapper for the `daemon.sh` script

## daemon.sh

This is an implementation of a monitoring client which periodically calls all  
the `enabled/*.sh` files from the `monitor.d` directory.

Each file that exists in the `enabled` directory is included (sourced) to set  
the environment for the call to the implementation of the monitoring. The first  
element of the file name, using `_` as separator, determines the monitoring  
subsystem. In the above example of `enabled/service_menderconnect.sh`, the file  
will be sourced and `monitor.d/service.sh` will be called to perform the  
monitoring.

### Configuration

See `src/config/config.sh`.

# Functionality

The monitor daemon functions as follows. To enable monitoring of a log file,  
D-Bus bus or systemd service, create an appropriate shell file in the enabled  
directory. Then install it with the `mender-monitorctl` utility, which then  
automatically enables files in the available directory through a symlink, which  
should be familiar to users of Unix like utilites. Example:

```auto
#tree /etc/mender-monitor/monitor.d/
/etc/mender-monitor/monitor.d
├── available
│ ├── log_auth.sh
│ ├── log_messages.sh
│ ├── service_apache.sh
│ ├── service_cron.sh
│ ├── service_sshd.sh
│ └── sshd_service.sh
├── enabled
│ ├── log_auth.sh -> /etc/mender-monitor/monitor.d/available/log_auth.sh
│ ├── log_messages.sh -> /etc/mender-monitor/monitor.d/available/log_messages.sh
│ ├── service_apache.sh -> /etc/mender-monitor/monitor.d/available/service_apache.sh
│ ├── service_cron.sh -> /etc/mender-monitor/monitor.d/available/service_cron.sh
│ └── service_sshd.sh -> /etc/mender-monitor/monitor.d/available/service_sshd.sh
├── log.sh
└── service.sh

2 directories, 15 files

```

where for instance the `cron` monitoring entry in the `monitor.d`  
directory is:

```bash
#cat /tmp/etc/mender-monitor/monitor.d/enabled/service_cron.sh
# This file was autogenerated by Monitoring daemon based on the configuration
SERVICE_NAME=cron
SERVICE_TYPE=systemd

```

Support for `interval_s` is coming, currently all services share 1s sleep interval  
between runs.

# The three sub-systems supported

Currently the monitor client supports three different sub-systems.

## Log file subsystem

You can configure log files monitoring in a similar way as with the systemd services.  
Let’s assume you want to raise an alert every time _session opened for user root by_  
appears in `/var/log/auth.log` file. To this end you need to create the enable file  
and use `log.sh` subsystem. The file can look like this:

```bash
SERVICE_NAME="auth"
LOG_PATTERN="session opened for user root by"
LOG_FILE="/var/log/auth.log"

```

With each iteration of the main loop `daemon.sh` will call `log.sh` to check  
for a given pattern in a given file. The pattern is interpreted  
as a grep extended regular expression.

## systemd services subsystem

To enable systemd service monitoring all you need to do is to enable the service.sh monitor  
for a given service. For example, to enable monitoring of cron systemd:

```bash
echo -ne 'SERVICE_NAME="cron"\nSERVICE_TYPE="systemd"' > monitor.d/enabled/service_cron.sh;

```

The “enable” directory file names format is: `service_"name-of-the-service".sh"`.  
The name of the service in the file name is arbitrary and does not have any other meaning  
than just to distinguish the file in the `monitor.d/enabled` directory. The files are read  
by the `daemon.sh`.

## D-Bus subsystem

You can configure D-Bus monitoring in a similar way as with the systemd  
services, and log files. Let’s assume you want to raise an alert every time  
`u-power` raises a signal. To this end you need to create the enable file and use  
the `dbus.sh` subsystem. The file can look like this:

```bash
DBUS_NAME="u-power"
DBUS_PATTERN="some pattern to match on | (empty to match on all notifications)"
DBUS_WATCH_EXPRESSION="type='signal',interface='org.freedesktop.DBus.Properties',member='PropertiesChanged',path=/org/freedesktop/UPower/devices/battery_BAT0 | (empty to match on all interfaces)"

```

# Shell API for the Monitoring backend

`monitor-lib.sh` provides a single call to send an alert via the backend  
with the help of `monitor_send_alert`. You can find an example usage  
in the `monitor.d/service.sh`.

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 12:29pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/8 "2021-12-01T12:29:39Z")

</div>

I recently wrote this monitor subsystem for disk space:

```auto
#!/usr/bin/env bash
#
# Copyright 2021 Northern.tech AS
#
# All Rights Reserved
#
#
# Monitor the disk space of a given disk.
#
#
# More specifically
#
# DISK_NAME=<some name>
# DISK_TRESHOLD=<1-100> (default: 80)
#

. common/common.sh
. lib/monitor-lib.sh
. lib/alert-lib.sh

log_debug "STATUS_BY_SERVICE_NAME[${SERVICE_NAME}]=${STATUS_BY_SERVICE_NAME[${SERVICE_NAME}]}"

alert_log_is_set "service_$SERVICE_NAME" || eval declare -A ts2alert_${service_identifier}

#
# Parse the input file
#
if [[-z "${DISK_NAME}"]]; then
    log_error "DISK_NAME not set, this is an error."
    exit 0
fi

function disk_usage() {
    df --output=pcent ${DISK_NAME} | tail -1 | cut -d% -f1
}

if [[$(disk_usage) > ${DISK_TRESHOLD:-80}]]; then
    log_debug "Disk storage has grown to fill more than ${DISK_TRESHOLD:-80}% of the disk" \
              monitor_send_alert \
              CRITICAL \
              "Disk storage has grown to fill more than ${DISK_TRESHOLD:-80}% of the disk" \
              "Disk ${DISK_NAME} is filling up" \
              "${DISK_NAME}" \
              DISK_USAGE_WARNING \
              "disk"
fi

```

Which, if added to `/etc/mender-monitor/monitor.d`, allows you to create a monitor for disk usage, by adding a file to `/etc/mender-monitor/monitor.d/available` with contents similar to:

```auto
# Monitor the whole rootfs
DISK_NAME="/"
# Report on 3/4 full disk
DISK_TRESHOLD=75

```

Then enable it with a symlink from `/etc/mender-monitor/monitor.d/available` → `/etc/mender-monitor/monitor.d/enabled`

---

<div class="post-metadata">

**Author:** ![mfloiot](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/mfloiot/32/1438_2.png) [@mfloiot](https://hub.mender.io/u/mfloiot)\
**Post date:** [December 1, 2021, 1:46pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/9 "2021-12-01T13:46:17Z")

</div>

Adding:  
echo -ne ‘SERVICE\_NAME=“cron”\nSERVICE\_TYPE=“systemd”’ \> monitor.d/enabled/service\_cron.sh;  
causes the error:  
**Service cron not running** CRITICAL 2021-12-01 14:33view details

But I will stop investigating it and wait for sufficient documentation or some ready to use examples.

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [December 1, 2021, 2:13pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/10 "2021-12-01T14:13:03Z")

</div>

> [@mfloiot](#):
>
> causes the error:  
> **Service cron not running** CRITICAL 2021-12-01 14:33view details

This actually means that an alarm should show up in your UI, notifying you that cron is not in fact running, which I think you can verify with:

```auto
systemctl status cron.service

```

---

<div class="post-metadata">

**Author:** ![marizonne.villafranc](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@marizonne.villafranc](https://hub.mender.io/u/marizonne.villafranc)\
**Post date:** [February 10, 2022, 6:49am UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/11 "2022-02-10T06:49:10Z")

</div>

Can I install this mender-monitor in my client devices and still send alerts to self-hosted mender server?

---

<div class="post-metadata">

**Author:** ![oleorhagen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/oleorhagen/32/297_2.png) [@oleorhagen](https://hub.mender.io/u/oleorhagen)\
**Post date:** [March 8, 2022, 6:46pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/12 "2022-03-08T18:46:40Z")

</div>

yes, there is nothing keeping you from doing this.

It is an `enterprise` feature though. So self-hosted still needs to be `Enterprise`.

---

<div class="post-metadata">

**Author:** ![marizonne.villafranc](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@marizonne.villafranc](https://hub.mender.io/u/marizonne.villafranc)\
**Post date:** [March 8, 2022, 11:37pm UTC](https://hub.mender.io/t/mender-monitor-does-not-work/4364/13 "2022-03-08T23:37:52Z")

</div>

Ok. Thank you!