Installing mender-monitor by the cmdline provieded by the “Conncet a device” wizard does not work as expected.
The cmdline installs the mender-monitor version:
mender-monitor is already the newest version (1.0.0-1).
The systemd service always reports the following failure.
Nov 23 08:24:39 iot2050-debian mender-monitord[177828]: mdb_env_open failed, error 2 No such file or directory
Hi @mfloiot
The error can be ignored, and is wrongly reported, this is just a result of accessing an empty database.
So does it still not work for you?
Like does the systemd service die, and the monitor service is restarted in a loop?
I would have expected some monitor items (like CPU usage). But I do not see any monitoring data in mender.
Is there tutorial for that feature?
There are the docs:
https://docs.mender.io/add-ons/monitor
Which shows you how to use the currently supported monitoring subsystems:
However, it does not seem we have any good tutorials for writing custom subsystems present (this should be amended imo):
In the meantime, I am actually writing some demo code that we are about to ship in our demo setup, which will be some variation of this Pr.
Here you can see how I have created a docker events subsystem, and added a log-monitor for USB disconnects.
Have a look at this, and tell me if it is helpful or not. I will help you out here, then you can help me out improving our documentation on the subject for future users 
Do I need to configure the device in some manner to collect data?
I do not see any hint in docs.
https://docs.mender.io/add-ons/monitor/30.Advanced-configuration/docs.md gives an error 404
https://github.com/mendersoftware/monitor-client/pull/99 gives an error 404
Oh, sorry, github private repository, I’ll share the code with you here in the next post.
Doc link:
https://docs.mender.io/add-ons/monitor/advanced-configuration
So short tutorial:
Mender monitor (monitor client) is a Mender add-on, which is loosely coupled
with the Mender client to provide monitoring of device events related to three
subsystems:
The daemon is designed to be a lightweight edge processing oriented monitoring solution, which
The functionality is mostly tied up into three files:
mender-monitorctl provides the user interface for enabling/disabling monitor
services, run mender-monitorctl -h for more information.
mender-monitord is the main entrance to the monitor client, and is essentially
a wrapper for the daemon.sh script
This is an implementation of a monitoring client which periodically calls all
the enabled/*.sh files from the monitor.d directory.
Each file that exists in the enabled directory is included (sourced) to set
the environment for the call to the implementation of the monitoring. The first
element of the file name, using _ as separator, determines the monitoring
subsystem. In the above example of enabled/service_menderconnect.sh, the file
will be sourced and monitor.d/service.sh will be called to perform the
monitoring.
See src/config/config.sh.
The monitor daemon functions as follows. To enable monitoring of a log file,
D-Bus bus or systemd service, create an appropriate shell file in the enabled
directory. Then install it with the mender-monitorctl utility, which then
automatically enables files in the available directory through a symlink, which
should be familiar to users of Unix like utilites. Example:
#tree /etc/mender-monitor/monitor.d/
/etc/mender-monitor/monitor.d
├── available
│ ├── log_auth.sh
│ ├── log_messages.sh
│ ├── service_apache.sh
│ ├── service_cron.sh
│ ├── service_sshd.sh
│ └── sshd_service.sh
├── enabled
│ ├── log_auth.sh -> /etc/mender-monitor/monitor.d/available/log_auth.sh
│ ├── log_messages.sh -> /etc/mender-monitor/monitor.d/available/log_messages.sh
│ ├── service_apache.sh -> /etc/mender-monitor/monitor.d/available/service_apache.sh
│ ├── service_cron.sh -> /etc/mender-monitor/monitor.d/available/service_cron.sh
│ └── service_sshd.sh -> /etc/mender-monitor/monitor.d/available/service_sshd.sh
├── log.sh
└── service.sh
2 directories, 15 files
where for instance the cron monitoring entry in the monitor.d
directory is:
#cat /tmp/etc/mender-monitor/monitor.d/enabled/service_cron.sh
# This file was autogenerated by Monitoring daemon based on the configuration
SERVICE_NAME=cron
SERVICE_TYPE=systemd
Support for interval_s is coming, currently all services share 1s sleep interval
between runs.
Currently the monitor client supports three different sub-systems.
You can configure log files monitoring in a similar way as with the systemd services.
Let’s assume you want to raise an alert every time session opened for user root by
appears in /var/log/auth.log file. To this end you need to create the enable file
and use log.sh subsystem. The file can look like this:
SERVICE_NAME="auth"
LOG_PATTERN="session opened for user root by"
LOG_FILE="/var/log/auth.log"
With each iteration of the main loop daemon.sh will call log.sh to check
for a given pattern in a given file. The pattern is interpreted
as a grep extended regular expression.
To enable systemd service monitoring all you need to do is to enable the service.sh monitor
for a given service. For example, to enable monitoring of cron systemd:
echo -ne 'SERVICE_NAME="cron"\nSERVICE_TYPE="systemd"' > monitor.d/enabled/service_cron.sh;
The “enable” directory file names format is: service_"name-of-the-service".sh".
The name of the service in the file name is arbitrary and does not have any other meaning
than just to distinguish the file in the monitor.d/enabled directory. The files are read
by the daemon.sh.
You can configure D-Bus monitoring in a similar way as with the systemd
services, and log files. Let’s assume you want to raise an alert every time
u-power raises a signal. To this end you need to create the enable file and use
the dbus.sh subsystem. The file can look like this:
DBUS_NAME="u-power"
DBUS_PATTERN="some pattern to match on | (empty to match on all notifications)"
DBUS_WATCH_EXPRESSION="type='signal',interface='org.freedesktop.DBus.Properties',member='PropertiesChanged',path=/org/freedesktop/UPower/devices/battery_BAT0 | (empty to match on all interfaces)"
monitor-lib.sh provides a single call to send an alert via the backend
with the help of monitor_send_alert. You can find an example usage
in the monitor.d/service.sh.
I recently wrote this monitor subsystem for disk space:
#!/usr/bin/env bash
#
# Copyright 2021 Northern.tech AS
#
# All Rights Reserved
#
#
# Monitor the disk space of a given disk.
#
#
# More specifically
#
# DISK_NAME=<some name>
# DISK_TRESHOLD=<1-100> (default: 80)
#
. common/common.sh
. lib/monitor-lib.sh
. lib/alert-lib.sh
log_debug "STATUS_BY_SERVICE_NAME[${SERVICE_NAME}]=${STATUS_BY_SERVICE_NAME[${SERVICE_NAME}]}"
alert_log_is_set "service_$SERVICE_NAME" || eval declare -A ts2alert_${service_identifier}
#
# Parse the input file
#
if [[ -z "${DISK_NAME}" ]]; then
log_error "DISK_NAME not set, this is an error."
exit 0
fi
function disk_usage() {
df --output=pcent ${DISK_NAME} | tail -1 | cut -d% -f1
}
if [[ $(disk_usage) > ${DISK_TRESHOLD:-80} ]]; then
log_debug "Disk storage has grown to fill more than ${DISK_TRESHOLD:-80}% of the disk" \
monitor_send_alert \
CRITICAL \
"Disk storage has grown to fill more than ${DISK_TRESHOLD:-80}% of the disk" \
"Disk ${DISK_NAME} is filling up" \
"${DISK_NAME}" \
DISK_USAGE_WARNING \
"disk"
fi
Which, if added to /etc/mender-monitor/monitor.d, allows you to create a monitor for disk usage, by adding a file to /etc/mender-monitor/monitor.d/available with contents similar to:
# Monitor the whole rootfs
DISK_NAME="/"
# Report on 3/4 full disk
DISK_TRESHOLD=75
Then enable it with a symlink from /etc/mender-monitor/monitor.d/available → /etc/mender-monitor/monitor.d/enabled
Adding:
echo -ne ‘SERVICE_NAME=“cron”\nSERVICE_TYPE=“systemd”’ > monitor.d/enabled/service_cron.sh;
causes the error:
Service cron not running CRITICAL 2021-12-01 14:33view details
But I will stop investigating it and wait for sufficient documentation or some ready to use examples.
This actually means that an alarm should show up in your UI, notifying you that cron is not in fact running, which I think you can verify with:
systemctl status cron.service
Can I install this mender-monitor in my client devices and still send alerts to self-hosted mender server?
yes, there is nothing keeping you from doing this.
It is an enterprise feature though. So self-hosted still needs to be Enterprise.
Ok. Thank you!