Getting started - log monitoring

*The Mender OTA software updater is used for the purpose of this worked out example in secure edge device monitoring.

An OTA software updater offers a secure capability to allow you to monitor services and log files. Typically an edge device monitor works by executing enabled checks, triggering, and delivering alerts directly to the management server of the OTA updates manager and, from there, to authorized users who have sufficient permissions. It can do much more than check if the cron daemon is running, and it is this “much more” that you should use to implement security monitoring and alerting.

Log subsystem

A vital feature of an edge device monitor is the ability to raise alarms when a given data set contains a pattern. Originally used to parse log files, the edge device monitor should be able to stream any command stdout and react to the lines that match specific patterns. For the sake of simplicity, we will mainly use the log subsystem in the examples below. Keep in mind that extending and integrating the monitor with your systems is easy and much more tightly than what we present here. We will also base most of the checks on a simple case, which we can summarize as follows:

# create a check
mender-monitorctl create log results-scan "CVE.*HIGH" "/tmp/results-scan" 2d

# enable the check
mender-monitorctl enable log results-scan

Where results-scan is a name of your choice, "CVE.*HIGH" is the pattern we are looking for in /tmp/results-scan, and 2d means that it will call off the alert after two days. We will als and as needed, swap the file path with a command to get the data from an arbitrary source:

# create a check
mender-monitorctl create log pid-change "PID changed" "@/usr/local/bin/pid-change" 2d

# enable the check
mender-monitorctl enable log pid-change

# restart mender-monitor
systemctl restart mender-monitor

It is @ as the first character in the fifth argument to mender-monitorctl that marks the rest of the string as a command which the edge monitor will run and which stdin will be parsed and checked against the pattern (fourth argument). All we have to do is implement /usr/local/bin/pid-change .

For further reading and more examples, please consult the log subsystem section in the Mender technical documentation. You need to ascertain if the pattern matches the data stream (either from a file or a command); if it does, the edge monitor will issue an alert. Once it reaches the server, it will turn into a UI notification in the device details and on the devices list, and if you have not disabled the email notifications in the settings, it will trigger an email notification to all the users of your tenant who have access to the device.

An OTA software updater can add value by helping to address critical edge security monitoring cases. It will help if you start with a general configuration for all the security checks. A device configuration tool in the OTA software updates manager can be used.

We will show you how to pass the scan settings to a device using the configuration capability within an OTA updates manager and conclude with an example implementation. Please remember that we endeavor to show the most straightforward possible solutions. You can extend and tailor all of it to your needs and device management ecosystem by introducing extra parameters and security checks and using the REST API of the OTA updates manager; you can do everything shown programmatically.