We are experiencing an unprecedented acceleration of technological development. Sometimes growth is faster than exponential, and sometimes it feels like an exponent of the exponent. The proliferation of connected devices in recent years is part of the phenomenon. Unfortunately, the increasing pervasiveness of technology brings new threats and dangers. More powerful devices mean more complex applications, which in turn means greater access to data, more customers, more users, more points of entry, and more dependencies. This means more places to check for security flaws but also more resources at our disposal on the devices to not only scan and monitor but also make decisions. In edge computing we are harnessing distributed and remote power while decentralizing the point of decision making. The complexity of the solutions walks in front of new challenges. You want to know what is happening with your fleet as much as be sure it is secure.
In the following sections we will present a subset of most common security monitoring scenarios assuming your remote device is running Linux.
Open ports
The number of open, bound, and listening ports on external IP addresses on embedded devices should be minimal. Preferably, IoT devices do not listen on any ports. Any unexpected TCP port in a listening mode could spell danger, and you will want to know as soon as possible what process is responsible for opening it. In this case, you need to have a list of known open ports with the name of the processes and path to the executable; anything not on the list is a cause of a security alert.
Set-uid executables
Most users would expect to find a limited number of executables that have setuid on exec bit set. As with open ports, you should expect only a few well-known executable paths and names you know have the bit on. Any sudden appearance of a set-uid executable in a given list of locations (assuming that you only allow execution from certain places) should raise suspicion and an alert.
Unexpected process running
IoT-connected devices are like “micro-servers” (in a similar way we use the term microservices): typically, they do not run a lot of applications, and most of the time, you will be aware of precisely what a given device should have in the process table, and the role the device has to play; usually, it is a small (micro) application or service.
Similarly to the open ports and set-uid executables, you can maintain a list of expected executables you accept to be running. There is a small catch here: there are ways to hide a process or run them fast enough to make them hard to detect with the standard utilities.
High data transfers
There are a large number of use cases where you would expect the outgoing or incoming (or both) bytes transferred per second to be next to nothing (e.g., transfer of the order of magnitude of 1 kilobyte per day). Especially when the network uplink is more expensive than my car, I would appreciate an alert before I see the bill from the satellite telecoms company! You can imagine a certain threshold of bytes transferred in an hour, above which you should treat a device as being in a critical state.
Executable changing location
Usually, in one life cycle of a software version on a device, you should not expect the core executables to change location and the files themselves to change. For instance, your cherished /usr/bin/mender-connect executable should not change unless you have explicitly upgraded it. It should also occupy the same inode until the subsequent deployment of the new software to the device. Any executable from a given list that does not match the expected inode and sha256sum should raise serious concerns.
Failed deployments
Software updates occupy a central place in the device lifecycle as they can deploy new applications and security fixes as well as supply new configuration and system updates. It may be a critical issue when a device fails to update for unknown reasons. Alarm bells should start to ring in this case.
Root login attempts
In many situations, any direct root login to a device is entirely unwanted and disabled. There are a few scenarios when you must log in to the root account. The logins may come in at least two flavours: remote (e.g., SSH) or local (e.g., sudo).
sudo
Not all users have permission to perform sudo commands, and you may have some other channel to get the upgrade to a super user. You can set up an alert with the monitoring capabilities found in an OTA software updater every time a suspicious sudo attempt occurs.
ssh
A remote terminal capability in an OTA software updater eradicates the need to use ssh. However, it may also be worthwhile to get an alert on whether or not there is a logged attempt to access a leftover sshd with root credentials.
Network access point change
There are cases when you know precisely to which access point a device connects. The fact that it all of a sudden changed an SSID on a wireless interface could be a sign of trouble ahead.
Process ID change
Assuming you have a long-running application on your device, it would be unusual for that application to restart, let’s say, on Thursday unexpectedly and without your knowledge. It may mean that an unauthorized user is trying to exploit some weakness. It also may be that this malicious actor does not expect that you can monitor the process id (PID) change. PIDs always stay the same during the lifetime of a process, so the chance that the new PID would be the same as the old one is next to impossible. You can safely assume that the new PID means there was a restart.
Periodical automatic vulnerability scans
Last but certainly not least, all the above checks may be in an ok state, yet your device may still have serious problems. What if a recent deployment introduced a critical vulnerability: an executable that contains a well-known security hole in this version? The only way to uncover that vector is to perform an active scan inside the device. Fortunately, you can perform such scans, parse the output and raise alerts if need be.
The above list of security monitoring scenarios is a partial one. It’s worth bearing in mind that even performing basic checks and changing configurations in order to get the desired alerts requires a lot of work in the absence of the monitoring capability found in an OTA software updater. The question arises: how to configure the monitoring, generation, and delivery of the alerts, and how to do it at scale?