Using Node-RED with Mender

drewmoseley · October 8, 2020, 7:54pm

Intro

In this tutorial, we will discuss how to use the Mender OTA with Node-RED to deploy updated flows. We will start at the beginning and use the Raspberry Pi OS on a Raspberry Pi 3A board. The instructions will be specific to that OS and hardware but will likely only require minor modifications to work with other solutions.

This is not intended to be a complete tutorial on Raspberry Pi, Node-RED, or even Mender. There are plenty of resources available to dig deep on each of those topics. We will only concern ourselves with the intersection of them all. Our goal will be to develop a golden master image of the Raspberry Pi OS customized to run Node-RED. We will then integrate Mender with that image and deploy it to a fleet of devices. We can use standard Mender full image payloads to deploy OS updates to the device. But we will also develop a custom payload type for Node-RED flows, allowing us to quickly deploy updated flows to our device fleet without dealing with the complexity of a full image update.

About Node-RED

Node-RED is a browser-based low-code platform for programming event-driven applications, written in Node.JS. Quoting from the main website:

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

Node-RED is used heavily in DIY/Maker applications as well as Industrial Internet of Things (IoT) markets.

Getting Started

Install Raspberry Pi OS

As a first step, we will produce our golden-master image. This is the image that will be used as input to the Mender conversion utilities. Let’s start by downloading the latest Raspberry Pi OS image and writing it to an SDCard. The easiest way to accomplish this is to use the Raspberry Pi Imager provided by the Raspberry Pi Foundation. Open a web browser and navigate to the Raspberry Pi Downloads page. Select the binary matching your desktop operating system and install it using the appropriate mechanism for your OS. Insert an SDCard into your development host and run the imager:

Select the CHOOSE OS button:

Then select Raspberry Pi OS (other) and finally select Raspberry Pi OS Lite (32-bit):

Next, click the button labeled CHOOSE SD CARD and then select the SD Card that you previously inserted into your desktop:

Finally, click the _WRITE _button and follow the prompts from the Imager.

Setup headless mode

Now we will configure the operating system to run an SSH server, as well as to automatically connect to the wifi network. Optionally you can enable the serial port for easy remote access to the device. Note that for production operating systems you will likely want to undo or lockdown these settings before deployment. For now, we will keep it simple and open.

Remove the SD Card and reinsert it in your desktop. If your development host is not configured to automatically mount removable media, then you will need to consult the respective documentation to manually mount it. There is a single FAT partition that will be mounted on most hosts and we can use this partition to configure SSH and wifi.

To enable SSH, we will create an empty file in the FAT partition named ssh. You can do this with the GUI (ie Finder in MacOS, Explorer in Windows, and Files in Ubuntu); consult the documentation for details.

To enable wifi, we will create a file in the FAT partition named wpa_supplicant.conf that contains our credentials.

To enable the UART/serial port, we will add a line to the existing config.txt file in the FAT partition.

Using the command line for these changes, enter the following commands. Note that the path to the SD Card will need to be changed to match your host

touch /media/user/boot/ssh
sudo cat > /media/user/boot/wpa_supplicant.conf <<EOF 
country=us
update_config=1
ctrl_interface=/var/run/wpa_supplicant
EOF
 wpa_passphrase ssid 'password' | grep -v '#psk' >> /media/user/boot/wpa_supplicant.conf
echo enable_uart=1 >> /media/user/boot/config.txt

Note: There is a space at the start of the _wpa_passphrase _command. This ensures that this command will not appear in your .bash_history file which is important to avoid possibly leaking your password.

More details about Raspberry Pi OS headless mode are available from Adafruit.

Accessing the Raspberry Pi OS

There are several methods for accessing a shell prompt on the Raspberry Pi. Choose the method that is most suitable for you.

Use an HDMI monitor, and USB keyboard. The device will boot with a login prompt on the HDMI console.
Use a serial cable. If you enabled UART in the steps above then you can use an appropriate cable and serial terminal program on your desktop. The device will boot with a long prompt on the UART.
Login over SSH. If you enabled SSH in the steps above then you can login using ssh. The default credentials are user pi with password raspberry. You need to determine the IP address of your Raspberry Pi. Usually your router configuration interface can help with this.

Boot Raspberry Pi OS and apply all operating system updates

General best practice would be to apply all updates provided by the vendor when creating your golden master image. In this case we will use the apt utility to make sure our device is up-to-date.

Remove the SD Card from your desktop, insert into the Raspberry Pi and power up the device. Connect to a login prompt using one of the methods discussed above and login with user pi and password raspberry.

Now perform the following change of password on your device.

passwd

Finally, enter the following commands to update all installed software to the latest versions:

sudo apt update
sudo apt dist-upgrade -y
sudo apt autoremove -y
sudo reboot

Install Node-RED

Now that we have the base OS installed and up-to-date, we can start installing our application stack. In our case we will install Node-RED using the instructions from their web site:

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

Follow the on-screen prompts. This will take some time so now would be a good time for a break. Once the installation completes, review the file /var/log/nodered-install.log to make sure no issues were encountered when installing.

Now we will enable the Node-RED service so that it automatically starts at boot time:

sudo systemctl enable nodered
sudo systemctl start nodered

Now if you open a browser on your PC and browse to port 1880 on the IP address of your Raspberry Pi device, you will see the Node-RED Programming interface. Ie http://192.168.1.100:1880

Create your first Node-RED flow

We’ll create a sample flow to make sure we have a functioning Node-RED setup. First, let’s make sure our Node-RED flows can access operating system information. This allows us to return data such as the device hostname. Edit the ~/.node-red/settings.js file in the Raspberry Pi filesystem. Both the nano and vi editors are installed. Search for the string functionGlobalContext in the editor of your choice and uncomment the line containing os.require(‘os’).

    // The following property can be used to seed Global Context with
    // predefined values. This allows extra node modules to be made
    // available with the Function node.
    // For example,
    //    functionGlobalContext: { os:require('os') }
    // can be accessed in a function block as:
    //    global.get("os")
    functionGlobalContext: {
        os:require('os'),
        // jfive:require("johnny-five"),
        // j5board:require("johnny-five").Board({repl:false})
    },

We will start with a simple flow that contains 5 nodes. We have two input nodes, a function node and two output nodes. All the node types are listed on the left side of the Node-RED UI. They are added to the flow by dragging the appropriate node type into the flow tab of the UI. You can then customize each node by double clicking to see details relevant to that node. Connections between nodes are made by selecting the circle icons displayed on the border of the nodes and dragging a line to a different node.

Node Descriptions

Create an “inject” node. This allows us to inject arbitrary data into the filesystem. In this flow, we are using this primarily for debugging. Customize the node and change the _msg.payload _type to be a string containing the venerable “Hello world!”.

Create an “mqtt in” node. This will read from a specific MQTT topic and pass the message contents to the next node in the flow. The first step is to configure the MQTT server. Select the pencil icon next to the “Add new mqtt-broker…” field.

Enter the server details. We will use the publicly accessible MQTT server at test.mosquitto.org for this demo.

WARNING: This MQTT server is open to the public and any data you send there will be visible to anyone.

Now, set the Topic field to something like “mender-node-red-demo/in”.

Next create a “function” node. This is where we will get the device information. Customize this node and set the javascript code to run as shown below.

Now create a “debug” node. This allows us to view the message payload in the Node-RED UI. To view the debug data, select the bug icon in the upper right corner of the UI.

Finally, create an “mqtt-out” node. Use the server we setup previously, and set the Topic to “mender-node-red-demo/out”

Now make sure you make connections between the nodes as shown in the following screenshot.

Testing the flow with the debug window

Before the flow is active, we need to click the “Deploy” button in the upper right corner. Once the “Successfully Deployed” banner is displayed, then your flow is running on the device. Click the blue box next to the Hello World node. This will inject a message into the flow. Then the hostname node will gather the device hostname and display it in the debug window.

Testing the flow with MQTT utilities

There are two utilities we can use from the command line to publish to MQTT topics and subscribe to them. In Ubuntu these can be installed using apt:

sudo apt install mosquitto-clients

These utilities are also available for both Mac and Windows. See https://mosquitto.org/ for more details.

Setup a command-line subscribed to the MQTT output topic we added in the flow above.

mosquitto_sub -h test.mosquitto.org -t mender-node-red-demo/out

Now publish a message to the MQTT input topic. Our flow does not do anything with the payload but simply uses the message as a trigger.

mosquitto_pub -h test.mosquitto.org -t mender-node-red-demo/in -m "doit"

If everything is working as expected, your shell running the mosquitto_sub command will display the hostname of your Raspberry Pi:

$ mosquitto_sub -h test.mosquitto.org -t mender-node-red-demo/out
raspberrypi

Integrate Mender into the operating system

Now that we have a functioning Node-RED installation, with a working flow for testing, we can create our golden-master image and integrate it with Mender using the mender-convert utility. Full documentation for this process is available in the Mender documentation.

Shutdown your Raspberry Pi, and insert the SD card into your PC. View your operating system logs to determine the device node connected to the SD card. Then, clone the release version of mender-convert, and create the input image from the device node previously determined.

Make sure to set the SD_DEVICE environment variable to the value you determined above from your logs.

SD_DEVICE="/dev/sdb"
git clone -b 2.2.0 https://github.com/mendersoftware/mender-convert.git
cd mender-convert
./scripts/bootstrap-rootfs-overlay-hosted-server.sh --output-dir ${PWD}/rootfs_overlay_demo --tenant-token "Paste token from Mender Professional"
mkdir input
sudo dd if=${SD_DEVICE} of=input/raspbian-node-red-golden.img bs=8M
./docker-build
MENDER_ARTIFACT_NAME=release-1 ./docker-mender-convert --disk-image input/raspbian-node-red-golden.img --config configs/raspberrypi3_config --overlay rootfs_overlay_demo/

Save the golden-master SD Card for later processing. Insert a new SD Card into your PC and provision it with the Mender-integrated image.

sudo dd if=deploy/raspbian-node-red-golden-raspberrypi3-mender.img of=${SD_DEVICE} bs=8M status=progress conv=fdatasync
sudo sync

This SDCard can be inserted into a Raspberry Pi device and deployed to the field. Once the Pi is up and running, you can retest with the MQTT command line tests we used in the previous session. If you navigate to the hosted Mender devices page, you will, in a short time, see the device listed in the “Pending” tab. Click through the device and select the “Accept” list to admit the device to your fleet.

Update the Node-RED flow

Now, let’s presume we have a customer feature request. We want to be able to get the IP address of the device using a similar mechanism. We will modify the flow so that the MQTT input node takes two values to indicate which device parameter we want to return. The device parameters will be stored in separate MQTT topics based on the input value. To do this we will power up our golden master device, and update the flow as follows.

First, change the MQTT input node to reference the topic mender-node-red-demo/cmd since it now will contain a specific command.

Next, update the function node. Change the number of outputs to 2, change the name to Process Command and replace the function code block with the following:

if (msg.payload == "hostname") {
    msg.payload = context.global.os.hostname();
    return [msg,null,null]
} else if (msg.payload == "ip") {
    msg.payload = context.global.os.networkInterfaces()["wlan0"][0]["address"]
    return [null,msg,null]
} else {
    msg.payload = `Unknown command ${msg.payload}`
    return [null,null,msg]
}

This changes our function node to have three outputs. All outputs will be connected to the debug node. The two valid outputs will additionally be connected to appropriate MQTT output nodes. We can also disable the debug node by clicking the green box at the right end of the node. The updated flow should look like this:

We can test with the following commands:

mosquitto_pub -h test.mosquitto.org -t mender-node-red-demo/cmd -m "doit"
mosquitto_pub -h test.mosquitto.org -t mender-node-red-demo/cmd -m "hostname" 
mosquitto_pub -h test.mosquitto.org -t mender-node-red-demo/cmd -m "ip"

And verify that the appropriate MQTT topics are updated:

$ mosquitto_sub -h test.mosquitto.org -t mender-node-red-demo/ip
192.168.7.182
$ mosquitto_sub -h test.mosquitto.org -t mender-node-red-demo/hostname
raspberrypi

Deploy the updated flow to your device fleet

In the following we will use Mender to deploy updated flows with rollback support. If a new flow fails to install, Mender will roll back to the previous flow.

Now we can export all the flows into a text file that can be converted into a Mender artifact and deployed over the air to your devices. Click the menu icon in the upper right of the UI and select the Export menu option. Make sure to select All Flows, and then Download. This will download a file named flows.json. We will use the single-file update module with state scripts to update the Node-RED instance running on the deployed device. The update module itself is installed by default with the mender-convert utility. The following steps are an abbreviated form of the details covered in the Mender hub post. Please refer to that post for full information.

Setup for generating an artifact

Run the following commands to download the necessary scripts:

wget https://raw.githubusercontent.com/mendersoftware/mender/master/support/modules-artifact-gen/single-file-artifact-gen
chmod +x single-file-artifact-gen 
wget https://d1b0l86ne08fsf.cloudfront.net/mender-artifact/3.4.0/linux/mender-artifact
chmod +x mender-artifact 
sudo mv mender-artifact /usr/local/bin/

Create the state scripts

A Mender State script is a generalized form of a post-install script. State scripts can be run before (Enter) or after (Leave) any state in the Mender update process.

We will create several state scripts. The first is run on the ArtifactInstall_Enter state and is used to backup the existing flows from the Node-RED server.

cat > ArtifactInstall_Enter_01_Backup_Current_Flows <<-EOF
#!/bin/sh
mkdir -p /data/node-red-flows-update
curl -X GET http://localhost:1880/flows \\
    -H 'Content-Type: application/json'  \\
    > /data/node-red-flows-update/backup-flows.json
exit \$?
EOF

Next, we will create the state script that installs the new flows. This will be done in the ArtifactCommit_Enter state so we can ensure that the new file has been deposited into the target filesystem.

cat > ArtifactCommit_Enter_01_Install_New_Flows <<-EOF
#!/bin/sh
if [ ! -f "/data/node-red-flows-update/flows.json" ]; then
    exit 1
else
    curl -X POST http://localhost:1880/flows \\
        -H 'Content-Type: application/json' \\
        --data @/data/node-red-flows-update/flows.json
    exit \$?
fi
EOF

Finally, we will create a state script to restore the backed up flows. Note that we don’t have any other post-install checks that will trigger this in the current setup. This will be very dependent on the content of your flows. To implement further sanity testing and possibly trigger the rollback, you will need to develop additional ArtifactCommit_Enter scripts that will do the appropriate checks and return an error code if needed. See the Mender state scripts documentation for full details.

cat > ArtifactRollback_Enter_01_Restore_Backup_Flows <<-EOF
#!/bin/sh
if [ ! -f "/data/node-red-flows-update/backup-flows.json" ]; then
    exit 1
else
    curl -X POST http://localhost:1880/flows \\
        -H 'Content-Type: application/json' \\
        --data @/data/node-red-flows-update/backup-flows.json
    exit \$?
fi
EOF

Create the Mender artifact

We now have all the pieces to create a deployable artifact to update the flows to your device fleet. Invoke the following command to create the artifact.

./single-file-artifact-gen -n updated-flows -t raspberrypi3 -d /data/node-red-flows-update/ -o updated-flows.mender flows.json -- --script ArtifactInstall_Enter_01_Backup_Current_Flows --script ArtifactCommit_Enter_01_Install_New_Flows --script ArtifactRollback_Enter_01_Restore_Backup_Flows

The file updated-flows.mender can now be uploaded to your Mender server and deployed through the Mender web UI. Once the deployment is complete, then all your devices will be running your latest and greatest Node-RED flows.

You can utilize the expansive features of Mender when creating a deployment based on your updating plans and requirements. For example, you can schedule deployments at a pre-defined start time and date allowing you to minimize fleet interruption during certain peak hours. You can also greatly reduce risks by dividing a deployment into time-delayed phases with a customizable share of the devices being updated in each phase. For example, deploy to 5% of the devices, wait 24 hours, then 15%, wait 48 hours, and so on. It gives you the ability to deploy based on your needs and risk levels.

Other Considerations

Please note that this tutorial only covered the basic functionality needed for a Node-RED installation with Mender. There are many other considerations needed before rolling this out beyond an initial pilot phase.

Security

Both the MQTT broker and the Node-RED installation used here are wide open and subject to abuse. Many resources exist to help you decide how to lock these down including the following:

Node-RED security: https://nodered.org/docs/user-guide/runtime/securing-node-red
MQTT security: https://www.google.com/search?q=mqtt+security

Reproducibility

Using a pre-built binary OS such as Raspberry Pi OS is a very quick way to prototype these kinds of applications. However, they have issues with build reproducibility, scaling to large development teams, and general security posture. There is generally a lot more software preinstalled in these operating system images than needed for any particular application and stripping them to the bare minimum can be tricky. Using build systems such as Buildroot[^10] or Yocto[^11] address these issues and can make for better product design. We have a number of resources that go into more depth on this topic and a few are listed below.

Embedded Device Use Cases

The most interesting aspect of using embedded boards such as the Raspberry Pi is its ability to interact with the real world. Using sensors and actuators at remote locations adds many interesting use cases to these kinds of designs. Our sample code here did not take advantage of that and simply used internet-based technologies. These flows could just as easily have run on your desktop PC. There are embedded-specific nodes that you can use with Node-RED that give you access to features such as GPIO, I2C, and SPI which are heavily used in IoT applications.

Conclusion

This tutorial has demonstrated how to set up a basic Node-RED installation on a Raspberry Pi board running the Raspberry Pi OS. We then integrated that installation with Mender to add over-the-air update functionality. We demonstrated a custom artifact setup that allows you to deploy updated Node-RED flows to remotely deployed devices. We did not discuss other types of payload updates however the Mender integration completed here is fully capable of handling full image updates as well as any other payload types you have to deploy.

Topic		Replies	Views
How to update a Docker container with a Raspberry Pi 3 Mender raspberry-pi-3 , docker , mender , update-modules , mender-artifact	14	8501	July 26, 2023
Mender with python app General Discussions raspberry-pi-3	9	1407	September 25, 2019
Best way to setup Mender on multiple devices General Discussions raspberry-pi-4	9	1365	December 10, 2020
Can we use a XXYY.img.xz for OTA update to a raspberry Pi 3? General Discussions	1	286	January 22, 2021
How to update .deb package with a Raspberry Pi 3 Mender	0	5598	October 4, 2019