How to write a custom client interfacing a Mender server

mirzak · September 11, 2020, 7:11am

Introduction

Mender is an end-to-end OTA solution and both the Mender client and server are provided as open-source, additional we provide reference board integrations that bind it all together.

But Mender was designed with interchangeability in mind, especially for the client part of the solution. We provide documentation for the device facing API’s of the server which are implemented by the Mender client, but it is also a reference for implementing or integrating thirdparty clients.

These resources can be useful when:

your connected devices is not running Linux which makes it incompatible with
the existing Mender client (e.g Android, Windows, MCU’s).
your want to integrate an existing thirdparty update client, e.g SWUpdate, RAUC or
a homegrown solution.

The API documentation is a good starting point but it does not cover the expected workflow and one would need to dig trough the Mender client source code to try to figure out how it is using the API’s.

This tutorial aims to provide a overview of the API workflow and how to approach integrating either a custom or a thirdparty client with the Mender server API. For demonstration purposes we will be using standard CLI tools that are available on any major Linux distribution and in the end we should a working Mender client implemented as a bash script.

This tutorial will not cover bootloader integrations, flash layout or how you implement the logic to write the payload that you get from the Mender server.

Prerequisites for this tutorial

It is assumed that you are running a Linux distribution (native or in a VM) with the following packages installed:

bash - might work with other shell implementations
jq - parse JSON structures
curl - does HTTP(s)
openssl - for keys and signatures

On a Debian based distribution you can run the following to install the dependencies:

sudo apt-get install bash jq curl openssl

Additionally, you will need to install mender-artifact , version 3.1.0 or later

Prerequisites for Mender client in a micro controller (RTOS)

To implement a custom Mender client your environment must have the equivalent capabilities to the tools listed above (excluding bash).

Your environment must have he following capabilities:

Able to communicate over TCP/IP using the HTTP protocol with the SSL extension (HTTPS)
Able to encrypt/decrypt/sign/verify data using the RSA or ECDSA256 cryptographic algorithms
Able to process JSON structures
Able to process tar file archives. Mender Artifact are uncompressed tar file archives
(Optional) Able to decompress gzip or lzma. By default the Mender Artifact payload is compressed using gzip (also supports lzma) but payload compression can be disabled.

If you have an device that is already using connectivity, and specifically HTTPS then you probably have the necessary components in place.

This is a unverified shortlist of stacks that are good candidates to support above requirements:

Zephyr Project
Mbed OS
Google Cloud IoT Device SDK for Embedded C
and similar

Overview

A client integrating with a Mender server only needs to implement five API calls:

And the workflow can be described with this simple diagram:

mender_client (1)

Step 0 - Preparations

As preparation we need to define a couple of variables that we will use trough out this tutorial.

Set Mender server URL:

MENDER_SERVER_URL="https://hosted.mender.io"

Set the tenant token variable (you can get it here):

MENDER_TENANT_TOKEN="< paste your token here >"

Set Mender client device type:

MENDER_DEVICE_TYPE="curl-client"

Set Mender Artifact name:

MENDER_ARTIFACT_NAME="release-v1"

Step 1 - Authentication

The device authentication workflow is probably the most involved process and is something that most people struggle with initially. This is also something that is hard to debug, because you need to cryptographically sign data and if something goes wrong you will only that the signature is wrong, but not indicators on where it went wrong along the way.

You can familiarize your self with the device authorization process by reading the Device authentication section in the official Mender documentation.

Here we need to use the /api/devices/v1/authentication/auth_requests (POST) API call.

As the API documentation indicates, we need to send two things:

X-MEN-Signature - Header request signature
auth_request - Request body
- device_id - You can read more about how device identity works in Mender
  here
- public key - Used to verify the X-MEN-Signature, this proves
  that who ever sent this request also owns the private key.
- tenant token (optional) - Needed to connect to a multitenant server, e.g https://hosted.mender.io

Lets get to it. To send an authorization request we first must generate a key pair.

Generating a private RSA key can be done by executing the command below:

openssl genpkey -algorithm RSA -out private.key -pkeyopt rsa_keygen_bits:3072
openssl rsa -in private.key -out private.key

Extract a public key from the private key use following command:

openssl rsa -in private.key -out public.key -pubout

Store public key in a variable and replace newlines with \n:

PUBLIC_KEY=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' public.key)

The PUBLIC_KEY content should be:

Generate request body (we use MAC address as identity data here):

# NOTE! This command will trim newlines, and it is very important to do this,
# otherwise the data will not match what is sent out, and this also means
# that the signature would be invalid
read -r -d '' REQUEST_BODY <<EOF
{
    "id_data": "{ \"mac\": \"00:11:22:33:44:55\"}",
    "pubkey": "${PUBLIC_KEY}",
    "tenant_token": "${MENDER_TENANT_TOKEN}"
}
EOF

The REQUEST_BODY content should be:

{ "id_data": "{ \"mac\": \"00:11:22:33:44:55\"}", "pubkey": "-----BEGIN PUBLIC KEY-----\nMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEA0dIShCaaRf5EyvscTMI+\nXPe5EVrIHUj3kOs4KgOM++dswq+OMh03UJqW73p63Seay8H44s7J/D2SbsxtvAoU\nxH7uCOySquqYtyxi0DkufeYn/MbxPhZAgaK1tIa/jGhGfxqvJkJF1lsWPyboa8+g\nNPxwH3G4/m9IZW5IwhJ99Rbh1q6UWMxhNRi+f4Y1b4/JcFBGPaUxEuO6nkLQbAS4\nBUl+KefzJuliPIenbQYMmO5bkfgD0PZMVXRTTFcQfnpt4fRxLUmqsdnjSMbrloVY\nW/Nu2Z25A6+Dvw1A2eiASLtH5+3B26B5syEGYFcVea3xog5KqkViUbZvXIZQ3uI3\nB/uO8JlUSjXy6La9ZHriRqlbZi3VUfD1OiPfO8qZEfS/8MP5ttvP8rs3nra8Whby\nRAwtfO5R11r1+Mq34wOUM9OZJNDgINHRzSerbvIf6jkneMCE0b1IoW5RBoYapmy5\nv8bEvG4gJf6n62jfIvgzBBCSFn54pwaRCwREXHXNH52XAgMBAAE=\n-----END PUBLIC KEY-----\n", "tenant_token": "< paste your token here >" }

NOTE! That the tenant token has the default value here, and instead it should print what you configured.

Generate signature of type RSASSA-PKCS1-V1_5-SIGN:

X_MEN_SIGNATURE=$(echo -n "${REQUEST_BODY}" | openssl dgst -sha256 -sign private.key | openssl base64 -A)

The X_MEN_SIGNATURE content should be:

MXWhOON/w2lQI6uj2Arbxauvl3Quu7hT35yoY3XBG+KxDzJjLiUQDDo45uHDYKIUV6V4OqKdnsz9LQ+4s9X8IlHPKqxPiUKCByAPb5O+OaWfOC1BbkGLYAVULtes1J2p1r4n0benr2E2T3VlTl3x1Zw3lxAJsRiQTPdpgyDvcV46R+lUWCYnCXXwrxAXna5RovgzyhBSkCFIE3bcSm5/M8jKyFNA3iEoSCuJpv4ZcAjpvQTp/wbFC134hHmP03FfbRGYFUNDR5gi5gUYDVfCUoTp739AaVpzt8ASuWTHtQ9PFEVYfhlr7U0L0w6sSNbFCuDSsbGTYgPNBlYHE0d5xAR7zRevgZmUB+++rg07l3FXgimLdd8YJUV7jp2pCfQze3d6DCiIWHpxl4WjuTgBlNDcrIi72MqmlIwzRxAKy9Y5R+A4bLbsFdaVt5JVBAg8QP84O81Xp/dP7TG4wryF25/3VHob3N7W1wF31EKPtVhqVQX1ltQgS7zxT8ttucMf

NOTE! Above value is only valid if all the default variables are used, including the tenant token so your signature might differ.

curl \
    -H "Content-Type: application/json" \
    -H "X-MEN-Signature: ${X_MEN_SIGNATURE}" \
    --data "${REQUEST_BODY}" \
    ${MENDER_SERVER_URL}/api/devices/v1/authentication/auth_requests

The expected response to above is:

{"error":"dev auth: unauthorized","request_id":"7d637207-6a62-4c16-8e1f-14d37bd0693f"}

This means that the device is not yet authorized, but we have successfully sent the authentication request.

If you go to the Pending on the server you should see device there with a matching device identity to what we prepare for our authentication request.

NOTE! Make sure to accept the device on the server before proceeding with the next steps.

Once the device is accepted on the server we can run the same command again to get an access token (JSON Web Token):

JWT=$(curl \
    -H "Content-Type: application/json" \
    -H "X-MEN-Signature: ${X_MEN_SIGNATURE}" \
    --data "${REQUEST_BODY}" \
    ${MENDER_SERVER_URL}/api/devices/v1/authentication/auth_requests)

The JSON Web token will be used for all subsequent API calls.

Step 2 - Publishing inventory data

Each device connected to the Mender server must publish inventory data to populate the necessary fields for the GUI to display in the Device tab.

At a minimum the device_type and the artifact_name name should be sent. You can read more about inventory data in the official Mender documentation

Inventory data is to be sent as an JSON array, and we can prepare it with the following command:

read -r -d '' INVENTORY_DATA <<EOF
[
    {
      "name":"device_type",
      "value":"${MENDER_DEVICE_TYPE}"
    },
    {
      "name":"artifact_name",
      "value":"${MENDER_ARTIFACT_NAME}"
    },
    {
      "name":"kernel",
      "value":"$(uname -a)"
    }
]
EOF

Now we can publish it to the server:

curl \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $JWT" \
    --data "${INVENTORY_DATA}" \
    -X PATCH \
    ${MENDER_SERVER_URL}/api/devices/v1/inventory/device/attributes

You should now see something similar to below in the Devices tab.

Step 3 - Check for deployments

To check if there are are pending deployments on the Mender server we can run (command will print return code):

curl -s -o /dev/null -w '%{http_code}' \
    -H "Authorization: Bearer $JWT" \
    -X GET \
    "${MENDER_SERVER_URL}/api/devices/v1/deployments/device/deployments/next?artifact_name=${MENDER_ARTIFACT_NAME}&device_type=${MENDER_DEVICE_TYPE}"

The expected result right now is to get return code 400, meaning that there are no pending deployments. We would keep doing this check periodically until we get return code 200 meaning that there is a pending deployment and we should handle it accordingly.

Step 4 - Download deployment

For demonstration purposes we need to prepare a Mender Artifact that we can deploy to our curl-client.

Create payload data:

dd if=/dev/urandom of=payload.img bs=1M count=1

Create a Mender Artifact:

mender-artifact write rootfs-image \
    --device-type ${DEVICE_TYPE} \
    --artifact-name release-v2 \
    --output-path release-v2.mender \
    --file payload.img

Upload the artifact to the Mender server, under the Releases tab.

You should have something like this:

Press the “CREATE DEPLOYMENT WITH THIS RELEASE” button and click trough the deployment wizard.

Lets check if there are any pending deployments now:

curl \
    -H "Authorization: Bearer $JWT" \
    -X GET \
    "${MENDER_SERVER_URL}/api/devices/v1/deployments/device/deployments/next?artifact_name=${MENDER_ARTIFACT_NAME}&device_type=${MENDER_DEVICE_TYPE}"

Expected result is something similar to below:

{
 "id": "a4d9c64f-3de5-40f0-9886-c0f77d00baf0",
 "artifact": {
   "artifact_name": "release-v2",
   "source": {
     "uri": "https://s3.amazonaws.com/hosted-mender-artifacts/5b48937d7e71f600014ab529/af0ccfb3-c410-40f8-9b30-649e5dd7878c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIAQWI25QR6NDTJ7DLD%2F20191213%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191213T204753Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&response-content-type=application%2Fvnd.mender-artifact&X-Amz-Signature=3923b3238890ea98e51c6943805900570f306d1aaa858eedfd6b3ef1417dfd1b",
     "expire": "2019-12-13T20:47:53.869391192Z"
   },
   "device_types_compatible": [
     "curl-client"
   ]
 }
}

As you can see that the server has responded that there is a deployment in progress and we have gotten an signed URL from which we can download the Mender Artifact.

Step 5 - Updating deployment status on the server

The client is responsible of updated the deployment state to the server and in the end marking it either as successful or failure.

Supported states are:

downloading
installing
rebooting
success
failure
already-installed

There is also support for publishing substate, which is an optional parameter of the API call. Definition of the state strings are up to the user, and they will be only be displayed by Mender server without further processing. The official Mender client uses substate to report execution of state-scripts.

For demonstration purposes lets update the state to downloading on our pending deployment that we created in previous steps:

curl \
    -H "Authorization: Bearer $JWT" \
    -H "Content-Type: application/json" \
    -d "{\"status\":\"downloading\"}" \
    -X PUT \
    "${MENDER_SERVER_URL}/api/devices/v1/deployments/device/deployments/a4d9c64f-3de5-40f0-9886-c0f77d00baf0/status"

NOTE! We are using the deployment id ( a4d9c64f-3de5-40f0-9886-c0f77d00baf0 ) that we got from the previous step

You can verify that above command was successful by checking the Deployments tab on the Mender server.

And finally we can mark the deployment as complete by updating the state to success:

  curl \
    -H "Authorization: Bearer $JWT" \
    -H "Content-Type: application/json" \
    -d "{\"status\":\"success\"}" \
    -X PUT \
    "${MENDER_SERVER_URL}/api/devices/v1/deployments/device/deployments/a4d9c64f-3de5-40f0-9886-c0f77d00baf0/status"

NOTE! We are using the deployment id (a4d9c64f-3de5-40f0-9886-c0f77d00baf0) that we got from the previous step

The last command should have marked the deployment as complete on the server. You can verify this by going to the Deployments tab.

The is also an possibility to send a device log to the Mender server using the API’s, which is something you would do in case of update failure. This way one can troubleshoot deployments using the logs that where pushed to Mender server.

Conclusion

In this tutorial we have gone trough the Mender server client interface in detail, using common CLI tools on Linux. Hopefully we have demonstrated the simplicity of the interface and that it is helpful in your integration efforts.

Can further recommend tools such as Postman, which are useful to debug HTTP commands. While developing this tutorial. curl -v and curl --trace-ascii provided very useful which provide very detailed information on how the HTTP commands are sent out, down to byte level.

If you want an overview of the workflow in code, you can take a look at the mender-client.sh script which implements the steps covered in this tutorial.

Additional there are a couple re-implementations of the Mender client:

And of course a the source code of the official Mender client can provide a good resource for information.

If this tutorial was useful to you, please press like, or leave a thank you note to the contributor who put valuable time into this and made it available to you. It will be much appreciated!

mirzak · September 11, 2020, 7:12am

3 posts were split to a new topic: Use ECC keys for authentication between Mender client server