Hello,
When applying mender to my yocto-based verdin-imx8mp device, I made a pair of state scripts to make the process more user friendly. I pulled pieces from this community, but didn’t find a complete example of either function yet. As you can see from my script comments, I did run into several pesky bugs that I had to solve. So I wanted to share this with the community to save everyone the same headaches. And if anyone would like to make improvements, I’d love to discuss that here.
The scripts do the following:
1) Prompt the device user for approval before installing an update
This device has an LVDS display, so it made sense for our application. The script writes the update information to /tmp/mender/update-available and you can have your user interface app (not provided here) read the file and prompt for user approval however you’d like. Then the update proceeds when the user interface app writes the file /tmp/mender/update-approved
2) Backup some important files when applying a mender update
This reads a list of files and replaces them with symlinks to a file structure in /data/persist, so the data in the files/folders persist across updates. Some settings I found very nice to back up:
- Unique device identifiers generated at boot time (SSH Host keys, RDP certificate)
- Saved wifi networks
- /home/ directory
Here are those state scripts:
- mender-await-user-approval.sh:
#!/bin/sh
#test this without a full yocto/mender rebuild by simplying SCP'ing to your device at:
# /etc/mender/scripts/Sync_Enter_50
#Example:
# scp mender-await-user-approval.sh root@192.168.1.88:/etc/mender/scripts/Sync_Enter_50
#IMPORTANT LOGGING NOTE:
#Mender captures the standard error (but not standard out) stream from state scripts. The standard
# error stream from state scripts is stored as part of the Mender deployment log, so it becomes
# available locally on the client as well as reported to the server (if the deployment fails) to
# ease diagnostics.
#Thus the state scripts should be written so they output diagnostics information to standard error,
# especially in case of failure (returning 1). The maximum size of the log is 10KiB per state
# script, anything above this volume will be truncated.
echo "$(mender-update show-artifact): $(basename "$0") was called." >&2
#For testing/development, upload this script to your device with the command:
# scp mender-await-user-approval.sh root@192.168.1.88:/etc/mender/scripts/Sync_Enter_50
#Note that this script must be in the old software on the device, the one in the incoming image
# isn't used because this executes before the download of the new image
updtAvailFile=/tmp/mender/update-available
updtApprovedFile=/tmp/mender/update-approved
#updtAlwaysApprovedFile is in /root so it persists for all updates. Used for development purposes
updtAlwaysApprovedFile=/root/update-always-approved
env >&2 #for debugging. See if mender passes us any helpful env vars
#remove old approval files from past runs
rm -rf $updtApprovedFile
#if this script has been triggered, it means mender-updated just entered the download
# state. Grab some version info for dashapp, then pause until the user accepts
#This gets a JSON Web Token to authenticate with the mender server. It is based on
# this post: https://hub.mender.io/t/validate-device-jwt-authentication-token-on-custom-server/5232/2
jwt=`dbus-send --system --dest=io.mender.AuthenticationManager --print-reply \
/io/mender/AuthenticationManager io.mender.Authentication1.GetJwtToken | grep -F string \
| grep -vF http | sed -e 's/[ ]*string //' -e 's/"//g'`;
#JWTs consist of three parts (header, payload, and signature). Cut and display for debug with:
echo $jwt | cut -d '.' -f1 | base64 -d | jq .; #header
echo $jwt | cut -d '.' -f2 | base64 -d | jq .; #data
echo $jwt | cut -d '.' -f3; #signature - not json
#then this uses the token to ask the mender server to share details on the pending update
#TODO: have yocto inject machine name instead of having to edit this script:
machineName="verdin-imx8mp";
#get data for "device_provides" field. You can send multiple curl requests to the mender API during
# an update, but each must have IDENTICAL values in the "device_provides" field. If you mess that
# up, mender's web UI just shows a mysterious error: "error":"Device provided conflicting request data"
#The mender-update binary sends it's request around line 100 of this source code file:
# git/src/mender-update/deployments/deployments.cpp
# so we have to match that identically. Every element of the "show-provides" command must be included.
#This comment on this forum posted tipped me off to this: https://hub.mender.io/t/using-api-to-get-artifact-name/5104/3
output=$(mender-update show-provides);
#AI for the win. It made this crazy awk string for me to convert that output to JSON :)
formatted_output=$(echo "$output" | awk -F= '
{
key = $1
value = $2
# Remove leading and trailing whitespace from key and value
sub(/^[\t ]+/, "", key)
sub(/[\t ]+$/, "", value)
# Print in the desired format
if (NR > 1) printf ",\n"
printf " \"%s\": \"%s\"", key, value
}
END { printf "\n }" } # Close the last object with newline for readability
')
echo $formatted_output;
#sleep here. Without sleep, this script worked when manually running/testing. But with an actual
# mender update, the $updtAvailFile was blank. Seems like everything was happening too fast when
# the process started and the server... idk wasn't ready? Not sure what the server is doing in
# the back end exactly. But I found that a 3 second sleep (or less) was too short and resulted in
# a blank file. 4 seconds or more was long enough that the file had data.
sleep 4;
curl -X POST \
https://hosted.mender.io/api/devices/v2/deployments/device/deployments/next \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $jwt" \
-o $updtAvailFile \
-d "{\"device_provides\": {
\"device_type\":\"$machineName\",
$formatted_output,
\"update_control_map\": false \
}";
#
#echo "Update avail file says:"
#cat $updtAvailFile >&2 #for debugging
# Check if the file is empty
if [ ! -s "$updtAvailFile" ]; then
echo "$updtAvailFile is empty. No pending update."
rm -rf "$updtAvailFile" #so user interface app isn't confused
exit 0
fi
echo "$updtAvailFile not empty"
#Forced update: For emergencies, like if we find that an old image is unsafe, or insecure, or
# simply has a screen driver bug and the user can't see the update button to click it.
# Without a 'force' option, the user will always have to click the update button to update
# software. And the update logic lives in the old image, not the new one we're pushing. So we would
# have no way to override. This if statement checks the incoming update to see if the
# artifact_name has the string "force" in it. Then the device could apply the update with no user
# interaction if it sees that string. Ideally it's a case where we only push force artifacts when
# we are on the phone with the customer and they confirm they're in a safe location
#
# To take an existing .mender file and edit it to have “force” in the artifact name, do this:
# re-generate a mender artifact with the mender-artifact command on the host PC. It’s not in Ubuntu’s apt repo, but
# yocto builds it under tmp/work/x86_64-linux/mender-artifact-native/3.11.2/build/bin/mender-artifact.
# Can also download from mender’s website: https://docs.mender.io/downloads#mender-artifact
# Use this command sequence format to use mender-artifact:
# cd <to wherever you've extracted $ARTIFACTIMG_NAME.ext4 from the existing mender artifact;
# MENDER_ARTIFACT_NAME="v0.2.0-alpha-129-g9a3a2a9-2025.01.18_13.57.41-force"; #edit for your version. don't forget "force"!!!
# MENDER_DEVICE_TYPES_COMPATIBLE="verdin-imx8mp";
# extra_args="-t $MENDER_DEVICE_TYPES_COMPATIBLE";
# ARTIFACTIMG_NAME="My-Image-Name-verdin-imx8mp";
# mender-artifact write rootfs-image -n $MENDER_ARTIFACT_NAME $extra_args -f $ARTIFACTIMG_NAME.ext4 -o $ARTIFACTIMG_NAME$MENDER_ARTIFACT_NAME.mender
if jq -r ".artifact.artifact_name" $updtAvailFile | grep -qi "force" ; then
echo "Forced update detected."
touch $updtApprovedFile;
fi
#Above we let user interface app know an update is available by passing the curl info to
# $updtAvailFile
#Now wait until user interface app discusses with the user and user wants to update
#TODO: Mender says this will time out after 24hrs. Be sure to test this and handle
# it appropriately. Will have to ensure we have a trigger to prompt the user again
# later, rather than just never checking for an update ever again
echo "In state $(basename "$0"), waiting for $updtApprovedFile... " >&2
while [ ! -f "$updtApprovedFile" ]; do
if [ -f "$updtAlwaysApprovedFile" ]; then
echo "$updtAlwaysApprovedFile detected. Proceeding with update download & install..." >&2
break;
else
sleep 1;
fi;
done
# File has appeared, now do some stuff
echo "$updtApprovedFile detected. Proceeding with update download & install..." >&2
rm -rf $updtAvailFile
and 2) mender-back-up-data.sh:
#!/bin/bash
#IMPORTANT LOGGING NOTE:
#Mender captures the standard error (but not standard out) stream from state scripts. The standard
# error stream from state scripts is stored as part of the Mender deployment log, so it becomes
# available locally on the client as well as reported to the server (if the deployment fails) to
# ease diagnostics.
#Thus the state scripts should be written so they output diagnostics information to standard error,
# especially in case of failure (returning 1). The maximum size of the log is 10KiB per state
# script, anything above this volume will be truncated.
#Anything we want to save needs to be copied from the current A/B rootfs to the persistent /data partition
persistDir="/data/persist"
newRootMountDir="/tmp/newRootfs"
defaultRestoreDir="/restore"
# Function create a symlink to persistent data for a given directory or file
backup_dir_or_file() {
# realpath --no-symlinks converts ~/myFile to /home/user1/myFile
dirOrFile=$(realpath --no-symlinks --canonicalize-missing "$1")
#first check if the file/dir exists on the current rootfs. If it does not, there is nothing
# to back up! A good example is /etc/NetworkManager/system-connections - that's where wifi
# networks are saved. So it doesn't exist until the first time a user connects to wifi. If we
# make that a symlink before nmcli ever creates the folder itself, the symlink will point to
# a nonexistent directory and nmcli will get confused & error when trying to create the dir
if [ -e "$dirOrFile" ]; then
#if this device is fresh and has never been updated, these files aren't yet symlinks
# this also happens if we add a new file to the backup list and update an old device in the field
alreadyLinked=false
if [[ -L "$dirOrFile" ]]; then
real_path=$(readlink "$dirOrFile")
if [[ $real_path == $persistDir* ]]; then
echo "mender-back-up-data.sh: $dirOrFile is already a symlink to $real_path " >&2
alreadyLinked=true
fi
fi
if [ "$alreadyLinked" = false ]; then #then it's the devices first run and isn't a link yet so copy files
rm -rf "$persistDir$dirOrFile" #make sure nothing is already there
echo "mender-back-up-data.sh: mkdir -p \"$(dirname "$persistDir$dirOrFile")\" " >&2
mkdir -p "$(dirname "$persistDir$dirOrFile")" #if parent dir doesn't exist
echo "mender-back-up-data.sh: running cp -rp \"$dirOrFile\" \"$persistDir$dirOrFile\" " >&2
cp -rp "$dirOrFile" "$persistDir$dirOrFile" #copy to persist data. leave in old rootfs in case new one has issues
fi
owner=$(stat -c '%U' "$persistDir$dirOrFile")
group=$(stat -c '%G' "$persistDir$dirOrFile")
echo "mender-back-up-data.sh: owner is $owner:$group for \"$persistDir$dirOrFile\" " >&2
# if [ -n "$user" ]; then #if username supplied
# # if [ "$owner:$group" != "$2" ]; then
# # echo "mender-back-up-data.sh: chowning \"$persistDir$dirOrFile\" to $owner:$group" >&2
# # chown -R "$persistDir$dirOrFile" "$owner:$group"
# # fi
# else
# echo "mender-back-up-data.sh: no user arg supplied for \"$persistDir$dirOrFile\" " >&2
# fi
#now put a symlink in the new rootfs
# Remove the original directory before making symlink. Remove by moving it to /restore
# just in case we ever run into issues where we need to restore what was originally in the
# default image
echo "mender-back-up-data.sh: running mkdir -p \"$newRootMountDir$defaultRestoreDir$dirOrFile\" " >&2
mkdir -p "$newRootMountDir$defaultRestoreDir$dirOrFile"
#TODO: sometimes this mv statement fails, like for /etc/ssh/ssh_host* keys - they don't exist in a fresh image
echo "mender-back-up-data.sh: running mv \"$newRootMountDir$dirOrFile\" \"$newRootMountDir$defaultRestoreDir$(dirname "$dirOrFile")\" " >&2
mv "$newRootMountDir$dirOrFile" "$newRootMountDir$defaultRestoreDir$(dirname "$dirOrFile")"
# Create a symbolic link pointing to /data/persist/ with the real path of dirOrFile
echo "mender-back-up-data.sh: running mkdir -p \"$(dirname "$newRootMountDir$dirOrFile")\" " >&2
mkdir -p "$(dirname "$newRootMountDir$dirOrFile")" #in case parent folder doesn't exist
echo "mender-back-up-data.sh: running ln -s \"$persistDir$dirOrFile\" \""$newRootMountDir$dirOrFile"\"" >&2
ln -s "$persistDir$dirOrFile" ""$newRootMountDir$dirOrFile""
echo "" >&2
else
echo "mender-back-up-data.sh: \"$dirOrFile\" doesnt exist." >&2
fi
}
restore_default_dir_or_file() {
# realpath --no-symlinks converts ~/myFile to /home/user1/myFile
#TOTO: somehow "/data/persist" is sneaking into the below paths. Debug later
dirOrFile=$(realpath --no-symlinks --canonicalize-missing "$1")
echo "mender-back-up-data.sh: Restoring $newRootMountDir$defaultRestoreDir$dirOrFile" >&2
echo "mender-back-up-data.sh: running rm -rf \"$newRootMountDir$dirOrFile\" " >&2
rm -rf "$newRootMountDir$dirOrFile"
if [ -e "$newRootMountDir$defaultRestoreDir$dirOrFile" ]; then
echo "mender-back-up-data.sh: running mkdir -p \"$(dirname "$newRootMountDir$dirOrFile")\" " >&2
mkdir -p "$(dirname "$newRootMountDir$dirOrFile")" #if parent dir doesn't exist
echo "mender-back-up-data.sh: running cp -rp \"$newRootMountDir$defaultRestoreDir$dirOrFile\" \"$newRootMountDir$dirOrFile\" " >&2
cp -rp "$newRootMountDir$defaultRestoreDir$dirOrFile" "$newRootMountDir$dirOrFile"
else
echo "mender-back-up-data.sh: error \"$newRootMountDir$defaultRestoreDir$dirOrFile\" doesnt exist " >&2
fi
echo "" >&2
}
echo "mender-back-up-data.sh: Starting data back up now." >&2
newRootMountDir="/tmp/newRootfs"
#mount the newly updated partition (one we're not running on)
currentRootfs=$(findmnt -n -o SOURCE / | tr -d '\n') #will be /dev/mmcblk2p2 or /dev/mmcblk2p3
if [[ "$currentRootfs" == "/dev/mmcblk2p2" ]]; then
newRootfs="/dev/mmcblk2p3"
elif [[ "$currentRootfs" == "/dev/mmcblk2p3" ]]; then
newRootfs="/dev/mmcblk2p2"
else
echo "Error: Current root filesystem $currentRootfs is not on expected device (/dev/mmcblk2p2 or /dev/mmcblk2p3)" >&2
exit 1
fi
echo "mender-back-up-data.sh: currentRootfs: $currentRootfs"
echo "mender-back-up-data.sh: newRootfs: $newRootfs"
echo "" >&2
mkdir -p $newRootMountDir #/tmp/newRootfs
mount $newRootfs $newRootMountDir
#back up home directories of all users, not just root
#TODO: make this a for loop that loops thru all /home/* users on the device. Can't just link
# /home and let linux create user folders - it won't. We have to make the /home/weston,
# /home/root, etc folders on the /data/persist drive
backup_dir_or_file "/home/weston"
#root's home directory is /root
backup_dir_or_file "/root"
#back up ssh keys for the device
backup_dir_or_file "/etc/ssh/ssh_host_rsa_key.pub"
backup_dir_or_file "/etc/ssh/ssh_host_rsa_key"
backup_dir_or_file "/etc/ssh/ssh_host_ecdsa_key.pub"
backup_dir_or_file "/etc/ssh/ssh_host_ecdsa_key"
backup_dir_or_file "/etc/ssh/ssh_host_ed25519_key.pub"
backup_dir_or_file "/etc/ssh/ssh_host_ed25519_key"
#RDP certificate & key
backup_dir_or_file "/etc/xdg/weston/weston-rdp.key"
backup_dir_or_file "/etc/xdg/weston/weston-rdp.crt"
#Saved wifi networks
backup_dir_or_file "/etc/NetworkManager/system-connections"
# We want to keep most of the user data in /home, but there is a "default-project" file from our
# custom application that we want to overwrite old versions and use the latest from the incoming
# image.
#WARING: do not use "~/" because realpath gets confused by the symlinks. Plus this script runs as
# root so even $HOME won't reflect the weston user's HOME dir
restore_default_dir_or_file "/home/weston/default-project"
#This boot.scr copy is temporary only if we make a bootloader change that we want to force.
# /uboot in the image is part of mmcblk2p1, which isn't an A/B partition and isn't
# normally updated by mender. But we can optionally choose to copy this over in a mender
# state script if there is a critical update. But we must be careful. Mender can't rollback
# changes to this file so we must be extra sure it won't cause boot failures/bricked devices
cp -fp $newRootMountDir/boot/boot.scr.bak /uboot/boot.scr
sync
umount $newRootfs
echo "mender-back-up-data.sh: Completed data back up." >&2
exit 0