This update module enables zero-reboot application updates on immutable systems by leveraging systemd’s system extension (systemd-sysext) mechanism. It allows you to deliver SquashFS extensions containing updated binaries, libraries, or configurations to a read-only /usr without requiring a full system restart or A/B rootfs flip.
Prepare the development environment on your workstation
This section describes how to set up your development environment on your workstation to package system extension artifacts.
All commands outlined in this section should be run in the development environment.
Prerequisites
-
mender-artifact version 3.0.0 or later (to build .mender artifacts).
-
mksquashfs (squashfs-tools) to create .raw SquashFS extension images.
On Debian/Ubuntu hosts, install squashfs-tools:
sudo apt update && sudo apt install -y squashfs-tools
Note: You can complete the host preparation and artifact generation steps on macOS as well. To install mksquashfs on macOS via Homebrew, run brew install squashfs instead of apt install squashfs-tools.
Configure the Target Device
Before deploying artifacts, prepare the target device by ensuring systemd-sysext is active, setting up persistent storage, installing the update module, and configuring /usr immutability.
Note: The target device must be running the Mender Client (v4.0.0 or later), but it does not require an A/B dual-partition layout as this update module runs entirely in Standalone or Connected mode on single-partition or immutable rootfs setups.
Step 1: Enable systemd-sysext
Ensure systemd-sysext is installed and enabled to automatically merge extensions on system boot:
# Verify systemd-sysext is present (included with systemd)
systemd-sysext --version
# Enable the service so extensions persist and refresh across reboots
sudo systemctl enable systemd-sysext.service
Step 2: Configure Persistent Extension Storage
Set up persistent storage on /var for extension images so they persist across updates without requiring write access to /usr:
# 1. Create a persistent directory on writable /var and add a placeholder file
# (The placeholder prevents systemd-sysext.service ConditionDirectoryNotEmpty from failing)
sudo mkdir -p /var/local/extensions
sudo touch /var/local/extensions/.keep
# 2. Symlink /var/lib/extensions to persistent storage
sudo mkdir -p /var/lib
sudo rm -rf /var/lib/extensions
sudo ln -s /var/local/extensions /var/lib/extensions
Step 3: Install the Mender Update Module
Install the sysext update module to /usr/share/mender/modules/v3/sysext.
# 1. Ensure the module directory exists
sudo mkdir -p /usr/share/mender/modules/v3
# 2. Write the dynamic update module script
sudo tee /usr/share/mender/modules/v3/sysext > /dev/null << 'EOF'
#!/bin/sh
set -e
STATE="$1"
case "$STATE" in
NeedsArtifactReboot)
echo "No"
;;
SupportsRollback)
echo "Yes"
;;
ArtifactInstall|ArtifactInstall_Enter)
# 1. Locate incoming .raw payload in Mender's temporary workspace
PAYLOAD=$(find . -type f -name "*.raw" | head -n 1)
if [ -n "$PAYLOAD" ]; then
# Extract raw filename dynamically (e.g., extension-v1.0.raw)
FILENAME=$(basename "$PAYLOAD")
# 2. Unmerge active extensions first to safely detach loop devices
systemd-sysext unmerge || true
# 3. Clean previous raw extensions in storage to avoid stale overlays
rm -f /var/local/extensions/*.raw
# 4. Copy new payload using its exact original filename
cp "$PAYLOAD" "/var/local/extensions/$FILENAME"
else
echo "Error: No .raw payload found in Mender working directory ($PWD)" >&2
exit 1
fi
# 5. Trigger live merge over /usr
systemd-sysext refresh
;;
ArtifactRollback|ArtifactRollback_Enter)
systemd-sysext unmerge || true
rm -f /var/local/extensions/*.raw
systemd-sysext refresh
;;
esac
exit 0
EOF
# 3. Make the update module executable
sudo chmod +x /usr/share/mender/modules/v3/sysext
Step 4: Configure Read-Only /usr and Lock System Down
Now that the system services and update module are configured, enforce system immutability by mounting /usr read-only via a systemd mount unit.
# 1. Create systemd mount unit to bind mount /usr as read-only
sudo tee /etc/systemd/system/usr.mount > /dev/null << 'EOF'
[Unit]
Description=Mount /usr Read-Only
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=systemd-remount-fs.service
[Mount]
What=/usr
Where=/usr
Type=none
Options=bind,ro
[Install]
WantedBy=local-fs.target
EOF
# 2. Reload systemd daemon and enable unit
sudo systemctl daemon-reload
sudo systemctl enable usr.mount
# 3. Activate the read-only mount immediately
sudo systemctl start usr.mount
# 4. Verify /usr is write-protected
sudo touch /usr/testfile
Expected output:
touch: cannot touch ‘/usr/testfile’: Read-only file system
Create Mender Artifacts
Follow these steps to structure a system extension payload, create the SquashFS image, and package it into a .mender artifact targeting the sysext update module.
Step 1: Create the Payload Workspace
Create the directory structure for your extension. Notice that the release metadata file inside /usr/lib/extension-release.d/ uses the prefix my-extension — which matches the .raw image name (my-extension.raw) built in the next step.
# 1. Create directory structure for application files and metadata
mkdir -p ./my-extension/usr/bin
mkdir -p ./my-extension/usr/lib/extension-release.d
# 2. Add your binary or application payload
cat << 'EOF' > ./my-extension/usr/bin/my-app
#!/bin/sh
echo "Hello from systemd-sysext via Mender!"
EOF
chmod +x ./my-extension/usr/bin/my-app
# 3. Add required extension-release metadata matching the extension name
cat << 'EOF' > ./my-extension/usr/lib/extension-release.d/extension-release.my-extension
ID=ubuntu
VERSION_ID=24.04
EOF
Note: The extension-release.my-extension file inside /usr/lib/extension-release.d/ must match the target system’s /etc/os-release fields (ID and VERSION_ID), and the filename suffix (my-extension) must match the name of the SquashFS .raw file (my-extension.raw).
Step 2: Build the SquashFS Image
Compress the payload folder into a raw SquashFS image file named my-extension.raw:
mksquashfs ./my-extension ./my-extension.raw -noappend
Step 3: Package the Mender Artifact
Use mender-artifact to generate the final update package targeting the sysext module type and your target device_type:
mender-artifact write module-image \\
-t qemu-ubuntu-x86_64 \\
-n my-extension-v1.0 \\
-T sysext \\
-f ./my-extension.raw \\
-o ./my-extension-v1.0.mender
You can deploy this Artifact using the Mender Server (upload under Releases in the UI) or locally on standalone clients:
sudo mender-update install ./my-extension-v1.0.mender
sudo mender-update commit
Artifact technical details
The artifact wraps a single SquashFS (.raw) payload that is placed into the persistent extension storage on the device (/var/lib/extensions/).
Key Behaviors
-
Zero Reboot: Reports NeedsArtifactReboot = No to Mender, ensuring immediate execution without restarting the target.
-
Atomic Overlay: Uses systemd-sysext refresh to dynamically overlay the SquashFS image onto /usr in user-space via OverlayFS.
-
Immutability Friendly: Target rootfs /usr remains strictly read-only; extensions reside safely in writable persistent storage (/var/local/extensions symlinked to /var/lib/extensions).
Verify the Deployment
Once the artifact installation completes and you run sudo mender-update commit, verify that the system extension was successfully merged into /usr:
Step 1: Check System Extension Status
Run systemd-sysext status to confirm that systemd-sysext recognized and mounted the .raw image:
systemd-sysext status
Expected output:
HIERARCHY EXTENSION VERSION TYPE
/usr my-extension - raw
Step 2: Execute the Installed Application
Verify that the binary inside the SquashFS payload is now visible in /usr/bin over the read-only filesystem and can be executed directly:
# 1. Verify path resolution
which my-app
# 2. Run the application
my-app
Expected output:
/usr/bin/my-app
Hello from systemd-sysext via Mender!