Mender Update Module for systemd-sysext: Zero-Reboot System Extensions

Tutorials Mender
1

This update module enables zero-reboot application updates on immutable systems by leveraging systemd’s system extension (systemd-sysext) mechanism. It allows you to deliver SquashFS extensions containing updated binaries, libraries, or configurations to a read-only /usr without requiring a full system restart or A/B rootfs flip.

Prepare the development environment on your workstation

This section describes how to set up your development environment on your workstation to package system extension artifacts.

All commands outlined in this section should be run in the development environment.

Prerequisites

  • mender-artifact version 3.0.0 or later (to build .mender artifacts).

  • mksquashfs (squashfs-tools) to create .raw SquashFS extension images.

On Debian/Ubuntu hosts, install squashfs-tools:

sudo apt update && sudo apt install -y squashfs-tools

Note: You can complete the host preparation and artifact generation steps on macOS as well. To install mksquashfs on macOS via Homebrew, run brew install squashfs instead of apt install squashfs-tools.

Configure the Target Device

Before deploying artifacts, prepare the target device by ensuring systemd-sysext is active, setting up persistent storage, installing the update module, and configuring /usr immutability.

Note: The target device must be running the Mender Client (v4.0.0 or later), but it does not require an A/B dual-partition layout as this update module runs entirely in Standalone or Connected mode on single-partition or immutable rootfs setups.

Step 1: Enable systemd-sysext

Ensure systemd-sysext is installed and enabled to automatically merge extensions on system boot:

# Verify systemd-sysext is present (included with systemd)

systemd-sysext --version

# Enable the service so extensions persist and refresh across reboots

sudo systemctl enable systemd-sysext.service

Step 2: Configure Persistent Extension Storage

Set up persistent storage on /var for extension images so they persist across updates without requiring write access to /usr:

# 1. Create a persistent directory on writable /var and add a placeholder file

# (The placeholder prevents systemd-sysext.service ConditionDirectoryNotEmpty from failing)

sudo mkdir -p /var/local/extensions

sudo touch /var/local/extensions/.keep



# 2. Symlink /var/lib/extensions to persistent storage

sudo mkdir -p /var/lib

sudo rm -rf /var/lib/extensions

sudo ln -s /var/local/extensions /var/lib/extensions

Step 3: Install the Mender Update Module

Install the sysext update module to /usr/share/mender/modules/v3/sysext.

# 1. Ensure the module directory exists
sudo mkdir -p /usr/share/mender/modules/v3

# 2. Write the dynamic update module script
sudo tee /usr/share/mender/modules/v3/sysext > /dev/null << 'EOF'
#!/bin/sh
#
# Mender Update Module: sysext
#
# Installs a systemd-sysext image (.raw) and merges it onto /usr live,
# without a reboot. The previously active image is kept as a backup until
# the update is committed, so a rollback restores the previous version
# instead of falling back to the bare base OS.
#
# Install to: /usr/share/mender/modules/v3/sysext   (chmod 755)
#
# Directories:
#   EXT_DIR  - active extension images (/var/lib/extensions -> /var/local/extensions)
#   BAK_DIR  - previous images, kept until ArtifactCommit.
#              Deliberately OUTSIDE EXT_DIR: systemd-sysext treats every
#              *.raw file and sub-directory there as an extension.
#
# Optional health check:
#   If HEALTHCHECK exists and is executable, it runs after the merge.
#   A non-zero exit fails ArtifactInstall and Mender rolls back.
#
# Limitation: manages all *.raw images in EXT_DIR as one unit
# (one extension per device, like the original module).

set -eu

STATE="${1:-}"
FILES="${2:-.}"

EXT_DIR="${SYSEXT_EXT_DIR:-/var/local/extensions}"
BAK_DIR="${SYSEXT_BAK_DIR:-/var/local/extensions-backup}"
HEALTHCHECK="${SYSEXT_HEALTHCHECK:-/etc/mender/sysext-healthcheck}"
RELEASE_DIR="${SYSEXT_RELEASE_DIR:-/usr/lib/extension-release.d}"

log() { echo "sysext: $*" >&2; }

# Move every *.raw from $1 to $2 (no-op if there are none).
move_images() {
    for img in "$1"/*.raw; do
        [ -e "$img" ] || continue
        mv -f "$img" "$2/"
    done
}

# Delete every *.raw in $1 (no-op if there are none).
remove_images() {
    for img in "$1"/*.raw; do
        [ -e "$img" ] || continue
        rm -f "$img"
    done
}

case "$STATE" in
    NeedsArtifactReboot)
        echo "No"
        ;;

    SupportsRollback)
        echo "Yes"
        ;;

    ArtifactInstall)
        # 1. Locate the incoming .raw payload Mender unpacked for us
        PAYLOAD=$(find "$FILES" -type f -name '*.raw' | head -n 1)
        if [ -z "$PAYLOAD" ]; then
            log "no .raw payload found in $FILES"
            exit 1
        fi
        FILENAME=$(basename "$PAYLOAD")
        NAME="${FILENAME%.raw}"
        log "installing $FILENAME"

        mkdir -p "$EXT_DIR" "$BAK_DIR"

        # 2. Detach the current overlay before touching the images
        systemd-sysext unmerge || true

        # 3. Back up the running image(s); drop any stale backup first
        remove_images "$BAK_DIR"
        move_images "$EXT_DIR" "$BAK_DIR"

        # 4. Copy the new image in under a temporary name, then rename,
        #    so a power cut never leaves a half-written .raw in EXT_DIR
        cp "$PAYLOAD" "$EXT_DIR/$FILENAME.tmp"
        sync
        mv -f "$EXT_DIR/$FILENAME.tmp" "$EXT_DIR/$FILENAME"

        # 5. Merge onto /usr live
        systemd-sysext refresh

        # 6. Verify the merge actually happened. systemd-sysext silently
        #    skips images whose extension-release does not match the host.
        if [ ! -e "$RELEASE_DIR/extension-release.$NAME" ]; then
            log "$NAME is not merged (extension-release mismatch or name mismatch)"
            exit 1
        fi

        # 7. Optional application health check
        if [ -x "$HEALTHCHECK" ]; then
            log "running health check $HEALTHCHECK"
            if ! "$HEALTHCHECK"; then
                log "health check failed"
                exit 1
            fi
        fi

        log "$NAME merged"
        ;;

    ArtifactCommit)
        # New image is good: the backup is no longer needed
        remove_images "$BAK_DIR"
        log "committed, backup removed"
        ;;

    ArtifactRollback)
        mkdir -p "$EXT_DIR" "$BAK_DIR"
        systemd-sysext unmerge || true

        # Remove the new image and restore the previous one (if any).
        # With no backup (first ever install) this returns to the base OS.
        remove_images "$EXT_DIR"
        for img in "$EXT_DIR"/*.raw.tmp; do
            [ -e "$img" ] || continue
            rm -f "$img"
        done
        move_images "$BAK_DIR" "$EXT_DIR"

        systemd-sysext refresh
        log "rolled back"
        ;;

    *)
        # Download, ArtifactVerifyReboot, ArtifactFailure, Cleanup, ...:
        # nothing to do. Mender cleans up its own working directory.
        ;;
esac

exit 0
EOF

# 3. Make the update module executable
sudo chmod +x /usr/share/mender/modules/v3/sysext

Step 4: Configure Read-Only /usr and Lock System Down

Now that the system services and update module are configured, enforce system immutability by mounting /usr read-only via a systemd mount unit.

# 1. Create systemd mount unit to bind mount /usr as read-only
sudo tee /etc/systemd/system/usr.mount > /dev/null << 'EOF'
[Unit]
Description=Mount /usr Read-Only
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
After=systemd-remount-fs.service

[Mount]
What=/usr
Where=/usr
Type=none
Options=bind,ro

[Install]
WantedBy=local-fs.target
EOF

# 2. Reload systemd daemon and enable unit
sudo systemctl daemon-reload
sudo systemctl enable usr.mount

# 3. Activate the read-only mount immediately
sudo systemctl start usr.mount

# 4. Verify /usr is write-protected
sudo touch /usr/testfile

Expected output:
touch: cannot touch ‘/usr/testfile’: Read-only file system

Create Mender Artifacts

Follow these steps to structure a system extension payload, create the SquashFS image, and package it into a .mender artifact targeting the sysext update module.

Step 1: Create the Payload Workspace

Create the directory structure for your extension. Notice that the release metadata file inside /usr/lib/extension-release.d/ uses the prefix my-extension — which matches the .raw image name (my-extension.raw) built in the next step.

# 1. Create directory structure for application files and metadata

mkdir -p ./my-extension/usr/bin

mkdir -p ./my-extension/usr/lib/extension-release.d



# 2. Add your binary or application payload

cat << 'EOF' > ./my-extension/usr/bin/my-app

#!/bin/sh

echo "Hello from systemd-sysext via Mender!"

EOF



chmod +x ./my-extension/usr/bin/my-app



# 3. Add required extension-release metadata matching the extension name

cat << 'EOF' > ./my-extension/usr/lib/extension-release.d/extension-release.my-extension

ID=ubuntu

VERSION_ID=24.04

EOF

Note: The extension-release.my-extension file inside /usr/lib/extension-release.d/ must match the target system’s /etc/os-release fields (ID and VERSION_ID), and the filename suffix (my-extension) must match the name of the SquashFS .raw file (my-extension.raw).

Step 2: Build the SquashFS Image

Compress the payload folder into a raw SquashFS image file named my-extension.raw:

mksquashfs ./my-extension ./my-extension.raw -noappend

Step 3: Package the Mender Artifact

Use mender-artifact to generate the final update package targeting the sysext module type and your target device_type:

mender-artifact write module-image \\

    -t qemu-ubuntu-x86_64 \\

    -n my-extension-v1.0 \\

    -T sysext \\

    -f ./my-extension.raw \\

    -o ./my-extension-v1.0.mender

You can deploy this Artifact using the Mender Server (upload under Releases in the UI) or locally on standalone clients:

sudo mender-update install ./my-extension-v1.0.mender

sudo mender-update commit

Artifact technical details

The artifact wraps a single SquashFS (.raw) payload that is placed into the persistent extension storage on the device (/var/lib/extensions/).

Key Behaviors

  • Zero Reboot: Reports NeedsArtifactReboot = No to Mender, ensuring immediate execution without restarting the target.

  • Atomic Overlay: Uses systemd-sysext refresh to dynamically overlay the SquashFS image onto /usr in user-space via OverlayFS.

  • Immutability Friendly: Target rootfs /usr remains strictly read-only; extensions reside safely in writable persistent storage (/var/local/extensions symlinked to /var/lib/extensions).

Verify the Deployment

Once the artifact installation completes and you run sudo mender-update commit, verify that the system extension was successfully merged into /usr:

Step 1: Check System Extension Status

Run systemd-sysext status to confirm that systemd-sysext recognized and mounted the .raw image:

systemd-sysext status

Expected output:

HIERARCHY EXTENSION VERSION TYPE

/usr my-extension - raw

Step 2: Execute the Installed Application

Verify that the binary inside the SquashFS payload is now visible in /usr/bin over the read-only filesystem and can be executed directly:

# 1. Verify path resolution

which my-app


# 2. Run the application

my-app

Expected output:

/usr/bin/my-app

Hello from systemd-sysext via Mender!