Composable Updates: Independent Update Domains for OS, Applications, and Containers

Solutions
1

The problem

An ML team has a new inference model ready every Wednesday. The platform team rebuilds the OS image once a quarter. The applications team ships a new microservice container every other afternoon. All three live on the same fleet of devices.

Ship that as one image and the slowest cadence wins. Every change, be it a one-line config tweak, a 30 MB model swap, or a 2 KB feature flag, rides along with the next kernel rollout. The model that was ready on Wednesday goes out in March, two iterations stale.

The pattern shows up everywhere. A medical imaging vendor wants to push a corrected DICOM parser without revalidating the bootloader. An industrial gateway operator wants the same security-patched kernel on every site, with site-specific containerized workloads on top. A connected-car program wants OTA updates that touch the infotainment stack and never the safety-critical layer underneath.

Composable updates do the opposite. The OS updates atomically with bootloader-backed rollback, applications through a lighter mechanism that survives OS swaps, containers on their own clock. Each layer declares what it provides and what it depends on, and an artifact that doesn’t fit the device it lands on gets rejected before a single byte is installed.

In this article we’ll walk through composable updates with a concrete example that exercises three update domains end-to-end, using Mender’s Update Modules and the provides/depends metadata system. The configuration snippets and command output that follow are all from a small reference setup that’s described inline; everything you need to reproduce or adapt it is in this article.

What composable updates are

The basic idea: treat the software on a device as a small number of distinct update domains. Each domain gets its own update mechanism, its own version namespace, and its own rollback strategy. The domains share a single coordination layer, Mender’s provides/depends metadata, which keeps incompatible combinations from reaching a device.

Domain Typical contents Update mechanism Rollback
Rootfs (A/B) Kernel, init, core libraries, container runtime Atomic write to inactive partition + bootloader switch Bootloader reverts on failed health check
Persistent applications Business logic, ML models, configuration File install to /data/apps/<name>/ via update module Backup/restore from the same module
Containers Microservices, rapidly-iterating application code Docker / docker-compose image pull Image rollback by the update module

We’ll cover the three domains the example below exercises. A composable strategy can include others, such as directory-overlay for runtime-mutable files on an otherwise read-only rootfs, or custom modules for vendor-specific mechanisms. The same principles apply.

Architecture

Composable updates: storage layout and update domains
Composable updates: storage layout and update domains1640×920 78.4 KB

The rootfs A/B pair handles atomic OS updates; the data partition (/data) is the persistence layer that survives every rootfs swap. The right-hand column shows the namespace each domain uses for version tracking.

The data partition does the heavy lifting. Anything that needs to outlive an A/B switch lives there: persistent applications under /data/apps/, container images under /data/docker/, the Mender Client’s own database under /data/mender/. The rootfs partitions are otherwise replaced wholesale: new image to the inactive side, bootloader flag flip, reboot, health check, commit or revert.

A rootfs update is heavyweight: reboot, deep integration, verified rollback. A persistent-app update is lightweight: no reboot, backup-and-restore inside the running system. A container update is lighter still: a docker pull against the local data partition.

Update cadence by domain (one year)
Update cadence by domain (one year)1760×720 40.8 KB

Three lanes on a shared time axis: rootfs quarterly, applications monthly, containers weekly. Ship all three as one image and every cadence collapses to the slowest one.

The coordination work — namespace discipline, dependency declarations, domain ownership — is the price you pay for every domain running at its own speed.

The coordination layer: provides and depends

Independent domains only work if the device can refuse incompatible combinations. Every Mender Artifact declares what it provides and what it depends on. Before installation begins, the Mender Client checks the incoming artifact’s depends against the device’s current provides.

Let’s see what mender-update show-provides returns on the example device:

rootfs-image.version=composable-demo-1.0
rootfs-image.api.version=2.0
rootfs-image.feature.docker=true
rootfs-image.feature.persistent-app=true
rootfs-image.python.version=3.12
data-partition.demo-app.version=1.0

Two namespaces show up: rootfs-image.* for capabilities advertised by the OS, data-partition.* for things tracked against the persistence layer. The distinction matters.

Namespaces and what they survive

A rootfs artifact clears everything in rootfs-image.* before installing the new set, which is the right thing to do, since the old OS’s claims about itself are stale. Anything in data-partition.* survives the swap, because the data partition itself does.

This is why --software-filesystem data-partition matters for persistent applications. Without it, the artifact’s provides land in rootfs-image.* and get wiped on the next OS update. With it, they land in data-partition.* and persist across rootfs cycles. We’ll come back to this when we build the persistent-app artifact.

Evaluating depends against provides

Provides/depends evaluation: accepted versus rejected
Provides/depends evaluation: accepted versus rejected1920×1120 91 KB

An artifact with --depends rootfs-image.python.version:3.12 installs on the example device. An artifact with --depends rootfs-image.cuda.version:11.8 is rejected with Missing 'rootfs-image.cuda.version' in provides, required by artifact depends. System not modified. Both verdicts are observed when running the example configured below.

In the artifact-build commands, that looks like this. The rootfs side declares what it provides:

mender-artifact write rootfs-image \
    -t qemux86-64 \
    -n composable-demo-1.0 \
    -f rootfs.ext4 \
    -o composable-demo-1.0.mender \
    --software-version 1.0 \
    --provides rootfs-image.python.version:3.12 \
    --provides rootfs-image.api.version:2.0 \
    --provides rootfs-image.feature.docker:true

And the application side declares what it requires:

mender-artifact write module-image \
    -T persistent-app \
    -t qemux86-64 \
    -n analytics-engine-3.0 \
    -o analytics-engine-3.0.mender \
    --software-filesystem data-partition \
    --software-name analytics-engine \
    --software-version 3.0 \
    --depends rootfs-image.python.version:3.12 \
    --depends rootfs-image.feature.docker:true \
    -f payload/

Deploy analytics-engine-3.0 to a device whose rootfs reports rootfs-image.python.version=3.11 and the client rejects the deployment before the install state runs. No operator intervention, no half-installed application.

The same mechanism handles feature gating (--provides rootfs-image.feature.tpm:true), API-compatibility coordination during phased rollouts, and cross-component dependencies between applications. The reference for the full set of flags is in Software Versioning and Combining System and Application Updates.

Domain 1: Rootfs (A/B)

This is the strongest update guarantee Mender offers. The standard rootfs-image Update Module (what Mender’s taxonomy calls an Operating System update) writes the new image to the inactive partition, switches the bootloader on reboot, and reverts automatically if the health check or the boot itself fails. The active partition stays untouched until the flag flips, so power loss during the write is non-fatal.

Use this domain for anything that needs a reboot or has deep OS integration: kernel, init system and its units, core shared libraries, the container runtime, security-critical components on a verified-boot chain. The rootfs Artifact also advertises capabilities for the upper domains to depend on:

MENDER_ARTIFACT_PROVIDES = " \
    rootfs-image.python.version:3.12 \
    rootfs-image.api.version:2.0 \
    rootfs-image.feature.docker:true \
    rootfs-image.feature.persistent-app:true \
"

Set in local.conf, these become the device’s rootfs-image.* provides on first boot.

Domain 2: Persistent applications

Some files have to survive a rootfs swap. That’s what the data partition is for. A binary at /opt/myapp/ is gone after the next A/B switch; the same binary at /data/apps/myapp/ is not, because the data partition is never overwritten. In Mender’s taxonomy this is an Application update: software delivered without an OS swap.

The persistent-app Update Module implements the pattern. On install it backs up the destination, copies in the new files, and on rollback restores the backup. Streaming installation under Mender Client 4.x/5.x is handled correctly. The module is community-maintained. It is not part of the supported Mender platform and is not covered by Mender support contracts.

We build artifacts with the bundled artifact-generator script:

./persistent-app-artifact-gen \
    -n demo-app-v2.0 \
    -t qemux86-64 \
    -d /data/apps/demo-app \
    --software-name demo-app \
    --software-version 2.0 \
    -o demo-app-v2.0.mender \
    ./payload/

The generator unconditionally passes --software-filesystem data-partition to mender-artifact, which is what puts the version provides in the surviving namespace. After deployment the device reports data-partition.demo-app.version=2.0, and that entry stays put through subsequent rootfs swaps. Drop the flag and you’ll spend an hour wondering why the version reverts after every OS update. That’s exactly the trap the namespace split exists to prevent.

To declare dependencies on rootfs capabilities, pass flags through to mender-artifact after --:

./persistent-app-artifact-gen \
    -n analytics-engine-v3.0 \
    -t qemux86-64 \
    -d /data/apps/analytics-engine \
    --software-name analytics-engine \
    --software-version 3.0 \
    -o analytics-engine-v3.0.mender \
    ./payload/ \
    -- --depends rootfs-image.python.version:3.12 \
       --depends rootfs-image.feature.docker:true

Now analytics-engine-v3.0 only installs on devices whose rootfs provides both python.version=3.12 and feature.docker=true. Anywhere else, the deployment fails fast with the Missing '<key>' in provides, required by artifact depends. error shown in the diagram above.

Domain 3: Containers

Containers iterate fastest. The catch is that container storage has to live on the data partition, or every image disappears at the next A/B switch. Mender’s taxonomy calls this a Container update, a subtype of Application update.

We configure Docker’s data root via /etc/docker/daemon.json:

{
    "data-root": "/data/docker",
    "storage-driver": "overlay2"
}

For Yocto builds, we install the file via a bbappend against docker-moby:

# recipes-containers/docker/docker-moby_%.bbappend
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"

SRC_URI += "file://daemon.json"

do_install:append() {
    install -d ${D}${sysconfdir}/docker
    install -m 0644 ${WORKDIR}/daemon.json ${D}${sysconfdir}/docker/daemon.json

    install -d ${D}/data/docker
}

FILES:${PN} += "/data/docker ${sysconfdir}/docker/daemon.json"

The runtime itself stays in the rootfs (updated atomically with the OS); only images and runtime state live on /data. With that split, a rootfs update never breaks the container fleet.

For deployments, the Docker Update Module handles single-container artifacts and the docker-compose Update Module handles multi-container stacks. Artifacts can pull from a registry at install time or carry the images inline. The inline option matters when the device is behind a constrained or air-gapped transport.

If Docker isn’t the runtime, the same pattern works with the Podman Update Module or the Kubernetes Update Module (K3s and other lightweight distributions). The composability story is identical; the runtime is an implementation choice.

A minimal three-domain build

The reference setup we’ve been quoting is a small Yocto build for qemux86-64, on the scarthgap branches of poky, meta-openembedded, meta-virtualization, and meta-mender. On top of those, a single small custom layer installs the persistent-app Update Module (from the script and module file linked in the Domain 2 section) and the Docker bbappend from the Domain 3 section. That’s the whole layer cake; everything else is what’s already in local.conf.

Two local.conf settings are worth highlighting. The rootfs declares its capabilities, which become the device’s rootfs-image.* provides on first boot:

MENDER_ARTIFACT_NAME = "composable-demo-1.0"
MENDER_ARTIFACT_PROVIDES = " \
    rootfs-image.python.version:3.12 \
    rootfs-image.api.version:2.0 \
    rootfs-image.feature.docker:true \
    rootfs-image.feature.persistent-app:true \
"

And the data partition is sized for the workload:

MENDER_DATA_PART_SIZE_MB = "4096"
MENDER_STORAGE_TOTAL_SIZE_MB = "8192"

Of the 8 GB total, 4 GB go to /data. Real sizing depends on what the upper domains actually deploy. /data/docker in particular grows with the union of all container layers ever pulled, not just the currently running set, so don’t size it for the steady state.

Booting this image under QEMU and exercising the three domains gives three observable outcomes that together validate the strategy:

  1. Rootfs provides land in the right namespace. mender-update show-provides after first boot returns the rootfs-image.* entries declared in MENDER_ARTIFACT_PROVIDES above.
  2. A persistent-app deployment surfaces in data-partition.*. Build a persistent-app artifact with the generator from Domain 2 (--software-name demo-app --software-version 1.0) and deploy it. The device then reports data-partition.demo-app.version=1.0, in the namespace that survives rootfs A/B swaps rather than the rootfs-image.* namespace that gets cleared.
  3. An artifact with an unsatisfied dependency is rejected pre-install. Build an artifact with --depends rootfs-image.cuda.version:11.8 (a capability the rootfs above does not provide) and deploy it. The Mender Client terminates the deployment with Missing 'rootfs-image.cuda.version' in provides, required by artifact depends. Streaming failed. System not modified. No partial install, no rollback needed.

Choosing what belongs in which domain

The three-domain split is a design tool, not a rule. For every piece of software in your stack, ask:

Question Bias toward
Does it require a reboot to take effect? Rootfs
Does it have deep OS integration (initramfs, kernel modules, systemd units)? Rootfs
Does it need to survive a rootfs update unchanged? Persistent apps or containers
Does it iterate faster than the OS release calendar? Persistent apps or containers
Is it isolated by a container boundary already? Containers
Is it large state (databases, ML models) that should not be re-shipped with code? Persistent apps

The edges aren’t always crisp. A Python application that depends on a system-level OpenSSL version sits at the seam between rootfs and persistent apps — declare it with --depends rootfs-image.openssl.version:<x> and let the coordination layer keep it honest. A container that bundles a kernel module is a sign the boundary is wrong; kernel-adjacent code belongs in the rootfs.

Cross-domain coordination at scale (orchestrated rollouts across backend, frontend, and sidecars, or capability migrations that retire an old API across the fleet over weeks) uses the same primitives: provides, depends, clears-provides, deployment dependencies. But assembling the release campaign is design work. For composable strategies at that scale, that work is appropriately handled as a professional-services engagement.

Combining composable updates with constrained transport

Composable updates are orthogonal to how artifacts reach the device. The domains, namespaces, and provides/depends checks work identically whether the device polls hosted Mender directly, sits behind a Mender Gateway, or receives updates through an operator-mediated bridge.

For the transport side, two adjacent pieces cover the constrained cases. Mender Gateway: OTA Updates for Segregated Networks handles the single-upstream-link case, with the gateway terminating Mender Client connections and caching artifacts at the boundary. The Trusted Intermediary Pattern handles the fully air-gapped case, where no system inside the boundary may reach the outside and operators carry artifacts across via portable media or a data diode.

A composable strategy is the shape of the payload, not the path it takes.

Wrapping up

Composable updates rest on three things: explicit domain separation by update mechanism, independent version namespaces per domain, and an enforced dependency graph between them. Get those right and an OS release stops dragging every application change with it, an application refuses to install on an incompatible rootfs, and a container fleet iterates at its own speed.

Pick the domains your software actually has, declare the provides that downstream artifacts depend on, and let the coordination layer keep the fleet consistent. The configuration above is the smallest end-to-end setup we could come up with, so use the snippets as starting points and adapt them to your stack.

Next steps