Edge AI Models: Custom Update Modules in Practice

Solutions
1

The problem

An interactive kiosk runs a local voice assistant.
A delivery van streams diagnostics through an on-board language model.
A car listens for spoken commands while driving with no reliable uplink.

The compute is on the device. There is no cloud round-trip to ask for, and no operator standing by to swap an SD card. The hardware stays where it is for years, but the models do not.

Three things change on three different clocks. The OS gets quarterly security patches. The inference runtime (llama.cpp, ONNX Runtime, whatever it happens to be) tracks upstream releases. The model weights turn over faster than either of the other two: Whisper for speech-to-text, a quantised Llama or Qwen for the language model, Piper for text-to-speech. And the system prompt that shapes the LLM’s behaviour can turn over weekly. Wrapping all of that in a single rootfs image makes the slowest cadence the only cadence the fleet ever sees.

The off-the-shelf Update Modules (rootfs-image, single-file, directory-overlay) don’t quite fit either. The unit of change is not “one file” or “one directory”. It is “swap the weights, restart the inference server, confirm that the service comes back healthy, and only then mark the deployment installed.” A failed model file is not a corrupted byte stream. It is a syntactically valid GGUF that crashes llama.cpp at warm-up. The installer has to use the artifact, not just write it, before it is allowed to commit.

This piece walks through what an Update Module that handles that responsibility actually looks like. The example throughout is a working voice-assistant stack: faster-whisper, a quantised Qwen2 LLM, Piper TTS, with four custom Update Modules and a small fleet of inventory hooks behind it. The patterns generalise to any edge-AI workload with component-level model swaps.

Version Notes

  • Mender Client 4.x or 5.x. The Update Module v3 protocol described below is stable across these releases.
  • mender-artifact 4.x for building module-image artifacts.
  • The example device is an NVIDIA Jetson Orin NX 8 GB running JetPack 5.1. Any Linux device with enough free space to hold the active and backup model sets at the same time will do; nothing in the patterns below is Jetson-specific.

What an Update Module has to do

Mender’s Update Module v3 protocol is a small contract between the Mender Client and an executable on the device. The client invokes the module by name, passing the current state of the deployment as the first argument and the path to the streamed artifact as the second. The module returns a non-zero exit code if anything is wrong.

A complete module implements a handful of states:

  • NeedsArtifactReboot and SupportsRollback: capability queries answered once, before the install begins.
  • ArtifactInstall: validate the payload and stage it somewhere reversible. Do not yet touch the running system.
  • ArtifactCommit: make the change visible to the rest of the system. This is the only state allowed to break things.
  • ArtifactRollback: restore the previous state if commit failed.
  • Cleanup: remove temporary files, always.

Modules live in /usr/share/mender/modules/v3/<name> and need only be executable. The streaming download itself is handled by the Mender Client; modules see the unpacked payload at the path passed in argument two.

Three properties of the contract matter for AI workloads. First, the split between install and commit gives the module two chances to back out before any service notices the change. Second, rollback is opt-in and the client respects what the module says: if SupportsRollback returns Yes, the client will call ArtifactRollback on failure. Third, Cleanup always runs, so transient state in a staging directory is safe to leave behind if anything goes wrong.

Validate at Install, swap at Commit, restore at Rollback. The running system never sees a half-applied change.

The example: a voice-assistant pipeline

The reference setup runs on an NVIDIA Jetson Orin NX 8 GB. Hardware is incidental to the patterns described here; anything with enough memory to hold the active model plus a backup of the previous one will do.

Edge AI voice assistant: pipeline and storage
Edge AI voice assistant: pipeline and storage1760×1080 75.4 KB

The audio path is mic → faster-whisper STT → llama.cpp running Qwen2 → Piper TTS → speaker. Two systemd services keep the pipeline running: voice-assistant.service (Python, FastAPI on port 8080, Silero VAD) and llama-server.service (llama.cpp HTTP server on port 8081, CUDA). All model state lives under /data/ai-models/. The Mender Client and its four custom modules sit alongside the inference services; the modules write into staging/ and backup/ and only the commit step touches current/.

The layout under /data/ai-models/ is the structural choice that makes everything else possible:

/data/ai-models/
├── current/                  # active set, read by services
│   ├── config.json
│   ├── stt/                  # Whisper model files
│   ├── tts/                  # Piper voice files
│   └── llm/
│       └── model.gguf
├── staging/                  # Update Module workspace
└── backup/                   # previous set, for rollback

current/ is the only directory the services know about. The inference path is hard-coded to read from there. Updates land in staging/, get validated, and only get promoted into current/ once the module is sure. Whatever was in current/ before the promotion gets snapshotted into backup/ first, so rollback is a directory copy.

Independent release cadences

The four components on the device change on four different clocks.

Update cadence by component (one year)
Update cadence by component (one year)1840×940 68.4 KB

The OS turns over quarterly with a 1 GB+ artifact. The STT model gets re-released maybe twice a year as a ~140 MB artifact. The LLM weights change as the user picks new quantisations or upstream Qwen2 revisions: eight times a year is realistic, at ~318–375 MB per swap. The TTS voice changes monthly at ~60 MB. The system prompt changes weekly at ~5 KB. Forcing all five through a single rootfs swap collapses every cadence to the slowest.

Each non-rootfs cadence gets its own custom Update Module. The voice-assistant stack ships four:

Module Payload Typical artifact
ai-model-stt Whisper model files (model.bin, config.json, tokenizer.json, vocabulary.txt) ~140 MB
ai-model-llm A single *.gguf file, optional llm-config.json ~318–375 MB
ai-model-tts A Piper *.onnx and matching *.onnx.json ~60 MB
ai-model-llm-prompt prompt-config.json only, no model weights ~5 KB

The prompt module is the one that justifies the whole approach. Treat it as a separate release lane and a non-engineering reviewer can land a tone change (“be more concise”, “address the user formally”) without anyone rebuilding or re-shipping a 375 MB model file. The same module skeleton handles both the 5 KB JSON delta and the 375 MB binary swap, because the protocol does not care.

This is the natural application layer for the patterns described in Composable Updates: Independent Update Domains for OS, Applications, and Containers: multiple components on one device, each shipped through its own Update Module, each with its own version namespace. Composable Updates covers the theory, including how to use --software-filesystem data-partition so that the version provides for each component land in a namespace that survives a rootfs swap. The piece you are reading is an example of one of those components done in depth.

Anatomy of a custom Update Module

The prompt module is the smallest of the four and is the right place to start, because it exercises every state the protocol offers without the distraction of a half-gigabyte binary.

Update Module v3 state machine
Update Module v3 state machine1920×1120 99.3 KB

The happy path runs left to right along the top: ArtifactInstall validates and stages the payload, ArtifactCommit backs up the previous state and applies the new one, Cleanup removes the staging directory. Either install or commit can fail; both routes go through ArtifactRollback, which restores the backup, and then rejoin Cleanup. The capability queries above the flow are answered once, before the install begins.

The module is a single Bash script at /usr/share/mender/modules/v3/ai-model-llm-prompt. The capability queries are at the top:

case "$STATE" in
    NeedsArtifactReboot)
        echo "No"
        ;;

    SupportsRollback)
        echo "Yes"
        ;;

No to reboot, Yes to rollback. A reboot is overkill for a config change; restarting voice-assistant.service is enough.

ArtifactInstall validates the payload and stages it. The script does not touch the running config:

    ArtifactInstall)
        log "info" "Starting LLM prompt update"

        if [ ! -f "$FILES/files/prompt-config.json" ]; then
            log "err" "prompt-config.json not found in payload"
            exit 1
        fi

        if ! jq empty "$FILES/files/prompt-config.json" 2>/dev/null; then
            log "err" "Invalid JSON in prompt-config.json"
            exit 1
        fi

        NEW_PROMPT=$(jq -r ".system_prompt // empty" "$FILES/files/prompt-config.json")
        if [ -z "$NEW_PROMPT" ]; then
            log "err" "system_prompt field missing or empty"
            exit 1
        fi

        mkdir -p "${MODEL_BASE}/staging/prompt"
        cp "$FILES/files/prompt-config.json" "${MODEL_BASE}/staging/prompt/"

        log "info" "Prompt staged: ${NEW_PROMPT:0:50}..."
        ;;

Three validation gates, in order: the file is present, the file parses as JSON, the JSON contains a non-empty system_prompt field. If any gate fails, the module exits non-zero, the Mender Client records the failure, and the running system has not been touched.

ArtifactCommit is where the change becomes visible. It is also the only place the module is allowed to fail without breaking the user’s expectation that updates are atomic:

    ArtifactCommit)
        log "info" "Committing LLM prompt update"

        STAGING_FILE="${MODEL_BASE}/staging/prompt/prompt-config.json"
        if [ ! -f "$STAGING_FILE" ]; then
            log "err" "Staging file not found"
            exit 1
        fi

        mkdir -p "$(dirname "$BACKUP_FILE")"
        cp "$CONFIG_FILE" "$BACKUP_FILE"
        log "info" "Config backed up"

        systemctl stop voice-assistant 2>/dev/null || true

        NEW_PROMPT=$(jq -r ".system_prompt" "$STAGING_FILE")
        jq --arg prompt "$NEW_PROMPT" ".llm.system_prompt = \$prompt" \
           "$CONFIG_FILE" > "${CONFIG_FILE}.tmp"
        mv "${CONFIG_FILE}.tmp" "$CONFIG_FILE"

        systemctl start voice-assistant

        log "info" "Waiting for voice assistant to become healthy..."
        if type check_service >/dev/null 2>&1; then
            if ! check_service voice-assistant 30; then
                log "err" "Voice assistant service failed to start"
                exit 1
            fi
            if ! check_voice_assistant_health; then
                log "err" "Voice assistant API health check failed"
                exit 1
            fi
        else
            sleep 5
            if ! systemctl is-active --quiet voice-assistant; then
                log "err" "Voice assistant failed to start"
                exit 1
            fi
        fi

        rm -rf "${MODEL_BASE}/staging/prompt"
        log "info" "LLM prompt updated successfully - health check passed"
        ;;

Three things to notice. First, the backup happens before the service stops, so the old config is on disk regardless of what happens to the running process. Second, the JSON edit goes through a temp file and a mv so the on-disk config is never half-written. Third, the health check has a fast path and a fallback. When the shared health-check.sh helper is sourced, the module calls check_voice_assistant_health(), which polls the FastAPI /api/health endpoint and confirms the assistant reports a non-initializing status. When the helper is not available, the module degrades to a five-second sleep followed by a systemctl is-active check. A service that starts and immediately wedges fails the primary gate; the fallback only catches the harder case where the process exits outright.

If any of those steps fails, the script exits non-zero. Because the module declared SupportsRollback: Yes, the Mender Client then calls ArtifactRollback:

    ArtifactRollback)
        log "warning" "Rolling back LLM prompt"

        systemctl stop voice-assistant 2>/dev/null || true

        if [ -f "$BACKUP_FILE" ]; then
            cp "$BACKUP_FILE" "$CONFIG_FILE"
            log "info" "Config restored from backup"
        fi

        rm -rf "${MODEL_BASE}/staging/prompt"
        systemctl start voice-assistant 2>/dev/null || true

        log "info" "Rollback completed"
        ;;

Restore the backup, restart the service, drop the staging directory. The 2>/dev/null || true on the systemctl calls is deliberate: rollback is the path that runs when something is already wrong, and a second stop or start failure should not mask the original error.

Cleanup runs unconditionally at the end of either path, and is correspondingly small:

    Cleanup)
        rm -rf "${MODEL_BASE}/staging/prompt"
        ;;

That is the entire module: roughly 130 lines of Bash, three validation gates on install, a backup-edit-validate cycle on commit, a directory-copy rollback, and a guaranteed cleanup. The Update Module protocol provides the spine; the script fills in the verbs.

Scaling the pattern to binary payloads

The three model modules (ai-model-stt, ai-model-llm, ai-model-tts) follow the same skeleton with a different payload shape. The interesting differences are in the install-time validation and the commit-time health check.

The LLM module validates that a GGUF file is present and large enough to plausibly be a model:

        GGUF_FILE=$(find "$FILES/files" -name "*.gguf" -type f 2>/dev/null | head -1)

        if [ -z "$GGUF_FILE" ]; then
            log "err" "No GGUF model file found in payload"
            exit 1
        fi

        FILE_SIZE=$(stat -c%s "$GGUF_FILE")
        if [ "$FILE_SIZE" -lt 100000000 ]; then
            log "err" "GGUF file too small: ${FILE_SIZE} bytes"
            exit 1
        fi

The 100 MB lower bound is a coarse sanity check: every quantisation worth shipping is larger than that. The intent is to catch obviously-truncated downloads before they reach commit.

The commit phase is where the AI-specific work happens. The LLM module stops llama-server and voice-assistant, swaps the file, restarts both services, and then waits for llama-server to answer on its health endpoint before declaring success:

        systemctl start llama-server
        sleep 3

        if ! systemctl is-active --quiet llama-server; then
            log "err" "llama-server failed to start"
            exit 1
        fi

        for i in 1 2 3 4 5; do
            if curl -sf http://localhost:8081/health > /dev/null 2>&1; then
                log "info" "llama-server health check passed"
                break
            fi
            sleep 2
        done

One trap: a GGUF file that parses but crashes on warm-up (typically a quantisation mismatch with the runtime, or an oversized context window for the available VRAM) fails this loop. The rollback path puts the previous model.gguf back and restarts both services.

The TTS module does the same dance for ONNX voice files, with one extra step: it edits config.json so the running service knows which voice file to load. Each module restricts itself to the component it owns. They share /data/ai-models/, but they do not stomp on each other’s directories.

Inventory hooks: matching the artifact to the device

A custom Update Module decides whether an artifact can be installed on the device it lands on. Inventory decides which devices the artifact lands on in the first place. Without inventory you can build the best Update Module in the world, but the only way to target a deployment is by device ID. That does not scale to fleets.

Inventory scripts live in /usr/share/mender/inventory/. Each one is an executable that prints key=value lines on stdout; the Mender Client harvests them at the configured InventoryPollInterval and ships them to the server. The voice-assistant stack ships four:

# /usr/share/mender/inventory/mender-inventory-ai-models
CONFIG_FILE="/data/ai-models/current/config.json"
if [ -f "$CONFIG_FILE" ]; then
    VERSION=$(jq -r '.version // "unknown"' "$CONFIG_FILE" 2>/dev/null)
    LLM_MODEL=$(jq -r '.llm.model // "unknown"' "$CONFIG_FILE" 2>/dev/null)
    STT_MODEL=$(jq -r '.stt.model_path // "unknown"' "$CONFIG_FILE" 2>/dev/null)
    TTS_MODEL=$(jq -r '.tts.model_path // "unknown"' "$CONFIG_FILE" 2>/dev/null | \
                xargs basename 2>/dev/null | sed 's/.onnx$//')

    echo "ai_config_version=$VERSION"
    echo "ai_llm_model=$LLM_MODEL"
    echo "ai_stt_model=$STT_MODEL"
    echo "ai_tts_voice=$TTS_MODEL"
fi

On a device that has installed llm-qwen2-q4km-v1, stt-whisper-base-v2, and tts-amy-voice-v1, the resulting inventory keys look like this:

ai_config_version=1.1.0
ai_llm_model=qwen2:0.5b
ai_stt_model=stt
ai_tts_voice=en_US-amy-medium
ai_llm_size_mb=375

The hardware script (mender-inventory-hardware) reports the Jetson model, GPU type, RAM, disk, and GPU temperature. The services script (mender-inventory-services) reports whether voice-assistant and llama-server are running and whether they answer their health endpoints. The system script reports JetPack release, kernel, CUDA version, and the IP address.

What this combination buys is targeted rollouts. Deploy llm-qwen2-q4km-v1 only to devices with hw_memory_total_mb >= 7000 and sys_cuda_version >= 11.4. Deploy a new prompt only to devices where ai_llm_model already matches the LLM the prompt was tuned for. Withhold any deployment from devices where svc_api_health is currently error, because chasing a broken device with another update tends to make things worse. The decisions happen on the server, against keys the modules and the inventory scripts agreed on.

Building the artifacts

Artifacts get built with mender-artifact write module-image. The -T flag names the target module on the device; the device-type filter (-t) keeps a TTS artifact from landing on hardware that does not run this stack.

A TTS artifact is two files plus three flags of metadata:

mender-artifact write module-image \
    -T ai-model-tts \
    -n "tts-amy-voice-v1" \
    -t "jetson-orin-nx" \
    -o tts-amy-voice-v1.mender \
    -f en_US-amy-medium.onnx \
    -f en_US-amy-medium.onnx.json

An LLM artifact is the same shape, with a different -T and the GGUF as the payload:

mender-artifact write module-image \
    -T ai-model-llm \
    -n "llm-qwen2-q4km-v1" \
    -t "jetson-orin-nx" \
    -o llm-qwen2-q4km-v1.mender \
    -f qwen2-0.5b-q4_k_m.gguf

The prompt artifact is the same command again, with a 5 KB JSON payload:

mender-artifact write module-image \
    -T ai-model-llm-prompt \
    -n "llm-prompt-professional-v1" \
    -t "jetson-orin-nx" \
    -o llm-prompt-professional-v1.mender \
    -f prompt-config.json

The artifacts above ship the version implicitly in their -n name. For fleet-scale work where the same module type ships repeatedly, pair these with the --software-filesystem data-partition, --software-name, and --software-version flags described in Composable Updates so each component carries a version provide in the namespace that survives an OS swap. Set that up once and the server can refuse to install an LLM artifact that depends on an STT version the device does not have, before the device starts streaming the payload.

What this isn’t

The four modules in this article are tier-3 custom engineering. They were written specifically for this voice-assistant stack: the validation logic that recognises a plausibly-large GGUF, the dual-service health check that waits for both llama-server and voice-assistant to answer, the prompt-only release path that exists only because someone decided treating system prompts as a release lane was worth the effort. None of that is something you get out of the box.

The Update Module v3 protocol they implement is supported and stable; the modules themselves are not. For AI workloads at production scale, the modules’ validation logic, model-warmup checks, and component-by-component release strategy are exactly the kind of work that benefits from professional services involvement. Not because the patterns are difficult, but because they are domain-specific and there is no generic right answer. The wiki page can show you what a good module looks like. It cannot tell you whether your TTS service needs a five-second or a thirty-second warm-up grace.

The inventory scripts are similarly custom. The keys they emit are not part of any Mender taxonomy; they exist because someone needed to target deployments by GPU temperature and chose what to call the field.

Combining edge-AI modules with broader update strategy

The four modules in this article are payload-side machinery: they define what happens inside the artifact once it lands on the device. They sit orthogonal to how that artifact reaches the device, and to what else the same device is updating.

For the composition question, Composable Updates covers the split between rootfs, persistent applications, and containers, along with the provides/depends metadata that keeps incompatible combinations off the fleet. An edge-AI stack is one instance of the persistent-application domain; the modules above ride on top of the primitives that piece describes.

For the transport question, two adjacent pieces cover the constrained cases. Mender Gateway: OTA Updates for Segregated Networks handles hospitals, factories, and other single-upstream-link environments where a fleet of accelerators cannot reach the Mender Server directly. The Trusted Intermediary Pattern handles fully air-gapped deployments where an operator carries artifacts across the boundary.

The modules above are the same in any of these topologies. Only the delivery path changes.

Conclusion

Three things to take away.

The Update Module v3 contract is rich enough for non-trivial workloads. The install/commit split gives the module two chances to back out before the running system notices. Rollback is opt-in and the client honours it. Cleanup always runs. That is enough scaffolding to express “stage a model, swap it under a stopped service, wait for the service to come back healthy, and accept or reject the change accordingly” in around 130 lines of Bash.

Decomposing an AI application into per-component modules unlocks independent release cadences. A weekly prompt change and a quarterly STT model upgrade do not need to share a release vehicle. Mender’s Update Modules give each component its own contract, its own validation, and its own rollback path.

Inventory is what turns “we can deploy” into “we can deploy to the right subset”. Custom inventory keys (GPU memory, active LLM identifier, service health) let the server target deployments by the state of the device’s AI pipeline rather than by device ID. Without that, the modules above are deployable but not steerable.

The four modules in this article are the smallest set of moving parts that exercises the protocol end to end. Use the snippets as starting points and adapt them. The interesting questions are workload-specific: what counts as a valid model, how long is too long for the service to take to come back, which inventory keys you need to target a deployment against. The protocol stays the same.

Next steps