The Trusted Intermediary Pattern: Managing Updates When Devices Can’t Reach the Server

Solutions
1

The problem

A power plant’s SCADA network has no route to the internet. A naval vessel’s operational technology segment is physically isolated from the ship’s administrative LAN. A defense contractor’s classified lab prohibits any outbound connection from the device subnet. A pharmaceutical manufacturing floor enforces strict network segmentation between production equipment and corporate IT.

In each of these environments, devices need software updates. They need inventory tracking. They need auditable deployment records. But they have no path — by design — to an update management server.

The standard approach to OTA software update management assumes the device can reach the server: the device polls for updates, downloads artifacts, reports status. When the network boundary makes that impossible, the update workflow does not disappear. It moves to a human operator or an intermediary system that can reach both sides of the gap.

The Trusted Intermediary (TI) pattern formalizes this. An external process — running on a workstation, a bastion host, or a portable device that crosses network boundaries — represents the air-gapped device to the Mender Server. It authenticates on the device’s behalf, retrieves deployments, carries artifacts across the gap, performs the update, and reports the outcome back. The Mender Server sees a managed device with full deployment history. The air-gapped device never makes a network connection it is not allowed to make.

Architecture

Trusted Intermediary Architecture
Trusted Intermediary Architecture1552×840 60.8 KB

The TI bridges the air gap. On the connected side, it speaks the Mender API. On the isolated side, it delivers updates through whatever transfer mechanism the environment allows.

The architecture has three zones:

Connected zone — A workstation or server with network access to the Mender Server (Hosted Mender or on-premises). The TI runs here for all API interactions: preauthorization, authentication, deployment retrieval, artifact download, and status reporting.

Air gap — The boundary between the connected network and the isolated network. Depending on the environment, this is crossed by physical media (USB drives, portable storage), a data diode, a controlled file transfer appliance, or a dual-homed bastion host. The transfer mechanism is environment-specific and often governed by site security policy.

Isolated zone — The air-gapped network where the target devices operate. Devices here have no outbound connectivity to the Mender Server. Updates arrive through the transfer mechanism and are applied locally — via SSH, a vendor-specific protocol, a local web interface, or direct file placement.

The TI uses two Mender API surfaces with separate credentials:

  • Management API — Authenticated with operator credentials (a Personal Access Token or management JWT). Used for preauthorizing devices, uploading artifacts, creating deployments, searching device records, and decommissioning.
  • Device API — Authenticated with per-device credentials (an RSA keypair and the resulting device JWT). Used for authenticating as the device, polling for updates, reporting deployment status, and submitting inventory data.

This dual-API design means the TI holds credentials at two privilege levels. In an air-gapped environment, where the TI workstation or the transfer media may be physically accessible to multiple personnel, the security implications are heightened. These are covered in the Security Considerations section.

Decision: Hosted or on-premises

The pattern works identically against Hosted Mender (hosted.mender.io / eu.hosted.mender.io) and on-premises Mender Server installations. For environments where the connected zone also has restricted internet access, an on-premises Mender Server in the connected zone is the typical choice.

Lifecycle overview

Trusted Intermediary lifecycle diagram
Trusted Intermediary lifecycle diagram1400×3084 172 KB

The full TI lifecycle. Phases 1–5 and 7 execute in the connected zone. Phase 6 crosses the air gap.

The lifecycle consists of seven phases. Phases 1 through 5 and phase 7 run entirely in the connected zone — they are API calls against the Mender Server. Phase 6 is where the air gap is crossed: the artifact is transferred to the isolated network and applied to the device.

Phase Zone API Surface Purpose
1. Key generation Connected Local Create RSA keypair for device identity
2. Preauthorization Connected Management API Register device with server before first contact
3. Device authentication Connected Device API Obtain device JWT using signed request
4. Inventory + update check Connected Device API Report device attributes, poll for deployments
5. Upload + deployment Connected Management API Upload pre-built artifact, create deployment
6. Update execution Crosses gap Device API (status) Transfer artifact, apply update, report result
7. Cleanup Connected Management API Remove auth set, optionally decommission

Prerequisites

You need:

  • A Mender Server account (Hosted Mender or on-premises) with a management PAT or JWT
  • The tenant token for your Mender organization
  • openssl for key generation and request signing
  • curl and jq for API interaction
  • A pre-built Mender Artifact (.mender file) — typically produced by a CI/CD pipeline or build system using mender-artifact
  • A transfer mechanism for moving artifacts across the air gap (USB drive, data diode, controlled file transfer — whatever your site security policy permits)

Set these as environment variables on the connected-zone workstation:

export MENDER_API_TOKEN="<your-management-api-token>"
export MENDER_TENANT_TOKEN="<your-tenant-token>"
export MENDER_SERVER_URL="https://hosted.mender.io"

The management token should come from a Personal Access Token (PAT) created in the Mender UI. PATs support expiry dates and can be revoked independently if compromised.

The API paths shown in this guide were validated against Mender Server 4.x (Hosted Mender, as of March 2026). Check the API documentation if you are working with a different server version.

Implementation

Phase 1: Key generation

Each air-gapped device managed through the TI needs a unique RSA keypair. The private key signs authentication requests. The public key is registered with the Mender Server during preauthorization. Both keys stay in the connected zone — the air-gapped device never sees them.

DEVICE_ID="device-001"
mkdir -p device_keys

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 \
    -out "device_keys/${DEVICE_ID}.key"

openssl rsa -pubout -in "device_keys/${DEVICE_ID}.key" \
    -out "device_keys/${DEVICE_ID}.pub"

The 3072-bit RSA key size matches what the Mender Client uses by default. If the TI manages multiple air-gapped devices, each device gets its own keypair. Store private keys securely — they are each device’s identity as far as the Mender Server is concerned.

Phase 2: Preauthorization

Preauthorization registers a device with the Mender Server before the device’s first authentication attempt. The server accepts the device’s public key and identity data, and the device is immediately placed in preauthorized state — no manual approval step required. The identity data is a set of key-value attributes that uniquely identify the device — a MAC address, a serial number, or any vendor-specific identifier. The examples below use a MAC address, but the structure is the same for any identity scheme.

DEVICE_IDENTITY='{"mac":"00:11:22:33:44:01"}'
PUBLIC_KEY_FILE="device_keys/${DEVICE_ID}.pub"
SERVER_URL="${MENDER_SERVER_URL:-https://hosted.mender.io}"

REQUEST_BODY=$(jq -n \
    --argjson id_data "$DEVICE_IDENTITY" \
    --rawfile pubkey "$PUBLIC_KEY_FILE" \
    '{
        identity_data: $id_data,
        pubkey: $pubkey,
        force: true
    }')

curl -s -w "\n%{http_code}" \
    -X POST "${SERVER_URL}/api/management/v2/devauth/devices" \
    -H "Authorization: Bearer ${MENDER_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$REQUEST_BODY"

The force: true parameter is critical for the TI pattern. It allows creating a new authentication set for a device identity that already exists on the server. Without it, re-registering a device that was previously managed — or reusing a device identity across TI sessions — fails with a conflict error. In air-gapped environments where devices are re-provisioned on a regular cycle, this is essential.

API reference: Preauthorize Device

Phase 3: Device authentication

With the device preauthorized, the TI authenticates as the device to obtain a device JWT. This mimics the Mender Client’s internal authentication flow: construct a JSON request body containing the device’s identity and tenant token, sign the entire body with the device’s private key, and submit both to the Device API.

IDENTITY_JSON="$DEVICE_IDENTITY"
PRIVATE_KEY="device_keys/${DEVICE_ID}.key"
PUBKEY=$(cat "device_keys/${DEVICE_ID}.pub")

REQUEST_BODY=$(jq -n \
    --arg id_data "$IDENTITY_JSON" \
    --arg tenant_token "$MENDER_TENANT_TOKEN" \
    --arg pubkey "$PUBKEY" \
    '{id_data: $id_data, tenant_token: $tenant_token, pubkey: $pubkey}' \
    --compact-output)

SIGNATURE=$(echo -n "$REQUEST_BODY" | \
    openssl dgst -sha256 -sign "$PRIVATE_KEY" | \
    openssl base64 -A)

RESPONSE=$(curl -s -w "\n%{http_code}" \
    -X POST "${SERVER_URL}/api/devices/v1/authentication/auth_requests" \
    -H "X-MEN-Signature: ${SIGNATURE}" \
    -H "Content-Type: application/json" \
    -d "$REQUEST_BODY")

MENDER_DEVICE_TOKEN=$(echo "$RESPONSE" | head -n -1)

The X-MEN-Signature header carries the RSA-SHA256 signature of the entire request body. The server verifies this against the public key registered during preauthorization. If the signature is valid and the device is in preauthorized or accepted state, the server returns a device JWT.

This token is used for all subsequent Device API calls. It has a limited lifetime; a production TI should implement token refresh by repeating this authentication step before the token expires.

API reference: Device Authentication

Phase 4: Inventory and update polling

Once authenticated, the TI reports inventory attributes and checks for pending deployments. In the air-gapped scenario, inventory data is typically collected from the isolated device during a previous transfer cycle and reported to the server in the next connected-zone session.

DEVICE_TYPE="pretend-client"
CURRENT_ARTIFACT="my-app-v1.0"

INVENTORY=$(jq -n \
    --arg hostname "$(hostname)" \
    --arg device_type "$DEVICE_TYPE" \
    '[
        {name: "hostname", value: $hostname},
        {name: "device_type", value: $device_type}
    ]')

curl -s -X PUT "${SERVER_URL}/api/devices/v1/inventory/device/attributes" \
    -H "Authorization: Bearer ${MENDER_DEVICE_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$INVENTORY"

REQUEST=$(jq -n \
    --arg device_type "$DEVICE_TYPE" \
    --arg artifact_name "$CURRENT_ARTIFACT" \
    '{device_type: $device_type, artifact_name: $artifact_name}')

DEPLOYMENT_INFO=$(curl -s -X POST \
    "${SERVER_URL}/api/devices/v1/deployments/device/deployments/next" \
    -H "Authorization: Bearer $MENDER_DEVICE_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$REQUEST")

echo "$DEPLOYMENT_INFO"

The device_type and artifact_name in the update check tell the Mender Server what the device currently runs. The server uses this to determine whether a pending deployment targets this device and whether the artifact differs from what is already installed. The response is stored in DEPLOYMENT_INFO for use in later phases — if a deployment is pending, it contains the deployment ID and artifact URI.

Phase 5: Uploading and deploying an update

The artifact to be deployed is typically built outside the TI workflow — in a CI/CD pipeline, a build server, or by a release engineering team. The TI’s role in this phase is to upload the pre-built artifact to the Mender Server and create a deployment targeting the air-gapped device.

ARTIFACT_NAME="pretend-release-v1"
ARTIFACT_FILE="pretend-artifact-v1.mender"

curl -s -X POST "${SERVER_URL}/api/management/v1/deployments/artifacts" \
    -H "Authorization: Bearer ${MENDER_API_TOKEN}" \
    -F "artifact=@${ARTIFACT_FILE}" \
    -F "description=Single-file config update for ${DEVICE_TYPE}"

DEPLOY_BODY=$(jq -n \
    --arg name "pretend-deploy-v1" \
    --arg artifact_name "$ARTIFACT_NAME" \
    '{name: $name, artifact_name: $artifact_name, all_devices: true}')

curl -s -X POST "${SERVER_URL}/api/management/v1/deployments/deployments" \
    -H "Authorization: Bearer ${MENDER_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$DEPLOY_BODY"

Note the API boundary crossing: artifact upload and deployment creation use the Management API with the operator token. The device then picks up the deployment in the next update poll using the Device API with its device token. If the artifact has already been uploaded to the Mender Server (e.g. by a CI/CD pipeline), the upload step can be skipped and only the deployment creation is needed.

API references: Upload Artifact, Create Deployment

Phase 6: Update execution — crossing the air gap

This is the phase where the air gap is actually crossed. The TI downloads the artifact in the connected zone, then the operator transfers it to the isolated network and applies it to the target device.

DEPLOYMENT_ID=$(echo "$DEPLOYMENT_INFO" | jq -r '.id // empty')
ARTIFACT_URI=$(echo "$DEPLOYMENT_INFO" | jq -r '.artifact.source.uri // empty')

curl -s -X PUT \
    "${SERVER_URL}/api/devices/v1/deployments/device/deployments/${DEPLOYMENT_ID}/status" \
    -H "Authorization: Bearer ${MENDER_DEVICE_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"status":"downloading"}'

curl -L -o "downloaded-artifact.mender" "$ARTIFACT_URI"

curl -s -X PUT \
    "${SERVER_URL}/api/devices/v1/deployments/device/deployments/${DEPLOYMENT_ID}/status" \
    -H "Authorization: Bearer ${MENDER_DEVICE_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"status":"installing"}'

# === Air-gap crossing happens here ===
# Transfer the artifact to the isolated network and apply the update.
# The mechanism is environment-specific: USB drive, data diode,
# controlled file transfer, bastion host, etc.

curl -s -X PUT \
    "${SERVER_URL}/api/devices/v1/deployments/device/deployments/${DEPLOYMENT_ID}/status" \
    -H "Authorization: Bearer ${MENDER_DEVICE_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"status":"success"}'

The status transitions — downloading, installing, success (or failure) — must be reported in sequence. If you skip directly to success without intermediate states, the deployment timeline in the Mender Server UI will show gaps. If you never report a terminal status, the deployment remains in pending state indefinitely.

The gap between the installing and success status reports is where the actual air-gap crossing and device update happens. This might take minutes (USB transfer to a device in the same facility) or days (portable media shipped to a remote site). The TI reports installing before the transfer and success (or failure) after the operator confirms the update was applied. In environments where the round-trip takes significant time, the deployment will appear in installing state in the Mender Server UI until the operator completes the cycle and reports back.

This is custom engineering — not a supported self-service path. The TI is responsible for bridging the air gap and translating a Mender Artifact into whatever update mechanism the target environment requires. For air-gapped deployments at this level of complexity, that is appropriate: you want engineering support, not a wiki page.

Phase 7: Cleanup

After the update cycle, the TI removes the authentication set and optionally decommissions the device entirely.

SEARCH_BODY=$(jq -n --argjson id_data "$DEVICE_IDENTITY" '{identity_data: $id_data}')

DEVICES_RESPONSE=$(curl -s \
    -X POST "${SERVER_URL}/api/management/v2/devauth/devices/search" \
    -H "Authorization: Bearer ${MENDER_API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$SEARCH_BODY")

MENDER_DEVICE_ID=$(echo "$DEVICES_RESPONSE" | jq -r '
  [.[] | select(.status == "accepted")] |
  sort_by(.updated_ts) | reverse | .[0].id // empty
')

AUTH_SET_ID=$(echo "$DEVICES_RESPONSE" | jq -r --arg device_id "$MENDER_DEVICE_ID" '
  .[] | select(.id == $device_id) |
  [.auth_sets[] | select(.status == "accepted" or .status == "preauthorized")] |
  sort_by(.ts) | reverse | .[0].id // empty
')

curl -X DELETE \
    "${SERVER_URL}/api/management/v2/devauth/devices/${MENDER_DEVICE_ID}/auth/${AUTH_SET_ID}" \
    -H "Authorization: Bearer ${MENDER_API_TOKEN}"

Two cleanup levels are available:

Action API Call Effect
Auth set removal DELETE /devauth/devices/{id}/auth/{auth_set_id} Removes the TI’s credentials. Device record remains. Re-registration possible.
Full decommission DELETE /devauth/devices/{id} Removes the device record entirely. Audit logs (Mender Enterprise) retain the history.

Choose auth set removal when the device will receive future update cycles through the TI. Choose decommissioning when the device is permanently retired from service.

Static vs. rotating Trusted Intermediaries

The full lifecycle described above — key generation, preauthorization, device authentication, and cleanup — is designed for rotating TIs. This is the typical model when service personnel from the device manufacturer visit multiple customer sites: each visit spins up a fresh TI session, authenticates on behalf of the devices at that site, performs the update, and tears down the credentials on departure. No long-lived device keys remain on the service engineer’s workstation between site visits.

When the TI is operated by the site owner or facility manager — a permanent fixture rather than a visiting process — the workflow simplifies. Key generation and preauthorization happen once during initial setup. The device credentials (RSA keypair and device JWT) are retained between update cycles, and Phases 1–3 and 7 can be skipped for subsequent runs. The TI then only needs to poll for updates, upload artifacts, create deployments, and report status — Phases 4 through 6.

This static assignment model is simpler to operate and reduces the per-cycle API interactions, but it does mean the device credentials are long-lived. The security considerations below apply with particular weight in this case, especially around key storage and credential scope.

Security considerations

Air-gapped environments exist because security is the primary constraint. The TI must not undermine that constraint.

Credential scope

The management PAT should be constrained with Role-Based Access Control (RBAC) to the minimum permissions the TI needs. RBAC is a Mender Enterprise feature. If the TI workstation is compromised, a scoped PAT limits the blast radius to only the operations the TI is authorized to perform — not full management access. PATs support expiry dates and can be revoked independently.

Key storage

Device private keys must be stored securely on the connected-zone workstation. Filesystem permissions are the baseline. In environments with strict key management requirements (defense, critical infrastructure), consider a hardware security module (HSM) or a secrets manager. The TI should never log, display, or transmit private keys. Keys should never cross into the isolated zone — they exist solely to authenticate with the Mender Server.

Transfer media integrity

The artifact transferred across the air gap must arrive intact and unmodified. Mender Artifacts are cryptographically signed; if your build pipeline signs artifacts, verify the signature on the isolated side before applying the update. For environments that use USB drives or portable media, consider write-once media or hardware-enforced write protection to prevent the transfer media from being used as an exfiltration vector on the return trip.

Audit trail

Mender Enterprise provides audit logging for device acceptance, authentication, and decommissioning. For air-gapped deployments, the Mender audit trail complements the site’s physical access logs and transfer records. The combination provides end-to-end traceability: who authorized the deployment (Mender audit log), who carried the media across the gap (site access log), and who applied the update to the device (local device log or operator attestation).

Chain of custody

In regulated environments (defense, pharma, critical infrastructure), the air-gap crossing is a custody transfer point. The TI provides the digital side — deployment ID, artifact hash, timestamps, status reports. The physical side — who carried the media, when it crossed the boundary, what device received it — is outside Mender’s scope and must be tracked by the site’s operational procedures.

When to use this pattern

The Trusted Intermediary pattern is the right approach when:

  • Devices operate on an air-gapped or network-segmented network with no outbound path to the Mender Server
  • You need centralized deployment tracking and audit trails for devices that cannot report their own status
  • Regulatory or security policy requires that devices never initiate outbound connections
  • You operate mixed fleets where some devices have connectivity (and run the Mender Client) and others sit behind an air gap

It is not a replacement for the Mender Client on devices that can reach the server. The Mender Client handles network interruption recovery, partial download resumption, automatic rollback, and state persistence across reboots — edge cases that matter when the device has a live connection. If your device has a network path to the Mender Server, use the client directly.

Extending this pattern to production scale — managing dozens of air-gapped devices across multiple sites, automating the transfer workflow, implementing verification and rollback on the isolated side — is custom engineering. The Mender APIs are supported and documented, but the air-gap bridging logic, the transfer automation, and the site-specific security procedures are yours to design and maintain. For air-gapped deployments at scale, professional services engagement provides the engineering support to build this correctly.

Next steps