OTA updates to kubernetes workloads at the edge

Solutions
, ,
1

The problem

You have a fleet of remote devices equipped with Kubernetes, for example Raspberry Pi devices with K3s, a lightweight Kubernetes distribution. You can have a central management cluster, GitOps, remote container registries reachable from your devices.
Keeping workloads updated is painful. Cloud-native delivery tools assume data-center grade connectivity – they fall apart when the network is intermittent, firewalled, or behind NAT. You need SSH access just to debug a failed rollout. You want to push a commit and have it land on the device and you don’t want to refactor your developer experience, based on Kubernetes, the standard Developer Platform..

Architecture of the solution

Two layers:

  1. OS layer with K3s: you flash your device with K3s and Mender pre-installed and the minimal configuration to connect to your Mender Server account. You want to update the OS or only K3s to a newer patched kubernetes version? Just create a new Mender rootfs artifact and send it to your device (infrequent)
  2. Application layer: you package your Kubernetes application into a Mender artifact. No Git server or Container registry involved. You just send your artifact to your fleet!

Embedding Mender Update Modules into your image builds is what makes this fly.

How does it work?

The linked repository contains a basic Kubernetes application, the required helper scripts and tooling to facilitate the K3s device onboarding. Because the deployment pipeline runs as a GitHub Actions workflow on your copy of the repo, start by forking it to your own GitHub account (https://github.com/oldgiova/mender-k3s-edge-demo → Fork), then clone your fork to your local development environment:

git clone https://github.com/<your-username>/mender-k3s-edge-demo.git
cd mender-k3s-edge-demo

Optionally you can create a Raspberry Pi bootable image equipped with K3s and Mender, ready to be flashed and delivered to your device:

curl -LO https://d4o6e0uccgv40.cloudfront.net/2025-10-01-raspios-lite/arm/2025-10-01-raspios-lite-raspberrypi4_trixie_64bit-mender-convert-5.2.1.img.xz

make customize-image \
  INPUT=2025-10-01-raspios-lite-raspberrypi4_trixie_64bit-mender-convert-5.2.1.img.xz \
  DEVICE_USER=edgy \
  DEVICE_HOSTNAME=edge-01 \
  MENDER_SERVER_URL=https://hosted.mender.io \
  MENDER_TENANT_TOKEN=<your-token> \
  K3S_VERSION=v1.35.0+k3s1 \
  DEMO=true

Here MENDER_TENANT_TOKEN is your tenant token (in Hosted Mender under Settings → My organization). It is baked into the device’s /etc/mender/mender.conf and only lets the device connect to your tenant. It is not the same credential the CI pipeline uses to deploy – see “Two tokens, two jobs” below.

DEMO=true is optional: it bakes in mender-convert’s demo polling intervals (update and inventory every 5 s, retry every 30 s) instead of the slow production defaults, so deployments land within seconds while you are testing. Drop it for production images to avoid needless polling traffic.

The make customize-image calls a bash script which helps you to customize your Raspberry Pi with a firstboot configuration with K3s and Mender. Most importantly, it installs in the the Mender Update Module that will allow the K3s service to deploy your application to the Kubernetes node:

#!/bin/sh
set -e

STATE="$1"
FILES="$2"

CONTAINERD_SOCK=/run/k3s/containerd/containerd.sock
KUBECONFIG=/etc/rancher/k3s/k3s.yaml

case "$STATE" in
  NeedsArtifactReboot)
    echo "No"
    ;;
  SupportsRollback)
    echo "No"
    ;;
  ArtifactInstall)
    if [ -f "$FILES/files/image.tar" ]; then
      ctr --address "$CONTAINERD_SOCK" --namespace k8s.io images import "$FILES/files/image.tar"
    fi
    kubectl --kubeconfig "$KUBECONFIG" apply -f "$FILES/files/manifests.yaml"
    ;;
esac

Flash it to an SD Card (change /dev/sdX to your actual device)

IMG=$(ls output/*-custom.img)
sudo dd if="$IMG" of=/dev/sdX bs=4M conv=fsync status=progress

Use a Raspberry Pi 4 with at least 4GB of RAM (8GB recommended) – running a K3s node together with your workloads on less is likely to hit memory pressure. Use a microSD card of at least 16GB: it holds two A/B OS copies plus the K3s data partition and its container images. On first boot the data partition (/data, where K3s keeps its state under /data/k3s) automatically grows to fill the whole SD card – mender-convert bakes this in by default, so there is no manual resize step regardless of card size.

When the edge device boots up, you can see it available on your Mender Server instance, and you can accept it to start interacting with it. (Bonus: if you have Mender Gateway with mTLS configuration, you can onboard it automatically and in a very secure way).

Configure GitHub Actions

The deployment pipeline runs in your fork and talks to your Mender server, so it needs credentials and a few coordinates. In your fork, go to Settings → Secrets and variables → Actions and add:

Kind Name Value
Secret MENDER_TOKEN A Personal Access Token from hosted Mender (Profile → Access tokens in the Mender UI)
Variable MENDER_SERVER_URL Your Mender server, e.g. https://hosted.mender.io (or https://eu.hosted.mender.io)
Variable MENDER_DEVICE_TYPE The device type, e.g. raspberrypi4_64 – must match the device type accepted in Mender
Variable MENDER_DEVICE_GROUP The Mender device group to deploy to, e.g. edge-devices

With those in place, every push to main will build the artifact and create the deployment automatically.

Two tokens, two jobs

Do not confuse the two Mender tokens this solution relies on. They authenticate different things, live in different places, and are not interchangeable:

Token Where to get it Where it goes What it authenticates
Tenant token Hosted Mender → Settings → My organization Baked into the device image via MENDER_TENANT_TOKEN (/etc/mender/mender.conf) The device enrolling into your tenant
Personal Access Token (PAT) Hosted Mender → your profile → Personal access tokens The GitHub Actions secret MENDER_TOKEN The CI pipeline calling the Mender management API (upload artifact, create deployment)

The tenant token is shared by every device in your tenant and only grants enrolment – it gives no management access. The PAT carries your user’s permissions and can manage deployments, so keep it secret: store it only as a GitHub Actions secret, never bake it into an image or commit it. Both must also point at the same Mender region (hosted.mender.io vs eu.hosted.mender.io).

Now you can deploy your application: edit app/html/index.html and push to main. Watch GitHub Actions build the arm64 image, render the Kustomize overlay manifests, package the artifact, and trigger the deployment. The page updates on the device. No Mender UI interaction required. The operator never left their normal tools:

pi@edge-01:~$ sudo kubectl get pods -n edge-app -o wide
NAME                          READY   STATUS    RESTARTS   AGE     IP          NODE      NOMINATED NODE   READINESS GATES
nginx-edge-56f877dc79-v9jvb   1/1     Running   0          3h43m   10.42.0.9   edge-01   <none>           <none>
pi@edge-01:~$ sudo kubectl get pods -n edge-app -o yaml | grep image:
    - image: nginx-edge:sha-b6851337a8b870e0c7d853071895a44f6a2bff61

Verify it’s deployed correctly by navigating your device IP address:

<h1>Hello from the edge &mdash; v1</h1>

Edit app/html/index.html and push v2:

image1
image1764×800 51.7 KB

The automation is creating a new Mender Deployment under the hood, targeting your edge device, with the application built in it - no remote registry required. When the Mender Deployment finishes, you can see the updated result:

pi@edge-01:~$ sudo kubectl get pods -n edge-app
NAME                          READY   STATUS    RESTARTS   AGE
nginx-edge-7cb567b94c-rht2v   1/1     Running   0          37s
pi@edge-01:~$ sudo kubectl get pods -n edge-app -o yaml | grep image:
    - image: nginx-edge:sha-c728e30224548f270bc2910b5b926db34dbf378e

From the browser you can see the demo message from the updated index.html:

<h1>Hello from the edge &mdash; v2</h1>

How does automation work under the hood?

When you push your app/html/index.html to main, a Github Action is building a new Artifact with a container OCI image and the Kubernetes manifests:

      - name: Build OCI image (arm64) and export to tarball
        run: |
          mkdir -p payload
          docker buildx build \
            --platform linux/arm64 \
            --output "type=docker,dest=payload/image.tar" \
            --tag nginx-edge:sha-${{ github.sha }} \
            app/

      - name: Render kustomize manifests
        run: |
          cd k8s/overlays/edge
          kustomize edit set image nginx-edge=nginx-edge:sha-${{ github.sha }}
          cd ../../..
          kustomize build k8s/overlays/edge/ > payload/manifests.yaml

Then the Github Action is creating a Mender artifact out of them:

      - name: Create Mender artifact
        run: |
          mender-artifact write module-image \
            --type             k8s-workload \
            --compatible-types "${{ vars.MENDER_DEVICE_TYPE }}" \
            --artifact-name    "${{ env.ARTIFACT_NAME }}" \
            --file             payload/image.tar \
            --file             payload/manifests.yaml \
            -o                 "${{ env.ARTIFACT_NAME }}.mender"

The next step is to call the Mender Server APIs to deliver the new release to the Raspberry Pi device:

      - name: Upload artifact to hosted Mender
        run: |
          status=$(curl -sS -X POST \
            -H "Authorization: Bearer ${{ secrets.MENDER_TOKEN }}" \
            -F "artifact=@${{ env.ARTIFACT_NAME }}.mender" \
            -o response.txt -w "%{http_code}" \
            "${{ vars.MENDER_SERVER_URL }}/api/management/v1/deployments/artifacts")
          cat response.txt
          test "$status" -lt 300 || test "$status" -eq 409

      - name: Create deployment
        run: |
          status=$(curl -sS -X POST \
            -H "Authorization: Bearer ${{ secrets.MENDER_TOKEN }}" \
            -H "Content-Type: application/json" \
            -o response.txt -w "%{http_code}" \
            "${{ vars.MENDER_SERVER_URL }}/api/management/v1/deployments/deployments/group/${{ vars.MENDER_DEVICE_GROUP }}" \
            -d '{
              "name":          "${{ env.ARTIFACT_NAME }}",
              "artifact_name": "${{ env.ARTIFACT_NAME }}"
            }')
          cat response.txt
          test "$status" -lt 400

That’s all the magic!

Observability: inventory and Device Connect

There are two bonus features that come for free. The inventory script reports the running image tag, replica count, and pod status directly in the Mender UI:

image2
image2855×164 10 KB

This is very useful for fleet-wide visibility without kubectl access. Device Connect provides a WebSocket shell for live debugging without VPN or exposed SSH. The image build pre-configures mender-connect.conf so this shell opens as your DEVICE_USER (here edgy) rather than root, so passwordless sudo and the k3s tooling behave just like they do over SSH:

image3
image31267×719 88.4 KB

Next steps