Industrial edge devices are no longer static appliances. They run customer-critical workloads, connect to cloud services, process local telemetry, and need regular updates for security, reliability, and product evolution. Yet many embedded image workflows still look like manufacturing workflows: flash a vendor image, apply manual changes, capture a golden image, and hope the result can be reproduced later.
That model does not scale to a growing OT fleet. Every undocumented change becomes operational risk. Every kernel upgrade becomes a special project. Every failed boot can mean a field visit.
At Octave, we wanted device images managed like cloud infrastructure: built from pinned inputs, configured declaratively, verified automatically, released immutably, and updated through a controlled rollout process. In other words, we wanted an embedded Linux image pipeline with the same governance and repeatability we expect from Terraform, container builds, and CI/CD.
This article describes how we built that pipeline. Starting from unmodified vendor images, we apply deterministic configuration with pyinfra, convert the result into Mender A/B artifacts, generate SBOMs, gate boot safety at multiple checkpoints, and publish releases to S3 and hosted Mender through GitHub Actions.
The result is not just an automated image build. It is a system architecture for fleet-scale OTA management: reproducible builds, auditable releases, rollback-aware OTA updates, and a safe path for controlled kernel migrations across real hardware in the field.
Our two target platforms are the Revolution Pi Connect SE (RevPi) — an industrial Raspberry Pi CM4 variant with CAN bus, piControl real-time I/O, and a hardware watchdog — and the Edgebox RPI-200 — a Raspberry Pi CM4 in a ruggedised DIN-rail enclosure.
The central thesis is simple: A/B OTA solves delivery. IaC solves reproducibility. Industrial edge fleets need both.
The pipeline at a glance
The build pipeline has six phases, all driven by a single declarative config file:
build.json (declarative config)
|
+-- Phase 1 -- image-sync download vendor image from S3
+-- Phase 2 -- image-builder run pyinfra provisioning (chroot)
+-- Phase 3 -- mender-tool build mender-convert -> A/B artifact
+-- Phase 4 -- image-builder sbom Syft SBOM (SPDX + CycloneDX + Syft)
+-- Phase 5 -- image-sync upload release artifacts to S3
+-- Phase 6 -- mender-cloud upload push to hosted Mender + tag release
At the system level, this pipeline connects four layers:
Source of truth
Git + build.json + pinned vendor images (S3)
|
Build system
pyinfra provisioning + mender-convert + validation gates
|
Artifact registry
S3 releases + hosted Mender + SBOMs
|
Fleet rollout
staged deployment + compatibility gates + rollback
|
Device runtime
hardware health checks + inventory feedback to hosted Mender
build.json is the single source of truth: it names the platforms, the Debian releases they support, where vendor images live on S3, which pyinfra profile to apply, and which mender-convert version and configs to use. tasks.sh reads this file, enumerates the platform/release matrix, and drives the six phases. Taskfile.yml exposes the tasks (task build, task release, task show-plan) with dependency checking and environment defaults.
Mender provides the OTA delivery primitives: A/B rootfs partitioning, atomic slot switching, and automatic rollback on failure. Our pipeline adds IaC discipline around the image creation side — reproducible builds from pinned inputs, declarative configuration, and a validation chain that means a bad image cannot reach a device without tripping a gate first.
What this changed
Before this work, producing an embedded release was too close to a bespoke image-building exercise. Vendor images had to be selected carefully, modified, converted, verified, and uploaded through steps that were difficult to reproduce end-to-end. That made platform changes — especially kernel and Debian upgrades — operationally expensive and risky.
The new pipeline turns device images into governed release artifacts.
For engineering, that means every device image is traceable to Git, pinned vendor inputs, CI logs, SBOMs, and Mender metadata. For operations, it means fewer one-off recovery procedures and a safer path for OTA updates. For the platform roadmap, it means we can move across Debian releases, kernel generations, hardware variants, and Greengrass versions without rebuilding the process each time.
The architectural shift is that the device image is no longer a manually prepared asset. It is a product of the delivery system.
| Outcome | Why it matters |
|---|---|
| Reproducible releases | Any image can be rebuilt from source, pinned inputs, and versioned configuration |
| Safer OTA updates | Boot, rootfs, and hardware compatibility are checked before a device is allowed to reboot |
| Lower field risk | Known-bad update combinations fail before activation, reducing the chance of site visits |
| Faster platform evolution | Kernel, Debian, and hardware changes become planned migrations instead of bespoke projects |
Phase 1 — vendor images as a first-class input
We do not download images from the vendor website at build time. Vendor images are curated, checksummed, and stored on our own S3 bucket (s3://octave-energy-infrastructure/deliverables/mender/original/). This gives us reproducible, air-gapped builds and protects against upstream changes or disappearing downloads.
The image-sync CLI handles download, verification, and decompression:
uv run image-sync download --platform revpi --release bullseye
It verifies the .sha256 sidecar before decompressing the .img.xz, and fetches any existing SBOM sidecars so the provenance chain starts from the original image.
Image selection is deliberate. The RevPi Bullseye base image is pinned to 2023-09-26 (kernel 5.10.152-rt75-v8) rather than newer builds with kernel 6.1.46-rt13-v8. Devices already in the field run the 5.10 kernel; shipping a base image with a different kernel major would force a kernel incompatibility migration on every first install across a large fleet. The selection lives in build.json under platforms.revpi.releases.bullseye.source.file — one line, version-controlled, reviewed like any other infrastructure change. When a vendor releases a new kernel, the upgrade is deliberate: update build.json, rebuild, validate, release.
Boot partition normalisation happens before provisioning. Vendor Bookworm/Trixie images ship a 512 MiB boot FAT partition; devices already in the field have 256 MiB. image-builder run shrinks the boot partition back to 256 MiB before mender-convert so that over-the-air A/B rootfs swaps land safely on existing hardware.
Phase 2 — provisioning with pyinfra
The provisioned image is what we manage, test, and release. Every configuration decision — kernel version pinning, network setup, security hardening, hardware overlay, application runtime — is encoded in Python using pyinfra, an agentless infrastructure tool that applies operations over SSH or, in our case, directly into a chroot-mounted disk image.
image-builder run mounts the tuned image, starts an image-builder-image-tools Docker container, and runs infra/deploy.py inside the chroot. This file includes profile modules in a fixed order, gated on host.data keys that differ per platform and deliverable:
| Profile | Purpose |
|---|---|
firstboot.py |
Remove baked SSH host keys; enable regenerate_ssh_host_keys.service for fresh flashes |
boot_config.py |
Replace vendor config.txt with Octave’s version |
kernel.py |
Pin kernel major, purge Pi5 packages, apt-mark hold vendor kernel stack |
base_os.py |
Baseline packages, apt upgrade |
hardening.py |
Security hardening for production builds |
watchdog.py |
Compile and install Go-based hardware watchdog (RevPi only) |
can.py |
SocketCAN, PiCtory configuration, 80-can.network (RevPi only) |
greengrass.py |
AWS IoT Greengrass system accounts and CA trust store |
image_setup.py |
Device commissioning scripts (image-setup, image-restore) |
This is infrastructure-as-code in the most literal sense. A developer can read infra/deploy.py, trace every profile module, and understand exactly what a device image contains — no snowflakes, no manual SSH sessions baked in. A pull request that changes a profile is a reviewable, testable, diffable change.
Kernel pinning
kernel.py deserves a closer look because it carries significant OTA safety implications. The profile upgrades the kernel within the same major version only, then holds it. It also purges any rpi-2712 (Pi 5) kernel packages and ensures exactly one v8 module tree remains under /usr/lib/modules. This module directory name becomes the rootfs-kernel.version Mender Artifact provide — metadata that the Mender Client uses to match rootfs artifacts with the correct boot partition payload.
Hardware overlays and config.txt
Raspberry Pi hardware features are controlled through /boot/firmware/config.txt (Bookworm) or /boot/config.txt (Bullseye). Each platform has a curated template:
- Edgebox: I2C enabled, RTC overlay (
i2c-rtc,pcf8563), UART, CMA allocation. - RevPi: I2C enabled, RevPi-specific overlays (
revpi-con-can,revpi-connect-se), MAC address placeholders for the two Ethernet interfaces.
The boot_config.py profile copies the platform template onto the image. MAC addresses for RevPi are written at device commissioning time by image-setup-revpi, which reads them from hardware and appends dtparam=eth{n}_mac_{hi|lo} entries to /uboot/config.txt.
This is where we encountered one of the most instructive failures of the project. config.txt sits in the boot partition — and as we will see shortly, the boot partition is the hardest part of Raspberry Pi OTA.
Phase 3 — mender-convert: from tuned image to A/B artifact
Mender implements A/B rootfs updates: two rootfs partitions (mmcblk0p2, mmcblk0p3), a persistent data partition, and atomic slot switching with automatic rollback on failure. mender-convert takes a standard Linux disk image and produces a Mender-compatible image with this partition layout, plus .mender artifacts for OTA delivery.
We run mender-convert inside Docker to isolate the privileged partition operations. The mender-tool build command:
- Clones
mendersoftware/mender-convertat the tag specified inbuild.json. - Runs
mender/mender-convert.shto merge our overlay (mender/octave-convert/) into the checkout. - Calls
docker-mender-convertwith stacked config files. - Collects the outputs (
deploy/) intooutput/.
Config layering is additive. The base config (mender_convert_config) sets storage layout (256 MiB boot, 6 GiB data), LZMA compression, and our custom mender_create_artifact() function. Platform/release configs layer on top, setting device type (revpi-connect-se-16, edgebox-rpi-200-16) and APT repository URLs for Mender packages.
The Octave overlay
mender/octave-convert/input/octave-overlay/ is merged into the converted rootfs. It ships:
- Mender Update Modules —
usr/share/mender/modules/v3/uboot, a custom module that handles boot partition updates as a separate artifact type. - Inventory scripts —
mender-inventory-kernel(exposes kernel version, RT flag, revision) andmender-inventory-hardware(exposes platform, revision, serial). These appear as device attributes in the Mender UI. - Greengrass installer — the nucleus tarball, FleetProvisioningByClaim plugin, and Amazon root CAs, placed in
data/greengrass/GreengrassInstaller/so the application layer can self-install on first boot. - Mender Connect configuration (
etc/mender/mender-connect.conf).
Greengrass binaries are downloaded at convert-time by mender-convert.sh, not baked into source control. The script fetches pinned versions from AWS and validates their checksums before incorporating them.
When A/B rootfs is not enough
The most important lesson from operating this fleet is that A/B rootfs is not the same as A/B platform state. On Raspberry Pi-derived hardware, the hardware description lives outside the rootfs, and kernel upgrades are only safe if the shared boot partition moves in lockstep with the rootfs.
Standard Mender A/B updates cover the rootfs: new software is written to the passive slot, the bootloader is told to try it, and the system reboots. If a validation script passes within the timeout, the new slot is committed; otherwise Mender rolls back.
Raspberry Pi adds a complication. The boot partition — mmcblk0p1 — is a single shared FAT volume that holds the Raspberry Pi firmware, U-Boot entrypoint (kernel8.img), device tree blobs, overlays, config.txt, and cmdline.txt. It is not part of the A/B rootfs scheme. In our converted layout, the real Linux kernel lives in the selected rootfs, but it is booted using hardware description and boot configuration from this shared partition. When a Debian or kernel-generation upgrade changes the expected device-tree bindings or boot-support stack, the new rootfs may no longer boot correctly with the old boot partition’s overlays and firmware configuration.
The unsafe state is a cross-generation mix: a new rootfs with old boot-support files, or an old rootfs with new ones. Either may boot by accident, but neither is a validated platform state.
The failure mode is not theoretical. With a 6.x rootfs booting against a 5.x-era /uboot, the RevPi piControl stack loaded but could not bind the pibridge backend — the device-tree overlays and firmware version in /uboot no longer matched what the new rootfs drivers expected. /dev/piControl0 never appeared, which meant our watchdog service could no longer pet the hardware watchdog — a physical safety risk, not just a software problem.
We solve this with a three-artifact migration pattern and a custom Mender Update Module.
The uboot Update Module
The /uboot partition is effectively shared platform state: hardware description, firmware assumptions, bootloader configuration, and compatibility markers that must match the rootfs generation. A .uboot.mender artifact carries the full boot partition tree as uboot.tar. The Update Module (ArtifactInstall) extracts this into /data/uboot/<kernel-version>/ — a staging area on the persistent data partition — and writes a staged-by-mender marker last, making the staging atomic. Because /data survives A/B rootfs swaps, it is the only safe place to stage boot payloads and keep rollback backups across the migration. This is a no-reboot, staging-only operation (NeedsArtifactReboot: No).
Bootstrap requirement. Because custom Mender Update Modules must already exist on the device, the first rollout step is a normal rootfs artifact that installs /usr/share/mender/modules/v3/uboot. Only after that can devices accept type: uboot artifacts. This is a real-world Mender gotcha worth stating explicitly: the Update Module ships with the rootfs, not with the uboot artifact.
Three-artifact migration sequence
Step 1 - Rootfs bootstrap artifact
Installs /usr/share/mender/modules/v3/uboot
(no kernel change yet; boot partition untouched)
Step 2 - Uboot artifact (.uboot.mender)
Extracts uboot.tar -> /data/uboot/<target-kernel>/
Writes staged-by-mender marker (atomic)
No reboot; staging only
Step 3 - Rootfs platform artifact
Writes new rootfs to passive slot
ArtifactInstall_Leave_90 detects kernel incompatibility
Validates staged /data/uboot/<target-kernel>/config.txt
Preserves device state (MACs, CAN overlay) from live /uboot
Copies staged tree onto /uboot immediately before reboot
System reboots into new rootfs + new kernel
ArtifactCommit_Enter_90 validates hardware liveness
Commits only if all checks pass; rolls back otherwise
The compatibility check (kernels_compatible()) compares the kernel major of the live /uboot against the kernel embedded in the new rootfs using a configurable policy (UBOOT_SWITCH_POLICY=major). If incompatible and no valid staged payload exists, the state script panics — causing Mender to roll back before the device reboots into an unvalidated platform state.
The bootstrap rootfs artifact is only required once, for devices that do not yet have the custom uboot Update Module installed. After that, boot-support generation migrations use the staged uboot artifact followed by the rootfs platform artifact — a two-step sequence.
State scripts
All state scripts share a common library (_lib.sh) and follow a naming convention that maps to Mender update states:
| Script | State | Purpose |
|---|---|---|
ArtifactInstall_Enter_20_prepare_uboot_backup |
Install Enter | Snapshot live /uboot to /data/uboot/migration/current/ |
ArtifactInstall_Leave_70_preserve_device_state |
Install Leave | Copy hostname, SSH keys, network config into new rootfs slot |
ArtifactInstall_Leave_90_switch_uboot_if_required |
Install Leave | Activate staged boot tree if kernel changed |
ArtifactCommit_Enter_90_validate_runtime |
Commit Enter | Post-reboot: check kernel match, test piControl/watchdog (RevPi) or RTC (Edgebox) |
ArtifactCommit_Leave_90_finalize_uboot_migration |
Commit Leave | Archive migration, prune retention |
ArtifactRollback_Enter_20_restore_uboot_backup |
Rollback Enter | Restore /uboot snapshot if switch occurred |
The post-reboot validation script (ArtifactCommit_Enter_90_validate_runtime) is where platform-specific hardware liveness checks live. On RevPi it calls piTest -1 -r RevPiLED — the -1 flag is critical. Without it, piTest enters a continuous polling loop and the script hangs indefinitely, leaving the Mender commit window open until timeout. On Edgebox it checks the hardware RTC. If either check fails, Mender rolls back.
Phases 4–6 — SBOM, S3, and hosted Mender
Software bill of materials
image-builder sbom mounts the converted rootfs (.ext4) inside the image-tools container and runs Syft, producing three SBOM formats:
{deploy}.sbom.spdx.json.xz{deploy}.sbom.cdx.json.xz{deploy}.sbom.syft.json.xz
The CI pipeline runs Grype against the Syft SBOM for vulnerability scanning. SBOM files are uploaded to S3 alongside the artifacts, giving an auditable component inventory for every released image version.
Release naming
Every build run is identified by ARTIFACT_VERSION — a semver on CI (computed from conventional-commit prefixes in the Git log), a timestamp locally. Names are derived deterministically:
| Artifact | Name pattern |
|---|---|
| Mender rootfs release | octave-{release}-{version} |
| Mender uboot release | octave-{release}-uboot-{version} |
| S3 deploy basename | {platform_short}-{release}-64b-{version} |
A single ARTIFACT_VERSION covers both platforms in one CI run, so octave-bookworm-1.2.3 means the same software generation regardless of platform.
S3 upload
image-sync upload publishes to:
s3://octave-energy-infrastructure/deliverables/mender/releases/{platform}/{octave-release}/
Files per platform: .mender rootfs artifact, .img.xz disk image, .cfg (device type metadata), SBOM files, and optionally .uboot.mender. The uploader refuses to overwrite an existing release unless --force is passed — releases are immutable by default.
Hosted Mender
mender-cloud uploads the .mender and .uboot.mender artifacts and tags the Mender release with either release (main branch) or dev (feature branches). The tag controls which devices are eligible to receive updates through Mender’s deployment filters.
The CI/CD pipeline
The GitHub Actions workflow runs on workflow_dispatch with inputs for platform selection, release codename, and optional flags. A setup job computes the version and builds the platform matrix; one pipeline job runs per platform on an ubuntu-24.04-arm64-8core runner (mender-convert needs ARM64 and significant memory).
setup job
+-- gha-compute-version (semver from conventional commits)
+-- platform matrix from build.json
pipeline job (per platform, parallel)
+-- OIDC AWS credentials
+-- task check-host-deps -- release
+-- task build-plan
+-- task release
| +-- Phase 1: image-sync download
| +-- Phase 2: image-builder run (pyinfra, boot partition normalise)
| +-- Phase 3: mender-tool build (mender-convert, overlay, state scripts)
| +-- Phase 4: image-builder sbom (Syft)
| +-- Phase 5: image-sync upload (S3)
| +-- Phase 6: mender-cloud upload (hosted Mender + tag)
+-- Grype vulnerability scan
+-- Upload logs + SBOMs as GitHub artifacts
summary job
+-- Aggregate per-platform run summaries
On the main branch, S3 and hosted Mender uploads are live and a mender_<version> Git tag is pushed. On feature branches, uploads are skipped and the Mender release is tagged dev, so integration testing can happen against real artifacts without polluting the production release catalogue.
What changed operationally
The biggest change is not that image creation became automated. The bigger change is that fleet operations became explainable.
Before this work, a device image was the result of a sequence of build and configuration steps that were difficult to audit as a whole. After this work, a device image is a traceable release artifact: built from pinned inputs, transformed by version-controlled code, verified by CI, published immutably, and rolled out through Mender with compatibility checks on the device itself.
| Concern | Previous model | New model |
|---|---|---|
| Base image | Manually selected / downloaded | Pinned in build.json, checksummed in S3 |
| Platform differences | Managed per hardware type | Encoded as pyinfra profiles and platform templates |
| Kernel upgrades | Risky one-off projects | Explicit rootfs + /uboot compatibility protocol |
| Boot validation | Mostly manual | Build-time, artifact-time, upload-time, and on-device gates |
| Release evidence | Logs and tribal knowledge | SBOMs, CI artifacts, S3 releases, Mender metadata |
| Rollback | Rootfs A/B only | Rootfs rollback plus best-effort /uboot restore |
| Fleet readiness | Hard to prove before rollout | Device inventory and artifact metadata expose compatibility |
The result is a platform where operational risk is handled as part of the release architecture, not as a checklist after the build.
IaC principles, applied
Looking at the pipeline as a whole, the classical IaC principles map cleanly onto embedded device management:
Declarative over imperative. build.json declares what each platform’s image should contain; the pipeline figures out how to build it. Adding a new Debian release is a two-line change to build.json.
Idempotency. pyinfra operations are idempotent: running boot_config.py twice produces the same result. octave_fixup_boot_config_txt repairs state rather than failing if the upstream hook already ran. image-setup-revpi removes existing MAC entries before re-adding them.
Immutability. Images are not modified after build. A released version is a fixed artifact in S3. Updates replace the rootfs, they do not patch it in place. The S3 uploader enforces this by refusing overwrites.
Version control everything. Every config file, profile module, state script, and build configuration is in Git. The vendor image version is pinned in build.json. The mender-convert version is pinned per release codename. Greengrass nucleus is pinned by version and checksum.
Gate early and often. Boot config checks run at build time (mender-convert hook), artifact time (mender-tool validate-boot), upload time (tasks.sh), and device time (state scripts). A glued kernel= line cannot reach a device without tripping one of these gates.
Rollback by design. Mender’s A/B scheme gives rollback for rootfs updates. The backup/restore state scripts extend this to the shared boot partition. The three-artifact migration pattern ensures that a bad boot payload never activates without a paired, validated rootfs ready to use it.
Limits and trade-offs
The most important limitation to state openly: the boot partition is shared, not A/B. If the device loses power after /uboot has been overwritten but before the new rootfs has booted and committed, no rollback script can restore the old boot partition — the Mender Client is not yet running. We mitigate this by staging the new boot tree on /data well in advance, validating it before activation, keeping a timestamped backup on the persistent partition, and only physically copying the staged tree onto /uboot in the last moments before reboot. But the risk cannot be eliminated without making the boot partition itself A/B, or adding an external hardware recovery path (BMC, watchdog-driven recovery partition, or USB recovery mode).
For our fleet, this trade-off is acceptable: the staging and validation chain makes a silent failure extremely unlikely. The hardware watchdog can recover some classes of runtime hangs by forcing a reboot, but it is not a substitute for a rollback-capable boot partition. If the device cannot boot Linux far enough to run the Mender Client, recovery still requires another path. The key discipline is to treat every /uboot switch as a one-way operation until commit — which is exactly what the state script sequencing enforces.
Lessons learned
The gap between IT and OT is a gap of consequence, not complexity. The technologies — Linux, Python, Docker, Git, AWS — are familiar. What changes is the blast radius of a failure. A misconfigured cloud VM is terminated and replaced in minutes. A misconfigured boot partition might mean a field visit. This raises the bar for pre-release validation but does not change the tools.
Multiple small defences beat one big one. The config.txt corruption incident could have been prevented by any one of the layers we eventually added. In practice, none of the layers existed before the incident. Adding them after meant thinking clearly about where in the pipeline each check belonged and what evidence it had access to. A validation that runs at upload time has access to the final artifact bytes; a state script has access to the live device; a CI test has access to source truth. Each layer checks something the others cannot.
Kernel upgrades are infrastructure migrations. Changing a kernel major on a fleet of devices is not a firmware flash; it is a coordinated migration across two artifacts with a defined rollback path. Treating it as such — with its own artifact type, compatibility detection, staging protocol, and rollback procedure — is what makes it safe to do at scale.
Foreshadow your hard problems. The boot partition challenge was obvious in hindsight but not designed for up front. Building the three-artifact protocol after encountering the incompatibility was more expensive than designing for it initially would have been. For future platforms, the first question to ask of any OTA scheme is: what shared state sits outside the A/B boundary?
Treat edge devices as a software supply chain, not as installed appliances. Once devices are deployed, every image, package, kernel, overlay, and configuration file becomes part of the operational risk model. The only sustainable way to manage that risk is to make the full chain reproducible and reviewable. This is not an engineering principle — it is an operational one. The teams who own the devices, not just the teams who build the software, benefit from it.
Conclusion
The real shift is cultural and architectural: device images are no longer artifacts produced by a person at a bench. They are infrastructure outputs produced by code, from pinned inputs, through a repeatable pipeline, with evidence at every step.
Mender gives us the OTA delivery primitives. pyinfra gives us idempotent, reviewable provisioning. GitHub Actions, S3, SBOMs, and validation gates turn the process into a governed software supply chain. Together, they make fleet-scale controlled kernel migrations and platform upgrades tractable on devices that sit in customer environments and cannot be treated like disposable cloud instances.
The outcome is a system where updating Greengrass, changing a hardware overlay, tightening a firewall rule, or moving to a new Debian release is a pull request: reviewed, verified automatically, released immutably, and deployed with rollback-aware safeguards.
That is Infrastructure as Code with real consequences: not virtual machines in a region, but industrial computers on customer sites, expected to update safely without anyone touching a USB stick.
Edit by @TheYoctoJester, 2026-08-13: formatting and move to board integration category