Mender, CM4, `dm-verity`

Hello there!

I am, with a lot of help from Claude, working on a secure boot chain for the CM4 with dm-verity. Once I get it working my plan is to untangle it from our proprietary layer and make it into its own, open source layer.

As has been discussed on this forum previously, we need a custom update module that will find the underlying block device, and the dm-verity root hash cannot go on in rootfs (because chicken/egg) nor in a single location (because A/B updates).

Claude proposed a per-slot FIT in boot.img containing the root hash. Which made me think perhaps we should move the kernel into that same file (rather than /boot/fitImage). Thoughts? One disadvantage is that the kernel is not protected by dm-verity but I am not sure this is an issue in practice (I don’t think there’s a splice attack).

In this case the layout would be as follows:

  FAT (kernel lives here, per slot)
                                                                                                                                                                                                                                                                                                                            
  config.txt        15
  boot.img    3,213,312                                                                                                                                                                                                                                                                                                     
  boot.sig          602                                                                                                                                                                                                                                                                                                     
  boot.scr        4,756                                                                                                                                                                                                                                                                                                     
  boot-2.fit 10,848,705                                                                                                                                                                                                                                                                                                     
  boot-3.fit 10,848,705                                                                                                                                                                                                                                                                                                     
                         109 MB free of 128 MB
                                                                                                                                                                                                                                                                                                                            
  boot.fit content summary                                                                                                                                                                                                                                                                                                  
  
  FIT description: xebra slot boot (kernel + dtb stub + bootargs script)                                                                                                                                                                                                                                                    
   Image 0 (kernel)   gzip,  10,788,495 B (10.29 MiB)  AArch64/Linux  load=0x20008000  sha256                                                                                                                                                                                                                               
   Image 1 (fdt)      none,      56,928 B (55.59 KiB)  AArch64        sha256                                                                                                                                                                                                                                                
   Image 2 (script)   none,         552 B              Script         sha256                                                                                                                                                                                                                                                
   Default Configuration: 'conf-1'                                                                                                                                                                                                                                                                                          
   Configuration 0 (conf-1)                                                                                                                                                                                                                                                                                                 
    Kernel:       kernel                                                                                                                                                                                                                                                                                                    
    FDT:          fdt
    (script:      script)              ← present per fdtget (dumpimage doesn't print it)                                                                                                                                                                                                                                    
    Sign algo:    sha256,rsa2048:boot_img                                                                                                                                                                                                                                                                                   
    Sign padding: pkcs-1.5                                                                                                                                                                                                                                                                                                  
    sign-images:  kernel fdt script    ← all three covered by one signature                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                            
  .mender artifact payload                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                            
  Type: rootfs-verity
  Files:                                                                                                                                                                                                                                                                                                                    
    rootfs.ext4.verity   2,654,769,152 B   sha256 e06927fc...
    boot.fit                10,848,705 B   sha256 81c2d91c...                                                                                                                                                                                                                                                               
  2 payload files, no separate fitImage. Signature on artifact present.
                                                                                                                                                                                                                                                                                                                            

Cheers,
Luke

Putting this in a separate post because it is Claude-generated (with some cleanups by me):

Secure-boot chain on RPi CM4 (with dm-verity, no initramfs)

End-to-end, every link verified by the previous one:

  1. VideoCore ROM verifies boot.img (the inner FAT image holding GPU firmware, U-Boot, base DTB, overlays, cmdline.txt, config.txt) against an RSA pubkey hash fused into OTP. That OTP fuse + SIGNED_BOOT=1 in EEPROM is what makes the CM4 a hardware root of trust.

  2. U-Boot runs from inside boot.img and sources a signed boot.scr from the outer FAT. boot.scr is a FIT-wrapped Hush script with a signature-1 node; U-Boot’s source command verifies it against the RSA pubkey embedded in U-Boot’s control DTB (u-boot.dtb) and refuses tampered or unsigned scripts.

  3. boot.scr loads a per-slot signed FIT from the FAT named boot ${mender_boot_part}.fit (one per Mender A/B slot). This is the ‘interesting’ (Claude’s words) bit: it’s a single signed FIT containing three sub-images — gzipped Linux kernel, a stub DTB (covered by the signature; runtime DTB comes from VC firmware via bootm’s external-FDT arg), and a Hush bootargs script with the per-build dm-verity roothash baked into its signed bytes. One configurations.conf-1carries kernel/fdt/script properties; one signature-1 covers all three. boot.scr does:

    load mmc 0:1 ${kernel_addr_r} boot-${mender_boot_part}.fit
    source ${kernel_addr_r}                  # verifies sig, runs script
    fdt move ${fdt_addr} ${fdt_addr_r}
    bootm ${kernel_addr_r} - ${fdt_addr_r}   # verifies same sig, boots
    

    source and bootm both resolve the default config and verify the same signature node — two consumers, one signature, one atomic per-slot file.

  4. The signed script sets the kernel cmdline (setenv bootargs '... dm-mod.create="..." ${mender_kernel_root} ... root=/dev/dm-0 ...'). The script is wrapped in single quotes so Hush doesn’t eat ${mender_kernel_root} at source time; CONFIG_BOOTARGS_SUBST=y makes bootm substitute it in place without re-tokenising, so the embedded dm-mod.create="..." double quotes survive into /proc/cmdline.

  5. The kernel boots with CONFIG_DM_INIT, which parses dm-mod.create= at late_initcall and brings up /dev/dm-0 before rootfs mount. Every block read from the rootfs is then verified against the signed roothash. No initramfs.

  6. Mender OTA ships a single .mender artifact carrying rootfs.ext4.verity + boot.fit. A custom Update Module streams the rootfs to the inactive partition and boot.fit to /uboot/boot-${passive}.fit on the FAT, then flips mender_boot_part. New slot has a matching {rootfs, kernel, roothash} triple, signed by the same RSA key.

Same RSA key (UBOOT_SIGN_KEYNAME) signs boot.scr, boot-${slot}.fit, and the kernel image inside it. (Luke notes: re-using the key may not be ideal?)

The public key lives in U-Boot’s control DTB, which is itself baked into the U-Boot binary inside VC-signed boot.img. Trust roots in OTP and propagates forward without ever relying on writable storage state (env vars, unsigned scripts, mutable boot partitions).

Hi @lukehatpadl,

Thanks for getting this started! I’m also looking into the topic a bit, and I actually think that the approach by Claude is not the right way to go here. Especially the part which repeatedly modifies the boot partition is not a good idea, I would say.

What it did get right:

  • the kernel+initrd should be bundled in a signed fitImage, which gets verified by u-boot, and only loaded+executed if the check was successful.
  • from there, the kernel then uses a dm-verity signed root filesystem, a read only one strongly preferred.
  • Mender ships a root filesystem+fitImage artifact, which mandates a new Update Module to be created

Where it did go wrong:

Technically a more RPi-specific structure could even be used, like the tryboot flow (pre-Alpha quality PoC at feat: Raspberry Pi tryboot A/B support by TheYoctoJester · Pull Request #501 · mendersoftware/meta-mender-community · GitHub), but such a fitImage+dm-verity flow should be a pretty generic concept in my opinion, and I’m looking into creating such an approach. Hopefully I can show something soon.

Greetz,
Josef

Some brief thoughts (away from desk this week so, brief).

  • dm-verity without initramfs seems to work fine here (see here)
  • having a FIT image on the boot partition with the root hash and kernel, vs A/B kernel partitions seems a matter of taste? at the end of the day, a supported third-party layer trumps taste for me, though
  • using the OTP ECDSA key seems very neat, so yes, but also not necessary cryptographically if we have a secure boot chain with a signed boot loader
  • tryboot I suspect would boot faster but, less portable. See also this upcoming talk (although I know they don’t want to support Mender, and FDE is not something I want or need)

If you want to reach out of email (lukeh at lukktone dot com) I can share my changes.

Hi @lukehatpadl,

Matching your brief thoughts:

  • yes, dm-verity in a fitImage without initramfs might work in this specific (RPi) case, but my gut feeling is that for a generic solution it (or a similar approach) is required
  • the question always is, what is the trust anchor for the signed boot loader
  • yes, I’m aware of Ayoub’s work (in fact, I reviewed and was involved with accepting that presentation :slight_smile: ). But again as you rightfully say, not generic/portable.

I’ll get in touch with you, definitely interested in how and what you put together.

Greetz,
Josef

Mender + dm-verity + U-Boot signed boot on Raspberry Pi CM4

Here’s a Claude-generated, human-edited writeup of the secure boot implementation I (the human) had Claude implement recently for our CM4-based embedded system. There’s nothing particularly novel here but I couldn’t find anything that put all the pieces in one place with the requirements we had (specifically, using Mender with U-Boot and dm-verity on a CM4, and not requiring initramfs or a separate boot partition per slot).

Survey of existing approaches

  • meta-raspberrypi-secure (Embetrix) — covers the RPi-side signed boot: boot.img packaging, rpi-eeprom-digest, OTP fuse workflow, EEPROM lockdown. Excellent reference for the VC ROM half of the chain. Does not integrate with Mender (it assumes a stock rootfs flow) and does not wire into U-Boot’s FIT signature pipeline.

  • meta-mender + meta-mender-community — A/B slot management, mender-artifact, bootcount-rollback, U-Boot env integration. Does not assume an integrity layer over the rootfs; the stock rootfs-image Update Module writes a raw blob to the inactive slot with no expectation that what’s at rest in those blocks has to match a hash the bootloader will later verify.

  • ejaaskel.dev’s dm-verity walkthrough — a clean guide to getting a verity-protected rootfs booting under Yocto via initramfs. A great baseline for understanding the build-side machinery (image_types_verity, hash-tree generation, kernel cmdline). Doesn’t cover Mender A/B (the roothash is single-slot), doesn’t address what to do about per-slot bootargs, and uses an initramfs rather than CONFIG_DM_INIT.

The combined problem none of the above solves:

Two A/B rootfs slots, each with its own dm-verity roothash, behind
a fully signed U-Boot chain, updatable as a single atomic OTA, on a
CM4 with SIGNED_BOOT=1 and OTP fused — and no initramfs.

The core problem: where does the per-slot roothash live?

A dm-verity rootfs commits to a roothash that the kernel must be told about at mount time. With A/B slots, the two slots have different roothashes (they’re different filesystem images). So an OTA that updates the rootfs has to also update the authoritative record of the roothash that the bootloader will pass to the new kernel. Otherwise the kernel dm-inits the wrong table and the rootfs fails to mount.

Three families of places to put the per-slot roothash:

  1. In the kernel cmdline that the bootloader composes. Requires bootloader to know which slot is active and pick the right hash. Stock boot.cmd.in doesn’t do this.

  2. In U-Boot env. Tempting — Mender already manages mender_boot_part there — but env is writable from user space (fw_setenv).

  3. In a per-slot file on the boot partition, covered by a U-Boot FIT signature.

We chose 3, above:

  • One signed FIT per slot on the shared FAT boot partition: boot-2.fit, boot-3.fit (named after the Mender partition number).
  • Each FIT contains three subimages: kernel, fdt (stub), script.
  • The script is a Hush setenv bootargs '...' carrying that slot’s dm-verity cmdline with the slot’s roothash baked in.
  • One signature covers all three subimages. U-Boot’s source command verifies the signature and runs the script (sets env bootargs). bootm against the same FIT address verifies the same signature and boots the kernel.
  • The OTA payload is the rootfs plus the new slot’s FIT. A custom Mender Update Module installs both atomically (well, as atomically as is possible, given power could be interrupted when installing a single image I don’t see this as too different).
VideoCore GPU ROM (hardware root of trust)
  → boot.img                 (RSA-2048, VC-verified post-OTP)
    → U-Boot kernel8.img     (inside boot.img)
      → boot.scr             (FIT-format, ECDSA P-256 signed)
        → boot-<N>.fit       (ECDSA P-256 signed; per-slot)
              kernel  : Linux Image (gzip-compressed)
              fdt     : stub DTB (FIT-format requirement)
              script  : setenv bootargs with this slot's
                        dm-verity roothash + cmdline
          → kernel + DTB → CONFIG_DM_INIT → /dev/dm-0 (verity rootfs)

There are thus two distinct signing keys:

  • boot_img — RSA-2048. Algorithm fixed by Broadcom silicon (OTP holds SHA-256 of an RSA-2048 pubkey). Signs boot.img using rpi-eeprom-digest.
  • fit_sign — ECDSA P-256, covering all internal FITs (boot.scr, boot-<N>.fit, the build-time linux fitImage). This public key is injected into U-Boot’s control DTB by the kernel-fit-image’s pass.

Rootfs filesystem choice: erofs + dm-verity

We use erofs (with -zlz4) for the rootfs, not ext4 (I learned about that here). dm-verity is block-level and fs-agnostic, so it works with either.

One wic-side gotcha: erofs (like squashfs) cannot be “formatted” empty, so meta-mender’s default of leaving the inactive slot empty trips with “It’s not possible to create empty erofs partition”. The fix is as follows:

# machine conf
ARTIFACTIMG_FSTYPE = "erofs"
EXTRA_IMAGECMD:erofs = "-zlz4"

# Pre-populate the inactive slot at factory-flash with the same rootfs
# image. Costs ~rootfs-size additional bytes in the .sdimg (not in OTAs).
MENDER_FEATURES_ENABLE:append = " mender-prepopulate-inactive-partition"

# /data must stay writable; pin it to ext4 regardless of rootfs fstype.
MENDER_DATA_PART_FSTYPE = "ext4"

# Mount the U-Boot FAT read-only by default. The Update Module brackets
# its boot-<N>.fit write with remount,rw + remount,ro.
MENDER_BOOT_PART_FSTAB_OPTS = "ro"

Build-side classes

Kernel config fragment

No initramfs — CONFIG_DM_INIT lets the kernel parse a dm-mod.create= cmdline argument and set up /dev/dm-0 before mounting /:

# dm-verity built in so the kernel can mount the rootfs through a
# dm-mod.create=... cmdline arg with no initramfs involved.
CONFIG_BLK_DEV_DM=y
CONFIG_DM_INIT=y
CONFIG_DM_VERITY=y

# erofs + LZ4 for the dm-verity-protected rootfs.
CONFIG_EROFS_FS=y
CONFIG_EROFS_FS_ZIP=y
CONFIG_EROFS_FS_ZIP_LZ4=y

verity-rootfs.bbclass

Pairs with meta-oe’s image_types_verity (which produces .${ARTIFACTIMG_FSTYPE}.verity + a .verity-params shell fragment) and overwrites the plain rootfs blob in IMGDEPLOYDIR with the verity variant so Mender’s sdimg/.mender/bootstrap-artifact consumers pick it up unchanged.

inherit image_types_verity

# Read the same rootfs filename the rest of the pipeline produces.
# Without this, image_types_verity defaults to ext4 even when
# ARTIFACTIMG_FSTYPE is something else (e.g. erofs).
VERITY_IMAGE_FSTYPE = "${ARTIFACTIMG_FSTYPE}"

# Hash tree overhead reservation: ~0.8% of data at SHA-256/4 KiB.
# 32 MiB covers rootfses up to ~4 GiB with generous headroom.
VERITY_HASH_TREE_OVERHEAD_KB ?= "32768"

# Mender computes IMAGE_ROOTFS_SIZE from partition geometry; shrink it
# by the hash-tree overhead so rootfs + hash_tree fits the partition.
python __anonymous() {
    if 'verity' not in (d.getVar('IMAGE_FSTYPES') or '').split():
        return
    calc_kb = d.getVar('MENDER_CALC_ROOTFS_SIZE')
    if not calc_kb:
        return
    overhead_kb = int(d.getVar('VERITY_HASH_TREE_OVERHEAD_KB') or 0)
    extra_kb = int(eval(d.getVar('IMAGE_ROOTFS_EXTRA_SPACE') or '0'))
    new_size = int(calc_kb) - extra_kb - overhead_kb
    d.setVar('IMAGE_ROOTFS_SIZE', str(new_size))
}

# Force consumers to wait for the verity overwrite.
IMAGE_TYPEDEP:sdimg:append              = "${@bb.utils.contains('IMAGE_FSTYPES', 'verity', ' verity', '', d)}"
IMAGE_TYPEDEP:mender:append             = "${@bb.utils.contains('IMAGE_FSTYPES', 'verity', ' verity', '', d)}"
IMAGE_TYPEDEP:bootstrap-artifact:append = "${@bb.utils.contains('IMAGE_FSTYPES', 'verity', ' verity', '', d)}"

# Replace the plain artifact with the .verity variant as a post-step
# of do_image_verity (we can't modify do_image_verity itself).
python verity_rootfs_replace_rootfs() {
    import os, shutil
    imgdeploydir = d.getVar('IMGDEPLOYDIR')
    image_name   = d.getVar('IMAGE_NAME')
    image_link   = d.getVar('IMAGE_LINK_NAME')
    fstype       = d.getVar('ARTIFACTIMG_FSTYPE')
    verity = os.path.join(imgdeploydir, image_name + '.' + fstype + '.verity')
    rootfs = os.path.join(imgdeploydir, image_name + '.' + fstype)
    link   = os.path.join(imgdeploydir, image_link + '.' + fstype)
    if not os.path.exists(verity):
        bb.fatal("verity-rootfs: %s not produced by do_image_verity" % verity)
    shutil.copyfile(verity, rootfs)
    if os.path.islink(link) or os.path.exists(link):
        os.remove(link)
    os.symlink(image_name + '.' + fstype, link)
}
do_image_verity[postfuncs] += "verity_rootfs_replace_rootfs"

The per-slot signed boot FIT

The build assembles a FIT containing { kernel, stub DTB, bootargs script } and signs it with UBOOT_SIGN_KEYNAME (ECDSA P-256). The bootargs script reads the verity parameters that image_types_verity emitted alongside the rootfs:

# Read verity params (data sectors, blocks, algo, roothash, salt, ...)
VERITY_PARAMS_FILE="${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${ARTIFACTIMG_FSTYPE}.verity-params"
. "${VERITY_PARAMS_FILE}"

# Pre-gzip the kernel for FIT compression="gzip".
gzip -9 -n -c "${KERNEL_IMG}" > "${KERNEL_GZ}"

# Hush single-quoted literal. ${mender_kernel_root} stays as text;
# CONFIG_BOOTARGS_SUBST resolves it at bootm time. The single quotes
# prevent Hush from re-parsing the embedded dm-mod.create="..." double
# quotes at `source` time (which would otherwise lose them and break
# the kernel cmdline argument).
cat > "${BOOT_SCRIPT}" <<BOOTARGS_EOF
setenv bootargs '$CMDLINE_BASE dm-mod.waitfor=\${mender_kernel_root} dm-mod.create="vroot,,,ro,0 ${VERITY_DATA_SECTORS} verity 1 \${mender_kernel_root} \${mender_kernel_root} ${VERITY_DATA_BLOCK_SIZE} ${VERITY_HASH_BLOCK_SIZE} ${VERITY_DATA_BLOCKS} ${VERITY_DATA_BLOCKS} ${VERITY_HASH_ALGORITHM} ${VERITY_ROOT_HASH} ${VERITY_SALT} 1 ignore_zero_blocks" root=/dev/dm-0'
BOOTARGS_EOF

The ITS — one config carrying three subimages under one signature:

/dts-v1/;
/ {
    description = "Per-slot boot FIT (kernel + dtb stub + bootargs script)";
    #address-cells = <1>;
    images {
        kernel {
            data = /incbin/("${KERNEL_GZ}");
            type = "kernel";
            arch = "arm64";
            os = "linux";
            compression = "gzip";
            load = <0x20008000>;
            entry = <0x20008000>;
            hash-1 { algo = "sha256"; };
        };
        fdt {
            description = "Stub DTB (FIT-internal; runtime uses VC-merged DTB at fdt_addr)";
            data = /incbin/("${KERNEL_DTB}");
            type = "flat_dt";
            arch = "arm64";
            compression = "none";
            hash-1 { algo = "sha256"; };
        };
        script {
            description = "dm-verity bootargs setenv";
            data = /incbin/("${BOOT_SCRIPT}");
            type = "script";
            arch = "arm64";
            compression = "none";
            hash-1 { algo = "sha256"; };
        };
    };
    configurations {
        default = "conf-1";
        conf-1 {
            kernel = "kernel";
            fdt    = "fdt";
            script = "script";
            hash-1 { algo = "sha256"; };
            signature-1 {
                algo = "sha256,ecdsa256";
                key-name-hint = "${UBOOT_SIGN_KEYNAME}";
                sign-images = "kernel", "fdt", "script";
            };
        };
    };
};

Note the device tree is present just so it builds and is a valid fitImage; the kernel really needs the merged device tree from the VC ROM (I tried).

A build-time check verifies the signature at build time:

mkimage -f "${BOOT_ITS}" -k "${UBOOT_SIGN_KEYDIR}" -r "${BOOT_FIT}"

# fit_check_sign runs the same verifier path U-Boot's bootm/source use.
fit_check_sign \
    -k "${DEPLOY_DIR_IMAGE}/u-boot.dtb" \
    -f "${BOOT_FIT}" > check.log 2>&1 \
    || (cat check.log >&2; bbfatal "boot.fit signature not verifiable against u-boot.dtb")

grep -q 'sha256,ecdsa256' check.log \
    || bbfatal "FIT ECDSA signature present but not enforceable. Check:
  - UBOOT_SIGN_ENABLE = \"1\"
  - u-boot.dtb has /signature/key-<KEY> with required=\"conf\"
  - mkimage produced a signature-1 node with algo=\"sha256,ecdsa256\""

Signing boot.scr itself

Stock mkimage -f auto -T script emits a FIT without a signature node, so to close the loop we hand-write the ITS in the bbappend:

DEPENDS += "u-boot-tools-native"

do_compile() {
    cat > "${WORKDIR}/boot.its" <<BOOT_ITS_EOF
/dts-v1/;
/ {
    description = "boot script";
    #address-cells = <1>;
    images {
        bootscript {
            data = /incbin/("${WORKDIR}/boot.cmd");
            type = "script";
            arch = "${UBOOT_ARCH}";
            compression = "none";
            hash-1 { algo = "sha256"; };
        };
    };
    configurations {
        default = "conf-1";
        conf-1 {
            script = "bootscript";
            hash-1 { algo = "sha256"; };
            signature-1 {
                algo = "sha256,ecdsa256";
                key-name-hint = "${UBOOT_SIGN_KEYNAME}";
                sign-images = "script";
            };
        };
    };
};
BOOT_ITS_EOF

    mkimage -f "${WORKDIR}/boot.its" \
        -k "${UBOOT_SIGN_KEYDIR}" -r boot.scr
}

boot.cmd — what U-Boot actually runs

run mender_setup

mmc dev 0
load mmc 0:1 ${kernel_addr_r} boot-${mender_boot_part}.fit \
    || run mender_try_to_recover
source ${kernel_addr_r}                       || run mender_try_to_recover

# VC firmware parks the overlay-merged DTB at a high ARM-visible RAM
# address; U-Boot 2026.01's lmb_reserve rejects it. Relocate first.
fdt move ${fdt_addr} ${fdt_addr_r}

# Third arg is the firmware-merged DTB, not the FIT-internal stub.
bootm ${kernel_addr_r} - ${fdt_addr_r}
run mender_try_to_recover

source, bootm, or load failing all route through mender_try_to_recover, which bumps bootcount and lets the standard Mender altbootcmd path roll back to the other slot. A bad new boot-<N>.fit cannot brick a factory-flashed unit.

U-Boot configuration

Functional knobs (apply to every build)

# CONFIG_FIT_SIGNATURE depends on CONFIG_FIT. Without the parent, Kconfig
# silently drops the child and bootm rejects FIT blobs at runtime.
CONFIG_FIT=y
CONFIG_FIT_SIGNATURE=y
CONFIG_FIT_SIGNATURE_MAX_SIZE=0x4000000

# ECDSA P-256 verification. CONFIG_ECDSA is the *parent* (depends on
# CONFIG_DM); CONFIG_ECDSA_VERIFY is the FIT-time verifier. Without
# the parent, Kconfig silently drops the child and bootm rejects
# signed FITs even though mkimage + host-side fit_check_sign happily
# produce them.
CONFIG_ECDSA=y
CONFIG_ECDSA_VERIFY=y

# bootm_process_cmdline_env runs cli_simple_process_macros to resolve
# ${var} in env "bootargs" at bootm time -- in-place, no Hush re-parse.
# Required so the dm-mod.create="..." double quotes survive to the
# kernel. Mender's expand_bootargs trick (Hush `run` on a stored setenv
# string) eats them.
CONFIG_BOOTARGS_SUBST=y

Hardening knobs

Largely derived from meta-raspberrypi-secure’s security-harden.cfg and redpesk’s trusted-boot reference, with Mender-specific environment flags added:

# -2 = no autoboot prompt at all and no abort check.
CONFIG_BOOTDELAY=-2
# Disable EFI / extlinux bootmeths. Our bootflow is boot.scr only;
# leaving these on means U-Boot iterates them first, spamming
# "Cannot persist EFI variables" / "Boot failed (err=-14)", and they
# also widen the parallel-bootflow attack surface.
CONFIG_BOOTMETH_EFILOADER=n
CONFIG_BOOTMETH_EFI_BOOTMGR=n
CONFIG_BOOTMETH_EXTLINUX=n
CONFIG_BOOTMETH_EXTLINUX_PXE=n
CONFIG_ENV_FLAGS_LIST_STATIC="bootcmd:sr,altbootcmd:sr,preboot:sr,bootargs:sr,boot_targets:sr,bootdelay:dr,bootlimit:dr,mender_setup:sr,mender_altbootcmd:sr,mender_try_to_recover:sr,mender_pre_setup_commands:sr,mender_post_setup_commands:sr"
CONFIG_BOOTM_EFI=n
CONFIG_BOOTM_NETBSD=n
CONFIG_BOOTM_OPENRTOS=n
CONFIG_BOOTM_OSE=n
CONFIG_BOOTM_PLAN9=n
CONFIG_BOOTM_RTEMS=n
CONFIG_BOOTM_VXWORKS=n
CONFIG_CMD_BOOTI=n
CONFIG_CMD_BOOTD=n
CONFIG_CMD_BOOTZ=n
CONFIG_CMD_ABOOTIMG=n
CONFIG_CMD_ADTIMG=n
CONFIG_CMD_BOOTEFI=n
CONFIG_CMD_BOOTEFI_BINARY=n
CONFIG_CMD_BOOTEFI_BOOTMGR=n
CONFIG_CMD_ELF=n
CONFIG_CMD_GO=n
CONFIG_CMD_NET=n
CONFIG_CMD_UBI=n
CONFIG_CMD_UBIFS=n
CONFIG_CMD_USB=n
CONFIG_CMD_SF=n
CONFIG_CMD_I2C=n
CONFIG_CMD_SPI=n
CONFIG_CMD_PCI=n
CONFIG_CMD_DIAG=n
CONFIG_CMD_MEMORY=n
CONFIG_CMD_IMI=n
CONFIG_CMD_SMC=n
CONFIG_CMD_HVC=n
CONFIG_CMD_EXT4=n
CONFIG_CMD_EEPROM=n
CONFIG_VIDEO=n

One non-obvious bit specific to RPi: meta-raspberrypi ships a maxsize.cfg that pins CONFIG_SYS_BOOTM_LEN=0x1000000 (16 MiB). For a ~25 MiB gzip-compressed kernel that’s too tight (“inflate() returned -5”). Drop the fragment:

SRC_URI:remove = " file://maxsize.cfg"

so the upstream ARM64 default (128 MiB) wins.

OTA: the custom rootfs-verity Mender Update Module

The build-time piece overrides IMAGE_CMD:mender to emit a module-image artifact (type rootfs-verity) carrying two payload files:

# signed-bootimg produces .boot.fit as a side effect; wait for it.
IMAGE_TYPEDEP:mender:append = " signed-bootimg"

IMAGE_CMD:mender() {
    ROOTFS_SRC="${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${ARTIFACTIMG_FSTYPE}.verity"
    BOOT_FIT_SRC="${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.boot.fit"

    # Pack payload files under known stable names. The rootfs basename
    # is derived from ARTIFACTIMG_FSTYPE so the on-target Update Module
    # can glob *.verity and stay fstype-agnostic.
    STAGE="${WORKDIR}/mender-rootfs-verity.stage"
    rm -rf "${STAGE}" && mkdir -p "${STAGE}"
    cp "${ROOTFS_SRC}"   "${STAGE}/rootfs.${ARTIFACTIMG_FSTYPE}.verity"
    cp "${BOOT_FIT_SRC}" "${STAGE}/boot.fit"

    extra_args=
    for dev in ${MENDER_DEVICE_TYPES_COMPATIBLE}; do
        extra_args="$extra_args -t $dev"
    done
    [ -n "${MENDER_ARTIFACT_SIGNING_KEY}" ] \
        && extra_args="$extra_args -k ${MENDER_ARTIFACT_SIGNING_KEY}"

    mender-artifact write module-image \
        -n ${MENDER_ARTIFACT_NAME} \
        -T rootfs-verity \
        $extra_args \
        -f "${STAGE}/rootfs.${ARTIFACTIMG_FSTYPE}.verity" \
        -f "${STAGE}/boot.fit" \
        ${MENDER_ARTIFACT_EXTRA_ARGS} \
        -o ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.mender
}
IMAGE_CMD:mender[vardepsexclude] += "IMAGE_ID"

The target-side Update Module is a shell script under /usr/share/mender/modules/v3/rootfs-verity. It’s a fork of stock rootfs-image with three intentional changes:

  1. Two payload files instead of one. The rootfs streams to the inactive block device; the boot FIT goes to /uboot/boot-<passive-slot>.fit.

  2. check_device_matches_root understands /dev/dm-*. The stock major/minor and findfs fallbacks both miss when / is on /dev/dm-0because they compare against the mapper device rather than the underlying block device it wraps. Walk/sys/block/dm-N/slaves/` first:

check_device_matches_root() {
    case "$1" in
        /dev/ubi*) ... ;;
        *)
            # Stock: match major/minor of $1 against /.
            if [ "$(stat -L -c %02t%02T "$1")" = "$(stat -L -c %04D /)" ]; then
                return 0
            fi

            # Added: if / is /dev/dm-*, the major/minor check above
            # compares against the mapper device, not the block device
            # it wraps. Walk the dm-N slaves.
            ROOT_MOUNTPOINT_DEV="$(mount | awk '$3 == "/" { print $1; exit }')"
            case "$ROOT_MOUNTPOINT_DEV" in
                /dev/dm-*)
                    for slave in "/sys/block/$(basename "$ROOT_MOUNTPOINT_DEV")/slaves/"*; do
                        [ -e "$slave" ] && [ "$1" = "/dev/$(basename "$slave")" ] && return 0
                    done
                    ;;
            esac

            # Stock fallback.
            ROOT_DEVICE="$(findfs "$(grep -o '\(^\| \)root=[^ ]*' /proc/cmdline | cut -d= -f2-)")"
            ;;
    esac
    [ "$1" = "$ROOT_DEVICE" ] && return 0
    echo "Mounted root ($ROOT_DEVICE) does not match boot loader environment ($1)!" 1>&2
    return 1
}
  1. DownloadWithFileSizes rather than Download, so the rootfs streams straight to the inactive block device (via mender-flash or cat > $passive) without staging the full image. The small boot.fit is buffered to the
    module’s $FILES dir and installed via plain cp to the already-mounted /uboot:
DownloadWithFileSizes)
    check_requirements

    boot_mnt="$(boot_fat_mountpoint)"
    [ -n "$boot_mnt" ] || { echo "$BOOT_FAT not mounted" 1>&2; exit 1; }

    while true; do
        line="$(cat stream-next)"
        [ -z "$line" ] && break
        file="$(echo "$line" | cut -d' ' -f1)"
        size="$(echo "$line" | cut -d' ' -f2)"
        base="$(basename "$file")"

        case "$base" in
            # Fstype-agnostic glob: matches rootfs.ext4.verity,
            # rootfs.erofs.verity, etc.
            *.verity)
                if [ "$MENDER_FLASH_AVAILABLE" = 1 ]; then
                    mender-flash --input-size "$size" --input "$file" --output "$passive"
                else
                    cat "$file" > "$passive"; sync
                fi
                ;;
            boot.fit)
                # /uboot is normally ro; remount rw for the write, then
                # back to ro. Shrinks the window where compromised
                # userspace can scribble on the boot partition.
                mount -o remount,rw "$boot_mnt"
                cat "$file" > "$boot_mnt/boot-$passive_num.fit"
                sync
                mount -o remount,ro "$boot_mnt"
                ;;
            *)
                echo "unexpected payload file '$base'" 1>&2; exit 1
                ;;
        esac
    done
    ;;

The recipe is one-line install:

SUMMARY = "Mender Update Module: rootfs + signed boot FIT installer"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"

SRC_URI = "file://rootfs-verity"
S = "${UNPACKDIR}"

MENDER_MODULE_DIR = "${datadir}/mender/modules/v3"

do_install() {
    install -d ${D}${MENDER_MODULE_DIR}
    install -m 0755 ${S}/rootfs-verity ${D}${MENDER_MODULE_DIR}/rootfs-verity
}

FILES:${PN} = "${MENDER_MODULE_DIR}/rootfs-verity"
RDEPENDS:${PN} = "libubootenv-bin util-linux-findfs jq"

Boot sequence

  1. VC ROM verifies boot.img against OTP-fused pubkey hash.
  2. U-Boot runs the signed boot.scr. source verifies the FIT signature against u-boot.dtb’s embedded pubkey.
  3. mender_setup sets mender_boot_part (= 2 or 3).
  4. load mmc 0:1 ${kernel_addr_r} boot-${mender_boot_part}.fit.
  5. source ${kernel_addr_r} — U-Boot verifies the FIT signature; the embedded Hush script runs and setenv bootargs '...' with the slot’s dm-verity cmdline. ${mender_kernel_root} is left as a literal placeholder (inside single quotes so Hush doesn’t expand at source-time).
  6. fdt move ${fdt_addr} ${fdt_addr_r} (lmb_reserve workaround).
  7. bootm ${kernel_addr_r} - ${fdt_addr_r} — verifies the same FIT signature (kernel subimage this time), extracts and decompresses the kernel, hands off.
  8. bootm_process_cmdline_env runs cli_simple_process_macros (via CONFIG_BOOTARGS_SUBST), resolving ${mender_kernel_root} in env bootargs in-place — preserving the embedded dm-mod.create="..." double quotes.
  9. Kernel boots. CONFIG_DM_INIT parses dm-mod.create=, waits for /dev/mmcblk0pN via dm-mod.waitfor=, creates /dev/dm-0.

What this design buys you (and what it doesn’t)

Things this chain prevents, on a fused unit:

Vector Outcome
Modified boot.img (config.txt / cmdline / U-Boot / base DTB) VC ROM rejects RSA-2048 signature, board halts
Modified boot.scr on FAT U-Boot source verifies ECDSA against control-DTB pubkey, refuses to run
Modified boot-<N>.fit on FAT source + bootm both verify the FIT config signature; mismatch routes to mender_try_to_recover
Modified rootfs blocks on mmcblk0pX dm-verity hash tree authenticates each block; mismatched block returns EIO
Tampered Mender OTA artifact Mender verifies signing key before applying
Userspace fw_setenv bootcmd=... (or other bootflow env vars) :sr env flags reject the write
A/B swap to load an older signed rootfs Mender state machine guards mender_boot_part transitions; rollback only honours the active slot
Half-written rootfs after power loss during OTA dm-verity blocks read of corrupt sectors; bootcount triggers rollback to the still-good slot

Hi @lukehatpadl,

Thanks a lot for sharing!

Greetz,
Josef