Mender + dm-verity integration

General Discussions
, ,
1

Hi,

We are improving the security of a system based on a Toradex Verdin iMX8M Mini module.

We have been successfully using mender in this system for a long time previously. But now, as part of the security hardening we have integrated dm-verity to avoid modifications of the rootfs.

Our problem is the device-mapper used by dm-verity, regardless of the physical device being used, mounts the rootfs from device /dev/dm-0 and this confuses the mender-client, as expected.

This is the result of mount:

/dev/dm-0 on / type ext4 (ro,relatime)

And this is the result of the command ā€œstat /ā€

stat /
File: /
Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: 253,0	Inode: 2           Links: 21
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-06 00:21:16.000000000 +0100
Modify: 2024-01-06 00:21:16.000000000 +0100
Change: 2024-01-06 00:21:16.000000000 +0100
 Birth: 2024-01-06 00:21:16.000000000 +0100

This is the log output from mender when I try to install the artifact:

root@XXXX-00005003:~# mender install /tmp/usb/XXXX-dev-01.00-20240107215702.mender
INFO[0000] Loaded configuration file: /var/lib/mender/mender.conf 
INFO[0000] Loaded configuration file: /etc/mender/mender.conf 
INFO[0000] 'UpdateControlMapExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 2*UpdatePollIntervalSeconds 
INFO[0000] 'UpdateControlMapBootExpirationTimeSeconds' is not set in the Mender configuration file. Falling back to the default of 600 seconds 
INFO[0000] Mender running on partition: /dev/dm-0       
INFO[0001] Mender running on partition: /dev/dm-0       
INFO[0001] Start updating from local image file: [/tmp/usb/XXXX-dev-01.00-20240107215702.mender] 
Installing Artifact of size 394884608...
INFO[0001] No public key was provided for authenticating the artifact 
ERRO[0001] Download failed: Payload: can not install Payload: XXXX-dev-01.00-verdin-imx8mm.ext4: Active root partition matches neither RootfsPartA nor RootfsPartB. 
ERRO[0001] Payload: can not install Payload: XXXX-dev-01.00-verdin-imx8mm.ext4: Active root partition matches neither RootfsPartA nor RootfsPartB.

As you can see, it detects /dev/dm-0 and obviously it doesnā€™t match RootfsPartA neither RootfsPartB that are defined as /dev/mmcblk0p2 and /dev/mmcblk0p3.

Using commands like dmsetup is possible to figure out the physical physical device attached to the dm device:

root@XXXX-00005003:~# dmsetup deps -o devname
verity: 1 dependencies  : (mmcblk0p2)

My question is: What would you consider the best approach to update the mender client so it notices the device is provided by device-mapper and tries to look ā€œdeeperā€ to find the real device? Is there a standard way that we could use? or we have to simply patch mender-client?

We would be more than happy to contribute the patch to the community so this is why I ask for ā€œbest approachā€.

2

Hi @Globulo,

Hmm, good question. Just guessing would be a custom fork of the rootfs update module. Since the new C+Ā±Client, this is a separate Update Module anyways, so modifying if for that use case should not be super complicated.

Please see mender/support/modules/rootfs-image at master Ā· mendersoftware/mender Ā· GitHub

Greets,
Josef

3

Hi,

Iā€™ve reviewed the current code of our mender client in Go (3.5.1 runtime: go1.17.13) and I can more or less see how it resolves the rootfs, etc. Iā€™ve added an ugly patch as a PoC to solve the situation. We could live with it but Iā€™m interested in your suggestion.

So I have some questions:

  • Is the new c++ client already ā€œproduction readyā€? My understanding is, its still an alpha.
  • What you had shown me is a shellscript. Is the new client handling the whole rootfs update as an external module (shellscript) as opossed to current client doing it from GO code?.
  • Is it possible to update the rootfs as a shellscript-module in current GO client?.

Also as a side note: Sounds strange to me that no one came with similar problem.

In the end, afaik, there are not much other alternatives when it comes to hardening an embedded system, might be we are going the wrong direction trying to integrate mender+dm-verity? How are other users of mender hardening their rootfs?

Thanks for the support.

4

Hi @Globulo

No, I think the approach is correct. Itā€™s not just integrity checking one needs for hardening but the whole bootchain needs to be carefully reviewed, like u-boot dropping into a shell, etc.

For your questions:

  • the C+Ā±client is pretty close to release. Weā€˜re ironing out the last glitches by now, but I would consider it RC-quality. For something that is currently in R&D, it is definitely my go-to suggestion.
  • Yes, Iā€˜ve shown you an Update Module in Shellscript. The new client will handle the whole rootfs update as an external module, not internally anymore. That is correct.
  • You can definitely handle a rootfs update as a module on the Go client too. This will need a different artifact type string for identification then as far as I can see, so it is not as streamlined.

For a PoC on the Go client, let me put it like that: it would be interesting to have it visible for others that face the same problem and cannot upgrade the client for whatever reason, but I donā€˜t see any immediate way for such to be implemented in the mainline client, sorry.

Greetz,
Josef

5

Yes, of course, we have the full pack of complementary features: Signed + hardened u-boot/kernel, etc. Signed roothashes for dm-verity and all the bells and whistles.

We are currently updating the required hashes and signatures in u-boot-env by simply using a state script now.

The problem Iā€™ve mentioned was just the last obstacle to finish the mender integration into the hardened system.

Cool, so Iā€™m going to integrate it in our current build and see how it goes.

No need. Decision is to try the c++ client. The fact that a script is used as a module for rootfs update sounds more attractive as we could change some other details. Definitely more granularity and flexibility.

Question: Is there some specific documentation for this new approach?

Again, thanks for the support.

1 Like