We recently decided to try paid Mender Professional for our company and we are in the process of testing it for our needs. As we will use Mender on Raspberry Pi 3 and its microSD card, one issue we wanted to fully cover is filesystem corruption which is not often but it is still possible. So far with read-only filesystem and dual partitions we tackled this pretty well.
We use the latest Mender (we need the diff updates at the end but we test rootfs-image) on a custom Yocto warrior build.
After initiating a cloud rootfs-image deployment, we tricked filesystem to be deployed to not being able to boot with 2 methods (see below) and the results were the same with 2 serious issues observed from our end.
Pi rolls-back successfully but “mender show-artifact” reports the new artifact (which never got to boot) and committs that to Mender Cloud server and changing Deployment status to “SUCCESS” giving a totally wrong impression if done remotely.
The 2 methods used are:
- While cloud server pushes the download and mender client writes it to /dev/mmcblk0p3 we run this to mess up the inactive partition (p3)
dd if=/dev/random of=/dev/mmcblk0p3
- Built a custom artifact without
/sbin/initthat should fail to boot
We also tried the above on an unmodified core-image-base for MACHINE=raspberrypi3 with the same results too.
From our point of view we see 2 potential problems here.
- The deployment is only hashed on download before written to disk (p3) “assuming” trying to write it actually writes it intact. This assumption leaves a huge security hole if attacker with access can modify filesystem before the reboot (maybe to bypass a possible upcoming security fix update). Even on signed deployments!
- “mender show-artifact” reports a value which is not “realtime” but stored beforehand hoping it will be valid when needed
Has anyone tried this case before?