# Robustness of deployments **URL:** \ **Category:** General Discussions\ **Created:** [October 9, 2020, 12:58pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645 "2020-10-09T12:58:39Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 9, 2020, 12:58pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/1 "2020-10-09T12:58:39Z")
We recently decided to try paid Mender Professional for our company and we are in the process of testing it for our needs. As we will use Mender on Raspberry Pi 3 and its microSD card, one issue we wanted to fully cover is filesystem corruption which is not often but it is still possible. So far with read-only filesystem and dual partitions we tackled this pretty well. We use the latest Mender (we need the diff updates at the end but we test rootfs-image) on a custom Yocto warrior build. After initiating a cloud rootfs-image deployment, we tricked filesystem to be deployed to not being able to boot with 2 methods (see below) and the results were the same with 2 serious issues observed from our end. Pi rolls-back successfully but “mender show-artifact” reports the new artifact (which never got to boot) and committs that to Mender Cloud server and changing Deployment status to “SUCCESS” giving a totally wrong impression if done remotely. The 2 methods used are: 1. While cloud server pushes the download and mender client writes it to /dev/mmcblk0p3 we run this to mess up the inactive partition (p3) `dd if=/dev/random of=/dev/mmcblk0p3` 2. Built a custom artifact without `/sbin/init` that should fail to boot We also tried the above on an unmodified core-image-base for MACHINE=raspberrypi3 with the same results too. From our point of view we see 2 potential problems here. 1. The deployment is only hashed on download before written to disk (p3) “assuming” trying to write it actually writes it intact. This assumption leaves a huge security hole if attacker with access can modify filesystem before the reboot (maybe to bypass a possible upcoming security fix update). Even on signed deployments! 2. “mender show-artifact” reports a value which is not “realtime” but stored beforehand hoping it will be valid when needed Has anyone tried this case before? ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [October 9, 2020, 1:19pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/2 "2020-10-09T13:19:22Z")
Hi @bender > 1. The deployment is only hashed on download before written to disk (p3) “assuming” trying to write it actually writes it intact. This assumption leaves a huge security hole if attacker with access can modify filesystem before the reboot (maybe to bypass a possible upcoming security fix update). Even on signed deployments! Mender is not intending to protect against all attack scenarios. The Mender signature guarantees that the image that is downloaded and install is properly signed but we cannot guarantee that at some later time it hasn’t been changed. You would need some kind of Secure-Boot setup to protect against this kind of attack. > 1. “mender show-artifact” reports a value which is not “realtime” but stored beforehand hoping it will be valid when needed This one is strange. I’ve been unable to reproduce it with either of the scenarios you mentioned. In all cases my device rolls back and shows the proper artifact name. @kacf @lluiscampos @oleorhagen do you guys have any suggestions how to proceed troubleshooting this? Drew ---
**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\ **Post date:** [October 9, 2020, 1:32pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/3 "2020-10-09T13:32:34Z")
I thought with respects to point 2, this couldn’t happen as the meta-data that is referenced for the mender show-artifact command was stored on their respective A/B partitions unless this has changed recently? ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 9, 2020, 1:51pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/4 "2020-10-09T13:51:49Z")
How can I check this? ---
**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\ **Post date:** [October 9, 2020, 3:46pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/5 "2020-10-09T15:46:55Z")
In older versions of the client i believe it to be stored in /etc/mender/artifact\_info however from other discussions i believe this to be deprecated in newer versions of the client to use a lmdb database. I don’t know where that is stored as i have no devices on a newer enough version of mender to be able to check this. Just speculating but If the lmdb database is now stored on the /data partition then that would be where i would start looking as its now a shared resource and you may of uncovered a bug. ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 9, 2020, 10:24pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/6 "2020-10-09T22:24:05Z")
isn’t this in `/data/mender/mender-store` ? Although is not to be human readable, I tried a simple `cat` on it and I could see the new artifact name before AND after the unsuccessful update. ---
**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\ **Post date:** [October 9, 2020, 11:59pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/7 "2020-10-09T23:59:12Z")
looks like you have found its location. > [@Purpose of "/data/mender/mender-store" and "/data/mender/mender-store-lock"?](https://hub.mender.io/t/purpose-of-data-mender-mender-store-and-data-mender-mender-store-lock/602): > > I would like to know why "/data/mender/mender-store" and "/data/mender/mender-store-lock" are required? I couldn’t able to get more information on these files. thinking As to how its able to able to set the wrong values in the shared database under your testing conditions, i will have to defer to the mender-team as i don’t use a version that has a database. ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 10, 2020, 1:51pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/8 "2020-10-10T13:51:13Z")
is there a way to properly query this db or simply parse all values? ---
**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\ **Post date:** [October 10, 2020, 2:46pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/9 "2020-10-10T14:46:45Z")
i havent use these, but the source code for lmdb-utils for working with a database seems to be here > **[LMDB/lmdb](https://github.com/LMDB/lmdb/tree/mdb.master/libraries/liblmdb)** > > Read-only mirror of official repo on openldap.org. Issues and pull requests here are ignored. Use OpenLDAP ITS for issues. - LMDB/lmdb > **[LMDB: LMDB Command Line Tools](http://www.lmdb.tech/doc/tools.html)** ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [October 12, 2020, 6:21am UTC](https://hub.mender.io/t/robustness-of-deployments/2645/10 "2020-10-12T06:21:32Z")
> [@bender](#): > > The deployment is only hashed on download before written to disk (p3) “assuming” trying to write it actually writes it intact. This assumption leaves a huge security hole if attacker with access can modify filesystem before the reboot (maybe to bypass a possible upcoming security fix update). Even on signed deployments! An attacker could just as easily do this while the filesystem was being read back to verify the checksum, so reading it back provides no extra security. At best it provides a _slight_ protection against corruption due to hardware failure, but given the huge performance hit, I do not believe this is worthwhile to do. > [@bender](#): > > isn’t this in > `/data/mender/mender-store` > ? > Although is not to be human readable, I tried a simple `cat` on it and I could see the new artifact name before AND after the unsuccessful update. Yes, this is correct, the database stores the current name, and the upcoming name (how else would it remember what the upcoming name is going to be?). However `mender show-artifact` will always show the current name. If `mender show-artifact` does not produce the correct value, then please share the exact steps to reproduce this. ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 12, 2020, 6:50am UTC](https://hub.mender.io/t/robustness-of-deployments/2645/11 "2020-10-12T06:50:24Z")
> [@kacf](#): > > If `mender show-artifact` does not produce the correct value, then please share the exact steps to reproduce this. > > > [@bender](#): > > > > After initiating a cloud rootfs-image deployment, we tricked filesystem to be deployed to not being able to boot with 2 methods (see below) and the results were the same with 2 serious issues observed from our end. > > Pi rolls-back successfully but “mender show-artifact” reports the new artifact (which never got to boot) and committs that to Mender Cloud server and changing Deployment status to “SUCCESS” giving a totally wrong impression if done remotely. > > > > The 2 methods used are: > > > > 1. While cloud server pushes the download and mender client writes it to /dev/mmcblk0p3 we run this to mess up the inactive partition (p3) > > `dd if=/dev/random of=/dev/mmcblk0p3` > > 2. Built a custom artifact without `/sbin/init` that should fail to boot Could you try that? ---
**Author:** ![kacf](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/kacf/32/146_2.png) [@kacf](https://hub.mender.io/u/kacf)\ **Post date:** [October 12, 2020, 7:28am UTC](https://hub.mender.io/t/robustness-of-deployments/2645/12 "2020-10-12T07:28:33Z")
> [@bender](#): > > While cloud server pushes the download and mender client writes it to /dev/mmcblk0p3 we run this to mess up the inactive partition (p3) > `dd if=/dev/random of=/dev/mmcblk0p3` There is nothing Mender can do to prevent you from writing to the filesystem if you have root access. The only remedy I can think of here is to limit the number of processes that have root access to a minimum. > [@bender](#): > > Built a custom artifact without `/sbin/init` that should fail to boot I cannot reproduce your issue, for me it rolls back just fine and reverts to the previous image. What messages do you get on the console when doing this? And which Mender client version are you using? ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 12, 2020, 7:58am UTC](https://hub.mender.io/t/robustness-of-deployments/2645/13 "2020-10-12T07:58:32Z")
> [@kacf](#): > > There is nothing Mender can do to prevent you from writing to the filesystem if you have root access. Our stress test is mainly focusing on SD card corruption which is not very easy to emulate otherwise. > [@kacf](#): > > I cannot reproduce your issue, for me it rolls back just fine I am waiting for @drewmoseley to provide a rootfs image disk to flash and a test release to retry on my end > [@kacf](#): > > And which Mender client version are you using? root@pi5798:~ # mender --version 2.3.0-dirty runtime: go1.12.9 ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 12, 2020, 1:27pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/14 "2020-10-12T13:27:16Z")
> [@dellgreen](#): > > If the lmdb database is now stored on the /data partition then that would be where i would start looking as its now a shared resource and you may of uncovered a bug. As /data partition is indeed shared, storing variables in db (`/data/mender/mender-store`) that should be relative to active/inactive partitions should not work unless they are absolutely linked to each partition addressed directly (`/dev/mmcblk0p2` or `/dev/mmcblk0p3`). Could that be the root of the issue here? ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [October 14, 2020, 1:52pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/15 "2020-10-14T13:52:01Z")
@bender, since your tests with my image did not exhibit the issue, let’s see if we can figure out what your image is doing differently that would interfere with the rollback in this case. Do you have any scripting that invokes the mender binary directly or that manipulates the systemd service for Mender? Can you share any state scripts that you have implemented? Do you have any scripting in either U-Boot or Linux that manipulates the U-Boot environment? Can you share the details? Drew ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [October 14, 2020, 2:53pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/16 "2020-10-14T14:53:08Z")
For others who may be able to contribute, the build I provided to @bender was straight from the [Mender hub post](https://hub.mender.io/t/raspberry-pi-3-model-b-b/57/29). Configured to connect to hosted Mender. No other customizations, state scripts, etc. ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 14, 2020, 3:58pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/17 "2020-10-14T15:58:22Z")
> [@drewmoseley](#): > > Do you have any scripting that invokes the mender binary directly or that manipulates the systemd service for Mender? Nope > [@drewmoseley](#): > > Can you share any state scripts that you have implemented? No state scripts are implemented (yet). > [@drewmoseley](#): > > Do you have any scripting in either U-Boot or Linux that manipulates the U-Boot environment? No Any advice is welcome as we don’t know where else to look to spot the difference. ---
**Author:** ![drewmoseley](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/drewmoseley/32/47_2.png) [@drewmoseley](https://hub.mender.io/u/drewmoseley)\ **Post date:** [October 14, 2020, 4:18pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/18 "2020-10-14T16:18:34Z")
At the moment I’m at a loss. Can you provide the serial console output from the board through the course of the deployment and the associated reboots? ---
**Author:** ![dellgreen](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/dellgreen/32/85_2.png) [@dellgreen](https://hub.mender.io/u/dellgreen)\ **Post date:** [October 14, 2020, 8:05pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/19 "2020-10-14T20:05:58Z")
Did you ever get around to dumping the contents of the lmdb database to inspect it? ---
**Author:** ![bender](https://yyz2.discourse-cdn.com/flex036/user_avatar/hub.mender.io/bender/32/847_2.png) [@bender](https://hub.mender.io/u/bender)\ **Post date:** [October 15, 2020, 2:53pm UTC](https://hub.mender.io/t/robustness-of-deployments/2645/20 "2020-10-15T14:53:55Z")
Copied mender-store over to my machine and installed my distro’s ready package (did not build from source) ``` $ mdb_load -f mender-store mender-test mdb_load: line 1: unexpected format ``` Any hints? There was a mender-store.lock file present in the Pi too. [Next page](https://hub.mender.io/t/robustness-of-deployments/2645.md?page=2)